AI item assistance management method combining ar and semantic understanding

Abstract

The embodiment of the application relates to an AI article auxiliary management method combining AR and semantic understanding, comprising the following steps: constructing an environment semantic map comprising article semantics and safety attributes; when a user gazes at or approaches an article, actively displaying relevant information through an AR interface; through an intention understanding model specially trained by a disordered language corpus, combining with environment context to analyze a user voice request, determining a unique target; according to a user intention, a target state and a historical operation ability portrait, generating a personalized atomic operation sequence matched with real-time cognitive load; through a spatial audio and an AR visual channel registered in a real space, cooperatively outputting a progressive guide, and only revealing a next step guide after a previous step is completed; through comparison of action deviation and analysis of physiological data, real-time evaluation of a user state and triggering of hierarchical intervention are realized. The application realizes deep environment understanding, robust intention analysis and adaptive personalized guidance, and effectively improves patient self-care ability and safety.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an artificial intelligence (AI) item-assisted management method that combines augmented reality (AR) and semantic understanding. Background Technology

[0002] Alzheimer's disease (AD) patients often experience symptoms such as memory loss, spatial disorientation, and decreased executive function, making it difficult for them to independently complete basic tasks such as finding, putting away, and organizing items in daily life. This not only reduces their quality of life but also poses safety risks (such as accidental ingestion of medication or contact with dangerous items).

[0003] Currently, some smart home or assistive devices have attempted to solve some of these problems, for example: Item finders help users locate lost items by attaching Bluetooth tags. However, this method requires pre-labeling, cannot identify new or unlabeled items, and does not provide user guidance.

[0004] General-purpose voice assistants can respond to simple commands, but their semantic understanding models are trained for normal language and struggle to interpret the vague, disjointed, and illogical speech requests commonly encountered by patients with cognitive impairments. Furthermore, they lack the ability to perceive the physical environment and cannot associate commands with specific objects and spatial locations in the real world.

[0005] Simple AR guidance applications can display arrows or icons on the screen, but they are usually based on preset paths or fixed labels, lacking an understanding of dynamic environments, unable to adapt to complex item sorting tasks, and lacking the ability to adjust guidance strategies according to the user's real-time status.

[0006] Therefore, existing technologies lack a comprehensive assistance solution that can deeply understand user intentions, accurately perceive the physical environment, and adapt to changes in user cognitive state. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing an AI-assisted item management method that combines AR and semantic understanding, thereby solving the problems existing in the prior art.

[0008] To achieve the above objectives, this invention provides an AI-assisted item management method combining AR and semantic understanding, the method comprising: Construct and update an environmental semantic map in real time. The environmental semantic map consists of multiple semantic nodes, each of which corresponds to a real item and includes at least the semantic tag and security attributes of the item. When a user looks at or approaches the item corresponding to the semantic node, the basic information of the item is displayed through the AR interface. The system receives the user's voice request, uses an intent understanding model trained on a corpus of language disordered by cognitive impairment, and combines the environmental semantic map to perform context fusion reasoning, parsing out the actionable user intent and identifying a unique target object. Based on the user's intent, the real-time status of the target object, and the user's historical operational capability profile, an executable task sequence comprising multiple atomic operations is generated; wherein, the task complexity and decomposition granularity of the executable task sequence are matched with the user's current cognitive load; Guidance information is output through a combination of spatial audio and AR visual channels registered in the real space; the visual guidance adopts a progressive revelation strategy, dynamically generating the next step guidance only after the user completes the previous step. Throughout the task execution, the user's state level is assessed in real time by comparing the deviation between the user's actions and the expected sequence, as well as physiological sensor data, and multi-level auxiliary interventions are triggered in accordance with the state level; wherein, the state level includes at least confusion, forgetfulness or anxiety.

[0009] In one possible implementation, the construction and real-time updating of the environmental semantic map specifically includes: The system uses an integrated RGB-D camera and inertial measurement unit to simultaneously acquire color images and depth images of the environment, as well as inertial data of the equipment. Based on visual inertial odometry and semantic SLAM algorithms, the color images and inertial data are processed to construct and update a three-dimensional geometric map of the environment in real time, and output the real-time six-degree-of-freedom pose of the device in the world coordinate system. The color image is analyzed frame by frame using an instance segmentation neural network to identify each item instance and output the category label and pixel-level segmentation mask for each instance. The segmentation mask of each instance is pixel-level aligned and fused with the synchronously acquired depth image; combined with the camera intrinsic parameter matrix and the real-time six-degree-of-freedom pose, the pixels in the mask are back-projected into three-dimensional space to generate a three-dimensional point cloud cluster for each item instance. Spatial clustering and geometric analysis are performed on the 3D point cloud clusters of each item instance to calculate the 3D geometric information of the item, which includes: 3D axial bounding box, centroid coordinates, main orientation and occupied spatial volume; The visual feature vector of each item instance is extracted, and the visual feature vector, together with the category label and three-dimensional geometric information, is matched and retrieved with a pre-set personalized item knowledge graph to obtain the matching result; the personalized item knowledge graph stores the general semantic attributes of the item, the security level classification, and the user's custom semantic tags or memory anchors for the item. Based on the matching results, each item instance is attached with a corresponding semantic tag and security attribute to form a semantic node; wherein, the semantic tag includes at least the item's standard name and functional category, and the security attribute includes at least a risk level identifier; An environmental semantic map is formed based on the spatial relationship of the centroid coordinates of each semantic node in the world coordinate system. As new data is collected, the confidence and location information of each semantic node are updated in real time through multi-view geometric consistency test and Bayesian filtering method.

[0010] In one possible implementation, displaying basic information about the object via an AR interface when the user looks at or approaches the object specifically includes: The user's three-dimensional gaze vector is obtained by eye-tracking technology. The spatial intersection of the three-dimensional gaze vector with the three-dimensional bounding box of each semantic node in the environmental semantic map is calculated. If the gaze continues for more than a first time threshold, it is determined to be a gaze. The Euclidean distance between the user's location and the centroid of each semantic node is calculated in real time. If the distance is less than the preset second distance threshold, the user is considered close. When a gaze or proximity is detected, query the complete semantic tag and security attributes corresponding to the semantic node. Based on the current task context, user cognitive load, and item safety attributes, key basic information is dynamically filtered out using a priority algorithm. The priority algorithm ensures at least that: high-risk items are given priority in displaying warning messages, and items strongly related to the current task are given priority in displaying task instructions. The selected basic information is overlaid and displayed near the real spatial location of the corresponding item using an AR rendering engine in a spatially locked manner. The information display has a delayed trigger and automatic fade-out mechanism, and the display of information on irrelevant items is suppressed when the user is performing the main task.

[0011] In one possible implementation, receiving a user's voice request involves using an intent understanding model trained on a corpus of language disordered by cognitive impairment, and combining this with the environmental semantic map to perform contextual fusion reasoning, parsing the actionable user intent, and identifying a unique target object. Specifically, this includes: The collected user voice data is denoised and then converted into text using a speech recognition model trained on pathological speech corpus. The text is input into an intent understanding model fine-tuned with a cognitive impairment dialogue corpus to obtain preliminary analysis results including intent categories and semantic slots that may have ambiguous referents. Construct a dynamic context vector, whose data includes: the semantic node corresponding to the user's current gaze focus, a list of semantic nodes in the neighborhood space, the current task status, and short-term interaction history; The preliminary analysis results and the dynamic context vector are input into the inference engine. By calculating the correlation between fuzzy references and each candidate semantic node in the semantic, spatial, and temporal dimensions, the fuzzy references are disambiguated and anchored to a specific semantic node, thereby determining a unique target object. When the reasoning confidence is below the threshold, multiple candidate semantic nodes are labeled through the AR interface and a clarification question is initiated in conjunction with voice.

[0012] In one possible implementation, generating the executable task sequence specifically includes: Retrieve a task template corresponding to the user intent from a hierarchical task knowledge base. The template consists of multiple atomic operations with a logical order. Based on the user's historical operational ability profile, a predicted difficulty score is added to each atomic operation in the task template; Real-time assessment of the user's current cognitive load index; Based on the current cognitive load index, the atomic operation sequence is dynamically optimized: when the load index is low, low-difficulty operations are merged to simplify the sequence; when the load index is high, high-difficulty operations are finely decomposed; and confirmation checkpoints are dynamically inserted according to the user's error pattern. Output an optimized sequence of executable tasks and replan online based on the user's actual performance during execution.

[0013] In one possible implementation, the step of collaboratively outputting guidance information via spatial audio and AR visual channels specifically includes: Generate a multimodal guidance instruction set for each atomic operation, including spatial audio scripts and AR visual parameters; The AR glasses internally run a finite state machine synchronized with the sequence of executable tasks; The state machine only transitions to the next state and triggers the rendering engine to output guidance information for the next step when the sensor data verifies that the user has completed the current atomic operation. Spatial audio provides sound guidance from the direction of the target, while AR visual elements are precisely registered at the target location in the real space.

[0014] In one possible implementation, the real-time assessment of the user's state level and triggering of multi-level auxiliary intervention specifically includes: Real-time acquisition of deviation data between user actions and expected sequences, as well as physiological response data; The data is input into the state assessment model, which outputs a comprehensive state level, which includes at least: mild hesitation, clear forgetting or confusion, significant anxiety or persistent errors; Interventions are triggered based on the status level; these interventions include Level 1, Level 2, and Level 3 interventions; Level 1 interventions provide non-intrusive prompts; Level 2 interventions offer concise step-by-step prompts and voice repetitions; Level 3 interventions pause the task and initiate structured dialogue guidance. Record the intervention effect; the intervention effect is used to optimize the assessment and intervention strategy for this user.

[0015] In one possible implementation, the method further includes: Using the user's historical operation data and physiological data, the intent understanding model, task planning logic, and state evaluation model are periodically incrementally trained and fine-tuned, and the optimized parameters are updated to the user device.

[0016] In one possible implementation, the method operates in a client-server collaborative architecture: The client device is responsible for sensor data acquisition, AR rendering, and real-time guidance; the server is responsible for running the intent understanding model, maintaining the knowledge graph and user profile, and performing complex task planning. The client and server exchange data and synchronize commands through an encrypted channel.

[0017] In one possible implementation, the method further includes: Through the collaborative management interface, instructions from family member terminals are received and executed, including: remotely viewing a simplified view of the environmental semantic map, user task status, and activity reports; pre-associating or recording personalized voice prompts for specific semantic nodes; and... When a high-risk warning is received from the system or a user initiates a request for help, the remote real-time assistance mode is activated, so that AR annotations or voice guidance initiated by family members can be superimposed and presented on the user's AR interface in real time.

[0018] By applying the AI-assisted item management method that combines AR and semantic understanding provided in this embodiment of the invention, an environmental semantic map that integrates geometric, semantic, and security attributes is constructed and associated with a personalized item knowledge graph. This lays a cognitive foundation for all subsequent precise interactions, far exceeding shallow perception solutions based on Bluetooth tags or fixed tags.

[0019] A specially trained intent understanding model tailored to the language characteristics of patients with cognitive impairment, combined with multimodal contextual fusion reasoning, can effectively parse and disambiguate ambiguous and disjointed speech commands. Proactive, non-invasive AR information cues and multi-sensory collaborative spatial guidance align with the user's natural attention mechanism, significantly reducing the learning and operational burden and resolving the failure issues of general voice assistants and simple AR applications in this scenario.

[0020] Based on historical operational ability profiles and real-time cognitive load assessments, the system dynamically adjusts task decomposition granularity and guidance strategies to achieve personalized adaptation for each individual. Through real-time monitoring of behavioral and physiological signal status assessments and a tiered intervention mechanism, the system can provide timely assistance when users encounter difficulties, forming an intelligent closed loop of perception-planning-guidance-monitoring-adjustment, effectively improving the timeliness, safety, and user independence of assistance.

[0021] The client-server architecture balances computational load and real-time requirements, ensuring the system's usability. The introduction of family collaborative management features not only enables remote monitoring and safety alerts, but also seamlessly integrates emotional support from family members into the assistance process through personalized memory prompts and real-time remote assistance, enhancing the system's social acceptance and sense of warmth. Attached Figure Description

[0022] Figure 1 Flowchart of the AI-assisted item management method combining AR and semantic understanding provided by the present invention; Figure 2 This is a structural diagram of AR glasses; Figure 3 for Figure 1 Flowchart for step 110; Figure 4 for Figure 1 Flowchart for step 120; Figure 5 for Figure 1 Flowchart for step 130; Figure 6 for Figure 1 Flowchart for step 140; Figure 7 for Figure 1 Flowchart for step 150; Figure 8 for Figure 1 Flowchart for step 160; Figure 9 This is a schematic diagram of the structure of the segmentation neural network in this invention. Figure 10 This is a schematic diagram of the speech recognition model of the present invention; Figure 11This is a schematic diagram of the structure of the inference engine for disambiguation in this invention. Figure 12 This is a schematic diagram illustrating the structure of the model intended for understanding the present invention; Figure 13 This is a schematic diagram of the cognitive load assessment model of the present invention; Figure 14 This is a schematic diagram of the state assessment model of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0024] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0025] Figure 1 The present invention provides an AI-assisted item management method combining AR and semantic understanding. This method can be applied to augmented reality devices, such as AR glasses. Figure 2 As shown, the AR glasses include a main control module, an RGB-D camera, an inertial measurement unit (IMU), an eye-tracking camera, a heart rate sensor, a microphone array, a processor and memory, a display module, a speaker / bone conduction headphones, a communication module, an instance segmentation neural network, an automatic speech recognition module, an intent understanding model, an inference engine, a historical operation ability profile model, a cognitive load assessment model, and a state assessment model. The memory contains a pre-set personalized item knowledge graph.

[0026] An instance segmentation neural network analyzes RGB images frame by frame, identifying and segmenting each item instance, outputting category labels and segmentation masks. An automatic speech recognition model converts user speech into text. An intent understanding model performs natural language understanding on the text output by ASR, performing a joint task of intent classification and slot filling, outputting structured parsing results, which may include ambiguous referencing. The inference engine receives the preliminary parsing results and dynamic context vectors from the intent understanding model, and performs referencing disambiguation by calculating semantic, spatial, and temporal correlations to determine a unique target object. A historical operational ability profiling model, based on the user's long-term operational history and physiological data, uses machine learning for temporal modeling to form a dynamic model quantifying the user's capabilities. A cognitive load assessment model integrates user behavior flow anomalies and physiological signal deviations in real time to calculate the user's current cognitive load index. A state assessment model integrates action-temporal deviation data and physiological reaction data to output the user's comprehensive state level, such as mild hesitation, confusion, or anxiety.

[0027] An RGB-D camera is used to simultaneously acquire color and depth images. An inertial measurement unit (IMU) is used to acquire motion data such as acceleration and angular velocity. An eye-tracking camera is used to track the user's gaze direction in real time. A heart rate sensor is used to monitor the user's heart rate and heart rate variability in real time. A microphone array is used to directionally acquire the user's voice and perform beamforming noise reduction. A processor and memory are used to run local SLAM, AR rendering, sensor fusion, and lightweight inference. A display module is used to present AR visual information to the user. A speaker / bone conduction headset is used to play spatial audio guidance.

[0028] Now, taking the main control module as the execution entity, this application's AI-assisted item management method combining AR and semantic understanding will be explained in detail. For example... Figure 1 As shown, the method includes: Step 110: Construct and update the environmental semantic map in real time. The environmental semantic map consists of multiple semantic nodes, each of which corresponds to a real item and includes at least the semantic tag and security attributes of the item.

[0029] Semantic tags include, for example, the name and function of an item. Security attributes include, for example, the security level of an item. Figure 3 As shown, step 110 includes the following: Step 1101: Simultaneously acquire color images and depth images of the environment and inertial data of the device through the RGB-D camera and IMU integrated in the hardware device; Specifically, the RGB-D camera integrated into the AR glasses simultaneously captures two key types of data: color images and depth images. Color images provide rich texture and color information for visual recognition. Depth images record the physical distance from each pixel in the scene to the camera. Meanwhile, the IMU provides inertial data such as the device's acceleration and angular velocity in real time. These three types of data are time-stamped to ensure that visual information and spatial location information at the same moment can accurately correspond, laying the foundation for subsequent fusion processing.

[0030] Step 1102: Based on visual inertial odometry and semantic SLAM algorithm, process the color image and inertial data, construct and update the three-dimensional geometric map of the environment in real time, and output the real-time six-degree-of-freedom pose of the device in the world coordinate system. Specifically, the Visual-Inertial Odometry (VIO) algorithm fuses motion information from visual feature points in RGB-D and inertial data from the IMU to estimate the device's six-degree-of-freedom pose, i.e., its position and orientation in three-dimensional space. Motion information from visual feature points refers to the motion patterns and geometric relationships inferred by tracking the positional changes (e.g., pixel displacements) of identical visual feature points such as corners, edges, and texture spots on the image plane between consecutive image frames. Simultaneously, the semantic SLAM system uses this information to incrementally construct a globally consistent, sparse or semi-dense three-dimensional geometric map from continuously observed scene feature points. This map forms the skeleton of the entire environment, and the system continuously tracks the device's precise position within this skeleton.

[0031] Step 1103: Analyze the color image frame by frame using an instance segmentation neural network to identify each item instance and output the category label and pixel-level segmentation mask for each instance. Specifically, a pre-trained lightweight instance segmentation neural network, such as MobileNetV3 with an efficient backbone, is used to analyze each frame of the color image. This network can perform two key tasks: first, classification, predicting a category label for each identified object region, such as "cup" or "book"; second, pixel-level segmentation, generating a "mask" image of the same size as the input image, in which all pixels belonging to the same object instance are labeled with the same value, thereby accurately outlining the contour of each object at the pixel level and achieving instance-level separation.

[0032] In this embodiment, the instance segmentation neural network adopts the Mask R-CNN architecture, with ResNet-50 as the backbone network, combined with a Feature Pyramid Network (FPN) for multi-scale feature extraction. In a specific implementation example, the following architecture and parameters can be used. It is understood that those skilled in the art can also choose other equivalent instance segmentation networks, such as YOLACT, SOLO, or adjust specific parameters, depending on actual accuracy and real-time requirements, as long as pixel-level item instance segmentation can be achieved. The specific structure is as follows: The input layer receives RGB color images with a size of 640×480 pixels, normalized to the [0,1] range.

[0033] The backbone network, ResNet-50, extracts feature maps for four stages: C2, C3, C4, and C5, with 256, 512, 1024, and 2048 channels respectively.

[0034] Feature Pyramid Network (FPN): A top-down path and lateral connection that generates multi-scale feature maps P2, P3, P4, and P5, each with 256 channels, used to detect objects of different sizes (from remote controls to trash cans).

[0035] The Region Proposal Network (RPN) slides anchor points on feature maps P2-P5. The anchor point sizes are 32², 64², 128², 256², and 512² pixels, with aspect ratios of 1:2, 1:1, and 2:1, respectively. Fifteen anchor points are generated at each location on each feature map. After classification and regression, the network outputs the top 1000 candidate item regions.

[0036] ROI alignment extracts candidate regions from the feature map into a fixed-size 7×7 feature map, maintaining spatial location accuracy (avoiding quantization errors).

[0037] Head network: includes three parallel branches: Classification branch, fully connected layer (1024 dimensions) - fully connected layer (number of categories + 1), outputs the category probability of each candidate region (51 categories in total, including background).

[0038] The bounding box regression branch consists of a fully connected layer (1024 dimensions) and a fully connected layer (4 × number of categories), which outputs the offset of the bounding box.

[0039] Masking branch, fully convolutional network (4 convolutional layers, 256 channels per layer, kernel size 3×3) → deconvolutional layer (2x upsampling) → 1×1 convolution, outputting a 28×28 binary segmentation mask.

[0040] During pre-training, the COCO 2017 dataset (118k images, 80 categories) was used for initial training. The loss function was L_total = L_cls + L_box + L_mask, where L_cls is the cross-entropy loss, L_box is the Smooth L1 loss, and L_mask is the binary cross-entropy loss. The optimizer used SGD with an initial learning rate of 0.02, momentum of 0.9, weight decay of 1e-4, batch size of 16, and training for 90 epochs. The learning rate decayed by a factor of 0.1 at epochs 60 and 80.

[0041] During fine-tuning, a self-built indoor home furnishings dataset was used, comprising 10,000 RGB-D images covering 50 common home furnishing categories (cups, medicine bottles, remote controls, knives, etc.). Each image was annotated with an instance-level segmentation mask (manually annotated using the LabelMe tool). During fine-tuning, the first two layers (C2, C3) of the backbone network were frozen, and only the FPN, RPN, and head network were trained. The optimizer was changed to Adam, with a learning rate of 1e-4, a batch size of 8, and training for 50 epochs. Early stopping was employed (training stopped if the loss did not decrease for 10 consecutive epochs). After fine-tuning, the average mask AP (Average Precision) on the test set reached 0.72, meeting the real-time processing requirements (single-frame inference time ≤ 50ms).

[0042] Step 1104: Align and fuse the segmentation mask of each instance with the synchronously acquired depth image at the pixel level; combine the camera intrinsic parameter matrix and the real-time six-degree-of-freedom pose, back-project the pixels in the mask to the three-dimensional space to generate a three-dimensional point cloud cluster for each item instance; Specifically, the two-dimensional segmentation mask obtained in step 1103 is aligned with the depth image synchronously acquired in step 1101. Then, using the camera's intrinsic parameter matrix (parameters describing the lens's optical characteristics) and the current device pose provided in step 1102, back-projection calculations are performed to transform each pixel within the mask belonging to the same object instance from its two-dimensional image coordinates to its three-dimensional world coordinate system. The collection of all transformed three-dimensional points forms a discrete three-dimensional point cloud cluster of the object in real space, thus transforming the object from a region in an image into a set of points in space.

[0043] Step 1105: Perform spatial clustering and geometric analysis on the 3D point cloud clusters of each item instance to calculate the 3D geometric information of the item. The 3D geometric information includes: 3D axial bounding box, centroid coordinates, main orientation and occupied spatial volume. Specifically, spatial clustering algorithms, such as density-based clustering of applications with noise (DBSCAN), are used to denoise and optimize the point clouds in the 3D point cloud clusters, filtering out outliers and ensuring the integrity of the point cloud clusters. Subsequently, geometric analysis is performed to calculate the smallest cuboid that can enclose all points with edges parallel to the coordinate axes, i.e., the 3D axial bounding box, used to quickly represent the spatial occupancy of the object; the centroid coordinates of the point cloud are calculated as a representation of the object's spatial location; the distribution of the point cloud is analyzed to estimate its main orientation; and its occupied volume is calculated. These attributes collectively constitute the 3D geometric information of the object in 3D space.

[0044] Step 1106: Extract the visual feature vector of each item instance, and match the visual feature vector together with the category label and three-dimensional geometric information with the preset personalized item knowledge graph to obtain the matching result; the personalized item knowledge graph stores the general semantic attributes of the item, the security level classification, and the user's custom semantic tags or memory anchors for the item. Specifically, visual feature vectors are extracted from the original color image region of an item instance using a neural network, serving as a digital fingerprint of the item's appearance. The visual feature vector is a high-dimensional numerical representation extracted from the color image region of the item instance using a neural network, used to quantify the item's appearance characteristics (such as shape, color, texture, and pattern). This visual feature vector, along with its category label and 3D geometric information, is then matched against a pre-constructed personalized item knowledge graph. This knowledge graph is a structured database that includes not only general semantic attributes of items, such as function, material, and security level, but also stores personal user information, such as preferred storage locations and memory prompts set by family members. Through matching, the most matching graph entry is bound to the current item instance, thereby attaching semantic labels and security attributes, ultimately generating a complete semantic node. This semantic node is the basic unit of the environmental semantic map, integrating geometry, recognition, semantics, and security.

[0045] The personalized item knowledge graph is the core knowledge base built to achieve deep semantic understanding and personalized assistance for items. It is a structured, dynamically updated data network in which each item entity corresponds to a knowledge node, and the node information consists of the following three levels: The general semantic layer includes the standard name of the item, such as "ceramic mug", the functional category, such as "drinking utensils", and physical attributes, such as material, color and other objective descriptive information.

[0046] The safety and operational layer defines the risk level of items, such as "high risk" or "requires supervision," specific safety precautions, such as "fragile" or "sharp," and common-sense recommended storage locations, such as "knives should be placed on shelves."

[0047] The personalized memory and habit layer stores information closely related to the specific user, including personalized nicknames given by the user or family members, such as "my old teacup," associative cues that can evoke contextual memories, such as a voice recording from a family member: "This is a gift from your daughter," and user-specific usage habits learned through long-term observation, such as "this medicine bottle is often used at 8 a.m.

[0048] During the deployment phase, family members or caregivers manually label important items in the home with the above information, especially personal memories and security attributes, through a dedicated configuration interface, thereby completing the personalized initialization of the map.

[0049] During operation, when the RGB-D camera captures new items or changes in item status, the main control module attempts to match and associate the item features with the knowledge graph. Simultaneously, by silently observing successful user actions and habitual patterns, it automatically performs incremental updates and optimizations to attributes such as "frequently used locations" and "usage patterns" of items in the graph, enabling the graph to adapt to changes in user lifestyles.

[0050] This knowledge graph constitutes a rich source of information for semantic nodes in the environmental semantic map, and is also the fundamental basis for contextualized prompts, safety judgments, and personalized task planning.

[0051] Step 1107: Based on the matching results, attach corresponding semantic tags and security attributes to each item instance to form a semantic node; wherein, the semantic tags include at least the standard name and functional category of the item, and the security attributes include at least a risk level identifier; Specifically, a semantic node data structure is created and populated for each instance. This semantic node includes at least two core attributes: first, a semantic tag, explicitly assigning the item a machine-understandable standard identity, such as its recognized standard name (e.g., "iPhone charger") and its functional category (e.g., "electronic product / charging device"); second, a safety attribute, including at least a categorized risk level identifier (e.g., "Medium Risk: Electronic Device"), indicating the level of monitoring or warning strategies required in subsequent interactions. This generates a complete digital entity that integrates geometry (including location and shape), perception (including visual features and cognition), semantics, and safety.

[0052] Step 1108: Based on the spatial relationship of the centroid coordinates of each semantic node in the world coordinate system, an environmental semantic map is formed; and as new data is collected, the confidence and location information of each semantic node are updated in real time through multi-view geometric consistency test and Bayesian filtering method.

[0053] Specifically, multi-view geometric consistency checks are performed, such as feature point reprojection error verification based on the RANSAC algorithm, and Bayesian filtering methods, such as extended Kalman filtering. Based on the centroid coordinates of each semantic node—its three-dimensional position in the world coordinate system—it is organized according to its natural spatial adjacency to form a global, queryable environmental semantic map. This environmental semantic map is essentially a spatial database recording what is where and what it is. More importantly, the environmental semantic map is not a static snapshot; it can continuously collect new data—new images and depth information—as the user moves and the device moves. The hardware device uses multi-view geometric consistency checks, cross-validating the reliability of the location and appearance of the same object by using information observed from different angles. Simultaneously, Bayesian filtering methods, such as Kalman filtering or its extensions, are used to continuously update two key states of each semantic node, using new observational evidence as input: first, the estimated value and uncertainty of its location information, determined by covariance for more accurate positioning; and second, its overall confidence level, which increases with continuous, consistent observations or decreases due to long periods without observation or conflicting observations. This dynamic update mechanism enables the environmental semantic map to adapt to actual situations such as objects being moved and environmental layouts changing, maintaining the accuracy and real-time nature of its representation of the real world.

[0054] Step 120: When the user looks at or approaches the item corresponding to the semantic node, the basic information of the item is displayed through the AR interface; Specifically, when the main control module determines that a user is looking at or approaching an object through eye tracking or distance perception, it proactively and non-intrusively displays the object's basic information, such as its core name and safety reminders, through an AR interface. Figure 4 As shown, step 120 includes the following: Step 1201: When the user looks at or approaches the object, the basic information of the object is displayed through the AR interface.

[0055] Specifically, the system continuously monitors two types of interaction intentions between users and objects in the environment: visual attention (cognitive interest shown through fixation) and physical location (operational intention shown through proximity). When either condition is met, it is determined that the user has an information need or intention to operate on the object, thereby triggering the subsequent information query and display process, realizing a shift from a passive response to an active perception interaction mode.

[0056] Step 1202: Obtain the user's three-dimensional gaze vector through eye-tracking technology, calculate the spatial intersection of the three-dimensional gaze vector with the three-dimensional bounding box of each semantic node in the environmental semantic map, and determine gaze if the gaze continues for more than a first time threshold. Specifically, the eye-tracking module typically calculates a 3D gaze vector in real time based on the pupil-corneal reflex principle, whereby the user's gaze converges. Using the device's real-time pose, this vector is transformed into a world coordinate system, forming a spatial ray originating from the user's eyes. Subsequently, this ray is rapidly intersected with the 3D bounding boxes of all semantic nodes in the environmental semantic map. If the ray intersects with the bounding box of an object for a duration exceeding a preset threshold (e.g., 500 milliseconds), it is determined that the user is looking at that object. This eliminates interference from brief saccades, ensuring the stability of gaze determination and the authenticity of intent.

[0057] Step 1203: Calculate the Euclidean distance between the user's location and the centroid of each semantic node in real time. If the distance is less than the preset second distance threshold, the user is considered close. Specifically, the system acquires the user's real-time location, typically represented by the device's location, and calculates the Euclidean distance between the user and the centroid coordinates of various semantic nodes in the environmental semantic map. If this distance is less than a preset second distance threshold, such as 0.5 to 1 meter, representing a typical arm span or direct operating range, it determines that the user has approached the object. This determination complements the gaze determination, ensuring that even if the user does not intentionally look at the object, necessary information or warnings can be proactively provided when their body approaches a dangerous or target object.

[0058] Step 1204: When it is determined to be a gaze or proximity, query the complete semantic tag and security attributes corresponding to the semantic node; Specifically, when a gaze or proximity condition is triggered, all associated information is retrieved from the environmental semantic map based on the semantic node ID of the triggered item. This includes complete semantic tags bound from the personalized item knowledge graph, such as detailed name, function, material, and safety attributes like risk level, specific warnings, and operating procedures. This step provides the data foundation for the next step of information filtering.

[0059] Step 1205: Based on the current task context, user cognitive load status, and item safety attributes, key basic information is dynamically filtered out using a priority algorithm; the priority algorithm ensures at least that: high-risk items are given priority in displaying warning information, and items strongly related to the current task are given priority in displaying task instructions; Specifically, this step is a core decision-making process to ensure accurate, timely, and uninterrupted information prompts. Instead of directly displaying all information, a priority algorithm dynamically filters out 1-3 of the most critical basic information points based on multiple contexts. The rules of this priority algorithm include at least the following: 1. Safety Priority Principle: If an item's safety attribute is marked as high-risk or requires monitoring, a clear safety warning must be displayed regardless of the current task, such as "Danger! Knife." 2. Task Relevance Principle: If the user is performing a task guided by the main control module, such as "taking medication," instructions directly related to that task are prioritized, such as the "dosage and time" on the medicine bottle, rather than general descriptions. 3. Cognitive Adaptation Principle: Based on real-time assessment of the user's cognitive load, when the load is high, the information is extremely simplified to icons and keywords; when the load is low, slightly more detailed phrases can be displayed.

[0060] Step 1206: The selected basic information is overlaid and displayed near the real spatial location of the corresponding item using an AR rendering engine in a spatially locked manner; the information display has a delayed trigger and automatic fade-out mechanism, and the display of information on irrelevant items is suppressed when the user is performing the main task.

[0061] Specifically, this step is responsible for presenting the selected information to the user in a user-friendly and efficient manner. After receiving the information content and the 3D spatial coordinates of the target object, the AR rendering engine uses a spatial locking method to stably render the virtual information panel, such as a semi-transparent label, at the real-world location near the object, maintaining its attachment as the user moves. Display mechanisms are used to reduce interference. For example, delayed triggering avoids frequent flickering when the user's gaze sweeps over the object; automatic fade-out clears the information after the user's gaze moves away; and task suppression temporarily blocks automatic prompts for other unrelated items when the user is focused on a guided task, preventing information overload. These mechanisms work together to ensure that AR prompts are auxiliary rather than disruptive.

[0062] Step 130: Receive the user's voice request, use the intent understanding model trained on the cognitive impairment disorder language corpus, and combine it with the environmental semantic map to perform context fusion reasoning, parse out the operable user intent and determine the unique target object; The cognitive impairment disordered language corpus is derived from publicly available Alzheimer's patient speech datasets, such as DementiaBank, and over 500 hours of clinically collected dialogue recordings. These recordings have been annotated with intent categories and slot information by professional physicians, covering typical features such as unclear pronunciation, disordered word order, and ambiguous reference. Figure 5 As shown, step 130 includes: Step 1301: The collected user voice is denoised and converted into text using a speech recognition model trained on pathological speech corpus. Specifically, speech is captured via a microphone array, and beamforming technology is used to enhance the sound signal oriented towards the user while suppressing ambient noise. The audio is then fed into an end-to-end automatic speech recognition model optimized for the characteristics of pathological speech. This model is trained using a large corpus of speech samples from Alzheimer's patients, including those with slurred pronunciation, uneven speech rate, and disjointed sentences, enabling it to adapt to and accurately recognize non-standard pronunciations and output preliminary text with a low error rate, laying the foundation for subsequent semantic understanding.

[0063] The speech recognition model adopts the Conformer-CTC architecture. Its specific structure is as follows: The input layer extracts features from an 80-Vimel filter bank with a window length of 25ms and a stride of 10ms. Each speech segment is truncated or padded to 3000 frames (corresponding to 30 seconds of speech).

[0064] The encoder is composed of 12 stacked Conformer blocks. Each Conformer block includes four modules: The first feedforward module (FFN1), with a dimension of 2048, is activated by Swish with dropout=0.1.

[0065] The multi-head self-attention module has 8 attention heads, a dimension of 512, and uses relative position encoding with a maximum relative position of 128 and dropout=0.1.

[0066] The convolution module features depthwise separable convolution with a 5×5 kernel size, gated linear unit (GLU) activation, 1024 pointwise convolution channels, and dropout=0.1.

[0067] The second feedforward module (FFN2) has a dimension of 2048, Swish activation, and dropout=0.1. Each module is followed by a normalization layer and residual connections.

[0068] The output layer uses a linear transformation to map the encoder output (512 dimensions) to a vocabulary size, such as 5000 word units, including Chinese characters, punctuation, and special symbols, and then aligns them using the CTC loss function.

[0069] During pre-training, the AISHELL-2 Chinese speech dataset was used, consisting of approximately 1000 hours of data covering various accents and scenarios. The optimizer was AdamW, and a warm-up-decay strategy was employed, where the learning rate linearly increased from 1e-6 to 1e-3 in the first 10k steps, then exponentially decreased to 1e-5. The batch size was 32, and training lasted for 120 epochs. The loss function was CTC loss, with label smoothing incorporated, such as a coefficient of 0.1.

[0070] For fine-tuning of pathological speech, a 200-hour speech corpus of Alzheimer's patients can be used. This corpus, collected from multiple patients with mild to moderate Alzheimer's disease (MMSE scores 15-24), covers everyday object requests and commands, such as "Give me that red cup." During fine-tuning, the encoder structure remains unchanged, the output layer is reinitialized, and the CTC loss function is used, with a learning rate of 5e-5, a batch size of 8, and training for 30 epochs. After fine-tuning, the word error rate (WER) on the patient speech test set decreased from 28.5% to 16.2%.

[0071] Step 1302: Input the text into the intention understanding model fine-tuned by the cognitive impairment dialogue corpus to obtain preliminary analysis results including intention categories and semantic slots that may have ambiguous referents; Specifically, this step performs preliminary structured understanding of the identified text. The text is input into an intent understanding model, which is based on a pre-trained language model, such as BERT, and has been fine-tuned using a large amount of labeled dialogue data from patients with cognitive impairment. This model performs two joint tasks: first, intent classification, determining the user's overall operational intent, such as retrieving or storing an object; and second, slot filling, identifying specific attribute parameters mentioned in the text, such as color and location. Its key capability lies in its ability to parse ambiguous references caused by language disorder, such as "that" or "the red thing," and output them as structured slots, forming a preliminary analysis result that includes intent and potentially incomplete or uncertain parameters.

[0072] Step 1303: Construct a dynamic context vector, the data of which includes: the semantic node corresponding to the user's current gaze focus, a list of semantic nodes in the neighborhood space, the current task status, and short-term interaction history; Specifically, this step provides real-time contextual information for disambiguation inference. The main control module integrates real-time data from multiple channels to construct a dynamic context vector. This vector mainly includes: 1. Visual attention context: the semantic node currently being gazed at by the user, determined through eye tracking; 2. Environmental spatial context: a list of all semantic nodes and their attributes within a certain radius centered on the user's current location; 3. Task and interaction context: the current task state being guided by the main control module, and the user's short-term interaction history in recent times, such as what items were recently manipulated. This vector digitizes the user's physical environment, visual focus, and behavioral state, providing key clues for understanding fuzzy language.

[0073] Step 1304: Input the preliminary analysis results and the dynamic context vector into the inference engine. By calculating the correlation between the fuzzy reference and each candidate semantic node in the semantic, spatial and temporal dimensions, the fuzzy reference is disambiguated and anchored to a specific semantic node, thereby determining the unique target object. Specifically, this step is crucial for understanding the user's precise intent. The main control module inputs the preliminary parsing results obtained in step 1302, including the fuzzy references, along with the dynamic context vector constructed in step 1303, into an inference engine. This engine can employ a neural network or graph neural network based on an attention mechanism. The engine calculates the relevance of the fuzzy reference, such as "that red one," to each candidate semantic node in the dynamic context vector across multiple dimensions. These dimensions include semantic relevance, spatial relevance, and temporal relevance. Semantic relevance refers to the matching degree between the description and the item's attributes; spatial relevance refers to the proximity of the item to the user's line of sight or location; and temporal relevance refers to the frequency of the item's appearance in recent interaction history. By weighted summarization of these relevances, the inference engine uniquely anchors the fuzzy reference to a specific semantic node in the environmental semantic map, thereby outputting an actionable and targeted user instruction.

[0074] In this embodiment, the inference engine used for referential disambiguation employs a Graph Attention Network (GAT). The specific structure is as follows: Graph construction: For each user voice request, a graph is dynamically constructed. The nodes of the graph are a set of candidate semantic nodes, drawn from the dynamic context vector. The semantic node corresponding to the user's current gaze focus can be at most 1.

[0075] All semantic nodes within a 2-meter radius of the user's location, up to a maximum of 20.

[0076] Deduplicat semantic nodes involved in the most recent 5 interaction histories.

[0077] Node features: Each node is represented by a 128-dimensional vector, which is composed of three parts: Semantic embedding (64-dimensional) is obtained by inputting the category labels of the items into a pre-trained Word2Vec model.

[0078] Spatial features (32-dimensional): the relative position of the node to the user (dx, dy, dz) and the cosine of the angle between the node and the user's gaze vector.

[0079] Temporal features (32 dimensions): the timestamp of the node's most recent interaction (normalized to [0,1]) and the frequency of interaction.

[0080] Edge weights, calculated based on spatial distance (Gaussian kernel) and semantic similarity (cosine similarity) between nodes, serve as biases for the attention mechanism.

[0081] The graph attention layer uses two layers of GAT, each with four attention heads, each outputting a 32-dimensional graph, resulting in a 128-dimensional graph after concatenation. The activation function is ELU.

[0082] The output layer outputs a scalar score for each node. After Softmax normalization, the node with the highest score is selected as the disambiguation result, and the confidence level of the score is also output.

[0083] The training method is as follows: Acquire the dataset. This could involve collecting, for example, 5000 user voice request texts with ambiguous referents. Each request corresponds to a real-world scenario, including 5-15 candidate items. The correct semantic node ID for each request is manually labeled. The dataset includes typical ambiguous referents such as "that," "this," "red," and "on the table," covering different spatial distances and degrees of semantic ambiguity.

[0084] The loss function is the pairwise hinge loss: L = Σ max(0, 0.5 - (s_pos - s_neg)), where s_pos is the score of the positive example node and s_neg is the highest score among the negative example nodes.

[0085] Specifically, when calculating the relevance of semantic, spatial, and temporal dimensions, the initial weights for each dimension were set as follows: semantic relevance weight 0.5, spatial relevance weight 0.3, and temporal relevance weight 0.2. These initial values ​​were obtained through experimental tuning on 200 patient voice requests with ambiguous referents.

[0086] To further improve the accuracy of disambiguation, the above weights will be dynamically adjusted based on the real-time context, according to the following rules: If the current user's 3D gaze vector intersects with the bounding box of a candidate semantic node, indicating that the user is gazing at the item, the spatial relevance weight is temporarily increased to 0.6, while the semantic and temporal weights are correspondingly reduced, with a total of 1.

[0087] If a user has interacted with an item in the last 30 seconds, such as picking it up or moving it, the time relevance weight will be temporarily increased to 0.4.

[0088] If the user's speech includes explicit spatial prepositions, such as "on the left" or "on the table," the spatial relevance weight increases to 0.55.

[0089] If the user's voice includes explicit attribute words, such as "red" or "round," the semantic relevance weight increases to 0.6.

[0090] The aforementioned weight adjustments will revert to their default values ​​after each inference iteration.

[0091] Step 1305: When the reasoning confidence is lower than the threshold, mark multiple candidate semantic nodes through the AR interface and initiate a clarification inquiry in conjunction with voice.

[0092] Specifically, when the confidence level of the inference engine's output is lower than a preset threshold—for example, when there are multiple candidate targets with similar relevance—the main control module does not forcibly execute potentially erroneous instructions. Instead, it proactively initiates a clarifying dialogue. This is achieved by visually annotating the semantic nodes corresponding to several high-confidence candidate items in the AR display interface, such as by flashing or highlighting them. Simultaneously, a concise clarifying question is posed to the user via spatial audio, such as, "Are you referring to the red apple on the table, or the red cup on the shelf?" By receiving user feedback on this question, such as voice, gaze, or gesture, the main control module ultimately determines the user's true intent, demonstrating the intelligence and robustness of human-machine collaboration.

[0093] The intent understanding model described below includes an input encoding layer, a core encoder, and a multi-task output head.

[0094] In the input encoding layer, the text from the Automatic Speech Recognition (ASR) system, such as "that...red...take it," is first segmented into words. Then, the model converts each word into a high-dimensional vector (word embedding) and adds positional encoding to preserve word order information. Crucially, the model's vocabulary is expanded to include vague pronouns commonly used by patients with cognitive impairment, such as "that thing" and "this," enabling it to better represent these non-standard expressions.

[0095] The core encoder is composed of multiple stacked Transformer encoders. Its core mechanism is multi-head self-attention, which allows the model to examine the relationship between any two words in the input sequence from different perspectives simultaneously. This is crucial for understanding discontinuous dependencies caused by thought interruptions, such as associating the action "take" with the distant attribute "red." The model underwent domain-adaptive fine-tuning on a large corpus of real, labeled dialogues from patients with cognitive impairment. Targeted training strategies may be employed during fine-tuning, such as: random masking training, randomly masking parts of the input to simulate forgetting and interruptions in patient speech, forcing the model to learn to make more robust inferences from incomplete context; and noise-resistant training, introducing noise similar to ASR errors into the input to improve the model's tolerance for inaccurate recognition.

[0096] The multi-task output head completes two key tasks: The intent classification header, typically attached after a specific symbol representing the entire sequence, such as the encoding vector of [CLS], consists of a fully connected layer and a Softmax activation function. It is responsible for classifying the entire utterance into a predefined intent category, such as retrieving, storing, or organizing.

[0097] The slot filling head is connected after the encoded vector of each word output by the encoder. It typically uses a conditional random field layer or a feedforward network to label each word in the sequence with its semantic slot type, using a BIO labeling system, such as B-color, I-item, O, etc., to extract structured parameters from unstructured text.

[0098] In one example, the input to the intent understanding model is a sequence of text from the front-end ASR master module. This text directly reflects the characteristics of the patient's speech: Content characteristics: Input text is typically short, loosely grammatically structured, includes numerous filler words such as "uh," "that," and may be logically incomplete or self-contradictory. For example: descriptions with omitted predicates: "Glasses...my..."; descriptions including self-correction: "Put it over there...no, on top."; scattered descriptions: "I want the round one, the one on the table." The text sequence described above can be a string sequence, such as: ["uh", "ba", "that", "red", "give me"]. The model's output is not a simple label, but a structured, machine-readable semantic representation.

[0099] The output includes: Intent, a clear, predefined action category representing the macro-level action the user wants to perform; Confidence, a numerical value reflecting the model's confidence in the parsing result; and a set of semantic slots, a list of key attribute-value pairs extracted from the input text. For example, from "put the red one on the table," the model might output {color: red} and {target location: on the table}. Crucially, what exactly "the red one" is is transformed into a structured query including "action type" and "object partial attributes." This query is passed to the downstream contextual fusion inference engine. The inference engine compares, matches, and disambiguates it in real-time against an environmental semantic map, which includes the precise identities, locations, and attributes of all items, ultimately locking onto the unique and precise target item indicated by the user's intent within the specific context, thus completing the closed loop from vague human language to precise machine-executable instructions.

[0100] It should be noted that for new users using the system for the first time, the historical operational ability profile and personalized item knowledge graph are initially empty. During this cold start phase, the system adopts the following default strategies: 1) Task planning and decomposition use a standard task template and difficulty baseline predefined based on general population behavior data, which is derived from initial survey data of multiple patients with mild cognitive impairment; 2) The personalized comparison threshold in the state assessment model temporarily uses the statistical norms of healthy elderly populations, and user personal data is rapidly accumulated during the first 5 task executions, gradually switching to the personal baseline; 3) Family members or caregivers can pre-enter personalized information of key items through the collaborative management interface, thereby partially filling the knowledge graph before the first use. When the accumulated user interaction records, including successful and failed tasks, exceed 20, the system automatically triggers the first personalized model fine-tuning, and subsequent periodic updates are performed as per claim 8.

[0101] Step 140: Based on the user intent, the real-time status of the target object, and the user's historical operational capability profile, generate an executable task sequence including multiple atomic operations; wherein the task complexity and decomposition granularity of the executable task sequence are matched with the user's current cognitive load. Atomic operations refer to indivisible basic operational units or those whose granularity can be dynamically adjusted according to cognitive load. A user's operational history ability profile is a personalized computational model that is dynamically generated and continuously evolved within the user's operational history ability profile model. It is used to quantitatively characterize the ability level of specific cognitively impaired patients to perform daily tasks. It transforms the user's abstract, vague, and fluctuating cognitive functions and operational abilities into a structured data model that can be understood and used for decision-making.

[0102] This user action history profile consists of multiple interrelated quantitative dimensions, obtained through long-term, passive collection of every user interaction. The acquired interaction data includes task-level metrics, step-level metrics, interaction-level metrics, and physiological correlation metrics.

[0103] Task-level metrics include: Success rate history refers to the number and proportion of successes and failures for different types of tasks, such as "finding the remote control," "taking medicine," and "folding clothes." Task time distribution refers to the average, variance, and trend of the time required to complete each type of task.

[0104] Step-level metrics, i.e., atomic operation levels, include: Step execution efficiency refers to the average time and stability taken to complete each atomic operation, such as "picking up a water glass" or "walking to the bookshelf." Step hesitation time refers to the delay between receiving an instruction and starting to execute the action. Step error pattern refers to recording the types of errors that are frequently made, such as "picking up the wrong object," "reversing the order of steps," or "completely forgetting the step," and their frequency.

[0105] Interaction layer metrics include: Request frequency and pattern refer to the frequency at which users proactively ask questions or express confusion via voice, thereby triggering prompts. The main control module also analyzes which steps within which tasks are most likely to elicit a request for help. Response pattern to guidance refers to the user's reaction speed and correction effectiveness to different types of prompts, such as voice commands, AR arrows, and icon flashing.

[0106] Physiological correlation indicators include: The baseline physiological signal refers to the typical range of physiological data such as heart rate and heart rate variability when performing simple, familiar tasks. Stress response correlation refers to the corresponding patterns of physiological signals, such as increased heart rate and changes in skin conductance, recorded when performing difficult steps or making mistakes.

[0107] The modeling process for this user operation history profiling model includes: During data collection, data from all the above dimensions are continuously recorded in the background, and each data point is associated with a specific task type, execution step, timestamp, and environmental context.

[0108] Extract meaningful features from raw behavior logs. For example, convert the time spent on the "pick up the medicine bottle" step into "percentage relative to the user's historical average" or "Z-score relative to the standard duration of the step".

[0109] Use these time-series feature data to train a lightweight machine learning model. This model can be a gradient boosting decision tree (GBDT), a hidden Markov model (HMM), or a shallow neural network.

[0110] The model's learning objective is to establish a mapping relationship between user characteristics and task performance. For example, the model can learn that when users are in the morning and their hesitation time is generally longer, their probability of failure on tasks requiring fine motor skills increases significantly. The ultimate goal of generating a user's operational history capability profile is to serve the adaptive decision-making of the main control module. Its main outputs and applications include: Predicting task and step difficulty: When the main control module plans a new task, the user operation history ability profile model can predict the potential difficulty coefficient and expected time consumption of each step in the user's execution of this task. This is the core basis for realizing personalized task decomposition.

[0111] Establish personalized baselines to create unique benchmarks for each user's normal performance, such as standard time ranges for each step and physiological resting baselines. Deviations monitored in real-time by the main control module are based on comparisons with these individual baselines, rather than universal standards, resulting in more accurate assessments.

[0112] By analyzing long-term trends and short-term anomalies in user profiles, the main control module can infer the daytime or intraday fluctuations in user capabilities, such as "afternoon fatigue period" or "improvement period after medication," thereby adjusting guidance strategies in advance.

[0113] Optimize the interaction parameters of the main control module. The profile data can be used to automatically fine-tune the interaction parameters of the main control module, such as setting the most suitable voice command speed, the complexity of AR prompts, and the sensitivity threshold for triggering auxiliary interventions for the user.

[0114] User operation history profiling goes beyond simple log functionality, achieving computational modeling of user cognitive-behavioral patterns. It abandons a "one-size-fits-all" approach, establishing unique assessment and prediction benchmarks for each user. This gives the main control module a long-term memory of user capabilities, enabling it to understand the meaning of current user behavior, forming the cornerstone of truly personalized, adaptive assistance. In the patent, it is not merely a data recorder, but a predictive intelligent agent, a key intelligent module connecting historical behavior with future adaptive guidance.

[0115] Current cognitive load is a multidimensional, computable fusion variable, including: Behavioral efficacy reflects changes in external operational performance caused by cognitive resource consumption. Timing deviation is the ratio between the actual execution time of the current step and the personalized expected time predicted based on the user's historical operational ability profile. Significant timeouts are a key behavioral indicator of increased workload. Action quality refers to the fluency, accuracy, and path optimization of the user's actions. For example, trembling, hesitation, or multiple corrections in the hand trajectory when picking up an object. Error frequency is the consecutive occurrence of primary errors such as incorrect object selection or incorrect step sequence within a short time window. Physiological arousal reflects the autonomic nervous system response caused by cognitive effort or stress. Cardiovascular signals are measured in real-time using photoplethysmography (PPG) sensors, including heart rate (HR) and heart rate variability (HRV). Increased cognitive load is typically accompanied by an increase in HR and a decrease in HRV (especially high-frequency components). Electrodermal activity measures minute changes in skin conductivity using Galvanic Skin Response (GSR) sensors; increases in GSR are correlated with attention and emotional arousal. Breathing patterns: Increased breathing rate or altered rhythm. Interaction patterns: Reflecting the user's cognitive state during interaction with AR glasses. Help signals: Questions, repetitive requests, or confused statements actively expressed by the user through voice. Distractions: Frequent eye movement away from task-related areas, detected by eye tracking, such as moving away from the target object or aimless wandering. Interaction delays: Extended response time to voice or visual cues.

[0116] The main control module transforms the aforementioned multi-source data into usable load indices through the cognitive load assessment engine. This process includes the following steps: Real-time feature extraction involves online processing of each input data stream, such as video frames and heart rate interval sequences, to extract key feature values. For example, it calculates the variance of hand movement speed from video and the RMSSD (HRV time-domain index) of the past 30 seconds from heart rate sequences.

[0117] Personalized baseline comparison compares extracted real-time feature values ​​with a personalized baseline range stored in the user's historical operational capability profile. For example, it calculates the standard deviation multiple by which the current step time exceeds the user's average time. This comparison based on personal history is the key difference between this method and a general threshold, making the assessment insensitive to individual user differences, such as a user who is consistently slow, but sensitive to fluctuations in their performance.

[0118] Multimodal information fusion and classification inputs multiple alignment bias values ​​from behavioral, physiological, and interaction dimensions into a pre-trained classification or regression model. This model can be a Support Vector Machine (SVM), a Random Forest, or a shallow neural network. The model outputs a comprehensive judgment result, typically represented in two forms: Discrete load levels, such as low, medium, and high load conditions. Continuous load levels, such as a scalar value between 0.0 and 1.0, where a larger value indicates a higher cognitive load.

[0119] The output includes a confidence score for downstream modules to use as a reference for decision-making.

[0120] The current cognitive load index directly drives the adaptive adjustment of the following key modules, forming a closed loop: For example, under high load, a refined decomposition mode is activated for subsequent steps, breaking down complex operations into more basic and easily executed action primitives, such as breaking down "pouring water" into "lifting the kettle - aligning it with the cup - tilting - straightening," and providing more intensive guidance. Under low load, multiple atomic operations are summarized into a macro-instruction, such as "please organize these books," providing a smoother and more efficient guidance experience.

[0121] For example, under high load, it automatically adopts simpler AR icons, slower and clearer voice prompts, stronger visual contrast, and increases the repetition of key information. Under low load, it reduces the frequency of prompts, uses more natural language and simpler visual cues, and avoids excessive intervention.

[0122] Specifically, such as Figure 6 As shown, the generation of the executable task sequence specifically includes: Step 1401: Retrieve a task template corresponding to the user intent from the hierarchical task knowledge base. The template consists of multiple atomic operations with a logical order. Specifically, a structured, hierarchical task knowledge base is maintained in memory. The top layer of this knowledge base is organized by daily activity categories, such as "personal care" and "household chores," while the lower layers store multiple standardized task templates corresponding to each category. Each task template is a predefined flowchart consisting of multiple atomic operations arranged in a logical order, including sequence, selection, and loop operations. When a clear user intent, such as "take medication," is received, the corresponding task template is retrieved from this knowledge base. This template may include atomic operations such as "confirm medication, open bottle cap, pour out pills, take medication, and close bottle cap," providing a basic framework for subsequent personalized adjustments.

[0123] Step 1402: Based on the user's historical operational ability profile, add a predicted difficulty score to each atomic operation in the task template; Specifically, the system invokes a historical operational capability profile built for the user. This profile is a predictive model that uses machine learning to model long-term user behavior data, such as success rate, time consumption, and error patterns. This historical operational capability profile model receives the current task template and context information, and outputs a predicted difficulty score for each atomic operation in the template. This score is a continuous value between 0 and 1, where 0 indicates that the operation is highly likely to succeed for the user, and 1 indicates that failure is highly probable. For example, for the operation "pouring out the pills," the model might predict a high probability of failure based on the user's historical data, thus assigning a high difficulty score, such as 0.85.

[0124] Step 1403: Real-time assessment of the user's current cognitive load index; Specifically, this step obtains instantaneous state parameters that affect task execution. The main control module analyzes the user's behavioral flow in real time, such as movement sluggishness, physiological signals such as heart rate variability, and recent interaction performance, and runs a lightweight cognitive load assessment model to calculate the user's current cognitive load index. This current cognitive load index is a quantitative value used to characterize the user's current level of mental resource strain in processing information, and is usually divided into discrete levels such as "low," "medium," and "high" or a continuous value. This is a key instantaneous basis for determining whether the guidance strategy should encourage independence or provide intensive support.

[0125] The cognitive load assessment model can be based on a rule engine or a lightweight classifier such as SVM. It integrates and analyzes multi-source information, including the user's behavioral flow (e.g., slowness of movement), physiological signals (e.g., heart rate variability), and recent interaction performance. Through a decision matrix or algorithm, it outputs a discrete, comprehensive state level. This level is at least divided into: Level 1, mild hesitation, characterized by slight timeouts or short pauses without physiological abnormalities; Level 2, clear forgetting or confusion, characterized by errors in the task, pauses in steps, or mild physiological stress signals; and Level 3, significant anxiety or persistent errors, characterized by task flow interruptions, repetitive errors, and strong physiological stress responses. The assessment model provides a direct basis for precise intervention.

[0126] The specific structure of the cognitive load assessment model is as follows: The cognitive load assessment model uses a random forest classifier to classify users' real-time cognitive load into three levels: low, medium, and high. The specific structure is as follows: Input features (12 dimensions in total) include behavioral features, physiological features, and interaction features.

[0127] The behavioral characteristics are 4 dimensions, including the ratio of the current step time to the historical average, the number of errors in the last 30 seconds, the variance of hand movement speed (calculated from the trajectory of hand key points), and the number of gaze shifts (the number of times the gaze leaves the task area).

[0128] Five physiological characteristics were included, including heart rate (HR), RMSSD of heart rate variability, high-frequency power (HF, 0.15-0.4Hz), skin conductance level (SCL), and peak amplitude of skin conductance response, such as the peak amplitude of skin conductance response in the last 30 seconds.

[0129] The interaction features are 3-dimensional, including the number of times a request for help has been made in the last 30 seconds, i.e., by voice or key, the response delay to prompts, and the number of repeated voice segments, such as "uh...that...".

[0130] Model parameters include, for example, 100 decision trees, a maximum depth of 10, a minimum number of splits of 5, a minimum number of leaf samples of 2, and the Gini coefficient as the splitting criterion. The number of feature samples is sqrt(12)=4.

[0131] The training method is as follows: During data collection, for example, 30 patients with mild cognitive impairment (MMSE score 18-24) were selected. Each subject performed five standard daily tasks with the assistance of AR glasses, such as taking medication, finding the remote control, tidying the table, pouring water, and identifying dangerous objects. Each task was repeated three times. The 12-dimensional features mentioned above were collected throughout the process. The cognitive load level of the data was labeled as low / medium / high. The labels agreed upon by at least two physicians were taken as the ground truth. A total of 4500 valid labeled samples were obtained.

[0132] During training, you can use the Python scikit-learn library, setting parameters such as n_estimators=100, with the remaining parameters at their default values. Five-fold cross-validation can be used to evaluate the model.

[0133] Step 1404: Based on the current cognitive load index, dynamically optimize the atomic operation sequence: when the load index is low, merge low-difficulty operations to simplify the sequence; when the load index is high, perform refined secondary decomposition on high-difficulty operations; and dynamically insert confirmation checkpoints based on user error patterns. Specifically, this step is the core of adaptive planning. It integrates the long-term capabilities of the user in step 1402 with the instantaneous state in step 1403 to perform real-time pruning of the standard task sequence.

[0134] The optimization strategies include: 1. Complexity Adjustment: When the cognitive load index is low, multiple adjacent atomic operations with low prediction difficulty scores are merged into a single macro-instruction, such as merging "pick up the medicine bottle - open the cap" into "prepare your medicine bottle" to improve interaction fluency; when the load index is high, specific atomic operations with high prediction difficulty scores are further refined into smaller sub-decompositions, using the "action primitive library" to break them down into more basic and easier-to-execute steps, such as breaking "pour out the pill" into "bottle opening to palm - lightly shake wrist - check". 2. Enhanced Risk and Fault Tolerance: Based on common error patterns recorded in the user profile, confirmation checkpoints are dynamically inserted before and after the corresponding steps, such as "please confirm you have a pill in your hand"; for target objects with high safety attributes, a safety pause prompt is automatically added. This generates a deeply customized, executable task sequence that precisely matches the user's current capabilities and state.

[0135] The action primitive library used for secondary decomposition is a predefined set of basic operations. Each primitive corresponds to a minimum action unit that does not require further decomposition, such as "raise your arm" or "rotate your wrist." The initial version of the primitive library was built based on human kinematics data and empirical rules from occupational therapists, and includes 50 commonly used action primitives.

[0136] When an atomic operation is successfully completed in multiple task executions, but the system records a systematic difference between the user's actual action sequence and the preset primitive decomposition—for example, the user habitually uses their left hand instead of their right to perform the "pick up" action—then the user's personalized action pattern can be registered as a new primitive variant and stored in the user's personalized profile. Furthermore, if family members or caregivers upload personalized operation demonstration videos for specific items through the collaborative management interface, the system can extract new action primitives from them and add them to the primitive library. This allows task decomposition to continuously adapt to changes in user behavior.

[0137] Step 1405: Output the optimized executable task sequence and perform online replanning based on the user's actual performance during execution.

[0138] Specifically, this step ensures that the planning is not static but dynamically adjustable. The main control module delivers the optimized task sequence to the guidance module for execution, while simultaneously activating online monitoring and replanning mechanisms. During task execution, if real-time feedback indicates that a step's actual time consumption significantly exceeds expectations, or if the user repeatedly makes mistakes leading to the conclusion that the original plan is no longer applicable, online replanning will be immediately triggered. The replanning engine will reassess the cognitive load based on the user's latest performance and quickly re-optimize the remaining unexecuted steps. For example, it might immediately switch all subsequent steps to a more refined decomposition mode, thereby achieving a dynamic response to fluctuations in the user's state and forming a closed loop of "planning-execution-monitoring-adjustment," ensuring the effectiveness and resilience of the assistance.

[0139] Step 150: Output guidance information in collaboration with spatial audio and AR visual channels registered in the real space; wherein, the visual guidance adopts a progressive revelation strategy, dynamically generating the next step guidance only after the user completes the previous step; Specifically, in this invention, to achieve intuitive, natural, and efficient spatial guidance for users, the main control module adopts a multimodal output method combining spatial audio and an AR visual channel registered in the real space. The definitions of these two are as follows: Spatial audio refers to an immersive sound rendering and presentation method based on Head-Related Transfer Function (HRTF) and real-time head tracking technology. Its core technology lies in dynamically generating auditory signals with precise orientation, distance, and spatial stability by simulating the subtle spectral changes and time differences that occur when sound travels through three-dimensional space to the user's ears, combined with real-time head posture data. Specifically, when the main control module needs to guide the user's attention to a certain direction or object, such as water located to the left front, the audio engine calculates and generates guiding voices or prompts that sound as if they are emanating from that exact spatial point, based on the target's three-dimensional position in the world coordinate system and the user's current real-time head orientation. Even if the user turns their head, the perceived location of the sound source remains fixed in the virtual auditory space, providing intuitive directional guidance that can be perceived without visual assistance.

[0140] The AR visual channel registered in real space refers to the graphics processing flow in which the main control module generates and renders all augmented reality visual elements, such as virtual arrows, highlighted borders, information labels, and illustrative animations. Its key technical feature is that each virtual visual element is stably and persistently aligned and attached to specific 3D coordinates or object surfaces in the physical world through precise spatial calculations. This functionality relies on two fundamental elements: first, an environmental semantic map including the precise 3D coordinates of objects; and second, the device's own high-precision six-DOF pose provided in real-time by visual inertial odometry (VIO). When visual cues need to be overlaid on a real object, such as a medicine bottle, the main control module first obtains the world coordinates of the object's semantic node from the environmental semantic map. Then, based on the current device's perspective, it calculates the screen position and 3D pose of the virtual graphic through 3D graphics projection transformation, ensuring that it stably "anchors" to the target object without drifting as the user moves. This means that the virtual guidance seen by the user through the AR device is deeply integrated with the physical space, just as if it were actually drawn on an environmental object.

[0141] In this main control module, spatial audio and the AR visual channel registered in the real space do not work independently, but rather collaboratively. For example, when the main control module guides the user to pick up a water cup, the AR visual channel renders a highlighted frame on the cup, while simultaneously, the spatial audio generates a voice command that sounds like it's coming from the same location as the cup: "Please pick up the water cup." This strict consistency between the visual focus and the auditory source in three-dimensional space constitutes redundant and reinforced spatial instructions. For users with impaired cognitive function, this multi-sensory consistent spatial cue can significantly reduce confusion caused by difficulties in understanding single-channel information or distraction, greatly improving the robustness of guidance and the user's execution efficiency. It is the core technological foundation for achieving natural and barrier-free assisted interaction.

[0142] like Figure 7 As shown, the method of outputting guidance information through the coordinated use of spatial audio and AR visual channels specifically includes: Step 1501: Generate a multimodal guidance instruction set for each atomic operation, including spatial audio scripts and AR visual parameters; Specifically, for each atomic operation in the executable task sequence, the main control module generates a corresponding multimodal guidance instruction set. This instruction set includes two core elements: first, a spatial audio script, which defines in detail the text content and pronunciation parameters of the guidance speech, and crucially specifies the virtual sound source location coordinates that the sound should be perceived in three-dimensional space; second, AR visual parameters, which define the types of visual elements to be rendered, such as arrows, highlighted boxes, icons, their three-dimensional dimensions, colors, dynamic effects such as pulsation and rotation, and most importantly—the precise three-dimensional coordinates and pose that the element should be registered in the world coordinate system. This step lays the foundation for subsequent immersive, spatially consistent guidance.

[0143] Step 1502: The main control module runs a finite state machine synchronized with the executable task sequence. Specifically, the main control module internally instantiates a finite state machine, whose set of states corresponds strictly one-to-one with the atomic operations in the executable task sequence. Each state is associated with the corresponding multimodal boot instruction set generated in step 1501. The initial state of the state machine corresponds to the first atomic operation in the sequence, and its state transition path is defined strictly according to the logical order of the atomic operations in the sequence. This design transforms the linear task flow into a precisely controllable, state-defined, and fault-tolerant discrete event main control module.

[0144] Step 1503: The state machine will only transition to the next state and trigger the rendering engine to output the next step's guidance information when the sensor data verifies that the user has completed the current atomic operation. Specifically, this step is crucial for ensuring strict synchronization between the guidance and the user's actual actions, achieving a gradual reveal. After outputting guidance information for the current state, such as playing audio or rendering AR elements, the main control module does not automatically proceed to the next step but enters a waiting and verification phase. The main control module comprehensively utilizes real-time sensor data streams from RGB-D, depth sensors, and IMUs, using lightweight algorithms to verify whether the user has completed the current atomic operation in the physical world. Verification methods include: confirming through visual recognition and position tracking that the target item has been picked up, moved, or placed in the correct area; or confirming through positioning data that the user has moved to the expected location. Only when such physical verification is successful does the state machine trigger a transition event, jumping from the current state to the state corresponding to the next atomic operation. The state transition event then triggers the planning module, dynamically generating a new set of guidance instructions for the next state and immediately sending it to the rendering engine. This ensures that the user only ever sees and hears guidance directly related to the current step, and the next step information only appears strictly after the previous step is completed.

[0145] If the user does not complete the current atomic operation within the preset timeout period, such as 60 seconds, or if the user clearly indicates by voice that "I don't know how to do it" or "I forgot", the state machine will trigger a timeout / help event and proceed to the evaluation and intervention process in step 160, instead of remaining in the original state and waiting indefinitely.

[0146] Step 1504: Spatial audio provides sound guidance from the target direction, while AR visual elements are precisely registered at the target location in the real space.

[0147] Specifically, upon receiving instructions, the spatial audio rendering engine utilizes head-related transfer function technology to generate binaural audio with a clear sense of direction and distance, based on the target location of the virtual sound source and the user's real-time head orientation. This allows the user to perceive the sound as coming from a target object or direction. Simultaneously, the AR rendering engine leverages the stable pose provided by visual inertial odometry and the target coordinates in the environmental semantic map. Through 3D graphics projection transformation, it precisely renders and anchors virtual visual elements to their corresponding positions in the real world, ensuring they do not drift with device movement. This strict alignment of the visual focus, the position of AR elements, the auditory source, and the perceived location of spatial audio in 3D space provides users with redundant and enhanced spatial cues, significantly reducing operational confusion caused by omissions or misunderstandings of single-sensory information. This is particularly beneficial for users with cognitive impairments to perform precise orientation and operations in complex environments.

[0148] To achieve seamless integration of multi-sensory guidance, the system requires the following synchronization accuracy between spatial audio and AR visual output: the time difference between the start time of the voice prompt corresponding to the same atomic operation and the start time of rendering visual elements, such as highlighted boxes and arrows, must not exceed 50 milliseconds. This metric is based on the conclusion in cognitive psychology research regarding the multi-sensory integration time window of approximately 100 milliseconds, with a 50ms engineering tolerance reserved.

[0149] The above synchronization is ensured by the following mechanisms: 1) The AR glasses main control module adopts real-time operating system task scheduling, allocating audio rendering and graphics rendering to the same priority thread and sharing the same hardware clock reference; 2) If the estimated synchronization error exceeds 50ms due to excessive instantaneous computing load, the system automatically executes a degradation strategy: prioritizing the real-time output of spatial audio, while delaying the display of AR visual elements to the next frame, and compensating for the user's perception continuity by increasing the continuous display time of visual elements, such as from 1 second to 1.5 seconds. Degradation events are recorded for subsequent optimization of local computing resource allocation.

[0150] Step 160: Throughout the task execution, the user's state level is assessed in real time by comparing the deviation between the user's actions and the expected sequence, as well as physiological sensor data, and multi-level auxiliary interventions adapted to the state level are triggered; wherein, the state level includes at least confusion, forgetfulness, or anxiety.

[0151] In this invention, the main control module objectively and quantitatively assesses the user's real-time cognitive state by collecting and analyzing deviation data between the user's actions and the expected sequence, as well as physiological sensor data. The definitions and compositions of these two types of data are as follows: The deviation data between user actions and the expected sequence refers to the quantitative difference calculated after the main control module, through cameras, depth sensors, and inertial measurement units, collects the actual user action flow during task execution and compares it with the predefined expected atomic operations in the executable task sequence corresponding to the current task, performing a multi-dimensional comparison. This data specifically includes the following measurable dimensions: Object deviation refers to whether the actual target item operated by the user matches the expected target item. The main control module uses computer vision to identify the item grasped or manipulated by the user's hand and compares it with the expected semantic node ID in the environmental semantic map.

[0152] Spatiotemporal path deviation refers to the degree to which the trajectory, speed, and destination of a user's movement or manipulation of an item match the expected path and destination. For example, whether the path to the target location is circuitous or meandering, or whether the item has been placed in the wrong area.

[0153] Sequence deviation refers to the order in which the user executes atomic operations, and whether this violates the logical order defined in the task sequence, such as attempting to pour medicine without opening the bottle cap.

[0154] Timing deviation is the ratio or difference between the actual time a user takes to complete each atomic operation and the user's personalized expected time baseline based on their "historical operation capability profile." This is a key indicator for judging hesitation and forgetting.

[0155] Physiological sensor data refers to signals that reflect a user's autonomic nervous system activity and arousal level, collected and pre-processed in real time by biosensors integrated into hardware devices, such as the temples of AR glasses or smart wristbands. This data provides objective physiological indicators of a user's internal cognitive load and emotional state, primarily including: Heart rate variability refers to the small fluctuations between consecutive heartbeats. A decrease in the power of specific frequency bands (such as the high-frequency components reflecting parasympathetic activity) is usually associated with high concentration, cognitive effort, or psychological stress.

[0156] Electrodermal activity, referring to changes in the conductivity of the skin, is related to sweat gland activity and is a sensitive indicator of emotional arousal and cognitive stress. Elevated signal amplitude often occurs during moments of confusion, frustration, or anxiety.

[0157] Heart rate is the number of times the heart beats per unit of time. An abnormally high heart rate at rest can serve as an auxiliary indicator of physiological stress.

[0158] like Figure 8 As shown, step 160, which involves real-time assessment of the user's status level and triggering multi-level auxiliary interventions, includes the following: Step 1601: Real-time acquisition of deviation data between user actions and expected sequences, as well as physiological response data; Specifically, the main control module performs parallel real-time acquisition of two types of data: first, action-time deviation data, which continuously tracks the user's actual operations through cameras, depth sensors, etc., and compares them in multiple dimensions with the currently guided expected atomic operation sequence. This includes whether the operation object is correct, whether the action path is reasonable, whether the step order is reversed, and the ratio of the actual time spent on each step to the personalized expected time. Second, physiological response data, which simultaneously collects signals such as the user's heart rate variability and skin conductance activity through biosensors integrated into the device, such as PPG and GSR. These signals are objective physiological indicators reflecting cognitive load and emotional arousal. The synchronization of these two types of data ensures the comprehensiveness of the assessment.

[0159] Step 1602: Input the data into the state assessment model and output a comprehensive state level, which includes at least: mild hesitation, clear forgetting or confusion, significant anxiety or persistent errors; Specifically, the main control module inputs the deviation data and physiological data collected in step 1601 into a pre-trained state assessment model.

[0160] The state evaluation model employs a Temporal Graph Convolutional Network (T-GCN). Its specific structure is as follows: The input layer takes a 60-second time window as input, with a sampling rate of 1 Hz and a feature dimension of 10 at each time step. Features include: Action deviation characteristics (4-dimensional), object deviation flag (0 / 1), path deviation amount (normalized), sequence deviation flag (0 / 1), and time sequence deviation ratio (actual time consumption / expected time consumption).

[0161] Physiological characteristics (4 dimensions): RMSSD of HRV, high-frequency power of HRV, mean SCL, and skin conductance response count.

[0162] Interaction features (2D), help-seeking flag (0 / 1), and voice confusion flag (0 / 1).

[0163] The graph convolutional layer first constructs a static graph with 10 nodes, each corresponding to one of the 10 features. The edge weights are based on the Pearson correlation coefficients between features in the training set. A two-layer graph convolutional network is used, with each layer having an output dimension of 64 and the activation function being ReLU.

[0164] The temporal modeling layer takes the graph convolution output (sequence length 60, dimension 64) as input to a single-layer LSTM with 128 hidden units and outputs the hidden state of the last time step.

[0165] The output layer is a fully connected layer (128 → 3) with Softmax activation, outputting the probability of three state levels: mild hesitation, clear forgetting or confusion, and significant anxiety or persistent error.

[0166] The training method is as follows: In the acquired cognitive load assessment data from multiple patients, state levels were additionally labeled. Labeling was performed on a per-atomic-operation basis, independently judged by three clinicians based on user behavior videos, physiological signals, and post-interview assessments, with the majority consensus result being used. A total of 2000 valid samples were obtained, divided into training, validation, and test sets in a 7:1.5:1.5 ratio.

[0167] The loss function used is weighted cross-entropy loss, assigning weights [1.0, 1.5, 2.0] to the three classes. The optimizer is Adam, with a learning rate of 1e-3 and weight decay of 1e-5.

[0168] In this embodiment, the cognitive load index and the state level are two assessment outputs with different granularities and purposes. They work together but have distinct responsibilities: The cognitive load index is typically a continuous value of 0-1, or three levels: low, medium, and high. It is mainly used for proactive adaptation in the task planning phase, namely, adjusting the task decomposition granularity, merging or splitting atomic operations, and inserting checkpoints in step 1404. Its update frequency is once every 5 seconds or once after each atomic operation is completed.

[0169] Status levels, including mild hesitation / clear forgetfulness or confusion / significant anxiety or persistent errors, are primarily used for reactive interventions in the execution phase, i.e., triggering different levels of assistive measures in step 1603. They are updated in real-time, for example, assessed once per second.

[0170] When the two assessments conflict—for example, a low cognitive load index but a significant anxiety level—the level of anxiety takes precedence, triggering a Level 3 intervention because physiological and behavioral bias indicators better reflect the user's immediate difficulty. Conversely, if the cognitive load index is high but the level of mild hesitation, no Level 2 or higher intervention will be triggered; only the subsequent task decomposition strategy will be adjusted. The specific joint decision-making rule can be expressed as: If the state level is "significant anxiety or persistent error" → Level 3 intervention else if the status level is "clearly forgotten or confused" → Level 2 intervention Otherwise, if the cognitive load index is greater than 0.7 and the state level is "mild hesitation", then proceed with Level 1 intervention and further refinement of subsequent steps. Otherwise, if the cognitive load index > 0.8, proceed with Level 1 intervention (preventative suggestion). else → No intervention, only data recording Step 1603: Trigger a matching intervention based on the status level; the intervention includes Level 1 intervention, Level 2 intervention, and Level 3 intervention; Level 1 intervention provides non-intrusive prompts; Level 2 intervention provides concise step prompt cards and voice repetition; Level 3 intervention pauses the task and initiates structured dialogue guidance. Specifically, based on the assessed state level, the main control module automatically triggers the most suitable auxiliary measures from a predefined multi-level intervention strategy library: Level 1 intervention targets mild hesitation by fine-tuning the dynamic characteristics of the current AR guidance elements, such as increasing the pulse frequency of the highlighted box or playing a short, non-verbal encouraging prompt, providing a textless, mild reminder. Level 2 intervention targets clear forgetting or confusion by dynamically generating a minimalist step-by-step prompt card in the off-center area of ​​the AR interface, containing only icons and 1-2 keywords, and repeating the core instructions of the current atomic operation through spatial audio at a calm, slow pace, providing a clear cognitive scaffold. Level 3 intervention targets significant anxiety or persistent errors by proactively pausing the current task flow and initiating a structured conversational guidance in the center of the AR interface, providing limited follow-up operation options to the user in the form of voice multiple-choice questions, such as "A. Demonstrate again; B. Skip this step," aiming to break the error loop and guide the user back to the correct path.

[0171] Step 1604: Record the intervention effect; the intervention effect is used to optimize the assessment and intervention strategy for this user.

[0172] Specifically, after each intervention is triggered, the main control module continuously monitors the user's recovery performance data over a subsequent period, such as whether the user returned to the correct operation after being prompted, whether physiological indicators tended to level off, and whether subsequent steps were completed smoothly. The triggering conditions, types, and subsequent effects of these interventions are recorded in detail. Using this data, the main control module periodically optimizes two key aspects: first, it adjusts the state assessment threshold for the user to better match their individual response patterns; second, it optimizes the intervention strategy matching rules. For example, if it finds that a user responds poorly to voice repetition but well to visual icon cue cards, then visual intervention strategies will be prioritized in similar situations in the future. Through this closed-loop learning mechanism, the auxiliary behavior of the main control module is continuously improved.

[0173] Furthermore, the method also includes: using the user's historical operation data and physiological data to perform periodic incremental training and fine-tuning on the intent understanding model, task planning logic and state evaluation model, and updating the optimized parameters to the user device.

[0174] Specifically, the main control module periodically, such as daily or weekly, uploads anonymized historical operation data generated locally by the user. This data includes task success / failure records, time taken for each step, error types, and synchronously collected physiological data, to the cloud analysis module via a secure channel. In the cloud, this newly generated data is used for incremental learning and fine-tuning of the three core intelligent modules. Fine-tuning includes the following: 1. For the intent understanding model, new voice interaction data, especially those with clarified and confirmed samples, are used to enhance the model's understanding of users' individual expression habits; 2. For the task planning logic, the decision rules or model parameters for task decomposition, step merging, and checkpoint insertion are optimized based on the user's actual performance in completing various tasks; 3. For the state evaluation model, the accuracy of its evaluation is improved by using labeled real-state behavioral-physiological data pairs inferred from the final task results. After optimization, a lightweight model parameter difference package is generated and silently updated to the user's client device. This allows each user's main control module to increasingly align with their unique cognitive decline curve and behavioral patterns over time, realizing the evolution from a general aid to a dedicated partner.

[0175] Furthermore, the method operates on a client-server collaborative architecture: the client device is responsible for sensor data acquisition, AR rendering, and real-time guidance; the server is responsible for running the intent understanding model, maintaining the knowledge graph and user profile, and performing complex task planning; the client and server interact with data and synchronize instructions through an encrypted channel.

[0176] Specifically, the client, representing the user's smart AR device, undertakes tasks with extremely high real-time requirements: including raw data acquisition from multiple sensors, real-time localization and mapping of the environmental semantic map, low-latency rendering of AR content, and state monitoring and basic guidance based on a local lightweight model. The server, on the other hand, is responsible for intensive computation and global data management: running computationally complex intent understanding models trained on large-scale corpora; centrally storing, maintaining, and updating personalized item knowledge graphs and historical operational capability profiles for all users; and performing time-consuming complex task planning and replanning calculations. The two ends collaborate efficiently through an encrypted communication channel: the client uploads necessary sensor summaries, voice features, and other data; after completing deep understanding and planning, the server sends a structured set of guidance instructions to the client for execution. This architecture ensures both a seamless and immediate user experience and fully leverages the powerful computing power and data aggregation advantages of the cloud.

[0177] Furthermore, the method also includes: remotely viewing a simplified view of the environmental semantic map, user task status, and activity reports; pre-associating personalized voice memory prompts for specific semantic nodes; and activating a remote assistance mode when an alert is issued or a user requests assistance, sharing the AR annotations or voice guidance from the family member's terminal to the user's AR interface in real time.

[0178] Specifically, in remote assistance mode, such as via a mobile app or webpage, authorized family members or caregivers can use the service. This function includes: 1. Remote non-invasive monitoring: Family members can log in to view a simplified view of the environmental semantic map presented in a privacy-preserving manner, such as a heatmap of object locations, statistics on the user's task completion status, and a summary report of daily activities, facilitating understanding of the patient's daily routines and cognitive fluctuations. 2. Personalized memory anchor configuration: Family members can associate specific semantic nodes in the environmental semantic map, such as old photos or souvenirs, or record a personalized voice prompt. When the user looks at the object, the main control module will play this prompt to help evoke contextual memories. 3. Remote real-time intervention assistance: When the main control module triggers a high-risk warning, such as when the user approaches a dangerous item or actively sends a distress signal, the family member's terminal will receive a notification and can activate the remote assistance mode with one click. In this mode, family members can make AR annotations on the synchronized scene view on their own terminals, such as drawing circles, arrows, or making direct voice calls. Their annotations and voices will be superimposed and presented in the user's AR vision and hearing with low latency and high fidelity, realizing hands-on guidance across space, providing immediate human support in emergencies, and greatly enhancing the safety net and humanistic care of the main control module.

[0179] By applying the AI-assisted item management method that combines AR and semantic understanding provided in this embodiment of the invention, an environmental semantic map that integrates geometric, semantic, and security attributes is constructed and associated with a personalized item knowledge graph. This lays a cognitive foundation for all subsequent precise interactions, far exceeding shallow perception solutions based on Bluetooth tags or fixed tags.

[0180] A specially trained intent understanding model tailored to the language characteristics of patients with cognitive impairment, combined with multimodal contextual fusion reasoning, can effectively parse and disambiguate ambiguous and disjointed speech commands. Proactive, non-invasive AR information cues and multi-sensory collaborative spatial guidance align with the user's natural attention mechanism, significantly reducing the learning and operational burden and resolving the failure issues of general voice assistants and simple AR applications in this scenario.

[0181] Based on historical operational ability profiles and real-time cognitive load assessments, the system dynamically adjusts task decomposition granularity and guidance strategies to achieve personalized adaptation for each individual. Through real-time monitoring of behavioral and physiological signal status assessments and a tiered intervention mechanism, the system can provide timely assistance when users encounter difficulties, forming an intelligent closed loop of perception-planning-guidance-monitoring-adjustment, effectively improving the timeliness, safety, and user independence of assistance.

[0182] The client-server architecture balances computational load and real-time requirements, ensuring the system's usability. The introduction of family collaborative management features not only enables remote monitoring and safety alerts, but also seamlessly integrates emotional support from family members into the assistance process through personalized memory prompts and real-time remote assistance, enhancing the system's social acceptance and sense of warmth.

[0183] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0184] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0185] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An AI-assisted item management method combining AR and semantic understanding, characterized in that, The method includes: Construct and update an environmental semantic map in real time. The environmental semantic map consists of multiple semantic nodes, each of which corresponds to a real item and includes at least the semantic tag and security attributes of the item. When a user looks at or approaches the item corresponding to the semantic node, the basic information of the item is displayed through the AR interface. The system receives the user's voice request, uses an intent understanding model trained on a corpus of language disordered by cognitive impairment, and combines the environmental semantic map to perform context fusion reasoning, parsing out the actionable user intent and identifying a unique target object. Based on the user's intent, the real-time status of the target object, and the user's historical operational capability profile, an executable task sequence comprising multiple atomic operations is generated; wherein, the task complexity and decomposition granularity of the executable task sequence are matched with the user's current cognitive load; Guidance information is output through a combination of spatial audio and AR visual channels registered in the real space; the visual guidance adopts a progressive revelation strategy, dynamically generating the next step guidance only after the user completes the previous step. Throughout the task execution, the user's state level is assessed in real time by comparing the deviation between the user's actions and the expected sequence, as well as physiological sensor data, and multi-level auxiliary interventions are triggered in accordance with the state level; wherein the state level includes at least confusion, forgetfulness, or anxiety.

2. The method according to claim 1, characterized in that, The construction and real-time updating of the environmental semantic map specifically includes: The system uses an integrated RGB-D camera and inertial measurement unit to simultaneously acquire color images and depth images of the environment, as well as inertial data of the equipment. Based on visual inertial odometry and semantic SLAM algorithms, the color images and inertial data are processed to construct and update a three-dimensional geometric map of the environment in real time, and output the real-time six-degree-of-freedom pose of the device in the world coordinate system. The color image is analyzed frame by frame using an instance segmentation neural network to identify each item instance and output the category label and pixel-level segmentation mask for each instance. The segmentation mask of each instance is pixel-level aligned and fused with the synchronously acquired depth image; combined with the camera intrinsic parameter matrix and the real-time six-degree-of-freedom pose, the pixels in the mask are back-projected into three-dimensional space to generate a three-dimensional point cloud cluster for each item instance. Spatial clustering and geometric analysis are performed on the 3D point cloud clusters of each item instance to calculate the 3D geometric information of the item, which includes: 3D axial bounding box, centroid coordinates, main orientation and occupied spatial volume; The visual feature vector of each item instance is extracted, and the visual feature vector, together with the category label and three-dimensional geometric information, is matched and retrieved with a pre-set personalized item knowledge graph to obtain the matching result; the personalized item knowledge graph stores the general semantic attributes of the item, the security level classification, and the user's custom semantic tags or memory anchors for the item. Based on the matching results, each item instance is attached with a corresponding semantic tag and security attribute to form a semantic node; wherein, the semantic tag includes at least the item's standard name and functional category, and the security attribute includes at least a risk level identifier; An environmental semantic map is formed based on the spatial relationship of the centroid coordinates of each semantic node in the world coordinate system. As new data is collected, the confidence and location information of each semantic node are updated in real time through multi-view geometric consistency test and Bayesian filtering method.

3. The method according to claim 1, characterized in that, When a user looks at or approaches an object, the basic information of the object is displayed through an AR interface, specifically including: The user's three-dimensional gaze vector is obtained by eye-tracking technology. The spatial intersection of the three-dimensional gaze vector with the three-dimensional bounding box of each semantic node in the environmental semantic map is calculated. If the gaze continues for more than a first time threshold, it is determined to be a gaze. The Euclidean distance between the user's location and the centroid of each semantic node is calculated in real time. If the distance is less than the preset second distance threshold, the user is considered close. When a gaze or proximity is detected, query the complete semantic tag and security attributes corresponding to the semantic node. Based on the current task context, user cognitive load, and item safety attributes, key basic information is dynamically filtered out using a priority algorithm. The priority algorithm ensures at least that: high-risk items are given priority in displaying warning messages, and items strongly related to the current task are given priority in displaying task instructions. The selected basic information is overlaid and displayed near the real spatial location of the corresponding item using an AR rendering engine in a spatially locked manner. The information display has a delayed trigger and automatic fade-out mechanism, and the display of information on irrelevant items is suppressed when the user is performing the main task.

4. The method according to claim 1, characterized in that, The process of receiving a user's voice request involves using an intent understanding model trained on a corpus of language disordered by cognitive impairment, combined with contextual fusion reasoning based on the environmental semantic map, to parse out the actionable user intent and determine a unique target object. Specifically, this includes: The collected user voice data is denoised and then converted into text using a speech recognition model trained on pathological speech corpus. The text is input into an intent understanding model fine-tuned with a cognitive impairment dialogue corpus to obtain preliminary analysis results including intent categories and semantic slots that may have ambiguous referents. Construct a dynamic context vector, whose data includes: the semantic node corresponding to the user's current gaze focus, a list of semantic nodes in the neighborhood space, the current task status, and short-term interaction history; The preliminary analysis results and the dynamic context vector are input into the inference engine. By calculating the correlation between fuzzy references and each candidate semantic node in the semantic, spatial, and temporal dimensions, the fuzzy references are disambiguated and anchored to a specific semantic node, thereby determining a unique target object. When the reasoning confidence is below the threshold, multiple candidate semantic nodes are labeled through the AR interface and a clarification question is initiated in conjunction with voice.

5. The method according to claim 1, characterized in that, The generation of the executable task sequence specifically includes: Retrieve a task template corresponding to the user intent from a hierarchical task knowledge base. The template consists of multiple atomic operations with a logical order. Based on the user's historical operational ability profile, a predicted difficulty score is added to each atomic operation in the task template; Real-time assessment of the user's current cognitive load index; Based on the current cognitive load index, the atomic operation sequence is dynamically optimized: when the load index is low, low-difficulty operations are merged to simplify the sequence; when the load index is high, high-difficulty operations are finely decomposed; and confirmation checkpoints are dynamically inserted according to the user's error pattern. Output an optimized sequence of executable tasks and replan online based on the user's actual performance during execution.

6. The method according to claim 1, characterized in that, The method of outputting guidance information through the coordinated use of spatial audio and AR visual channels specifically includes: Generate a multimodal guidance instruction set for each atomic operation, including spatial audio scripts and AR visual parameters; The AR glasses internally run a finite state machine synchronized with the sequence of executable tasks; The state machine only transitions to the next state and triggers the rendering engine to output guidance information for the next step when the sensor data verifies that the user has completed the current atomic operation. Spatial audio provides sound guidance from the direction of the target, while AR visual elements are precisely registered at the target location in the real space.

7. The method according to claim 1, characterized in that, The real-time assessment of user status levels and triggering of multi-level auxiliary interventions specifically includes: Real-time acquisition of deviation data between user actions and expected sequences, as well as physiological response data; The data is input into the state assessment model, which outputs a comprehensive state level, which includes at least: mild hesitation, clear forgetting or confusion, significant anxiety or persistent errors; Interventions are triggered based on the status level; these interventions include Level 1, Level 2, and Level 3 interventions; Level 1 interventions provide non-intrusive prompts; Level 2 interventions offer concise step-by-step prompts and voice repetitions; Level 3 interventions pause the task and initiate structured dialogue guidance. Record the intervention effect; the intervention effect is used to optimize the assessment and intervention strategy for this user.

8. The method according to claim 1, characterized in that, The method further includes: Using the user's historical operation data and physiological data, the intent understanding model, task planning logic, and state evaluation model are periodically incrementally trained and fine-tuned, and the optimized parameters are updated to the user device.

9. The method according to claim 1, characterized in that, The method operates in a client-server collaborative architecture: The client device is responsible for sensor data acquisition, AR rendering, and real-time guidance; the server is responsible for running the intent understanding model, maintaining the knowledge graph and user profile, and performing complex task planning. The client and server exchange data and synchronize commands through an encrypted channel.

10. The method according to claim 1, characterized in that, The method further includes: Through the collaborative management interface, instructions from family member terminals can be received and executed, including: remotely viewing a simplified view of the environmental semantic map, user task status, and activity reports; pre-associating or recording personalized voice prompts for specific semantic nodes; and... Upon receiving a high-risk warning or a user's request for help, the remote real-time assistance mode is activated, enabling AR annotations or voice guidance initiated by family members' terminals to be overlaid and displayed on the user's AR interface in real time.

Images