Personal memory map generation system based on intelligent glasses

Abstract

The invention discloses a personal memory map generation system based on intelligent glasses. The system comprises a multi-source data acquisition layer, a memory map generation engine and an intelligent analysis and interaction system. The multi-source data acquisition layer acquires camera visual information and GPS + UWB + SLAM data through an information acquisition module of the intelligent glasses; the memory map generation engine constructs a three-dimensional semantic map and a spatio-temporal index architecture based on the collected multi-source data; and the intelligent analysis and interaction system identifies an object moving track, interacts with user questions and answers, and performs abnormal behavior detection based on the track. The precision and comprehensiveness of environment perception are improved through multi-source data fusion, the scene understanding ability and safety protection of the system are enhanced through track recognition and anomaly detection by the intelligent analysis module, and the interaction experience of a user and a memory map is optimized through a question and answer interaction function. Intelligent and personalized memory management and environment perception solutions are provided for individuals, and application innovation of wearable equipment in the field of memory assistance and intelligent security and protection is promoted.

Description

Technical Field

[0001] This invention relates to a map generation system, and more specifically to a personal memory map generation system based on smart glasses. Background Technology

[0002] In the digital age, the recording and management of personal memories faces the challenges of fragmentation and unstructured nature inherent in traditional text and image-based methods. This results in a lack of systematic connection in memory storage, low retrieval efficiency, and insufficient accuracy, failing to meet people's needs for complete and traceable memory recall. Simultaneously, the reliance on single information dimensions for daily behavior recall, coupled with the inability to dynamically link to physical space, creates technological bottlenecks in natural language-based intelligent question answering and scene recall. For special populations such as Alzheimer's patients and those with attention deficit hyperactivity disorder (ADHD), existing methods have gaps in automated behavior tracking and anomaly alerts, making it difficult to effectively compensate for the cognitive and behavioral management challenges posed by short-term memory loss. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a personal memory map generation system based on smart glasses that can solve one or more of the above-mentioned problems.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a personal memory map generation system based on smart glasses, characterized in that it includes: Multi-source data acquisition layer, used to acquire visual information from the camera through the information acquisition module of the intelligent agent, GPS+UWB+SLAM; The memory map generation engine communicates with the multi-source data acquisition layer to build a 3D semantic map and spatiotemporal index architecture based on information collected by the multi-source data acquisition layer. The intelligent analysis and interaction system communicates with the multi-source data acquisition layer and the memory map generation engine to identify the movement trajectory of objects, interact with users through question and answer, and perform abnormal behavior detection based on the identified object movement trajectory.

[0005] As a further improvement of the present invention, the multi-source data acquisition layer includes: An environmental perception module, which includes a camera, a nine-axis IMU and a barometer, is used to acquire RGB-D images through the camera, acquire attitude correction information through the nine-axis IMU and identify floors through the barometer. The positioning enhancement module includes a GPS module, a UWB module, and a SLAM module, which acquires outdoor positioning information through the GPS module, acquires indoor positioning information through the UWB module, and maps areas without signal through the SLAM module. The attitude data acquired by the nine-axis IMU is used to transform the point cloud data in the camera coordinate system to the world coordinate system. The transformation formula is as follows: \(P_{world}=R\cdotP_{camera}+T\) In the formula, \(P_{camera}\) represents the point cloud coordinates in the camera coordinate system, and \(T\) represents the translation vector.

[0006] As a further improvement of the present invention, the specific steps of the memory map generation engine in generating the memory map are as follows: Step 1: By fusing multi-view images and depth information using neural radiation field technology, a 3D environment model with semantic labels is generated to complete the construction of a 3D semantic map. Step two: A four-dimensional spatiotemporal database, including longitude λ, latitude φ, altitude h, and timestamp t, is used to store events. Spatial coordinates (x, y, z) are then converted to latitude, longitude, and altitude using the following formula: Where R is the Earth's radius.

[0007] As a further improvement of the present invention, the intelligent analysis and interaction system includes: A memory reasoning engine is used to automatically identify the movement trajectory of objects based on a memory association model of graph neural networks; A multimodal dialogue interface is used to provide an interface for question-and-answer interaction with users; The storage module adopts a "hot data caching + cold data compression" strategy. For scene data exceeding 30 days, the number of vertices (n) is reduced by simplifying the polygon mesh for storage.

[0008] As a further improvement of the present invention, the storage module reduces the number of vertices (n) by simplifying the polygonal mesh in the following specific way: Assuming the original model has n_0 vertices and the compressed model has n_1 vertices, the compression ratio (rho) is calculated using the following formula: \(\rho=\frac{n_1}{n_0}\).

[0009] As a further improvement of the present invention, the specific steps of the memory reasoning engine to automatically identify the movement trajectory of an object are as follows: Step 3: Abstract the elements such as objects, positions, time, and actions into nodes and edges in a graph structure to form a memory association graph neural network; Step four: Analyze the relationships between nodes using a memory association model to identify the object's movement trajectory; Step 5: Complete the abnormal behavior detection by comparing historical behavior patterns.

[0010] As a further improvement to the present invention, the specific steps of analyzing the association relationship between nodes and identifying the object's movement trajectory in step four are as follows: Step 41, perform time series correlation analysis, specifically: Traverse the spatiotemporal coordinates of object nodes in timestamp order and detect position changes; Query the location transfer records of adjacent time points using a four-dimensional spatiotemporal database; Step 42 involves learning behavioral patterns, specifically: The graph network model is trained based on historical data to learn common behavior patterns, which are placement → movement → final position. Step 43 involves identifying abnormal trajectories, specifically: By comparing the current trajectory with historical patterns, movements that do not conform to normal logic are detected, and anomaly alerts are triggered.

[0011] The beneficial effects of this invention are that by integrating GPS positioning, map information, and first-person perspective images captured by smart glasses, it accurately maps and efficiently stores the information seen by the user onto a map, effectively solving the problems of fragmented personal memory and difficulty in retrieval. Users can rely on the constructed personal memory map to quickly recall historical scenes and behavioral information. For example, when searching for the location of items at home, the system can accurately locate the last time and place of the item based on image recognition and spatiotemporal indexing. Simultaneously, by combining a large model to achieve natural language interaction, users only need to ask questions such as "What was different on my way to work today?" or "Did I lock the door?" The system can automatically analyze and compare historical data, accurately identify anomalies, and provide answers, significantly improving the efficiency and accuracy of daily behavior recall. Furthermore, the layered storage optimization strategy reduces data redundancy, saving storage space while ensuring data availability, providing users with a convenient and efficient personal memory management experience. Attached Figure Description

[0012] Figure 1 A block diagram of a personal memory map generation system based on smart glasses. Detailed Implementation

[0013] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0014] Reference Figure 1 As shown in this embodiment, a personal memory map generation system based on smart glasses specifically includes the following: S1 Multi-Source Data Acquisition Layer Environmental perception module: camera (RGB-D image acquisition), 9-axis IMU (attitude correction), barometer (floor recognition). Positioning enhancement module: GPS (outdoor) + UWB (indoor high-precision positioning) + SLAM (mapping in areas without signal). The attitude data (rotation matrix R) acquired by the IMU is used to transform the point cloud data in the camera coordinate system to the world coordinate system. The transformation formula is as follows: P_{world}=R\cdotP_{camera}+T\ In the formula, \(P_{camera}\) represents the point cloud coordinates in the camera coordinate system, and \(T\) represents the translation vector.

[0015] S2 Memory Map Generation Engine 3D semantic map construction: By fusing multi-view images and depth information through Neural Radiation Field (NeRF) technology, a 3D environment model with semantic labels is generated (e.g., "Sofa-Living Room-2023 / 10 / 05 08:30").

[0016] Spatiotemporal index architecture: Events are stored using a four-dimensional spatiotemporal database (longitude λ, latitude φ, altitude h, timestamp t). The conversion relationship between spatial coordinates (x, y, z) and latitude / longitude / altitude is as follows: Where R is the Earth's radius, and searches can be performed using a combination of "location + time + semantics". For example, "find all records of using a kitchen knife in October 2023".

[0017] S3 Intelligent Analysis and Interaction System Memory Reasoning Engine: A memory association model based on graph neural networks: automatically identifies the movement trajectory of an object, from "keys placed on the dining table → moved to the coffee table → finally located in the entryway." Specifically, it abstracts elements such as objects, locations, times, and actions as nodes and edges in a graph structure, forming a memory association graph neural network. Node type Object nodes: such as entities like "key", "dining table", "coffee table", and "entryway".

[0018] Spatiotemporal node: includes location (latitude and longitude, indoor coordinates) and timestamp (e.g., "2023 / 10 / 05 08:30").

[0019] Action nodes: such as behavior tags like "place", "move", "pick up" etc.

[0020] Definition of edge Spatiotemporal related edges: connect objects to their corresponding spatiotemporal nodes (e.g., "key" → "dining table - 2023 / 10 / 05 08:30").

[0021] Behavioral association edges: describe changes in the state of an object (e.g., "place" → "the key moves from the dining table to the coffee table").

[0022] By analyzing the relationships between nodes using a memory association model, the temporal logic of the object's movement trajectory can be identified. 1) Temporal correlation analysis Traverse the spatiotemporal coordinates of object nodes in order of timestamps and detect position changes (e.g., the "key" moves from the "dining table" coordinates to the "coffee table" coordinates).

[0023] Query the location transfer records of adjacent time points using a four-dimensional spatiotemporal database (longitude λ, latitude φ, altitude h, timestamp t).

[0024] 2) Behavioral pattern learning The graph network model is trained based on historical data to learn common behavior patterns (such as the sequence of "placement → movement → final position").

[0025] For example: if it is detected that the "key" is marked as "placed" at the "dining table" position, then marked as "moved" at the "coffee table" position, and finally stabilized at the "entryway" position, it is inferred that "the key was moved from the dining table to the entryway, stopping at the coffee table along the way".

[0026] 3) Abnormal trajectory recognition By comparing the current trajectory with historical patterns, detect movements that do not conform to normal logic (such as the "key" suddenly teleporting from the "dining table" to the "outdoors") and trigger an anomaly alert.

[0027] Multimodal dialogue interface: Supports voice interaction: Enables question-and-answer interaction with users. Abnormal behavior detection: By comparing historical behavior patterns (such as average door locking time 17:30±5 minutes), a vibration alert is triggered when the door lock status is not updated at 18:00. S4 Tiered Storage Optimization Solution A strategy of "hot data caching + cold data compression" is adopted. For scene data exceeding 30 days, the number of vertices (n) is reduced by simplifying the polygon mesh. Assuming the original model has n_0 vertices and the compressed model has n_1 vertices, the compression ratio (rho) is calculated as follows: \(\rho=\frac{n_1}{n_0}\) When converted to a low-poly lightweight model, the storage space can be reduced by up to \(1-\rho\), achieving an 80% storage optimization effect.

[0028] Based on the above system, this embodiment provides the following application examples: 1) Example 1: Home Item Tracking System User's account: "Where was the last place I saw my wedding ring?" System response: Extract the keyword "wedding ring" → Use the item recognition model to search for all images containing the word "ring". Based on the memory map, the last time it appeared was at the master bathroom sink (2023 / 9 / 2022:15). Answer user questions.

[0029] Example 2: Commuting Anomaly Analysis A user asked, "Was anything unusual on my way to work today?" Automatic system: Compare the current day's commuting route with historical routes (GPS trajectory overlap 98%). A traffic jam was detected on Jingsan Road at 08:12, lasting for 2 minutes (due to a traffic accident). Generate a comparison report: average commute time 35 minutes → daily commute time 52 minutes, analyze and respond to abnormal events.

[0030] Example 3: Cognitive Impairment Assistance When it is detected that the user has opened and closed the refrigerator three times in a row (suspected of forgetting to retrieve the item): Retrieving memory fragments from the previous 30 minutes, the system identified that the user had placed medication in the refrigerator. Prompt: "Do you need to retrieve the insulin stored at 11:00?"

[0031] In summary, this embodiment presents a personal memory map generation system based on smart glasses: Solving the challenges of digital storage and retrieval of personal memories: By constructing a traceable spatiotemporal memory network through multimodal data fusion (GPS + image + depth information + timestamp), it overcomes the problems of fragmented and unstructured traditional text / image records.

[0032] Improve the accuracy and efficiency of daily behavior backtracking: Combine 3D environmental reconstruction and semantic analysis technology to dynamically bind user behavior with physical space, supporting intelligent question answering and scene backtracking based on natural language.

[0033] Assisting in the cognitive and behavioral management of special populations: Providing automated behavior tracking and abnormal alerts for Alzheimer's patients and people with attention deficit disorder to compensate for short-term memory loss.

[0034] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A personal memory map generation system based on smart glasses, characterized in that: include: A multi-source data acquisition layer is used to acquire visual information from the camera through the information acquisition module of the intelligent agent, using GPS + UWB + SLAM. The memory map generation engine communicates with the multi-source data acquisition layer to build a 3D semantic map and spatiotemporal index architecture based on information collected by the multi-source data acquisition layer. The intelligent analysis and interaction system communicates with the multi-source data acquisition layer and the memory map generation engine to identify the movement trajectory of objects, interact with users through question and answer, and perform abnormal behavior detection based on the identified object movement trajectory.

2. The personal memory map generation system based on smart glasses according to claim 1, characterized in that: The multi-source data acquisition layer includes: An environmental perception module, which includes a camera, a nine-axis IMU and a barometer, is used to acquire RGB-D images through the camera, acquire attitude correction information through the nine-axis IMU and identify floors through the barometer. The positioning enhancement module includes a GPS module, a UWB module, and a SLAM module, which acquires outdoor positioning information through the GPS module, acquires indoor positioning information through the UWB module, and maps areas without signal through the SLAM module. The attitude data acquired by the nine-axis IMU is used to transform the point cloud data in the camera coordinate system to the world coordinate system. The transformation formula is as follows: \( P_{world} = R \cdot P_{camera} + T \) In the formula, \(P_{camera}\) represents the point cloud coordinates in the camera coordinate system, and \(T\) represents the translation vector.

3. The personal memory map generation system based on smart glasses according to claim 1 or 2, characterized in that: The specific steps for the memory map generation engine to generate a memory map are as follows: Step 1: By fusing multi-view images and depth information using neural radiation field technology, a 3D environment model with semantic labels is generated to complete the construction of a 3D semantic map. Step two: A four-dimensional spatiotemporal database, including longitude λ, latitude φ, altitude h, and timestamp t, is used to store events. Spatial coordinates (x, y, z) are then converted to latitude, longitude, and altitude using the following formula: ； Where R is the Earth's radius.

4. The personal memory map generation system based on smart glasses according to claim 1 or 2, characterized in that: The intelligent analysis and interaction system includes: A memory reasoning engine is used to automatically identify the movement trajectory of objects based on a memory association model of graph neural networks; A multimodal dialogue interface is used to provide an interface for question-and-answer interaction with users; The storage module employs a "hot data caching + cold data compression" strategy. For scene data exceeding 30 days, it reduces the number of vertices (n) by simplifying the polygonal mesh.

5. The personal memory map generation system based on smart glasses according to claim 4, characterized in that: The specific method by which the storage module reduces the number of vertices (n) by simplifying the polygonal mesh is as follows: Assuming the original model has n_0 vertices and the compressed model has n_1 vertices, the compression ratio (rho) is calculated using the following formula: \(\rho = \frac{n_1}{n_0}\).

6. The personal memory map generation system based on smart glasses according to claim 4, characterized in that: The specific steps of the memory reasoning engine in automatically recognizing the trajectory of an object's movement are as follows: Step 3: Abstract the elements such as objects, positions, time, and actions into nodes and edges in a graph structure to form a memory association graph neural network; Step four: Analyze the relationships between nodes using a memory association model to identify the object's movement trajectory; Step 5: Complete the abnormal behavior detection by comparing historical behavior patterns.

7. The personal memory map generation system based on smart glasses according to claim 6, characterized in that: The specific steps for analyzing the relationships between nodes and identifying the object's movement trajectory using a memory association model in step four are as follows: Step 41, perform time series correlation analysis, specifically: Traverse the spatiotemporal coordinates of object nodes in timestamp order and detect position changes; Query the location transfer records of adjacent time points using a four-dimensional spatiotemporal database; Step 42 involves learning behavioral patterns, specifically: The graph network model is trained based on historical data to learn common behavior patterns, which are placement → movement → final position. Step 43 involves identifying abnormal trajectories, specifically: By comparing the current trajectory with historical patterns, movements that do not conform to normal logic are detected, and anomaly alerts are triggered.

Images