Wearable device interaction method and device based on memory information, computer device

Abstract

This invention proposes a wearable device interaction method and apparatus, and a computer device based on memory information. The method includes: collecting multimodal raw data through a wearable device; performing event recognition and semantic extraction on the raw data of each modality, and outputting a single-modal semantic description; comprehensively analyzing all single-modal semantic descriptions within the same time period to generate an event chain and event context description; saving an event table, an association information table, and an index table to a database, wherein the event table stores the event chain and event context description, the association information table stores the relationships between events, and the index table establishes a data retrieval channel; automatically generating reminder tasks based on the data in the event table, association information table, and index table, and pushing them to the user when the task triggering conditions are met, wherein the task triggering conditions include at least one of time, location, scenario, and unfinished tasks. This invention achieves efficient and accurate active and passive intelligent interaction, improving user experience and system efficiency.

Description

Technical Field

[0001] This invention relates to the field of smart wearable device technology, and in particular to a wearable device interaction method and apparatus based on memory information, and a computer device. Background Technology

[0002] Wearable devices (such as AI smart glasses) are emerging smart terminals and are gradually taking on more diverse functional needs. With users' growing demand for memory assistance and information retrieval, how to leverage wearable devices to achieve efficient monitoring and intelligent interaction of diverse daily information has become a crucial technological area requiring breakthroughs.

[0003] Currently, conventional information monitoring and interaction solutions used in smart glasses or similar devices have significant limitations. On the one hand, existing solutions mostly focus on single-type data (such as visual or auditory data only), making it difficult to integrate multimodal data such as visual, auditory, location, and environmental information for comprehensive and coordinated capture, resulting in the inability to reconstruct complete and rich scene memories. On the other hand, their data processing is mostly limited to simple storage or basic classification, lacking deep semantic analysis and intelligent interactive linkage. Therefore, it is difficult to proactively and accurately output valuable memory records to users, and it is also unable to flexibly and intelligently respond to different user interaction scenarios (such as proactively triggering reminders or passively asking and answering questions). These shortcomings collectively lead to problems such as high data redundancy, poor memory availability, passive interactive experience, and high device resource consumption in existing systems. Summary of the Invention

[0004] The purpose of this invention is to provide a wearable device interaction method and apparatus, and a computer device based on memory information, to solve the above-mentioned problems. By collecting multimodal information through event triggering, semantic processing and structured storage, it achieves efficient and accurate active and passive intelligent interaction. It has achieved substantial improvements in data collection, storage retrieval, task triggering and system efficiency, thereby enhancing user experience and system efficiency.

[0005] This invention proposes a wearable device interaction method based on memory information, comprising the following steps: Multimodal raw data is collected through wearable devices, including at least one of visual data, auditory data, geographic location data, physiological health data, and environmental data. Perform event recognition and semantic extraction on the raw data of each modality, and output a single-modal semantic description; By comprehensively analyzing all single-modal semantic descriptions within the same time period, an event chain and event context description are generated. The event context description includes at least one of the following: time, location, participants, content description, causal relationship between events, event time sequence, event importance level, context in which the event occurs, and event association information. Save the event table, the association information table, and the index table to the database. The event table is used to store event chains and event context descriptions, the association information table is used to store the relationships between events, and the index table is used to establish data retrieval channels. Reminder tasks are automatically generated based on data from the event table, related information table, and index table, and pushed to the user when the task triggering conditions are met. The task triggering conditions include at least one of time, location, scenario, and unfinished items.

[0006] In one embodiment, an automatic generation of reminder tasks based on data from an event table, a related information table, and an index table specifically includes: Candidate events are obtained by retrieving them from the event table based on preset conditions and using an index table. The preset conditions include at least one of the following: records containing a time field, a location field, or semantic content containing an item to be completed. Semantic analysis is performed on candidate events to identify users' unfulfilled behavioral needs and extract task elements, which include at least one of time elements, geographical location elements, scene elements, and to-do elements. Based on the task generation rules, a reminder task is generated using the extracted task elements.

[0007] In one embodiment, candidate events are obtained by retrieving them from an event table based on preset conditions and using an index table, specifically including: The time, location, and / or semantic content containing records of pending tasks in the preset conditions are mapped to the corresponding index fields of the index table for preliminary retrieval, resulting in an initial set of events that meet each individual condition; Perform a logical intersection operation on the initial event set to filter out events that simultaneously meet all preset conditions as primary candidate events; Based on the association information table, other events that are related to the primary candidate events are obtained; The primary candidate events and their associated events are combined to form the final candidate events.

[0008] In one embodiment, semantic parsing of candidate events is performed to identify unfulfilled user behavioral needs, specifically including: Extract the content description, related information, and participating objects of the event from the event scenario description corresponding to the candidate event, and combine them into semantic text to be parsed; The semantic text is input into a natural language understanding model to identify and label sentence fragments in the text that represent intentions, commitments, or needs but do not contain completion status markers, thereby identifying the user's unfulfilled behavioral needs.

[0009] In one embodiment, after automatically generating reminder tasks based on data from the event table, the associated information table, and the index table, the generated reminder tasks are also saved to the task table, which includes at least one of the following: task identifier, trigger condition, task content, trigger parameter, associated event identifier, task priority, and confidence level.

[0010] In one embodiment, the wearable device interaction method based on memory information further includes: storing an event table in a database according to time to form a time-coherent event chain; For queries submitted by users in natural language, time and / or semantic information is extracted, user intent is parsed, events related to the user intent are retrieved from the event table using the index table, and then converted into natural language and fed back to the user.

[0011] In one embodiment, when collecting multimodal raw data, the raw data of each modality is collected based on preset change conditions.

[0012] In one embodiment, the preset change condition for acquiring visual data is as follows: visual data is compared with the similarity of adjacent image frames through an image change detection algorithm. When the similarity of adjacent frames is less than a preset similarity threshold, visual data acquisition is triggered. When the similarity of adjacent frames of N consecutive frames is greater than or equal to the preset similarity threshold, visual data acquisition is terminated, where N is a positive integer. The process of event recognition and semantic extraction from visual data is as follows: visual data outputs a visual semantic description through image recognition and scene understanding.

[0013] In one embodiment, the preset change condition for collecting auditory data is as follows: auditory data is monitored by a sound activity detection algorithm. When the sound energy exceeds a preset energy threshold or the speech characteristics change, auditory data collection is triggered. When no sound activity is detected for T consecutive seconds, auditory data collection is terminated, where T is a positive integer. The process of event recognition and semantic extraction from auditory data is as follows: auditory data is converted into auditory semantic descriptions through speech transcription and semantic analysis.

[0014] In one embodiment, the preset change condition for collecting geographic location data is: when the spatial displacement of the user's location relative to the previous recorded point exceeds a preset displacement change threshold or the time spent in the same location exceeds a preset stay duration, a new location point is recorded. The process of event identification and semantic extraction from geographic location data is as follows: geographic location data is processed through geographic semantics to output the semantics corresponding to the location information.

[0015] In one embodiment, the preset change condition for collecting physiological health data is: when the physiological health data exceeds a preset health threshold or when the user is detected to be in motion, physiological health data collection is triggered. The process of event identification and semantic extraction from physiological health data is as follows: physiological health data is analyzed by data pattern analysis and medical semantic mapping to output physiological health semantic information.

[0016] In one embodiment, the preset change condition for collecting environmental data is: when the change in environmental parameters exceeds a preset environmental change threshold, environmental data collection is triggered. The process of event identification and semantic extraction from environmental data is as follows: environmental data is processed through semanticization of environmental parameters to output environmental semantic information.

[0017] In one embodiment, comprehensive analysis of all single-modal semantic descriptions within the same time period generates an event chain and event context description, specifically including: By using a cross-modal attention mechanism, all single-modal semantic descriptions within the same time period are aligned and fused, and then extracted and integrated into a set of event context description elements. Construct a logically coherent event chain based on the chronological order and / or causal relationships in the event scenario description elements.

[0018] The present invention also proposes a wearable device interaction device based on memory information, for implementing the wearable device interaction method based on memory information as described above, comprising: The data acquisition module collects multimodal raw data, which includes at least one of visual data, auditory data, geographic location data, physiological health data, and environmental data. The data processing module includes a single-modal semantic processing unit and a multimodal event integration unit. The single-modal semantic processing unit is used to perform event recognition and semantic extraction on the raw data of each modality and output a single-modal semantic description. The multimodal event integration unit is used to comprehensively analyze all single-modal semantic descriptions within the same time period and generate an event chain and an event context description. The event context description includes at least one of the following: time, location, participants, content description, causal relationship between events, event time sequence, event importance level, context in which the event occurred, and event association information. The data storage module stores an event table, an association information table, and an index table. The event table stores event chains and event context descriptions, the association information table stores the relationships between events, and the index table is used to establish data retrieval channels. The task generation module automatically generates reminder tasks based on data from the event table, related information table, and index table; The intelligent reminder module pushes a notification to the user when a task trigger condition is met. The task trigger condition includes at least one of time, location, scenario, and unfinished items.

[0019] In one embodiment, the task generation module is configured to retrieve candidate events from an event table based on preset conditions and using an index table. The preset conditions include at least one of records containing a time field, a location field, and semantic content containing items to be completed. The module performs semantic parsing on the candidate events to identify the user's unfinished behavioral needs, extracts task elements, and the task elements include at least one of time elements, geographical location elements, scene elements, and to-do elements. Based on the extracted task elements, the module generates a reminder task according to task generation rules.

[0020] In one embodiment, the wearable device interaction device based on memory information further includes a query and retrieval module, which extracts time and / or semantic information from the query request made by the user through natural language, parses the user's intent, and retrieves and locates events related to the user's intent in the data storage module; The intelligent reminder module is connected to the query and retrieval module and is also used to convert events related to the user's intent retrieved by the query and retrieval module into natural language feedback to the user.

[0021] In one embodiment, the data storage module also stores a task table and a raw data table; The task table is used to store reminder tasks, including at least one of the following: task identifier, trigger condition, task content, trigger parameters, associated event identifier, task priority, and confidence level. The original data table is used to store the original data of each modality, including at least one of the following: data identifier, device identifier, data type, time information, data storage reference information, and metadata information; The event table includes at least one of the following: event identifier, time information, location information, content description, event tags and keywords, and semantic vector information; The association information table includes at least one of the following: association identifier, source event identifier and target event identifier, relationship type, relationship strength or confidence level, and evidence information; The index table includes at least one of the following: index identifier, target object identifier, time index, geographic location index, type index, and vector index.

[0022] In one embodiment, the data acquisition module is configured to trigger the acquisition of raw data for each modality based on preset change conditions.

[0023] In one embodiment, the data acquisition module includes at least one of a visual acquisition unit for acquiring visual data, an auditory acquisition unit for acquiring auditory data, a geographic location acquisition unit for acquiring geographic location data, a physiological health acquisition unit for acquiring physiological health data, and an environmental acquisition unit for acquiring environmental data. The visual acquisition unit is configured to compare the similarity of adjacent image frames using an image change detection algorithm. When the similarity of adjacent frames is less than a preset similarity threshold, visual data acquisition is triggered. When the similarity of adjacent frames of N consecutive frames is greater than or equal to the preset similarity threshold, visual data acquisition is terminated, where N is a positive integer. The auditory acquisition unit is configured to monitor the sound signal through a sound activity detection algorithm. When the sound energy exceeds a preset energy threshold or the speech characteristics change, auditory data acquisition is triggered. When no sound activity is detected for T consecutive seconds, auditory data acquisition is terminated, where T is a positive integer. The geographic location acquisition unit is configured to monitor changes in the user's location. When the spatial displacement of the user's location relative to the previous recorded point exceeds a preset displacement change threshold or the time spent in the same location exceeds a preset dwell time, a new location point is recorded. The physiological health acquisition unit is configured to monitor the user's physiological health parameters. When the physiological health data exceeds a preset health threshold or the user is detected to be in motion, physiological health data acquisition is triggered. The environmental acquisition unit is configured to monitor external environmental parameters. When the change in environmental parameters exceeds a preset environmental change threshold, environmental data acquisition is triggered.

[0024] In one embodiment, the single-modal semantic processing unit is configured to output visual semantic descriptions through visual data from image recognition and scene understanding, output auditory semantic descriptions through auditory data from speech transcription and semantic analysis, output semantics corresponding to location information through geographic semantic processing of geographic location data, output physiological health semantic information through data pattern analysis and medical semantic mapping of physiological health data, and / or output environmental semantic information through environmental parameter semantic processing of environmental data.

[0025] The present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the wearable device interaction method based on memory information as described above.

[0026] Compared with the prior art, the beneficial effects of the wearable device interaction method and apparatus and computer equipment based on memory information of the present invention are as follows: 1) This invention breaks through the traditional all-weather continuous acquisition mode by establishing an intelligent event-driven acquisition mechanism based on multimodal data change detection algorithms (such as image changes, sound activity, geographical location displacement, physiological parameter thresholds, and environmental parameter changes). Acquisition and recording are triggered only when effective changes in images, sounds, environmental parameters, etc. are detected. This significantly reduces the generation of redundant data from the source, which not only improves the utilization of storage space, but also reduces the processor's computing load and network bandwidth consumption by reducing the transmission and processing of invalid data, thereby improving data processing efficiency and significantly extending the battery life of smart glasses.

[0027] 2) This invention, through a vector database architecture, a multi-dimensional indexing system (time index, geographic location index, type index, semantic vector index, etc.), and a two-level artificial intelligence processing method, can achieve semantic similarity-based retrieval and structured condition filtering of structured event descriptions. This not only improves retrieval accuracy but also ensures rapid response, thereby meeting users' needs for real-time and accurate queries in multi-round natural language interactions.

[0028] 3) This invention supports both proactive triggering and passive querying interaction modes. Through the proactive triggering mechanism, combined with extracted task elements such as time, location, scenario, and to-do list, personalized reminder tasks can be automatically created based on time and location triggers. This achieves contextualized and personalized proactive services, significantly reducing the probability of users forgetting daily tasks and making the human-computer interaction of wearable devices more natural and intelligent. The passive querying mode supports natural language queries, accurately locating relevant memories in a vector database through semantic similarity matching, adapting to diverse user needs and scenarios.

[0029] 4) This invention integrates multimodal data (visual, auditory, location, physiological, and environmental) and performs event correlation analysis. It combines multiple pieces of information within the same event for semantic processing. Through two-level intelligent processing (the first level identifies events based on raw data and the second level performs comprehensive analysis of multiple events within a time period), all data is converted into a structured event description of "when-where-what". This extracts key information rather than simply storing raw data, which significantly improves the completeness of event records and enhances the accuracy of key information extraction. This provides a reliable and rich data foundation for subsequent intelligent retrieval and task triggering.

[0030] 5) This invention uses a timeline as the main thread and combines it with the event dimension to construct a complete memory management system, capable of outputting a user's continuous event record throughout the day, forming a personal memory archive. The database adopts a layered structure of raw data layer, event description layer (including semantic vectors), associated information layer, and index layer, supporting efficient retrieval based on both time and semantics. It provides unified and powerful data support for diverse user needs (such as meeting memory assistance, life recording, health management, shopping reminders, etc.), with high adaptability and broad application prospects.

[0031] 6) This invention moves some computations to the acquisition end by using front-end intelligent filtering, hierarchical processing architecture and incremental memory storage mechanism, which can reduce invalid data transmission, reduce processor utilization, extend device battery life and reduce network bandwidth consumption. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating a wearable device interaction method based on memory information according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a wearable device interaction device based on memory information according to an embodiment of the present invention.

[0033] Figure Labels

[0034] 100. Data Acquisition Module; 200. Data Processing Module; 300. Data Storage Module; 400. Task Generation Module; 500. Intelligent Reminder Module; 600. Query and Retrieval Module; 101. Visual Acquisition Unit; 102. Auditory Acquisition Unit; 103. Geographic Location Acquisition Unit; 104. Physiological Health Acquisition Unit; 105. Environmental Acquisition Unit; 201. Single-Modal Semantic Processing Unit; 202. Multi-Modal Event Integration Unit. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention more readily understood, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that while many specific details are set forth in the following description to provide a thorough understanding of the invention, the invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of the invention; therefore, the invention is not limited to the specific embodiments disclosed below.

[0036] Secondly, the phrase "an embodiment" or "a particular embodiment" in this application refers to a specific feature, structure, or characteristic that can be included in at least one implementation of the present invention. The phrases "in one embodiment" and "a particular embodiment" appearing in different places in this specification do not all refer to the same embodiment, nor are they embodiments that are mutually exclusive, either alone or selectively, with other embodiments. The terms "comprising" or "including" indicate the presence of the claimed feature but do not exclude the presence of one or more other features. The term "and / or" as used in this application includes any and all combinations of one or more of the related listed items.

[0037] This invention proposes a wearable device interaction method based on memory information, see [link to relevant documentation]. Figure 1 It includes the following steps: Step S1: Collect multimodal raw data through wearable devices (such as AI smart glasses, smart bracelets, smartwatches, etc.). Multimodal raw data includes at least one of visual data (images or videos), auditory data (audio), geographic location data, physiological health data, and environmental data.

[0038] Step S2: Perform event recognition and semantic extraction on the raw data of each modality, and output a single-modal textual semantic description.

[0039] Step S3: Comprehensively analyze all single-modal semantic descriptions within a certain time period to generate a complete event chain and event context description. Specifically, the semantic description information obtained in Step S2 (e.g., all results within a day) is fed back into the large model for processing, outputting context description information for at least one event. The event context description includes at least one of the following: time, location, participants, content description (e.g., "The user attended a meeting in the office in the morning and exercised at the gym in the afternoon"), causal relationships between events, event time sequence, event importance level (e.g., high, medium, low), the context of the event (e.g., "receiving a phone call during a meeting," "heart rate significantly increased during exercise"), and related information about the event (e.g., information related to the user's health, schedule, and environmental changes). This event context description information serves as the core storage content of the database for subsequent retrieval, inference, and user service invocation.

[0040] Step S4: Save the event table, association information table, and index table to the database. The event table stores the semantic event chain and event context descriptions after the aforementioned processing, forming the core records of user behavior and context. The association information table stores the relationships between events (such as logical relationships and contextual connections). The index table establishes an efficient data retrieval channel to support multi-dimensional queries based on time, space, and semantics.

[0041] Step S5: Automatically generate reminder tasks based on data in the event table, related information table, and index table, and push them to the user when the task trigger conditions are met; this is the proactive triggering mechanism. Task trigger conditions include at least one of the following: time, location, scenario, and unfinished tasks.

[0042] Step S5 of one embodiment of the present invention, which involves automatically generating a reminder task based on data in the event table, the associated information table, and the index table, specifically includes: Candidate event retrieval: Based on preset conditions and using the time index, geographic index and / or semantic vector index in the index table, candidate events are retrieved from the event table. The preset conditions include at least one of the following: records containing a time field, a location field, or records whose semantic content contains an item to be completed. Element analysis: Semantic analysis is performed on candidate events to identify users' unfinished behavioral needs and extract task elements. Task elements include at least one of the following: time elements (start and end time fields from the event table, or time information obtained through semantic content analysis), geographical location elements (latitude and longitude information or location tags from the event table), scene elements (compared by event semantic content and tags and current visual input), and to-do elements (such as behavioral intentions identified in semantics such as "need," "remember," and "to be completed"). Task Generation: Based on task generation rules, reminder tasks are generated using extracted task elements. Specific task generation rules include: if a specific time field is included, a time-based reminder is generated; if the current user's location matches the event location, a location-based reminder is generated; based on visual features or semantic vectors, if the current visual input matches the event scene to a certain degree, a scene-based reminder is generated; if a pending task is detected, a new pending task is generated.

[0043] For example, when the system detects that a user's current location is within the office area, it will search the event table for recent event records related to keywords such as "meeting," "report," or "client communication." If a record shows "10:00 AM" in the time field, "Company Meeting Room A" in the location field, and the semantic content includes "preparing next week's report materials," the system will treat this event as a candidate event and proceed to the element analysis and task generation steps to create a "Reminder for Preparing Report Materials" task.

[0044] One embodiment of the present invention retrieves candidate events from an event table based on preset conditions and utilizing time indexes, geographic indexes, and / or semantic vector indexes in an index table, specifically including: The time, location, and / or semantic content containing records of pending tasks in the preset conditions are mapped to the corresponding index fields of the index table for preliminary retrieval, resulting in an initial set of events that meet each individual condition; Perform a logical intersection operation on the initial event set to filter out events that simultaneously meet all preset conditions as primary candidate events; Based on the association information table, retrieve other events that are related to the primary candidate events; The primary candidate events and their related events are combined to form the final candidate events.

[0045] One embodiment of the present invention performs semantic parsing on candidate events to identify users' unfulfilled behavioral needs, specifically including: Extract the content description, related information, and participating objects from the event context descriptions corresponding to the candidate events, and combine them into semantic text to be parsed; The semantic text is input into a natural language understanding model to identify and label sentence fragments in the text that represent intentions, commitments, or needs but do not contain completion status markers, thereby identifying the user's unfulfilled behavioral needs.

[0046] In one embodiment of the present invention, after automatically generating reminder tasks based on data from an event table, an associated information table, and an index table, the generated reminder tasks are also saved to a task table. The task table includes at least one of the following: task identifier, trigger conditions (time, location, scenario, to-do), task content, trigger parameters (time point, geofence, scenario matching degree), associated event identifier, task priority, and confidence level.

[0047] In addition, the generated tasks are further filtered, merged and deduplicated. Specifically, if multiple tasks are highly similar in time, location or content, they are merged into a single task and the content with the highest priority is retained.

[0048] An embodiment of the wearable device interaction method based on memory information of the present invention further includes: storing an event table in a database according to time to form a time-coherent event chain; User input triggers a passive query interaction mechanism. Specifically, in passive interaction mode, for user queries submitted via natural language, the user's voice input is transcribed into text, the user's intent is parsed, and elements such as time, location, event type, and semantic information are extracted. Using an index table, the system locates events related to the user's intent in the database through semantic retrieval and structured conditional retrieval, and then converts these events back into natural language feedback to the user. Passive interaction mode can also support multiple rounds of follow-up questions.

[0049] In summary, the wearable device interaction method based on memory information of the present invention can realize information interaction through two mechanisms: active triggering of task generation and passive interaction.

[0050] An embodiment of the wearable device interaction method based on memory information of the present invention further includes: storing the original data table to a database according to time. The original data table is used to store the original data of each modality, organized by timestamp, and only the newly added or modified parts are written to the database. The original data table includes at least one of the following: data identifier (used to uniquely identify each piece of original data), device identifier (used to record the data source), data type (such as image, audio, video, geographical location, physiological parameters, environmental parameters, etc.), time information (such as the start time and end time of collection), data storage reference information (used to point to the storage location of the original file or object), and metadata information (such as auxiliary parameters such as resolution, sampling rate, file size, geographical coordinates, sensor accuracy, etc.).

[0051] An event table in one embodiment of the present invention includes at least one of the following: event identifier (used to uniquely identify each event), time information (the time range in which the event occurred), location information (the geographical coordinates and semantic location tags of the event), content description (semantic description information of the event), related original data references (used to point to the original data associated with the event), event tags and keywords (for easy retrieval and classification), and semantic vectors (i.e., the vectorized representation of the event, generated by a semantic embedding model, used for similarity search). The association information table includes at least one of the following: association identifier (used to uniquely identify the association relationship), source event identifier and target event identifier, relationship type (including chronological, causal, concurrent, and contextual continuation relationships), relationship strength or confidence (used to describe the importance of the relationship), and evidence information (e.g., the original data fragment that triggered the relationship or the basis for model judgment). The index table includes at least one of the following: index identifier, target object identifier (such as an event or original data), time index (such as the timestamp of the event or data), geographical location index (such as latitude and longitude coordinates), type index (such as event type or data type classification), vector index (i.e., an index based on semantic vectors, used for similarity retrieval), and other extended fields (such as user tags, device origin, etc.).

[0052] Specifically, the database of this invention adopts a hierarchical storage structure, including a raw data layer, an event description layer, a relational information layer, and an index layer. The raw data layer stores raw data tables, the event description layer stores event tables, the relational information layer stores relational information tables, and the index layer stores index tables.

[0053] One embodiment of the present invention for collecting multimodal raw data specifically involves: employing an event-triggered selective data acquisition mechanism, whereby raw data for each modality is collected based on preset change conditions, and the acquisition conditions are determined by a front-end device algorithm.

[0054] Visual data (such as image data) is collected through a continuously operating camera to provide a constant input of images. Image change detection algorithms (such as structural similarity algorithms and histogram difference algorithms) are used to compare the similarity of adjacent image frames: when the similarity between adjacent frames is less than a preset similarity threshold (e.g., 90%), it is determined that the image content has changed significantly, triggering visual data acquisition and saving the current image; when the similarity between adjacent frames for N consecutive frames (e.g., N=30, corresponding to approximately 1 second) is greater than or equal to the preset similarity threshold, it is determined that the image content has stabilized and no significant changes have occurred, terminating visual data acquisition, where N is a positive integer. Auditory data (audio data) is monitored through a sound activity detection algorithm: when the sound energy exceeds a preset energy threshold or when there is a significant change in speech characteristics, auditory data acquisition is triggered and saved; when no sound activity is detected for T consecutive seconds (e.g., T=2), auditory data acquisition is terminated, where T is a positive integer. The conditions for collecting geographic location data are as follows: when the user's spatial displacement relative to the last recorded point exceeds a preset displacement change threshold (e.g., 500 meters) or the time spent in the same location exceeds a preset dwell time (e.g., 10 minutes), a new location point is recorded, thus forming the user's movement trajectory. The conditions for collecting physiological health data (such as heart rate, respiratory rate, blood pressure, etc.) are as follows: when physiological health data exceeds a preset health threshold or when the user is detected to be in motion, physiological health data collection is triggered. The conditions for collecting environmental data (such as weather conditions, temperature, humidity, light intensity, etc.) are as follows: when changes in environmental parameters exceed a preset environmental change threshold, environmental data collection is triggered.

[0055] In one specific embodiment, the visual information acquired is in the form of images. The camera operates in real-time, continuously inputting images. An image change detection algorithm, as the main module, compares the similarity of adjacent frames (e.g., using structural similarity simulcasting (SSIM), histogram difference, or other equivalent algorithms). When the similarity between adjacent frames is greater than or equal to 90%, it is determined that the image has not changed significantly, and acquisition is not triggered; when the similarity between adjacent frames is less than 90%, it is determined that the image content has changed significantly, triggering the camera to save the current image. The termination condition for image acquisition is: in the detection of N consecutive frames (e.g., N=30, corresponding to approximately 1 second), if the similarity between adjacent frames is greater than or equal to 90%, the system determines that the image content has stabilized and issues a command to stop acquisition. This mechanism ensures that image acquisition only occurs when the content has undergone a valid change, thus breaking through the traditional all-weather continuous acquisition mode and reducing redundant storage and computational burden.

[0056] The auditory information collected is in audio format. A sound activity detection method is used; when sound energy exceeds a set threshold or there is a significant change in speech characteristics, the acquisition device is activated and the audio data is saved. If no sound activity is detected for T consecutive seconds (e.g., T=2), the system terminates the acquisition. This mechanism avoids invalid data storage during prolonged periods of silence.

[0057] The collected geolocation information data consists of the user's geographic location points. A location change threshold is set. When the user's spatial displacement relative to the last recorded point exceeds the set threshold (e.g., 500 meters) or the user stays in the same location for more than a set duration (e.g., 10 minutes), a new location point is recorded, thus forming the user's movement trajectory.

[0058] The physiological health data collected includes the user's physiological parameters, including but not limited to heart rate, respiratory rate, and blood pressure. The system monitors physiological signals in real time through sensors and triggers recording when specific events are detected, such as heart rate exceeding a preset threshold (e.g., greater than 120 beats / minute), a significant increase in respiratory rate, or the identification of exercise status. This mechanism ensures effective data recording during exercise or physiological fluctuations while avoiding unnecessary continuous data collection.

[0059] The environmental information collected consists of external environmental parameters, including but not limited to weather conditions, temperature, humidity, and light intensity. The system acquires these parameters through integrated environmental sensors or by interacting with external data sources. Recording is triggered when an external environmental parameter exceeds a preset threshold (e.g., a temperature change exceeding 2 degrees Celsius, a humidity change exceeding 5%, or a significant change in light intensity); continuous data collection is not performed when parameters remain stable. This mechanism ensures that environmental information is recorded only when significant changes occur, thus avoiding the generation of redundant data.

[0060] In one embodiment of the present invention, step S2 involves feeding the raw data of each modality collected in step S1 into a large model, performing event recognition and semantic extraction on the raw data of each modality respectively, and outputting a single-modal textual semantic description, specifically including: Visual data outputs visual semantic descriptions through image recognition and scene understanding, such as "the user is working indoors," "the user is walking outdoors," and "human interaction was detected." Auditory data is used to output auditory semantic descriptions through speech transcription and semantic analysis, such as "the user is making a phone call", "music is playing in the environment", "sudden noise occurs", etc. Geographic location data is processed through geographic semantics to output the semantics corresponding to the location information, such as "the user is located in a restaurant", "the user moves from a residence to an office", and "the user is in a state of high-speed movement". Physiological health data is analyzed through data pattern analysis and medical semantic mapping to output physiological health semantic information, such as "the user is in a state of exercise", "the user's heart rate is higher than the resting threshold", and "an abnormal increase in the user's respiratory rate is detected". Environmental data is processed through semantic processing of environmental parameters to output semantic information about the environment, such as "sudden increase in external temperature", "significant decrease in ambient humidity", and "reduction in light intensity may indicate evening".

[0061] Regardless of whether the input data is video, audio, location, physiological or environmental parameters, the output of the large model is semantic information, that is, a textual description of the original data that is abstracted, interpreted and contextualized.

[0062] Step S3 of one embodiment of the present invention involves comprehensively analyzing all single-modal semantic descriptions within a certain equal time period to generate a complete event chain and event context description, specifically including: The semantic description information obtained in step S2 is fed back into the large model. Through the cross-modal attention mechanism, the visual semantic description, auditory semantic description, semantic information corresponding to location information, physiological health semantic information and / or environmental semantic information within the same time period are aligned and fused to extract and integrate them into a set of event context description elements. Construct a logically coherent event chain based on the chronological order and / or causal relationships in the event scenario description elements.

[0063] The method of this invention can be used in various scenarios such as meeting memory assistance, daily life recording, health data management, and shopping reminders. For example, when used for meeting memory assistance, it collects multimodal data during the meeting, processes it to generate meeting minutes, and automatically generates action item reminders. When used for daily life recording, it collects daily activity data, identifies important events, outputs a personal life timeline, and generates life-related reminders when necessary. When used for health data management, it generates health reminders when physiological health parameters exceed thresholds and provides adjustment suggestions in conjunction with exercise records. When used as a shopping reminder assistant, it automatically triggers shopping reminders when the user approaches a shopping location, based on the user's mentioned shopping needs and geographical location.

[0064] This invention also proposes a wearable device interaction device based on memory information, used to implement the wearable device interaction method based on memory information as described above. See [link to relevant documentation]. Figure 2The system includes a data acquisition module 100, a data processing module 200, a data storage module 300, a task generation module 400, and a smart reminder module 500. The data acquisition module 100 is used to collect multimodal raw data through wearable devices (such as AI smart glasses, smartwatches, smart bracelets, etc.). The multimodal raw data includes at least one of visual data (images or videos), auditory data (audio), geographic location data, physiological health data, and environmental data. The data processing module 200 includes a single-modal semantic processing unit 201 and a multimodal event integration unit 202. The single-modal semantic processing unit 201 performs event recognition and semantic extraction on each modality of raw data, outputting a single-modal textual semantic description. The multimodal event integration unit 202 comprehensively analyzes all single-modal semantic descriptions within a certain time period to generate a complete event chain and event context description. The event context description includes at least one of the following: time, location, participants, content description, causal relationship between events, event time sequence, event importance level, context of the event, and related information (such as information related to user health, schedule, and environmental changes). The data storage module 300 stores an event table, a related information table, and an index table. The event table stores the semantic event chain and event context description processed by the data processing module. The related information table stores the relationships between events (such as logical relationships and contextual connections). The index table establishes an efficient data retrieval channel to support multi-dimensional queries based on time, space, and semantics. The task generation module 400 automatically generates reminder tasks based on data from the event table, related information table, and index table. The intelligent reminder module 500 pushes reminders to the user when task trigger conditions are met. Task trigger conditions include at least one of the following: time, location, scenario, and unfinished tasks.

[0065] In one embodiment of the present invention, the task generation module 400 is configured to retrieve candidate events from an event table based on preset conditions and using an index table. The preset conditions include at least one of a time field, a location field, and a record whose semantic content contains a task to be completed. The module performs semantic parsing on the candidate events to identify the user's unfinished behavioral needs, extracts task elements, and the task elements include at least one of a time element, a geographical location element, a scene element, and a task to be completed element. Based on the extracted task elements, the module generates a reminder task according to the task generation rules.

[0066] An embodiment of the wearable device interaction apparatus based on memory information of the present invention further includes a query retrieval module 600. The query retrieval module 600 is used to extract time, semantic information, etc., from a user's query request submitted via natural language, parse the user's intent, and retrieve and locate events related to the user's intent in the data storage module 300. The intelligent reminder module 500 is connected to the query retrieval module 600 and is also used to convert the events related to the user's intent retrieved by the query retrieval module into natural language and feed them back to the user.

[0067] In one embodiment of the present invention, the data storage module 300 further stores a task table and a raw data table. The task table stores reminder tasks and includes at least one of the following: task identifier, trigger condition, task content, trigger parameters, associated event identifier, task priority, and confidence level. The raw data table stores raw data for each modality and includes at least one of the following: data identifier, device identifier, data type, time information, data storage reference information, and metadata information. The event table includes at least one of the following: event identifier, time information, location information, content description, event tags and keywords, and semantic vector information. The association information table includes at least one of the following: association identifier, source event identifier and target event identifier, relationship type, relationship strength or confidence level, and evidence information. The index table includes at least one of the following: index identifier, target object identifier, time index, geographic location index, type index, and vector index.

[0068] In one embodiment of the present invention, the data acquisition module 100 is configured to trigger the acquisition of raw data of each modality based on preset change conditions.

[0069] The data acquisition module 100 includes at least one of the following: a visual acquisition unit 101 for acquiring visual data, an auditory acquisition unit 102 for acquiring auditory data, a geographic location acquisition unit 103 for acquiring geographic location data, a physiological health acquisition unit 104 for acquiring physiological health data, and an environmental acquisition unit 105 for acquiring environmental data. The visual acquisition unit 101 (e.g., a camera) is configured to compare the similarity of adjacent image frames using an image change detection algorithm (e.g., structural similarity algorithm, histogram difference algorithm, etc.). Visual data acquisition is triggered when the similarity between adjacent frames is less than a preset similarity threshold, and visual data acquisition is terminated when the similarity between adjacent frames for N consecutive frames is greater than or equal to the preset similarity threshold, where N is a positive integer. The auditory acquisition unit 102 is configured to monitor sound signals using a sound activity detection algorithm. Auditory data acquisition is triggered when the sound energy exceeds a preset energy threshold or when there is a significant change in speech characteristics, and auditory data acquisition is terminated when no sound activity is detected for T consecutive seconds, where T is a positive integer. The geographic location acquisition unit 103 is configured to monitor changes in the user's location. When the user's spatial displacement relative to the last recorded point exceeds a preset displacement change threshold or the time spent at the same location exceeds a preset dwell time, a new location point is recorded. The physiological health acquisition unit 104 is configured to monitor the user's physiological health parameters (such as heart rate, respiratory rate, blood pressure, etc.). When physiological health data exceeds a preset health threshold or when the user is detected to be in motion, physiological health data acquisition is triggered. The environmental acquisition unit 105 is configured to monitor external environmental parameters (such as weather conditions, temperature, humidity, light intensity, etc.). When changes in environmental parameters exceed a preset environmental change threshold, environmental data acquisition is triggered.

[0070] One embodiment of the present invention provides a single-modal semantic processing unit 201 configured to output visual semantic descriptions through visual data from image recognition and scene understanding, output auditory semantic descriptions through auditory data from speech transcription and semantic analysis, output semantics corresponding to location information through geographic semantic processing of geographic location data, output physiological health semantic information through data pattern analysis and medical semantic mapping of physiological health data, and / or output environmental semantic information through environmental parameter semantic processing of environmental data.

[0071] The present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the wearable device interaction method based on memory information as described above.

[0072] The present invention has the following beneficial effects: 1) This invention breaks through the traditional all-weather continuous acquisition mode by establishing an intelligent event-driven acquisition mechanism based on multimodal data change detection algorithms (such as image changes, sound activity, geographical location displacement, physiological parameter thresholds, and environmental parameter changes). Acquisition and recording are triggered only when effective changes in images, sounds, environmental parameters, etc. are detected. This significantly reduces the generation of redundant data from the source, which not only improves the utilization of storage space, but also reduces the processor's computing load and network bandwidth consumption by reducing the transmission and processing of invalid data, thereby improving data processing efficiency and significantly extending the battery life of smart glasses.

[0073] 2) This invention, through a vector database architecture, a multi-dimensional indexing system (time index, geographic location index, type index, semantic vector index, etc.), and a two-level artificial intelligence processing method, can achieve semantic similarity-based retrieval and structured condition filtering of structured event descriptions. This not only improves retrieval accuracy but also ensures rapid response, thereby meeting users' needs for real-time and accurate queries in multi-round natural language interactions.

[0074] 3) This invention supports both proactive triggering and passive querying interaction modes. Through the proactive triggering mechanism, combined with extracted task elements such as time, location, scenario, and to-do list, personalized reminder tasks can be automatically created based on time and location triggers. This achieves contextualized and personalized proactive services, significantly reducing the probability of users forgetting daily tasks and making the human-computer interaction of wearable devices more natural and intelligent. The passive querying mode supports natural language queries, accurately locating relevant memories in a vector database through semantic similarity matching, adapting to diverse user needs and scenarios.

[0075] 4) This invention integrates multimodal data (visual, auditory, location, physiological, and environmental) and performs event correlation analysis. It combines multiple pieces of information within the same event for semantic processing. Through two-level intelligent processing (the first level identifies events based on raw data and the second level performs comprehensive analysis of multiple events within a time period), all data is converted into a structured event description of "when-where-what". This extracts key information rather than simply storing raw data, which significantly improves the completeness of event records and enhances the accuracy of key information extraction. This provides a reliable and rich data foundation for subsequent intelligent retrieval and task triggering.

[0076] 5) This invention uses a timeline as the main thread and combines it with the event dimension to construct a complete memory management system, capable of outputting a user's continuous event record throughout the day, forming a personal memory archive. The database adopts a layered structure of raw data layer, event description layer (including semantic vectors), associated information layer, and index layer, supporting efficient retrieval based on both time and semantics. It provides unified and powerful data support for diverse user needs (such as meeting memory assistance, life recording, health management, shopping reminders, etc.), with high adaptability and broad application prospects.

[0077] 6) This invention moves some computations to the acquisition end by using front-end intelligent filtering, hierarchical processing architecture and incremental memory storage mechanism, which can reduce invalid data transmission, reduce processor utilization, extend device battery life and reduce network bandwidth consumption.

[0078] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0079] The constructions and arrangements of this application shown in several different exemplary embodiments are merely illustrative. Although only a few embodiments are described in detail in this disclosure, those who consult this disclosure will readily understand that many modifications are possible (e.g., changes in size, structure, shape, and proportions, as well as parameter values, installation arrangements, use of materials, orientation, etc.) without substantially departing from the novel teachings and advantages of the subject matter described in this application. For example, an element shown as integrally formed may be composed of multiple parts or elements, the position of the elements may be inverted or otherwise changed, and the nature or number or position of discrete elements may be altered or changed. Therefore, all such modifications are intended to be included within the scope of this invention. The order or sequence of any process or method steps may be changed or rearranged according to alternative embodiments. In the claims, any "device plus function" clause is intended to cover the structure described herein that performs the function, and not only structurally equivalent but also equivalent in structure. Various corresponding modifications and variations can be made by those skilled in the art according to this invention without departing from the spirit and essence of the invention, but such corresponding modifications and variations should fall within the protection scope of this invention.

Claims

1. A wearable device interaction method based on memory information, characterized in that, Includes the following steps: Multimodal raw data is collected through wearable devices, including at least one of visual data, auditory data, geographic location data, physiological health data, and environmental data. Perform event recognition and semantic extraction on the raw data of each modality, and output a single-modal semantic description; By comprehensively analyzing all single-modal semantic descriptions within the same time period, an event chain and event context description are generated. The event context description includes at least one of the following: time, location, participants, content description, causal relationship between events, event time sequence, event importance level, context in which the event occurs, and event association information. Save the event table, the association information table, and the index table to the database. The event table is used to store event chains and event context descriptions, the association information table is used to store the relationships between events, and the index table is used to establish data retrieval channels. Reminder tasks are automatically generated based on data from the event table, related information table, and index table, and pushed to the user when the task triggering conditions are met. The task triggering conditions include at least one of time, location, scenario, and unfinished items.

2. The wearable device interaction method based on memory information according to claim 1, characterized in that, Automatically generate reminder tasks based on data from the event table, related information table, and index table, specifically including: Candidate events are obtained by retrieving them from the event table based on preset conditions and using an index table. The preset conditions include at least one of the following: records containing a time field, a location field, or semantic content containing an item to be completed. Semantic parsing is performed on candidate events to identify users' unfulfilled behavioral needs and extract task elements, which include at least one of time elements, geographical location elements, scene elements, and to-do elements. Based on the task generation rules, a reminder task is generated using the extracted task elements.

3. The wearable device interaction method based on memory information according to claim 2, characterized in that, Candidate events are obtained based on preset conditions and by retrieving them from the event table using an index table. Specifically, these include: The time, location, and / or semantic content containing records of pending tasks in the preset conditions are mapped to the corresponding index fields of the index table for preliminary retrieval, resulting in an initial set of events that meet each individual condition; Perform a logical intersection operation on the initial event set to filter out events that simultaneously meet all preset conditions as primary candidate events; Based on the association information table, other events that are related to the primary candidate events are obtained; The primary candidate events and their associated events are combined to form the final candidate events.

4. The wearable device interaction method based on memory information according to claim 2, characterized in that, Semantic parsing of candidate events identifies unfulfilled user behavioral needs, specifically including: Extract the content description, related information, and participating objects of the event from the event scenario description corresponding to the candidate event, and combine them into semantic text to be parsed; The semantic text is input into a natural language understanding model to identify and label sentence fragments in the text that represent intentions, commitments, or needs but do not contain completion status markers, thereby identifying the user's unfulfilled behavioral needs.

5. The wearable device interaction method based on memory information according to claim 2, characterized in that, After automatically generating reminder tasks based on data from the event table, associated information table, and index table, the generated reminder tasks are also saved to the task table. The task table includes at least one of the following: task identifier, trigger condition, task content, trigger parameters, associated event identifier, task priority, and confidence level.

6. The wearable device interaction method based on memory information according to claim 1, characterized in that, It also includes: storing event tables in the database according to time to form a time-coherent event chain; For queries submitted by users in natural language, time and / or semantic information is extracted, user intent is parsed, events related to the user intent are retrieved from the event table using the index table, and then converted into natural language and fed back to the user.

7. The wearable device interaction method based on memory information according to claim 1, characterized in that, When collecting multimodal raw data, the raw data of each modality is collected based on preset change conditions.

8. The wearable device interaction method based on memory information according to claim 7, characterized in that, The preset change conditions for collecting visual data are as follows: visual data is compared with the similarity of adjacent image frames through an image change detection algorithm. When the similarity of adjacent frames is less than the preset similarity threshold, visual data collection is triggered. When the similarity of adjacent frames of N consecutive frames is greater than or equal to the preset similarity threshold, visual data collection is terminated, where N is a positive integer. The process of event recognition and semantic extraction from visual data is as follows: visual data outputs a visual semantic description through image recognition and scene understanding.

9. The wearable device interaction method based on memory information according to claim 7, characterized in that, The preset change conditions for collecting auditory data are as follows: auditory data is monitored by a sound activity detection algorithm. When the sound energy exceeds the preset energy threshold or the speech characteristics change, auditory data collection is triggered. When no sound activity is detected for T consecutive seconds, auditory data collection is terminated, where T is a positive integer. The process of event recognition and semantic extraction from auditory data is as follows: auditory data is converted into auditory semantic descriptions through speech transcription and semantic analysis.

10. The wearable device interaction method based on memory information according to claim 7, characterized in that, The preset change condition for collecting geographic location data is: when the spatial displacement of the user's location relative to the previous recorded point exceeds the preset displacement change threshold or the time spent in the same location exceeds the preset stay duration, a new location point is recorded. The process of event identification and semantic extraction from geographic location data is as follows: geographic location data is processed through geographic semantics to output the semantics corresponding to the location information.

11. The wearable device interaction method based on memory information according to claim 7, characterized in that, The preset change conditions for collecting physiological health data are: when the physiological health data exceeds the preset health threshold or when the user is detected to be in motion, physiological health data collection is triggered. The process of event identification and semantic extraction from physiological health data is as follows: physiological health data is analyzed by data pattern analysis and medical semantic mapping to output physiological health semantic information.

12. The wearable device interaction method based on memory information according to claim 7, characterized in that, The preset change condition for collecting environmental data is: when the change in environmental parameters exceeds the preset environmental change threshold, environmental data collection is triggered. The process of event identification and semantic extraction from environmental data is as follows: environmental data is processed through semanticization of environmental parameters to output environmental semantic information.

13. The wearable device interaction method based on memory information according to claim 1, characterized in that, By comprehensively analyzing all single-modal semantic descriptions within the same time period, an event chain and event context description are generated, specifically including: By using a cross-modal attention mechanism, all single-modal semantic descriptions within the same time period are aligned and fused, and then extracted and integrated into a set of event context description elements. Construct a logically coherent event chain based on the chronological order and / or causal relationships in the event scenario description elements.

14. A wearable device interaction device based on memory information, characterized in that, A method for implementing a wearable device interaction method based on memory information as described in any one of claims 1-13, comprising: The data acquisition module collects multimodal raw data, which includes at least one of visual data, auditory data, geographic location data, physiological health data, and environmental data. The data processing module includes a single-modal semantic processing unit and a multimodal event integration unit. The single-modal semantic processing unit is used to perform event recognition and semantic extraction on the raw data of each modality and output a single-modal semantic description. The multimodal event integration unit is used to comprehensively analyze all single-modal semantic descriptions within the same time period and generate an event chain and an event context description. The event context description includes at least one of the following: time, location, participants, content description, causal relationship between events, event time sequence, event importance level, context in which the event occurred, and event association information. The data storage module stores an event table, an association information table, and an index table. The event table stores event chains and event context descriptions, the association information table stores the relationships between events, and the index table is used to establish data retrieval channels. The task generation module automatically generates reminder tasks based on data from the event table, related information table, and index table; The intelligent reminder module pushes a notification to the user when a task trigger condition is met. The task trigger condition includes at least one of time, location, scenario, and unfinished items.

15. The wearable device interaction device based on memory information according to claim 14, characterized in that, The task generation module is configured to retrieve candidate events from the event table based on preset conditions and using an index table. The preset conditions include at least one of the following: records containing a time field, a location field, and records whose semantic content contains items to be completed. The module performs semantic parsing on the candidate events to identify the user's unfinished behavioral needs and extracts task elements. The task elements include at least one of the following: time element, geographical location element, scene element, and to-do element. Based on the extracted task elements, the module generates a reminder task according to the task generation rules.

16. The wearable device interaction device based on memory information according to claim 14, characterized in that, It also includes a query and retrieval module, which extracts time and / or semantic information from user queries submitted in natural language, parses user intent, and retrieves and locates events related to user intent in the data storage module; The intelligent reminder module is connected to the query and retrieval module and is also used to convert events related to the user's intent retrieved by the query and retrieval module into natural language feedback to the user.

17. The wearable device interaction device based on memory information according to claim 14, characterized in that, The data storage module also stores a task table and a raw data table; The task table is used to store reminder tasks, including at least one of the following: task identifier, trigger condition, task content, trigger parameters, associated event identifier, task priority, and confidence level. The original data table is used to store the original data of each modality, including at least one of the following: data identifier, device identifier, data type, time information, data storage reference information, and metadata information; The event table includes at least one of the following: event identifier, time information, location information, content description, event tags and keywords, and semantic vector information; The association information table includes at least one of the following: association identifier, source event identifier and target event identifier, relationship type, relationship strength or confidence level, and evidence information; The index table includes at least one of the following: index identifier, target object identifier, time index, geographic location index, type index, and vector index.

18. The wearable device interaction device based on memory information according to claim 14, characterized in that, The data acquisition module is configured to trigger the acquisition of raw data for each modality based on preset change conditions. The data acquisition module includes at least one of the following: a visual acquisition unit for acquiring visual data, an auditory acquisition unit for acquiring auditory data, a geographic location acquisition unit for acquiring geographic location data, a physiological health acquisition unit for acquiring physiological health data, and an environmental acquisition unit for acquiring environmental data. The visual acquisition unit is configured to compare the similarity of adjacent image frames using an image change detection algorithm. When the similarity of adjacent frames is less than a preset similarity threshold, visual data acquisition is triggered. When the similarity of adjacent frames of N consecutive frames is greater than or equal to the preset similarity threshold, visual data acquisition is terminated, where N is a positive integer. The auditory acquisition unit is configured to monitor the sound signal through a sound activity detection algorithm. When the sound energy exceeds a preset energy threshold or the speech characteristics change, auditory data acquisition is triggered. When no sound activity is detected for T consecutive seconds, auditory data acquisition is terminated, where T is a positive integer. The geographic location acquisition unit is configured to monitor changes in the user's location. When the spatial displacement of the user's location relative to the previous recorded point exceeds a preset displacement change threshold or the time spent in the same location exceeds a preset dwell time, a new location point is recorded. The physiological health acquisition unit is configured to monitor the user's physiological health parameters. When the physiological health data exceeds a preset health threshold or the user is detected to be in motion, physiological health data acquisition is triggered. The environmental acquisition unit is configured to monitor external environmental parameters. When the change in environmental parameters exceeds a preset environmental change threshold, environmental data acquisition is triggered.

19. The wearable device interaction device based on memory information according to claim 14, characterized in that, The single-modal semantic processing unit is configured to output visual semantic descriptions through image recognition and scene understanding visual data, output auditory semantic descriptions through speech transcription and semantic analysis auditory data, output semantics corresponding to location information through geographic semantic processing of geographic location data, output physiological health semantic information through data pattern analysis and medical semantic mapping of physiological health data, and / or output environmental semantic information through environmental parameter semantic processing of environmental data.

20. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the wearable device interaction method based on memory information as described in any one of claims 1-13.

Images