Intelligent following robot object searching method and system based on graph structure memory

Abstract

The invention discloses an intelligent following robot object searching method and system based on graph structure memory. The method comprises the steps that the robot follows a user to move in real time, actively senses behaviors of the user and obtains an observation result; organizing a graph structure based on an observation result to form an initial pragmatic memory graph; forming structured memory based on the initial pragmatic memory graph, wherein the structured memory comprises continuously maintained and updated working memory and long-term memory for storing behavior habits of people; querying in the working memory or the long-term memory according to the question proposed by the user to obtain a query result; and outputting a voice prompt according to the query result. The system comprises an object observation module, a memory creation module, a structured memory storage module, a memory recall module and a reminding module. According to the method, the observation angle is flexible and active, the storage space is greatly saved, the query efficiency is improved, and based on graph structure memory, the method is closer to the memory and recall mode of human beings.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to an object-finding method and system for an intelligent following robot based on graph structure memory. Background Art

[0002] In the current object finding system, there are two main common solutions: 1) Observation system based on fixed cameras or wearable devices: Such systems usually rely on fixed cameras placed in the environment or wearable cameras on the user. The user's behavior is recorded through video, supplemented by language description or subtitle information, to achieve the recording and recall of objects. 2) Video-language based memory expression: Most systems use the "video + text description" method for memory storage and recall. For example, when the user places the key, the system records the entire video and attaches a semantic description. When the user makes a query, the system completes the reminder by playing back the relevant video segment or voice output description.

[0003] The main shortcomings of existing technologies are: 1) Limited viewing angle: Fixed cameras cannot move with the user, and their field of view is easily blocked, making it impossible to capture the complete scene; wearable devices are prone to losing key perspectives or causing user discomfort. 2) High storage costs and low efficiency: Video data is highly redundant, especially large-scale continuous recording of behavior, which leads to huge storage and processing burdens and is not suitable for deployment on service robots or wearable devices with limited computing resources. 3) Inconsistency with human cognitive mechanisms: Video and language are linear sequence data, which makes it difficult to support structure-based multi-level information extraction and reasoning, and lacks the efficient indexing and generalization capabilities of human-like memory. Summary of the Invention

[0004] The present invention aims to address the technical problems of the existing technology, such as lack of flexible observation capabilities, large amount of stored data, low query efficiency, and insufficiently structured memory expression. The present invention provides an object-finding method and system for an intelligent following robot based on graph structure memory. The technical solution adopted is:

[0005] An object-finding method for an intelligent following robot based on graph structure memory includes the following steps:

[0006] S1. The robot follows the user's movements in real time, actively perceives the user's behavior, and obtains observation results;

[0007] S2. organizing the observation results into a graph structure to form an initial pragmatic memory map;

[0008] S3. forming a structured memory based on the initial pragmatic memory map, wherein the structured memory includes a working memory that is continuously maintained and updated and a long-term memory that stores the character's behavioral habits;

[0009] S4. performing a query in the working memory or the long-term memory according to the question raised by the user to obtain a query result;

[0010] S5. Output a voice reminder according to the query result.

[0011] In one embodiment of the present invention, step S1 includes:

[0012] S11. The robot is equipped with a camera and a sensor, and inputs the extracted video frames into a visual model. The visual model determines whether the user has a hand-object interaction action.

[0013] S12. When the hand-object interaction action occurs, the video frame is determined to be a key frame, key frame recognition is triggered, the user behavior, environment and handheld object in the key frame are identified, the properties of the object in interaction and the background environment are recorded, and the observation result is obtained.

[0014] In one embodiment of the present invention, step S2 includes: organizing the observation results into a graph structure in a node-edge manner, wherein the graph structure represents the association between objects, scenes, people, and actions to form an initial pragmatic memory map.

[0015] In one embodiment of the present invention, the graph structure is converted into a behavior of updating memory library nodes, and the memory library is a five-layer pyramid structure of scenes, rooms, areas, large objects and small objects; the nodes represent geographic locations or objects, and the edges represent subordinate relationships.

[0016] In one embodiment of the present invention, step S3 includes:

[0017] Based on the behavioral data obtained from the video, a graph structure overlay mechanism is used to update the state of the graph nodes, continuously maintain and update the graph structure, and form a working memory;

[0018] The system records the language scene of each behavior into a log, and regularly inputs the log into the language model. The language model regularly organizes the log, summarizes the user's behavior patterns, and forms long-term memory.

[0019] In one embodiment of the present invention, step S4 includes:

[0020] In the working memory or the long-term memory, the robot uses the language large model and vector database technology to retrieve memory information close to the user's question, and continuously and actively interacts with the user to induce detailed information to accurately lock the object in the memory, thereby achieving the purpose of the query.

[0021] In one embodiment of the present invention, all fifth-layer leaf nodes in the memory library are text-vectorized and stored in the vector library. Then, the user's question is also converted into a vector, and similarity matching is performed in the vector library to obtain the closest leaf node and lock the strongly related objects in the memory library.

[0022] In one embodiment of the present invention, step S5 includes: after locking a strongly related object in the memory bank, inputting the object, its attributes, and its location information in the memory bank into the large language model, interacting with the user through the large language model, and finally accurately answering the user's question and outputting a voice reminder.

[0023] An intelligent following robot object-finding system based on graph structure memory includes:

[0024] An object observation module is used to follow the user's movements in real time, actively perceive the user's behavior, and obtain observation results;

[0025] A memory creation module, configured to organize the observation results into a graph structure to form an initial pragmatic memory map;

[0026] A structured memory storage module, configured to form structured memory based on the initial pragmatic memory map, wherein the structured memory includes a continuously maintained and updated working memory and a long-term memory that stores the character's behavioral habits;

[0027] A memory recall module, configured to query the working memory or the long-term memory according to the question raised by the user and obtain query results;

[0028] The reminder module is used to output a voice reminder according to the query result.

[0029] In one embodiment of the present invention, the memory creation module uses person following and trajectory positioning, scene recognition, handheld object recognition and user behavior recognition to achieve memory creation.

[0030] Beneficial effects of the present invention:

[0031] 1. The present invention's intelligent following robot object search method based on graph structure memory breaks through the limitations of fixed perspectives by having the robot actively follow the user's observation, making the observation angle flexible and active, and realizing the dynamic collection of key pragmatic events under the user's natural behavior; constructing a multi-level structured memory graph from semantic elements such as objects, scenes, and behaviors, and using structured graphs instead of video language data, greatly saving storage space and supporting longer time series and larger-scale information accumulation. Moreover, storing information based on graph structures facilitates efficient query, reasoning, and generalization, and is closer to human memory and recall methods; adopting the separation and linkage of working memory and long-term memory, and the state transfer and superposition update strategy based on graph structure, avoids video storage data redundancy and improves query efficiency.

[0032] 2. The intelligent following robot object-finding system based on graph structure memory of the present invention can adapt to a variety of home environments, user habits, and item categories, and has strong scalability and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of an object-finding method for an intelligent following robot based on graph structure memory provided by an embodiment of the present invention;

[0034] Figure 2 It is a schematic diagram of the structure of the initial pragmatic memory map provided by an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of graph structure data provided by an embodiment of the present invention;

[0036] Figure 4 This is a structural block diagram of an intelligent following robot object-finding system based on graph structure memory provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] The present invention provides an object-finding method for an intelligent following robot based on graph structure memory, which realizes an object recording mechanism for active observation and dynamic following, and can comprehensively capture user behavior; storing information based on the graph structure facilitates efficient query, reasoning, and generalization, and is closer to human memory and recall methods.

[0039] Refer to the attached Figure 1 The object-finding method of an intelligent following robot based on graph structure memory includes the following steps:

[0040] S1. The robot follows the user's movements in real time, actively perceives the user's behavior, and obtains observation results;

[0041] S2. Organize the observation results into a graph structure to form an initial pragmatic memory map;

[0042] S3. Based on the initial pragmatic memory map, structured memory is formed. Structured memory includes working memory that is constantly maintained and updated, and long-term memory that preserves the character's behavioral habits.

[0043] S4. Based on the question raised by the user, query the working memory or long-term memory to obtain the query results;

[0044] S5. Output a voice reminder according to the query result.

[0045] Step S1 of the embodiment of the present invention includes:

[0046] S11. The robot is equipped with cameras and sensors, which input the video frames extracted every second into the visual model. The visual model then determines whether the user has any hand-object interaction.

[0047] S12. When there is a hand-object interaction action, the video frame is determined to be a key frame, triggering key frame recognition, identifying the user behavior, environment and handheld object in the key frame, recording the attributes of the object being interacted and the background environment, and obtaining observation results, such as whether there is a bed and a black table behind.

[0048] This embodiment continuously acquires visual information through a depth camera installed on the mobile robot chassis, uses a human skeleton recognition model to obtain the user's position in the field of view, combines it with the map scanned in advance by the robot chassis, calculates the user's position on the map, and controls the chassis to continuously follow the user and record the user's behavior.

[0049] After obtaining a continuous video stream of user actions, a large multimodal visual model will be used to determine in real time whether the user has any actions that interact with objects in the environment. For example, the Qwen2.5-VL-7B model can achieve this effect. This technology belongs to existing technology and will not be described in detail here. Focus on two behaviors: picking up something from something and putting something on something. Whenever such a behavior is recognized, the nearby video frame will be determined to be a key frame, triggering key frame recognition. At this time, the multimodal model will be used to make a detailed description of the person holding the object and the overall environment in the key frame, that is, user behavior, environment, and handheld object recognition.

[0050] Step S2 of the embodiment of the present invention includes: forming graph structure data based on the observation results, and updating and maintaining the graph structure data at any time as the observation results are continuously generated. The observation results are organized into a graph structure in a node-edge manner. The graph structure represents the relationship between objects, scenes, people, and actions, forming an initial pragmatic memory map. Figure 2 Attached Figure 2In the graph, a five-layer pyramid structure is formed, which includes scenes, rooms, regions, large objects, and small objects.

[0051] Step S1 identifies information that is obtained in real time based on the video stream. After obtaining the information, the behavior is converted into the behavior of updating the memory node. The memory is a five-layer pyramid structure of scenes, rooms, areas, large objects and small objects. Nodes represent geographic locations or objects, and edges represent subordinate relationships. For example, there is a table in the room and a cup on the table. The graph structure data of this embodiment refers to the attached Figure 3 .

[0052] Memory maps are stored in two layers of memory: working memory and long-term memory. Working memory represents the immediate spatial state of objects in a specific scene. Long-term memory records abstract representations and interaction rules for all known objects and scenes (e.g., keys often appear in places like sofas and entryways).

[0053] Attachment Figure 3 For example, a mobile phone node can be added to the bedside table in the bedroom leisure area, or other nodes can be deleted. This ever-changing pyramid-structured scene memory library is working memory. Working in parallel is long-term memory. The system records the verbal scene description of each action (such as the time and minute when something was taken away) and regularly organizes these logs to summarize the behavioral patterns of people, that is, users. This is long-term memory. Long-term memory helps short-term memory and human-computer interaction modules. For example, when a user asks a question that is not clearly stored in memory, it can infer based on user habits.

[0054] Step S3 of the embodiment of the present invention includes:

[0055] Based on behavioral data captured through video, a graph structure overlay mechanism is employed to update the state of graph nodes, continuously maintaining and updating the graph structure to form working memory. Working memory enables memory evolution as events evolve. In addition to updating a single graph structure, each action is also stored. The system logs the language context of each action and regularly feeds this log into a large language model. The large language model regularly organizes the logs, summarizes user behavioral patterns, and forms long-term memory. Logs are not restricted by storage method and can be stored in databases such as txt and json. The storage format is to record [at one moment: did something] [at another moment: picked up something].

[0056] Long-term memory has two functions. One is to assist the storage of working memory. For example, when the background environment of a certain action is unclear and it is difficult to infer the user's interactive location in the scene, inferences can be made based on past experience. At the same time, the behavior of objects that often appear in one place suddenly appearing somewhere else can be marked as highlights. In addition, during the query step, when the current location of the object is not found in the working memory, the assistant can still infer and remind based on long-term memory to guide the user to actively search.

[0057] In step S4 of this embodiment of the present invention, if the user cannot recall the location of an object, they can ask the service robot for help. The robot then uses a large language model and vector database technology to retrieve information related to the user's question from its working or long-term memory. The robot then proactively interacts with the user to obtain detailed information, accurately identifying the object in memory and achieving the query. For example, the Qwen2.5-72B model can achieve this effect. This technology is currently available and will not be further described here.

[0058] All fifth-level leaf nodes in the memory library are vectorized and stored in the vector library. The user's questions are then vectorized and matched against the vector library for similarity. The closest leaf nodes are obtained, and strongly related objects in the memory library are locked. This embodiment uses an embedding model for vectorization, such as the Qwen / Qwen3-Embedding-8B model. In other words, the content of the working memory is vectorized and stored, the user's questions are vectorized and converted, and then records with similar semantics are matched, all of which serve as input to the large language model.

[0059] Step S5 of the embodiment of the present invention includes: after locking the strongly related objects in the memory library, the objects and their attributes and location information in the memory library are input into the large language model, and the large language model is used to interact with the user, and finally accurately answer the user's questions and output voice reminders.

[0060] Specifically, the user asks the robot a question via voice (e.g., "Where are my keys?"). The system converts the voice into text, locates the query intent, and performs a graph traversal and state activation in structured memory. After finding the relevant item nodes, it uses reasoning to derive the final location information, uses natural language to convey the exact location to the user, and announces the result via voice, thus fulfilling the function of a reminder assistant. The system utilizes a natural language-driven memory retrieval and recall process: user voice input → query intent recognition → structured memory activation → answer generation.

[0061] The object-finding method of the intelligent following robot based on graph structure memory of the present invention is completed through a combination of scene recognition, object recognition, behavior judgment, voice understanding, graph update, language generation, etc., which realizes the dynamic collection of key pragmatic events under the user's natural behavior, greatly saves storage space, supports longer time series and larger-scale information accumulation, avoids video storage data redundancy, improves query efficiency, and is closer to human memory and recall methods.

[0062] Based on the same inventive concept, the present invention also provides an intelligent following robot object searching system based on graph structure memory, which is applied to an intelligent following robot object searching method based on graph structure memory provided in the above embodiment. Please refer to the above for the method, and the similarities will not be repeated here.

[0063] Refer to the attached Figure 4 The intelligent following robot object-finding system based on graph structure memory includes:

[0064] The object observation module is used to follow the user's movements in real time, actively perceive the user's behavior, and obtain observation results;

[0065] The memory creation module is used to organize the observation results into a graph structure to form an initial pragmatic memory map;

[0066] A structured memory storage module is used to form structured memory based on the initial pragmatic memory map. Structured memory includes working memory that is constantly maintained and updated, and long-term memory that stores character behavior habits.

[0067] The memory recall module is used to query the working memory or long-term memory based on the questions raised by the user and obtain the query results;

[0068] The reminder module is used to output voice reminders based on the query results.

[0069] The memory creation module in this embodiment of the present invention uses person tracking and trajectory positioning, scene recognition, handheld object recognition, and user behavior recognition to achieve memory creation. This intelligent, follow-me robot object-finding system, based on graph-structured memory, enhances the memory system's semantic understanding capabilities, enabling the combined reasoning of object states, scene rules, and human preferences. It can adapt to a variety of home environments, user habits, and item categories, demonstrating strong scalability and application prospects.

[0070] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. An intelligent following robot object search method based on graph structure memory, characterized in that: Including steps: S1. The robot follows the user's movements in real time, actively perceives the user's behavior, and obtains observation results; S2. organizing the observation results into a graph structure to form an initial pragmatic memory map; S3. forming a structured memory based on the initial pragmatic memory map, wherein the structured memory includes a working memory that is continuously maintained and updated and a long-term memory that stores the character's behavioral habits; S4. performing a query in the working memory or the long-term memory according to the question raised by the user to obtain a query result; S5. Output a voice reminder according to the query result.

2. The method for finding an object by an intelligent following robot based on graph structure memory according to claim 1, characterized in that: The step S1 comprises: S11. The robot is equipped with a camera and a sensor, and inputs the extracted video frames into a visual model. The visual model determines whether the user has a hand-object interaction action. S12. When the hand-object interaction action occurs, the video frame is determined to be a key frame, key frame recognition is triggered, the user behavior, environment and handheld object in the key frame are identified, the properties of the object in interaction and the background environment are recorded, and the observation result is obtained.

3. The object-finding method for an intelligent following robot based on graph structure memory according to claim 2, characterized in that: The step S2 includes: organizing the observation results into a graph structure in a node-edge manner, wherein the graph structure represents the association between objects, scenes, people, and actions, and constitutes an initial pragmatic memory map.

4. The object-finding method for an intelligent following robot based on graph structure memory according to claim 3, characterized in that: The graph structure is converted into a behavior of updating memory library nodes, where the memory library is a five-layer pyramid structure of scenes, rooms, areas, large objects, and small objects; the nodes represent geographic locations or items, and the edges represent subordinate relationships.

5. The method for finding an object by an intelligent following robot based on graph structure memory according to claim 4, characterized in that: The step S3 comprises: Based on the behavioral data obtained from the video, a graph structure overlay mechanism is used to update the state of the graph nodes, continuously maintain and update the graph structure, and form a working memory; The system records the language scene of each behavior into a log, and regularly inputs the log into the language model. The language model regularly organizes the log, summarizes the user's behavior patterns, and forms long-term memory.

6. The method for finding an object by an intelligent following robot based on graph structure memory according to claim 5, characterized in that: The step S4 comprises: In the working memory or the long-term memory, the robot uses the language large model and vector database technology to retrieve memory information close to the user's question, and continuously and actively interacts with the user to induce detailed information to accurately lock the object in the memory, thereby achieving the purpose of the query.

7. The method for finding an object by an intelligent following robot based on graph structure memory according to claim 6, characterized in that: All fifth-layer leaf nodes in the memory library are vectorized and stored in the vector library. Then, the user's question is also converted into a vector, and similarity matching is performed in the vector library to obtain the closest leaf node and lock the strongly related objects in the memory library.

8. The object-finding method for an intelligent following robot based on graph structure memory according to claim 7, characterized in that: The step S5 includes: after locking the strongly related objects in the memory bank, inputting the objects, their attributes, and their location information in the memory bank into the large language model, interacting with the user through the large language model, and finally accurately answering the user's questions and outputting voice reminders.

9. An intelligent following robot object-finding system based on graph structure memory, characterized by: include: An object observation module is used to follow the user's movements in real time, actively perceive the user's behavior, and obtain observation results; A memory creation module, configured to organize the observation results into a graph structure to form an initial pragmatic memory map; A structured memory storage module, configured to form structured memory based on the initial pragmatic memory map, wherein the structured memory includes a continuously maintained and updated working memory and a long-term memory that stores the character's behavioral habits; A memory recall module, configured to query the working memory or the long-term memory according to the question raised by the user and obtain query results; The reminder module is used to output a voice reminder according to the query result.

10. The intelligent following robot object search system based on graph structure memory according to claim 9, characterized in that: The memory creation module uses character following and trajectory positioning, scene recognition, handheld object recognition and user behavior recognition to achieve memory creation.

Images