An object finding method, system, device and readable storage medium

Abstract

The application discloses an object searching method, system, device and readable storage medium in the technical field of computer application, and the method comprises the steps of obtaining an object image located in a target space, inputting the object image into a spatial visual language model for three-dimensional space reasoning to obtain a natural language text, extracting structured data from the natural language text, recording object memories by using the structured data, and writing the object memories into a memory bank with a relational database and a vector index bank, receiving a query text for querying a target object located in the target space, extracting structured semantic elements from the query text, respectively querying target object memory records corresponding to the structured semantic elements from the memory bank by using the relational database and the vector index bank, generating a response text comprising a scene and a spatial position by using the target object memory records, and outputting the response text. The application realizes an end-to-end automatic process from original perception to memory service.

Description

Technical Field

[0001] This application relates to the field of computer application technology, and in particular to a method, system, device and readable storage medium for object finding. Background Technology

[0002] The core of the "What You See Is What You Record - What You Ask Is What You Answer" end-to-end spatial memory service is to enable the system to automatically record spatial information it sees and accurately answer related questions. "What You See Is What You Record" means that through visual / spatial perception, it automatically captures and structures and stores information such as scenes, objects, locations, and layouts to form a searchable spatial memory bank; "What You Ask Is What You Answer" means that when a user asks a question in natural language, the system quickly retrieves its memory and directly provides an accurate answer based on the corresponding spatial information. However, currently, the spatial cognition capabilities of robots are not coherent and cannot meet the real-time requirements for object retrieval in practical applications.

[0003] In summary, how to provide effective end-to-end spatial memory services is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide an object finding method, system, device, and readable storage medium to provide a full-process spatial memory service and achieve real-time response to object finding needs.

[0005] To solve the above-mentioned technical problems, this application provides the following technical solution: An object finding method includes: An image of an object located in a target space is acquired, and the object image is input into a spatial visual language model for three-dimensional spatial reasoning to obtain natural language text; wherein, the natural language text describes the scene and spatial location of the object in the object image; Structured data is extracted from the natural language text, and the object memory is recorded using the structured data and written into a memory bank with a relational database and a vector index library; Receive query text that queries a target object located in the target space, and extract structured semantic elements from the query text; The memory records of the target objects corresponding to the structured semantic elements are queried from the memory bank using the relational database and the vector index library, respectively. The target object's memory records are used to generate response text including scene and spatial location, and the response text is output.

[0006] Preferably, querying the memory record of the target object corresponding to the structured semantic element from the memory bank using the relational database and the vector index library respectively includes: Transform the structured semantic elements into semantic vectors; The semantic vectors are used to perform an approximate nearest neighbor search in the vector index library to obtain candidate object memory records; Generate a structured query language statement corresponding to the structured semantic elements; Using the structured query language statements, a joint filtering process is performed in the relational database to obtain candidate object memory records; The memory record of the target object is determined by using the memory record of the candidate object.

[0007] Preferably, determining the target object memory record using candidate object memory records includes: A multi-dimensional relevance assessment was performed on the candidate object memory records to obtain multiple relevance scores; Multiple relevance scores of the same candidate object's memory records are weighted and summed to obtain a comprehensive matching score; The target object memory record is determined from the candidate object memory records using the comprehensive matching score.

[0008] Preferably, querying the memory record of the target object corresponding to the structured semantic element from the memory bank using the relational database and the vector index library respectively includes: The target object memory record is queried from the cached object memory record using the structured semantic elements; the cache stores hot data corresponding to the relational database and the vector index library, and evicts the least recently used data; If the object is not found in the cache, the object memory record that is found in the cache will be identified as the target object memory record. If the cache misses, the memory record of the target object corresponding to the structured semantic element is retrieved from the memory bank using the relational database and the vector index library, respectively.

[0009] Preferably, the generation of response text including scene and spatial location is achieved by utilizing the target object's memory record, including: Determine the confidence level of the memory record of the target object; Select a target template that matches the confidence level from the templates; The key fields from the memory record of the target object are filled into the placeholders in the target template to obtain the initial text; The initial text is polished to obtain the response text.

[0010] Preferably, the object memory is recorded using structured data and written into a memory bank with a relational database and a vector index library, including: After assigning vector numbers to the object names corresponding to the structured data, they are added to the vector index library; Extract the complete fields corresponding to each individual object from the structured data to obtain object memory records, and write the object memory records into the relational database.

[0011] Preferably, writing the object memory record into the relational database includes: Query the relational database to see if there exists an object memory record with the same name as the object memory record to be written; If so, compare the similarity between the memory records of objects with the same name to be written into the memory records; If the similarity is lower than a first threshold, the memory record of the object to be written is written into the relational database; If the similarity is higher than the second threshold, update the timestamp of the memory record of the same-named object in the relational database.

[0012] An object-finding system, comprising: The modal perception and spatial understanding central module is used to acquire images of objects located in the target space, and input the object images into a spatial visual language model for three-dimensional spatial reasoning to obtain natural language text; wherein, the natural language text describes the scene and spatial location of the objects in the object images; The hierarchical memory storage and adaptive update module is used to extract structured data from the natural language text, use the structured data to record object memories, and write them into a memory bank with a relational database and a vector index library; The intelligent retrieval and multi-dimensional matching engine module is used to receive query text that queries for target objects located in the target space, and extract structured semantic elements from the query text; and use the relational database and the vector index library to query the memory records of target objects corresponding to the structured semantic elements from the memory bank respectively. The Natural Language Generation and Integrated Response Module is used to generate response text including scene and spatial location using the memory records of the target object, and output the response text.

[0013] An electronic device, comprising: Memory, used to store computer programs; A processor for implementing the steps of the object-finding method described above when executing the computer program.

[0014] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the object-finding method described above.

[0015] The method provided in this application provides an image of an object located in a target space. This image is then input into a spatial visual language model for 3D spatial reasoning to obtain natural language text. The natural language text describes the scene and spatial location of the object in the image. Structured data is extracted from the natural language text, and object memory records are created using this structured data and written into a memory database containing a relational database and a vector index. A query text requesting information about a target object located in the target space is received, and structured semantic elements are extracted from the query text. The memory records corresponding to the structured semantic elements are retrieved from the memory database using both the relational database and the vector index. A response text including the scene and spatial location is generated using the target object memory records, and the response text is then output.

[0016] This application's analysis reveals that, in terms of spatial perception and understanding, most computer systems only perform object category recognition and rough localization, lacking precise descriptions and semantic expressions of spatial relationships between objects, and unable to translate users' natural language queries into specific spatial location information. Regarding the reliability of the memory system, the robot's visual processing flow is easily interrupted by environmental changes and computational resource limitations, lacking a state management mechanism to ensure the continuity and consistency of long-term memory. In the interaction and memory output stages, the system response is mechanical, memory retrieval efficiency is low, and it cannot provide an end-to-end memory service experience with spatiotemporal consistency and context awareness. The fragmentation of visual perception, spatial understanding, memory storage, and retrieval stages results in inconsistent spatial cognitive abilities in robots, failing to meet the real-time requirements for object localization queries in practical applications.

[0017] To address the aforementioned issues, this application automatically parses and transforms the acquired image information into a structured description containing object names, spatial relationships, and scene context, achieving automatic conversion from pixel data to semantic spatial memory. Then, to improve retrieval reliability and efficiency, the object memory records are stored in a memory database with relational databases and vector indexes based on the structured data. This allows for searching the object memory using both fuzzy and precise search methods in subsequent real-time queries. Furthermore, upon receiving the query text, an integrated retrieval pipeline of semantic parsing, multi-dimensional matching, and result output is automatically executed. Specifically, structured semantic elements are extracted from the query text using natural language processing technology; subsequently, the system performs both fuzzy and precise searches in the memory database; finally, the target object memory record matching the user's intent is obtained. Finally, a response text including scene and spatial location is generated based on the target object memory record and output, thereby quickly responding to query requests.

[0018] In other words, this application deeply integrates visual perception, spatial semantic understanding, and memory storage management. It achieves real-time acquisition and preprocessing of visual data, followed by depth estimation and spatial understanding. The generated structured memory is automatically stored in a hierarchical memory bank, and finally, by providing query services, it realizes an end-to-end automated process from raw perception to memory services.

[0019] Accordingly, embodiments of this application also provide an object finding system, device, and readable storage medium corresponding to the above-described object finding method, which have the aforementioned technical effects, and will not be elaborated further here. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the implementation of an object-finding method in an embodiment of this application. Figure 2 This is a schematic diagram of the structure of an object finding system according to an embodiment of this application; Figure 3 This is a schematic diagram of a robot spatial memory system according to an embodiment of this application; Figure 4 This is a schematic diagram of a robot spatial memory system according to an embodiment of this application; Figure 5 This is a schematic diagram of a sensing method in an embodiment of this application; Figure 6 This is a diagram of a multimodal perception and spatial understanding central architecture in an embodiment of this application; Figure 7 This is a schematic diagram illustrating a hierarchical memory storage and adaptive update method in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application; Figure 9 This is a schematic diagram of the specific structure of an electronic device in an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] To facilitate understanding and implementation of the object finding method provided in the embodiments of this application, the relevant technical terms involved herein will be explained below.

[0024] SpatialRGPT: A visual language model optimized for understanding spatial relationships. This model takes only RGB images as input and, through implicit learning of monocular depth cues and geometric priors, accurately understands and describes the three-dimensional spatial relationships between objects in an image, outputting a structured spatial semantic description.

[0025] Implicit depth perception refers to the ability of a machine vision system to infer three-dimensional spatial relationships by analyzing visual cues (such as occlusion, relative size, and perspective) in two-dimensional images without directly acquiring or inputting depth information. In this paper, it specifically refers to the technique of encoding depth perception capabilities into network weights through learning in spatial visual language models.

[0026] Tiered memory storage architecture: A system design that manages memory data by dividing it into multiple layers according to access frequency and storage requirements. In this paper, it specifically refers to a three-layer memory storage system consisting of a short-term cache (fast response layer), a vector index (semantic retrieval layer), and a relational database (persistent storage layer).

[0027] Multi-dimensional fusion matching: In the information retrieval process, this method considers multiple different features or criteria for comprehensive evaluation and ranking. In this paper, it specifically refers to the method of simultaneously calculating scores for multiple dimensions such as semantic relevance, temporal relevance, scene consistency, and description conformity during memory retrieval, and then weighting and fusing them to obtain a comprehensive matching score.

[0028] Adaptive update threshold: A judgment parameter dynamically adjusted by the system based on data characteristics and application scenarios to determine when to update stored information. In this article, it specifically refers to the similarity threshold used to judge whether the difference between the new location description and the historical record is significant during memory updates. If the similarity is lower than this threshold, it is determined that the location has changed significantly.

[0029] Semantic vector retrieval: an information retrieval technique based on semantic similarity. It finds relevant content by converting text into vector representations in a high-dimensional semantic space and then calculating the distance or similarity between vectors. In this paper, it is used to achieve fast approximate matching based on the semantic similarity of item names.

[0030] FAISS (Facebook AI Similarity Search) index: A library for efficient similarity search and dense vector clustering. It can quickly find the top K most similar vectors to a query vector from a massive amount of vectors, and is widely used in recommender systems, image retrieval, and other fields. In this paper, it is used to implement fast retrieval based on the semantic vectors of item names.

[0031] Structured parser: A processing module that converts unstructured or semi-structured data into structured data in a predetermined format. In this paper, it specifically refers to the post-processing component that parses natural language descriptions generated by spatial visual language models into standard JSON format.

[0032] Time decay function: In information retrieval or recommendation systems, this is a mathematical function used to quantify the decline in the value or relevance of information over time. In this paper, it is used to calculate the degree of matching between the timestamp of a memory record and the query time limit.

[0033] Confidence threshold range: A numerical range categorized according to different levels of certainty in the system's assessment of the results. In this paper, it is used to select expression templates with different levels of certainty based on the comprehensive matching score of the search results during the response generation phase.

[0034] Please refer to Figure 1 , Figure 1 This is a flowchart of an object-finding method according to an embodiment of this application. The method can be applied to a computer system with an image acquisition terminal (such as a robot with a vision sensor), and includes the following steps: S101. Obtain an image of an object located in the target space, and input the object image into a spatial visual language model for three-dimensional spatial reasoning to obtain natural language text.

[0035] Natural language text describes the scene and spatial location of objects in object images.

[0036] The target space can be any space where objects need to be found, such as a home environment, office space, or shop.

[0037] In practical applications, visual sensors in a computer system can be used to acquire images of objects in a target space. Multiple objects can be captured simultaneously in a single image, or only one object can be captured. The object itself can be any type of object located within the target space, such as a remote control or a key.

[0038] After obtaining the object image, an architecture combining implicit depth perception and prompt-guided generation can be adopted, integrating SpatialRGPT and a structured parser to achieve end-to-end transformation from raw pixel input to high-level spatial cognition. That is, pixel data is transformed into semantic spatial memory to obtain a natural language text description of the scene and spatial location of the object in the object image.

[0039] Using object images (such as RGB images) as input, the spatial visual language model, through large-scale training, has integrated a deep understanding of monocular depth (such as relative size, perspective, and occlusion), geometric prior knowledge (such as common object dimensions and standard layout patterns), and visual-language correspondences, thereby enabling implicit 3D spatial reasoning. The spatial visual language model takes RGB image features as input and combines them with pre-designed text prompt templates to generate natural language descriptions containing the spatial relationships between scenes and objects.

[0040] The output format can be strictly constrained using prompt templates. For example, the prompt template is: You are a robot's spatial memory module. Please analyze the current image, identify key objects, and accurately describe their spatial relationships. The output must be in strict JSON format: {'scene': 'scene name', 'objects': [{'name': 'object A', 'location':'location description A'}, …]}.

[0041] S102. Extract structured data from natural language text, use the structured data to record object memories, and write them into a memory bank with a relational database and a vector index library.

[0042] Because the output of the language model may have format deviations, in practical applications, a lightweight structured parser can be integrated into the system backend. This parser uses regular expressions or a JSON parsing library to extract and validate structured JSON objects from the natural language text generated by the spatial vision language model. If parsing fails, a backup plan is activated, such as keyword-based extraction or returning a predefined secure structure. Ultimately, a standardized spatial description JSON data stream is obtained, which serves as input for subsequent memory storage. Furthermore, during this process, raw images and intermediate feature data can be selectively output for system debugging and performance analysis.

[0043] By deeply integrating visual appearance features with implicit geometric spatial knowledge, unstructured sensor data is efficiently transformed into structured spatial semantic representations that robots can store and reason about, laying a solid perceptual and cognitive foundation for building a spatial memory system that truly has the ability to remember what it sees.

[0044] The structured data may include fields such as id (ID), item_name (object name), position_description (position description), scene_context (scene context), and timestamp (time stamp).

[0045] Once the structured data is obtained, the object memory record can be clearly defined. The structured data corresponding to a specific object can be regarded as an object memory record. The object memory record can include key information such as the object's name, scene, timestamp, and location.

[0046] To ensure the accuracy and comprehensiveness of subsequent object memory retrieval, this embodiment utilizes structured data to record the object memory and write it into a memory database containing a relational database and a vector index library. Specifically, the relational database provides precise retrieval, while the vector index library provides fuzzy retrieval.

[0047] For example, think of the memory bank as a large warehouse, and FAISS (a specific vector database) and SQLite (a specific relational database) as two different types of shelves within that warehouse. SQLite specifically stores structured data with explicit labels (such as time, location, and category). When you need to find a record based on specific conditions (like yesterday's cup in the living room), the system will perform precise key matching here. FAISS is the semantic association shelf, specifically storing vector data processed by AI. When you describe a vague concept (like simply saying "cup of hot water"), the system will calculate similarity here to help you associate it with a mug. Thus, together, they form the foundation of the memory bank: one handles precise location, and the other handles vague association. The results from both are aggregated and sorted to provide the most comprehensive and accurate answer.

[0048] In one specific embodiment of this application, object memory records are recorded using structured data and written into a memory library containing a relational database and a vector index library, including: After assigning vector numbers to the object names corresponding to the structured data, add them to the vector index library; Extract the complete fields corresponding to each individual object from the structured data to obtain the object memory record, and write the object memory record into the relational database.

[0049] Specifically, for each object description in the objects list in the JSON data, the data is first normalized by cleaning the object names (removing leading and trailing spaces, standardizing capitalization, etc.), filling in missing fields, and generating a unique timestamp.

[0050] Subsequently, a vector index update is performed, whereby the object name is encoded into a 512-dimensional semantic vector using a pre-trained sentence encoder model and added to the FAISS vector index which employs inner product similarity, laying the foundation for fast approximate nearest neighbor retrieval based on semantic similarity.

[0051] Next, relational database operations are performed to insert object memory records containing complete fields such as id, timestamp, item_name, item_description, location, position_description, scene_context, and created_at into the item_memories table of the SQLite database.

[0052] In one specific embodiment of this application, writing object memory records into a relational database includes: Query the relational database to see if there exists an object memory record with the same name as the object memory record to be written; If so, compare the similarity between the memory records of objects with the same name to be written into the memory records; If the similarity is below the first threshold, the memory record of the object to be written will be written into the relational database; If the similarity is higher than the second threshold, update the timestamp of the memory record of the same-named object in the relational database.

[0053] Specifically, during data storage, adaptive update judgments can be implemented to effectively update data. For example, the database can be first queried to see if there is a record of an item with the same name. If it exists, the similarity between the new location description and the latest historical record is calculated (by comparing text embedding vectors or keywords). If the similarity is lower than a set first threshold (e.g., 0.7, or other thresholds), it is determined to be a significant location change and a new record is inserted. If the similarity is high, such as above a second threshold (e.g., 0.7), only the timestamp may be updated or the record may be ignored, thus avoiding the storage of redundant or minor location change information.

[0054] In this embodiment, a short-term caching operation can also be performed simultaneously. The object name and its latest location description are stored as key-value pairs in a fixed-capacity double-ended queue (deque). This cache employs an LRU (Least Recently Used) eviction policy to provide sub-second query response capabilities for recently observed objects. Furthermore, the module periodically triggers maintenance tasks in the background (e.g., hourly execution), including merging overly similar records, archiving old records exceeding their retention period (e.g., 30 days) to control data size, and retraining and optimizing the FAISS index to ensure retrieval efficiency.

[0055] In other words, a hierarchical memory storage and adaptive update approach can be adopted to ensure memory persistence and consistency maintenance, enabling efficient storage, intelligent updates, and long-term management of structured data. Employing the design principles of hierarchical storage, asynchronous processing, and adaptive maintenance, a three-tiered storage architecture is constructed, comprising a short-term cache, vector indexes, and a relational database. Intelligent update strategies ensure the timeliness and accuracy of the stored information.

[0056] S103. Receive the query text of the target object located in the target space, and extract structured semantic elements from the query text.

[0057] Users can input object search requests through voice, text input, or other means, such as speaking: "Where is the remote control I put on the table?" or inputting text: "Where is the pencil?"

[0058] For speech, the query text can be obtained based on content recognition. For input text, the entire content of the input text or the part related to the query object can be directly determined as the query text.

[0059] After obtaining the query text, key information can be extracted from the query text to facilitate subsequent memory recording of the queried objects, thereby obtaining structured semantic elements.

[0060] Specifically, basic linguistic processing can be performed on the input natural language statements (i.e., query text), including word segmentation, filtering of irrelevant function words, and annotation of grammatical components. Based on this, the core semantic elements constituting the query intent are extracted from the statement: target entity (identifying the specific item the user intends to find, such as extracting the water glass from "the blue water glass I used yesterday" and associating it with the attribute "blue"), scene space constraint (parses explicit or implicit scene location information in the query statement, such as "in the living room" or "on the desk in the study"), time dimension constraint (identifies and parses time adverbs, such as "just now," "yesterday afternoon," and "last week," and converts them into specific timestamps or time intervals that the system can process), and attribute and relationship constraint (extracts descriptive information such as the item's color, size, owner, and the subordinate or referential relationships between items). These extracted structured semantic elements together constitute an internal query representation object, providing accurate navigation information for subsequent retrieval.

[0061] S104. Use the relational database and vector index library respectively to query the memory records of the target objects corresponding to the structured semantic elements from the memory bank.

[0062] After obtaining the structured semantic elements, object memory records can be queried in the memory database. Specifically, in this embodiment, to improve the comprehensiveness and reliability of the query results, two databases can be used to query the target object memory records corresponding to the structured semantic elements.

[0063] In one specific embodiment of this application, the method of querying the memory record of the target object corresponding to the structured semantic elements from the memory bank using a relational database and a vector index library respectively includes: Transform structured semantic elements into semantic vectors; Using semantic vectors, an approximate nearest neighbor search is performed in a vector index library to obtain candidate object memory records; Generate structured query language statements corresponding to structured semantic elements; Using structured query language statements, a joint filtering process is performed in a relational database to obtain candidate object memory records; The memory records of the target object are determined by using the memory records of the candidate objects.

[0064] Among them, determining the target object memory record using candidate object memory records includes: A multi-dimensional relevance assessment was performed on the candidate object memory records to obtain multiple relevance scores; Multiple relevance scores of the same candidate object's memory records are weighted and summed to obtain a comprehensive matching score; The target object memory record is determined from the candidate object memory records using the comprehensive matching score.

[0065] To maximize recall while maintaining retrieval efficiency, a multi-path parallel retrieval strategy is adopted, simultaneously scanning the memory from different angles and index structures for the same query intent: The semantic vector similarity retrieval path uses a pre-trained semantic encoding model to convert the extracted target entity text into high-dimensional semantic vectors, and then performs an approximate nearest neighbor search in the FAISS high-efficiency index library built on vector similarity, effectively overcoming the differences in lexical expression (such as associating a colloquial mug with a water cup in the memory); The structured relational database precise retrieval path dynamically generates precise structured query language (SQL) statements based on extracted structured constraints such as scene and time, and performs multi-condition joint filtering in the relational database to achieve high-precision retrieval.

[0066] The candidate set obtained during the recall phase may contain a large number of records. These can be finely sorted according to the query intent, selecting the few most relevant and reliable results. A comprehensive relevance score can be calculated for each candidate memory record. This score is a weighted fusion of sub-scores (i.e., relevance scores) from four dimensions: semantic relevance score (measures the semantic proximity between the item name stored in the memory record and the target entity in the query, calculated based on vector similarity), temporal relevance score (assesses the degree of conformity between the timestamp of the memory record and the query time limit, quantified using a time decay function), scene consistency score (determines whether the scene context to which the memory record belongs matches the user-defined scene), and description consistency score (checks whether the location description text in the memory record contains the attribute information mentioned in the query). The scores of each dimension are multiplied by preset, configurable weight coefficients and then summed to obtain the comprehensive matching score. The system finally sorts all candidate records in descending order of this score, selects the top N records (usually 1 to 3) as the final search results, and outputs them to the response generation module. Through this multi-dimensional fusion sorting mechanism, the system can intelligently balance multiple factors such as semantics, time, and context to ensure that the returned results not only match in name but also best fit the user's true intent in the overall context.

[0067] In one specific embodiment of this application, the method of querying the memory record of the target object corresponding to the structured semantic elements from the memory bank using a relational database and a vector index library respectively includes: The target object memory record is retrieved from the cached object memory record using structured semantic elements; the cache stores hot data corresponding to relational databases and vector indexes, and evicts the least recently used data. If the cache misses, the object memory record that is cache hit will be identified as the target object memory record; In the event of a cache miss, the memory record of the target object corresponding to the structured semantic elements is retrieved from the memory using both the relational database and the vector index library.

[0068] In other words, to accelerate query speed, caching can be used to quickly obtain the memory record of the target object. Specifically, in a high-speed short-term cache, the direct retrieval path can perform a fast matching check directly in the short-term cache that maintains the most recently observed records, providing sub-second response times for recent queries. If the cache is not hit, then the query is performed in the memory database, thereby shortening the query time.

[0069] In practical applications, the retrieved results are presented as a sorted list of target object memory records. This allows for the initial determination of the results' validity. If the result list is empty, a "No response found" strategy is automatically triggered, selecting a suitable comforting prompt template, such as "I haven't learned the location information for [item name] yet." If the result list is not empty, the top-ranked object memory record is identified as the target object memory record.

[0070] S105. Generate response text including scene and spatial location using the target object's memory record, and output the response text.

[0071] After obtaining the memory record of the target object, a response text containing the scene and spatial location of the queried object can be generated based on this memory record. Then, by outputting this response text, the object query can be completed.

[0072] In one specific embodiment of this application, response text including scene and spatial location is generated using target object memory records, including: Determine the confidence level of the memory record of the target object; Select the target template that matches the confidence level from the templates; Fill the placeholders in the target template with the key fields from the target object's memory record to obtain the initial text; The initial text is polished to obtain the response text.

[0073] The retrieved target object memory records are transformed into natural, fluent language that conforms to human communication habits, thereby obtaining and outputting response text, improving the friendliness and usability of the service.

[0074] In practical applications, the confidence level can be determined based on the comprehensive matching score of the target object's memory records. The response template can be dynamically selected according to the preset confidence threshold range: high confidence response (when the matching score exceeds a high threshold such as 0.8, a positive and clear statement template is used to directly inform the item's location), medium confidence response (when the matching score is in the medium range, a slightly suggestive or advisory statement template is used, such as "According to the most recent record, [item] may be located at a certain location"), and low confidence prompt (when the matching score is low, a cautious prompt template is used, clearly stating that the information may be inaccurate and suggesting that the user verify).

[0075] After selecting a template, the key fields (such as item name, specific location description, scene, and recording time) from the target object's memory record are accurately filled into the corresponding placeholders in the template.

[0076] Furthermore, to enhance the naturalness of the response, the module performs language polishing: humanizing time expressions (converting precise timestamps into colloquial expressions, such as 2:30 pm today, yesterday), simplifying location descriptions (simplifying and rewriting lengthy or technical descriptions into colloquial language), and optimizing tone and coherence (ensuring smooth and coherent sentences, and adding appropriate tone words and connecting sentences according to the response strategy).

[0077] After generating the final response text, the corresponding interaction can be completed according to the delivery channel integrated with the system. In text interaction scenarios, the text response is presented directly through a chatbot or application interface; in voice interaction scenarios, the response text is sent to a speech synthesis engine to generate a voice reply, which is then played through the robot's speaker. Simultaneously, the complete interaction context (including the original query, search results, and generated response) is recorded in the system log for service analysis, quality assessment, and system optimization.

[0078] The method provided in this application provides an image of an object located in a target space. This image is then input into a spatial visual language model for 3D spatial reasoning to obtain natural language text. The natural language text describes the scene and spatial location of the object in the image. Structured data is extracted from the natural language text, and object memory records are created using this structured data and written into a memory database containing a relational database and a vector index. A query text requesting information about a target object located in the target space is received, and structured semantic elements are extracted from the query text. The memory records corresponding to the structured semantic elements are retrieved from the memory database using both the relational database and the vector index. A response text including the scene and spatial location is generated using the target object memory records, and the response text is then output.

[0079] This application's analysis reveals that, in terms of spatial perception and understanding, most computer systems only perform object category recognition and rough localization, lacking precise descriptions and semantic expressions of spatial relationships between objects, and unable to translate users' natural language queries into specific spatial location information. Regarding the reliability of the memory system, the robot's visual processing flow is easily interrupted by environmental changes and computational resource limitations, lacking a state management mechanism to ensure the continuity and consistency of long-term memory. In the interaction and memory output stages, the system response is mechanical, memory retrieval efficiency is low, and it cannot provide an end-to-end memory service experience with spatiotemporal consistency and context awareness. The fragmentation of visual perception, spatial understanding, memory storage, and retrieval stages results in inconsistent spatial cognitive abilities in robots, failing to meet the real-time requirements for object localization queries in practical applications.

[0080] To address the aforementioned issues, this application automatically parses and transforms the acquired image information into a structured description containing object names, spatial relationships, and scene context, achieving automatic conversion from pixel data to semantic spatial memory. Then, to improve retrieval reliability and efficiency, the object memory records are stored in a memory database with relational databases and vector indexes based on the structured data. This allows for searching the object memory using both fuzzy and precise search methods in subsequent real-time queries. Furthermore, upon receiving the query text, an integrated retrieval pipeline of semantic parsing, multi-dimensional matching, and result output is automatically executed. Specifically, structured semantic elements are extracted from the query text using natural language processing technology; subsequently, the system performs both fuzzy and precise searches in the memory database; finally, the target object memory record matching the user's intent is obtained. Finally, a response text including scene and spatial location is generated based on the target object memory record and output, thereby quickly responding to query requests.

[0081] In other words, this application deeply integrates visual perception, spatial semantic understanding, and memory storage management. It achieves real-time acquisition and preprocessing of visual data, followed by depth estimation and spatial understanding. The generated structured memory is automatically stored in a hierarchical memory bank, and finally, by providing query services, it realizes an end-to-end automated process from raw perception to memory services.

[0082] Corresponding to the above method embodiments, this application also provides an object finding system, and the object finding system described below can be referred to in correspondence with the object finding method described above.

[0083] See Figure 2 As shown, the system includes the following modules: The modal perception and spatial understanding central module 101 is used to acquire object images located in the target space and input the object images into the spatial visual language model for three-dimensional spatial reasoning to obtain natural language text; wherein, the natural language text describes the scene and spatial location of the object in the object image; The hierarchical memory storage and adaptive update module 102 is used to extract structured data from natural language text, use the structured data to record object memories, and write them into a memory bank with a relational database and a vector index library; The intelligent retrieval and multi-dimensional matching engine module 103 is used to receive query text that queries for target objects located in the target space, and extract structured semantic elements from the query text; it uses a relational database and a vector index library to query the memory records of target objects corresponding to the structured semantic elements from the memory bank respectively; The Natural Language Generation and Integrated Response Module 104 is used to generate response text including scene and spatial location by utilizing the target object's memory record, and output the response text.

[0084] The system provided in this application acquires an image of an object located in a target space, inputs the object image into a spatial visual language model for three-dimensional spatial reasoning, and obtains natural language text. The natural language text describes the scene and spatial location of the object in the object image. Structured data is extracted from the natural language text, and object memory records are created using the structured data and written into a memory bank containing a relational database and a vector index library. A query text is received querying for a target object located in the target space, and structured semantic elements are extracted from the query text. The memory records corresponding to the structured semantic elements are queried from the memory bank using both the relational database and the vector index library. A response text including the scene and spatial location is generated using the target object memory records, and the response text is output.

[0085] This application's analysis reveals that, in terms of spatial perception and understanding, most computer systems only perform object category recognition and rough localization, lacking precise descriptions and semantic expressions of spatial relationships between objects, and unable to translate users' natural language queries into specific spatial location information. Regarding the reliability of the memory system, the robot's visual processing flow is easily interrupted by environmental changes and computational resource limitations, lacking a state management mechanism to ensure the continuity and consistency of long-term memory. In the interaction and memory output stages, the system response is mechanical, memory retrieval efficiency is low, and it cannot provide an end-to-end memory service experience with spatiotemporal consistency and context awareness. The fragmentation of visual perception, spatial understanding, memory storage, and retrieval stages results in inconsistent spatial cognitive abilities in robots, failing to meet the real-time requirements for object localization queries in practical applications.

[0086] To address the aforementioned issues, this application automatically parses and transforms the acquired image information into a structured description containing object names, spatial relationships, and scene context, achieving automatic conversion from pixel data to semantic spatial memory. Then, to improve retrieval reliability and efficiency, the object memory records are stored in a memory database with relational databases and vector indexes based on the structured data. This allows for searching the object memory using both fuzzy and precise search methods in subsequent real-time queries. Furthermore, upon receiving the query text, an integrated retrieval pipeline of semantic parsing, multi-dimensional matching, and result output is automatically executed. Specifically, structured semantic elements are extracted from the query text using natural language processing technology; subsequently, the system performs both fuzzy and precise searches in the memory database; finally, the target object memory record matching the user's intent is obtained. Finally, a response text including scene and spatial location is generated based on the target object memory record and output, thereby quickly responding to query requests.

[0087] In other words, this application deeply integrates visual perception, spatial semantic understanding, and memory storage management. It achieves real-time acquisition and preprocessing of visual data, followed by depth estimation and spatial understanding. The generated structured memory is automatically stored in a hierarchical memory bank, and finally, by providing query services, it realizes an end-to-end automated process from raw perception to memory services.

[0088] In one specific embodiment of this application, the intelligent retrieval and multi-dimensional matching engine module is specifically used to convert structured semantic elements into semantic vectors; Using semantic vectors, an approximate nearest neighbor search is performed in a vector index library to obtain candidate object memory records; Generate structured query language statements corresponding to structured semantic elements; Using structured query language statements, a joint filtering process is performed in a relational database to obtain candidate object memory records; The memory records of the target object are determined by using the memory records of the candidate objects.

[0089] In one specific embodiment of this application, the intelligent retrieval and multi-dimensional matching engine module is specifically used to perform multi-dimensional relevance evaluation on candidate object memory records to obtain multiple relevance scores. Multiple relevance scores of the same candidate object's memory records are weighted and summed to obtain a comprehensive matching score; The target object memory record is determined from the candidate object memory records using the comprehensive matching score.

[0090] In one specific embodiment of this application, the intelligent retrieval and multi-dimensional matching engine module is specifically used to query the target object memory record from the cached object memory record using structured semantic elements; the cache stores hot data corresponding to the relational database and vector index library, and evicts the least recently used data; If the cache misses, the object memory record that is cache hit will be identified as the target object memory record; In the event of a cache miss, the memory record of the target object corresponding to the structured semantic elements is retrieved from the memory using both the relational database and the vector index library.

[0091] In one specific embodiment of this application, the natural language generation and integrated response module is specifically used to determine the confidence level of the target object memory record; Select the target template that matches the confidence level from the templates; Fill the placeholders in the target template with the key fields from the target object's memory record to obtain the initial text; The initial text is polished to obtain the response text.

[0092] In one specific embodiment of this application, the hierarchical memory storage and adaptive update module is specifically used to add the object names corresponding to the structured data to the vector index library after vector numbering. Extract the complete fields corresponding to each individual object from the structured data to obtain the object memory record, and write the object memory record into the relational database.

[0093] In one specific embodiment of this application, the hierarchical memory storage and adaptive update module is specifically used to query whether there is an object memory record with the same name as the object memory record to be written in the relational database; If so, compare the similarity between the memory records of objects with the same name to be written into the memory records; If the similarity is below the first threshold, the memory record of the object to be written will be written into the relational database; If the similarity is higher than the second threshold, update the timestamp of the memory record of the same-named object in the relational database.

[0094] To facilitate those skilled in the art to better understand and implement the technical solutions provided in this application, the following uses a robot vision system as an example to explain in detail how to implement the technical solutions provided in this application.

[0095] Robot vision systems have evolved from simple object detection to intelligent systems with a certain level of scene understanding. However, in achieving a true end-to-end spatial memory service where what you see is what you remember and what you ask is what you answer, the relevant technologies still have the following shortcomings: In terms of spatial perception and understanding, most systems only complete object category identification and rough localization, lacking precise description and semantic expression of spatial relationships between objects, and are unable to convert users' natural language queries into specific spatial location information.

[0096] Regarding the reliability of memory systems, robot vision processing is susceptible to interruptions due to environmental changes and computational resource limitations, lacking a state management mechanism to ensure the continuity and consistency of long-term memory. In the interaction and memory output stages, the system response is mechanical, memory retrieval efficiency is low, and it cannot provide an end-to-end memory service experience with spatiotemporal consistency and context awareness.

[0097] The fragmentation of visual perception, spatial understanding, memory storage, and retrieval in related technologies leads to inconsistent spatial cognitive abilities in robots, failing to meet the real-time requirements for object location and retrieval in practical applications. Specifically, in these technologies, robot spatial perception and memory solutions typically suffer from systemic defects in achieving the complete process from visual observation to memory retrieval, including modal fragmentation, insufficient understanding accuracy, low retrieval efficiency, and poor memory timeliness. These defects manifest themselves as follows: (1) The gap between spatial relationship understanding and natural language description: The spatial understanding ability of robot vision systems largely relies on preset coordinate position descriptions or simple directional relationship words (such as left, right, front, and back). This approach cannot generate accurate and natural spatial position descriptions (such as the keyboard being on the left side of the desk, close to the monitor). Even if some systems integrate visual language models, they mostly remain at the object recognition level, lacking deep semantic analysis and natural language generation capabilities for complex spatial relationships between objects. This results in the system being unable to accurately translate visual observations into spatial descriptions that humans can understand, leading to a stiff user experience and limited information content.

[0098] (2) The continuity and consistency of spatial memory in dynamically changing environments: Robots operating in dynamically changing environments such as homes and offices often face real-world situations such as object movement and scene layout adjustments. Memory systems typically employ simple overwrite strategies or time decay mechanisms for memory updates, lacking an active maintenance mechanism for the spatiotemporal consistency of memory content. When the robot re-observes the same object, it cannot determine in real time whether memory needs to be updated or how to reasonably integrate old and new observations, leading to outdated or contradictory memory information. The accuracy and practicality of long-term memory cannot be guaranteed, seriously affecting system reliability and user experience.

[0099] (3) Matching accuracy issues between natural language query and spatial memory retrieval: Robot memory systems mostly use keyword matching or fixed-format commands for querying. To obtain the location information of a specific object, users must use the system's preset query statements and cannot use natural and flexible everyday language (such as "Where is the black remote control I left in the living room yesterday?") for interaction. Related vector retrieval technology also focuses on text semantic similarity and lacks the ability to comprehensively match multiple dimensions of temporal and scene contexts, failing to meet users' actual needs for fuzzy queries and precise returns, resulting in low levels of intelligence and humanization.

[0100] (4) Fragmented multimodal information processing links, lacking an integrated process of perception-understanding-memory: Related technical solutions generally treat visual perception, spatial understanding, memory storage and information retrieval as separate processing links. The robot needs to perform visual detection first, then pass the results to the understanding module, and finally store them in the memory bank. The process is cumbersome and there is information loss. The system fails to integrate depth perception, semantic understanding and memory management as a deeply integrated end-to-end process, which makes it impossible to provide users with a coherent service experience of real-time observation-instant memory-fast retrieval, thus weakening the practical value and application potential of the system.

[0101] In summary, existing technologies have yet to provide a highly integrated robotic spatial memory system capable of accurately understanding spatial relationships within a scene, maintaining memory consistency in dynamic environments, efficiently matching natural language queries, and seamlessly integrating multimodal processing. This application aims to systematically address these interconnected technical bottlenecks.

[0102] In other words, this application aims to address the systemic technical problems of related robot spatial memory systems, such as inaccurate perception, inconsistent memory, unintelligent retrieval, and disjointed processes. It proposes a spatial memory system and method that achieves an automated closed loop from multimodal perception to intelligent retrieval. The specific solution is as follows: (1) The problem of semantic conversion of visual observation into structured spatial description: The image information acquired by the robot vision system is automatically parsed and converted into a structured description containing object names, spatial relationships and scene context (e.g., {object: keyboard, position: on the left side of the desk near the monitor, scene: study}), realizing the automatic conversion from pixel data to semantic spatial memory.

[0103] (2) The problem of maintaining the timeliness and consistency of spatial memory in dynamic environments: In real-world scenarios where robots operate for a long time and the environment changes continuously, a memory update and fusion mechanism is designed to ensure that spatial memory can maintain accuracy and consistency over time. Specifically, this includes: new observation judgment based on similarity thresholds; memory priority adjustment with time decay weights; and intelligent resolution and fusion strategies for conflicting memories, thereby achieving continuous optimization and reliable maintenance of memory content in dynamic systems.

[0104] (3) Intelligent matching of natural language queries based on multi-dimensional similarity: An integrated query understanding and matching engine is built in the memory retrieval layer. After receiving a user's natural language query (e.g., where do I keep my water cup?), the engine can automatically execute an integrated retrieval pipeline of semantic parsing, multi-dimensional matching, and result ranking. The engine first needs to extract key item names, time modifiers, scene qualifiers, and other information from the query through natural language processing technology; then, the system performs multi-dimensional comprehensive retrieval in the memory bank, including semantic similarity matching, time relevance matching, and scene consistency matching; finally, the candidate results are ranked based on a weighted scoring mechanism, and the memory record that best matches the user's intent is returned. By deeply integrating multi-dimensional matching strategies, intelligent retrieval from fuzzy natural language to precise spatial memory is achieved, ensuring that the returned results are highly consistent with the user's needs in multiple dimensions such as semantics, time, and scene.

[0105] (4) Integration of multimodal perception and memory processing: At the system architecture level, visual perception, spatial semantic understanding, and memory storage management are deeply integrated, and a unified data flow and state management mechanism is designed. Specifically, real-time acquisition and preprocessing of visual data are required on the robot side, followed by depth estimation and spatial understanding. The generated structured memory is automatically stored in a hierarchical memory bank, and finally, query services are provided through standardized interfaces to achieve an end-to-end automated process from raw perception to memory services.

[0106] Based on this object-finding method, this application can construct, as follows: Figure 3 The SpatialRGPT and multi-dimensional retrieval robot spatial memory system and method are shown. This system constructs a collaborative workflow of multimodal perception, spatial semantic understanding, hierarchical memory management, and intelligent retrieval response, achieving complete automation from real-time robot observation to high-quality spatial memory services. Please refer to [reference needed]. Figure 2 The system specifically includes the following modules: (1) Multimodal perception and spatial understanding center, such as Figure 4 As shown, this module serves as the core of the system's intelligent sensing, responsible for... Figure 5 The robot's vision sensor (Figure C1) captures raw RGB images, which are then transformed into structured semantic descriptions containing precise three-dimensional spatial relationships. This module employs an innovative architecture that combines implicit depth perception with prompt-guided generation, integrating SpatialRGPT and a structured parser to achieve end-to-end conversion from raw pixel input to advanced spatial cognition.

[0107] To achieve a precise and structured description of scene spatial relationships, the system invokes a pre-loaded spatial visual language model. The core innovation of this model lies in the fact that, although it only uses RGB images as input, it has integrated a deep understanding of monocular depth (such as relative size, perspective, and occlusion relationships), geometric prior knowledge (such as common object dimensions and standard layout patterns), and visual-language correspondences through large-scale training, thereby enabling implicit 3D spatial reasoning. For example... Figure 6 As shown, this model takes RGB image features as input and combines them with a pre-designed text prompt template to generate a natural language description containing the spatial relationships between scenes and objects. The prompt template strictly constrains the output format, for example: You are a robot's spatial memory module. Please analyze the current image, identify key objects, and accurately describe their spatial relationships. The output must be in strict JSON format: {'scene': 'scene name', 'objects': [{'name': 'object A', 'location': 'location description A'},…]}.

[0108] Because the output of large language models may have format deviations, the system backend integrates a lightweight structured parser that uses regular expressions or a JSON parsing library to extract and validate structured JSON objects from the model-generated text. If parsing fails, a backup plan is activated, such as keyword-based extraction or returning a predefined secure structure. Finally, the module outputs a standardized spatial description JSON data stream as input to the subsequent memory storage module. Simultaneously, the module can selectively output raw images and intermediate feature data for system debugging and performance analysis.

[0109] This module, by deeply integrating visual appearance features with implicit geometric spatial knowledge, efficiently transforms unstructured sensor data into structured spatial semantic representations that robots can store and reason about, laying a solid perceptual and cognitive foundation for building a spatial memory system that truly has the ability to remember what it sees.

[0110] (2) Hierarchical memory storage and adaptive update module, such as Figure 7 As shown, this module is primarily responsible for the efficient storage, intelligent updating, and systematic management of structured spatial knowledge from the sensory center. Adopting a design principle of hierarchical storage, asynchronous processing, and adaptive decision-making, the module constructs a multi-layered, highly reliable memory management system to ensure the accuracy and availability of spatial memory during long-term operation.

[0111] Upon receiving structured JSON-formatted spatial description data, the module immediately initiates an independent asynchronous processing flow to avoid blocking the system's real-time sensing and response threads. The processing flow first standardizes the input data, including cleaning object names (removing extra spaces and standardizing character format), completing necessary fields, and generating globally unique timestamps. Subsequently, the module executes three layers of storage operations in parallel to achieve different levels of memory management: First, it updates the vector memory index, using a pre-trained sentence encoder model to convert the object name text into a high-dimensional semantic vector and adding it to a vector index built on the FAISS library. This index uses inner product similarity calculation, establishing the core capability for subsequent fast approximate nearest neighbor retrieval based on semantic similarity. Second, it performs relational memory storage, writing memory records containing complete fields (such as item_name, position_description, scene_context, timestamp, etc.) into a dedicated table in the SQLite relational database, providing a structured data foundation for supporting complex conditional queries and deep data analysis. During this process, the system simultaneously performs adaptive update judgment—by querying the database to see if there are historical records of items with the same name, and calculating the comprehensive similarity between the new location description and the latest historical record in terms of textual semantics and spatial relationships. If the similarity is lower than a preset threshold (e.g., 0.7), it is determined that the item's location has changed significantly, and the new observation is inserted as an independent record; if the similarity is high, it may only update the timestamp of the existing record or ignore the minor change, thus effectively avoiding data redundancy while ensuring the freshness of the memory. Finally, a short-term cache update is performed, storing the object name and its latest location description as key-value pairs in a fixed-capacity cache that follows the Least Recently Used (LRU) eviction policy, providing sub-second response capabilities for high-frequency queries of recent items.

[0112] All data operations are guaranteed to be independent and consistent through transaction mechanisms and asynchronously processed through message queues, maximizing overall system response performance while ensuring data reliability. This module, through its layered storage architecture and intelligent update strategy, effectively solves key problems common in traditional memory systems such as data bloat, retrieval efficiency degradation, and information obsolescence, providing a solid guarantee for the long-term stable operation of the spatial memory system.

[0113] (3) Intelligent retrieval and multi-dimensional matching engine: This module serves as the core of the system's intelligent query and decision-making. It is responsible for accurately parsing the user's query intent expressed in natural language and performing high-precision, multi-dimensional matching with the discrete spatial memory records stored in the memory bank. This module breaks through the limitations of traditional retrieval systems that rely on single keyword matching. Through a three-stage collaborative workflow of refined semantic parsing, multi-path parallel retrieval, and multi-dimensional fusion ranking, it achieves a reliable mapping from fuzzy, colloquial user queries to accurate and credible spatial location information.

[0114] The semantic parsing and query element extraction stage aims to deeply understand the semantic connotation and contextual constraints of the user's original query. The module receives natural language queries input by the user via voice or text (e.g., "Where is the black notebook I left on my desk in the study yesterday?"). First, it performs basic linguistic processing, including word segmentation, part-of-speech tagging, and named entity recognition. Based on this, the system employs a strategy combining rules and models to accurately extract the core semantic elements constituting the query intent: target entity (identifying the specific item the user intends to find, such as the notebook, and associating it with the attribute "black"), spatial context constraints (parsing the explicit or implicit location range in the query, such as "on the desk in the study"), temporal constraints (identifying and standardizing time modifiers, such as "yesterday," converting them into a processable time interval), and attribute and relation constraints (extracting modifier information such as color and owner). These elements together constitute a structured internal query representation object, providing a precise navigation framework for subsequent retrieval.

[0115] To maximize recall while maintaining efficiency, the multi-path parallel memory retrieval module employs a multi-path parallel retrieval strategy, simultaneously scanning the memory database from different dimensions and storage structures for the same query. Three retrieval paths are initiated simultaneously: the semantic vector retrieval path uses a pre-trained encoding model to convert the target entity text into semantic vectors, then performs an approximate nearest neighbor search in the FAISS high-efficiency index, effectively overcoming the differences in lexical expression (e.g., associating a mug with a water cup); the structured database retrieval path dynamically generates precise SQL query statements based on extracted structured conditions such as scene and time, performing multi-condition joint filtering in the SQLite relational database to achieve high-precision matching; and the high-speed cache retrieval path directly performs fast key-value matching in a short-term cache (circular buffer) that maintains recent records, providing sub-second response times for retrieving the latest observations. The three paths execute, returning a set of candidate memory records, forming a broad preliminary recall set.

[0116] The recall set may contain a large number of records. The core task of the multi-dimensional fusion sorting and result generation stage is to perform refined evaluation and sorting based on query intent. The system calculates a comprehensive matching score for each candidate memory record, which is a weighted fusion of sub-scores from four dimensions: semantic relevance score (based on vector similarity, measuring the semantic closeness of item names), temporal relevance score (using a time decay function to quantify the degree of conformity between the memory timestamp and the query time constraint), scene consistency score (evaluating the matching degree between the scene context of the memory record and the query scene constraint), and description conformity score (checking whether the location description text contains or implies the attributes and location information mentioned in the query). The scores of each dimension are multiplied by a configurable weight coefficient and then summed to obtain the final comprehensive matching score. The system sorts all candidate records in descending order of this score and selects the top N records (usually 1-3) as the optimal result, which is then output to the response generation module. Through this multi-dimensional fusion intelligent sorting mechanism, the system can go beyond simple name matching and return spatial memory information that best matches the user's true intent under the combined constraints of semantics, time, scene, and description.

[0117] (4) Natural Language Generation and Integrated Response Module: As the final delivery link for the system's memory service, this module, in addition to returning structured memory data via API, provides natural language responses during direct user interactions. When it needs to provide feedback on query results, this module converts the highest-ranking memory record into a fluent and natural language description (e.g., according to the record, the notebook you placed on your desk in the study yesterday is still in the center of the desk, near the lamp). If no matching record is found, a prompt message is generated (e.g., I haven't observed the item you mentioned yet; please confirm whether the item name or scene description is correct). This module realizes the conversion from internal data structures to user-friendly output, providing an intuitive and efficient human-computer interaction experience.

[0118] Through the deep collaboration and data closure of the four modules mentioned above, the entire system integrates discrete sensory inputs, complex spatial understanding, intelligent memory management, and user-friendly retrieval responses into a smooth, reliable, and complete organic whole. It not only realizes the full automation and intelligence of the robot's spatial memory process, but also significantly improves the robot's environmental cognition level and practical value through the closed-loop service capability of "what you see is what you remember" and "what you ask is what you answer".

[0119] Through the deep collaboration and coordination of the four major modules mentioned above, the entire system constructs a fully intelligent workflow from multimodal perception to intelligent memory services. It organically integrates discrete sensor data, specialized spatial understanding models, hierarchical memory storage mechanisms, and intelligent retrieval response modules into a highly collaborative, stable, reliable, and user-friendly closed-loop system architecture. This solution not only achieves end-to-end automation and intelligence of the robot's spatial memory workflow but also significantly expands the application boundaries of robots in fields such as home services, warehouse management, and intelligent navigation through the fusion of real-time perception, instant memory, and intelligent retrieval services. It provides users with a new environmental interaction experience that combines accuracy, timeliness, and ease of use, thereby enhancing the overall competitiveness and practical value of the robot system.

[0120] In actual implementation, the following steps can be referred to: Step 1: Environmental perception and spatial semantic understanding.

[0121] The robot captures RGB images of its environment using its onboard camera and inputs them into the spatial understanding central module. This module activates a pre-loaded Spatial Visual Language Model (SpatialRGPT), which, through its implicitly encoded depth perception capabilities—including learning monocular depth cues and visual-language feature alignment—can intelligently analyze the current field of view and perform spatial reasoning. Guided by specific text prompt templates, the model outputs a scene description in natural language. Subsequently, the system uses a structured parser to convert this description into a standardized, machine-readable format. For example, the system might generate a JSON structure like this: the scene is identified as a living room, and it includes key objects such as a television (located: hanging in the center of the wall opposite the sofa), a coffee table (located: approximately one meter in front of the sofa), and a remote control (located: lying flat on the upper left edge of the coffee table). This description is encapsulated as a JSON data packet and enters the system data stream.

[0122] Step 2: Hierarchical memory storage and adaptive updates.

[0123] The hierarchical memory storage and adaptive update module receives the aforementioned JSON data packet and performs systematic processing on each object description within it. Taking the remote control as an example, the module executes the following sequentially: 1) Short-term cache: Store the remote control and its location on the top left edge of the coffee table in a fixed-capacity cache based on the LRU strategy to provide millisecond-level response for immediate queries; 2) Vector Index: The remote control name is converted into a semantic vector by a sentence encoder and added to the FAISS vector index library, supporting fast retrieval based on semantic similarity; 3) Relational Database Storage and Conflict Resolution: The system queries the remote control's historical records in the `item_memories` table of the SQLite database. Assuming there's a record from yesterday showing the remote control in a sofa cushion crevice, the system doesn't simply overwrite it. Instead, it initiates an adaptive update decision: calculating the semantic similarity between the old and new location descriptions. If the similarity is below a preset threshold (e.g., 0.7), it's considered a significant location change, and a new record is inserted. If the similarity is high, it may only update the timestamp or ignore the record to avoid redundancy.

[0124] Step 3: Natural Language Query and Intelligent Retrieval.

[0125] Users can submit queries using natural language, such as: "Where is the remote control in the living room?" The intelligent search and multi-dimensional matching engine will then be activated. 1) Deep semantic parsing: Extracting the target entity (remote control), scene constraints (living room), and implicit time intent from the query; 2) Multi-path parallel retrieval: Synchronous execution—the semantic vector path searches for semantically similar items in the FAISS index; the structured database path executes SQL queries, such as SELECT... FROM item_memories WHERE item_name LIKE '%remote_control%' AND scene_context LIKE '%living room%'; This directly searches for the most recent record using the cache path. 3) Multi-dimensional fusion ranking: For all recalled candidate records, a comprehensive matching score is calculated from four dimensions: semantic relevance, temporal relevance, scene consistency, and description conformity, and then sorted in descending order of score. Finally, assuming that the remote control recorded in step 2 has the highest score at the top left corner of the coffee table in the living room, it is determined as the optimal result.

[0126] Step 4: Natural Language Response Generation and Delivery.

[0127] The natural language generation and response module receives the optimal result and its matching score. The module selects a response template based on the confidence range of the score: affirmative statements for high confidence, suggestive statements for medium confidence, and cautious prompts for low confidence. After selecting a template, key fields such as item name, location, scene, and time from the result are filled in, and the language is polished (e.g., making the time more conversational and the description more concise). The generated response text is delivered according to the interaction scenario: displayed directly in the text interface; or converted into speech and played back via a TTS engine in voice interaction. Simultaneously, a complete interaction log is recorded for analysis and optimization. Thus, the system completes a full technical loop from environmental perception → spatial understanding → memory storage → intelligent retrieval → natural response.

[0128] As can be seen, this application, through the deep collaboration of four major modules, achieves end-to-end automation from real-time environmental awareness to high-quality memory query services, and has achieved the following significant technical effects and application advantages: This system achieves an organic integration of perception and memory, as well as semantic cognition: It directly transforms raw RGB images into structured semantic descriptions rich in spatial relationships through the Spatial Visual Language Model (SpatialRGPT), overcoming the limitations of traditional vision systems that only output detection boxes or category labels. This ability to remember what is seen bridges the semantic gap between raw perceptual data and high-level memory knowledge, establishing a true foundation for spatial cognition in robots.

[0129] This ensures the long-term consistency, timeliness, and efficiency of the memory system: A unique three-layer storage architecture and adaptive update mechanism work together: short-term caching guarantees sub-second response times for recent queries; vector indexes support fast fuzzy retrieval based on semantic similarity; and a relational database ensures data persistence and complex query capabilities. The intelligent update strategy, through location similarity threshold judgment, effectively distinguishes between significant changes in item location and minor alterations, avoiding data redundancy and contradictions, enabling the memory to continuously optimize rather than degrade over time.

[0130] It provides a precise, intelligent, and human-like natural language interaction experience: Based on an intelligent retrieval engine that combines refined parsing, parallel retrieval, and multi-dimensional ranking, the system can deeply understand complex natural language queries from users, including vague references, contextual information, and temporal intent. Through multi-path parallel retrieval and comprehensive scoring across four dimensions (semantics, time, context, and description), the system intelligently returns results that best match the user's true intent. Combined with a dynamic template-based and language-polishing response generation mechanism, the chatbot can provide feedback in a clear, natural, and approachable manner, greatly enhancing the friendliness and practicality of human-computer interaction.

[0131] A general system framework of perception-cognition-memory-recall-response is constructed: This application contributes a complete system architecture with clear layers, decoupled modules, and standardized interfaces. This architecture integrates the entire process of multimodal perception, spatial semantic understanding, hierarchical memory management, and intelligent retrieval interaction into a self-sustaining and continuously evolving organic whole. Each module collaborates through standardized data interfaces, forming a complete closed loop of perception-cognition-memory-recall-response. This provides a universal, reliable, and scalable core capability for spatial environment cognition and memory services for various service robots (such as those for home companionship, warehouse management, and security patrol).

[0132] In summary, this application effectively addresses the core pain points in the robotics field, such as fragmented environmental cognition and rigid memory services. By deeply integrating cutting-edge visual language models, efficient retrieval technologies, and robotic systems engineering, it creates a core module of an intelligent agent that truly possesses the abilities of observation, understanding, memory, and recall. This lays a solid technical foundation for improving the environmental autonomy, task practicality, and human-machine collaboration experience of robots, and has broad prospects for industrial application.

[0133] Corresponding to the above method embodiments, this application also provides an electronic device. The electronic device described below and the object finding method described above can be referred to in correspondence.

[0134] See Figure 8 As shown, the electronic device includes: Memory 332 is used to store computer programs; The processor 322 is used to implement the steps of the object search method in the above method embodiment when executing a computer program.

[0135] For details, please refer to Figure 9 , Figure 9 This is a schematic diagram of the specific structure of an electronic device provided in this embodiment. The electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the electronic device 301.

[0136] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0137] The steps in the object-finding method described above can be implemented using the structure of an electronic device.

[0138] Corresponding to the above method embodiments, this application also provides a readable storage medium. The readable storage medium described below can be referred to in conjunction with the object finding method described above.

[0139] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the object search method described in the above method embodiments.

[0140] The readable storage medium can specifically be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other readable storage medium capable of storing program code.

[0141] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0142] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0143] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0144] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "include," "contain," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, object, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, object, or apparatus.

[0145] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for finding objects, characterized in that, include: An image of an object located in a target space is acquired, and the object image is input into a spatial visual language model for three-dimensional spatial reasoning to obtain natural language text; wherein, the natural language text describes the scene and spatial location of the object in the object image; Structured data is extracted from the natural language text, and the object memory is recorded using the structured data and written into a memory bank with a relational database and a vector index library; Receive query text that queries a target object located in the target space, and extract structured semantic elements from the query text; The memory records of the target objects corresponding to the structured semantic elements are queried from the memory bank using the relational database and the vector index library, respectively. The target object's memory records are used to generate response text including scene and spatial location, and the response text is output.

2. The method according to claim 1, characterized in that, Retrieving the target object memory record corresponding to the structured semantic element from the memory bank using the relational database and the vector index library respectively, including: Transform the structured semantic elements into semantic vectors; The semantic vectors are used to perform an approximate nearest neighbor search in the vector index library to obtain candidate object memory records; Generate a structured query language statement corresponding to the structured semantic elements; Using the structured query language statements, a joint filtering process is performed in the relational database to obtain candidate object memory records; The memory record of the target object is determined by using the memory record of the candidate object.

3. The method according to claim 2, characterized in that, Determining the target object memory record using candidate object memory records includes: A multi-dimensional relevance assessment was performed on the candidate object memory records to obtain multiple relevance scores; Multiple relevance scores of the same candidate object's memory records are weighted and summed to obtain a comprehensive matching score; The target object memory record is determined from the candidate object memory records using the comprehensive matching score.

4. The method according to claim 1, characterized in that, Retrieving the target object memory record corresponding to the structured semantic element from the memory bank using the relational database and the vector index library respectively, including: The target object memory record is queried from the cached object memory record using the structured semantic elements; the cache stores hot data corresponding to the relational database and the vector index library, and evicts the least recently used data; If the object is not found in the cache, the object memory record that is found in the cache will be identified as the target object memory record. If the cache misses, the memory record of the target object corresponding to the structured semantic element is retrieved from the memory bank using the relational database and the vector index library, respectively.

5. The method according to claim 1, characterized in that, The target object's memory records are used to generate response text including scene and spatial location, including: Determine the confidence level of the memory record of the target object; Select a target template that matches the confidence level from the templates; The key fields from the memory record of the target object are filled into the placeholders in the target template to obtain the initial text; The initial text is polished to obtain the response text.

6. The method according to claim 1, characterized in that, The object memory is recorded using structured data and written into a memory library with a relational database and a vector index library, including: After assigning vector numbers to the object names corresponding to the structured data, they are added to the vector index library; Extract the complete fields corresponding to each individual object from the structured data to obtain object memory records, and write the object memory records into the relational database.

7. The method according to any one of claims 1 to 6, characterized in that, Writing object memory records into the relational database includes: Query the relational database to see if there exists an object memory record with the same name as the object memory record to be written; If so, compare the similarity between the memory records of objects with the same name to be written into the memory records; If the similarity is lower than a first threshold, the memory record of the object to be written is written into the relational database; If the similarity is higher than the second threshold, update the timestamp of the memory record of the same-named object in the relational database.

8. An object-finding system, characterized in that, include: The modal perception and spatial understanding central module is used to acquire images of objects located in the target space, and input the object images into a spatial visual language model for three-dimensional spatial reasoning to obtain natural language text; wherein, the natural language text describes the scene and spatial location of the objects in the object images; The hierarchical memory storage and adaptive update module is used to extract structured data from the natural language text, use the structured data to record object memories, and write them into a memory bank with a relational database and a vector index library; The intelligent retrieval and multi-dimensional matching engine module is used to receive query text that queries for target objects located in the target space, and extract structured semantic elements from the query text; and use the relational database and the vector index library to query the memory records of target objects corresponding to the structured semantic elements from the memory bank respectively. The Natural Language Generation and Integrated Response Module is used to generate response text including scene and spatial location using the memory records of the target object, and output the response text.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the object-finding method as described in any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the object finding method as described in any one of claims 1 to 7.

Images