Construction method and system of robot memory bank

Abstract

The invention relates to the technical field of robots, and discloses a robot memory bank construction method and system.The method comprises the steps that scene graph information of a robot at all moments is determined based on image information and positioning information collected by all sensors of the robot; according to a segmentation mask in the object information and the three-dimensional point cloud data, dividing an object corresponding to the object information into large object information and small object information; based on a deduplication strategy and a deduplication queue corresponding to the scene information, performing deduplication on the scene information to obtain scene information after deduplication; and on the basis of the similarity between the existing scene information and the scene information after de-weighting in the memory bank and the similarity between the existing object information and the large object information and the small object information, adding the scene information after de-weighting, the large object information and the small object information into the memory bank. The memory bank can record the latest scene graph information in the working environment of the robot in real time.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method and system for constructing a robot memory bank. Background Technology

[0002] With the development of artificial intelligence and robotics, intelligent robots are gradually being applied to various fields such as home services, industrial automation, and medical care. When performing tasks across multiple scenarios, robots need to understand scene information and object information from multiple scenarios, and based on this information, generate cross-scenario tasks to complete tasks such as "bringing a glass of water from the living room to the bedroom" or "taking an apple from the kitchen to the living room dining table."

[0003] Therefore, the prerequisite for generating and executing tasks is that the database needs to store item information for each object in the current scene, as well as scene information to which each item belongs. This allows the robot to formulate and execute cross-scene tasks based on the aforementioned information. However, traditional databases are typically relational databases, storing structured text data, which is usually updated and maintained by manual and system-entered data. Robots, on the other hand, should be able to input multimodal information through their sensors and convert this multimodal information into hierarchical memory, thereby constructing the robot's memory bank. Therefore, a method for constructing the robot's memory bank is needed, enabling the robot to identify its current environment through its various sensors, such as sensors for localization and sensors for image acquisition, and store the identified scene and item information into the memory bank. How to scientifically construct this memory bank is the problem this application aims to solve. Summary of the Invention

[0004] In view of this, this application provides a method and system for constructing a robot memory bank.

[0005] This application discloses a method for constructing a robot memory bank, applied to a server, comprising: Step 1: Based on the image and positioning information collected by the robot's various sensors, determine the scene graph information of the robot at each time point; the scene graph information includes at least scene information and object information; the scene information includes at least the scene image of each frame and the pose information of the robot's camera; the object information includes at least the object's bounding box, segmentation mask, and position information; Step 2: Based on the segmentation mask in the object information and the 3D point cloud data, divide the object information into large object information and small object information; Step 3: Based on the deduplication strategy and deduplication queue corresponding to the scene information, deduplicate the scene information to obtain the deduplicated scene information; Step 4: Based on the similarity between the existing scene information in the memory bank and the deduplicated scene information, as well as the similarity between the existing object information and the large object information and the small object information, add the deduplicated scene information, large object information and small object information to the memory bank.

[0006] Further, step 1 includes: Using the various sensors mounted on the robot, the robot acquires scene images of the robot's working environment and the robot's positioning information when each frame is captured, at a fixed or set frame rate. Each frame of scene image is processed by a detection and segmentation model to obtain basic environmental perception information; the basic environmental perception information includes RGB image, depth map, object category, detection box, and segmentation mask; Based on the depth map and the positioning information of the robot when it captures each frame of image, the position information of each object in the depth map is determined; The scene image of each frame, the position information of each object in the depth map, and the basic environmental perception information are combined to form a structured scene map information.

[0007] Further, step 2 includes: If the number of pixels in the segmentation mask in the object information is less than the preset number of pixels, then the object information corresponding to the segmentation mask is determined to be small object information; If the number of pixels in the segmentation mask in the object information is greater than or equal to a preset number of pixels, and the volume of the 3D point cloud data of the segmentation mask is less than a first preset volume, then the object information corresponding to the segmentation mask is determined to be small object information; otherwise, the object information corresponding to the segmentation mask is determined to be large object information.

[0008] Further, based on the deduplication strategy and deduplication queue corresponding to the scene information, the scene information is deduplicated to obtain the deduplicated scene information, including: The scene information in the continuous frame scene graph is stored in the deduplication queue, and the scene information in the deduplication queue is sequentially input into the buffer. The scene information corresponding to multiple consecutive frames of scene images in the buffer is clustered using a preset scene similarity threshold to obtain multiple clusters; Calculate the intersection-union ratio (IUGR) between different scene images in each of the multiple clusters; The scene image corresponding to the maximum intersection-union ratio is selected from each cluster and used as the most representative scene image of each cluster to obtain the deduplicated scene information.

[0009] Furthermore, based on the similarity between the existing scene information in the memory and the deduplicated scene information, the deduplicated scene information is added to the memory, including: The most representative scene image of each cluster is matched with the scene information in the memory bank; if the match is successful, the corresponding scene image in the memory bank is updated using the most representative scene image of each cluster; otherwise, the most representative scene image of each cluster is added to the memory bank.

[0010] Furthermore, based on the similarity between existing object information and large object information in the memory bank, the large object information is added to the memory bank, including: Large object information in consecutive frame scene images is stored in a deduplication queue, and large object information in the deduplication queue is sequentially input into a buffer. Obtain the location information of the major objects to be removed from each scene graph in the buffer; Using the detection box of the major object information to be removed, the image of the major object to be removed is extracted from the image where the major object information is located. The image of the major object to be removed is then input into a multimodal coding model for encoding to obtain the semantic feature vector of the major object information to be removed. Based on the semantic feature vector of the information of the major object to be removed and the semantic feature vector of the object information in the memory bank, it is determined whether to add the information of the major object to be removed to the memory bank.

[0011] Further, based on the semantic feature vector of the information about the critical object to be removed and the semantic feature vector corresponding to the object information in the memory bank, determining whether to add the information about the critical object to be removed to the memory bank includes: Based on the similarity between the semantic feature vector of the major object information to be removed and the semantic feature vector of the major object information in the memory bank, major object information that matches the major object information to be removed is selected from the memory bank and used as candidate major object information; if major object information that matches the major object information to be removed cannot be selected from the memory bank, the major object information to be removed is added to the memory bank as new object information. Based on the first intersection-union ratio (IUU) between the 3D point cloud data of the major object information to be removed and the 3D point cloud data of the candidate major object information, a first target object information matching the major object information to be removed is selected from the candidate major object information; if a first target object information matching the major object information to be removed cannot be selected from the candidate major object information, the major object information to be removed is added to the memory bank as new object information. Based on the second intersection-union ratio (IUU) between the 3D point cloud data of the major object information to be removed and the 3D point cloud data of multiple images from different scene perspectives corresponding to the first target object information, the major object information to be removed is used to replace the first target object information whose second IUU ratio is greater than a preset IUU ratio.

[0012] Furthermore, based on the similarity between existing object information and small object information in the memory bank, the deduplicated small object information is added to the memory bank, including: If an object with the same unique ID as the small object information to be deduplicated exists in the memory, then the small object information to be deduplicated is used to replace the object with the same unique ID in the memory. Otherwise, based on the semantic feature vector of the small object information to be deduplicated, it is determined whether to add the small object information to be deduplicated as new object information to the memory.

[0013] Further, based on the semantic feature vector of the small object information to be deduplicated, determining whether to add the small object information to be deduplicated as new object information to the memory bank includes: Calculate the semantic similarity between the semantic feature vector of the small object information to be deduplicated and the semantic feature vector of the small object information in the memory bank; If the semantic similarity is greater than a preset value, then the small object information with the semantic similarity greater than the preset value is used as candidate object information; otherwise, the small object information to be deduplicated is regarded as new object information and added to the memory. Calculate the distance between the small object information to be deduplicated and each of the candidate object information, and filter out the candidate object information that is less than a preset distance from all the obtained distances and use it as the second target object information; For each second target object information, the semantic similarity and distance are normalized and weighted, and the second target object information with the highest score is selected. The smaller object information to be deduplicated is then used to replace the second target object information with the highest score.

[0014] This application also provides an embodiment of a robot memory bank construction system, implementing the robot memory bank construction method described in the above embodiments, applied to a server, and comprising: The scene graph information determination module is used to determine the scene graph information of the robot at each time point based on the image information and positioning information collected by the robot's various sensors. The scene graph information includes at least scene information and object information. The scene information includes at least the scene image of each frame and the pose information of the camera mounted on the robot. The object information includes at least the object's detection box, segmentation mask, and position information. The object segmentation module is used to segment the object information into large object information and small object information based on the segmentation mask in the object information and the three-dimensional point cloud data. The deduplication module is used to deduplicatize the scene information based on the deduplication strategy and deduplication queue corresponding to the scene information, and obtain the deduplicated scene information. The memory bank construction module is used to add deduplicated scene information, large object information, and small object information to the memory bank based on the similarity between existing scene information and deduplicated scene information, as well as the similarity between existing object information and large object information and small object information.

[0015] Due to the adoption of the above technical solution, this application has the following advantages: This application can store scene map information of the working scene acquired by the robot in real time through the memory bank, and the structure in the memory bank is clear, the information of each object is accurate and non-repetitive, which ultimately improves the stability, accuracy and real-time performance of the memory bank. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0017] Figure 1 This is a flowchart illustrating a method for constructing a robot memory bank according to an embodiment of this application; Figure 2 This is a schematic diagram illustrating another method for constructing a robot memory bank according to an embodiment of this application. Detailed Implementation

[0018] The present application will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of the present application.

[0019] See Figure 1 This application provides an embodiment of a method for constructing a robot memory bank, which includes: S1: Based on the image and positioning information collected by the robot's various sensors, determine the scene map information of the robot at each moment.

[0020] The scene graph information includes at least scene information and object information; the scene information includes at least the scene image of each frame and the pose information of the camera mounted on the robot; the object information includes at least the object's bounding box, segmentation mask, and position information.

[0021] Image information: acquired by a camera mounted on the robot (e.g., an RGB camera, a depth camera, a stereo camera, or a fisheye camera). The image information can be a series of consecutive image frames (i.e., a video stream) or a single image captured at a specific time interval. Preferably, the image information includes RGB images and / or depth images.

[0022] Location information: provided by the robot's positioning system. This positioning system may include, but is not limited to, one or more of an inertial measurement unit (IMU), a wheeled odometry system (WMPS), a lidar system, a GPS receiver, and a visual odometry module. The location information is used to determine the robot's pose in a global coordinate system and typically includes position coordinates and attitude angles.

[0023] Scene image: refers to the original image frame captured by the robot's camera at a certain time t, or the image after preprocessing (such as distortion correction and color enhancement). It serves as a direct visual record of the scene.

[0024] The pose information of the robot's camera refers to the camera's pose relative to a preset global coordinate system (e.g., the initial position coordinate system when the robot starts) at time t. This pose is typically represented by a six-degree-of-freedom transformation, including a three-dimensional translation vector and a three-dimensional rotation vector. The calculation method can be as follows: first, obtain the robot's pose through a positioning system; then, combine this with a pre-calibrated extrinsic parameter matrix between the camera and the robot, and obtain the camera's pose through coordinate transformation. The significance of obtaining the camera pose lies in providing a crucial coordinate system transformation basis for subsequent object localization and scene reconstruction.

[0025] Object information is used to describe detailed information about each object (or region of interest) identified in the environment.

[0026] Object detection bounding box: refers to the bounding box of the region where an object is located in the scene image, identified by an object detection algorithm (e.g., based on deep learning models such as YOLO and Faster R-CNN). The detection bounding box is usually represented by a rectangle and can be defined by the pixel coordinates (x1, y1, x2, y2) of its upper left and lower right corners, or by the coordinates of its center point and its width and height (cx, cy, w, h).

[0027] A segmentation mask is a finer, pixel-level segmentation of an object within a bounding box. The segmentation mask can be a binary image where foreground pixels (e.g., values ​​of 1) represent regions belonging to the object, and background pixels (e.g., values ​​of 0) represent regions not belonging to the object. Segmentation masks can be obtained using instance segmentation algorithms such as Mask R-CNN. Compared to bounding boxes, segmentation masks describe the outline shape of an object more accurately.

[0028] Object position information: refers to the object's position in three-dimensional space. Its calculation method can be as follows: combining the position of the object's bounding box or segmentation mask in the image, the camera's intrinsic parameter matrix, and the camera's pose information, the object's three-dimensional coordinates in the global coordinate system are estimated through geometric calculations (e.g., for a monocular camera, prior object dimensions or ground assumptions can be used; for a depth camera, the depth values ​​corresponding to pixels can be directly used). This position information can be represented as the three-dimensional coordinates (X, Y, Z) of the object's bounding box center point, or the coordinates of the point where the object's bottom contacts the ground.

[0029] Specifically, the robot collects raw environmental data using its various onboard sensors; the processor processes this raw data to extract image and positioning information; based on the extracted information, the scene graph information of the robot at various times is determined. This scene graph information is a structured environmental model used to comprehensively describe the static and dynamic elements of the robot's environment.

[0030] S2: Based on the segmentation mask in the object information and the 3D point cloud data, the object information is divided into large object information and small object information.

[0031] A segmentation mask is a computer vision technique used for pixel-level image recognition. It is typically represented by a two-dimensional matrix with the same resolution as the original scene image, where each pixel is assigned a specific label value. For an object identified by an object detection algorithm, its corresponding segmentation mask is used to accurately identify the pixel region occupied by that object in the image.

[0032] 3D point cloud data is a digital representation of the environment in three-dimensional space, consisting of a large set of discrete 3D points. Each 3D point contains its coordinate information in 3D space.

[0033] S3: Based on the deduplication strategy and deduplication queue corresponding to the scene information, the scene information is deduplicated to obtain the deduplicated scene information.

[0034] The deduplication strategy defines the criteria for determining whether new scene information is duplicated with existing information. The deduplication queue is a data structure (such as a first-in, first-out queue) used to store representative scene information that is determined to be non-duplicate under the deduplication strategy.

[0035] Specifically, during movement, especially at low speeds or when stationary, the robot continuously collects a large amount of highly similar scene information (images and poses). The differences between these consecutive frames are minimal; directly storing and processing all frames would result in a huge and unnecessary waste of storage space and computational resources (e.g., in subsequent mapping, localization, and loop closure detection modules). By employing a deduplication strategy, only keyframes where the scene undergoes significant changes (i.e., information in the deduplication queue) are retained. This reduces the data volume by an order of magnitude, significantly reducing storage pressure and freeing up valuable computational resources, enabling the system to run more smoothly on embedded robot platforms with limited computing power.

[0036] S4: Based on the similarity between the existing scene information in the memory bank and the deduplicated scene information, as well as the similarity between the existing object information and the large object information and the small object information, add the deduplicated scene information, large object information and small object information to the memory bank.

[0037] The memory is a structured database used by robots for long-term storage and organization of environmental knowledge. It is not a simple list of data, but a graphical or relational model containing various data types and their relationships. Similarity is a quantitative metric used to measure the degree of similarity or closeness between two data instances (scenes or objects). It is a numerical value, usually calculated using a specific similarity measurement algorithm. "Adding to memory" is a data writing operation that includes decision-making logic, not just simple storage. It includes two core operations: "Adding" and "Association Update." "Adding": When an information instance (scene or object) is determined to have no corresponding entry in the memory, it is added as a new node. "Association Update": When a new information instance is determined to match an existing instance in the memory, instead of adding a new node, an association is established between them, and the attributes of the existing instance (such as last appearance time and latest location) are updated.

[0038] The specific implementation process of S1 is as follows: S11: Utilize the various sensors mounted on the robot body to acquire scene images in the robot's working environment and the positioning information of the robot when taking each frame image at a fixed or set frame rate.

[0039] Each sensor includes at least the following two types: Image sensor: Used to acquire scene images. Specifically, it can be an RGB camera, a depth camera, an infrared camera, or a combination thereof. Scene images include not only traditional two-dimensional color (RGB) images, but also depth images, point cloud maps, or infrared thermal images. Positioning sensor: Used to acquire positioning information. Positioning information refers to the robot's pose in space. This pose typically includes its position (X, Y, Z coordinates) and attitude (such as roll, pitch, and yaw angles) in three-dimensional space.

[0040] Specifically, the sensor is triggered to acquire data at a constant frequency (e.g., 30 frames per second), generating uniform time-series data for subsequent processing. For example, the camera exposes at a frequency of 30Hz, while the positioning system (e.g., visual odometry) also outputs a pose estimate at a frequency of 30Hz. The acquisition frequency can be dynamically adjusted according to the robot's state or task requirements.

[0041] S12: Each frame of scene image is processed by the detection and segmentation model to obtain basic environmental perception information; the basic environmental perception information includes RGB image, depth map, object category, detection box, and segmentation mask.

[0042] The detection and segmentation model is a computer vision model based on a convolutional neural network architecture, preferably an instance segmentation model such as Mask R-CNN or YOLACT. This model achieves integrated output of object detection, classification, and segmentation in an end-to-end manner. Detection and segmentation models are typically pre-trained on large datasets, learning to recognize and segment various common objects from pixels. During inference, the model first extracts deep features from the image, then generates candidate regions through a region proposal network or anchor mechanism, and finally performs classification, bounding box regression, and segmentation mask prediction on these regions.

[0043] Basic environmental perception information is a standardized description of the current frame scene obtained after processing by detection and segmentation models. It serves as an intermediary bridge connecting low-level sensor data with high-level environmental understanding (such as scene graphs and memory).

[0044] RGB Image: The original color image that forms the basis of perception. Depth Map: A depth information map aligned with the pixels of the RGB image. It can be obtained directly from a depth camera or calculated using algorithms such as stereo vision. The depth map provides the distance of each pixel from the camera and is crucial for subsequent calculations of 3D information. Object Category: The classification label predicted by the model for each detected object, such as "person," "chair," "table," "television," etc. Detection Box: Represents the boundary position of each object in the image in coordinate form (e.g., [x_min, y_min, x_max, y_max], where x_min and x_max represent the minimum and maximum values ​​in the X-axis direction, respectively; y_min and y_max represent the minimum and maximum values ​​in the Y-axis direction, respectively), providing a coarse localization of the object. Segmentation Mask: Compared to detection boxes, it provides the precise shape and pixel-level localization of the object.

[0045] S13: Determine the position information of each object in the depth map based on the depth map and the positioning information of the robot when it takes each frame of image.

[0046] Specifically, using the object segmentation mask, the depth values ​​(i.e., the distance Z of each point from the camera) of all pixels belonging to that object are extracted from the depth map. Based on the camera's intrinsic parameters, the coordinates (u,v) of each pixel and its depth value Z are calculated through back projection to obtain the three-dimensional coordinates (X_c,Y_c,Z_c) of that point in the camera coordinate system.

[0047] Obtain the positioning information when the robot captured this frame of image. This is actually the pose of the camera coordinate system relative to a fixed world coordinate system (including the rotation matrix R and translation vector T).

[0048] Using this pose, the coordinates (X_c, Y_c, Z_c) of all object point clouds are transformed to obtain their coordinates (X_w, Y_w, Z_w) in the world coordinate system. Finally, a series of 3D point clouds of the objects in the world coordinate system are obtained.

[0049] Calculate the 3D bounding box of a series of 3D point clouds, and then use the 3D coordinates (X,Y,Z) of the center point or bottom center point of the bounding box as the final position information of the object.

[0050] S14: Combine each frame of scene image, the position information of each object in the depth map, and the basic environmental perception information to form a structured scene graph information.

[0051] The scene graph information is a structured data object that records a scene that the robot sees at a certain moment and from a certain pose. This scene contains several objects, each with its own identity, shape, and precise three-dimensional position.

[0052] The specific implementation process of S2 is as follows: If the number of pixels of the segmentation mask in the object information is less than the preset number, determine the object information corresponding to the segmentation mask as small object information; If the number of pixels of the segmentation mask in the object information is greater than or equal to the preset number, and the volume of the three-dimensional point cloud data of the segmentation mask is less than the first preset volume, determine the object information corresponding to the segmentation mask as small object information; otherwise, determine the object information corresponding to the segmentation mask as large object information.

[0053] In an example, object A (a cup far away): The number of pixels Pixel_C of its segmentation mask is 300.

[0054] Since 300 < T_p (assumed to be 600), directly determine that object A is a small object, and its corresponding information is small object information. The process ends, and there is no need to calculate the three-dimensional volume.

[0055] Object B (a chair nearby): The number of pixels Pixel_C of its segmentation mask is 15000, which is greater than T_p, then calculate its three-dimensional point cloud volume Volume_3D = 0.03 cubic meters.

[0056] Since 0.03 < T_volume_s (assumed to be 0.06), determine that object B is a small object, and its corresponding information is small object information.

[0057] Object C (a desk): The number of pixels Pixel_C of its segmentation mask is 20000, which is greater than T_p, then calculate its three-dimensional point cloud volume Volume_3D = 0.6 cubic meters.

[0058] Since 0.6 > T_volume_s (0.06), determine that object C is a large object, and its corresponding information is large object information.

[0059] This application adopts a two-stage classification strategy from coarse to fine: The first stage (based on the number of pixels): fast and low-cost, mainly filtering out objects that appear small due to distance. The second stage (based on three-dimensional volume): accurate and reliable, solving the misjudgment problem of "small objects nearby appear large in the image", and ensuring that the classification result strictly corresponds to the physical size of the object in the real world.

[0060] In the embodiment of this application, based on the duplicate removal strategy and duplicate removal queue corresponding to the scene information, the scene information is de-duplicated to obtain the de-duplicated scene information, including: Store the scene information in the continuous frame scene graph into the duplicate removal queue, and input the scene information in the duplicate removal queue into the buffer in turn; The scene information corresponding to multiple consecutive frames of scene images in the buffer is clustered using a preset scene similarity threshold to obtain multiple clusters; Calculate the intersection-over-union ratio (IoU) between different scene images in each of the multiple clusters; The scene image corresponding to the maximum intersection-union ratio is selected from each cluster and used as the most representative scene image of each cluster to obtain the deduplicated scene information.

[0061] Take a specific embodiment as an example: Suppose a robot is traversing a "room-corridor-room" environment, and the buffer processes 100 consecutive frames of images.

[0062] Clustering results: 100 frames were clustered into 3 clusters.

[0063] Cluster 1: Contains 40 frames, corresponding to the scenes in the first room.

[0064] Cluster 2: Contains 30 frames, corresponding to the scene of walking in the corridor.

[0065] Cluster 3: Contains 30 frames, corresponding to the screen after entering the second room.

[0066] Representative Frame Election: The system selects the 21st frame as the representative (possibly a frame when looking around in the center of the room) from the 40 frames in cluster 1 by calculating the Intersection over Union (IoU).

[0067] In cluster 2, the 16th frame was selected as the representative (possibly the center view of the corridor).

[0068] Frame 26 was selected as the representative from the 30 frames in cluster 3.

[0069] Final output: The original 100 frames of data were deduplicated into only 3 representative keyframes of scene information. These 3 frames basically summarize the core scene changes throughout the entire journey, and a large amount of redundant data was removed.

[0070] This application automatically selects the most representative key scene images with the greatest content differences from a continuous video stream (continuous frame scene map), thereby greatly reducing data redundancy and providing a high-quality data foundation for subsequent mapping, localization or scene memorization.

[0071] In this embodiment of the application, based on the similarity between the scene information already in the memory and the deduplicated scene information, the deduplicated scene information is added to the memory, including: The most representative scene image of each cluster is matched with the scene information in the memory bank. If the match is successful, the corresponding scene image in the memory bank is updated using the most representative scene image of each cluster. Otherwise, the most representative scene image of each cluster is added to the memory bank.

[0072] Specifically, high-level, robust visual features are extracted from the "most representative scene images" and "all scene images stored in the memory bank." The similarity between the feature vector of the new scene image and the feature vector of each old scene image in the memory bank is calculated (e.g., cosine distance or Euclidean distance). If the similarity between the new scene image and an old scene image in the memory bank exceeds a preset threshold, it is considered a "successful match," meaning the robot has recognized that it is currently in a scene it has previously visited, and the corresponding scene image in the memory bank needs to be updated using the most representative scene image of each cluster. Conversely, if the similarity is less than a threshold, it is considered a "failed match," indicating a completely new scene, and the most representative scene image of each cluster needs to be added to the memory bank.

[0073] In this embodiment of the application, based on the similarity between existing object information in the memory bank and large object information, the large object information is added to the memory bank, including: Store the information of large objects in the scene graph of consecutive frames into a deduplication queue, and then input the information of large objects in the deduplication queue into the buffer in sequence. Obtain the location information of the major objects to be removed from each scene graph in the buffer; Using the detection bounding box for removing information about major objects, the image of the major object to be removed is extracted from the image where the information about the major object is located. The image of the major object to be removed is then input into a multimodal coding model for encoding to obtain the semantic feature vector of the information about the major object to be removed. Based on the semantic feature vectors of the information on the major objects to be removed and the semantic feature vectors of the major object information in the memory bank, it is determined whether to add the information on the major objects to be removed to the memory bank.

[0074] Specifically, all large object information identified in consecutive frames (including its detection bounding box, segmentation mask, position, category, etc.) is temporarily stored in a "deduplication queue." This queue acts as a buffer and sorting mechanism, ensuring that objects are processed one by one to avoid congestion. Large object information in the deduplication queue is then sent in batches to a "buffer." The buffer can be considered a temporary workspace, where the system processes one or more objects at a time. From the information of a specific large object in the buffer, its detection bounding box coordinates and the original scene image in which it is located are retrieved.

[0075] Using a detection box as a boundary, an image patch containing only the object is precisely cropped from the original image, i.e., the image of the object to be removed.

[0076] The image of the object to be removed is input into a multimodal coding model (e.g., an image encoder of the CLIP model, or a specially trained visual encoder).

[0077] Multimodal coding models map object images into a high-dimensional vector space and output a fixed-length vector that can highly summarize the visual appearance and semantic information of the object, namely the semantic feature vector.

[0078] Calculate the similarity between the semantic feature vector of the current large object information to be removed and the semantic feature vectors of all large object information already in the memory (e.g., calculate cosine similarity).

[0079] Scenario A: Match successful (it's an old object) If the feature vector of the major object to be removed has a similarity to the feature vector of an existing major object in the memory bank that exceeds a preset threshold, then the major object to be removed will not be added to the memory bank.

[0080] Optionally, while no new records are added, an "association" operation is performed. For example, the "last observed time" and "latest location" of the object in memory are updated to enhance tracking of it.

[0081] Case B: Matching failed (it's a new object) Condition: If the similarity between the feature vector of the major object to be removed and the feature vectors of all existing major object information in the memory is lower than a preset threshold, then the major object information to be removed is added to the memory. The major object information to be removed (position, detection box, and newly generated semantic feature vector, etc.) is stored in the memory as a new object node.

[0082] In this embodiment of the application, determining whether to add the information of the major object to be removed to the memory bank based on the semantic feature vector of the information of the major object to be removed and the semantic feature vector of the major object information in the memory bank includes: Based on the similarity between the semantic feature vector of the major object information to be removed and the semantic feature vector of the major object information in the memory bank, the major object information that matches the major object information to be removed is selected from the memory bank and used as candidate major object information; if the major object information that matches the major object information to be removed cannot be selected from the memory bank, the major object information to be removed is added to the memory bank as new object information. Based on the first intersection-union ratio between the 3D point cloud data of the major object information to be removed and the 3D point cloud data of the candidate major object information, the first target object information that matches the major object information to be removed is selected from the candidate major object information; if the first target object information that matches the major object information to be removed cannot be selected from the candidate major object information, the major object information to be removed is added to the memory bank as a new object information. Based on the second intersection-union ratio (IUU) between the 3D point cloud data of the major object information to be removed and the 3D point cloud data of multiple different scene perspectives corresponding to the first target object information, the major object information to be removed is used to replace the first target object information whose second IUU ratio is greater than a preset IUU ratio.

[0083] In the above embodiments, the three-dimensional point cloud data of the object to be removed can be calculated using the following method: Using the location information, segmentation mask, depth map, and extrinsic parameters of the object to be removed, the pixel coordinates of the object to be removed on the image are converted into 3D point cloud data in the world coordinate system; the extrinsic parameters are the position and orientation of the camera in the world coordinate system.

[0084] Specifically, the cosine similarity or Euclidean distance between the information of the large object to be removed and the semantic feature vector of each object in the memory is calculated. All objects in the memory with similarity exceeding a preset threshold are selected to form a "candidate list", i.e., candidate large object information.

[0085] If none of the similarity scores exceed the preset threshold in the memory database, then the object information to be removed is determined to be a completely new object information, and it is directly added to the memory database, thus ending the process.

[0086] Calculate the intersection-over-union (IoU) ratio between the 3D point cloud of the large object to be removed and the 3D point cloud of each object in the candidate list, i.e., (intersection volume of the two bounding boxes) / (union volume of the two bounding boxes). Select the candidate large object with the highest IoU value as the first target object. If the IoU between the large object to be removed and all candidate large objects is low, it indicates that it may be a completely different new object with a similar appearance, and therefore it is added to the database as a new object.

[0087] Acquire multi-view point cloud: Retrieve 3D point cloud data corresponding to the information of the first target object from the memory bank, observed from multiple different historical perspectives.

[0088] Calculate the second Intersection over Union (IoU): Calculate the IoU between the current point cloud containing the information of the large, heavy object to be removed and each multi-view point cloud in the memory. Then, select the largest IoU value as the second IoU. This measures the degree of overlap between the current observation view and the historical best observation view. If the second IoU is greater than the preset IoU (the value of the IoU can be set according to the actual situation), it indicates that the currently observed object view is of higher quality and more complete. The information of the large, heavy object to be removed can be updated to the first target object information corresponding to the second IoU being greater than the preset IoU. At this time, the information of the large, heavy object to be removed is used to replace or merge and update the first target object information in the memory, so that the information of the large, heavy object to be removed is updated accordingly in the memory.

[0089] This application achieves self-optimization of the memory bank, ensuring that the memory bank always stores the clearest, most complete, and most representative observation data of the object, thereby making the robot's "memory" increasingly accurate.

[0090] In this embodiment of the application, based on the similarity between existing object information and small object information in the memory bank, the deduplicated small object information is added to the memory bank, including: If an object with the same unique ID as the small object information to be deduplicated exists in the memory, then the small object information to be deduplicated is used to replace the object with the same unique ID in the memory. Otherwise, based on the semantic feature vector of the small object information to be deduplicated, it is determined whether to add the small object information to be deduplicated as new object information to the memory.

[0091] Specifically, before a small object is first added to the memory, each small object is assigned a unique ID. A unique hash value can be generated as the ID using information such as the object's category and initial position coordinates.

[0092] Take a specific embodiment as an example: Scene: The robot sees a red cup on the table.

[0093] First time seeing (a completely new object): There is no matching ID in the memory, and semantic analysis also fails to find a similar object.

[0094] Result: The system assigns it a new ID (such as 001) and adds it to the memory bank as a new object.

[0095] The same object was seen a second time (tracking successful): The robot recognized the cup again and successfully associated it with its ID.

[0096] Result: The old information of 001 in the memory bank was directly replaced with the new location and information.

[0097] The third time it was seen (the same object, but re-identified after tracking was interrupted): Because someone completely covered the cup with their hand for one second, the robot's tracker lost track of the target. After the hand was removed, the robot detected the cup again, but generated a temporary new ID: temp_002.

[0098] temp_002 does not exist in the memory, proceed to semantic judgment.

[0099] Calculate the similarity between the features of temp_002 and the features of all small objects in the memory bank, and find that the similarity with 001 is as high as 0.97 (> the threshold of 0.9).

[0100] Result: Instead of adding temp_002, the position and timestamp of 001 were updated with the current information. This successfully prevented the same cup from being recorded twice.

[0101] In this embodiment of the application, determining whether to add the information of the small objects to be deduplicated as new object information to the memory bank based on the semantic feature vector of the small object information to be deduplicated includes: Calculate the semantic similarity between the semantic feature vector of the small object information to be deduplicated and the semantic feature vector of the small object information in the memory bank; If the semantic similarity is greater than the preset value, the small object information with the same semantic similarity as the preset value will be used as candidate object information; otherwise, the small object information to be deduplicated will be treated as new object information and added to the memory. Calculate the distance between the small object information to be deduplicated and each candidate object information, and filter out the candidate object information that is smaller than the preset distance from all the obtained distances and use it as the second target object information; For each second target object information, the semantic similarity and distance are normalized and weighted, and the second target object information with the highest score is selected. The smaller object information to be deduplicated is then used to replace the second target object information with the highest score.

[0102] Specifically, the semantic feature vectors of the small objects to be deduplicated are extracted, and the similarity between them and the semantic feature vectors of each small object in the memory is calculated. The similarity measure is usually cosine similarity (range [-1,1]) or the reciprocal of the normalized Euclidean distance (range [0,1]), resulting in a similarity list [similarity1, similarity2,..., similarityN], where N is the number of small objects in the memory.

[0103] The above similarity is compared with a preset semantic similarity threshold (e.g., T_s=0.83).

[0104] Case A (with candidates): Select all small object information with a similarity greater than T_s into the candidate object information list. This means that all known objects that visually "look similar" to the small object to be deduplicated have been identified.

[0105] Case B (No Candidates): If no object has a similarity greater than T_s, it indicates that this is a completely new object, and nothing resembling it can be found in the memory. In this case, the information of the smaller object to be deduplicated is directly added to the memory as the new object information, and the process ends.

[0106] For each candidate object information, calculate the Euclidean distance between the 3D position of the small object information to be deduplicated and the latest 3D position of the candidate object information recorded in the memory.

[0107] Compare all calculated Euclidean distances with a preset distance threshold (e.g., T_d = 2 meters).

[0108] From the candidate object information list, filter out those candidate object information whose spatial distance to the small object to be deduplicated is less than T_d, and form the second target object information list.

[0109] For each second target object, the two metrics are normalized and weighted for weighted summation: Normalized semantic score: Score_s = (semantic similarity of objects) / (sum of semantic similarity of all second target object information). The greater the semantic similarity, the higher the score.

[0110] Normalized distance score: Convert distance to "proximity": Proximity = 1 / (1 + distance). Then normalize: Score_d = (object's Proximity) / (sum of the Proximities of all secondary target objects). The closer the distance, the higher the score.

[0111] Weighted total score: Total_S=W1 Score_s+W2 Score_d. Where W1 and W2 are preset weights, and W1+W2=1.

[0112] Select the second target object information with the highest weighted total score, and replace the second target object information with the highest score in the memory bank with the small object information to be deduplicated.

[0113] This application also provides an embodiment of a robot memory bank construction system, implementing the robot memory bank construction method of the above embodiment, applied to a server, which includes: The scene graph information determination module is used to determine the scene graph information of the robot at each time point based on the image information and positioning information collected by the robot's various sensors. The scene graph information includes at least scene information and object information. The scene information includes at least the scene image of each frame and the pose information of the camera mounted on the robot. The object information includes at least the object's detection box, segmentation mask, and position information. The object segmentation module is used to divide object information into large object information and small object information based on the segmentation mask in the object information and the 3D point cloud data. The deduplication module is used to deduplicat scene information based on the deduplication strategy and deduplication queue corresponding to the scene information, and obtain the deduplicated scene information. The memory bank construction module is used to add deduplicated scene information, large object information, and small object information to the memory bank based on the similarity between existing scene information and deduplicated scene information, as well as the similarity between existing object information and large object information and small object information.

[0114] Those skilled in the art should clearly understand that, for the sake of convenience and brevity, the specific working process of the robot memory bank construction system described in the above embodiments can be referred to the corresponding process in the aforementioned robot memory bank construction method embodiments, and will not be repeated here.

[0115] For ease of understanding, this application provides the following more specific embodiments: like Figure 2 As shown, this application proposes a method for constructing a cross-scene memory bank for home service robots. The main application scenario is the multimodal perception and long-term storage of real-world scenes by robots in indoor environments. This memory bank, as the core data source during the task planning phase, continuously receives real-time scene information transmitted by the perception module and performs scene information classification and object deduplication before storage, thereby achieving structured and persistent management of various environmental perception information. Figure 2 The memory vector library mentioned here refers to the memory library mentioned below.

[0116] The overall process is as follows: S100, acquisition and encapsulation of perceived information.

[0117] The robot's perception module uses cameras mounted on the robot to acquire real-world scene images at a fixed or set frame rate. Each frame of the scene image is processed by a detection and segmentation model to obtain basic environmental perception information, including RGB images, depth maps, object categories, detection boxes, and segmentation masks. The system organizes the frame image, the position information of each object in the image, and the basic environmental information to form structured scene graph information, and sends it to the memory module.

[0118] Specifically, the robot body can acquire scene images at a fixed frame rate by patrolling and rotating its cameras. The cameras can directly acquire binocular RGB images, and through preprocessing of these images, corresponding depth maps can be obtained. Inputting the binocular RGB images into a detection and segmentation model allows for the labeling of detection boxes and item categories for each object in the image. Furthermore, the detection and segmentation model can also obtain the segmentation mask corresponding to the edges of each item within its detection box.

[0119] Packing the above data can serve as basic environmental perception information. Furthermore, the robot can acquire its own position information when each frame of an image is captured, and further determine the position information of each object in the depth map using the depth map and position information.

[0120] For each frame, the binocular RGB image, depth map, object category, detection box, segmentation mask, object location information, and the body location information in that frame can be used as scene graph information.

[0121] In one example, the process of obtaining the segmentation mask can optionally be as follows: 1. Input RGB image: The model receives a 3-channel (R, G, B) color image; 2. Feature extraction: The image is downsampled using a backbone (such as ResNet) to extract multi-level semantic features; 3. Object detection: Candidate boxes are generated using RPN (Region Proposal Network), and then the candidate boxes are classified and bounding box regressed to determine the location and category of the target; 4. Mask generation: For each detected target, the model's mask branch generates a small-sized mask feature map (such as 28×28), which is then upsampled to restore the size of the target bounding box, finally obtaining an instance mask aligned with the original image; 5. Output mask: The mask is usually a binary image (0 represents the background, 1 represents the target) or a multi-channel image (each channel corresponds to a category / instance).

[0122] Example of a scene graph information data structure: { "agent_room": "bobs_room", / / Custom scene, which can be determined based on the coordinate range of the electronic fence. "image_rgb_url": "url_rgb.jpg", / / RGB image "image_depth_url": "url_depth.png", / / Depth map "world2camera": [0.866, -0.5, 0, 2, 0.5, 0.866, 0, 3, 0, 0, 1, 5, 0,0, 0, 1], / / The pose information corresponding to the camera mounted on the robot body, 16 / 4 A 4-dimensional matrix vector, pose information includes position and orientation information, such as coordinates and angles.

[0123] "camera_intrinsics":[606.0801391601562,0.0, 318.43902587890625,0.0,606.3120727539062, 246.55946350097656, 0.0, 0.0, 1.0], / / Camera intrinsics, hardware parameters, generally not changed; "instance": [ / / example] { "object_id": "12", / / ID "name": "coffee_mug1", "room": "bobs_room", / / Environment "position":{"x":34.53,"y":23.82, "z":10.31}, / / Item position information "detection_2d":{"left_top":[45, 560],"right_bottom": [78, 780]}, / / The bounding box of the item in the RGB image. "segmentation": [[23,345], [234, 345]] / / Segmentation mask for this item; }, { "object_id": "15", "name": "coffee_mug2", "room": "bobs_room", "position":{"x":34.53,"y":23.82,"z":10.31}, "detection_2d":{"left_top":[45, 60],"right_bottom": [78, 780]}, "segmentation": [[27,145], [24, 325]]} ] S200, scene information classification and deduplication.

[0124] S201, the memory module categorizes the received scene graph information into three types based on information type: scene information, large object information, and small object information. The following is an explanation of these three types of information: 1. Scene Information: This refers to the continuous frames acquired by the robot's camera during scene scanning. Each frame includes the corresponding depth map, RGB image, robot camera pose in world coordinates, camera intrinsics, and the name of the room to which the frame belongs. (That is, all fields in the scene information graph except for instance are scene information).

[0125] 2. Large Object Information and Small Object Information: Both originate from the various objects under the "instance" field in the scene graph. The system combines the depth map and camera pose to calculate the actual size of the objects in the real world and classifies them as either large or small object information. Information corresponding to large objects is designated as large object information, information corresponding to small objects is designated as small object information, and information corresponding to objects is designated as object information.

[0126] S202. Since different types of information have different update and deduplication strategies, the system will call the corresponding deduplication process respectively, perform data filtering in parallel, and finally put the processing results into the database.

[0127] The following are the processes for three deduplication logics. These three deduplication logics are processed in parallel, with each performing deduplication in its own queue and buffer: A) Scene deduplication: 1) Information extraction: For each frame of original scene information, extract its "header information" (i.e., scene description fields other than the "instance" field), and put the continuous frame information into the scene information queue to wait for processing. When there is space in the buffer, the continuous frame information in the queue is input into the buffer in turn for deduplication.

[0128] 2) Buffer Deduplication: The scene information queue is continuously read by a dedicated sub-thread, and the scene data read is stored in the buffer. The system performs deduplication on the buffer at set time intervals, calculates the IOU (Intersection over Union) between scenes to determine the similarity of scene perspectives, selects the most representative scene perspectives to proceed to the next step, and then clears the buffer.

[0129] Specifically, "most representative" here means that since the received images are continuous frames, adjacent frames often have high correlation, which can easily lead to information redundancy. However, during the robot's movement, the scene perspective still differs at different time periods. Therefore, the system clusters continuous frames based on a set scene similarity threshold, and selects the scene perspective that best represents the characteristics over a period of time from each cluster as the "most representative" scene for subsequent processing.

[0130] For example, given 24 consecutive frames contained in a 1-second buffer, the server can cluster these 24 frames into three groups, resulting in three classes of consecutive frames. For each group, the server can calculate the Interchange of Units (IOU) between the consecutive frames and select the frame with the highest similarity as the most representative scene image. Based on this, the server can select three of the most representative scene images every second, thus obtaining multiple scene images. For each scene image, the corresponding scene perspective can be determined. The scene perspective can be calculated based on multiple scene images or directly obtained from the pose information of the robot when capturing the scene image.

[0131] In this embodiment, taking a 360-degree viewpoint as an example, only six scene images from different perspectives can be retained at 60-degree intervals. Based on this, for the same scene, the server ultimately selects only six scene images with 60-degree intervals from the buffer.

[0132] 3) Scene update: Perform IOU matching between the above representative scenes and existing scenes in the memory bank; if the match is successful, update the corresponding scene node; if no match is found, add a new scene record.

[0133] Specifically, after obtaining the six scene images from the current scene as described above, each of these six scene images is compared with the existing scene images in the memory database using Interchange of Union (IoU). If the IoU is greater than a preset threshold, the current scene is determined to match the scene in the memory database, and the scene image is updated to the memory database. If the IoU is less than the preset threshold, a mismatch is determined, and a new scene information entry is directly added to the memory database.

[0134] B) Deduplication of large object information: 1) Basic information extraction: For newly detected large object information, obtain its detection box, segmentation mask and the transformation relationship from pixel coordinates to world coordinates from the perception data, that is, obtain the position information of the large object.

[0135] 2) Semantic encoding: The object image is cropped based on the detection box and input into the GME multimodal coding model to generate the semantic feature vector (embedding) of the object information.

[0136] 3) Point cloud generation: Combining segmentation mask, depth map and extrinsic parameters, 2D pixel coordinates are converted into point cloud data in world coordinate system.

[0137] 4) Similarity and geometric matching: The semantic feature vector (embedding) is used to calculate the similarity with the existing large object information in the memory. Existing large object information with a similarity higher than a set threshold is used as candidate large objects; if the similarity is lower than the set threshold, the current object information is determined to be new object information and can be directly added to the memory.

[0138] For example, calculate the similarity between the embeddings of the current object information and the object information in the memory. When the similarity is greater than a set threshold, determine the object information in the memory as a candidate large object information.

[0139] The point cloud intersection-over-union (IoU) ratio between candidate large object information and the current object information is further calculated, and the object information with the largest IoU ratio and the IoU ratio exceeding a set threshold is selected as the matching target information. If no match is found, the current object information is treated as new object information and added to the memory.

[0140] 5) Multi-view update: For successfully matched large object information (which may contain multiple images from different perspectives), calculate the intersection-union ratio (IU) of the new object information and all its existing images on the point cloud; replace old images with IU exceeding the threshold with images of the new object to ensure that the representative multi-view information stored in the library can always accurately represent the large object.

[0141] C) Deduplication of small objects: 1) ID Matching and Update: Since the detection model tracks objects in continuous frames, each object is assigned a tracking ID, which is a unique ID for each small object within its local scene. Each time new scene image information is received, the system searches the database for all locally unique IDs of small objects. If the same locally unique ID is found, the old record is deleted and the new information is used to construct a node and add it to the database.

[0142] 2) New Object Detection: If a new object's ID is different from all IDs in the database, it is considered a new object. Both new and old objects are added to the memory database. Before being added, a multimodal coding model is used to extract its semantic feature vector.

[0143] 3) Similarity screening and weighted determination: Use the semantic feature vector to perform vector retrieval in the memory bank to screen out candidate objects with a similarity higher than the threshold; calculate the Euclidean distance for these candidate objects and screen out the target objects with a distance less than the set threshold. Then, perform normalized weighted scoring on the similarities and distances of multiple target objects, select the object with the highest score, and replace the current object with the object with the highest score. If there is no suitable match, directly store the current item as a new object in the library.

[0144] In one example, the process of distinguishing large object information and small object information is as follows: First, screen according to the two-dimensional area of the instance segmentation mask. If the number of mask pixels is less than the set threshold MIN_AREA, it is considered a small object; then generate three-dimensional point cloud data for the remaining objects, calculate the volume volume of the axis-aligned bounding box. If the volume is less than the threshold MIN_VOLUME, it is also classified as a small object, otherwise it is identified as a large object.

[0145] Example: In a scene, a table, a teacup, and a pen appear simultaneously: The mask area of the pen is very small, not exceeding MIN_AREA → judged as a small object; The mask area of the teacup is large enough, but the point cloud volume is only 9.6e5 mm³ < MIN_VOLUME → still judged as a small object; The mask area of the table is large and the point cloud volume is 5.4e8 mm³ ≥ MIN_VOLUME → judged as a large object.

[0146] In an exemplary embodiment, In actual situations, the scene graph data of the perception module sends data to the memory bank in units of frames. For each frame of image, the deduplication logic of three types of objects is listed below: In each frame of scene data, in the "instance" field, there are various examples detected by the model in the picture. For each individual object example, there is a certain data structure (see the example of the scene graph structure data). After the memory bank receives the scene graph data of this frame, it will first extract the scene information, that is, the field information other than the "instance" field constitutes the scene data. For the distinction between large and small objects in "instance", the segmentation mask is used to filter the point cloud, and a threshold is set to separate the large and small objects for processing.

[0147] The deduplication logic for small objects is as follows: 1. Each newly added object in the memory database will first have a local tracking ID provided by the detection and tracking model. This local tracking ID is the object's locally unique ID number. Each object will have a unique ID number in the continuous frame environment perception data (information). If the ID numbers are the same, the objects are directly considered to be the same, and the existing small object in the database will be directly replaced. 2. If the local tracking ID of the newly added object is different from every ID number in the memory, it is added as a new object node. Before being added to the memory, it is necessary to search the vector library for possible candidate matching objects with similarity higher than a set threshold. 3. If there are no candidate objects, the object is considered to be a truly new object that has not been recorded in the memory bank, and the object is directly added to the memory bank as a newly added object; 4. If there are candidate objects, calculate the distance to all candidate objects, sort the distance values, and select the object with the smallest distance value as the object to determine whether it is a possible duplicate of the newly added object. If the minimum distance value is less than a set threshold, the newly added object is considered to be the same object as the object with the corresponding minimum distance. If the minimum distance value is greater than the set threshold, the newly added object is considered to be an object that does not exist in the memory bank.

[0148] The logic for deduplicating large objects is as follows: 1. For each newly added large object, extract the bounding box, segmentation mask, and the transformation relationship from image pixel coordinates to world coordinates for each object in the environmental perception data; 2. Based on the detection box, extract the image of the corresponding object, and feed it into the multimodal coding model for encoding to obtain the corresponding semantic vector information; 3. Calculate the point cloud of the corresponding object in the world coordinate system by using the object segmentation mask and the transformation relationship from image pixel coordinates to world coordinates; 4. Utilize the semantic features of the object and calculate the similarity with existing large objects in the existing memory bank. Multiple objects with similarity greater than a set threshold are selected as candidates. Then, calculate the intersection-union ratio (IU) of their point clouds. Select the large object with the largest IU that exceeds the set threshold as the large object to match the newly added large object. If there is no match, the newly added large object is considered a new large object and is directly added to the database. 5. For the matched large object (each large object will have multiple images from different perspectives), calculate the intersection-union ratio (IU) of the new large object with all images inside the matched large object on the point cloud, replace all images with the new large object whose IU is greater than a set threshold with the previous images, and update to obtain new image information that can represent the large object.

[0149] In one example, scene information, large object information, and small object information are stored in the memory in the following format (similar to scene graph information): (1) The scene information only contains scene-related information: { "agent_room": "bobs_room", / / Custom scene, which can be determined based on the coordinate range of the electronic fence. "image_rgb_url": "url_rgb.jpg", / / RGB image "image_depth_url": "url_depth.png", / / Depth map "world2camera": [0.866, -0.5, 0, 2, 0.5, 0.866, 0, 3, 0, 0, 1, 5, 0,0, 0, 1], / / The pose information corresponding to the camera mounted on the robot body, 16 / 4 A 4-dimensional matrix vector, pose information includes position and orientation information, such as coordinates and angles.

[0150] "camera_intrinsics":[606.0801391601562,0.0, 318.43902587890625,0.0,606.3120727539062, 246.55946350097656, 0.0, 0.0, 1.0], / / Camera intrinsics, hardware parameters, generally not changed; } (2) Large object information consists of scene information + large object information. The data structure example is as follows: { "agent_room": "bobs_room", / / Custom scene, which can be determined based on the coordinate range of the electronic fence. "image_rgb_url": "url_rgb.jpg", / / RGB image "image_depth_url": "url_depth.png", / / Depth map "world2camera": [0.866, -0.5, 0, 2, 0.5, 0.866, 0, 3, 0, 0, 1, 5, 0,0, 0, 1], "instance": [ / / example] { "object_id": "12", / / ID "name": "bed_mug1", "room": "bobs_room", / / Environment "position":{"x":34.53,"y":23.82, "z":10.31}, / / Large object position information "detection_2d":{"left_top":[45, 560],"right_bottom": [78, 780]}, / / The bounding box of a large object in the RGB image. "segmentation": [[23,345], [234, 345]] / / Segmentation mask for the large item; } (3) Small object information is composed of scene information and small object information. The data structure example is as follows: { "agent_room": "bobs_room", / / Custom scene, which can be determined based on the coordinate range of the electronic fence. "image_rgb_url": "url_rgb.jpg", / / RGB image "image_depth_url": "url_depth.png", / / Depth map "world2camera": [0.866, -0.5, 0, 2, 0.5, 0.866, 0, 3, 0, 0, 1, 5, 0,0, 0, 1], "instance": [ / / example] { "object_id": "13", / / ID "name": "cup_mug1", "room": "bobs_room", / / Environment "position":{"x":34.53,"y":23.82, "z":10.31}, / / Large object position information "detection_2d":{"left_top":[45, 560],"right_bottom": [78, 780]}, / / The bounding box of a large object in the RGB image. "segmentation": [[23,345], [234, 345]] / / Segmentation mask for the large item; } The cross-scene memory bank and memory bank retrieval function constructed in this application serve as an important data source for robot cross-scene task planning, continuously receiving real-time scene data from the perception module during robot operation. The robot system performs parallel classification and deduplication processing on the received scene data and persistently stores the processing results. This memory bank aims to record the latest perspective images seen by the robot in different scenes, ensuring that the latest image information of objects in each target scene can be accurately obtained by retrieving the memory bank in the subsequent task planning stage. This supports the generation of task execution plans that are highly consistent with the real real-time environment and have greater feasibility.

[0151] This application utilizes a multimodal embedding model (simultaneously processing perceptual information such as images and text) to uniformly encode environmental perception results, and combines this with an object deduplication mechanism to achieve real-time memory updates across scenes. This application not only avoids repeatedly storing information about the same objects, but also allows for the sharing and reuse of existing object knowledge across different scenes. This provides a stable, complete, and scalable long-term memory foundation for intelligent agents in general indoor environments, significantly improving their contextual understanding and execution capabilities in complex task planning.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.

Claims

1. A method for constructing a robot memory bank, applied to a server, characterized in that, include: Step 1: Based on the image and positioning information collected by the robot's various sensors, determine the scene map information of the robot at each moment; The scene graph information includes at least scene information and object information; the scene information includes at least the scene image of each frame and the pose information of the robot's camera; the object information includes at least the object's bounding box, segmentation mask, and position information. Step 2: Based on the segmentation mask in the object information and the 3D point cloud data, divide the object information into large object information and small object information; Step 3: Based on the deduplication strategy and deduplication queue corresponding to the scene information, deduplicate the scene information to obtain the deduplicated scene information; Step 4: Based on the similarity between the existing scene information in the memory bank and the deduplicated scene information, as well as the similarity between the existing object information and the large object information and the small object information, add the deduplicated scene information, large object information and small object information to the memory bank.

2. The method for constructing a robot memory bank according to claim 1, characterized in that, Step 1 includes: Using the various sensors mounted on the robot, the robot acquires scene images of the robot's working environment and the robot's positioning information when each frame is captured, at a fixed or set frame rate. Each frame of scene image is processed by a detection and segmentation model to obtain basic environmental perception information; the basic environmental perception information includes RGB image, depth map, object category, detection box, and segmentation mask; Based on the depth map and the positioning information of the robot when it captures each frame of image, the position information of each object in the depth map is determined; The scene image of each frame, the position information of each object in the depth map, and the basic environmental perception information are combined to form a structured scene map information.

3. The method for constructing a robot memory bank according to claim 1, characterized in that, Step 2 includes: If the number of pixels in the segmentation mask in the object information is less than the preset number of pixels, then the object information corresponding to the segmentation mask is determined to be small object information; If the number of pixels in the segmentation mask in the object information is greater than or equal to a preset number of pixels, and the volume of the 3D point cloud data of the segmentation mask is less than a first preset volume, then the object information corresponding to the segmentation mask is determined to be small object information; otherwise, the object information corresponding to the segmentation mask is determined to be large object information.

4. The method for constructing a robot memory bank according to claim 1, characterized in that, Based on the deduplication strategy and deduplication queue corresponding to the scene information, the scene information is deduplicated to obtain the deduplicated scene information, including: The scene information in the continuous frame scene graph is stored in the deduplication queue, and the scene information in the deduplication queue is sequentially input into the buffer. The scene information corresponding to multiple consecutive frames of scene images in the buffer is clustered using a preset scene similarity threshold to obtain multiple clusters; Calculate the intersection-union ratio (IUGR) between different scene images in each of the multiple clusters; The scene image corresponding to the maximum intersection-union ratio is selected from each cluster and used as the most representative scene image of each cluster to obtain the deduplicated scene information.

5. The method for constructing a robot memory bank according to claim 4, characterized in that, Based on the similarity between existing scene information in the memory and the deduplicated scene information, the deduplicated scene information is added to the memory, including: The most representative scene image of each cluster is matched with the scene information in the memory bank; if the match is successful, the corresponding scene image in the memory bank is updated using the most representative scene image of each cluster; otherwise, the most representative scene image of each cluster is added to the memory bank.

6. The method for constructing a robot memory bank according to claim 1, characterized in that, Based on the similarity between existing object information and large object information in the memory bank, the large object information is added to the memory bank, including: Large object information in consecutive frame scene images is stored in a deduplication queue, and large object information in the deduplication queue is sequentially input into a buffer. Obtain the location information of the major objects to be removed from each scene graph in the buffer; Using the detection box of the major object information to be removed, the image of the major object to be removed is extracted from the image where the major object information is located. The image of the major object to be removed is then input into a multimodal coding model for encoding to obtain the semantic feature vector of the major object information to be removed. Based on the semantic feature vector of the information of the major object to be removed and the semantic feature vector of the object information in the memory bank, it is determined whether to add the information of the major object to be removed to the memory bank.

7. The method for constructing a robot memory bank according to claim 6, characterized in that, Based on the semantic feature vector of the critical object information to be removed and the semantic feature vector of the object information in the memory bank, determine whether to add the critical object information to the memory bank, including: Based on the similarity between the semantic feature vector of the major object information to be removed and the semantic feature vector of the major object information in the memory bank, major object information that matches the major object information to be removed is selected from the memory bank and used as candidate major object information; if major object information that matches the major object information to be removed cannot be selected from the memory bank, the major object information to be removed is added to the memory bank as new object information. Based on the first intersection-union ratio (IUU) between the 3D point cloud data of the major object information to be removed and the 3D point cloud data of the candidate major object information, a first target object information matching the major object information to be removed is selected from the candidate major object information; if a first target object information matching the major object information to be removed cannot be selected from the candidate major object information, the major object information to be removed is added to the memory bank as new object information. Based on the second intersection-union ratio (IUU) between the 3D point cloud data of the major object information to be removed and the 3D point cloud data of multiple images from different scene perspectives corresponding to the first target object information, the major object information to be removed is used to replace the first target object information whose second IUU ratio is greater than a preset IUU ratio.

8. The method for constructing a robot memory bank according to claim 6, characterized in that, Based on the similarity between existing object information and small object information in the memory, the deduplicated small object information is added to the memory, including: If an object with the same unique ID as the small object information to be deduplicated exists in the memory, then the small object information to be deduplicated is used to replace the object with the same unique ID in the memory. Otherwise, based on the semantic feature vector of the small object information to be deduplicated, it is determined whether to add the small object information to be deduplicated as new object information to the memory.

9. The method for constructing a robot memory bank according to claim 8, characterized in that, Based on the semantic feature vector of the small object information to be deduplicated, determine whether to add the small object information to be deduplicated as new object information to the memory bank, including: Calculate the semantic similarity between the semantic feature vector of the small object information to be deduplicated and the semantic feature vector of the small object information in the memory bank; If the semantic similarity is greater than a preset value, then the small object information with the semantic similarity greater than the preset value is used as candidate object information; otherwise, the small object information to be deduplicated is regarded as new object information and added to the memory. Calculate the distance between the small object information to be deduplicated and each of the candidate object information, and filter out the candidate object information that is less than a preset distance from all the obtained distances and use it as the second target object information; For each second target object information, the semantic similarity and distance are normalized and weighted, and the second target object information with the highest score is selected. The smaller object information to be deduplicated is then used to replace the second target object information with the highest score.

10. A system for constructing a robot memory bank, implementing the method for constructing a robot memory bank according to any one of claims 1-9, applied to a server, characterized in that, include: The scene map information determination module is used to determine the scene map information of the robot at each time point based on the image information and positioning information collected by the robot's various sensors. The scene graph information includes at least scene information and object information; the scene information includes at least the scene image of each frame and the pose information of the robot's camera; the object information includes at least the object's bounding box, segmentation mask, and position information. The object segmentation module is used to segment the object information into large object information and small object information based on the segmentation mask in the object information and the three-dimensional point cloud data. The deduplication module is used to deduplicatize the scene information based on the deduplication strategy and deduplication queue corresponding to the scene information, and obtain the deduplicated scene information. The memory bank construction module is used to add deduplicated scene information, large object information, and small object information to the memory bank based on the similarity between existing scene information and deduplicated scene information, as well as the similarity between existing object information and large object information and small object information.

Images