Target search method and system based on robot center scene memory, robot, and storage medium

Abstract

The application relates to the technical field of robot perception and autonomous navigation. The application discloses a target searching method and system based on robot central scene memory, a robot and a storage medium, which can improve the searching efficiency and positioning reliability of a target object and improve user experience. The method comprises the following steps: acquiring a robot central scene memory map and query features of a target object; the robot central scene memory map is updated in real time with the movement of the robot; similarity calculation and processing are performed on each filled grid unit in the robot central scene memory map and the query features, so that a first similarity score corresponding to each filled grid unit is obtained; all the first similarity scores are filtered according to a preset rule, a plurality of target scores are obtained, and each filled grid unit corresponding to each target score is used as a candidate unit; and the positioning information of the target object is searched based on all the candidate units.

Description

Technical Field

[0001] This application relates to the field of robot perception and autonomous navigation technology. More specifically, this application relates to a target search method, system, robot, and storage medium based on robot central scene memory. Background Technology

[0002] Existing target search methods for robots generally consist of two stages. The first stage involves extracting the target object and other semantic regions during robot navigation or mapping using visual recognition, detection, and segmentation modules. This target object and other semantic regions are then fused to construct a queryable environment representation, which can be a 2D semantic raster map, a 3D semantic point cloud, a voxel map, an object-level memory structure, or a scene graph. The second stage involves matching and retrieving within this environment representation based on the target object's category during task execution, thereby obtaining the possible locations or candidate regions where the target object may exist.

[0003] More specifically, during task execution, the target category of the target object is typically matched against a pre-defined category vocabulary or prompt word set corresponding to the environment representation. When the target category does not match the category vocabulary or prompt word set (i.e., the target category does not exist in the category vocabulary or prompt word set), or when using synonyms, attribute combinations, long-tail categories, or other expressions of the target category to match the category vocabulary or prompt word set, this method is prone to problems such as no matching (or retrieval) results or positioning errors. That is, even if the target object has been observed during the mapping process, if its target category is not included in the category vocabulary or prompt word set, it is difficult to directly locate the target object from the environment representation (map or memory) during the task execution phase, resulting in search failure or the need for a large amount of blind exploration, reducing the search efficiency and user experience of the target object. Summary of the Invention

[0004] The purpose of this application is to provide a target search method, system, robot, and storage medium based on robot-centric scene memory, which can improve the search efficiency and positioning reliability of target objects, and enhance user experience. This application is mainly achieved through the following technical solutions: A first aspect of this application provides a target search method based on robot-centric scene memory, comprising: The robot's central scene memory map and the query features of the target object are obtained, and the robot's central scene memory map is updated in real time as the robot moves; A similarity calculation is performed on each filled grid cell in the robot's central scene memory map and the query feature to obtain a first similarity score corresponding to each filled grid cell; All first similarity scores are filtered according to preset rules to obtain multiple target scores, and the filled grid cells corresponding to each target score are all taken as candidate cells. The location information of the target object is obtained by searching all candidate units.

[0005] According to one embodiment of this application, the step of obtaining the robot's central scene memory map includes: Acquire the memory map to be updated, the robot's environmental observation data at the current moment, and its current pose information; The environmental observation data is processed to extract instance features to obtain a first target instance feature set; The memory map to be updated is updated based on the first target instance feature set and the current pose information to obtain the robot center scene memory map.

[0006] According to one embodiment of this application, the step of performing instance feature extraction processing on the environmental observation data to obtain a first target instance feature set includes: The environmental observation data is subjected to instance mask extraction processing to obtain the final instance mask set; The final instance mask set is deduplicated to obtain a deduplicated instance mask set; The deduplicated instance mask set is subjected to instance feature extraction processing to obtain the first target instance feature set.

[0007] According to one embodiment of this application, the query feature is text information or image information.

[0008] According to one embodiment of this application, the step of searching for the location information of the target object based on all candidate units includes: Sort the positions of all candidate units according to the access order to obtain the set to be visited; All locations in the set to be visited are accessed sequentially, and the locations in the set that meet the verification rules are used as the location information of the target object.

[0009] According to one embodiment of this application, the step of sequentially accessing all locations in the set to be accessed and using the locations in the set to be accessed that conform to the verification rules as the location information of the target object includes: Take the first position in the set to be visited as the current position; The following steps are performed iteratively to visit all positions in the set to be visited: Access the current location and obtain the current field of view image corresponding to the current location; Instance feature extraction processing is performed on the current field-of-view image corresponding to the current position to obtain the second target instance feature set corresponding to the current position; If a target mask that conforms to the verification rules exists in the second target instance feature set, the target mask that conforms to the verification rules in the second target instance feature set is converted into the positioning information of the target object, and the loop ends; If there is no target mask that meets the verification rules in the second target instance feature set, the current position is modified to the next position to continue accessing the set to be accessed until the set to be accessed is completely accessed.

[0010] According to one embodiment of this application, the step of converting the target mask that conforms to the verification rules in the second target instance feature set into the positioning information of the target object includes: The pixels of the target mask are back-projected into a three-dimensional point set using a depth map, and the three-dimensional point set is used as the positioning information of the target object.

[0011] A second aspect of this application provides a target search system based on robot-centric scene memory, comprising: The scene perception and memory construction subsystem is used to acquire the robot's central scene memory map and the query features of the target object. The robot's central scene memory map is updated in real time as the robot moves. The target retrieval and navigation verification subsystem is used to perform similarity calculation processing on each filled grid cell in the robot's central scene memory map and the query feature to obtain a first similarity score corresponding to each filled grid cell; to filter all first similarity scores according to preset rules to obtain multiple target scores, and to use the filled grid cells corresponding to each target score as candidate cells; and to search for the location information of the target object based on all candidate cells.

[0012] A third aspect of this application provides a robot, including a processor and a memory, the memory for storing a computer program, and the processor for calling and running the computer program stored in the memory to perform the steps of the target search method based on robot central scene memory provided in the first aspect of this application.

[0013] A fourth aspect of this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the steps of the target search method based on robot central scene memory provided in the first aspect of this application.

[0014] The beneficial effects of the embodiments of this application include: This application embodiment acquires a robot central scene memory map and query features of the target object. The robot central scene memory map is updated in real time as the robot moves. Similarity calculations are performed on each filled grid cell in the robot central scene memory map and the query features to obtain a first similarity score corresponding to each filled grid cell. All first similarity scores are filtered according to preset rules to obtain multiple target scores, and the filled grid cells corresponding to each target score are used as candidate cells. The location information of the target object is then searched based on all candidate cells. Compared with existing technologies that require preset category word lists or prompt word sets, this application embodiment uses a robot central scene memory map that is updated in real time as the robot moves to search for the location information of the target object, without relying on a fixed category word list or prompt word set. Therefore, this application embodiment can improve the search efficiency and location reliability of the target object, and enhance the user experience. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 The flowcharts for some embodiments of the target search method based on robot-centric scene memory of this application are shown below. Figure 2 The flowcharts for the target search method based on robot-centric scene memory in some other embodiments of this application are shown. Figure 3 The flowcharts for the target search method based on robot-centric scene memory in some further embodiments of this application are shown. Figure 4 This is a block diagram illustrating the principle of the target search system based on robot-centric scene memory in some embodiments of this application; Figure 5 This is a schematic diagram of the robot described in this application. Detailed Implementation

[0017] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0018] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0019] The terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0020] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or apparatus.

[0021] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items.

[0022] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0023] refer to Figure 1 The diagram shown is a flowchart of a target search method based on robot-centric scene memory, provided in the first aspect of an embodiment of this application. Figure 1 The target search method based on robot central scene memory includes the following steps S1, S2, S3 and S4.

[0024] S1. Obtain the robot's central scene memory map and the query features of the target object (see reference). Figure 2 In the "Text Language Query / Reference Image Query" step, the robot's central scene memory map is updated in real time as the robot moves.

[0025] Furthermore, the step of acquiring the robot's central scene memory map includes: acquiring the memory map to be updated, the robot's environmental observation data at the current moment, and its current pose information (see reference). Figure 2 The "robot pose" step in the process); instance feature extraction processing is performed on the environmental observation data to obtain the first target instance feature set (refer to...). Figure 2 The "object feature extraction" step in the text); based on the first target instance feature set and the current pose information, the memory map to be updated is updated to obtain the robot center scene memory map (refer to...). Figure 2 The "Robot Central Scene Memory Map Construction" step in the process.

[0026] The environmental observation data can be an RGB image. RGB refers to the three colors red, green, and blue. In other embodiments, the environmental observation data may also include other data, which can be specifically set by those skilled in the art according to actual needs.

[0027] The current pose information is used to characterize the position and attitude of the robot (or camera) in the environment. In this embodiment, the robot is equipped with a LiDAR and an IMU (Inertial Measurement Unit), and the FAST-LIO (Fast LiDAR-Inertial Odometry) algorithm is run to perform laser inertial odometry estimation to obtain continuous, real-time pose output (see reference). Figure 2 The "LiDAR, IMU" step in the text refers to this. The pose output is the current pose information. In other embodiments, the current pose information can also be obtained using laser SLAM (Simultaneous Localization and Mapping), visual SLAM, wheel speed odometer and IMU fusion, or multi-sensor fusion positioning.

[0028] Further, the step of performing instance feature extraction processing on the environmental observation data to obtain a first target instance feature set includes: performing instance mask extraction processing on the environmental observation data to obtain a final instance mask set; performing deduplication processing on the final instance mask set to obtain a deduplicated instance mask set; and performing instance feature extraction processing on the deduplicated instance mask set to obtain the first target instance feature set.

[0029] Further, the step of performing instance mask extraction processing on the environmental observation data to obtain the final instance mask set includes: using a class-independent target detector that does not require prompt words to perform candidate box segmentation processing on the environmental observation data to obtain a target candidate box set (see reference). Figure 3 "Class-independent object detection" steps or reference Figure 2 (In the "Class-Independent Object Detection and Instance Segmentation" step) a bounding box-guided segmenter is used to refine instance segmentation of all candidate boxes in the target candidate box set, obtaining a bounding box-driven instance mask set (refer to...). Figure 3 (In the "Detection Box-Guided Instance Segmentation" step), a full-image fast class-independent segmenter is used to extract instance masks from the environmental observation data to obtain a full-image instance mask set (refer to...). Figure 3 (In the "Full Image Instance Segmentation" step), the bounding box-driven instance mask set and the full image instance mask set are fused to obtain the final instance mask set (refer to...). Figure 3 The "mask merging" step in the middle.

[0030] To control redundancy and computational load, confidence filtering can be performed on all candidate boxes, and preliminary non-maximum suppression (NMS) can be performed on highly overlapping candidate boxes to reduce the number of invalid segmentation calls.

[0031] The cue-free class-independent object detector is a MAVL (Multi-scale Attention Vision Transformer). In other embodiments, the cue-free class-independent object detector can also be other detectors, which can be set by those skilled in the art according to actual needs.

[0032] The bounding box segmenter is EfficientViT-SAM (a segmentation model based on an efficient visual Transformer). EfficientViT-SAM is an accelerated version of the Segment Anything Model (SAM), whose core objective is to significantly improve image segmentation speed without sacrificing model performance. The Segment Anything Model (SAM) is a segmentation model for everything. EfficientViT-SAM is an efficient image segmentation model. In other embodiments, the bounding box segmenter can also be other segmenters, which can be specifically set by those skilled in the art according to actual needs.

[0033] The full-image fast class-independent segmenter is FastSAM (Fast Segment Anything Model). FastSAM is an image segmentation model based on convolutional neural networks. In other embodiments, the full-image fast class-independent segmenter can also be other segmenters, which can be set by those skilled in the art according to actual needs.

[0034] Furthermore, the step of "using a full-image fast class-independent segmenter to perform instance mask extraction processing on the environmental observation data to obtain a full-image instance mask set" can be used as a supplement to the missed areas in the step of "using a class-independent target detector without prompt words to perform candidate box segmentation processing on the environmental observation data to obtain a target candidate box set; using a box-guided segmenter to perform instance segmentation and refinement processing on all candidate boxes in the target candidate box set to obtain a box-driven instance mask set".

[0035] Furthermore, the calculation formula for the step of fusing the box-driven instance mask set and the full-image instance mask set to obtain the final instance mask set is as follows: ; in, It is the final set of instance masks; It is the set of mask instances of the box-driven instance; It is the set of full-image instance masks; It is a fusion function; The representation uses the box-driven instance mask in the box-driven instance mask set as the main component, and the full-image instance mask in the full-image instance mask set as the filler for the missing range.

[0036] Furthermore, the fusion process can be implemented in the following way: first retain All bounding instance masks in the set are used as the main set; then iterate through... Masks for all full-image instances in the image. Masks that are "non-repeating and can be expanded to cover" are added to the main set. In cases where there are local missing extensions, the extension of the full-image instance mask can be used first to fill in the bounding box driven instance mask, thereby improving the integrity of the instance region.

[0037] Further, the step of deduplicating the final instance mask set to obtain a deduplicated instance mask set includes: using an intersection-union suppression strategy to filter highly overlapping masks in the final instance mask set to obtain a filtered instance mask set; performing pixel dilation on each mask in the filtered instance mask set and checking the boundary overlap; deleting masks with boundary overlap in the filtered instance mask set to obtain the deduplicated instance mask set.

[0038] Regarding the handling of "boundary overlap", it can be replaced by deduplication based on mask similarity and contour distance, priority filtering based on confidence or area, or a learning-based candidate merging strategy.

[0039] Further, the step of performing instance feature extraction processing on the deduplicated instance mask set to obtain the first target instance feature set includes: based on all masks in the deduplicated instance mask set, cropping the environmental observation data to obtain the cropped region corresponding to the environmental observation data (refer to...). Figure 3 The "instance cropping" step involves using a visual encoder to perform language alignment processing on the cropped regions corresponding to the environmental observation data to obtain the first target instance feature set (this step can be understood as...). Figure 3 The "feature extraction" step in the process.

[0040] In other implementations, "cropping region" can be replaced by methods such as "cropping region + mask matting", "multi-scale cropping", or "local block aggregation".

[0041] Furthermore, the visual encoder is a CLIP (Contrastive Language–Image Pre-training) image encoder. In other embodiments, the visual encoder may be other encoders, which can be specifically set by those skilled in the art according to actual needs.

[0042] Furthermore, the calculation formula for the step of using a visual encoder to perform language alignment processing on the cropped regions corresponding to the environmental observation data to obtain the first target instance feature set is as follows: ; in, It is the first in the set of deduplication instance masks. One element in the feature set of the first target instance corresponding to each mask; It is the visual encoder (i.e., the CLIP image encoder). It is the cropped area corresponding to the environmental observation data; It is the first in the set of deduplication instance masks. A mask; yes A set of dimensional real vectors. It is the feature dimension.

[0043] Furthermore, the cropping region corresponding to the environmental observation data can be obtained by cropping from the environmental observation data according to the mask's outer region, and the mask shape can be selectively preserved (e.g., setting the background to zero or performing alpha masking) to reduce background interference. Alpha masking is an image processing technique that uses the image's alpha channel (transparency channel) to control which parts of the image are visible and which parts are transparent, thereby achieving a refined image compositing or masking effect.

[0044] Furthermore, the first target instance feature set may be image features aligned with the text encoder.

[0045] Furthermore, the step of updating the memory map to be updated based on the first target instance feature set and the current pose information to obtain the robot center scene memory map includes: discretizing the memory map to be updated into a set of grid cells; and mapping the first target instance feature set and the current pose information to the grid cells in the set of grid cells corresponding to the current pose information. This update process helps to keep the memory size under control and ensure timely updates, enhancing the memory map's responsiveness to scene changes.

[0046] In other implementations, a topological node memory structure indexed by keyframes or return points, a sparse trajectory keypoint memory structure indexed by odometer path segments, or a hierarchical memory structure can be used to replace the grid cell structure.

[0047] Furthermore, the step of discretizing the memory map to be updated into a set of grid cells includes: dividing the memory map to be updated into multiple grid cells at a fixed resolution, and forming the set of grid cells from the multiple grid cells.

[0048] The resolution can be adaptively adjusted (e.g., a smaller resolution when closer to the robot and a larger resolution when farther away from the robot) or dynamically adjusted according to regional importance.

[0049] The current pose information is written to the grid cell in the set of grid cells that it falls into.

[0050] When the robot revisits the same grid cell, it can use an overwrite method to write the current pose information into the grid cell.

[0051] Furthermore, to avoid frequent jitter writes, "pose change threshold, time threshold and / or view change threshold" can be set to determine whether to trigger the overwrite write mode.

[0052] Furthermore, the query feature can be text information or image information. In other embodiments, the query feature can also be a multi-turn dialogue description, attribute constraints (color, shape, or material, etc.), or a combination of queries.

[0053] Furthermore, when the query feature is text information, the step of obtaining the query feature includes: obtaining initial text; encoding the initial text using a CLIP text encoder to obtain the query feature (see reference). Figure 2 The "target query coding" step in the process.

[0054] Furthermore, the calculation formula for obtaining the query features by encoding the initial text using the CLIP text encoder is as follows: ; in, These are the query features; It is the CLIP text encoder; This is the initial text.

[0055] The initial text may contain paraphrases, phrases, or sentence descriptions.

[0056] Furthermore, when the query feature is image information, the step of obtaining the query feature includes: obtaining a reference image; and encoding the reference image using a CLIP image encoder to obtain the query feature.

[0057] Furthermore, the calculation formula for obtaining the query feature by encoding the reference image using a CLIP image encoder is as follows: ; in, This is the reference image.

[0058] The reference image can be a target local cropped image provided to the user.

[0059] The CLIP text encoder and the CLIP image encoder encode the initial text and the reference image into the same feature space.

[0060] Furthermore, before encoding, the initial text can be cleaned by removing uppercase letters (or lowercase letters), removing special punctuation marks, and normalizing whitespace. The reference image can be scaled and normalized to improve feature consistency.

[0061] S2. Perform similarity calculation on each filled grid cell in the robot's central scene memory map and the query feature to obtain the first similarity score corresponding to each filled grid cell.

[0062] The filled mesh cells are those that have been mapped by the corresponding first target instance feature set and pose information.

[0063] The similarity calculation process may involve calculating the inner product similarity between the most similar instance feature within each filled grid cell and the query feature.

[0064] S3. Filter all first similarity scores according to preset rules to obtain multiple target scores, and take the filled grid cells corresponding to each target score as candidate cells.

[0065] The preset rule is to select the highest first similarity score among all first similarity scores as the multiple target scores.

[0066] The combined calculation formula for steps S2 and S3 is: ; in, It is the first Multiple target scores corresponding to each filled grid cell; It is the first The first target instance feature set within a filled grid cell; yes One of the elements; It is an operation to find the maximum value; It is the inner product symbol.

[0067] Each candidate unit is a candidate's arrival location. Thus, this embodiment of the application has achieved coarse localization of the target object.

[0068] Furthermore, to improve the stability of retrieval (which can be understood as screening), the embodiments of this application can perform temperature scaling (or normalization) on the first similarity score of each filled grid cell, skip empty cells, and filter cells below the minimum confidence threshold, but do not change the core mechanism of "maximum similarity score within the cell + Top-K recall" (i.e., steps S2 and S3). Top-K refers to the total number of candidate cells.

[0069] It should also be understood that during the robot's movement, the robot's perception side continuously updates and revisits the grid cells, and recalculates the first similarity score, thereby enabling the ranking of all candidate cells to be refreshed online to adapt to changes in the dynamic scene. From an engineering perspective, this online refresh can employ periodic refresh or event-triggered refresh (e.g., refresh when an update occurs or refresh after a candidate cell access failure) to balance the real-time performance and computational consumption of this embodiment.

[0070] In other implementations, sliding window updates, exponential decay updates, fusion updates, or conditional updates based on change detection can be used instead of overlay updates.

[0071] S4. Based on all candidate units, search for the location information of the target object.

[0072] Steps S3 and S4 can reduce the time consumed by blind searching.

[0073] Further, step S4 includes: sorting the locations of all candidate units according to the access order to obtain a set to be accessed; sequentially accessing all locations in the set to be accessed, and using the locations in the set that meet the verification rules as the location information of the target object (see reference). Figure 2 The steps of "candidate scheduling and navigation execution" and "local region processing and target verification (fine search)" in the process.

[0074] In other implementations, the access order can also be implemented using a heuristic approach. Heuristic approaches include nearest-first access, score-first access, or distance-ratio-first access.

[0075] Furthermore, the formula for calculating the step of sorting the positions of all candidate units according to the access order to obtain the set to be visited is as follows: ; in, It is the set to be accessed; It is about finding the permutation that minimizes the sum of the elements inside the parentheses. ; It is a trade-off coefficient. ; It is a function for calculating planar distance; The robot is in the first Current pose information corresponding to the given moment; The robot is in the first Current pose information corresponding to the given moment; It is the first The first similarity score corresponding to the grid cell containing the current pose information at that time.

[0076] It is the current pose information of the robot at the current moment.

[0077] Further, the step of sequentially accessing all locations in the set to be accessed and using the locations in the set to be accessed that meet the verification rules as the location information of the target object includes the following steps S41, S42, S43, S44, S45 and S46.

[0078] S41. Take the first position in the set to be visited as the current position.

[0079] S42. Repeat steps S43, S44, S45 and S46 to sequentially access all positions in the set to be accessed.

[0080] S43. Access the current location and obtain the current field of view image and current text description corresponding to the current location.

[0081] The current text description may be a phrase or sentence describing the current location.

[0082] The access operation in this application embodiment can be implemented based on a general mobile robot navigation framework (such as ROSNavigation (i.e., Robot Operating System Navigation)), which includes a global path planning module, a local obstacle avoidance module, and a controller tracking module, etc.

[0083] In other implementations, "accessing the current location" can be replaced by accessing the boundary or viewpoint of the current location, or a fine search can be dynamically triggered during the approach to the current location to reduce the number of stops for the robot.

[0084] In other implementations, when accessing the current location, the neighborhood of the current location may also be accessed.

[0085] S44. Perform instance feature extraction processing on the current view image and current text description corresponding to the current position to obtain the second target instance feature set corresponding to the current position.

[0086] Further, step S44 includes: using an open-vocabulary object detector to perform bounding box segmentation on the current view image and current text description corresponding to the current position, obtaining multiple candidate boxes to be processed and the confidence score of each candidate box; if there are candidate boxes with confidence scores that meet a preset value, then using the box-guided segmenter to perform instance segmentation refinement on the candidate boxes with confidence scores that meet the preset value, obtaining a first set of candidate instance masks to be processed; if there are no candidate boxes with confidence scores that meet the preset value, then using the class-independent object detector without prompt words to perform candidate box segmentation on the current view image corresponding to the current position, obtaining candidate boxes to be processed. A set of bounding boxes is obtained; the bounding box guide segmenter is used to perform instance segmentation and refinement processing on all candidate boxes in the set of candidate bounding boxes to be processed, to obtain a second set of instance masks to be processed; the full-image fast class-independent segmenter is used to perform instance mask extraction processing on the current view image corresponding to the current position, to obtain a third set of instance masks to be processed; the first set of instance masks to be processed and the third set of instance masks to be processed are used as the second target instance feature set corresponding to the current position, or the second set of instance masks to be processed and the third set of instance masks to be processed are used as the second target instance feature set corresponding to the current position.

[0087] The open vocabulary object detector can be Grounding DINO (also known as a ground-based learning algorithm) or GLIP (Grounded Language-Image Pre-training). The goal of the ground-based learning algorithm is to detect objects specified by the input text from the image.

[0088] The preset value is a preset confidence threshold. The specific value of the preset value can be set by those skilled in the art according to actual needs.

[0089] S45. If there is a target mask that meets the verification rules in the second target instance feature set, convert the target mask that meets the verification rules in the second target instance feature set into the positioning information of the target object, and end the loop.

[0090] S46. If there is no target mask that meets the verification rules in the second target instance feature set, then the current position is modified to the next position of the current position to continue accessing the set to be accessed until the set to be accessed is completely accessed.

[0091] Further, the step of determining whether there is a target mask conforming to the verification rules in the second target instance feature set includes: cropping the current view image corresponding to the current position based on all masks in the second target instance feature set to obtain the region to be processed; performing language alignment processing on the region to be processed using the visual encoder to obtain the third target instance feature set; calculating the similarity between each element in the third target instance feature set and the query feature to obtain a second similarity score corresponding to each element in the third target instance feature set; taking the second similarity score with the largest value among all second similarity scores as the final score; if the final score meets the verification conditions, it is determined that there is a target mask conforming to the verification rules in the second target instance feature set; otherwise, it is determined that there is no target mask conforming to the verification rules in the second target instance feature set.

[0092] Furthermore, the "maximum value" calculation method can be replaced by average similarity, weighted maximum value, or by aggregating features within a unit before calculating similarity.

[0093] Furthermore, the calculation formula for the step of performing language alignment processing on the region to be processed using the visual encoder to obtain the third target instance feature set is as follows: ; in, It is the first in the feature set of the second target instance One element of the feature set of the third target instance corresponding to each mask; It is the area to be processed; It is the first in the feature set of the second target instance A mask.

[0094] Furthermore, the verification condition is: ;in, This is the final score; This is the preset score.

[0095] The preset score can be a fixed threshold or an adaptive threshold (adjusted according to environment, lighting, or query type). The specific value of the preset score can be set by those skilled in the art according to actual needs.

[0096] Furthermore, the step of converting the target mask conforming to the verification rules in the second target instance feature set into the positioning information of the target object includes: back-projecting the pixels of the target mask into a three-dimensional point set using a depth map, and using the three-dimensional point set as the positioning information of the target object. This step can improve the accuracy and feasibility of target positioning.

[0097] The depth map can be directly provided by an RGB-D camera.

[0098] Furthermore, the calculation formula for the step of back-projecting the pixels of the target mask into a three-dimensional point set using the depth map, and using the three-dimensional point set as the positioning information of the target object, is as follows: ; ; in, It is the location information of the target object; It is the first in the target mask The three-dimensional coordinates of each pixel; It is the first in the depth map 1 pixel; These are the intrinsic parameters of the camera in the robot. It is the first in the target mask 1 pixel coordinate; It is the first in the target mask 1 pixel coordinate; Represents transpose; It is the target mask.

[0099] In other embodiments, the positioning information of the target object may also be a three-dimensional bounding box, the target center point, or the target's pose in the map coordinate system; or it may be a more stable output obtained by filtering and clustering based on the point set.

[0100] In practical engineering implementation, the positioning information of the target object can be processed by simple outlier removal, voxel downsampling, or cluster center estimation to obtain a more stable 3D position representation of the target (e.g., output point set centroid or bounding box center), without changing the basic output form of "3D point set obtained by mask pixels + depth back projection".

[0101] Through the above implementation methods, this application embodiment uses a robot-centric scene memory map that is updated in real time as the robot moves to search for the location information of the target object. It does not rely on a fixed category word list or prompt word set. Therefore, this application embodiment can improve the search efficiency and location reliability of the target object and enhance the user experience.

[0102] In some implementations, after the steps of acquiring the memory map to be updated, the robot's environmental observation data at the current moment, and the current pose information, the step of acquiring the robot's central scene memory map further includes: synchronizing and aligning the environmental observation data and the current pose information according to timestamps to ensure that "current frame observation - current frame pose" correspond one-to-one, thereby being used for grid index writing, candidate position generation, and navigation distance cost calculation.

[0103] In some implementations, before the step of extracting instance features from the environmental observation data to obtain a first target instance feature set, the step of obtaining the robot's central scene memory map further includes: preprocessing the environmental observation data to obtain preprocessed environmental observation data. Then, in this embodiment, the preprocessed environmental observation data is subjected to instance feature extraction processing to obtain the first target instance feature set.

[0104] The preprocessing includes resolution scaling, distortion correction, exposure normalization (or white balance normalization), and timestamp alignment.

[0105] The preprocessing steps ensure the stability of segmentation and feature encoding.

[0106] In some implementations, the target search method based on robot-centric scene memory further includes binding the first target instance feature set and the current pose information to form a frame-level memory entry.

[0107] Furthermore, the calculation formula for the step of binding the first target instance feature set and the current pose information to form a frame-level memory entry is as follows: ; in, It is the frame-level memory entry; It is the feature set of the first target instance; This refers to the current pose information.

[0108] refer to Figure 4 The diagram shown is a schematic block diagram of a target search system based on robot-centric scene memory, provided in the second aspect of an embodiment of this application. Figure 4 The target search system 100 based on robot-centric scene memory includes: The scene perception and memory construction subsystem 101 is used to acquire the robot's central scene memory map and the query features of the target object. The robot's central scene memory map is updated in real time as the robot moves. The target retrieval and navigation verification subsystem 102 is used to perform similarity calculation processing on each filled grid cell in the robot's central scene memory map and the query feature to obtain a first similarity score corresponding to each filled grid cell; to filter all first similarity scores according to preset rules to obtain multiple target scores, and to use the filled grid cells corresponding to each target score as candidate cells; and to search for the location information of the target object based on all candidate cells.

[0109] In some implementations, the scene perception and memory construction subsystem 101 further includes a multi-source perception and pose acquisition module, used to acquire the memory map to be updated, the robot's environmental observation data at the current moment, and the current pose information.

[0110] In some embodiments, the scene perception and memory construction subsystem 101 further includes an instance segmentation and feature extraction module, which is used to perform candidate box segmentation processing on the environmental observation data using a class-independent target detector without prompting words to obtain a target candidate box set; to perform instance segmentation and refinement processing on all candidate boxes in the target candidate box set using a box-guided segmenter to obtain a box-driven instance mask set; to perform instance mask extraction processing on the environmental observation data using a full-image fast class-independent segmenter to obtain a full-image instance mask set; and to fuse the box-driven instance mask set and the full-image instance mask set to obtain the final instance mask set.

[0111] In some implementations, the scene perception and memory construction subsystem 101 further includes a scene memory map construction and update module, used to update the memory map to be updated based on the first target instance feature set and the current pose information, to obtain the robot's central scene memory map. The scene memory map construction and update module enables the memory map to reflect recent observations and adapt to dynamic environmental changes, thereby maintaining the timeliness and consistency of the memory content within a limited storage capacity.

[0112] In some embodiments, the target retrieval and navigation verification subsystem 102 further includes a target query encoding and memory retrieval module for obtaining initial text; and uses a CLIP text encoder to encode the initial text to obtain the query features.

[0113] The target query encoding and memory retrieval module is also used to obtain a reference image; the reference image is encoded using a CLIP image encoder to obtain the query features.

[0114] In some implementations, the target retrieval and navigation verification subsystem 102 further includes a candidate scheduling and navigation execution module, which sorts the access order of all candidate units to obtain a set to be accessed.

[0115] In some implementations, the target retrieval and navigation verification subsystem 102 further includes a local area processing and target verification module, which is used to sequentially access all locations in the set to be accessed and use the locations in the set to be accessed that meet the verification rules as the location information of the target object.

[0116] refer to Figure 5 The diagram shown is a schematic block diagram of a robot provided in the third aspect of an embodiment of this application. Figure 5 In the present invention, the robot 200 includes a processor 201 and a memory 202, the memory 202 being used to store computer programs, and the processor 201 being used to call and run the computer programs stored in the memory 202 to execute the steps of the target search method based on robot central scene memory provided in the first aspect of the present application.

[0117] Those skilled in the art will understand that Figure 5 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the robot to which the present invention is applied. A specific robot may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0118] A fourth aspect of this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the steps of the target search method based on robot central scene memory provided in the first aspect of this application.

[0119] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0120] The technical features of the above embodiments can be combined without changing the basic principles of this application. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the patent protection scope of this application should be determined by the appended claims.

Claims

1. A target search method based on robot-centric scene memory, characterized in that, include: The robot's central scene memory map and the query features of the target object are obtained, and the robot's central scene memory map is updated in real time as the robot moves; A similarity calculation is performed on each filled grid cell in the robot's central scene memory map and the query feature to obtain a first similarity score corresponding to each filled grid cell; All first similarity scores are filtered according to preset rules to obtain multiple target scores, and the filled grid cells corresponding to each target score are all taken as candidate cells. The location information of the target object is obtained by searching all candidate units; The steps for obtaining the robot's central scene memory map include: obtaining the memory map to be updated, the robot's environmental observation data at the current moment, and the current pose information; performing instance feature extraction processing on the environmental observation data to obtain a first target instance feature set; and updating the memory map to be updated based on the first target instance feature set and the current pose information to obtain the robot's central scene memory map.

2. The target search method based on robot-centric scene memory according to claim 1, characterized in that, The steps of performing instance feature extraction processing on the environmental observation data to obtain the first target instance feature set include: The environmental observation data is subjected to instance mask extraction processing to obtain the final instance mask set; The final instance mask set is deduplicated to obtain a deduplicated instance mask set; The deduplicated instance mask set is subjected to instance feature extraction processing to obtain the first target instance feature set.

3. The target search method based on robot-centric scene memory according to claim 1, characterized in that, The query features are text information or image information.

4. The target search method based on robot-centric scene memory according to claim 1, characterized in that, The steps for searching the location information of the target object based on all candidate units include: Sort the positions of all candidate units according to the access order to obtain the set to be visited; All locations in the set to be visited are accessed sequentially, and the locations in the set that meet the verification rules are used as the location information of the target object.

5. The target search method based on robot-centric scene memory according to claim 4, characterized in that, The step of sequentially accessing all locations in the set to be accessed and using the locations in the set that conform to the verification rules as the location information of the target object includes: Take the first position in the set to be visited as the current position; The following steps are performed iteratively to visit all positions in the set to be visited: Access the current location and obtain the current field of view image corresponding to the current location; Instance feature extraction processing is performed on the current field-of-view image corresponding to the current position to obtain the second target instance feature set corresponding to the current position; If a target mask that conforms to the verification rules exists in the second target instance feature set, the target mask that conforms to the verification rules in the second target instance feature set is converted into the positioning information of the target object, and the loop ends; If there is no target mask that meets the verification rules in the second target instance feature set, the current position is modified to the next position to continue accessing the set to be accessed until the set to be accessed is completely accessed.

6. The target search method based on robot-centric scene memory according to claim 5, characterized in that, The step of converting the target mask that conforms to the verification rules in the second target instance feature set into the positioning information of the target object includes: The pixels of the target mask are back-projected into a three-dimensional point set using a depth map, and the three-dimensional point set is used as the positioning information of the target object.

7. A target search system based on robot-centric scene memory, characterized in that, include: The scene perception and memory construction subsystem is used to acquire the robot's central scene memory map and the query features of the target object. The robot's central scene memory map is updated in real time as the robot moves. The target retrieval and navigation verification subsystem is used to perform similarity calculation on each filled grid cell in the robot's central scene memory map and the query feature to obtain the first similarity score corresponding to each filled grid cell. All first similarity scores are filtered according to preset rules to obtain multiple target scores, and the filled grid cells corresponding to each target score are all regarded as candidate cells; the location information of the target object is searched based on all candidate cells; The scene perception and memory construction subsystem is also used to acquire the memory map to be updated, the environmental observation data of the robot at the current moment, and the current pose information; and to perform instance feature extraction processing on the environmental observation data to obtain a first target instance feature set; The memory map to be updated is updated based on the first target instance feature set and the current pose information to obtain the robot center scene memory map.

8. A robot, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to perform the steps of the target search method based on robot central scene memory as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the steps of the target search method based on robot central scene memory as described in any one of claims 1 to 6.

Images