Robot target navigation method using multi-thread semantic matching and related device

Abstract

The application provides a robot target navigation method using multi-thread semantic matching and a related device, and relates to the field of object target navigation. In the method, an electronic device obtains an image similarity map and a category similarity map of an explored area according to an image within a current field of view of a robot; obtains exploration scores of a plurality of front points according to the image similarity map and the category similarity map; selects a best front point with the highest exploration score from the plurality of front points according to the exploration scores, and controls the robot to move to the best front point; if the best front point is not located at a position of a target navigation object, iterates the above steps until the robot moves to the position of the target navigation object or explores the entire preset area. In this way, on the basis of the image similarity map, the category similarity map is used to provide association information between a new target or an unknown environment and the target navigation object, thereby improving the adaptability to different environments.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of object target navigation, in particular, to a robot target navigation method using multi-cue semantic matching and related device. BACKGROUND

[0002] In the field of object target navigation, existing technical strategies mainly revolve around three categories: classical learning, end-to-end learning and modular learning. Each method has its unique characteristics and applicable scenarios.

[0003] The classical learning method focuses on the fusion of the positioning and map construction (SLAM) module and heuristic exploration strategies, such as using a frontier-based exploration method to gradually build an environmental map and use it as a basis to guide the object to the target object. This method relies on accurate map construction and effective exploration algorithms to ensure the robustness and accuracy of the navigation process.

[0004] The end-to-end learning method, which is centered on reinforcement learning (RL) or imitation learning (IL), directly learns the navigation strategy in the real environment by receiving RGBD images, semantic segmentation and object detection results as input. The advantage of this method is that it can adapt to complex environmental changes, and the learning process does not require manually labeled data, but rather self-optimization through interaction with the environment.

[0005] Modular learning, which divides the navigation task into perception, path planning and navigation sub-tasks, with the semantic mapping module that builds semantic maps being a key component. This module is responsible for understanding objects and their locations in the environment, while the semantic strategy module based on semantic maps determines the best navigation path based on object location information. The advantage of modular design is that it improves the scalability and flexibility of the system, allowing each part to be optimized and updated independently.

[0006] In practice, it is found that many methods based on modular learning rely on training for specific tasks, so they can only handle the object classes trained, making it difficult to extend to new targets or unknown environments, and thus have poor adaptability. SUMMARY

[0007] To overcome at least one of the deficiencies in the prior art, the present application provides a robot target navigation method using multi-cue semantic matching and related device, specifically including:

[0008] In a first aspect, the present application provides a robot target navigation method using multi-cue semantic matching, the method comprising:

[0009] According to images within a current field of view of the robot, obtain an image similarity map and a category similarity map of an explored area, wherein the image similarity map records image similarity between picture information in the images and navigation prompt texts, and the category similarity map records name similarity between a target navigation object in the navigation prompt texts and each object in the images;

[0010] According to the image similarity map and the category similarity map, obtain exploration scores of a plurality of frontier points, wherein each of the frontier points represents a candidate position to be explored by the robot, and the exploration score of each of the frontier points represents proximity to the target navigation object;

[0011] According to the exploration scores of the plurality of frontier points, select a best frontier point with a highest exploration score therefrom, and control the robot to move to the best frontier point;

[0012] If the best frontier point is not located at a position of the target navigation object, return to obtaining the image similarity map and the category similarity map of the explored area according to images within a current field of view of the robot until the robot moves to the position of the target navigation object or explores the entire preset area.

[0013] With reference to the optional implementation manner of the first aspect, the obtaining of the exploration scores of the plurality of frontier points according to the image similarity map and the category similarity map comprises:

[0014] determining the plurality of frontier points from the explored area;

[0015] for each of the frontier points, obtaining an image similarity score of the frontier point according to image similarity of pixels within a preset range from the frontier point in the image similarity map;

[0016] obtaining a category similarity score of the frontier point according to name similarity of pixels within a preset range from the frontier point in the category similarity map;

[0017] obtaining a distance score of the frontier point according to distance between the current position of the robot and the frontier point;

[0018] obtaining a density score of the frontier point according to distance between the frontier point and other frontier points;

[0019] weighting the image similarity score, the category similarity score, the distance score and the density score to obtain the exploration score of the frontier point.

[0020] According to the optional implementation of the first aspect, the image similarity score, the category similarity score, the distance score and the density score are weighted to obtain the exploration score of the front point, including:

[0021] Obtaining a background similarity between at least one background object in the image and the target navigation object, and a blank similarity between the navigation prompt text and a blank image;

[0022] According to the background similarity of each background object and the blank similarity, a weight strategy threshold is obtained;

[0023] If the plurality of front points include a semantic tendency node, the image similarity score, the category similarity score, the distance score and the density score are weighted by a first weighting strategy to obtain the exploration score of the front point, wherein the image similarity score or the category similarity score of the semantic tendency node is greater than the weight strategy threshold, and in the first weighting strategy, the weight of the image similarity score and the category similarity score is greater than the weight of the distance score and the density score;

[0024] If the plurality of front points do not include the semantic tendency node, the image similarity score, the category similarity score, the distance score and the density score are weighted by a second weighting strategy to obtain the exploration score of the front point, wherein in the second weighting strategy, the weight of the image similarity score and the category similarity score is less than the weight of the distance score and the density score.

[0025] According to the optional implementation of the first aspect, the image similarity score of the front point is obtained according to the image similarity of the pixels within a preset range from the front point in the image similarity map, including:

[0026] A plurality of pixels within a preset range from the front point in the image similarity map are taken as a plurality of first neighborhood pixels;

[0027] The mean value of the image similarity of the plurality of first neighborhood pixels is taken as the image similarity score of the front point.

[0028] According to the optional implementation of the first aspect, the category similarity score of the front point is obtained according to the name similarity of the pixels within a preset range from the front point in the category similarity map, including:

[0029] A plurality of pixels within a preset range from the front point in the category similarity map are taken as a plurality of second neighborhood pixels;

[0030] An average of the maximum name similarity of each of the plurality of second neighborhood pixels is taken as a category similarity score of the frontier point.

[0031] In combination with the optional implementation of the first aspect, the proven area includes an explored area of the robot and an unexplored area corresponding to the image, and the plurality of frontier points are determined from the proven area, including:

[0032] A plurality of boundaries between the explored area and the unexplored area are determined.

[0033] The plurality of frontier points are determined from the plurality of boundaries.

[0034] In combination with the optional implementation of the first aspect, the proven area includes an explored area and an unexplored area corresponding to the image, and the category similarity map of the proven area is obtained according to the image within the current field of view of the robot, including:

[0035] The image is segmented to determine an object to which each pixel in the image belongs.

[0036] Each pixel in the image is projected to a plane corresponding to the unexplored area according to depth information of the image to obtain a semantic sub-map of the unexplored area, wherein each pixel in the semantic sub-map corresponds to a spatial position and records an object located at the spatial position.

[0037] For each pixel in the semantic sub-map, a similarity between a name of each object located at a corresponding spatial position and a name of the target navigation object is calculated to obtain a category similarity sub-map of the unexplored area.

[0038] The category similarity sub-map of the unexplored area and a category similarity sub-map of the explored area are merged to obtain the category similarity map of the proven area.

[0039] In a second aspect, the application further provides a robot target navigation device using multi-cue semantic matching, including:

[0040] A similarity map module is configured to obtain an image similarity map and a category similarity map of a proven area according to an image within a current field of view of a robot, wherein the image similarity map records an image similarity between picture information in the image and navigation prompt text, and the category similarity map records a name similarity between a target navigation object in the navigation prompt text and each object in the image.

[0041] The exploration navigation module obtains exploration scores of a plurality of frontier points according to the image similarity map and the category similarity map, wherein each of the frontier points represents a candidate position to be explored by the robot, and the exploration score of each of the frontier points represents the closeness to the target navigation object;

[0042] The exploration navigation module is further configured to select a best frontier point with the highest exploration score from the plurality of frontier points according to the exploration scores of the plurality of frontier points, and control the robot to move to the best frontier point.

[0043] If the best frontier point is not located at the position of the target navigation object, the method returns to obtaining the image similarity map and the category similarity map of the explored area according to the images in the current field of view of the robot until the robot moves to the position of the target navigation object or explores the entire preset area.

[0044] With reference to the optional implementation of the second aspect, the exploration navigation module is further configured to:

[0045] determine the plurality of frontier points from the explored area;

[0046] for each of the frontier points, obtain an image similarity score of the frontier point according to the image similarity degrees of the pixels within a preset range from the frontier point in the image similarity map;

[0047] obtain a category similarity score of the frontier point according to the name similarity degrees of the pixels within a preset range from the frontier point in the category similarity map;

[0048] obtain a distance score of the frontier point according to the distance between the current position of the robot and the frontier point;

[0049] obtain a density score of the frontier point according to the distance between the frontier point and other frontier points;

[0050] weight the image similarity score, the category similarity score, the distance score and the density score to obtain the exploration score of the frontier point.

[0051] With reference to the optional implementation of the second aspect, the exploration navigation module is further configured to:

[0052] obtain a background similarity between at least one preset background object in the image and the target navigation object, and a blank similarity between the navigation prompt text and a blank image;

[0053] obtain a weight strategy threshold according to the background similarity of each of the background objects and the blank similarity;

[0054] If the semantic tendency node is included in the plurality of the frontier points, the image similarity score, the category similarity score, the distance score and the density score are weighted by a first weighting strategy to obtain an exploration score of the frontier point, wherein the image similarity score or the category similarity score of the semantic tendency node is greater than the weight strategy threshold, and in the first weighting strategy, the weight of the image similarity score and the category similarity score is greater than the weight of the distance score and the density score.

[0055] If the semantic tendency node is not included in the plurality of the frontier points, the image similarity score, the category similarity score, the distance score and the density score are weighted by a second weighting strategy to obtain an exploration score of the frontier point, wherein in the second weighting strategy, the weight of the image similarity score and the category similarity score is less than the weight of the distance score and the density score.

[0056] With reference to the optional implementation manner of the second aspect, the exploration navigation module is further specifically configured to:

[0057] a plurality of first neighborhood pixels in the image similarity map within a preset range from the frontier point are determined as the plurality of first neighborhood pixels;

[0058] an average value of the image similarity of the plurality of first neighborhood pixels is determined as the image similarity score of the frontier point.

[0059] With reference to the optional implementation manner of the second aspect, the exploration navigation module is further specifically configured to:

[0060] a plurality of second neighborhood pixels in the category similarity map within a preset range from the frontier point are determined as the plurality of second neighborhood pixels;

[0061] an average value of the maximum name similarity of each of the plurality of second neighborhood pixels is determined as the category similarity score of the frontier point.

[0062] With reference to the optional implementation manner of the second aspect, the explored area and the area to be explored corresponding to the image are included in the proven area, and the exploration navigation module is further specifically configured to:

[0063] a plurality of boundaries between the explored area and the area to be explored are determined;

[0064] the plurality of frontier points are determined from the plurality of boundaries.

[0065] With reference to the optional implementation manner of the second aspect, the explored area and the area to be explored corresponding to the image are included in the proven area, and the exploration navigation module is further specifically configured to:

[0066] segmenting the image to determine an object to which each pixel in the image belongs;

[0067] projecting each pixel in the image to a plane corresponding to the region to be explored according to depth information of the image to obtain a semantic sub-map of the region to be explored, wherein each pixel in the semantic sub-map corresponds to a spatial position and records an object located at the spatial position;

[0068] for each pixel in the semantic sub-map, calculating a similarity between a name of each object located at a corresponding spatial position and a name of the target navigation object to obtain a category similarity sub-map of the region to be explored;

[0069] merging the category similarity sub-map of the region to be explored and the category similarity sub-map of the region already explored to obtain a category similarity map of the region already explored.

[0070] In a third aspect, the present application also provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the robot target navigation method using multi-thread semantic matching.

[0071] In a fourth aspect, the present application also provides an electronic device, wherein the electronic device comprises a processor and a memory, the memory stores a computer program, and the computer program is executed by the processor to implement the robot target navigation method using multi-thread semantic matching.

[0072] Compared with the prior art, the present application has the following beneficial effects:

[0073] This application provides a robot target navigation method and related apparatus utilizing multi-clue semantic matching. In this method, an electronic device obtains an image similarity map and a category similarity map of the explored area based on images within the robot's current field of view. The image similarity map records the image similarity between the visual information in the image and the navigation prompt text, while the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image. Based on the image similarity map and the category similarity map, exploration scores for multiple leading edges are obtained. Each leading edge represents a candidate location to be explored by the robot, and the exploration score of each leading edge characterizes the proximity to the target navigation object. Based on the exploration scores of multiple leading edges, the optimal leading edge with the highest exploration score is selected, and the robot is controlled to move to the optimal leading edge. If the optimal leading edge is not located at the target navigation object's location, the process returns to obtaining the image similarity map and the category similarity map of the explored area based on images within the robot's current field of view, until the robot moves to the location of the target navigation object. Thus, based on the image similarity map, the category similarity map provides association information between new targets or unknown environments and the target navigation object, thereby improving adaptability to different environments. Attached Figure Description

[0074] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a schematic diagram of the method flow provided in the embodiments of this application;

[0076] Figure 2 This is a schematic diagram illustrating the principle of weight strategy selection provided in the embodiments of this application;

[0077] Figure 3 This is a schematic diagram of the structure of the virtual device provided in the embodiments of this application;

[0078] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0079] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application but not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0080] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0081] It should be noted that similar reference numerals and letters in the following drawings represent similar items, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0082] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the present application is usually placed, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0083] In addition, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0084] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the component must be absolutely horizontal or overhanging, but can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0085] In the description of the present application, it also needs to be explained that, unless otherwise explicitly specified and limited, the terms "set", "install", "connect", "connect" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0086] Based on the above statement, as introduced in the background art, the current semantic navigation method based on modular learning, many methods rely on training for specific tasks, therefore, only the trained object class can be processed, it is difficult to extend to new targets or unknown environment, and the adaptability is poor.

[0087] Based on the discovery of the above technical problems, the inventors propose the following technical solutions after creative labor to solve or improve the above problems. It should be noted that the defects of the above prior art solutions are the result of the inventors' careful research after practice, therefore, the discovery process of the above problems and the solutions proposed by the embodiments of the present application to solve the above problems should be the contribution of the inventors to the present application during the invention and creation process, and should not be understood as technical content known to those skilled in the art.

[0088] In view of this, the embodiment provides a robot target navigation method using multi-cue semantic matching. In the method, an electronic device obtains an image similarity map and a category similarity map of an explored area according to an image in a current field of view of a robot, wherein the image similarity map records an image similarity between picture information in the image and a navigation prompt text, and the category similarity map records a name similarity between a target navigation object in the navigation prompt text and each object in the image; according to the image similarity map and the category similarity map, an exploration score of a plurality of front points is obtained, wherein each front point represents a candidate position to be explored by the robot, and the exploration score of each front point represents an approaching degree to the target navigation object; according to the exploration scores of the plurality of front points, a best front point with the highest exploration score is selected from the plurality of front points, and the robot is controlled to move to the best front point; if the best front point is not located at a position of the target navigation object, the image similarity map and the category similarity map of the explored area are obtained according to the image in the current field of view of the robot, until the robot moves to the position of the target navigation object. In this way, on the basis of the image similarity map, the category similarity map is used to provide association information between a new target or an unknown environment and the target navigation object, thereby improving the adaptability to different environments.

[0089] It should be noted that the electronic device implementing the method can be a controller integrated with the robot body, and can also be a mobile terminal, a tablet computer, a laptop computer, a desktop computer, a server, etc. in communication connection with the controller of the robot. When it is a server, the server can be a single server, or a server group. The server group can be centralized or distributed (for example, the server can be a distributed system). In some embodiments, the server can be local or remote relative to the user terminal. In some embodiments, the server can be implemented on a cloud platform; by way of example only, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-cloud, a multi-cloud, etc., or any combination thereof. In some embodiments, the server can be implemented on an electronic device with one or more components.

[0090] To make the scheme provided by the present embodiment clearer, the following will take the controller of the robot as the electronic device implementing the method, and combine the method with the robot to describe the method in detail. Figure 1 The various steps of the method will be described in detail. However, it should be understood that the operations of the flowchart can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowchart or one or more operations can be removed from the flowchart under the guidance of the content of the present application. As shown in the flowchart, the method includes: Figure 1

[0091] S1, obtaining an image similarity map and a category similarity map of the proven area according to an image within the current field of view of the robot.

[0092] The image similarity map records the image similarity between the picture information in the image and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image.

[0093] In addition, it should be noted that the proven area includes an explored area and a to-be-explored area corresponding to the image. It can be understood that the explored area represents an area that has been explored by the robot, and the to-be-explored area represents a new area photographed by the camera of the robot, but the area has not been explored. The following optional implementation of step S1 is provided for obtaining the category similarity map:

[0094] S1-1, segmenting the image to determine the object to which each pixel in the image belongs.

[0095] ​At present, there are a large number of mature image segmentation methods, and technical personnel can choose adaptively according to the implementation scene when implementing the present solution. In the present embodiment, the controller of the robot obtains an image in the current field of view by shooting through the camera carried by the robot; then, the image is subjected to instance recognition and segmentation by using Semantic SAM to obtain an instance mask image with semantics, so as to determine the object to which each pixel in the image belongs.

[0096] It should be noted that the above-mentioned Semantic SAM combines semantic segmentation and general object segmentation, can automatically recognize and segment various object objects in the image, and at the same time assign a semantic label (such as "table", "chair") to each object, and is widely used in the fields of automatic driving, robot vision, etc. Considering the limited computing power of the controller of the robot, Mobile SAM with less demand for computing resources can also be used to segment the image, which can significantly improve the inference speed. It should be understood that on the resource-limited device such as a mobile robot, the lightweight architecture of Mobile SAM is more suitable for real-time processing tasks, while reducing the consumption of memory and computing resources, thereby improving the system performance without significantly reducing the segmentation accuracy.

[0097] S1-2, according to the depth information of the image, projecting each pixel in the image to a plane corresponding to the region to be explored to obtain a semantic submap of the region to be explored.

[0098] To this end, the present embodiment can generate a three-dimensional point cloud (x, y, z, cls_id) by combining the depth information of the image and the segmentation mask segmented by the SAM model, and the camera intrinsic parameter back projection, wherein (x, y, z) represents the spatial coordinates of the point cloud, and cls_id represents the class label of the object to which the point cloud belongs. In addition, the present embodiment introduces a confidence degree to cope with the noise problem in the measurement of the depth camera, which represents the possibility of the real existence of each point cloud in space. Therefore, the finally generated point cloud not only contains spatial position information, but also contains the class and confidence degree of the object. Finally, it is projected to the plane where the region to be explored is located to obtain a semantic submap of the region to be explored.

[0099] It should be understood that the point clouds in the same vertical line in the three-dimensional space will be projected to the same position on the two-dimensional plane. Therefore, in the present embodiment, each pixel in the semantic submap corresponds to a spatial position, and records the objects located at the spatial position. It can be understood that in the present embodiment, according to the types of objects in the environment where the robot is located, a corresponding number of channels are provided for each pixel in the semantic submap, each channel representing an object, and being used to record whether the object represented by the channel exists at the spatial position corresponding to the pixel. For example, for the three three-dimensional point clouds of (x, y, z1, cls_id1), (x, y, z2, cls_id3), and (x, y, z3, cls_id3), since they have the same (x, y), when projected onto the two-dimensional plane, they will all be projected to the position where (x, y) is located. However, in the actual scene, cls_id1 can be a "desk", cls_id2 can be a "computer" on the "desk", and cls_id3 can be a "light" located above the "computer", i.e., the three are in the same vertical line.

[0100] Based on the above introduction of the semantic submap, step S1 further includes:

[0101] S1-3, for each pixel in the semantic submap, calculating the similarity between the name of each object located at the corresponding spatial position and the name of the target navigation object to obtain a class similarity submap of the region to be explored.

[0102] It should be understood that there are currently a large number of mature methods that can be used to evaluate the semantic similarity between two names, and the skilled person can make adaptive choices according to the implementation scene when implementing the present solution. In the present embodiment, the types of objects in the environment where the robot is located come from the ADE20K dataset, which contains a total of 150 different semantic categories, such as people, vehicles, plants, buildings, etc. In the present embodiment, the names (i.e., class labels) of the object types in the ADE20K dataset and the name of the target navigation object are encoded by the Sentence Transformer model to convert them into high-dimensional semantic vectors; then, the semantic similarity between the name of each object and the name of the target navigation object is measured by calculating the cosine similarity. The closer the cosine similarity is to 1, the higher the semantic similarity, and vice versa, the closer to 0, which means no correlation. In this way, the semantic submap can be converted into a class similarity submap.

[0103] S1-4, merging the class similarity submap of the region to be explored and the class similarity submap of the region that has been explored to obtain a class similarity map of the region that has been explored.

[0104] In this embodiment, the class-similar sub-map of the region to be explored is merged with the class-similar sub-map of the explored region, and thus the class-similar map of the explored region is obtained, which has a dimension of H*W*class_num, that is, there are class_num channels for each pixel (H, W), and each channel records the similarity between the name of the corresponding object and the name of the target navigation object.

[0105] For the above image similarity map, there are also mature methods for evaluating the similarity between image content and description text. However, in related technologies, some methods require powerful computing resources, especially when relying on language models for semantic reasoning, which may require remote server support, limiting the autonomy of the robot. In addition, these technologies often need to convert visual information into text for processing, increasing the computational complexity and time cost, and reducing the efficiency.

[0106] In this embodiment, the controller captures an image within the current field of view through the robot camera, calculates the cosine similarity between the image and the navigation prompt text through the BLIP-2 model, and records it as BLIP2ITM Score. In this embodiment, it is referred to as a semantic score, which represents the matching degree of the image content in the image and the navigation prompt text. Then, the controller projects the semantic score to the plane where the image similarity sub-map of the explored region is located, and combines it with the depth information to exclude the area blocked by obstacles; finally, the semantic score of each pixel is weighted with the confidence in the corresponding three-dimensional point cloud to obtain an image similarity map of the explored region, which is used for subsequent navigation decision-making.

[0107] For example, assume that the navigation prompt text is "Seems like there is a [TV] ahead" (it seems that there is a TV ahead), where [TV] represents the target navigation object, that is, the robot needs to navigate to the location of [TV]. If the image within the current field of view of the robot happens to show that there is a TV in front, the semantic scores of the two are close to 1, otherwise, if there is no TV in front of the image and there is no object related to the TV, the semantic scores of the two are close to 0.

[0108] Based on the above introduction of the image similarity map and the class-similar map in the embodiment, continue to refer to Figure 1 The robot target navigation method using multi-cue semantic matching provided in this embodiment further comprises:

[0109] S2, obtaining an exploration score of a plurality of front points according to the image similarity map and the class-similar map.

[0110] Each frontier point represents a candidate position to be explored by the robot, and an exploration score of each frontier point represents a closeness to the target navigation object. In addition, the exploration score is a comprehensive result of multi-aspect scores. In this regard, the present embodiment further provides the following optional implementation of step S2:

[0111] S2-1, determining a plurality of frontier points from the explored area.

[0112] Since the explored area includes the explored area and the unexplored area of the robot, the controller of the robot can determine the boundaries between the explored area and the unexplored area; and determine the plurality of frontier points from the plurality of boundaries. For example, for each boundary, a midpoint or an endpoint of the boundary can be taken as a frontier point.

[0113] Based on the above introduction of the frontier point, step S2 further includes:

[0114] S2-2, for each frontier point, obtaining an image similarity score of the frontier point according to image similarities of pixels within a preset range from the frontier point in the image similarity map.

[0115] In this embodiment, the controller of the robot can take a plurality of pixels within a preset range from the frontier point in the image similarity map as a plurality of first neighborhood pixels; and take a mean value of image similarities of the plurality of first neighborhood pixels as the image similarity score of the frontier point. The preset range refers to a circular range with the frontier point as the center, and a radius of the circular range is a preset value.

[0116] Based on the above introduction of the image similarity score of the frontier point, step S2 further includes:

[0117] S2-3, obtaining a category similarity score of the frontier point according to name similarities of pixels within a preset range from the frontier point in the category similarity map.

[0118] In this embodiment, the controller of the robot can take a plurality of pixels within a preset range from the frontier point in the category similarity map as a plurality of second neighborhood pixels; and take a mean value of maximum name similarities of the plurality of second neighborhood pixels as the category similarity score of the frontier point.

[0119] To this end, it can be understood that, since the above embodiment has introduced the category similarity map of the proven area, the dimension thereof is H*W*class_num, that is, there are class_num channels for each pixel (H, W), and each channel records the similarity between the name of the corresponding object and the name of the target navigation object. Therefore, among the class_num channels of each second neighborhood pixel, there is a maximum name similarity, and the embodiment accumulates the maximum name similarity of each second neighborhood pixel, and divides the accumulated accumulated similarity by the number of the plurality of second neighborhood pixels, to obtain the category similarity score of the front point.

[0120] Based on the above embodiment introducing the front point image similarity score and the category similarity score from the semantic angle, the embodiment also evaluates the exploration value of the front point from the spatial distance angle. Therefore, step S2 further comprises:

[0121] S2-4, obtaining a distance score of the front point according to the distance between the current position of the robot and the front point.

[0122] Exemplarily, the distance between the front point and the current position of the robot is the Euclidean distance, and the expression for calculation is:

[0123]

[0124] In the formula, (x i ,y i ) represents the coordinates of the position of the i-th front point, (x r ,y r ) represents the coordinates of the position of the robot, and distance(i) represents the Euclidean distance between the i-th front point and the robot.

[0125] Considering that the above Euclidean distance is an absolute distance, which is not convenient for subsequent analysis, the Euclidean distance of other front points and the robot is also normalized, and the expression is as follows:

[0126]

[0127] In the formula, distance(i) represents the Euclidean distance between the i-th front point and the robot, min(distance) represents the minimum distance among the Euclidean distances from all front points to the robot, and max(distance) represents the maximum distance among the Euclidean distances from all front points to the robot.

[0128] Based on the above embodiment introducing the distance score, step S further comprises:

[0129] S2-5, obtaining a density score of the front point according to the distance between the front point and other front points.

[0130] To this end, the present embodiment provides the following calculation expression of the density score:

[0131]

[0132] In the expression, 1 (distance(i,j)≤radius) indicates that if the distance between the ith front point and the jth front point is less than or equal to the specified radius radius, it is counted as 1, distance(i,j) represents the Euclidean distance between the ith front point and the jth front point, and the expression is summed and then subtracted by 1 to subtract the effect of the ith front point itself.

[0133] Based on the image similarity score, the category similarity score, the distance score, and the density score obtained by the above implementation, step S2 further includes:

[0134] S2-6, weighting the image similarity score, the category similarity score, the distance score, and the density score to obtain an exploration score of the front point.

[0135] To this end, the present embodiment provides an adaptive weight strategy, that is, in the case of tending to follow semantic clues and tending to explore the scene, different weighting strategies are adopted respectively. In the optional implementation provided by the present embodiment, step S2-6 can include:

[0136] S2-6-1, obtaining a background similarity between at least one background object in the image and the target navigation object, and a blank similarity between the navigation prompt text and a blank image.

[0137] In the present embodiment, the at least one background object can include at least one of a wall, a ceiling, and a floor object. The blank similarity between the navigation prompt text and the blank image can be obtained by calculating through the BLIP-2 model.

[0138] S2-6-2, obtaining a weight strategy threshold value according to the background similarity and the blank similarity of each background object.

[0139] As an optional implementation, the mean value between the background similarity and the blank similarity of the at least one background object can be calculated as the weight strategy threshold value. Of course, as other optional implementations, the maximum value can also be selected from the background similarity and the blank similarity of the at least one background object as the weight strategy threshold value.

[0140] S2-6-3, judging whether the plurality of front points include a semantic tendency node.

[0141] If the image similarity score or the category similarity score of the semantic tendency node is greater than the weight strategy threshold, step S2-6-4 is performed, otherwise, step S2-6-5 is performed.

[0142] S2-6-4, the image similarity score, the category similarity score, the distance score and the density score are weighted by a first weight strategy to obtain an exploration score of the frontier point.

[0143] In the first weight strategy, the weight of the image similarity score and the category similarity score is greater than the weight of the distance score and the density score.

[0144] S2-6-5, the image similarity score, the category similarity score, the distance score and the density score are weighted by a second weight strategy to obtain an exploration score of the frontier point.

[0145] In the second weight strategy, the weight of the image similarity score and the category similarity score is less than the weight of the distance score and the density score.

[0146] For the above step S2-6, the entire step will be fully described below in combination with a specific example. As shown in FIG. Figure 2 On the left side of the figure, for the navigation prompt text "Seems like there is a [TV] ahead", the BLIP2 ITM Score between the text and the blank image is calculated by the BLIP-2 model first; and the semantic similarity between the target navigation object [TV] and the preset background objects such as "Wall", "Ceiling" and "Foor" is calculated; finally, the weight strategy threshold is obtained by averaging.

[0147] On the right side of the figure, for the image in the current field of view, the BLIP2 ITM Score between the image and the prompt text "Seems like there is a [TV] ahead" is calculated by the BLIP-2 model; and the BLIP2 ITM Score is converted into an image similarity map. The image similarity score of each frontier point can be obtained through the image similarity map.

[0148] For each pixel in the semantic map, the cosine similarity between the name of each object located at the corresponding spatial position and the name of the target navigation object is calculated, and then a category similarity map is obtained. The category similarity score of each frontier point can be obtained through the category similarity map.

[0149] Finally, the image similarity score of each frontier point is compared with a weight strategy threshold, and the category similarity score of each frontier point is compared with a weight strategy threshold, and if there is a result greater than the weight strategy threshold, the first weighting strategy tending to follow semantic clues is selected, otherwise the second weighting strategy tending to explore the scene is selected.

[0150] With reference to the above embodiment of the exploration score, Figure 2 , in the first weighting strategy, the weight of the image similarity score is 0.4, the weight of the category similarity score is 0.4, the weight of the distance score is 0.1, and the weight of the density score is 0.1. While in the second weighting strategy, the weight of the image similarity score is 0.1, the weight of the category similarity score is 0.1, the weight of the distance score is 0.4, and the weight of the density score is 0.4. In this way, the controller of the robot can decide which of the semantic information and the distance information in the current image is more trustworthy from the matching weight strategy.

[0151] Based on the above embodiment of the exploration score, with reference to the above embodiment of the best frontier point, Figure 1 , the method provided by the embodiment further comprises;

[0152] S3, selecting the best frontier point with the highest exploration score from the plurality of frontier points according to the exploration scores of the plurality of frontier points, and controlling the robot to move to the best frontier point.

[0153] For this purpose, in view of the fact that there are many mature path planning methods at present, the skilled person can make adaptive selection according to the implementation scene when implementing the present solution. In the present embodiment, the controller of the robot can use the PointNav strategy to guide the robot to navigate from a position to the best frontier point. It should be understood that the PointNav strategy relies on visual and odometry observations, so it does not require semantic understanding of the environment, but only needs to navigate according to geometric clues (depth image and distance and direction of relative target point). Moreover, the strategy is trained through deep reinforcement learning, using the Variable Experience Rollout (VER) algorithm, and after a large number of training in simulated environments, it can effectively navigate to the target point in different scenes.

[0154] With reference to the above embodiment of the best frontier point, Figure 1 , based on the above embodiment of the best frontier point, the robot target navigation method using multi-clue semantic matching provided by the embodiment further comprises:

[0155] S4, determining whether the best frontier point is located at the position of the target navigation object. If not, return to execute step S5, and if yes, execute S6.

[0156] S5, determining whether the entire preset area has been explored. If not, return to execute step S1, otherwise execute step S6.

[0157] S6, end the exploration task of the target navigation object.

[0158] It can be understood that the controller of the robot may find the target navigation object to complete the task under the prompting of the navigation prompt text. It can also be understood that the robot may explore the entire preset area and still not find the target navigation object, so that the task fails. The preset area is defined by the user in advance, for example, the entire room or the entire floor where the target navigation object is located.

[0159] Based on the same inventive concept as the robot target navigation method provided by the embodiment, the embodiment also provides a robot target navigation device using a similarity map. The device includes at least one software function module stored in the memory in the form of software or solidified in the electronic device. The processor in the electronic device is used to execute the executable module stored in the memory. For example, the device includes software function modules and computer programs. Please refer to Figure 3 Functionally, the device can include:

[0160] The similarity map module 11 is configured to obtain an image similarity map and a category similarity map of the explored area according to the image within the current visual range of the robot, wherein the image similarity map records the image similarity between the picture information in the image and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image.

[0161] The exploration navigation module 12 is configured to obtain an exploration score of a front point according to the image similarity map and the category similarity map, wherein the front point represents a candidate position to be explored by the robot, and the exploration score of the front point represents the proximity to the target navigation object.

[0162] The exploration navigation module 12 is further configured to select a best front point with the highest exploration score from the plurality of front points according to the exploration scores of the plurality of front points, and control the robot to move to the best front point.

[0163] If the best front point is not located at the position of the target navigation object, return to obtaining the image similarity map and the category similarity map of the explored area according to the image within the current visual range of the robot, until the robot moves to the position of the target navigation object or explores the entire preset area.

[0164] In the embodiment, the similarity map module 11 is configured to implement step S1 in Figure 1 The exploration navigation module 12 is configured to implement Figure 1The detailed description of the above modules can be referred to the detailed description of the corresponding steps. In addition, the robot target navigation device using multi-cue semantic matching can also implement other steps or sub-steps of the method through the above modules or other modules, and the embodiments will not be described in detail.

[0165] In addition, the functional modules in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0166] It should also be understood that the above embodiments, if implemented in the form of software functional modules and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application.

[0167] Therefore, the embodiments also provide a storage medium, which is a computer readable storage medium. The storage medium stores a computer program, and the computer program is executed by a processor to implement the robot target navigation method using multi-cue semantic matching provided by the embodiments. The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0168] The embodiments also provide an electronic device for implementing the robot target navigation method using multi-cue semantic matching. As shown in Figure 4 The electronic device can include a processor 22 and a memory 21. The memory 21 stores a computer program, and the processor implements the robot target navigation method using multi-cue semantic matching by reading and executing the computer program corresponding to the above embodiments in the memory 21.

[0169] Continuing to refer to Figure 4 The electronic device also includes a communication unit 23. The memory 21, the processor 22 and the communication unit 23 are directly or indirectly electrically connected to each other through a system bus 24 to realize data transmission or interaction.

[0170] The memory 21 can be any electronic, magnetic, optical, or other physical information record storage device that stores executable instructions, data, etc. In some embodiments, the memory 21 can be, but is not limited to, a volatile memory, a non-volatile memory, a storage drive, etc.

[0171] In some embodiments, the volatile memory can be a Random Access Memory (RAM); in some embodiments, the non-volatile memory can be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electric Erasable Programmable Read-Only Memory (EEPROM), a flash memory, etc.; in some embodiments, the storage drive can be a magnetic disk drive, a solid state drive, any type of storage disk (such as an optical disk, a DVD, etc.), or similar storage media, or a combination thereof, etc.

[0172] The communication unit 23 is configured to transceive data over a network. In some embodiments, the network can include a wired network, a wireless network, a fiber optic network, a telecommunications network, an intranet, the Internet, a Local Area Network (LAN), a Wide Area Network (WAN), a Wireless Local Area Network (WLAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a Public Switched Telephone Network (PSTN), a Bluetooth network, a ZigBee network, or a Near Field Communication (NFC) network, etc., or any combination thereof. In some embodiments, the network can include one or more network access points. For example, the network can include wired or wireless network access points, such as base stations and / or network switching nodes, through which one or more components of the service request processing system can connect to the network to exchange data and / or information.

[0173] The processor 22 can be an integrated circuit chip with signal processing capability and can include one or more processing cores (e.g., a single-core processor or a multi-core processor). By way of example only, the processor can include a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), an Application Specific Instruction-set Processor (ASIP), a Graphics Processing Unit (GPU), a Physics Processing Unit (PPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a microcontroller unit, a Reduced Instruction Set Computing (RISC), or a microprocessor, or any combination thereof.

[0174] It can be understood that Figure 4 The structure shown is merely schematic. The electronic device can also have more or fewer components than those shown, or have a different configuration from that shown. Figure 4 The components shown can be implemented in hardware, software, or a combination thereof. Figure 4 The components shown can be implemented in hardware, software, or a combination thereof. Figure 4 The components shown can be implemented in hardware, software, or a combination thereof.

[0175] It should be understood that all the devices and methods disclosed in the above embodiments can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the drawings show the possible implementation architectures, functions and operations of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders from those noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0176] The above describes only various embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A robot target navigation method utilizing multi-clue semantic matching, characterized in that, The method includes: Based on the images within the robot's current field of view, an image similarity map and a category similarity map of the explored area are obtained. The image similarity map records the image similarity between the image information and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image. Based on the image similarity map and the category similarity map, exploration scores for multiple frontier points are obtained, wherein each frontier point represents a candidate location to be explored by the robot, and the exploration score of each frontier point characterizes the proximity to the target navigation object; Based on the exploration scores of the multiple frontier points, the best frontier point with the highest exploration score is selected, and the robot is controlled to move to the best frontier point. If the optimal leading edge point is not located at the location of the target navigation object, the system returns to the image similarity map and category similarity map of the explored area based on the image within the robot's current field of view, until the robot moves to the location of the target navigation object or explores the entire preset area.

2. The robot target navigation method using multi-clue semantic matching according to claim 1, characterized in that, The step of obtaining exploration scores for multiple frontier points based on the image similarity map and the category similarity map includes: The plurality of frontier points were determined from the explored area; For each leading edge point, an image similarity score is obtained based on the image similarity of pixels within a preset range from the leading edge point in the image similarity map. The category similarity score of the leading edge point is obtained based on the name similarity of pixels within a preset range from the leading edge point in the category similarity map. The distance score of the leading edge point is obtained based on the distance between the robot's current position and the leading edge point; The density score of the leading edge point is obtained based on the distance between the leading edge point and other leading edge points; The image similarity score, the category similarity score, the distance score, and the density score are weighted together to obtain the exploration score of the frontier point.

3. The robot target navigation method using multi-clue semantic matching according to claim 2, characterized in that, The step of weighting the image similarity score, the category similarity score, the distance score, and the density score to obtain the exploration score of the frontier point includes: Obtain the background similarity between at least one preset background object in the image and the target navigation object, and the blankness similarity between the navigation prompt text and the blank image; The weighting strategy threshold is obtained based on the background similarity and the blank similarity of each background object; If a semantically inclined node is included among the multiple frontier points, the image similarity score, the category similarity score, the distance score, and the density score are weighted using a first weighting strategy to obtain the exploration score of the frontier point. The image similarity score or the category similarity score of the semantically inclined node is greater than the weighting strategy threshold. In the first weighting strategy, the weights assigned to the image similarity score and the category similarity score are greater than the weights assigned to the distance score and the density score. If none of the aforementioned semantic tendency nodes are included among the multiple frontier points, the image similarity score, the category similarity score, the distance score, and the density score are weighted using a second weighting strategy to obtain the exploration score of the frontier point. In the second weighting strategy, the weights assigned to the image similarity score and the category similarity score are less than the weights assigned to the distance score and the density score.

4. The robot target navigation method using multi-clue semantic matching according to claim 2, characterized in that, The step of obtaining the image similarity score of the leading edge point based on the image similarity of pixels within a preset range from the leading edge point in the image similarity map includes: Multiple pixels in the image similarity map that are within a preset range from the leading edge point are taken as multiple first neighboring pixels; The average image similarity of the plurality of first neighboring pixels is used as the image similarity score of the leading edge point.

5. The robot target navigation method using multi-clue semantic matching according to claim 2, characterized in that, The step of obtaining the category similarity score of the leading edge point based on the name similarity of pixels within a preset range from the leading edge point in the category similarity map includes: Multiple pixels in the category-similar map that are within a preset range from the leading edge point are used as multiple second neighbor pixels; The average of the maximum name similarity of each of the multiple second neighboring pixels is used as the category similarity score of the front edge point.

6. The robot target navigation method using multi-clue semantic matching according to claim 2, characterized in that, The explored area includes the area already explored by the robot and the area to be explored corresponding to the image. Determining the plurality of frontier points from the explored area includes: Multiple boundaries between the explored region and the region to be explored are determined; The plurality of frontier points are determined from the plurality of boundaries.

7. The robot target navigation method using multi-clue semantic matching according to claim 1, characterized in that, The explored area includes the already explored area and the area to be explored corresponding to the image. Based on the image within the robot's current field of view, a category similarity map of the explored area is obtained, including: The image is segmented to determine the object to which each pixel in the image belongs; Based on the depth information of the image, each pixel in the image is projected onto a plane corresponding to the area to be explored to obtain a semantic sub-map of the area to be explored. Each pixel in the semantic sub-map corresponds to a spatial location and records the object located at the spatial location. For each pixel in the semantic sub-map, the similarity between the name of each object located at the corresponding spatial location and the name of the target navigation object is calculated to obtain the category similarity sub-map of the area to be explored. The category similarity submap of the area to be explored is merged with the category similarity submap of the explored area to obtain the category similarity map of the explored area.

8. A robot target navigation device utilizing multi-cue semantic matching, characterized in that, The device includes: The similarity map module is used to obtain an image similarity map and a category similarity map of the explored area based on the image within the robot's current field of view. The image similarity map records the image similarity between the image information and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image. The exploration navigation module obtains exploration scores for multiple leading edge points based on the image similarity map and the category similarity map. Each leading edge point represents a candidate position to be explored by the robot, and the exploration score of each leading edge point represents the degree of proximity to the target navigation object. The exploration and navigation module is also used to select the best front point with the highest exploration score from among the multiple front point exploration scores, and control the robot to move to the best front point; If the optimal leading edge point is not located at the location of the target navigation object, the system returns to the image similarity map and category similarity map of the explored area based on the image within the robot's current field of view, until the robot moves to the location of the target navigation object or explores the entire preset area.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the robot target navigation method using multi-clue semantic matching as described in any one of claims 1-7.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing a computer program, which, when executed by the processor, implements the robot target navigation method using multi-clue semantic matching as described in any one of claims 1-7.

Images