Robot target navigation method using multi-clue semantic matching and related device

Abstract

The invention provides a robot target navigation method utilizing multi-clue semantic matching and a related device, and relates to the field of object target navigation. According to the method, an electronic device obtains an image similar map and a category similar map of an ascertained area according to an image in a current visual field range of a robot; according to the image similar map and the category similar map, obtaining exploration scores of the plurality of leading edge points; according to the exploration scores of the plurality of front edge points, selecting an optimal front edge point with the highest exploration score from the plurality of front edge points, and controlling the robot to move to the optimal front edge point; and if the optimal leading edge point is not located at the position where the target navigation object is located, iterating the steps until the robot moves to the position where the target navigation object is located or explores the whole preset area. Therefore, on the basis of the image similar map, the associated information between the new target or the unknown environment and the target navigation object is provided through the category similar map, so that the adaptability to different environments is improved.

Description

Technical Field

[0001] The present application relates to the field of object target navigation, and in particular, to a robot target navigation method and related devices using multi-cue semantic matching. Background Art

[0002] In the field of object target navigation, existing technical strategies mainly revolve around three categories: classical learning, end-to-end learning, and modular learning. Each method has its own unique characteristics and applicable scenarios.

[0003] Classical learning methods focus on integrating localization and mapping (SLAM) modules and heuristic exploration strategies, such as using frontier-based exploration methods to gradually build an environmental map and use it as a basis to guide objects to reach the target object. This method relies on accurate map construction and effective exploration algorithms to ensure the robustness and accuracy of the navigation process.

[0004] The end-to-end learning method is based on reinforcement learning (RL) or imitation learning (IL), that is, by receiving RGBD images, semantic segmentation and object detection results as input, it directly learns navigation strategies in the real environment. The advantage of this method is that it can adapt to complex environmental changes. The learning process does not require manually labeled data, but is self-optimized through interaction with the environment.

[0005] Modular learning decomposes the navigation task into subtasks such as perception, path planning and navigation. The semantic mapping module that builds the semantic map is a key component. This module is responsible for understanding the objects and their locations in the environment, while the semantic strategy module based on the semantic map determines the best navigation path based on the location information of the objects. The advantage of modular design is that it improves the scalability and flexibility of the system, allowing each part to be optimized and updated independently.

[0006] It has been found in practice that many of the current semantic navigation methods based on modular learning rely on training for specific tasks. Therefore, they can only process trained object categories and are difficult to expand to new targets or unknown environments, resulting in poor adaptability. Summary of the invention

[0007] In order to overcome at least one of the deficiencies in the prior art, the present application provides a robot target navigation method and related device using multi-cue semantic matching, specifically including:

[0008] In a first aspect, the present application provides a robot target navigation method using multi-cue semantic matching, the method comprising:

[0009] According to the image within the current field of view of the robot, an image similarity map and a category similarity map of the explored area are obtained, wherein the image similarity map records the image similarity between the picture information in the image and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image;

[0010] Obtaining exploration scores of a plurality of frontier points according to the image similarity map and the category similarity map, wherein each frontier point represents a candidate position to be explored by the robot, and the exploration score of each frontier point represents a degree of proximity to the target navigation object;

[0011] According to the exploration scores of the plurality of frontier points, selecting an optimal frontier point with the highest exploration score, and controlling the robot to move to the optimal frontier point;

[0012] If the optimal frontier point is not located at the position of the target navigation object, the robot returns to the image within the current field of view of the robot to obtain the image similarity map and the category similarity map of the explored area until the robot moves to the position of the target navigation object or explores the entire preset area.

[0013] In combination with the optional implementation manner of the first aspect, obtaining exploration scores of multiple frontier points according to the image similarity map and the category similarity map includes:

[0014] Determining the plurality of frontier points from the explored area;

[0015] For each of the frontier points, an image similarity score of the frontier point is obtained according to the image similarity of pixels within a preset range from the frontier point in the image similarity map;

[0016] Obtaining a category similarity score for the frontier point according to the name similarity of pixels within a preset range from the frontier point in the category similarity map;

[0017] Obtaining a distance score of the frontier point according to the distance between the current position of the robot and the frontier point;

[0018] Obtaining a density score of the frontier point according to the distance between the frontier point and other frontier points;

[0019] The image similarity score, the category similarity score, the distance score and the density score are weighted to obtain the exploration score of the frontier point.

[0020] In combination with the optional implementation manner of the first aspect, the weighting of the image similarity score, the category similarity score, the distance score, and the density score to obtain the exploration score of the frontier point includes:

[0021] Acquire a background similarity between at least one background object preset in the image and the target navigation object, and a blank similarity between the navigation prompt text and a blank image;

[0022] Obtaining a weight strategy threshold according to the background similarity of each background object and the blank similarity;

[0023] If a plurality of the frontier points include a semantic tendency node, the image similarity score, the category similarity score, the distance score and the density score are weighted by a first weighting strategy to obtain an exploration score of the frontier point, wherein the image similarity score or the category similarity score of the semantic tendency node is greater than the weighting strategy threshold, and in the first weighting strategy, the weights assigned to the image similarity score and the category similarity score are greater than the weights assigned to the distance score and the density score;

[0024] If the semantic tendency node is not included in the multiple frontier points, the image similarity score, the category similarity score, the distance score and the density score are weighted using a second weighting strategy to obtain the exploration score of the frontier point, wherein in the second weighting strategy, the weights assigned to the image similarity score and the category similarity score are smaller than the weights assigned to the distance score and the density score.

[0025] In combination with an optional implementation manner of the first aspect, obtaining the image similarity score of the frontier point according to the image similarity of pixels within a preset range from the frontier point in the image similarity map includes:

[0026] Using a plurality of pixels within a preset range from the frontier point in the image similarity map as a plurality of first neighborhood pixels;

[0027] The average of the image similarities of the plurality of first neighborhood pixels is used as the image similarity score of the frontier point.

[0028] In combination with an optional implementation manner of the first aspect, obtaining the category similarity score of the frontier point according to the name similarity of pixels within a preset range from the frontier point in the category similarity map includes:

[0029] Using a plurality of pixels within a preset range from the frontier point in the category similarity map as a plurality of second neighborhood pixels;

[0030] The average of the maximum name similarities of the plurality of second neighborhood pixels is used as the category similarity score of the frontier point.

[0031] In combination with an optional implementation manner of the first aspect, the explored area includes an explored area of ​​the robot and an area to be explored corresponding to the image, and determining the multiple frontier points from the explored area includes:

[0032] Determining a plurality of boundaries between the explored area and the area to be explored;

[0033] The plurality of frontier points are determined from the plurality of boundaries.

[0034] In combination with the optional implementation manner of the first aspect, the explored area includes an explored area and an area to be explored corresponding to the image, and a category similarity map of the explored area is obtained according to the image within the current field of view of the robot, including:

[0035] Segmenting the image to determine the object to which each pixel in the image belongs;

[0036] According to the depth information of the image, each pixel in the image is projected onto a plane corresponding to the area to be explored, so as to obtain a semantic submap of the area to be explored, wherein each pixel in the semantic submap corresponds to a spatial position and records an object located at the spatial position;

[0037] For each pixel in the semantic submap, calculating the similarity between the name of each object located at the corresponding spatial position and the name of the target navigation object, to obtain a category similarity submap of the area to be explored;

[0038] The category-similar submap of the to-be-explored area is merged with the category-similar submap of the explored area to obtain a category-similar submap of the explored area.

[0039] In a second aspect, the present application further provides a robot target navigation device using multi-cue semantic matching, the device comprising:

[0040] A similarity map module is used to obtain an image similarity map and a category similarity map of the explored area based on the image within the current field of view of the robot, wherein the image similarity map records the image similarity between the picture information in the image and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image;

[0041] an exploration navigation module, which obtains exploration scores of a plurality of frontier points according to the image similarity map and the category similarity map, wherein each frontier point represents a candidate position to be explored by the robot, and the exploration score of each frontier point represents a degree of proximity to the target navigation object;

[0042] The exploration navigation module is further used to select the best frontier point with the highest exploration score from the exploration scores of the plurality of frontier points, and control the robot to move to the best frontier point;

[0043] If the optimal frontier point is not located at the position of the target navigation object, the robot returns to the image within the current field of view of the robot to obtain the image similarity map and the category similarity map of the explored area until the robot moves to the position of the target navigation object or explores the entire preset area.

[0044] In conjunction with the optional implementation manner of the second aspect, the exploration and navigation module is further specifically configured to:

[0045] Determining the plurality of frontier points from the explored area;

[0046] For each of the frontier points, an image similarity score of the frontier point is obtained according to the image similarity of pixels within a preset range from the frontier point in the image similarity map;

[0047] Obtaining a category similarity score for the frontier point according to the name similarity of pixels within a preset range from the frontier point in the category similarity map;

[0048] Obtaining a distance score of the frontier point according to the distance between the current position of the robot and the frontier point;

[0049] Obtaining a density score of the frontier point according to the distance between the frontier point and other frontier points;

[0050] The image similarity score, the category similarity score, the distance score and the density score are weighted to obtain the exploration score of the frontier point.

[0051] In conjunction with the optional implementation manner of the second aspect, the exploration and navigation module is further specifically configured to:

[0052] Acquire a background similarity between at least one background object preset in the image and the target navigation object, and a blank similarity between the navigation prompt text and a blank image;

[0053] Obtaining a weight strategy threshold according to the background similarity of each background object and the blank similarity;

[0054] If a plurality of the frontier points include a semantic tendency node, the image similarity score, the category similarity score, the distance score and the density score are weighted by a first weighting strategy to obtain an exploration score of the frontier point, wherein the image similarity score or the category similarity score of the semantic tendency node is greater than the weighting strategy threshold, and in the first weighting strategy, the weights assigned to the image similarity score and the category similarity score are greater than the weights assigned to the distance score and the density score;

[0055] If the semantic tendency node is not included in the multiple frontier points, the image similarity score, the category similarity score, the distance score and the density score are weighted using a second weighting strategy to obtain the exploration score of the frontier point, wherein in the second weighting strategy, the weights assigned to the image similarity score and the category similarity score are smaller than the weights assigned to the distance score and the density score.

[0056] In conjunction with the optional implementation manner of the second aspect, the exploration and navigation module is further specifically configured to:

[0057] Using a plurality of pixels within a preset range from the frontier point in the image similarity map as a plurality of first neighborhood pixels;

[0058] The average of the image similarities of the plurality of first neighborhood pixels is used as the image similarity score of the frontier point.

[0059] In conjunction with the optional implementation manner of the second aspect, the exploration and navigation module is further specifically configured to:

[0060] Using a plurality of pixels within a preset range from the frontier point in the category similarity map as a plurality of second neighborhood pixels;

[0061] The average of the maximum name similarities of the plurality of second neighborhood pixels is used as the category similarity score of the frontier point.

[0062] In conjunction with an optional implementation manner of the second aspect, the explored area includes an explored area of ​​the robot and an area to be explored corresponding to the image, and the exploration and navigation module is further specifically configured to:

[0063] Determining a plurality of boundaries between the explored area and the area to be explored;

[0064] The plurality of frontier points are determined from the plurality of boundaries.

[0065] In conjunction with the optional implementation manner of the second aspect, the explored area includes an explored area and an area to be explored corresponding to the image, and the similarity map module is further specifically configured to:

[0066] Segmenting the image to determine the object to which each pixel in the image belongs;

[0067] According to the depth information of the image, each pixel in the image is projected onto a plane corresponding to the area to be explored, so as to obtain a semantic submap of the area to be explored, wherein each pixel in the semantic submap corresponds to a spatial position and records an object located at the spatial position;

[0068] For each pixel in the semantic submap, calculating the similarity between the name of each object located at the corresponding spatial position and the name of the target navigation object, to obtain a category similarity submap of the area to be explored;

[0069] The category-similar submap of the to-be-explored area is merged with the category-similar submap of the explored area to obtain a category-similar submap of the explored area.

[0070] In a third aspect, the present application also provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the robot target navigation method using multi-cue semantic matching is implemented.

[0071] In a fourth aspect, the present application also provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the robot target navigation method using multi-cue semantic matching is implemented.

[0072] Compared with the prior art, this application has the following beneficial effects:

[0073] The present application provides a robot target navigation method and related device using multi-cue semantic matching. In the method, the electronic device obtains an image similarity map and a category similarity map of the explored area according to the image within the current field of view of the robot, wherein the image similarity map records the image similarity between the picture information in the image and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image; according to the image similarity map and the category similarity map, the exploration scores of multiple frontier points are obtained, wherein each frontier point represents a candidate position to be explored by the robot, and the exploration score of each frontier point represents the degree of proximity to the target navigation object; according to the exploration scores of multiple frontier points, the best frontier point with the highest exploration score is selected from them, and the robot is controlled to move to the best frontier point; if the best frontier point is not located at the location of the target navigation object, it returns to the image similarity map and the category similarity map of the explored area according to the image within the current field of view of the robot, until the robot moves to the location of the target navigation object. In this way, on the basis of the image similarity map, the category similarity map is used to provide the association information between the new target or unknown environment and the target navigation object, thereby improving the adaptability to different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0075] Figure 1 A schematic diagram of a method flow chart provided in an embodiment of the present application;

[0076] Figure 2 A schematic diagram of the weight strategy selection principle provided in the embodiment of the present application;

[0077] Figure 3 A schematic diagram of the structure of a virtual device provided in an embodiment of the present application;

[0078] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0080] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0081] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0082] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the invented product is usually placed when in use. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0083] In addition, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0084] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0085] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0086] Based on the above statement, as introduced in the background technology, many current semantic navigation methods based on modular learning rely on training for specific tasks. Therefore, they can only process trained object categories and are difficult to expand to new targets or unknown environments, resulting in poor adaptability.

[0087] Based on the discovery of the above technical problems, the inventors have proposed the following technical solutions to solve or improve the above problems through creative work. It should be noted that the defects in the solutions in the above prior art are the results obtained by the inventors after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the embodiments of the present application for the above problems below should all be the contributions made by the inventors to the present application in the process of invention and creation, and should not be understood as technical contents known to those skilled in the art.

[0088] In view of this, this embodiment provides a robot target navigation method using multi-cue semantic matching. In the method, the electronic device obtains an image similarity map and a category similarity map of the explored area according to the image within the current field of view of the robot, wherein the image similarity map records the image similarity between the picture information in the image and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image; according to the image similarity map and the category similarity map, the exploration scores of multiple frontier points are obtained, wherein each frontier point represents a candidate position to be explored by the robot, and the exploration score of each frontier point represents the degree of proximity to the target navigation object; according to the exploration scores of multiple frontier points, the best frontier point with the highest exploration score is selected, and the robot is controlled to move to the best frontier point; if the best frontier point is not located at the location of the target navigation object, it returns to the image similarity map and the category similarity map of the explored area according to the image within the current field of view of the robot, until the robot moves to the location of the target navigation object. In this way, on the basis of the image similarity map, the category similarity map is used to provide the association information between the new target or unknown environment and the target navigation object, thereby improving the adaptability to different environments.

[0089] It should be noted that the electronic device implementing the method may be a controller integrated with the robot body, or a mobile terminal, a tablet computer, a laptop computer, a desktop computer, and a server, etc., which are communicatively connected to the robot's controller. When it is a server, the server may be a single server or a server group. The server group may be centralized or distributed (for example, the server may be a distributed system). In some embodiments, the server may be local or remote relative to the user terminal. In some embodiments, the server may be implemented on a cloud platform; as an example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud (CommunityCloud), a distributed cloud, an inter-cloud (Inter-Cloud), a multi-cloud (Multi-Cloud), etc., or any combination thereof. In some embodiments, the server may be implemented on an electronic device having one or more components.

[0090] To make the solution provided by this embodiment clearer, the following uses the robot controller as an electronic device to implement the method, and combines Figure 1 Each step of the method is described in detail. However, it should be understood that the operations of the flowchart may not be implemented in order, and steps without logical contextual relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart, or remove one or more operations from the flowchart under the guidance of the content of this application. Figure 1 As shown, the method includes:

[0091] S1, based on the images within the robot’s current field of view, obtain an image similarity map and a category similarity map of the explored area.

[0092] Among them, the image similarity map records the image similarity between the picture information in the image and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image.

[0093] In addition, it should be noted that the above-mentioned explored area includes the explored area and the unexplored area corresponding to the image. It can be understood that the explored area means the area that the robot has explored, and the unexplored area means the new area captured by the robot's camera, but the area has not been explored yet. This embodiment provides the following optional implementation of step S1 for obtaining a category similarity map:

[0094] S1-1, segment the image to determine the object to which each pixel in the image belongs.

[0095] At present, there are a large number of mature image segmentation methods, and technicians can make adaptive choices according to the implementation scenario when implementing this solution. In this embodiment, the robot controller obtains the image within the current field of view through the camera on board; then, Semantic SAM is used to perform instance recognition and segmentation on the image to obtain an instance mask image with semantics, thereby determining the object to which each pixel in the image belongs.

[0096] It should be noted that the above-mentioned Semantic SAM combines semantic segmentation and general object segmentation, which can automatically identify and segment various objects in the image, and assign semantic labels (such as "table", "chair") to each object. It is widely used in autonomous driving, robot vision and other fields. Considering the limited computing power of the robot's controller, Mobile SAM, which requires less computing resources, can also be used to segment the image, which can significantly improve the reasoning speed. It should be understood that on resource-constrained devices such as mobile robots, the lightweight architecture of Mobile SAM is more suitable for real-time processing tasks, while reducing the consumption of memory and computing resources, thereby improving system performance without significantly reducing segmentation accuracy.

[0097] S1-2, according to the depth information of the image, each pixel in the image is projected onto a plane corresponding to the area to be explored, so as to obtain a semantic submap of the area to be explored.

[0098] To this end, this embodiment can generate a three-dimensional point cloud (x, y, z, cls_id) through the depth information of the image and the segmentation mask segmented by the SAM model, combined with the camera intrinsic parameter back projection, where (x, y, z) represents the spatial coordinates of the point cloud, and cls_id represents the category label of the object to which the point cloud belongs. In addition, in order to deal with the noise problem in the depth camera measurement, this implementation also introduces confidence, which indicates the possibility that each point cloud actually exists in space. Therefore, the point cloud finally generated contains not only spatial location information, but also the category and confidence of the object. Finally, it is projected to the plane where the area to be explored is located to obtain a semantic submap of the area to be explored.

[0099] It should be understood that the point clouds on the same vertical line in the three-dimensional space will be projected to the same position of the two-dimensional plane. Therefore, in this embodiment, each pixel in the semantic submap corresponds to a spatial position and records the object located at the spatial position. It can be understood that in this embodiment, according to the type of object in the robot's environment, a corresponding number of channels are provided for each pixel in the semantic submap, each channel represents an object, and is used to record whether the object represented by the channel exists in the spatial position corresponding to the pixel. For example, for a total of 3 three-dimensional point clouds (x, y, z1, cls_id1), (x, y, z2, cls_id3), and (x, y, z3, cls_id3), since they have the same (x, y), when projected to the two-dimensional plane, they will all be projected to the position of (x, y). In the actual scene, cls_id1 may be a "desk", cls_id2 may be a "computer" on the "desk", and cls_id3 may be a "light" located above the "computer", that is, the three are on the same vertical line.

[0100] Based on the introduction of the semantic submap in the above embodiment, step S1 further includes:

[0101] S1-3, for each pixel in the semantic submap, calculate the similarity between the name of each object located at the corresponding spatial position and the name of the target navigation object to obtain a category similarity submap of the area to be explored.

[0102] It should be understood that there are currently a large number of mature methods that can be used to evaluate the semantic similarity between two names. When implementing this solution, technicians can make adaptive choices based on the implementation scenario. In this embodiment, the object types in the robot's environment come from the ADE20K dataset, which contains a total of 150 different semantic categories, such as people, vehicles, plants, buildings, etc. This embodiment encodes the names of the object types (i.e., category labels) and the names of the target navigation objects in the ADE20K dataset through the Sentence Transformer model, and converts them into high-dimensional semantic vectors; then, by calculating the cosine similarity, the semantic similarity between the name of each object and the name of the target navigation object is measured. Among them, the closer the cosine similarity is to 1, the higher the semantic similarity, and conversely, the closer it is to 0, the irrelevant it is. In this way, the semantic submap can be converted into a category-similar submap.

[0103] S1-4, merging the category-similar submap of the area to be explored with the category-similar submap of the explored area to obtain a category-similar submap of the explored area.

[0104] In this embodiment, the category similarity submap of the area to be explored is merged with the category similarity submap of the explored area to obtain a category similarity map of the explored area, whose dimension is H*W*class_num, that is, for each pixel (H, W), there are class_num channels, and each channel records the similarity between the name of the corresponding object and the name of the target navigation object.

[0105] For the above-mentioned image similarity map, there are also mature methods that can be used to evaluate the similarity between image content and description text. However, some related technologies require powerful computing resources, especially when relying on language models for semantic reasoning, which may require remote server support, limiting the autonomy of the robot. In addition, these technologies often need to convert visual information into text for processing, which increases computational complexity and time cost, and reduces efficiency.

[0106] In this regard, in this embodiment, the controller captures the image within the current field of view through the robot camera, and calculates the cosine similarity between the image and the navigation prompt text through the BLIP-2 model, which is recorded as BLIP2ITM Score. In this embodiment, it is called a semantic score, which indicates the degree of match between the image content in the image and the navigation prompt text. Then, the controller projects the semantic score to the plane where the image similarity submap of the explored area is located, and combines it with the depth information to exclude areas blocked by obstacles; finally, the semantic score of each pixel is weighted with the confidence in the corresponding three-dimensional point cloud to obtain an image similarity map of the explored area for subsequent navigation decisions.

[0107] For example, assume that the navigation prompt text is "Seems like there is a [TV] ahead", where [TV] represents the target navigation object, that is, the robot needs to be controlled to navigate to the location of [TV]. If the image within the robot's current field of view happens to show a TV ahead, the semantic scores of the two are close to 1. On the contrary, if there is no TV ahead in the image and there are no objects related to the TV, the semantic scores of the two are close to 0.

[0108] Based on the introduction of the image similarity map and the category similarity map in the above embodiment, continue to refer to Figure 1 The robot target navigation method using multi-cue semantic matching provided in this embodiment also includes:

[0109] S2, based on the image similarity map and the category similarity map, obtains the exploration scores of multiple frontier points.

[0110] Each frontier point represents a candidate position to be explored by the robot, and the exploration score of each frontier point represents the degree of proximity to the target navigation object. In addition, the above exploration score is a comprehensive result of multiple scores. In this regard, this implementation also provides the following optional implementation of step S2:

[0111] S2-1, identify multiple frontier points from the explored area.

[0112] Since the explored area includes the robot's explored area and the unexplored area, the robot's controller can determine the boundary between the explored area and the unexplored area, and determine multiple frontier points from the multiple boundaries. For example, for each boundary, the midpoint or end point of the boundary can be used as the frontier point.

[0113] Based on the above introduction to the frontier point, step S2 further includes:

[0114] S2-2, for each frontier point, an image similarity score of the frontier point is obtained according to the image similarity of pixels within a preset range from the frontier point in the image similarity map.

[0115] In this embodiment, the controller of the robot can use multiple pixels within a preset range from the frontier point in the image similarity map as multiple first neighborhood pixels, and use the average of the image similarities of the multiple first neighborhood pixels as the image similarity score of the frontier point. The preset range refers to a circular range with the frontier point as the center, and the radius of the circular range is a preset value.

[0116] Based on the introduction of the image similarity score of the frontier point in the above embodiment, step S2 further includes:

[0117] S2-3, obtaining a category similarity score of the frontier point according to the name similarity of pixels within a preset range from the frontier point in the category similarity map.

[0118] In this embodiment, the robot's controller can use multiple pixels within a preset range from the frontier point in the category similarity map as multiple second neighborhood pixels; and use the average of the maximum name similarities of the multiple second neighborhood pixels as the category similarity score of the frontier point.

[0119] In this regard, it can be understood that, since the above embodiment has introduced the category similarity map of the explored area, its dimension is H*W*class_num, that is, for each pixel (H, W), there are class_num channels, and each channel records the similarity between the name of the corresponding object and the name of the target navigation object. Therefore, in the class_num channel of each second neighborhood pixel, there is a maximum name similarity. This embodiment accumulates the maximum name similarities of each of these second neighborhood pixels, and divides the accumulated cumulative similarity by the number of multiple second neighborhood pixels to obtain the category similarity score of the frontier point.

[0120] Based on the introduction of the frontier point image similarity score and the category similarity score from the semantic perspective in the above embodiment, this embodiment also evaluates the exploration value of the frontier point from the perspective of spatial distance. Therefore, step S2 also includes:

[0121] S2-4, obtain the distance score of the frontier point according to the distance between the current position of the robot and the frontier point.

[0122] Exemplarily, the distance between the frontier point and the current position of the robot is the Euclidean distance, and the calculation expression is:

[0123]

[0124] In the formula, (x i ,y i ) represents the coordinates of the i-th frontier point, (x r ,y r ) represents the coordinates of the robot’s location, and distance(i) represents the Euclidean distance between the i-th frontier point and the robot.

[0125] Considering that the above Euclidean distance is an absolute distance, it is not convenient for subsequent analysis. Therefore, the Euclidean distance between other frontier points and the robot is also combined to normalize it. The expression is as follows:

[0126]

[0127] Where distance(i) represents the Euclidean distance between the i-th frontier point and the robot, min(distance) represents the minimum distance among the Euclidean distances from all frontier points to the robot, and max(distance) represents the maximum distance among the Euclidean distances from all frontier points to the robot.

[0128] Based on the introduction of the distance score in the above embodiment, step S further includes:

[0129] S2-5, obtain the density score of the frontier point based on the distance between the frontier point and other frontier points.

[0130] In this regard, this embodiment provides the following calculation expression for the density score:

[0131]

[0132] Where 1(distance(i,j)≤radius) means that if the distance between the i-th frontier point and the j-th frontier point is less than or equal to the specified radius, it is counted as 1. Distance(i,j) represents the Euclidean distance between the i-th frontier point and the j-th frontier point. In the expression, the sum is subtracted by 1 to subtract the influence of the i-th frontier point itself.

[0133] Based on the image similarity score, category similarity score, distance score and density score obtained in the above implementation, step S2 further includes:

[0134] S2-6, weight the image similarity score, category similarity score, distance score and density score to obtain the exploration score of the frontier point.

[0135] In this regard, this embodiment provides an adaptive weight strategy, that is, different weight strategies are adopted in the two cases of tending to follow semantic clues and tending to explore scenes. In the optional implementation provided by this embodiment, step S2-6 may include:

[0136] S2-6-1, obtaining the background similarity between at least one background object preset in the image and the target navigation object, and the blank similarity between the navigation prompt text and the blank image.

[0137] In this embodiment, the at least one background object may include at least one of a wall, a ceiling, a floor, etc. The blank similarity between the navigation prompt text and the blank image may be calculated by a BLIP-2 model.

[0138] S2-6-2, obtaining a weight strategy threshold according to the background similarity and blank similarity of each background object.

[0139] As an optional implementation, the mean between the background similarity and the blank similarity of at least one background object can be calculated and used as the weight strategy threshold. Of course, as another optional implementation, the maximum value between the background similarity and the blank similarity of at least one background object can be selected as the weight strategy threshold.

[0140] S2-6-3, determine whether multiple frontier points include semantically inclined nodes.

[0141] Wherein, the image similarity score or category similarity score of the semantic tendency node is greater than the weight strategy threshold. If so, step S2-6-4 is executed, otherwise, step S2-6-5 is executed.

[0142] S2-6-4, weight the image similarity score, category similarity score, distance score and density score using the first weighting strategy to obtain the exploration score of the frontier point.

[0143] In the first weighted strategy, the weights assigned to the image similarity score and the category similarity score are greater than the weights assigned to the distance score and the density score.

[0144] S2-6-5, weight the image similarity score, category similarity score, distance score and density score using the second weighting strategy to obtain the exploration score of the frontier point.

[0145] Among them, in the second weighted strategy, the weights assigned to the image similarity score and the category similarity score are smaller than the weights assigned to the distance score and the density score.

[0146] For the above step S2-6, the entire step is fully described below in conjunction with a specific example. Figure 2 As shown in the figure, on the left side, for the navigation prompt text "Seems like there is a [TV] ahead", the BLIP2 ITM Score between it and the blank image is first calculated through the BLIP-2 model; and the semantic similarity between the target navigation object [TV] and the preset background objects such as "Wall", "Ceiling", and "Foor" is calculated; finally, the average is taken to obtain the weighted strategy threshold.

[0147] On the right side of the figure, for the image in the current field of view, the BLIP2 ITM Score between it and the prompt text "Seems like there is a [TV] ahead" is calculated through the BLIP-2 model; and the BLIP2 ITM Score is converted into an image similarity map. The image similarity map can be used to obtain the image similarity score of each frontier point.

[0148] For each pixel in the semantic map, the cosine similarity between the name of each object at the corresponding spatial position and the name of the target navigation object is calculated, and then a category similarity map is obtained. The category similarity score of each frontier point is obtained through the category similarity map.

[0149] Finally, the image similarity score of each frontier point is compared with the weight strategy threshold, and the category similarity score of each frontier point is compared with the weight strategy threshold. As long as there is a case where the comparison result is greater than the weight strategy threshold, the first weighted strategy that tends to follow semantic clues is selected. Otherwise, the second weighted strategy that tends to explore the scene is selected.

[0150] Continue to see Figure 2 In the first weighted strategy, the weight of the image similarity score is 0.4, the weight of the class similarity score is 0.4, the weight of the distance score is 0.1, and the weight of the density score is 0.1. In the second weighted strategy, the weight of the image similarity score is 0.1, the weight of the class similarity score is 0.1, the weight of the distance score is 0.4, and the weight of the density score is 0.4. In this way, the robot controller can decide which of the semantic information and distance information in the current image is more trustworthy, from the matching weight strategy.

[0151] Based on the introduction of the exploration score in the above embodiment, continue to refer to Figure 1 , the method provided in this embodiment also includes:

[0152] S3, according to the exploration scores of multiple frontier points, select the best frontier point with the highest exploration score, and control the robot to move to the best frontier point.

[0153] In this regard, in view of the fact that there are many mature path planning methods at present, technicians can make adaptive choices according to the implementation scenario when implementing this solution. In this embodiment, the robot's controller can use the PointNav strategy to guide the robot to navigate from one position to the optimal frontier point. It should be understood that the PointNav strategy relies on visual and odometer observations. Therefore, there is no need for semantic understanding of the environment, and only needs to navigate based on geometric clues (depth images and distance and direction relative to the target point). In addition, the strategy is trained through deep reinforcement learning, using the Variable ExperienceRollout (VER) algorithm, and after training in a large number of simulated environments, it can effectively navigate to the target point in different scenarios.

[0154] Continue to see Figure 1 Based on the introduction of the optimal frontier point in the above implementation, the robot target navigation method using multi-cue semantic matching provided in this embodiment also includes:

[0155] S4, determine whether the best front point is located at the location of the target navigation object. If not, return to step S5, if yes, execute S6.

[0156] S5, judging whether the entire preset area has been explored. If not, returning to step S1, otherwise, executing step S6.

[0157] S6, ends the exploration task of the target navigation object.

[0158] It can be understood that the robot controller may find the target navigation object to complete the task under the prompt of the navigation prompt text. It may also fail to find the target navigation object after exploring the entire preset area, so that the task fails. The preset area here is pre-defined by the user, for example, the preset area is the entire room or the entire floor where the target navigation object is located.

[0159] Based on the same inventive concept as the robot target navigation method using multi-cue semantic matching provided in this embodiment, this embodiment also provides a robot target navigation device using a similarity map, the device includes at least one software function module that can be stored in a memory or fixed in an electronic device in the form of software. The processor in the electronic device is used to execute the executable module stored in the memory. For example, the software function module and computer program included in the device. Please refer to Figure 3 , functionally speaking, the device may include:

[0160] The similarity map module 11 is used to obtain an image similarity map and a category similarity map of the explored area according to the image within the current field of view of the robot, wherein the image similarity map records the image similarity between the picture information in the image and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image;

[0161] An exploration navigation module 12 obtains an exploration score of a frontier point according to the image similarity map and the category similarity map, wherein the frontier point represents a candidate position to be explored by the robot, and the exploration score of the frontier point represents the degree of proximity to the target navigation object;

[0162] The exploration navigation module 12 is further used to select the best frontier point with the highest exploration score according to the exploration scores of the multiple frontier points, and control the robot to move to the best frontier point;

[0163] If the optimal frontier point is not located at the location of the target navigation object, it returns to the image within the current field of view of the robot to obtain the image similarity map and category similarity map of the explored area until the robot moves to the location of the target navigation object or explores the entire preset area.

[0164] In this embodiment, the similar map module 11 is used to implement Figure 1 Step S1 in the exploration navigation module is used to implement Figure 1Therefore, for the detailed description of each of the above modules, please refer to the specific implementation of the corresponding step. In addition, since the invention concept is the same as that of the robot target navigation method using multi-cue semantic matching, the robot target navigation device using multi-cue semantic matching can also implement other steps or sub-steps of the method through the above modules or other modules, which will not be described in detail in this embodiment.

[0165] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0166] It should also be understood that if the above implementation is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0167] Therefore, this embodiment also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by the processor, the robot target navigation method using multi-cue semantic matching provided in this embodiment is implemented. Among them, the storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc., which can store program codes.

[0168] This embodiment also provides an electronic device for implementing a robot target navigation method using multi-cue semantic matching. Figure 4 As shown, the electronic device may include a processor 22 and a memory 21. In addition, the memory 21 stores a computer program, and the processor implements the robot target navigation method using multi-cue semantic matching provided in this embodiment by reading and executing the computer program corresponding to the above implementation in the memory 21.

[0169] Continue to see Figure 4 The electronic device further includes a communication unit 23. The memory 21, the processor 22 and the communication unit 23 are electrically connected to each other directly or indirectly through a system bus 24 to achieve data transmission or interaction.

[0170] The memory 21 may be an information recording device based on any electronic, magnetic, optical or other physical principle, used to record execution instructions, data, etc. In some embodiments, the memory 21 may be, but is not limited to, a volatile memory, a non-volatile memory, a storage drive, etc.

[0171] In some embodiments, the volatile memory may be a random access memory (RAM); in some embodiments, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a flash memory, etc.; in some embodiments, the storage drive may be a disk drive, a solid-state drive, any type of storage disk (such as a CD, a DVD, etc.), or a similar storage medium, or a combination thereof, etc.

[0172] The communication unit 23 is used to send and receive data through a network. In some embodiments, the network may include a wired network, a wireless network, a fiber optic network, a telecommunication network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof. In some embodiments, the network may include one or more network access points. For example, the network may include a wired or wireless network access point, such as a base station and / or a network switching node, through which one or more components of the service request processing system may be connected to the network to exchange data and / or information.

[0173] The processor 22 may be an integrated circuit chip having a signal processing capability, and the processor may include one or more processing cores (e.g., a single-core processor or a multi-core processor). By way of example only, the processor may include a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), or a microprocessor, or any combination thereof.

[0174] Understandably, Figure 4 The structure shown is for illustration only. The electronic device may also have Figure 4 More or fewer components than shown, or with Figure 4 Different configurations are shown. Figure 4 The components shown may be implemented in hardware, software or a combination thereof.

[0175] It should be understood that the apparatus and method disclosed in the above-mentioned embodiments can also be implemented in other ways. The apparatus embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A robot target navigation method using multi-cue semantic matching, characterized in that: The method comprises: According to the image within the current field of view of the robot, an image similarity map and a category similarity map of the explored area are obtained, wherein the image similarity map records the image similarity between the picture information in the image and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image; Obtaining exploration scores of a plurality of frontier points according to the image similarity map and the category similarity map, wherein each frontier point represents a candidate position to be explored by the robot, and the exploration score of each frontier point represents a degree of proximity to the target navigation object; According to the exploration scores of the plurality of frontier points, selecting an optimal frontier point with the highest exploration score, and controlling the robot to move to the optimal frontier point; If the optimal frontier point is not located at the position of the target navigation object, the robot returns to the image within the current field of view of the robot to obtain the image similarity map and the category similarity map of the explored area until the robot moves to the position of the target navigation object or explores the entire preset area.

2. The robot target navigation method using multi-cue semantic matching according to claim 1 is characterized in that: The step of obtaining exploration scores of multiple frontier points according to the image similarity map and the category similarity map includes: Determining the plurality of frontier points from the explored area; For each of the frontier points, an image similarity score of the frontier point is obtained according to the image similarity of pixels within a preset range from the frontier point in the image similarity map; Obtaining a category similarity score for the frontier point according to the name similarity of pixels within a preset range from the frontier point in the category similarity map; Obtaining a distance score of the frontier point according to the distance between the current position of the robot and the frontier point; Obtaining a density score of the frontier point according to the distance between the frontier point and other frontier points; The image similarity score, the category similarity score, the distance score and the density score are weighted to obtain the exploration score of the frontier point.

3. The robot target navigation method using multi-cue semantic matching according to claim 2 is characterized in that: The step of weighting the image similarity score, the category similarity score, the distance score and the density score to obtain the exploration score of the frontier point includes: Acquire a background similarity between at least one background object preset in the image and the target navigation object, and a blank similarity between the navigation prompt text and a blank image; Obtaining a weight strategy threshold according to the background similarity of each background object and the blank similarity; If a plurality of the frontier points include a semantic tendency node, the image similarity score, the category similarity score, the distance score and the density score are weighted by a first weighting strategy to obtain an exploration score of the frontier point, wherein the image similarity score or the category similarity score of the semantic tendency node is greater than the weighting strategy threshold, and in the first weighting strategy, the weights assigned to the image similarity score and the category similarity score are greater than the weights assigned to the distance score and the density score; If the semantic tendency node is not included in the multiple frontier points, the image similarity score, the category similarity score, the distance score and the density score are weighted using a second weighting strategy to obtain the exploration score of the frontier point, wherein in the second weighting strategy, the weights assigned to the image similarity score and the category similarity score are smaller than the weights assigned to the distance score and the density score.

4. The robot target navigation method using multi-cue semantic matching according to claim 2 is characterized in that: The obtaining the image similarity score of the frontier point according to the image similarity of pixels within a preset range from the frontier point in the image similarity map comprises: Using a plurality of pixels within a preset range from the frontier point in the image similarity map as a plurality of first neighborhood pixels; The average of the image similarities of the plurality of first neighborhood pixels is used as the image similarity score of the frontier point.

5. The robot target navigation method using multi-cue semantic matching according to claim 2, characterized in that: The obtaining of the category similarity score of the frontier point according to the name similarity of pixels within a preset range from the frontier point in the category similarity map includes: Using a plurality of pixels within a preset range from the frontier point in the category similarity map as a plurality of second neighborhood pixels; The average of the maximum name similarities of the plurality of second neighborhood pixels is used as the category similarity score of the frontier point.

6. The robot target navigation method using multi-cue semantic matching according to claim 2, characterized in that: The explored area includes an explored area of ​​the robot and an area to be explored corresponding to the image, and determining the plurality of frontier points from the explored area includes: Determining a plurality of boundaries between the explored area and the area to be explored; The plurality of frontier points are determined from the plurality of boundaries.

7. The robot target navigation method using multi-cue semantic matching according to claim 1, characterized in that: The explored area includes an explored area and an area to be explored corresponding to the image. According to the image within the current field of view of the robot, a category similarity map of the explored area is obtained, including: Segmenting the image to determine the object to which each pixel in the image belongs; According to the depth information of the image, each pixel in the image is projected onto a plane corresponding to the area to be explored, so as to obtain a semantic submap of the area to be explored, wherein each pixel in the semantic submap corresponds to a spatial position and records an object located at the spatial position; For each pixel in the semantic submap, calculating the similarity between the name of each object located at the corresponding spatial position and the name of the target navigation object, to obtain a category similarity submap of the area to be explored; The category-similar submap of the to-be-explored area is merged with the category-similar submap of the explored area to obtain a category-similar submap of the explored area.

8. A robot target navigation device using multi-cue semantic matching, characterized in that: The device comprises: A similarity map module is used to obtain an image similarity map and a category similarity map of the explored area based on the image within the current field of view of the robot, wherein the image similarity map records the image similarity between the picture information in the image and the navigation prompt text, and the category similarity map records the name similarity between the target navigation object in the navigation prompt text and each object in the image; an exploration navigation module, which obtains exploration scores of a plurality of frontier points according to the image similarity map and the category similarity map, wherein each frontier point represents a candidate position to be explored by the robot, and the exploration score of each frontier point represents a degree of proximity to the target navigation object; The exploration navigation module is further used to select the best frontier point with the highest exploration score from the exploration scores of the plurality of frontier points, and control the robot to move to the best frontier point; If the optimal frontier point is not located at the position of the target navigation object, the robot returns to the image within the current field of view of the robot to obtain the image similarity map and the category similarity map of the explored area until the robot moves to the position of the target navigation object or explores the entire preset area.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the robot target navigation method using multi-cue semantic matching as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the robot target navigation method using multi-cue semantic matching as described in any one of claims 1-7 is implemented.

Images