Multi-robot visual semantic navigation method and device and storage medium

Abstract

The present disclosure relates to a multi-robot visual semantic navigation method, multiple robots have their own passable areas, each robot establishes a semantic map according to its past observations of the scene, and improves the semantic map through communication with each other. When performing top-level planning, the robot not only considers the probability of the existence of target objects at each location, but also comprehensively considers the uncertainty of the prediction of the location, and selects the point with the highest score in the feasible region as the target location for the next step after weighting and summing the probability and uncertainty. The feasible region map and the semantic map are continuously updated during navigation until one of the robots finds the target object. The present disclosure balances the relationship between exploration and utilization, allowing the robot to explore as much as possible in the early stage, and enabling faster positioning to the target object after having certain perception information of the environment. It is used for target object finding in various scenes, and the category of the target object is any category in the semantic map, without additional training.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of robot navigation, and particularly relates to a multi-robot visual semantic navigation method and device and a storage medium. BACKGROUND

[0002] Semantic segmentation refers to dividing a given image according to the semantic categories of objects appearing in the image. Semantic segmentation can extract semantic information corresponding to each position from the image, and has a wide and important application in many downstream tasks such as target positioning, image understanding, human-computer interaction, etc.

[0003] In recent years, with the development of deep learning, the accuracy of semantic segmentation has been greatly improved, and the semantic segmentation model can accurately give the semantic category to which each pixel in the image belongs. By combining with the depth map to map the pixels, semantic segmentation can be converted into a semantic map, which further helps the robot to understand the scene and navigate the target object. However, the current target object navigation is often directly learned for a given target object label type, which has a high correlation with the label of the target object, and lacks exploration of the scene. This feature also leads to a great decline in the performance of the robot when it is in a new scene or looking for a new target object. If the robot can prioritize the exploration of the scene in the case of less perception information of the scene, rather than directly navigating to the position of the target object it considers based on the knowledge during training, the generalization ability of the robot in the new scene and the adaptability to new types of target objects will be improved.

[0004] Current target object navigation methods mainly encode the RGB observation image through a convolutional neural network (CNN) to obtain an RGB observation vector, and then encode the historical RGB observation vector through a long short-term memory neural network (LSTM), and predict the next action of the robot through the hidden state output by the LSTM. When navigating the target, the regions of the RGB image after convolution can also be assigned weights according to the given target object category, and a weighted average of these regions can be performed to help the network pay more attention to the regions related to the target object. Some navigation methods also introduce a target detector or a semantic segmentation model to help the robot learn a more robust navigation model through more diverse information input. However, all these models only focus on how to search for the location of the target object as quickly as possible when testing. In an unseen scene, the arrangement and distribution of objects may be very different, and when the perception information of the environment is less at the beginning of the search, directly using the knowledge learned during training to search may go in the wrong direction, ultimately leading to the failure of the search task. This also leads to the poor generalization ability of current target object navigation algorithms in new scenes. At the same time, most target object navigation methods are for single robots, and even some for multiple robots do not consider that multiple robots may be heterogeneous, with different motion capabilities and passable areas. SUMMARY

[0005] The present disclosure aims to at least partially solve one of the technical problems in the related art.

[0006] To this end, the present disclosure provides a multi-robot visual semantic navigation method, device and storage medium. The present disclosure first considers the heterogeneity of each robot and plans according to the motion capability and passable area of each robot. Furthermore, the present disclosure balances the relationship between exploration and exploitation, allowing the robot to perform some necessary exploration at the beginning of navigation, and as the number of navigation steps increases, the robot can use the perception of the scene during the exploration process to quickly locate the target object.

[0007] To achieve the above object, the present disclosure adopts the following technical solutions:

[0008] The first aspect of the present disclosure provides a multi-robot visual semantic navigation method, comprising:

[0009] The training phase: collect the RGB image and the depth image at each position in the scene, perform semantic segmentation and point cloud mapping, obtain the real global passable area map and the real global semantic map of the scene, collect the RGB image and the depth image at each position on each path in the scene, perform semantic segmentation and point cloud mapping, obtain the local passable area map and the local semantic map of each path at each time, fuse the local passable area map and the local semantic map of each path at each time with the global passable area map and the global semantic map stored by the robot before sampling the respective path respectively, obtain the fused global passable area map and the global semantic map at each time, and use the fused global passable area map and the global semantic map at each time and the real global semantic map of the scene as training samples to construct a training data set; use the training data set to train the deep neural network model, and obtain a semantic map prediction model; wherein each point in the semantic map represents the probability of the existence of each object at the point, and each point in the semantic map corresponds to the position of the corresponding point in the passable area map one by one;

[0010] The navigation phase: the robot obtains a real-time local passable area map and a real-time local semantic map, fuses the real-time local passable area map and the real-time local semantic map with the global passable area map and the global semantic map stored by the robot respectively, obtains a real-time global passable area map and a fused semantic map, and inputs the real-time global passable area map and the fused semantic map into the semantic map prediction model to obtain a predicted real-time global semantic map; the robots communicate with each other to update the global semantic map of each robot; each robot selects a point with the highest weighted sum value in the local passable area map as a target position in the next step after weighting and summing the probability of the existence of a target object at each position in the updated global semantic map of the robot and the uncertainty of each position; each robot goes to the target position and searches for a target object, and stops navigation when one of the robots finds the target object or reaches the upper limit of the number of navigation steps.

[0011] In some embodiments, in the navigation phase, the real global passable area map and the real global semantic map of the scene are obtained according to the following steps:

[0012] In the scene where the robot is located, each position is traversed, the RGB image I t and the depth image D t corresponding to the position at time t in the scene are collected by using the vision sensor carried by the robot, the coordinates of each position in the RGB image I t and the depth image D t are obtained through point cloud mapping to obtain a local passable area map m o , the local passable area map m o is subjected to semantic segmentation to obtain a local semantic map m s , and the local passable area map m oand the local semantic map m s The data is then fused separately to obtain the true global traversable domain map M for the entire scene. o and the true global semantic map M s ;

[0013] The following steps are used to obtain the fused global traversable domain map and global semantic map for each time point:

[0014] Using the robot's onboard vision sensors, RGB images of each path at each moment in the scene are acquired. t，i and depth image D t，i The subscript 'i' represents the path number in the scene, and each RGB image is represented by 'i'. t，i and depth image D t，i The coordinates of each location are used to obtain a local traversable area map through point cloud mapping. For the local traversable area map Semantic segmentation is performed to obtain a local semantic map. The local accessible area map and the local semantic map Compare this to the global traversable region map stored by the robot at the previous time step. and global semantic map By merging the data, a global traversable region map at time t is obtained. and fusion semantic map

[0015] In some embodiments, training the deep neural network model using the training dataset to obtain a semantic map prediction model specifically includes:

[0016] Using U-Net as the backbone network of the deep neural network model, each training sample from the training dataset is input into the deep neural network model for training, so that the global semantic map predicted by the deep neural network model is closer to the real global semantic map M of the entire scene. s The semantic map has dimensions C×H×W, and each point in the semantic map is represented by a C-dimensional vector Cls. k ={Cls k，j} represents the total number of object categories in the scene, where C is the total number of categories, and Cls is the total number of categories. k Each element Cls in k，j Representative position l k The probability that an object of class j exists at location l k k∈1,2,…,H×W, where H and W are the width and height of the gridded semantic map, respectively, and multiple categories of objects are allowed at each location;

[0017] The training process of the deep neural network model is constrained by a multi-label loss function L, expressed as follows:

[0018]

[0019] where T is the total number of steps of time; is an indicator function, which is equal to 1 only when position l k there is an object of class j at position l , otherwise is equal to 0, and the value of the indicator function at each position is determined according to the real global semantic map Ms; is the probability that the deep neural network model predicts that there is an object of class j at position l k in the global semantic map at time t;

[0020] The training is stopped when it reaches a set upper limit, and the semantic map prediction model is obtained.

[0021] In some embodiments, the real-time local passable region map and the real-time local semantic map are obtained by the following steps:

[0022] At time t Before navigation, each robot maps the RGB image and the depth image observed by itself into a feasible region map and a semantic map centered on itself in combination with the semantic segmentation result, and converts them into a real-time local feasible region map and a real-time local semantic map

[0023] In some embodiments, in the navigation stage, when each robot fuses the real-time local feasible region map with the global feasible region map stored by itself, if a certain position in the global feasible region map stored by itself is in an unknown state, the value of the corresponding position in the real-time local feasible region map is directly assigned by the value of the certain position in the global feasible region map stored by itself; if the value of the corresponding position in the local feasible region map and the global passable region map stored by itself is inconsistent, the position is marked as having an obstacle, and thus a real-time global passable region map of each robot is obtained.

[0024] In some embodiments, in the navigation stage, the plurality of robots communicate with each other to update the global semantic map of each robot, specifically including:

[0025] Each robot compresses and encodes its own global semantic map before sending it to other robots. It then integrates the global semantic maps received from all other robots with its own, fusing them together. During fusion, each robot only updates the parts of its own global semantic map that are not present in its own map, leaving the parts it has already observed unupdated, thus obtaining an updated global semantic map.

[0026] In some embodiments, each robot determines its respective target position through the following steps.

[0027] For the robot's updated global semantic map Each given position l in m m∈1,2,…H×W, where H and W are the width and height of the meshed semantic map, respectively, and the points in the updated global semantic map are... This corresponds to a C-dimensional vector Cls m ={Cls m，j}, which represents the position l m The probability of the existence of various objects is calculated by the vector Cls. m The entropy is used to obtain the semantic map prediction model for location l. m Uncertainty in the distribution of objects:

[0028]

[0029] in, Indicates position l m The corresponding uncertainty;

[0030] Determining target location by combining probability and uncertainty The calculation formula is as follows:

[0031]

[0032] Among them, NR q This represents the traversable area for robot q, based on the real-time global traversable area map of robot q. Sure; Indicates the category of the target object; Indicates position l m The probability that the target object exists, i.e., the robot's updated global semantic map. Middle position l m Place The value of ; α is the weighting coefficient.

[0033] In some embodiments, the multiple robots are heterogeneous robots.

[0034] The second aspect of the present disclosure provides a multi-robot visual semantic navigation device, comprising:

[0035] a training module configured to collect RGB images and depth images at each position in the scene, perform semantic segmentation and point cloud mapping thereon, obtain a real global passable area map and a real global semantic map of the scene, collect RGB images and depth images at each position on each path in the scene, perform semantic segmentation and point cloud mapping thereon, obtain a local passable area map and a local semantic map of each path at each time, fuse the local passable area map and the local semantic map of each path at each time with the global passable area map and the global semantic map stored by the robot before sampling the respective paths respectively, obtain a global passable area map and a global semantic map fused at each time, and use the global passable area map and the global semantic map fused at each time and the real global semantic map of the scene as training samples to construct a training data set; train a deep neural network model using the training data set to obtain a semantic map prediction model; wherein each point in the semantic map represents the probability of existence of each object at the point, and each point in the semantic map corresponds one-to-one to a corresponding point in the passable area map;

[0036] a navigation module configured to make the robot obtain a real-time local passable area map and a real-time local semantic map, fuse the real-time local passable area map and the real-time local semantic map with the global passable area map and the global semantic map stored by the robot respectively, obtain a real-time global passable area map and a fused semantic map, and input the real-time global passable area map and the fused semantic map into the semantic map prediction model to obtain a predicted real-time global semantic map; the plurality of robots communicate with each other to update the global semantic map of each robot; each robot selects a point with the highest weighted sum value in the local passable area map as a target position for the next step after weighting and summing the probability of existence of a target object at each position in the updated global semantic map of the robot and the uncertainty of each position prediction; each robot goes to the target position and searches for a target object, and stops navigation when one of the robots finds the target object or reaches an upper limit of the number of navigation steps.

[0037] The third aspect of the present disclosure provides a computer-readable storage medium storing computer instructions for causing the computer to execute the multi-robot visual semantic navigation method according to any one of the embodiments of the first aspect of the present disclosure.

[0038] The present disclosure has the characteristics and advantages that:

[0039] The present disclosure utilizes a deep learning-based semantic segmentation model to perform semantic segmentation on an RGB image, and obtains a semantic map by mapping and gridding the pixel points through a depth map. The model predicts a complete semantic map based on a seen semantic map by utilizing the correlation between various objects, and takes the probability of a target object existing at a position in the predicted semantic map as the confidence, and takes the probability of each object at the position as the uncertainty, and through the weighted sum of the two, helps the agent to balance exploration and utilization of perception information for search, and improves the generalization ability of the robot in a new scene.

[0040] The present disclosure can be used in the field of multi-robot target navigation, allowing multiple heterogeneous robots to start from their respective positions, jointly explore and search the scene, and quickly search for the position of the target object while establishing a complete and accurate semantic map of the scene. The present disclosure balances the relationship between exploration and utilization, allowing the robot to explore as much as possible in the early stage, and quickly locate the target object after having certain perception information of the environment. The present disclosure can be used for finding target objects in indoor and outdoor scenes, and the category of the target object can be any category in the semantic map without additional training. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flowchart of a multi-robot visual semantic navigation method provided by a first aspect embodiment of the present disclosure.

[0042] Figure 2 is a structural schematic diagram of an electronic device provided by a third aspect embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0044] On the contrary, the present application covers any alternative, modification, equivalent method and scheme defined by the claims on the essence and scope of the present application. Further, in order to make the public better understand the present application, some specific details are described in detail in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.

[0045] Referring to Figure 1 The multi-robot visual semantic navigation method provided by the first aspect embodiment of the present disclosure comprises:

[0046] The training phase: the robot traverses each position in the scene where the robot is located, collects the RGB image and the depth image at each position, performs semantic segmentation and point cloud mapping, and obtains the real global passable area map and the real global semantic map of the scene; the robot collects the RGB image and the depth image at each position on each path in the scene, performs semantic segmentation and point cloud mapping, and obtains the local passable area map and the local semantic map of each path at each time, and fuses the local passable area map and the local semantic map of each path at each time with the historical global passable area map and the historical global semantic map stored by the robot before sampling the respective path, to obtain the global passable area map and the fused semantic map at each time, and uses the global passable area map and the fused semantic map at each time and the real global semantic map of the scene as training samples to construct a training dataset; the deep neural network model is trained using the training dataset, and a semantic map prediction model is obtained; wherein each point in the semantic map represents the probability of the existence of each object at the point, and each point in the semantic map corresponds to the position of the corresponding point in the passable area map one by one;

[0047] The navigation phase: the robot obtains a real-time local passable area map and a real-time local semantic map, fuses them with the global passable area map and the global semantic map stored by the robot respectively, obtains a real-time global passable area map and a fused semantic map, and inputs them into the semantic map prediction model to obtain a predicted real-time global semantic map; the robots communicate with each other to update their global semantic maps; the robot weights and sums the probability of the existence of a target object at each position in the updated global semantic map of the robot and the uncertainty of the prediction of each position, and selects the point with the highest sum value in the real-time local passable area map as the target position for the next step; each robot goes to the target position and searches for the target object, and continuously updates the global passable area map and the global semantic map during navigation, until one of the robots finds the target object or reaches the upper limit of the number of navigation steps.

[0048] In some embodiments, the training phase is mainly used to obtain a semantic map prediction model to predict the complete semantic map of the scene, and specifically includes the following steps:

[0049] (1-1) In the scene where the robot is located, first traverse each position, use the visual sensor carried by the robot to obtain the RGB image I t and the depth image D t at the position corresponding to time t in the scene, obtain the local passable area map m o by point cloud mapping of the coordinates of each position in the RGB image I t and the depth image D t , and fuse the local passable area map m operform semantic segmentation to obtain a local semantic map m s , and fuse them to obtain a real global passable area map M o and a real global semantic map M s of the entire scene. The dimension of the semantic map is CxHxW, C is the total number of object categories contained in the scene, H and W are the height and width of the semantic map respectively, and each point in the semantic map is a C-dimensional vector Cls∈R C Each element in the vector represents the probability of containing each type of object at that point in the semantic map. For the real global semantic map M s , the value of each element in the vector Cls is 0 or 1. When it is 0, it means that the point in the semantic map does not contain an object of that type, and when it is 1, it means that the point in the semantic map contains an object of that type. It should be noted that each point in the semantic map may contain multiple objects, i.e. multiple elements in the vector may have a value of 1.

[0050] Further, the scene in which the robot is located can be a real scene or a simulated scene. In one specific embodiment of the present disclosure, the scene in which the robot is located is a simulated scene formed using an outdoor park rendered based on Unity Engine 4.

[0051] (1-2) Training dataset construction

[0052] Using the vision sensor carried by the robot, the RGB image I t，i and the depth image D t,i at each time on each path in the scene are obtained, the subscript t represents the time, and the subscript i represents the path number in the scene. The coordinates of each position in the RGB image I t，i and the depth image D t,i are obtained by point cloud mapping to obtain a local passable area map The local passable area map is subjected to semantic segmentation to obtain a local semantic map The local passable area map and the local semantic map are respectively fused with the global passable area map and the global semantic map stored by the robot at the previous time to obtain the global passable area map and the global semantic map at time t and path i. The global passable area map and the global semantic map and the real global semantic map M sAs a training sample, a training data set is constructed, and in this embodiment, the length of each type of map (local passable area map, global passable area map, local semantic map and global semantic map) is H and the width is W.

[0053] (1-3) Semantic map prediction model construction and training

[0054] The U-Net network is selected as the deep neural network model, and the global passable area map in the training data set is taken as the input of the deep neural network model, so that the U-Net can predict the global semantic map of the path i in the scene at time t according to the historical observation. And the global semantic map is taken as the input of the deep neural network model, so that the U-Net can predict the global semantic map of the path i in the scene at time t according to the historical observation. The U-Net is composed of a series of convolutional layers and deconvolutional layers, and each pair of convolutional layers and deconvolutional layers is symmetrical and connected by a skip connection, so as to ensure that there are detailed features of the bottom layer and abstract features of the high layer in each layer of the network. Specifically, after obtaining the point cloud mapped by the RGB image and the depth image, the point cloud is voxelized by a predefined grid size, and each point is mapped to the global semantic map grid. Let the size of the global semantic map after gridization be HxW, where H is the height of the global semantic map after gridization and W is the width of the global semantic map after gridization. For each position l k , k∈1,2, …, HxW, the probability of the object included in this position is represented by a C-dimensional vector Cls k ={Cls k ,j}, C is the total number of object categories in the scene, each element in Cls k represents the probability of the existence of an object of category j at position l k , Cls k,j =1 represents the existence of an object of category j at position l k , and Cls k，j =0 represents the non-existence of an object of category j at position l k . Multiple categories of objects can exist in each position, and the training process of the U-Net network is constrained by a multi-label loss function L, so that the semantic map predicted by the U-Net network is closer to the true global semantic map M s of the entire scene. The multi-label loss function L is as follows:

[0055]

[0056] Where T is the total number of time steps; is an indicator function, and the value of k is 1 only when there is an object of category j at position l , otherwise is 0, the value of the indicator function at each position is determined according to the true global semantic map Ms obtained in step (1-1); the global semantic map at time t predicted by the U-Net network k the probability of the existence of an object of class j at position l.

[0057] The U-Net network is trained for several iterations until a set upper limit is reached, and the training is stopped. The trained U-Net network is used as a semantic map prediction model.

[0058] In some embodiments, the navigation stage mainly completes the selection of sub-targets by each robot, and the environment is continuously explored to finally locate the position of the target object, and specifically includes the following steps:

[0059] (2-1) At time t Before navigation, the robots map the RGB image and the depth image observed by themselves into a passable area map and a semantic map centered on themselves in combination with the semantic segmentation result, and transfer them into real-time local passable area maps and real-time local semantic maps

[0060] (2-2) Each robot merges the real-time local passable area map with the global passable area map stored by itself to obtain a real-time global passable area map of each robot. Specifically, if a certain position in the global passable area map stored by itself is in an unknown state, the value (passable / obstacle) of the corresponding position in the real-time local passable area map is directly assigned, and the value is used as the value of the corresponding position in the real-time global passable area map If the values of the corresponding positions in the local passable area map and the global passable area map stored by itself are inconsistent (which may be caused by errors in the depth image), the position is marked as an obstacle. Thus, the real-time global passable area map of each robot at time t is obtained.

[0061] Meanwhile, each robot merges the real-time local semantic map with the global semantic map stored by itself to obtain a fused global semantic map of each robot.

[0062] (2-3) The real-time global passable area map obtained in step (2-2) and the fused global semantic map are input into the semantic map prediction model. The model first performs a series of convolution operations to downsample the features of the maps, then performs a series of inverse convolution to upsample the features, and finally obtains the predicted Global semantic map of the whole scene at a moment Distribute the global semantic map to the corresponding robot.

[0063] (2-4) Multiple robots communicate to update their own global semantic map, and obtain the updated global semantic map of themselves Specifically: each robot compresses and encodes its own global semantic map and sends it to other robots, and integrates all the global semantic maps received from other robots and fuses them with its own global semantic map. Since each robot has different observation angles and positions of the scene, and the heights and shapes of the heterogeneous robots are different, the semantic segmentation results of the same position may be different, which further leads to the fact that the semantic maps corresponding to the public part are not necessarily completely consistent. Therefore, each robot updates the part of its own global semantic map that has not been observed, and does not update the part that has been observed.

[0064] (2-5) For the updated global semantic map of the robot Each given position l m , m∈1, 2, …H×W, the point in the global semantic map corresponds to a C-dimensional vector Cls m = {Cls m，j}, which represents the probability of the existence of objects of each class at the position l m . By calculating the entropy of the vector Cls m , the uncertainty of the object distribution at the position predicted by the semantic map model can be obtained:

[0065]

[0066] wherein, represents the uncertainty corresponding to the position l m , which is in the form of a matrix.

[0067] After obtaining the uncertainty, the target position can be determined by combining the probability and the uncertainty. Each robot determines its own target position in the following way:

[0068]

[0069] wherein, NR q represents the navigable region of the robot q, which is determined according to the real-time global navigable region map of the robot q ; and represents the category of the target object; represents the position l mthe probability that there is a target object at the position, i.e. the updated global semantic map of the robot the position l m at The value of α is a weighting coefficient, which balances the relationship between the probability and uncertainty. The larger α encourages the model to explore more and eliminate uncertainty; the smaller α encourages the model to search for the target object according to the existing perception.

[0070] (2-6) The robot goes to the target position through path planning and updates the global passable region map and the global semantic map in the manner of steps (2-1) to (2-3) at a set interval. The navigation stops when the target position is reached or the upper limit of the number of navigation steps is reached.

[0071] (2-7) After stopping navigation, the target detector is used to observe whether the target object appears in the field of view. If the target object has been found, the other robots are notified to stop navigation, and the task ends. Otherwise, go to (2-1) to continue searching.

[0072] It can be understood that the multi-robot visual semantic navigation method provided by the first aspect of the present disclosure takes into account the passable region of each robot of the heterogeneous multiple robots, lets each robot establish a semantic map according to its stored observation of the scene, and improves the semantic map through communication with each other. When performing top-level planning, the robot not only considers the probability that there is a target object at each position, but also comprehensively considers the uncertainty predicted for the position, and selects the point with the highest score in the feasible region as the target position for the next step after weighting and summing the probability and the uncertainty. The passable region map and the semantic map are continuously updated during navigation until one of the robots finds the target object. The embodiments of the present disclosure balance the relationship between exploration and utilization, let the robot explore as much as possible in the early stage, and can quickly locate the target object after having certain perception information of the environment. The method can be used for target object finding in indoor and outdoor scenes, and the category of the target object can be any category in the semantic map without additional training.

[0073] The multi-robot visual semantic navigation device provided by the second aspect of the present disclosure comprises:

[0074] The training module is configured to collect RGB images and depth images at each position in the scene, perform semantic segmentation and point cloud mapping on the images, and obtain a real global passable area map and a real global semantic map of the scene; collect RGB images and depth images at each position on each path in the scene, perform semantic segmentation and point cloud mapping on the images, and obtain a local passable area map and a local semantic map of each path at each time; fuse the local passable area map and the local semantic map of each path at each time with the global passable area map and the global semantic map stored by the robot before sampling the respective paths, respectively, to obtain a fused global passable area map and a fused global semantic map at each time; and use the fused global passable area map and the fused global semantic map at each time and the real global semantic map of the scene as training samples to construct a training data set.

[0075] The navigation module is configured to make the robot obtain a real-time local passable area map and a real-time local semantic map, fuse the real-time local passable area map and the real-time local semantic map with the global passable area map and the global semantic map stored by the robot, respectively, to obtain a real-time global passable area map and a fused semantic map, and input the real-time global passable area map and the fused semantic map into the semantic map prediction model to obtain a predicted real-time global semantic map; the plurality of robots communicate with each other to update the global semantic map of each robot; each robot selects a point with the highest weighted sum value in the local passable area map of the robot as a target position in the next step after performing weighted summation on the probability of the existence of a target object at each position in the updated global semantic map of the robot and the uncertainty of the prediction of each position; and each robot goes to the target position and searches for a target object, and stops navigation when one of the robots finds the target object or reaches an upper limit of the number of navigation steps.

[0076] It should be noted that the foregoing embodiment of the method for visual semantic navigation of multiple robots is also applicable to the visual semantic navigation device of multiple robots of the present embodiment, and thus will not be described herein again.

[0077] To implement the above-mentioned embodiments, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to perform the method for visual semantic navigation of multiple robots provided in the first aspect of the present disclosure.

[0078] The following will be described with reference to the accompanying drawings Figure 2The diagram illustrates a structural schematic suitable for implementing the electronic device provided in the third aspect of the present disclosure. The electronic device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs, desktop computers, and servers. Figure 2 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0079] like Figure 2 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 into a random access memory (RAM) 103. The RAM 103 also stores various programs and data required for the operation of the electronic device. The processing unit 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0080] Typically, the following devices can be connected to I / O interface 105: input devices 106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 108 including, for example, magnetic tapes, hard disks, etc.; and communication devices 109. Communication device 109 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 2 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0081] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, this embodiment includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via communication device 109, or installed from storage device 108, or installed from ROM 102. When the computer program is executed by processing device 101, it performs the functions defined above in the methods of embodiments of this disclosure.

[0082] It should be noted that the computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, a RF (radio frequency) or the like, or any suitable combination of the above.

[0083] The computer-readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device.

[0084] The computer-readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the multi-robot visual semantic navigation method described above.

[0085] Computer program code for carrying out operations of the present disclosure can be written in any one or more programming languages or combinations of languages including object or visual programming languages such as Java, Smalltalk, C++, Python, conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0086] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0087] In addition, the terms "first", "second" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0088] Any process or method descriptions or descriptions of the flow diagrams described herein or otherwise described in the specification can be understood as representing the modules, segments, or portions of code that include executable instructions for performing specific logical functions or steps in the processes, and the scope of the preferred embodiments of the present application includes additional implementation in which the functions performed by the illustrated or discussed order are performed in a different order, including substantially simultaneously or in reverse order, as will be understood by those skilled in the art of the embodiments to which the present application belongs.

[0089] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (electrical connections), a portable computer diskette (a magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.

[0090] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0091] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the developed programs can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of steps of the method embodiments.

[0092] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0093] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A multi-robot visual semantic navigation method, characterized in that, include: Training phase: Collect RGB images and depth images at various locations in the scene, perform semantic segmentation and point cloud mapping on them to obtain the scene's real global traversable domain map and real global semantic map; RGB and depth images are collected at each location along each path in the scene. Semantic segmentation and point cloud mapping are performed on these images to obtain local traversable domain maps and local semantic maps for each path at each time step. These local traversable domain maps and local semantic maps for each path at each time step are then fused with the global traversable domain map and global semantic map stored by the robot before sampling its respective path, resulting in fused global traversable domain maps and global semantic maps at each time step. A training dataset is constructed using the fused global traversable domain maps and global semantic maps at each time step, along with the scene's real global semantic map, as training samples. A deep neural network model is trained using this training dataset to obtain a semantic map prediction model. Each point in the semantic map represents the probability of the existence of objects at that point, and each point in the semantic map corresponds one-to-one with the corresponding point in the traversable domain map. Navigation Phase: The robot acquires a real-time local traversable domain map and a real-time local semantic map, and fuses them with its own stored global traversable domain map and global semantic map to obtain a real-time global traversable domain map and a fused semantic map. These are then input into the semantic map prediction model to obtain a predicted real-time global semantic map. Multiple robots communicate to update their own global semantic maps. Each robot calculates a weighted sum of the probability of a target object at each location in its updated global semantic map and the uncertainty of the prediction at each location, and selects the point with the highest weighted sum score in its local traversable domain map as the next target location. Each robot proceeds to the target location and searches for the target object until one robot finds the target object or reaches the navigation step limit, at which point navigation stops. Each robot determines its target position through the following steps. : For the robot's updated global semantic map Each given position in , and These represent the width and height of the meshed semantic map, and the points in the updated global semantic map, respectively. Corresponding to one dimensional vector It indicates position The probability of the existence of various objects is calculated by vectors. The entropy is used to obtain the semantic map prediction model for location. Uncertainty in the distribution of objects: in, Indicates position The corresponding uncertainty; Determining target location by combining probability and uncertainty The calculation formula is as follows: in, This represents the traversable area for robot q, based on the real-time global traversable area map of robot q. Sure; Indicates the category of the target object; Indicates position The probability that the target object exists, i.e., the robot's updated global semantic map. Middle position Place The value; These are the weighting coefficients.

2. The multi-robot visual semantic navigation method according to claim 1, characterized in that, During the navigation phase, the following steps are taken to obtain the true global navigable domain map and the true global semantic map of the scene: Within the robot's environment, each location is traversed, and the robot's onboard vision sensors are used to acquire the RGB image of the location at time t. and depth images RGB image and depth images The coordinates of each location are used to obtain a local traversable area map through point cloud mapping. For the local traversable area map Semantic segmentation is performed to obtain a local semantic map. Map of the local traversable area and the local semantic map The data is then fused separately to obtain a true global traversable area map of the entire scene. and the true global semantic map ; The following steps are used to obtain the fused global traversable domain map and global semantic map for each time point: Using the robot's onboard vision sensors, RGB images of each path in the scene are acquired at each moment. and depth images Subscript Represents the path number in the scene, and assigns each RGB image... and depth images The coordinates of each location are used to obtain a local traversable area map through point cloud mapping. For the local traversable area map Semantic segmentation is performed to obtain a local semantic map. Map of the local traversable area and the local semantic map Compare this to the global traversable region map stored by the robot at the previous time step. and global semantic map To merge and obtain the time Globally accessible area map and fusion semantic map .

3. The multi-robot visual semantic navigation method according to claim 1, characterized in that, The step of training the deep neural network model using the training dataset to obtain the semantic map prediction model specifically includes: U-Net is used as the backbone network of the deep neural network model. Each training sample from the training dataset is input into the deep neural network model for training, making the global semantic map predicted by the deep neural network model more closely resemble the true global semantic map of the entire scene. The semantic map has the following dimensions: Each point in the semantic map is represented by a... dimensional vector express, This represents the total number of object categories in the scene. Each element in Representative position The probability that an object of category j exists at a given location. , and These represent the width and height of the meshed semantic map, respectively, allowing for multiple categories of objects at each location; Using a multi-label loss function The training process of the deep neural network model is constrained by the following expression: Where T is the total number of time steps; For indicator functions, only at position Existence category is When objects The value is 1, otherwise The value is 0, according to the real global semantic map. Determine the value of the indicator function at each location; The time predicted by the deep neural network model Location in global semantic map The types that exist there are The probability of an object; Training stops after reaching a set maximum number of iterations, resulting in the semantic map prediction model.

4. The multi-robot visual semantic navigation method according to claim 1, characterized in that, The real-time local traversable domain map and the real-time local semantic map are obtained by following these steps: At any moment Before navigation, multiple robots combine their observed RGB and depth images with semantic segmentation results to map a feasible region map and a semantic map centered on themselves. They then combine these with their own coordinates to transform the map into a real-time local feasible region map in the world coordinate system. and real-time local semantic map .

5. The multi-robot visual semantic navigation method according to claim 1, characterized in that, During the navigation phase, each robot will map the local feasible region in real time. When merging with its own stored global feasible region map, if a certain location in the stored global feasible region map is unknown, the value of that location in the stored global feasible region map is used to adjust the real-time local feasible region map. The values ​​at the corresponding positions in the map are directly assigned; if the local feasible region map... If the value at a location is inconsistent with the value at the corresponding location in the globally accessible domain map that the location is stored, then that location is marked as obstructed, thus obtaining the respective real-time globally accessible domain map. .

6. The multi-robot visual semantic navigation method according to claim 1, characterized in that, During the navigation phase, the multiple robots communicate with each other to update their global semantic maps, specifically including: Each robot compresses and encodes its own global semantic map before sending it to other robots. It then integrates the global semantic maps received from all other robots with its own, fusing them together. During fusion, each robot only updates the parts of its own global semantic map that are not present in its own map, leaving the parts it has already observed unupdated, thus obtaining an updated global semantic map. .

7. The multi-robot visual semantic navigation method according to any one of claims 1 to 6, characterized in that, Multiple robots are heterogeneous robots.

8. A multi-robot visual semantic navigation device, characterized in that, include: The training module is configured to collect RGB and depth images at various locations in the scene, perform semantic segmentation and point cloud mapping on them, and obtain the scene’s real global traversable domain map and real global semantic map. RGB and depth images are collected at each location along each path in the scene. Semantic segmentation and point cloud mapping are performed on these images to obtain local traversable domain maps and local semantic maps for each path at each time step. These local traversable domain maps and local semantic maps for each path at each time step are then fused with the global traversable domain map and global semantic map stored by the robot before sampling its respective path, resulting in fused global traversable domain maps and global semantic maps at each time step. A training dataset is constructed using the fused global traversable domain maps and global semantic maps at each time step, along with the scene's real global semantic map, as training samples. A deep neural network model is trained using this training dataset to obtain a semantic map prediction model. Each point in the semantic map represents the probability of the existence of objects at that point, and each point in the semantic map corresponds one-to-one with the corresponding point in the traversable domain map. The navigation module is configured to enable the robot to acquire a real-time local traversable domain map and a real-time local semantic map, and to fuse these with its own stored global traversable domain map and global semantic map to obtain a real-time global traversable domain map and a fused semantic map. These are then input into the semantic map prediction model to obtain a predicted real-time global semantic map. Multiple robots communicate to update their own global semantic maps. Each robot calculates a weighted sum of the probability of a target object at each location in its updated global semantic map and the uncertainty of the prediction at each location, and selects the point with the highest weighted sum score in its local traversable domain map as the next target location. Each robot proceeds to the target location and searches for the target object until one robot finds the target object or reaches the navigation step limit, at which point navigation stops. Each robot determines its target position through the following steps. : For the robot's updated global semantic map Each given position in , and These represent the width and height of the meshed semantic map, and the points in the updated global semantic map, respectively. Corresponding to one dimensional vector It indicates position The probability of the existence of various objects is calculated by vectors. The entropy is used to obtain the semantic map prediction model for location. Uncertainty in the distribution of objects: in, Indicates position The corresponding uncertainty; Determining target location by combining probability and uncertainty The calculation formula is as follows: in, This represents the traversable area for robot q, based on the real-time global traversable area map of robot q. Sure; Indicates the category of the target object; Indicates position The probability that the target object exists, i.e., the robot's updated global semantic map. Middle position Place The value; These are the weighting coefficients.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the multi-robot visual semantic navigation method according to any one of claims 1 to 7.

Images