Multi-robot visual semantic navigation method and device and storage medium

Abstract

The invention relates to a multi-robot visual semantic navigation method, a plurality of robots have respective passable areas, and each robot establishes a semantic map according to past scene observation and perfects the semantic map through communication among the robots. When top planning is carried out, the robot not only considers the probability that a target object exists at each position, but also comprehensively considers the uncertainty of position prediction, and after weighted summation is carried out on the probability and the uncertainty, a point with the highest score in a feasible region is selected as the target position of the next step. And continuously updating the feasible region map and the semantic map in the navigation process until one of the robots finds the target object. According to the method and the device, the relationship between exploration and utilization is balanced, the robot can perform exploration as much as possible at the initial stage, the target object can be positioned more quickly after certain perception information is provided for the environment, the method and the device are used for searching the target object in each scene, and the target object is any one type in a semantic map without additional training.

Description

Technical Field

[0001] The present disclosure belongs to the field of robot navigation technology, and in particular relates to a multi-robot visual semantic navigation method, device and storage medium. Background Art

[0002] Semantic segmentation refers to dividing the objects in a given image according to their semantic categories. Semantic segmentation can extract the semantic information corresponding to each position in the image and has extensive and important applications in many downstream tasks such as target positioning, image understanding, and human-computer interaction.

[0003] In recent years, with the development of deep learning, the accuracy of semantic segmentation has been greatly improved, and the semantic segmentation model can accurately give the semantic category to which each pixel in the image belongs. By mapping pixels in combination with the depth map, semantic segmentation can be converted into a semantic map, which further helps the robot understand the scene and navigate the target object. However, the current target object navigation is often learned directly for a given target object label type, which has a high correlation with the target object label and lacks exploration of the scene. This feature also leads to a significant decrease in the performance of the robot in new scenes or when searching for new target objects. If the robot can give priority to exploring the scene when it has less perceptual information about the scene, rather than navigating directly to the location of the target object it believes to be based solely on the knowledge learned during training, it will improve the robot's generalization ability in new scenes and its ability to adapt to new types of target objects.

[0004] The current target object navigation method mainly encodes the RGB observation image based on the convolutional neural network (CNN) to obtain the RGB observation vector, and then encodes the historical RGB observation vector through the long short-term memory neural network (LSTM), and predicts the robot's next action through the hidden state output by the LSTM. When performing target navigation, weights can also be assigned to each area of ​​the RGB image after passing through the convolutional network according to the given target object category, and these areas are weighted averaged to help the network pay more attention to those areas related to the target object. Some navigation methods also introduce additional target detectors or semantic segmentation models to help robots learn more robust navigation models through more diverse information inputs. However, all these models only focus on how to search for the location of the target object as quickly as possible when testing. In an unseen scene, the display and distribution relationship of objects may be very different. When there is less perceptual information about the environment at the beginning of the search, directly using the knowledge during training to search is likely to go in the wrong direction, ultimately leading to the failure of the search task. This also leads to the poor generalization ability of the current target object navigation algorithm in new scenes. At the same time, most target object navigation methods are for single robots. Even if there are some works for multiple robots, they do not take into account that multiple robots may be heterogeneous and have differences in their motion capabilities and traversable areas. Summary of the invention

[0005] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the embodiments of the present disclosure provide a multi-robot visual semantic navigation method, device and storage medium. The present disclosure first takes the heterogeneity of each robot into consideration and plans for each robot based on its motion capability and traversable area. Furthermore, the present disclosure balances the relationship between exploration and exploitation, allowing the robot to perform some necessary exploration in the early stages of navigation, and as the number of navigation steps increases, it can use the perception of the scene during the exploration process to locate the target object faster.

[0007] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0008] The first aspect of the present disclosure provides a multi-robot visual semantic navigation method, comprising:

[0009] Training phase: collecting RGB images and depth images at each position in the scene, performing semantic segmentation and point cloud mapping on them, and obtaining a real global traversable domain map and a real global semantic map of the scene; collecting RGB images and depth images at each position on each path in the scene, performing semantic segmentation and point cloud mapping on them, and obtaining a local traversable domain map and a local semantic map of each path at each moment; fusing the local traversable domain map and the local semantic map of each path at each moment with the global traversable domain map and the global semantic map stored by the robot before sampling each path, respectively, to obtain a fused global traversable domain map and a global semantic map at each moment; using the fused global traversable domain map and the global semantic map at each moment and the real global semantic map of the scene as training samples to construct a training data set; using the training data set to train a deep neural network model to obtain a semantic map prediction model; wherein each point in the semantic map represents the probability of the existence of each object at that point, and each point in the semantic map corresponds one-to-one to the position of the corresponding point in the traversable domain map;

[0010] Navigation phase: The robot obtains a real-time local traversable domain map and a real-time local semantic map, and fuses them with its own stored global traversable domain map and global semantic map to obtain a real-time global traversable domain map and a fused semantic map, and inputs them into the semantic map prediction model to obtain a predicted real-time global semantic map; multiple robots communicate to update their own global semantic maps; each robot weightedly sums the probability of the existence of a target object at each position in its updated global semantic map and the uncertainty of the prediction for each position, and then selects the point with the highest weighted sum score in its own local feasible domain map as the next target position; each robot goes to the target position and searches for the target object until one of the robots finds the target object or stops navigating when the upper limit of navigation steps is reached.

[0011] In some embodiments, during the navigation phase, the real global traversable domain map and the real global semantic map of the scene are obtained according to the following steps:

[0012] In the scene where the robot is located, traverse each position and use the visual sensor carried by the robot to collect the RGB image I at the position corresponding to time t in the scene t and the depth image D t , the RGB image I t and the depth image D t The coordinates of each position in the local traversable area map m is obtained through point cloud mapping o , for the local traversable area map m o Perform semantic segmentation to obtain a local semantic map m s , the local traversable area map m oand the local semantic map m s The real global traversable domain map M of the entire scene is obtained by fusion. o and the true global semantic map M s ;

[0013] The global traversable domain map and global semantic map fused at each moment are obtained by following the steps below:

[0014] Use the robot's visual sensor to obtain the RGB image I at each moment on each path in the scene t，i and the depth image D t，i , the subscript i represents the path number in the scene, and each RGB image I t，i and the depth image D t，i The coordinates of each position in the local traversable area map are obtained through point cloud mapping The local traversable area map Perform semantic segmentation to obtain a local semantic map The local traversable area map and the local semantic map The global traversable area map stored by the robot at the last moment and global semantic map Fusion is performed to obtain the global traversable domain map at time t and fused semantic maps

[0015] In some embodiments, the using the training data set to train the deep neural network model to obtain the semantic map prediction model specifically includes:

[0016] U-Net is used as the backbone network of the deep neural network model, and each training sample in the training data set is input into the deep neural network model for training, so that the global semantic map predicted by the deep neural network model is closer to the real global semantic map M of the entire scene. s The dimension of the semantic map is C×H×W, and each point in the semantic map is represented by a C-dimensional vector Cls k = {Cls k，j} indicates that C is the total number of object categories in the scene, Cls k Each element in Cls k，j Representative position k The probability that there is an object of category j at k , k∈1,2,…,H×W, H and W are the width and height of the gridded semantic map, respectively. Multiple categories of objects are allowed to exist at each position;

[0017] The multi-label loss function L is used to constrain the training process of the deep neural network model, and the expression is as follows:

[0018]

[0019] Where T is the total number of time steps; is the indicator function. Only at position l k When there is an object of category j The value of is 1, otherwise The value of is 0, and the value of each position of the indicator function is determined according to the real global semantic map Ms; is the position l in the global semantic map at time t predicted by the deep neural network model k The probability that there is an object of type j at ;

[0020] When the training reaches the set upper limit number of times, it stops and the semantic map prediction model is obtained.

[0021] In some embodiments, the real-time local traversable domain map and the real-time local semantic map are obtained according to the following steps:

[0022] At the moment Before navigating, multiple robots first map the RGB images and depth images they observe into a feasible domain map and a semantic map centered on themselves in combination with the semantic segmentation results, and then transfer them into a real-time local feasible domain map in the world coordinate system in combination with their own coordinates. and real-time local semantic maps

[0023] In some embodiments, during the navigation phase, each robot will map the local feasible area in real time. When merging with the global feasible domain map stored in itself, if a certain position in the global feasible domain map stored in itself is in an unknown state, the value of the certain position in the global feasible domain map stored in itself is used to merge the real-time local feasible domain map. Directly assign the value of the corresponding position in the local feasible region map If the value of the corresponding position in the global traversable area map stored in the system is inconsistent, the position is marked as having an obstacle, thereby obtaining the respective real-time global traversable area maps.

[0024] In some embodiments, during the navigation phase, the multiple robots communicate with each other to update their own global semantic maps, specifically including:

[0025] Each robot compresses and encodes its own global semantic map and sends it to other robots. It also integrates the global semantic maps received from all other robots and fuses them with its own global semantic map. During the fusion, each robot only updates the parts that are not in its own global semantic map, and does not update the parts that it has observed, thereby obtaining an updated global semantic map of its own.

[0026] In some embodiments, each robot determines its target position by the following steps:

[0027] For the robot's updated global semantic map For each given position l in m , m∈1,2,…H×W, H and W are the width and height of the gridded semantic map, respectively, and the points in the updated global semantic map Corresponding to a C-dimensional vector Cls m = {Cls m，j}, which indicates the position l m The probability of the existence of various objects at each location is calculated by calculating the vector Cls m The entropy of the semantic map prediction model is obtained for the position l m Uncertainty in the distribution of objects at:

[0028]

[0029] in, Indicates the position m corresponding uncertainty;

[0030] Determine target location by combining probability and uncertainty The calculation formula is as follows:

[0031]

[0032] Among them, NR q Indicates the traversable area of ​​robot q, based on the real-time global traversable area map of robot q Sure; Indicates the category of the target object; Indicates the position m The probability that the target object exists on the robot, that is, the robot’s updated global semantic map Middle position m Department The value of; α is the weighting coefficient.

[0033] In some embodiments, the plurality of robots are heterogeneous robots.

[0034] A second aspect of the present disclosure provides a multi-robot visual semantic navigation device, comprising:

[0035] The training module is configured to collect RGB images and depth images at each position in the scene, perform semantic segmentation and point cloud mapping on them, and obtain a real global traversable domain map and a real global semantic map of the scene; collect RGB images and depth images at each position on each path in the scene, perform semantic segmentation and point cloud mapping on them, and obtain a local traversable domain map and a local semantic map of each path at each moment, and fuse the local traversable domain map and the local semantic map of each path at each moment with the global traversable domain map and the global semantic map stored by the robot before sampling the respective paths, respectively, to obtain a fused global traversable domain map and a global semantic map at each moment, and use the fused global traversable domain map and the global semantic map at each moment and the real global semantic map of the scene as training samples to construct a training data set; use the training data set to train a deep neural network model to obtain a semantic map prediction model; wherein each point in the semantic map represents the probability of the existence of each object at the point, and each point in the semantic map corresponds one-to-one to the position of the corresponding point in the traversable domain map;

[0036] The navigation module is configured to enable the robot to obtain a real-time local traversable domain map and a real-time local semantic map, and fuse them with its own stored global traversable domain map and global semantic map respectively to obtain a real-time global traversable domain map and a fused semantic map, and input them into the semantic map prediction model to obtain a predicted real-time global semantic map; multiple robots communicate to update their own global semantic maps; each robot performs a weighted summation of the probability of the existence of a target object at each position in its own updated global semantic map and the uncertainty of the prediction of each position, and then selects the point with the highest weighted sum score in its own local feasible domain map as the next target position; each robot goes to the target position and searches for the target object until one of the robots finds the target object or stops navigating when the upper limit of the navigation steps is reached.

[0037] A third aspect of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multi-robot visual semantic navigation method according to any embodiment of the first aspect of the present disclosure.

[0038] The characteristics and beneficial effects of the present disclosure are:

[0039] The present invention uses a semantic segmentation model based on deep learning to perform semantic segmentation on RGB images, and maps and grids the pixels through the depth map to obtain a semantic map. The model uses the correlation between various objects to predict a complete semantic map based on the semantic map it has seen, and takes the probability of the target object existing at a certain position in the predicted semantic map as the confidence, and the probability of each object at that position as the uncertainty. The weighted sum of the two helps the intelligent agent balance exploration and search using perceptual information, thereby improving the robot's generalization ability in new scenarios.

[0040] The disclosure can be used in the field of multi-robot target navigation, allowing multiple heterogeneous robots to explore and search the scene together from their respective positions, and to search for the location of the target object as quickly as possible while establishing a scene semantic map that is as complete and accurate as possible. The disclosure balances the relationship between exploration and utilization, allowing the robot to conduct as much exploration as possible in the early stage, and to locate the target object more quickly after having certain perceptual information about the environment. It can be used to search for target objects in various indoor and outdoor scenes, and the category of the target object can be any category in the semantic map, without the need for additional training. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the multi-robot visual semantic navigation method provided by the embodiment of the first aspect of the present disclosure.

[0042] Figure 2 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the third aspect of the present disclosure. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] On the contrary, the present application covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present application as defined by the claims. Further, in order to make the public have a better understanding of the present application, some specific details are described in detail in the detailed description of the present application below. Those skilled in the art can fully understand the present application without the description of these details.

[0045] See also Figure 1 The multi-robot visual semantic navigation method provided by the first aspect of the present disclosure includes:

[0046] Training phase: the robot traverses each position in the scene where it is located, collects RGB images and depth images at each position, performs semantic segmentation and point cloud mapping on them, and obtains the real global traversable domain map and real global semantic map of the scene; the robot collects RGB images and depth images at each position on each path in the scene, performs semantic segmentation and point cloud mapping on them, and obtains the local traversable domain map and local semantic map of each path at each moment, and fuses the local traversable domain map and local semantic map of each path at each moment with the historical global traversable domain map and historical global semantic map stored by the robot before sampling each path, respectively, to obtain the global traversable domain map and fused semantic map at each moment, and uses the global traversable domain map and fused semantic map at each moment and the real global semantic map of the scene as training samples to construct a training data set; uses the training data set to train the deep neural network model to obtain a semantic map prediction model; wherein each point in the semantic map represents the probability of the existence of each object at that point, and each point in the semantic map corresponds one-to-one to the position of the corresponding point in the traversable domain map;

[0047] Navigation phase: The robot obtains a real-time local traversable domain map and a real-time local semantic map, and fuses them with its own stored global traversable domain map and global semantic map to obtain a real-time global traversable domain map and a fused semantic map, and inputs them into the semantic map prediction model to obtain a predicted real-time global semantic map; multiple robots communicate to update their own global semantic maps; the robot weightedly sums the probability of the existence of a target object at each position in its updated global semantic map and the uncertainty of the prediction for each position, and the robot selects the point with the highest sum in the real-time local traversable domain map as the target position for the next step; each robot goes to the target position and searches for the target object, and continuously updates the global traversable domain map and the global semantic map during the navigation process until one of the robots finds the target object or stops navigating when the upper limit of the navigation steps is reached.

[0048] In some embodiments, the training phase is mainly used to obtain a semantic map prediction model to predict a complete semantic map of a scene, and specifically includes the following steps:

[0049] (1-1) In the scene where the robot is located, first traverse each position and use the robot's visual sensor to obtain the RGB image I corresponding to the position at time t in the scene. t and the depth image D t , the RGB image I t and the depth image D t The coordinates of each position in the local traversable area map m is obtained through point cloud mapping o , for this local traversable area map m oPerform semantic segmentation to obtain a local semantic map m s , and fuse them to obtain the real global traversable domain map M of the entire scene o and the true global semantic map M s The dimension of the semantic map is C×H×W, where C is the total number of object categories in the scene, H and W are the height and width of the semantic map respectively, and each point in the semantic map is a C-dimensional vector Cls∈R C , each element in the vector represents the probability of containing various types of objects at that point in the semantic map. For the real global semantic map M s , the value of each element in the vector Cls is 0 or 1. When it is 0, it means that the point in the semantic map does not contain a certain type of object. When it is 1, it means that the point in the semantic map contains a certain type of object. It should be noted that each point in the semantic map may contain multiple objects, that is, there may be multiple elements in the vector with a value of 1.

[0050] Furthermore, the scene where the robot is located can be a real scene or a simulated scene. In a specific embodiment of the present disclosure, the scene where the robot is located is a simulated scene formed by an outdoor park rendered based on Unity Engine 4.

[0051] (1-2) Construction of training dataset

[0052] Use the robot's visual sensor to obtain the RGB image I at each moment on each path in the scene t，i and the depth image D t,i , the subscript t represents the time, the subscript i represents the path number in the scene, and each RGB image I t，i and the depth image D t,i The coordinates of each position in the local traversable area map are obtained through point cloud mapping Map of the local traversable area Perform semantic segmentation to obtain a local semantic map Local traversable area map and local semantic maps The global traversable area map stored by the robot at the last moment and global semantic map Fusion is performed to obtain the global traversable domain map corresponding to time t and path i and global semantic map Using the global traversable area map and global semantic map And the true global semantic map M of the entire scene sA training data set is constructed as training samples. In this embodiment, the length of each type of map (local drivable area map, global drivable area map, local semantic map and global semantic map) is H and the width is W.

[0053] (1-3) Construction and training of semantic map prediction model

[0054] Select U-Net network as the deep neural network model and map the global traversable domain in the training dataset. and global semantic map As the input of the deep neural network model, let U-Net predict the global semantic map of path i in the scene at time t based on historical observations U-Net consists of a series of convolutional layers and deconvolutional layers, and each pair of convolutional layers and deconvolutional layers are symmetrical and connected by skip connections, so as to ensure that each layer of the network has both underlying detailed features and high-level abstract features. Specifically, after obtaining the point cloud mapped by the RGB image and the depth image, the point cloud is voxelized by a predefined grid size, and each point is mapped to the global semantic map grid. Assume that the size of the global semantic map after gridding is H×W, where H is the height of the global semantic map after gridding, and W is the width of the global semantic map after gridding. For each position l k , k∈1,2,…,H×W, the probability of the object included in this position is represented by a C-dimensional vector Cls k = {Cls k ,j}, C is the total number of object categories in the scene, Cls k Each element in represents the position l k The probability that there is an object of category j at k,j =1 indicates position l k There are objects of category j on Cls k，j =0 means position l k There is no object of category j on the scene, and there can be objects of multiple categories at each position. The multi-label loss function L is used to constrain the training process of the U-Net network, making the semantic map predicted by the U-Net network closer to the real global semantic map M of the entire scene. s , the multi-label loss function L formula is as follows:

[0055]

[0056] Where T is the total number of time steps; is the indicator function. Only at position l k When there is an object of category j The value of is 1, otherwise The value of is 0, and the value of each position of the indicator function is determined according to the real global semantic map Ms obtained in step (1-1); is the position l in the global semantic map at time t predicted by the U-Net network k The probability that there is an object of type j at .

[0057] The training is iterated several times until the set upper limit is reached, and the trained U-Net network is obtained, which is used as the semantic map prediction model.

[0058] In some embodiments, the navigation phase mainly completes each robot selecting its own sub-goal, continuously exploring the environment, and finally locating the position of the target object, which specifically includes the following steps:

[0059] (2-1) At the time Before navigating, multiple robots first map the RGB images and depth images they observe into a traversable area map and a semantic map centered on themselves, and then transfer them into a real-time local traversable area map in the world coordinate system based on their own coordinates. and real-time local semantic maps

[0060] (2-2) Each robot will map the local traversable area in real time Merge with the global traversable domain map stored in itself to obtain their own real-time global traversable domain map Specifically: If a location in the stored global traversable domain map is unknown, the corresponding value (passable / obstructed) in the real-time local traversable domain map is directly assigned, and this value is used as the real-time global traversable domain map. If the local traversable area map and the corresponding value in the global traversable area map stored in the local traversable area map are inconsistent (which may be caused by the error caused by the depth image), the position is marked as having an obstacle. Real-time global traversable area map of each robot at any time

[0061] At the same time, each robot will use the local semantic map in real time Merge with its own stored global semantic map to obtain their respective fused global semantic maps.

[0062] (2-3) The real-time global traversable area map obtained in step (2-2) The fused global semantic map is fed into the semantic map prediction model, which first downsamples the map features through a series of convolution operations, then upsamples through a series of inverse convolutions, and finally obtains the predicted Global semantic map of the entire scene at all times The global semantic map is sent to the corresponding robot.

[0063] (2-4) Multiple robots communicate with each other to update their own global semantic maps and obtain the updated global semantic maps. Specifically: Each robot compresses and encodes its own global semantic map and sends it to other robots, and integrates the global semantic maps received from all other robots and merges them with its own global semantic map. Since each robot observes the scene from different angles and positions, and the heights and shapes of the heterogeneous robots themselves are different, the semantic segmentation results of the same position may be different, which further leads to the semantic maps corresponding to the common parts not necessarily being completely consistent. Therefore, each robot updates the parts that are not in its own global semantic map, and does not update the parts that it has already observed.

[0064] (2-5) For the robot’s updated global semantic map For each given position l in m , m∈1,2,…H×W, a point in the global semantic map Corresponding to a C-dimensional vector Cls m = {Cls m，j}, which indicates that the position l m The probability of the existence of various objects at each location. By calculating the vector Cls m The entropy of the semantic map prediction model can be used to obtain the uncertainty of the distribution of objects at this location:

[0065]

[0066] in, Indicates the position m The corresponding uncertainty is in matrix form.

[0067] After obtaining the uncertainty, the target position can be determined by combining probability and uncertainty. Each robot determines its target position in the following way:

[0068]

[0069] Among them, NR q Indicates the navigable region of robot q, based on the real-time global navigable region map of robot q Sure; Indicates the category of the target object; Indicates the position mThe probability that the target object exists on the robot, that is, the robot’s updated global semantic map Middle position m Department α is a weighting coefficient that reconciles the relationship between probability and uncertainty. The larger α is, the more it encourages the model to explore more and eliminate uncertainty; the smaller α is, the more it encourages the model to search for the target object based on the existing perception.

[0070] (2-6) The robot moves to the target location through path planning The global traversable area map and the global semantic map are updated at set intervals in the manner of steps (2-1) to (2-3). Navigation stops after reaching the target location or reaching the upper limit of the navigation step number.

[0071] (2-7) After stopping navigation, use the target detector to observe whether the target object appears in the field of view. If the target object has been found, notify other robots to stop navigation and the task is completed. Otherwise, go to (2-1) to continue searching.

[0072] It is understandable that the multi-robot visual semantic navigation method provided by the embodiment of the first aspect of the present disclosure takes into account the traversable areas of multiple heterogeneous robots, allows each robot to establish a semantic map based on its stored observations of the scene, and improves the semantic map through communication between each other. When performing top-level planning, the robot not only considers the probability of the existence of the target object at each position, but also comprehensively considers the uncertainty of the prediction of the position, and selects the point with the highest score in the feasible domain as the target position for the next step after weighted summation of the probability and uncertainty, and continuously updates the traversable domain map and the semantic map during the navigation process until one of the robots finds the target object and stops. The embodiment of the present disclosure balances the relationship between exploration and utilization, allowing the robot to explore as much as possible in the early stage, and can locate the target object faster after having certain perceptual information about the environment. It can be used to find target objects in various indoor and outdoor scenes, and the category of the target object can be any category in the semantic map, without the need for additional training.

[0073] The multi-robot visual semantic navigation device provided by the second embodiment of the present disclosure includes:

[0074] The training module is configured to collect RGB images and depth images at each position in the scene, perform semantic segmentation and point cloud mapping on them, and obtain a real global traversable domain map and a real global semantic map of the scene; collect RGB images and depth images at each position on each path in the scene, perform semantic segmentation and point cloud mapping on them, and obtain a local traversable domain map and a local semantic map of each path at each moment, and fuse the local traversable domain map and the local semantic map of each path at each moment with the global traversable domain map and the global semantic map stored by the robot before sampling the respective paths, respectively, to obtain a fused global traversable domain map and a global semantic map at each moment, and use the fused global traversable domain map and the global semantic map at each moment and the real global semantic map of the scene as training samples to construct a training data set; use the training data set to train a deep neural network model to obtain a semantic map prediction model; wherein each point in the semantic map represents the probability of the existence of each object at the point, and each point in the semantic map corresponds one-to-one to the position of the corresponding point in the traversable domain map;

[0075] The navigation module is configured to enable the robot to obtain a real-time local traversable domain map and a real-time local semantic map, and fuse them with its own stored global traversable domain map and global semantic map respectively to obtain a real-time global traversable domain map and a fused semantic map, and input them into the semantic map prediction model to obtain a predicted real-time global semantic map; multiple robots communicate to update their own global semantic maps; each robot performs a weighted summation of the probability of the existence of a target object at each position in its own updated global semantic map and the uncertainty of the prediction of each position, and then selects the point with the highest weighted sum score in its own local feasible domain map as the next target position; each robot goes to the target position and searches for the target object until one of the robots finds the target object or stops navigating when the upper limit of the navigation steps is reached.

[0076] It should be noted that the aforementioned explanation of the embodiment of a multi-robot visual semantic navigation method is also applicable to a multi-robot visual semantic navigation device of this embodiment, and will not be repeated here.

[0077] In order to implement the above embodiments, the embodiments of the present disclosure also propose a computer-readable storage medium, on which a computer program is stored. The program is executed by a processor to execute the multi-robot visual semantic navigation method provided by the embodiment of the first aspect of the present disclosure.

[0078] Reference below Figure 2, which shows a schematic diagram of the structure of an electronic device suitable for implementing the third aspect of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, servers, etc. Figure 2 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0079] like Figure 2 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 to a random access memory (RAM) 103. In the RAM 103, various programs and data required for the operation of the electronic device are also stored. The processing device 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0080] Typically, the following devices may be connected to the I / O interface 105: an input device 106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; an output device 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 109. The communication device 109 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 2 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0081] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present embodiment includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 109, or installed from the storage device 108, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0082] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0083] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0084] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the multi-robot visual semantic navigation method.

[0085] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, python, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0086] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0087] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0088] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0089] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0090] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0091] A person of ordinary skill in the art may understand that all or part of the steps carried out in the above-mentioned embodiment method may be completed by instructing the relevant hardware through a program, and the developed program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0092] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0093] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A multi-robot visual semantic navigation method, characterized in that: include: Training phase: Collect RGB images and depth images at various locations in the scene, perform semantic segmentation and point cloud mapping on them, and obtain the real global traversable domain map and real global semantic map of the scene; Collect RGB images and depth images at each position on each path in the scene, perform semantic segmentation and point cloud mapping on them, obtain the local traversable domain map and local semantic map of each path at each moment, fuse the local traversable domain map and local semantic map of each path at each moment with the global traversable domain map and global semantic map stored by the robot before sampling the respective paths, obtain the fused global traversable domain map and global semantic map at each moment, use the fused global traversable domain map and global semantic map at each moment and the real global semantic map of the scene as training samples to construct a training data set; use the training data set to train a deep neural network model to obtain a semantic map prediction model; wherein each point in the semantic map represents the probability of the existence of each object at that point, and each point in the semantic map corresponds one-to-one to the position of the corresponding point in the traversable domain map; Navigation phase: The robot obtains a real-time local traversable domain map and a real-time local semantic map, and fuses them with its own stored global traversable domain map and global semantic map to obtain a real-time global traversable domain map and a fused semantic map, and inputs them into the semantic map prediction model to obtain a predicted real-time global semantic map; Multiple robots communicate with each other to update their own global semantic maps; each robot takes a weighted sum of the probability of the existence of a target object at each position in its updated global semantic map and the uncertainty of the prediction of each position, and then selects the point with the highest weighted sum score in its local feasible domain map as the target position for the next step; each robot goes to the target position and searches for the target object until one of the robots finds the target object or stops navigating when the upper limit of the navigation steps is reached.

2. The multi-robot visual semantic navigation method according to claim 1, characterized in that: In the navigation phase, the real global traversable domain map and the real global semantic map of the scene are obtained by following the steps below: In the scene where the robot is located, traverse each position and use the visual sensor carried by the robot to collect the RGB image I at the position corresponding to time t in the scene t and the depth image D t , the RGB image I t and the depth image D t The coordinates of each position in the local traversable area map m is obtained through point cloud mapping o , for the local traversable area map m o Perform semantic segmentation to obtain a local semantic map m s , the local traversable area map m o and the local semantic map m s The real global traversable domain map M of the entire scene is obtained by fusion. o and the true global semantic map M s ; The global traversable domain map and global semantic map fused at each moment are obtained by following the steps below: Use the robot's visual sensor to obtain the RGB image I at each moment on each path in the scene t,i and the depth image D t,i , the subscript i represents the path number in the scene, and each RGB image I t,i and the depth image D t,u The coordinates of each position in the local traversable area map are obtained through point cloud mapping The local traversable area map Perform semantic segmentation to obtain a local semantic map The local traversable area map and the local semantic map The global traversable area map stored by the robot at the last moment and global semantic map Fusion is performed to obtain the global traversable domain map at time t and fused semantic maps 3. The multi-robot visual semantic navigation method according to claim 1, characterized in that: The using the training data set to train the deep neural network model to obtain a semantic map prediction model specifically includes: U-Net is used as the backbone network of the deep neural network model, and each training sample in the training data set is input into the deep neural network model for training, so that the global semantic map predicted by the deep neural network model is closer to the real global semantic map M of the entire scene. s The dimension of the semantic map is C×H×W, and each point in the semantic map is represented by a C-dimensional vector Cls k = {Cls k,j } indicates that C is the total number of object categories in the scene, Cls k Each element in Cls k,j Representative position k The probability that there is an object of category j at k ,k∈1,2,…,H×W, H and W are the width and height of the gridded semantic map, respectively. Multiple categories of objects are allowed to exist at each position; The multi-label loss function L is used to constrain the training process of the deep neural network model, and the expression is as follows: Where T is the total number of time steps; is the indicator function. Only at position l k When there is an object of category j The value of is 1, otherwise The value of is 0, according to the real global semantic map M s Determine the value of the indicator function at each position; is the position l in the global semantic map at time t predicted by the deep neural network model k The probability that there is an object of type j at ; When the training reaches the set upper limit number of times, it stops and the semantic map prediction model is obtained.

4. The multi-robot visual semantic navigation method according to claim 1, characterized in that: The real-time local traversable domain map and the real-time local semantic map are obtained by following the steps below: At the moment Before navigating, multiple robots first map the RGB images and depth images they observe into a feasible domain map and a semantic map centered on themselves in combination with the semantic segmentation results, and then transfer them into a real-time local feasible domain map in the world coordinate system in combination with their own coordinates. and real-time local semantic maps 5. The multi-robot visual semantic navigation method according to claim 1, characterized in that: During the navigation phase, each robot will map the local feasible area in real time. When merging with the global feasible domain map stored in itself, if a certain position in the global feasible domain map stored in itself is in an unknown state, the value of the certain position in the global feasible domain map stored in itself is used to merge the real-time local feasible domain map. Directly assign the value of the corresponding position in the local feasible region map If the value of the corresponding position in the global traversable area map stored in the system is inconsistent, the position is marked as having an obstacle, thereby obtaining the respective real-time global traversable area maps.

6. The multi-robot visual semantic navigation method according to claim 1, characterized in that: In the navigation phase, the multiple robots communicate with each other to update their own global semantic maps, which specifically includes: Each robot compresses and encodes its own global semantic map and sends it to other robots. It also integrates the global semantic maps received from all other robots and fuses them with its own global semantic map. During the fusion, each robot only updates the parts that are not in its own global semantic map, and does not update the parts that it has observed, thereby obtaining an updated global semantic map of its own.

7. The multi-robot visual semantic navigation method according to claim 1, characterized in that: Each robot determines its target position by the following steps: For the robot's updated global semantic map For each given position l in m ,m∈1,2,…H×W, H and W are the width and height of the gridded semantic map, respectively. The points in the updated global semantic map Corresponding to a C-dimensional vector Cls m = {Cls m,j }, which indicates the position l m The probability of the existence of various objects at each location is calculated by calculating the vector Cls m The entropy of the semantic map prediction model is obtained for the position l m Uncertainty in the distribution of objects at: in, Indicates the position m corresponding uncertainty; Determine target location by combining probability and uncertainty The calculation formula is as follows: Among them, NR q Indicates the traversable area of ​​robot q, based on the real-time global traversable area map of robot q Sure; Indicates the category of the target object; Indicates the position m The probability that the target object exists on the robot, that is, the robot’s updated global semantic map Middle position m Department The value of; α is the weighting coefficient.

8. The multi-robot visual semantic navigation method according to any one of claims 1 to 7, characterized in that: Multiple robots are heterogeneous robots.

9. A multi-robot visual semantic navigation device, characterized in that: include: A training module is configured to collect RGB images and depth images at various locations in the scene, perform semantic segmentation and point cloud mapping on them, and obtain a true global traversable domain map and a true global semantic map of the scene; Collect RGB images and depth images at each position on each path in the scene, perform semantic segmentation and point cloud mapping on them, obtain the local traversable domain map and local semantic map of each path at each moment, fuse the local traversable domain map and local semantic map of each path at each moment with the global traversable domain map and global semantic map stored by the robot before sampling the respective paths, obtain the fused global traversable domain map and global semantic map at each moment, use the fused global traversable domain map and global semantic map at each moment and the real global semantic map of the scene as training samples to construct a training data set; use the training data set to train a deep neural network model to obtain a semantic map prediction model; wherein each point in the semantic map represents the probability of the existence of each object at that point, and each point in the semantic map corresponds one-to-one to the position of the corresponding point in the traversable domain map; The navigation module is configured to enable the robot to obtain a real-time local traversable domain map and a real-time local semantic map, and fuse them with its own stored global traversable domain map and global semantic map respectively to obtain a real-time global traversable domain map and a fused semantic map, and input them into the semantic map prediction model to obtain a predicted real-time global semantic map; multiple robots communicate to update their own global semantic maps; each robot performs a weighted summation of the probability of the existence of a target object at each position in its own updated global semantic map and the uncertainty of the prediction of each position, and then selects the point with the highest weighted sum score in its own local feasible domain map as the next target position; each robot goes to the target position and searches for the target object until one of the robots finds the target object or stops navigating when the upper limit of the navigation steps is reached.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multi-robot visual semantic navigation method according to any one of claims 1 to 8.

Images