Navigation method and device based on natural language and map, medium and program product

Abstract

The embodiment of the invention discloses a navigation method and device based on a natural language and a map, a medium and a program product. The method comprises the steps of performing semantic segmentation on a color image for an image in a current scene collected by a robot, performing processing based on a ground projection technology to obtain a semantic aerial view, and performing processing on a depth image based on the ground projection technology to obtain a depth aerial view; performing scene completion on the deep aerial view based on a language instruction input by a user to obtain a completed deep aerial view in a global scene; and according to the complemented depth aerial view, the semantic aerial view and the language instruction, generating a semantic map in the global scene, and predicting a path and navigating for the robot based on the semantic map. According to the scheme, on the basis of the image of the current scene collected by the robot, construction and complementation of the map can be guided based on the global semantic information, reasonable path reasoning is carried out on an invisible or unobserved area, and global navigation of the robot is realized under the condition of a prior global map.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of map construction and navigation, and particularly relates to a navigation method, device, medium and program product based on natural language and a map. BACKGROUND

[0002] Vision-and-Language Navigation (VLN) is a multi-modal artificial intelligence task that requires an agent to autonomously navigate in a visual environment according to natural language instructions and accurately reach the target location from the starting point. This task integrates multiple research fields such as computer vision, natural language processing and robot navigation, and is an important foundation for realizing human-computer interaction and autonomous decision-making in real scenes. Early vision language navigation methods use sequence-to-sequence LSTM to predict low-level actions or high-level actions of panoramas. Later researchers proposed various mechanisms to improve the learning of visual text correspondence. Reinforcement learning is also explored to improve policy learning. In addition, scholars have studied different strategies such as environment pre-exploration and active perception to form a more powerful navigation policy. In order to enhance the generalization ability of the agent to new environments, various data augmentation strategies are studied to simulate new environments or synthesize new instructions.

[0003] The prior art has many deficiencies. End-to-end models need to implicitly learn maps, planning and control, which are low in training efficiency and dependent on massive data; there is a lack of spatial reasoning and map perception ability, only the next path point is predicted, and there is a lack of global path modeling; in addition, the existing methods are unstable in performance without prior maps. SUMMARY

[0004] The embodiment of the present application provides a navigation method, device, medium and program product based on natural language and a map, which complete the map under the current perspective of the robot based on the language instruction, and predict the global path of the robot based on the completed map and navigate the robot.

[0005] According to an aspect of the present application, a navigation method based on natural language and a map is provided, which comprises:

[0006] For the color image and the depth image of the current scene collected by the robot, after the semantic segmentation of the color image, a semantic bird's eye view is obtained by processing based on the ground projection technology, and the depth image is processed based on the ground projection technology to obtain a depth bird's eye view;

[0007] Based on the language instruction input by the user, the depth bird's eye view is scene-completed to obtain a completed depth bird's eye view under the global scene;

[0008] According to the completed depth bird's eye view, the semantic bird's eye view and the language instruction, a semantic map under a global scene is generated, and a path for the robot is predicted and navigation is performed based on the semantic map.

[0009] According to an aspect of the present application, a natural language and map based navigation device is provided, the device comprising:

[0010] A bird's eye view generation module is configured to, for color images and depth images of a current scene collected by a robot, perform semantic segmentation on the color images and obtain a semantic bird's eye view based on ground projection technology, and perform processing on the depth images based on ground projection technology to obtain a depth bird's eye view.

[0011] A completed depth bird's eye view generation module is configured to perform scene completion on the depth bird's eye view based on a language instruction input by a user to obtain a completed depth bird's eye view under a global scene.

[0012] A semantic map generation module is configured to, according to the completed depth bird's eye view, the semantic bird's eye view and the language instruction, generate a semantic map under a global scene, and predict a path for the robot and perform navigation based on the semantic map.

[0013] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory in communication with the at least one processor; wherein

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the natural language and map based navigation method of any embodiment of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, the computer readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the natural language and map based navigation method of any embodiment of the present application when executed by the processor.

[0018] According to another aspect of the present application, a computer program product is provided, the computer program product comprises a computer program, and the computer program is executed by a processor to implement the natural language and map based navigation method of any embodiment of the present application.

[0019] The technical scheme of the embodiment of the application is that, aiming at the color image and the depth image of the current scene collected by the robot, the color image is subjected to semantic segmentation and then processed based on ground projection technology to obtain a semantic bird's eye view, the depth image is processed based on ground projection technology to obtain a depth bird's eye view, thereby obtaining the bird's eye view of the current scene under the current view angle of the robot, the depth bird's eye view is subjected to scene completion based on the language instruction input by the user to obtain a completed depth bird's eye view under the global scene, the scene that cannot be seen by the robot but needs to be passed through by the robot is predicted, the global map is constructed, the semantic map under the global scene is generated according to the completed depth bird's eye view, the semantic bird's eye view and the language instruction, and the path of the robot is predicted and navigated based on the semantic map, and in the case of no prior map, the construction and completion of the global map are accurately realized, the global path of the robot is predicted in advance, the instruction of the robot is generated according to the predicted global path to navigate, and the navigation precision is improved.

[0020] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0022] Figure 1 A flow chart of a navigation method based on natural language and map provided by an embodiment of the application;

[0023] Figure 2 A color image processing schematic diagram provided by an embodiment of the application;

[0024] Figure 3 A depth image processing schematic diagram provided by an embodiment of the application;

[0025] Figure 4 A flow chart of a navigation method based on natural language and map provided by another embodiment of the application;

[0026] Figure 5 A flow chart of a navigation method based on natural language and map provided by still another embodiment of the application;

[0027] Figure 6 A network structure diagram of a specific implementation provided by an embodiment of the application;

[0028] Figure 7 A structural schematic diagram of a navigation device based on natural language and a map provided for an embodiment of the present application is provided.

[0029] Figure 8 A structural schematic diagram of an electronic device provided for an embodiment of the present application is provided. DETAILED DESCRIPTION

[0030] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms “first”, “second”, “third”, “fourth”, “actual”, “preset” and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0032] Figure 1 A flowchart of a navigation method based on natural language and a map provided for an embodiment of the present application, the embodiment of the present application can be applicable to the case of navigating during the travel of a robot, typically, it can be applicable to the case of navigating based on the map under the current view during the travel of a robot, according to the language instruction to complete the global map. The method can be performed by a navigation device based on natural language and a map, which can be realized in the form of hardware and / or software, and the navigation device based on natural language and a map can be configured in an electronic device. As shown in the figure, the method comprises: Figure 1

[0033] S110, for the color image and the depth image of the current scene collected by the robot, after the semantic segmentation of the color image, the semantic bird's eye view is obtained by processing based on the ground projection technology, and the depth bird's eye view is obtained by processing the depth image based on the ground projection technology.​

[0034] wherein the robot is an intelligent device configured with an image collector and can move on the ground. The current scene is the scene within the field of view of the image collector when the robot is at the current position. The color image is an image composed of three color channels of red, green and blue, which can reflect the color information of the object surface. The color image is obtained by a color camera. The depth image is a kind of gray image, and the value of each pixel point represents the distance from the point to the camera. The depth information image does not contain color information, but reflects the distance relationship between the object and the image collector through the high and low of the gray value. The depth image can be obtained by a depth sensor (such as a ToF camera, a structured light camera, a binocular camera, etc.).

[0035] In the embodiments of the present application, the robot at the current position carries out image collection on the current scene within the current field of view of the image collector to obtain a color image and a depth image respectively, carries out semantic segmentation on the color image to obtain a semantic map corresponding to the color image, and processes the semantic map based on a ground projection technology to obtain a semantic bird's eye view. Semantic segmentation is to classify each pixel in the color image, and output a "mask image" with the same size as the input image, where different colors on the pixel point represent different categories. The network used for semantic segmentation can be selected according to actual conditions, such as FCN, U-Net, Mask R-CNN, etc. Processing the semantic map corresponding to the color image based on the ground projection technology to obtain the semantic bird's eye view is to map the semantic map to the global horizontal ground through the projection technology to form the visual effect of viewing the global scene from a high altitude. A schematic diagram of obtaining the semantic bird's eye view by performing semantic segmentation on the color image and then performing ground projection on the semantic map is shown in Figure 2 . Figure 2 The RGB image in

[0036] Processing the depth image based on the ground projection technology to obtain the depth bird's eye view can map the depth image to the global horizontal ground to form the visual effect of viewing the global scene from a high altitude. A schematic diagram of obtaining the depth bird's eye view by processing the depth map based on the ground projection technology is shown in Figure 3 .

[0037] In the above scheme, the semantic bird's eye view and the depth bird's eye view are obtained by the ground projection technology, so as to map the scene under the current perspective of the robot to the global scene, which is convenient for subsequent further completion to obtain the global map information.

[0038] S120, based on the language instruction input by the user, the scene of the depth bird's eye view is completed to obtain a completed depth bird's eye view under the global scene.

[0039] The language instruction input by the user can be an instruction for guiding the robot to walk, and the language instruction includes guidance for a subsequent walking direction of the robot and a destination reached. For example, the language instruction includes "turn right, leave the bathroom and enter the bedroom. After entering the bedroom, turn right and walk to the door on the right. After leaving the door, turn left and walk to the top of the stairs and stop."

[0040] For example, the language instruction includes an instruction for guiding the robot to act in a global scene, which includes various scene locations in the global scene, and is not limited to the scene location currently occupied by the robot. For example, in the language instruction in the above example, the bedroom, the door, the stairs, and the positional relationship therebetween after the bathroom, which is the scene location currently occupied by the robot, are included. In this case, the image collected in the scene currently occupied by the robot can be scene-completed based on the various scene locations passed after the scene location currently occupied by the robot and the positional relationship therebetween, so as to obtain more scenes and generate a map including more scenes, thereby facilitating map completion to obtain a complete global map required to be passed by the robot in the absence of a prior map.

[0041] In the embodiment of the present application, the depth bird's-eye view is scene-completed based on the language instruction input by the user, and the various scenes and the positional relationship described in the language instruction are expanded in the depth bird's-eye view according to the positional description and the scene location description of the scene location after the current scene location of the robot in the language instruction, so as to complete the various scene locations after the language instruction. For example, in the above example, the language instruction reflects that the current scene occupied by the robot is the bathroom, and "turn right and leave the bathroom to enter the bedroom" reflects that the bedroom is on the right side of the bathroom, so the depth map of the bedroom is supplemented on the right side of the bathroom. "After entering the bedroom, turn right and walk to the door on the right" reflects that there is a door on the right side in the bedroom, so the door is supplemented on the right side in the bedroom. "After leaving the door, turn left and walk to the top of the stairs" reflects that there are stairs after turning left, so the stairs are supplemented on the left side after leaving the door. In the embodiment of the present application, the information of the position and the location reflected by the language instruction can be embodied by depth information, but cannot be embodied by color information, so scene completion is performed based on the depth bird's-eye view, thereby realizing prediction of a scene that cannot be observed after the current position of the robot and determining global scene information.

[0042] S130, generating a semantic map in a global scene according to the completed depth bird's-eye view, the semantic bird's-eye view, and the language instruction, and predicting a path and navigating for the robot based on the semantic map.

[0043] For example, the completed depth bird's eye view reflects global scene information, the semantic bird's eye view reflects color information of the scene at the current position of the robot, and the language instruction reflects all scene information that needs to be passed through by the robot. Based on the above, a semantic map in the global scene can be generated, so as to predict a path and navigate the robot based on the semantic map.

[0044] For example, based on the depth information of each pixel point in the completed depth bird's eye view and the category to which each pixel point in the semantic bird's eye view belongs, the pixel points of the same category can be predicted, the pixel points of the category in other scenes in the completed depth bird's eye view can be color-assigned based on the color corresponding to the category in the semantic bird's eye view, and the semantic map can be supplemented. The semantic map can be further supplemented and verified in combination with the language instruction to obtain a more accurate semantic map.

[0045] The technical scheme of the embodiment of the application is that, for the color image and the depth image of the current scene collected by the robot, the color image is subjected to semantic segmentation and then processed based on ground projection technology to obtain a semantic bird's eye view, and the depth image is processed based on ground projection technology to obtain a depth bird's eye view, so as to obtain a bird's eye view of the current scene at the current perspective of the robot, the depth bird's eye view is subjected to scene completion based on a language instruction input by a user to obtain a completed depth bird's eye view in the global scene, the scene that cannot be seen by the robot but needs to be passed through by the robot is predicted, the global map is constructed, a semantic map in the global scene is generated based on the completed depth bird's eye view, the semantic bird's eye view and the language instruction, and the path and navigation of the robot are predicted based on the semantic map. In the case of no prior map, the construction and completion of the global map are accurately realized, the global path of the robot is predicted in advance, the robot is navigated based on the generated instruction according to the predicted global path, and the navigation accuracy is improved.

[0046] Figure 4 A flowchart of a navigation method based on natural language and a map provided for another embodiment of the application, the embodiment of the application is optimized based on the above-described embodiment, and the schemes not described in detail in the embodiment of the application are described in the above-described embodiment. As shown in Figure 4 The method of the embodiment of the application specifically includes the following steps:

[0047] S210, for the color image and the depth image of the current scene collected by the robot, the color image is subjected to semantic segmentation and then processed based on ground projection technology to obtain a semantic bird's eye view, and the depth image is processed based on ground projection technology to obtain a depth bird's eye view.

[0048] S220, determining a cross-modal map attention value according to the depth bird's eye view and a language instruction input by a user.

[0049] The depth bird's eye view reflects visually presented spatial map features, and the language instruction reflects acoustic spatial map features. The data of the two modalities can be fused through cross attention distribution to establish a dynamic mapping of spatial coordinates and language keywords, thereby forming a joint representation that combines environmental structure understanding and semantic intention capture.

[0050] Specifically, one of the depth bird's eye view and the language instruction can be selected for calculating a query vector, one of the depth bird's eye view and the language instruction can be selected for calculating a key vector, and one of the depth bird's eye view and the language instruction can be selected for calculating a value vector. Then, cross-modal map attention values are obtained through attention weighting.

[0051] In the embodiments of the present application, the cross-modal map attention values are determined according to the depth bird's eye view and the language instruction input by the user, and the method comprises the following steps:

[0052] The depth bird's eye view is subjected to feature extraction to obtain depth bird's eye view features, and the language instruction input by the user is subjected to language encoding to obtain language instruction features.

[0053] The cross-modal map attention values are calculated based on a scaled dot product attention mechanism according to the depth bird's eye view features and the language instruction features.

[0054] For example, the depth bird's eye view is subjected to feature extraction to obtain depth bird's eye view features, for example, using , wherein o t is the depth bird's eye view, is the depth bird's eye view features. The language instruction input by the user is subjected to language encoding to obtain language instruction features, for example, using X. The depth bird's eye view features and the language instruction features are calculated based on a scaled dot product attention mechanism to obtain cross-modal map attention, for example, the depth bird's eye view features and the language instruction features X are subjected to linear transformation to respectively generate Q, K, and V vectors: K=XW k , V=XW v , and then cross-modal map attention values are obtained through attention weighting: The above scheme has the beneficial effect that it can fuse multi-modal data to calculate attention values, thereby forming a joint representation that combines environmental structure understanding and semantic intention capture, and facilitating the integration of global scene information contained in the language instruction in the subsequent map completion process, thereby realizing map scene completion based on language semantics.

[0055] S230, based on the cross-modal map attention values and the depth bird's eye view, a completed depth bird's eye view in a global scene is generated.

[0056] For example, the cross-modal map attention and the depth bird's-eye view map can be processed based on a convolutional neural network, thereby using the cross-modal map attention to dynamically weight the input of data of different modalities, distinguishing the attention weights of the depth bird's-eye view features and the language command features, and using the attention weights as a guide, inferring scenes that are occluded or not collected by the robot from the locally observed depth bird's-eye view map, realizing the completion and prediction of the global scene, and generating a completed depth bird's-eye view map under the global scene.

[0057] In an embodiment of the present application, a complete depth bird's-eye view in a global scene is generated based on the cross-modal map attention value and the depth bird's-eye view based on a convolutional neural network, including:

[0058] The depth bird's-eye view map and the cross-modal map attention value are input into the convolutional neural network to obtain a completed depth bird's-eye view map in a global scene; wherein the convolutional neural network is a U-Net network.

[0059] Among them, U-Net is a convolutional neural network with a symmetrical encoder-decoder structure, whose core features are U-shaped design and jump connection. The encoder extracts features by gradually downsampling through convolution and pooling, and the decoder recovers spatial details through upsampling and jump connection (directly fusing the multi-layer features of the encoder), and finally outputs pixel-level classification results. The depth bird's-eye view map and the attention value of the cross-modal map are input into the convolutional neural network to recover the spatial details of the vacant scene part in the depth bird's-eye view map, and distinguish different depths as different classifications to obtain a completed depth bird's-eye view map, which can be used To express, where To complete the depth bird's-eye view, g o It is the U-Net network.

[0060] S240. Generate a semantic map in a global scenario based on the cross-modal map attention value, the completed depth bird's-eye view, and the semantic bird's-eye view based on a convolutional neural network.

[0061] For example, the cross-modal map attention value is used to dynamically weight the contribution of data from different modalities, and the depth bird's-eye view is completed to provide geometric information of the repaired uncollected areas. The semantic bird's-eye view provides the label of the object category. The above three types of features are integrated through the convolutional neural network, and the local observation is promoted to a dense semantic map covering the whole world through the encoding-decoding structure. The complete scene expression containing three-dimensional geometry and semantic attributes is output to obtain the semantic map of the global scene. Specifically, the semantic map in, is a semantic bird's-eye view, g s It is the U-Net network.

[0062] The embodiment of the application provides a navigation method based on natural language and a map, a cross-modal map attention value is determined according to the deep bird's-eye view and a language instruction input by a user; a complete deep bird's-eye view in a global scene is generated based on the cross-modal map attention value and the deep bird's-eye view according to a convolutional neural network, the deep bird's-eye view is completed based on scene information contained in the language instruction, prediction of a scene not collected by a robot is realized, a semantic map in the global scene is generated based on the cross-modal map attention value, the complete deep bird's-eye view and the semantic bird's-eye view according to the convolutional neural network, so that global scene information and object category labels are fused to realize presentation of the global scene, a semantic map in a panoramic scene is accurately and quickly constructed without a priori map, and subsequent global navigation of the robot is facilitated.

[0063] Figure 5 A flowchart of a navigation method based on natural language and a map is provided for another embodiment of the application, the embodiment of the application is optimized based on the above-mentioned embodiment, and the schemes not described in detail in the embodiment of the application are described in the above-mentioned embodiment. As shown in Figure 5 The method of the embodiment of the application specifically includes the following steps:

[0064] S310, for a color image and a depth image in a current scene collected by a robot, a semantic bird's-eye view is obtained by performing semantic segmentation on the color image and then performing processing based on ground projection technology, and a deep bird's-eye view is obtained by performing processing on the depth image based on the ground projection technology.

[0065] S320, a complete deep bird's-eye view in a global scene is obtained by performing scene completion on the deep bird's-eye view based on a language instruction input by a user.

[0066] S330, cross-modal path attention is determined according to semantic map features obtained by performing feature extraction on the semantic map and language instruction features obtained by encoding the language instruction.

[0067] For example, semantic map features are obtained by performing feature extraction on the semantic map, language instruction features X are obtained by encoding the language instruction, and the cross-modal path attention is determined according to the semantic map features and the language instruction features. Specifically, the semantic map features and the language instruction features X are respectively generated into Q, K and V vectors after linear transformation: K=XW k , V=XW v , and the cross-modal path attention is obtained by attention weighting:

[0068] S340, generating a path point heat map based on the cross-modal path attention and the current position of the robot according to the convolutional neural network.

[0069] For example, a path prediction model is constructed based on the convolutional neural network, and a loss function in the training process of the path prediction model is composed of the Euclidean distance between the predicted path point and the actual path point, and the cross-entropy loss of the path point that has passed. The cross-modal path attention and the current position of the robot are input into the path prediction model, and the path point heat map is output. wherein, is the current position of the robot.

[0070] S350, detecting the current path point closest to the current position of the robot in the predicted path point heat map.

[0071] For example, in the predicted path point heat map, the current path point closest to the current position of the robot is detected, and the current path point is approximately equivalent to the current position of the robot, that is, the current path point is approximately equivalent to the trajectory point passed by the robot, so as to plan the action of the robot for the path points after the current path point in the path point heat map. For example, the label of the target path point is ζ = 1 + argmin i △(p i,t ,p t ), wherein △ represents the Euclidean distance, p i,t represents the i-th path point, p t represents the current position. argmin i △(p i,t ,p t ) represents the label i of the path point closest to the current position, and i+1 is the label of the target path point, that is, the (i+1)-th path point as the target path point.

[0072] S360, taking the path points after the current path point in the path heat map as target path points, and inputting the target path points into a proximal policy optimization algorithm to obtain low-level action instructions for robot navigation.

[0073] Exemplarily, a path point behind the current path point in the path heat map is taken as a target path point, which is a path point that the robot needs to pass next on the premise that the robot is at the current position, and a target for guiding the robot is to guide the robot to move from the current position to the target path point. After the target for moving is determined, the target path point can be input into the proximal policy optimization algorithm to obtain a low-level action instruction for guiding the robot to move, the path planning task is converted into a continuous decision problem through the reinforcement learning framework, and finally specific commands for directly controlling the robot actuator are generated, including advancing (0.25 meters), turning left / turning right (15°), stopping, and the like.

[0074] The embodiment of the present application provides a navigation method based on natural language and a map. According to semantic map features after feature extraction of the semantic map and language instruction features after encoding of the language instruction, cross-modal path attention is determined. According to the cross-modal path attention and a current position of the robot, a path point heat map is generated based on a convolutional neural network, so as to navigate based on the path point heat map. Therefore, the semantic map features are taken as queries, the language instruction features are taken as key-value pairs, an attention relationship from language to a map region is constructed, and the map is updated in the case of unchanged language, so as to guide the robot to move. The closest current path point to the current position of the robot is detected in the predicted path point heat map. A path point behind the current path point in the path heat map is taken as a target path point. The target path point is input into a proximal policy optimization algorithm to obtain a low-level action instruction for robot navigation. Therefore, the robot is navigated in subsequent behavior in the globally completed map. The path prediction and navigation are optimized through spatial alignment, and the navigation precision is improved.

[0075] The embodiment of the present application provides a specific implementation mode as shown in Figure 6 The embodiment specifically comprises:

[0076] 1, constructing a semantic map part: considering that the visual input of the robot is only a current visual angle, and there are often occlusions or unknown regions in the scene, the semantic information of the unobserved region needs to be completed through the language instruction. Therefore, the semantic map prediction module is designed as a two-stage U-Net network. The module uses a cross-modal attention mechanism to introduce the language instruction into the semantic prediction process. The semantic completion network receives the semantic segmentation results of the depth image and the color image of the current frame, and combines the language instruction to guide the semantic completion and prediction of the unobserved region through the cross-modal attention. This process embodies the enhancement effect of the spatial prior (for example, "passing through the kitchen into the living room") in the language on the scene mapping. Specifically:

[0077] (1) The robot acquires an RGB image through an image collector, obtains a semantic map corresponding to a color image based on the RGB image using semantic segmentation, and further processes to obtain a top-down semantic bird's eye view using ground projection technology

[0078] (2) The robot acquires a depth image through a depth camera, and obtains a top-down depth bird's eye view using ground projection t , and extracts features to obtain a depth bird's eye view feature

[0079] (3) For the received language instruction, the language instruction feature X is extracted using the BERT model.

[0080] (4) A standard scaled dot product attention form is used, and in the calculation process, the depth bird's eye view feature and the language instruction feature X are linearly transformed to generate Q, K, and V vectors respectively: K=XW k , V=XW v , and the cross-modal map attention value is obtained by attention weighting:

[0081] (5) The U-Net network g o is used to complete to obtain a completed depth bird's eye view

[0082] (6) Combined with the information extracted in (1) and (5), the U-Net is used to combine the completed depth bird's eye view , the semantic bird's eye view , and the cross-modal map attention to generate a complete semantic map: The semantic map feature of the map is

[0083] 2. Path point prediction part: The path prediction module aims to predict a complete path on the constructed semantic map according to the language instruction, which is represented as a set of two-dimensional path points. The core mechanism is a cross-modal attention network, which constructs an attention relationship from language to map area by taking the current semantic map feature as the query and the language instruction feature as the key and value pair. The motivation of this design is that the language remains unchanged during navigation, while the map is constantly updated, which is suitable for using language to guide key areas in the map. Specifically:

[0084] (1) Similar to the previous section, the complete semantic map feature is extracted and the language instruction feature X is weighted by attention to obtain

[0085] (2) The representation is input to the U-Net decoding structure for generating the path point heat map. Each channel in the heat map corresponds to a path point, and the highlighted area thereof represents the possible position of the path point. Meanwhile, the model also predicts whether each path point has been passed by the robot to enhance the rationality of the current path prediction. The path prediction loss is composed of the Euclidean distance loss between the heat map and the actual path point position and the cross-entropy loss of the traversed points. The path point prediction module is a U-Net network, and the input thereof is and the current position The output generates a path point heat map:

[0086] 3, the navigation instruction generation part: once the path prediction is completed, the system needs to select the next short-term target point and generate the navigation action. The system can automatically issue a stop command according to the distance between the current position of the robot and the end point of the path point heat map during navigation, thereby completing the entire navigation task. Specifically:

[0087] (1) By finding the closest current path point to the current position of the robot in the predicted path point heat map, and selecting the next point of the current path point in the path point heat map as the target path point: ζ = 1 + argmin i △(p i,t ,p t ), where △ represents the Euclidean distance, p i,t represents the i th path point, p t represents the current position, and ζ represents the ζ th path point as the target path point.

[0088] (2) Subsequently, the target path point P ζ is input to the reinforcement learning strategy DD-PPO, which has been trained on the point target navigation task and is responsible for converting the target into low-level action instructions.

[0089] Figure 7 A structure schematic diagram of a navigation device based on natural language and a map provided by an embodiment of the present application, which can execute the navigation method based on natural language and a map provided by any embodiment of the present application, has the corresponding function modules and beneficial effects of the execution method. As shown in Figure 7 The device comprises:

[0090] An aerial view generation module 410 is configured to, for a color image and a depth image of a current scene collected by a robot, perform semantic segmentation on the color image and perform processing based on a ground projection technology to obtain a semantic aerial view, and perform processing on the depth image based on the ground projection technology to obtain a depth aerial view.

[0091] The completion depth bird's eye view generation module 420 is configured to perform scene completion on the depth bird's eye view based on the language instruction input by the user, to obtain a completion depth bird's eye view in a global scene.

[0092] The semantic map generation module 430 is configured to generate a semantic map in a global scene according to the completion depth bird's eye view, the semantic bird's eye view and the language instruction, and predict a path and navigate the robot based on the semantic map.

[0093] In the embodiment of the present application, the completion depth bird's eye view generation module 420 performs scene completion on the depth bird's eye view based on the language instruction input by the user, to obtain a completion depth bird's eye view in a global scene, including:

[0094] determining a cross-modal map attention value according to the depth bird's eye view and the language instruction input by the user;

[0095] generating a completion depth bird's eye view in a global scene based on a convolutional neural network according to the cross-modal map attention value and the depth bird's eye view.

[0096] In the embodiment of the present application, the completion depth bird's eye view generation module 420 determines a cross-modal map attention value according to the depth bird's eye view and the language instruction input by the user, including:

[0097] performing feature extraction on the depth bird's eye view to obtain a depth bird's eye view feature, and performing language encoding on the language instruction input by the user to obtain a language instruction feature;

[0098] calculating a cross-modal map attention value based on a scaled dot product attention mechanism according to the depth bird's eye view feature and the language instruction feature.

[0099] In the embodiment of the present application, the completion depth bird's eye view generation module 420 generates a completion depth bird's eye view in a global scene based on a convolutional neural network according to the cross-modal map attention value and the depth bird's eye view, including:

[0100] inputting the depth bird's eye view and the cross-modal map attention value into the convolutional neural network to obtain a completion depth bird's eye view in a global scene; wherein the convolutional neural network is a U-Net network.

[0101] In the embodiment of the present application, the semantic map generation module 430 generates a semantic map in a global scene according to the completion depth bird's eye view, the semantic bird's eye view and the language instruction, including:

[0102] determining a cross-modal map attention value according to the depth bird's eye view and the language instruction input by the user;

[0103] generate a semantic map in a global scene based on the cross-modal map attention value, the completed depth bird's eye view, and the semantic bird's eye view according to a convolutional neural network.

[0104] In the embodiments of the present application, the semantic map generation module 430 predicts a path and navigation for the robot based on the semantic map, including:

[0105] determine cross-modal path attention based on semantic map features after feature extraction according to the semantic map, and language instruction features after language instruction encoding;

[0106] generate a path point heat map based on the cross-modal path attention and the current position of the robot according to a convolutional neural network, to perform navigation based on the path point heat map.

[0107] In the embodiments of the present application, the semantic map generation module 430 predicts a path and navigation for the robot based on the semantic map, including:

[0108] detect a current path point closest to the current position of the robot in the predicted path point heat map;

[0109] take a path point after the current path point in the path heat map as a target path point, and input the target path point into a proximal policy optimization algorithm to obtain a low-level action instruction for robot navigation.

[0110] The navigation device based on natural language and a map provided in the embodiments of the present application can execute the navigation method based on natural language and a map provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0111] Figure 8 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0112] As Figure 8As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0113] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless natural language and map based navigation transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0114] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the natural language and map based navigation method.

[0115] In some embodiments, the natural language and map based navigation method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the natural language and map based navigation method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the natural language and map based navigation method by any other appropriate means, such as by means of firmware.

[0116] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0117] Computer programs used to implement the processes of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a general purpose computer, special purpose computer, or other programmable processing apparatuses to produce a machine, such that the computer program, when executed, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0120] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0121] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0122] The embodiments of the present application further provide a computer program product, comprising a computer program which, when executed by a processor, implements the natural language and map-based navigation method as provided in any of the embodiments of the present application.

[0123] The computer program code can also be implemented in, and used with, one or more computer program languages, including object oriented, visual, scripting or other types of computer program languages. The computer program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0124] It should be understood that the various forms of flow shown in the figures can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired information is achieved in accordance with the present disclosure, which is not limited herein.

[0125] The specific embodiments have been shown and described for the purposes of illustrating the embodiments, which are susceptible to various modifications and alternative forms. It is to be understood that specific further implementations can be developed with the exercise of skill in the art, and that any further implementations that are within the scope of the following claims are intended to be encompassed thereby.

Claims

1. A navigation method based on natural language and map, characterized in that: The method comprises: For the color image and depth image of the current scene collected by the robot, the color image is semantically segmented and then processed based on the ground projection technology to obtain a semantic bird's-eye view image, and the depth image is processed based on the ground projection technology to obtain a depth bird's-eye view image; Performing scene completion on the depth bird's-eye view based on a language instruction input by a user to obtain a completed depth bird's-eye view in a global scene; A semantic map of a global scene is generated according to the completed depth bird's-eye view, the semantic bird's-eye view, and the language instructions, and a path is predicted and navigation is performed for the robot based on the semantic map.

2. The method according to claim 1, characterized in that The scene completion is performed on the depth bird's-eye view based on the language instruction input by the user to obtain a completed depth bird's-eye view in a global scene, including: Determining a cross-modal map attention value based on the depth bird's-eye view image and a language instruction input by the user; Based on the convolutional neural network, a completed depth bird's-eye view is generated in the global scene according to the cross-modal map attention value and the depth bird's-eye view.

3. The method according to claim 2, characterized in that Determining a cross-modal map attention value according to the depth bird's-eye view image and a language instruction input by the user includes: Performing feature extraction on the depth bird's-eye view image to obtain depth bird's-eye view features, and performing language encoding on the language instruction input by the user to obtain language instruction features; The cross-modal map attention value is calculated based on the deep bird's-eye view features and the language instruction features based on the scaled dot product attention mechanism.

4. The method according to claim 2, characterized in that Generating a completed depth bird's-eye view in a global scene based on the cross-modal map attention value and the depth bird's-eye view based on a convolutional neural network, including: The depth bird's-eye view map and the cross-modal map attention value are input into the convolutional neural network to obtain a completed depth bird's-eye view map in a global scene; wherein the convolutional neural network is a U-Net network.

5. The method according to claim 1, wherein Generating a semantic map of a global scene according to the completed depth bird's-eye view, the semantic bird's-eye view, and the language instruction, including: Determining a cross-modal map attention value based on the depth bird's-eye view image and a language instruction input by the user; Based on a convolutional neural network, a semantic map of the global scene is generated according to the cross-modal map attention value, the completed depth bird's-eye view map and the semantic bird's-eye view map.

6. The method according to claim 1, characterized in that Predicting a path and navigating for the robot based on the semantic map includes: Determining cross-modal path attention based on semantic map features extracted from the semantic map and language instruction features encoded from the language instruction; Based on the convolutional neural network, a waypoint heat map is generated according to the cross-modal path attention and the current position of the robot, so as to navigate based on the waypoint heat map.

7. The method according to claim 1, characterized in that Predicting a path and navigation for the robot based on the semantic map includes: Detecting the current waypoint closest to the current position of the robot in the predicted waypoint heat map; The path point located after the current path point in the path heat map is used as the target path point, and the target path point is input into the proximal strategy optimization algorithm to obtain low-level action instructions for robot navigation.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the natural language and map-based navigation method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the natural language and map-based navigation method according to any one of claims 1 to 7 when executed.

10. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the natural language and map-based navigation method as described in any one of claims 1 to 7.

Images