A visual language navigation method, system, device and storage medium

Abstract

The application discloses a visual language navigation method, system and device and a storage medium, wherein the method comprises the following steps: acquiring visual image data of an intelligent agent in an environment; the visual image data comprises an RGB image and a depth image; constructing a map representation according to the visual image data and extracting instruction markers; performing a weakly supervised instruction marker positioning task according to the map representation and the instruction markers, and constructing an instruction marker map; predicting a sub-target point position and a navigation progress, and performing a corresponding action. The application establishes a corresponding relationship between relevant objects in an instruction and a semantic map, thereby improving the representation ability of instruction features and map features, realizing alignment and fusion of two modalities, and being widely applied to the technical field of intelligent navigation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent navigation, and in particular to a visual language navigation method, system, device and storage medium. BACKGROUND

[0002] In the early development of artificial intelligence, researchers often only focus on how to make computers process data faster and make predictions more accurately, while ignoring the problem of interaction with the real world. However, in reality, people need to interact and perceive the environment when performing various tasks. The emergence of embodied intelligence brings new ideas to the development of artificial intelligence, which emphasizes the interaction between agents and the environment and fills this gap.

[0003] One of the research hotspots of embodied intelligence is visual language navigation, which aims to enable agents to navigate autonomously through visual and language information. This technology has broad application prospects, such as in home service robots, where users can instruct the robot to navigate to a specific location and perform specific tasks, such as turning on the lights or closing the windows. In addition, this technology can also be applied in the field of autonomous driving, enabling autonomous vehicles to drive autonomously based on instructions and visual information. The development of visual language navigation technology brings new possibilities for human-computer interaction, and there are many potential application scenarios worth exploring in the future.

[0004] Currently, existing methods propose a modular approach based on maps to implement visual language navigation by constructing semantic maps to represent environmental information. However, semantic maps can only represent a limited number of object categories, and it is difficult to represent the objects described in the instructions through semantic information alone. Existing methods often ignore the potential correspondence between the landmarks in the received navigation instructions and the constructed semantic maps. How to align and fuse text and visual modalities is one of the research hotspots and difficulties in the current visual language navigation task. SUMMARY

[0005] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a visual language navigation method based on weakly supervised instruction landmark positioning, system, device and storage medium.

[0006] The technical solution adopted by the present application is:

[0007] A visual language navigation method, comprising the following steps:

[0008] Obtaining visual image data of an agent in an environment; the visual image data includes RGB images and depth images;

[0009] Constructing a map representation based on visual image data and extracting instruction landmarks;

[0010] According to the map representation and the instruction marker, a weakly supervised instruction marker positioning task is performed, and an instruction marker map is constructed.

[0011] The sub-target point position and the navigation progress are predicted, and a corresponding action is performed.

[0012] Further, the map representation is used to map information of an RGB picture to a 2D map; the map representation is a semantic map representing object category information, or a high-dimensional feature map obtained by passing an object through a deep neural network and mapped to a 2D map.

[0013] Further, when the map representation is a semantic map representing object category information, the constructing a map representation according to visual image data comprises:

[0014] An object existing in the RGB image is detected by a preset semantic segmentation model as a semantic information;

[0015] In the constructing of the map, a semantic segmentation result of the semantic segmentation model is selected as a basis, and a spatial three-dimensional coordinate of each pixel in the image is obtained through a camera parameter and depth information in a depth image;

[0016] The spatial three-dimensional coordinate is converted to a map coordinate, so that each segmentation result is mapped to a corresponding position in the map to obtain a semantic map.

[0017] Further, the extracting an instruction marker comprises:

[0018] An instruction marker category included in the instruction is parsed using an instruction parser;

[0019] The instruction parser is a large language model or a part-of-speech analysis tool.

[0020] Further, the performing a weakly supervised instruction marker positioning task and constructing an instruction marker map comprises:

[0021] Based on a real path corresponding to the instruction, a coarse-grained instruction marker positioning label is constructed, a shortest straight line distance between each position in the map and the real path is calculated, distances of all positions are normalized, and finally a softmax operation is performed in a spatial dimension to obtain coarse-grained positioning labels of multiple markers;

[0022] A cosine similarity between an instruction marker feature and a feature of each position in the map is calculated to obtain a predicted positioning result;

[0023] The predicted positioning result is taken as a weight and multiplied by the marker feature to obtain an instruction marker map.

[0024] Further, the cosine similarity between the calculated instruction marker feature and each location feature of the map is used to obtain a prediction positioning result, including:

[0025] For the text of the nth marker, the marker text feature f n is calculated by a text encoder

[0026] The cosine similarity between the text feature and each location feature of the map representation is calculated, thereby obtaining the positioning prediction result of the marker

[0027] Wherein, the positioning prediction loss is calculated by the following formula:

[0028]

[0029] In the formula, P n represents the coarse-grained marker positioning label, represents the prediction positioning result, and N is the number of markers.

[0030] Further, the sub-target point position and navigation progress are predicted, and corresponding actions are performed, including:

[0031] The map representation, instruction marker map and instruction feature are input into a recurrent neural network to obtain the current state feature of the agent;

[0032] The current state feature is input into a sub-target point predictor to predict the relative coordinate offset of the sub-target point from the current position;

[0033] The relative coordinate offset obtained by prediction is used to mark the sub-target point position on the map, and the next action of the agent is obtained and executed through a preset path planning algorithm;

[0034] The current state feature is input into a navigation progress predictor to predict the navigation progress under the current state, and if the predicted progress is greater than a preset threshold, the current navigation is ended.

[0035] Another technical solution adopted by the present application is:

[0036] A visual language navigation system, comprising:

[0037] A data acquisition module is configured to acquire visual image data of an agent in an environment; the visual image data includes RGB images and depth images;

[0038] A representation construction module is configured to construct a map representation according to the visual image data, and extract instruction markers;

[0039] A weakly supervised instruction marker positioning module is configured to perform a weakly supervised instruction marker positioning task according to the map representation and the instruction markers, and construct an instruction marker map;

[0040] A sub-target position prediction module is configured to predict a sub-target position and a navigation progress, and perform a corresponding action.

[0041] Another technical solution adopted by the present application is:

[0042] A visual language navigation device comprises:

[0043] At least one processor;

[0044] At least one memory for storing at least one program;

[0045] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0046] Another technical solution adopted by the present application is:

[0047] A computer-readable storage medium, wherein a processor-executable program is stored, and the processor-executable program is used to execute the above method when executed by a processor.

[0048] The present application has the beneficial effect that the present application establishes the correspondence between the relevant objects in the instruction and the semantic map, thereby improving the representation ability of the instruction features and the map features, and realizing the alignment and fusion of the two modalities. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing some embodiments in the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0050] Figure 1 is a step flow chart of a visual language navigation method based on weakly supervised instruction marker positioning in an embodiment of the present application;

[0051] Figure 2 is a flowchart of a visual language navigation method based on weakly supervised instruction marker positioning in an embodiment of the present application;

[0052] Figure 3 is a schematic diagram of a weakly supervised instruction marker positioning module in an embodiment of the present application. DETAILED DESCRIPTION

[0053] Embodiments of the present application are described below in detail with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explaining the description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0054] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by the upper, lower, front, rear, left, right, etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0055] In the description of the present application, several meanings are one or more, and the meaning of multiple is two or more. Greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of indicated technical features.

[0056] In addition, in the description of the present application, unless otherwise specified, "a plurality of" means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the front and rear associated objects are in an "or" relationship.

[0057] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0058] As shown in Figure 1 and Figure 2 The embodiment provides a visual language navigation method based on weakly supervised instruction marker positioning, which comprises the following steps:

[0059] S1, acquiring visual image data of an agent in an environment; the visual image data comprises an RGB image and a depth image.

[0060] Obtain visual images observed by the agent in the simulation environment, including RGB images and depth images. As an optional implementation, this embodiment uses the public simulator habitat-sim and uses the public dataset VLN-CE as training and testing data.

[0061] S2, construct a map representation according to the visual image data, and extract instruction markers.

[0062] The map representation of this embodiment can be any type of map representation, which aims to map the information of the RGB picture to a 2D map, including but not limited to a semantic map representing object category information, and a high-dimensional feature map obtained by passing the object through a deep neural network and mapped to a 2D map. The following takes a semantic map as an example to illustrate the specific implementation process.

[0063] Detect the objects present in the RGB image through a semantic segmentation model (such as U-Net), and select the semantic segmentation result of the semantic segmentation network as the basis when constructing the map to reflect semantic information. Obtain the spatial three-dimensional coordinates of each pixel in the image through the camera parameters and the depth information in the depth image, and finally convert the spatial three-dimensional coordinates to map coordinates, thereby mapping each segmentation result to the corresponding position in the map to obtain a semantic map.

[0064] Use an instruction parser to parse the marker categories contained in the instruction. A large language model such as ChatGPT can be selected as the instruction parser, and the corresponding prompt words are input to make it output the instruction markers; or a part-of-speech analysis tool such as StanfordNLP can be selected as the instruction parser, and the nouns in the instruction are obtained through analysis as the instruction markers. It is worth noting that using a large language model such as ChatGPT can further detect the time sequence order of each instruction marker.

[0065] S3, perform a weakly supervised instruction marker localization task according to the map representation and the instruction markers, and construct an instruction marker map.

[0066] In order to locate the instruction markers in the map representation, a 2D probability distribution map is predicted to represent the possible positions of the instruction markers in the map according to the map representation and the instruction markers. However, the lack of annotation information makes it difficult to obtain the accurate positions of each object in the instruction in the map. However, since each instruction corresponds to a real path in the map, we observe that the areas closer to the real path have a higher probability of containing instruction markers, the instruction markers appearing earlier in the time sequence are closer to the starting point of the path, and all instruction markers are basically uniformly distributed around the path.

[0067] Based on the above observations, refer to Figure 3, the real path corresponding to the instruction is used as a guide to construct the coarse-grained instruction marker positioning label. Specifically, for an instruction with N markers, the path is evenly divided into N segments. For the nth appearing instruction marker, the coarse-grained marker positioning label P n ∈R H×W is supervised as the instruction marker positioning, where H and W are the length and width of the map respectively, P n Each point on the map represents the probability of the nth marker appearing at that location on the map. P n The calculation method is as follows: calculate the shortest straight line distance between each location in the map and the nth segment of the path and normalize it. The closer to the real path, the closer to 1, and vice versa. Finally, perform a softmax operation on the spatial dimension to obtain the coarse-grained positioning label of the nth marker.

[0068] For the text of the nth marker, the marker text feature f n is calculated by a text encoder (such as BERT). The cosine similarity between the text feature and the feature of each location in the map representation is further calculated to obtain the positioning prediction result of the marker, and the positioning prediction loss is calculated by the following formula:

[0069]

[0070] Where P n represents the coarse-grained marker positioning label, represents the predicted positioning result, and N is the number of markers.

[0071] The predicted 2D probability distribution map of the instruction marker can reflect the marker position information, but lacks the marker category information. Therefore, the marker text feature f n is multiplied by the corresponding probability distribution map to obtain the nth marker marker map. The average of the N marker maps is obtained to obtain the instruction marker map, which contains the category information and position information of the marker.

[0072] S4, predict the sub-target point position and navigation progress, and perform the corresponding action.

[0073] The agent predicts a sub-target point position at each time, and plans a path to the sub-target point by using a path planning algorithm, so as to calculate the action that should be performed at present. Specifically, the map representation, the instruction marker map and the instruction feature are input into a recurrent neural network (such as LSTM) to obtain the current state feature of the agent, the state feature is input into a sub-target point predictor to predict the relative coordinate offset of the sub-target point from the current position. Thus, the sub-target point position can be marked on the map, and the next action of the agent can be obtained by using an existing path planning algorithm (such as DDPPO). Meanwhile, the state feature is input into a navigation progress predictor to predict the navigation progress under the current state, and when the predicted progress is greater than a certain threshold, a STOP action is performed to end the current navigation.

[0074] The embodiment also provides a visual language navigation system, comprising:

[0075] A data acquisition module is configured to acquire visual image data of the agent in the environment, wherein the visual image data comprises an RGB image and a depth image.

[0076] A representation construction module is configured to construct a map representation according to the visual image data, and extract an instruction marker.

[0077] A weakly supervised instruction marker positioning module is configured to perform a weakly supervised instruction marker positioning task according to the map representation and the instruction marker, and construct an instruction marker map.

[0078] A sub-target position prediction module is configured to predict a sub-target point position and a navigation progress, and perform a corresponding action.

[0079] The visual language navigation system of the embodiment can perform the visual language navigation method based on weakly supervised instruction marker positioning provided by the method embodiment of the present application, and can perform any combination of the steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0080] The embodiment also provides a visual language navigation device, comprising:

[0081] At least one processor;

[0082] At least one memory configured to store at least one program;

[0083] When the at least one program is executed by the at least one processor, the at least one processor is caused to implement the method. Figure 1 The method.

[0084] The visual language navigation device of the embodiment can execute the visual language navigation method based on weakly supervised instruction marker positioning provided by the method embodiment of the application, and can execute the implementation steps of any combination of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0085] The application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the method shown in the method embodiment of the application. Figure 1

[0086] The embodiment also provides a storage medium storing instructions or programs executable for the visual language navigation method based on weakly supervised instruction marker positioning provided by the method embodiment of the application. When the instructions or programs are executed, the implementation steps of any combination of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method are possessed.

[0087] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the application are provided by way of example, and the purpose is to provide a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0088] In addition, although the application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It can also be understood that detailed discussion of the actual implementation of each module is unnecessary for understanding the application. Rather, given the properties, functions and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary skill without undue experimentation, using ordinary skill. It can also be understood that the disclosed specific concepts are merely illustrative and are not intended to limit the scope of the application, the scope of the application being determined by the full scope of the appended claims and their equivalents. ​

[0089] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0090] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be specifically embodied in any computer readable medium for use by or in conjunction with an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from an instruction execution system, device or apparatus. For the purpose of this specification, the "computer readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in conjunction with an instruction execution system, device or apparatus, or in conjunction with these instruction execution systems, devices or apparatus.

[0091] More specific examples (a non-exhaustive list) of the computer readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or processing, if necessary, in other suitable ways, and then stored in a computer memory.

[0092] It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or their equivalents, can be employed for implementation: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0093] In the above description of the present specification, the description of the terms "one embodiment / one example", "another embodiment / another example", or "certain embodiments / certain examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0094] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the claims and their equivalents.

[0095] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present application.

Claims

1. A visual language navigation method, characterized in that, Includes the following steps: Acquire visual image data of the intelligent agent in the environment; the visual image data includes RGB images and depth images; A map representation is constructed based on visual image data, and instruction markers are extracted. The temporal order of each instruction marker is obtained. The earlier the instruction marker appears, the closer it is to the starting point of the path. All instruction markers are basically evenly distributed near the path. Based on map representation and command markers, perform weakly supervised command marker localization tasks and construct a command marker map; Predict the location of sub-target points and navigation progress, and execute corresponding actions; The task of performing weakly supervised instruction marker localization and constructing an instruction marker map includes: Based on the actual path corresponding to the instruction, a coarse-grained instruction marker positioning label is constructed. For an instruction with N markers, the path is evenly divided into N segments. For the nth instruction marker, a coarse-grained marker positioning label is calculated as a supervision for the positioning of the instruction marker. Calculate the shortest straight-line distance between each location on the map and the actual path, normalize the shortest straight-line distances of all locations, and finally perform a softmax operation in the spatial dimension to obtain coarse-grained positioning labels for multiple landmarks. Calculate the cosine similarity between the features of the command markers and the features of each location on the map to obtain the predicted positioning result; The predicted positioning results are used as weights and multiplied with the landmark features to obtain the command landmark map.

2. The visual language navigation method according to claim 1, characterized in that, The map representation is used to map the information of the RGB image onto the 2D map; the map representation is a semantic map representing object category information, or a high-dimensional feature map obtained by the object through a deep neural network and mapped onto the 2D map.

3. The visual language navigation method according to claim 2, characterized in that, When the map representation is a semantic map representing object category information, the step of constructing the map representation based on visual image data includes: Objects in RGB images are detected using a pre-defined semantic segmentation model to represent semantic information. When constructing the map, the semantic segmentation results of the semantic segmentation model are selected as the basis, and the spatial three-dimensional coordinates of each pixel in the image are obtained through camera parameters and depth information in the depth image. The spatial three-dimensional coordinates are transformed into map coordinates, thereby mapping each segmentation result to the corresponding location on the map, resulting in a semantic map.

4. The visual language navigation method according to claim 1, characterized in that, The extraction instruction marker includes: Use the instruction parser to parse the category of the flags contained in the instruction; The instruction parser is either a large language model or a part-of-speech analysis tool.

5. The visual language navigation method according to claim 1, characterized in that, The predicted positioning result is obtained by calculating the cosine similarity between the feature of the command marker and the feature of each location on the map, including: For the text of the nth marker, its marker text features are calculated using a text encoder. ; The cosine similarity between the text features and the map representation of each location feature is calculated to obtain the localization prediction result of the landmark. ; The location prediction loss is calculated using the following formula: In the formula, This indicates a coarse-grained marker positioning label. Indicates the predicted positioning result. The number of markers.

6. The visual language navigation method according to claim 1, characterized in that, The prediction of sub-target point locations and navigation progress, and the execution of corresponding actions, include: The map representation, the command marker map, and the command features are input into a recurrent neural network to obtain the current state features of the agent. Input the current state features into the sub-target point predictor to predict the relative coordinate offset of the sub-target point from the current position; Based on the predicted relative coordinate offset, the sub-target point positions are marked on the map, and the next action of the agent is obtained and executed through a preset path planning algorithm. Input the current state characteristics into the navigation progress predictor to predict the navigation progress under the current state. If the predicted progress is greater than a preset threshold, end the current navigation.

7. A visual language navigation system, applied to the method described in any one of claims 1-6, characterized in that, include: The data acquisition module is used to acquire visual image data of the intelligent agent in the environment; the visual image data includes RGB images and depth images; The representation construction module is used to construct map representations based on visual image data, extract command markers, and obtain the temporal order of each command marker. The command marker that appears earlier in the temporal order is closer to the starting point of the path. The weakly supervised command marker localization module is used to perform the weakly supervised command marker localization task based on map representation and command markers, and to construct a command marker map; The sub-target location prediction module is used to predict the location of sub-target points and navigation progress, and execute corresponding actions.

8. A visual language navigation device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1-6.

9. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-6.

Images