Target-oriented visual language navigation indoor data set preparation method and visual language navigation method

Abstract

The invention relates to a target-oriented visual language navigation indoor data set preparation method and a visual language navigation method. The preparation method comprises the following steps: acquiring real environment information and a plurality of panoramic images to be processed, and constructing a discrete topological map; rotating the to-be-processed panorama and carrying out object labeling; covering the marked panorama by using the color, and extracting position and size information of a covered area; generating a path-target object-instruction pair training sample; respectively extracting image features of the rotated panorama and the marked object picture to obtain image features of each navigation point and an included object; and based on the path-target object-instruction pair training samples and the corresponding panorama image features, constructing a real indoor target-oriented visual language navigation data set. Compared with the prior art, the method has the advantages of quickly constructing a complete target-oriented data set, providing an effective tool for enriching the data set in the field of visual language navigation and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of visual language navigation, and in particular to a method for preparing a target-oriented visual language navigation indoor data set and a visual language navigation method. Background Art

[0002] In recent years, deep learning has achieved breakthroughs in image recognition, natural language processing, and speech recognition. In addition, as robotics technology matures, it has excellent spatial environment perception capabilities, can understand and communicate in natural language, and interact with the real world in a multimodal manner. Finally, embodied intelligence tasks that can complete complex tasks in real physical scenes based on these capabilities have been proposed.

[0003] As an important branch of embodied intelligence, the visual language navigation task aims to build an embedded intelligent agent that can communicate with humans in natural language and navigate in a real 3D environment. It can learn general navigation rules from the instruction-path dataset, thus having the ability to explore the environment autonomously, and can be used to build service robots. Goal-oriented visual language navigation requires the robot to process visual and language information in a discrete environment to complete the task of finding object targets.

[0004] One of the important challenges of visual language navigation is that most of its research relies on simulators, such as Matterport3D and Habitat simulation environments, which lack physical verification. There is a big difference between the high-level discrete action space learned by the agent and the low-level continuous action space of the robot. Although a few studies have been verified in real environments, in order to obtain more accurate and reliable physical verification results, it is necessary to establish a complete and adaptable visual language navigation dataset. However, there is currently no complete method for preparing a goal-oriented visual language navigation dataset. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a goal-oriented visual language navigation indoor dataset preparation method and a visual language navigation method, which can establish a complete and adaptable visual language navigation dataset, efficiently complete the collection, annotation and organization of the dataset, and can be efficiently reused.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, a method for preparing a target-oriented visual language navigation indoor dataset is provided, comprising the following steps: S1, obtaining real environment information and a plurality of panoramas to be processed, and constructing a discrete topological map, wherein the discrete topological map comprises a plurality of predefined navigation points, and each navigation point corresponds to a panorama; S2, rotating the panorama to be processed, and annotating the rotated panorama with objects to obtain an annotated object image, and automatically generating first object annotation information; S3, according to the first object annotation information, covering the annotated panorama with a preset color, extracting the position and size information of the covered area, and the position and size information is used to train a preset visual language navigation model; S4, according to The discrete topological map and the second object annotation information are used to generate path-target object-instruction pair training samples, and each generated path is manually annotated to obtain the instruction annotation information corresponding to each generated path, and the path end points that meet the preset object detection conditions are retained. The second object annotation information is obtained by integrating the first object annotation information; S5, image feature extraction is performed on the rotated panoramic image and the annotated object image respectively to obtain the image features of each navigation point in the discrete topological map and the object contained in the navigation point; S6, encoding the instruction annotation information, and constructing a real indoor target-oriented visual language navigation dataset based on the path-target object-instruction pair training samples and the corresponding panoramic image features.

[0008] As a preferred technical solution, the real environment information and multiple panoramic images to be processed are obtained using a preset real environment data acquisition hardware platform. The real environment data acquisition hardware platform includes at least a chassis and a panoramic camera. The chassis is used for mapping and autonomous positioning and navigation, and collecting real environment information. The panoramic camera is used to take panoramic images, and the shooting angle includes 360 degrees in the horizontal direction and 90 degrees in the vertical direction.

[0009] As a preferred technical solution, in step S2, the panorama to be processed is rotated, and the specific process includes: based on coordinate processing, all the panorama to be processed are rotated to the 0° direction of the global map.

[0010] As a preferred technical solution, in step S2, the object labeling rules include: uniformly defining a list of labelable objects, ignoring objects outside 3m, and labeling the objects as "object category_ab", wherein the object category is limited to the list of labelable objects, "a" represents the room code, and "b" represents the bth identical object category that has been labeled in room a.

[0011] As a preferred technical solution, step S3 specifically includes: using a preset special RGB color to cover each panoramic image, dividing the processed panoramic image into multiple sub-images, and classifying and extracting the position and size information of the covered area in each sub-image according to the size of the object.

[0012] As a preferred technical solution, the multiple sub-images are distributed at three pitch angles of [-30°, 0°, 30°], corresponding to the downward, horizontal and upward viewing angles respectively; at each pitch angle, 12 heading angles are evenly distributed along the horizontal direction, and the heading angle starts from 0° and ends at 330°. The resolution of each sub-image is 640 pixels wide and 480 pixels high, and the vertical field of view of each sub-image is 60°.

[0013] As a preferred technical solution, in step S4, the generated path meets four specifications, specifically including: the starting point and the end point of the path are in different rooms, and the rooms are obtained by dividing the indoor area when constructing the discrete topological map; the length of the path is greater than the preset shortest length of the path; the number of navigation points passed by the path is between the preset minimum number of points and the maximum number of points; the navigation point at the end point contains at least two objects, and the target object appears only once.

[0014] As a preferred technical solution, in step S4, when manually labeling each generated path, at least three instructions are labeled for each path at the same time.

[0015] As a preferred technical solution, in step S5, the image features are extracted using the ViT-Base model pre-trained on ImageNet.

[0016] According to a second aspect of the present invention, a visual language navigation method is provided, comprising at least the following steps: obtaining current natural language instructions and environmental images; inputting the natural language instructions and environmental images into a trained visual language navigation model to obtain the next navigation action; wherein the visual language navigation model uses the real indoor target-oriented visual language navigation dataset for migration training.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. The present invention proposes a complete process for constructing a goal-oriented visual language navigation indoor dataset. The method is concise and can be reused efficiently. It can quickly construct a complete and adaptable goal-oriented dataset, provide an effective tool for enriching the dataset in the field of visual language navigation, and help improve the accuracy of entity verification.

[0019] 2. The present invention uses a preset special RGB color to cover each panorama, divides the processed panorama into multiple sub-images, and classifies objects of different sizes, so that the position and size information extracted from the sub-images is more accurate, thereby improving the accuracy of the data set;

[0020] 3. The present invention sets an efficient annotation process for manual object annotation and path annotation. The annotator can quickly view the location of the navigation point to be annotated, the panoramic image of the path and the target object at the end point, which can save a lot of time;

[0021] 4. The present invention targets each path-target object-instruction pair training sample and manually labels each generated path. The manual labeling method is used to label the instructions, which has higher accuracy than the existing labeling. At the same time, each path is labeled with at least three instructions with different styles, which can improve the richness of the input instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic flow chart of the method provided by the present invention;

[0023] Figure 2 A discrete topological map constructed in an embodiment of the present invention;

[0024] Figure 3 A three-dimensional reconstructed map constructed in an embodiment of the present invention;

[0025] Figure 4 It is a schematic diagram of the panorama effect after being covered by special RGB colors in an embodiment of the present invention;

[0026] Figure 5 Schematic diagram of the real environment and the generated path-target object in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0028] Example

[0029] like Figure 1 As shown, this embodiment provides a method for preparing a goal-oriented visual language navigation indoor dataset, comprising the following steps:

[0030] Step S1, obtaining real environment information and a plurality of panorama images to be processed, and constructing a discrete topology map, wherein the discrete topology map includes a plurality of predefined navigation points, and each navigation point corresponds to a panorama image.

[0031] Specifically, after collecting real environment information and multiple panoramas to be processed, a discrete topological map is constructed, the indoor area is divided into different rooms, and navigable points and connectivity are defined in the map. The originally continuous real environment is converted into a discrete topological map that can be used for navigation. Each navigable point corresponds to a panoramic picture.

[0032] In practice, the preset real environment data acquisition hardware platform can be used to obtain real environment information and multiple panoramic images to be processed. Optionally, the real environment data acquisition hardware platform consists of an AGV mobile chassis with autonomous positioning and navigation and a panoramic camera, which is used for SLAM (Simultaneous Localization and Mapping) mapping and construction of environmental discrete topology maps. Among them, the AGV chassis is equipped with a laser radar and an industrial computer for mapping and autonomous positioning and navigation capabilities, and can collect real environment information. The panoramic camera is used to take panoramic images with a shooting angle of 360 degrees horizontally and 90 degrees vertically. The panoramic camera is fixed to the AGV chassis with a tripod, connected to the industrial computer via a data cable, and the taken photos are stored in the industrial computer hard disk. The environmental information includes the connectivity of each topological point (i.e., navigation point) of the defined discrete topological map and the room in which it is located. In one embodiment, the constructed discrete environment topology map is as follows Figure 2 As shown in the figure, the connectivity of the navigation points is displayed in the form of straight lines. The roomID in the figure represents the room code. Furthermore, the corresponding visual 3D reconstruction map can be obtained, such as Figure 3 as shown to enhance the intuitiveness of the constructed map.

[0033] Step S2, rotating the panorama to be processed, and annotating the objects in the rotated panorama to obtain an annotated object image, and automatically generating first object annotation information.

[0034] Performing panoramic image rotation in preprocessing can reduce the probability of model error. The panoramic image is rotated according to the robot's orientation using a panoramic image rotation algorithm so that the image is aligned to the 0° direction of the map coordinate system. Since the scale of the specific style data set collected on the spot is not large, it can be preprocessed before training to rotate it to a uniform direction. Specifically:

[0035] Based on coordinate processing, all panoramas to be processed are rotated to the 0° direction of the global map. Objects are annotated in the rotated panoramas. Each object needs to be globally uniquely represented, and the global representation of the same object photographed at different navigation points must be consistent.

[0036] Panoramic object labeling is done manually. The optional labeling tool is labelimg, the labeling format is CreateML, and the object labeling rules are as follows: uniformly define the list of labelable objects, ignore objects outside 3m, and the labeling name is "object category_ab", where the object category is limited to the above list, "a" represents the room code, and "b" represents the bth identical object category that has been labeled in room a. Since each object has a unique labeling name, it can be sorted and numbered in sequence.

[0037] In this step, the first object annotation information is automatically generated by the annotation tool.

[0038] Step S3, according to the first object annotation information, using a preset color to cover the annotated panoramic image, extracting the position and size information of the covered area, and the position and size information is used to train a preset visual language navigation model.

[0039] Specifically, each panorama is covered with a preset special RGB color, the processed panorama is divided into multiple sub-images, and the position and size information of the covered area in each sub-image is extracted by classification according to the size of the object. Figure 4 As shown in the figure, objects are covered with special RGB colors according to the manual annotation information. The panorama can be divided at intervals of 30 degrees in both horizontal and vertical directions to obtain 36 color RGB images with a resolution of 640×480 (i.e., 36 sub-images). From this, the covered area can be fitted into a rectangle in the split sub-images to obtain the size and position information of the object. For large objects such as cabinets, the conditions for seeing the object in the screening sub-images are more stringent. The center point of the fitted rectangle must be greater than a sufficiently large threshold from the boundary to be recorded in the list. The threshold corresponding to small objects is smaller and can be defined by yourself.

[0040] Optionally, the 36 sub-images divided from a panoramic image are distributed in three pitch angles of [-30°, 0°, 30°], corresponding to the robot's top, horizontal, and top view angles, respectively. At each pitch angle, 12 heading angles are evenly distributed along the horizontal direction, starting from 0° and ending at 330°. The resolution of the sub-image is fixed to 640 pixels wide and 480 pixels high, and the vertical field of view of each sub-image is set to 60°.

[0041] Step S4, based on the discrete topological map and the second object annotation information, generate path-target object-instruction pair training samples, manually annotate each generated path, obtain the instruction annotation information corresponding to each generated path, retain the path end points that meet the preset object detection conditions, and the second object annotation information is obtained by integrating the first object annotation information.

[0042] In this step, the second object annotation information is obtained by integrating the first object annotation information. Specifically, the first object annotation information is automatically generated, and each panorama corresponds to a file. The second object annotation information integrates all panorama annotation information in the same environment into one file, including the category, location, and size information of the object. In addition, the path endpoints that meet the preset object detection conditions are retained in order to filter out the path endpoints that are not suitable for object detection.

[0043] Optionally, to ensure the quality of the generated path, the generated path (ie, the navigation path) meets four specifications, including:

[0044] The starting point and the end point of the path are in different rooms, and the division of the rooms has been completed when constructing the discrete topological map in step S1;

[0045] The length of the path is greater than the preset minimum path length;

[0046] The number of navigation points that the path passes through is between the preset minimum and maximum points;

[0047] The navigation point at the end point contains at least two objects, and the target object appears only once.

[0048] In one embodiment, the real environment and the generated paths and target objects are as follows: Figure 5 As shown. Figure 2 The topological map of Figure ⑤ and the 3D reconstruction map of the current environment, Figure 5 Part ① is near the restaurant table. Figure 5 Part ② is the front door of the restaurant near the meeting room. Figure 5 Part ③ is the corridor near the front door of the restaurant. Figure 5 Part ④ is the corridor near the back door of the conference room. Figure 5 Part ⑤ is near the target object in the conference room. In order to enrich the sample style and reduce the overfitting of the model to the training samples, the same path is annotated by three different annotators, that is, three instructions are annotated for each path with different styles, so as to construct the path-target object-instruction sample pairs that can be used for visual language navigation model training. When annotating, the task requirements are described in Chinese, and then the cloud translation interface is used to batch translate and generate the corresponding English instructions.

[0049] Step S5, extracting image features from the rotated panorama and the annotated object images, respectively, to obtain image features of each navigation point and the object contained in the navigation point in the discrete topological map. Optionally, this step uses a ViT-Base model pre-trained on the ImageNet dataset to extract image features.

[0050] Step S6, encoding the instruction annotation information of the path, and constructing a real indoor target-oriented visual language navigation dataset based on the path-target object-instruction pair training samples and the corresponding panoramic image features. Optionally, this step uses the BERT model to encode the instruction annotation information.

[0051] Furthermore, this embodiment also provides a visual language navigation method, which, after acquiring the current natural language instructions and environmental images, inputs the natural language instructions and environmental images into a trained visual language navigation model to obtain the next navigation action.

[0052] Optionally, the visual language navigation model consists of two modules: a topological mapping module and a global action planning module. The topological mapping module is used to update the environment topology map based on historical information and current input, and the global action planning module is used to fuse dual scales and predict the next action. Specifically, at the beginning of the task, the visual language navigation model accepts natural language instructions. The topological mapping module builds a topological map by adding newly observed locations in the map and updating the visual representation of nodes. At each step, the global action planning module predicts the next location in the map or stops the action. It dynamically fuses the fine-grained representation of the current location and the coarse-grained representation of the global map and uses Transformer to capture cross-modal visual language relationships.

[0053] The visual language navigation model uses the indoor dataset prepared by the above method for transfer training. In the process of transfer training of the visual language navigation model, a smaller learning rate, a smaller batch size, and an increased proportion of randomly discarded features can prevent the model from overfitting.

[0054] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for preparing a goal-oriented visual language navigation indoor dataset, characterized in that: The following steps are involved: S1, obtaining real environment information and a plurality of panorama images to be processed, and constructing a discrete topology map, wherein the discrete topology map includes a plurality of predefined navigation points, each of which corresponds to a panorama image; S2, rotating the panorama to be processed, and marking objects in the rotated panorama to obtain a marked object image, and automatically generating first object marking information; S3, covering the annotated panorama with a preset color according to the first object annotation information, extracting position and size information of the covered area, wherein the position and size information is used to train a preset visual language navigation model; S4, generating path-target object-instruction pair training samples according to the discrete topological map and the second object annotation information, manually annotating each generated path, obtaining instruction annotation information corresponding to each generated path, retaining the path end points that meet the preset object detection conditions, and the second object annotation information is obtained by integrating the first object annotation information; S5, extracting image features from the rotated panoramic image and the annotated object image respectively, to obtain image features of each navigation point in the discrete topological map and the object contained in the navigation point; S6, encoding the instruction annotation information, and constructing a real indoor target-oriented visual language navigation dataset based on the path-target object-instruction pair training samples and corresponding panoramic image features.

2. The method for preparing a goal-oriented visual language navigation indoor dataset according to claim 1, characterized in that: The real environment information and multiple panoramic images to be processed are obtained using a preset real environment data acquisition hardware platform, which includes at least a chassis and a panoramic camera. The chassis is used for mapping and autonomous positioning and navigation, and for collecting real environment information. The panoramic camera is used for taking panoramic images, and the shooting angle includes 360 degrees in the horizontal direction and 90 degrees in the vertical direction.

3. The method for preparing a goal-oriented visual language navigation indoor dataset according to claim 1, characterized in that: In step S2, the panoramic images to be processed are rotated, and the specific process includes: based on coordinate processing, all the panoramic images to be processed are rotated to the 0° direction of the global map.

4. The method for preparing a goal-oriented visual language navigation indoor dataset according to claim 1, characterized in that: In step S2, the object labeling rules include: A list of labelable objects is uniformly defined, objects outside 3m are ignored, and the labeling name is "object category_ab", where the object category is limited to the list of labelable objects, "a" represents the room code, and "b" represents the bth identical object category that has been labeled in room a.

5. The method for preparing a goal-oriented visual language navigation indoor dataset according to claim 1, characterized in that: Step S3 specifically includes: using a preset special RGB color to cover each panoramic image, dividing the processed panoramic image into multiple sub-images, and classifying and extracting the position and size information of the covered area in each sub-image according to the size of the object.

6. The method for preparing a goal-oriented visual language navigation indoor dataset according to claim 5, characterized in that: The multiple sub-images are distributed at three pitch angles of [-30°, 0°, 30°], corresponding to the downward, horizontal and upward viewing angles respectively; at each pitch angle, 12 heading angles are evenly distributed along the horizontal direction, and the heading angle starts from 0° and ends at 330°. The resolution of each sub-image is 640 pixels wide and 480 pixels high, and the vertical field of view of each sub-image is 60°.

7. The method for preparing a goal-oriented visual language navigation indoor dataset according to claim 1, characterized in that: In step S4, the generated path meets four specifications, including: The starting point and the end point of the path are in different rooms, and the rooms are obtained by dividing the indoor area when constructing the discrete topological map; The length of the path is greater than the preset minimum path length; The number of navigation points that the path passes through is between the preset minimum and maximum points; The navigation point at the end point contains at least two objects, and the target object appears only once.

8. The method for preparing a goal-oriented visual language navigation indoor dataset according to claim 1, characterized in that: In step S4, when manually labeling each generated path, at least three instructions are labeled for each path at the same time.

9. The method for preparing a goal-oriented visual language navigation indoor dataset according to claim 1, characterized in that: In step S5, the image features are extracted using the ViT-Base model pre-trained on ImageNet.

10. A visual language navigation method, characterized in that: At least the following steps are included: Get the current natural language instructions and environment image; Inputting the natural language instruction and the environment image into a trained visual language navigation model to obtain the next navigation action; The visual language navigation model is transferred and trained using a real indoor target-oriented visual language navigation dataset as described in any one of claims 1 to 9.

Images