Visual language navigation data enhancement method and device for indoor mobile service robot and storage medium

Abstract

The invention relates to a visual language navigation data enhancement method and device for an indoor mobile service robot and a storage medium. Training an instruction generation model by utilizing a visual language navigation data set; generating a new environment collection first navigation path according to the simulation environment; generating a first navigation instruction by using the trained instruction generation model according to the first navigation path, expanding the simulation environment data set and training a preset navigation model; synchronous positioning and mapping are carried out in the actual deployment environment of the indoor mobile service robot, and a topological navigation map is constructed according to positioning and mapping results; collecting a panorama in an actual deployment environment according to the topological navigation map; and collecting a second navigation path in the actual deployment environment, generating a second navigation instruction by using the trained instruction generation model, expanding the deployment environment data set, and finely adjusting the trained navigation model. Compared with the prior art, the method has the advantages that the generalization of the navigation model and the performance of the navigation model in a deployment environment are improved, and the like.

Description

Technical Field

[0001] The present invention relates to the field of visual language navigation technology, and in particular to a visual language navigation data enhancement method, device and storage medium for an indoor mobile service robot. Background Art

[0002] In the field of visual language navigation, dataset annotation is a time-consuming and complex task. When annotating language instructions, it is necessary to consider both visual and language information, as well as ensure that the language description is consistent with the actual environment. This process requires professionals to complete and often requires subjective judgment, which increases the uncertainty of annotation. In the complex deployment environment of indoor mobile service robots, ensuring the accuracy of annotated language instructions is crucial for training reliable navigation models.

[0003] The currently available visual language navigation datasets are relatively small, which limits the training of deep learning models. In the field of computer vision, the COCO (Common Objects in Context) dataset is one of the widely used datasets, mainly used for tasks such as object detection, segmentation, and image annotation. The COCO dataset contains 118,287 training images, 5,000 validation images, and 40,670 test images, with a total of about 1.23 million object instances annotated, covering 80 common categories in daily life. Each image contains an average of 7.2 object instances, and provides detailed bounding boxes and segmentation masks, as well as key point detection annotations. In the field of natural language processing, the scale of datasets ranges from tens of thousands to millions, and some even exceed 100 million. For example, the SQuAD (Stanford Question Answering Dataset) 2.0 dataset contains 150,000 question-answer pairs, and the OpenAI GPT-3 training data contains hundreds of GB of text data from various sources on the Internet. BERT training data uses large-scale corpora such as Wikipedia (about 250 million words) and BooksCorpus (about 800 million words). The Room-to-Room (R2R) dataset, which is commonly used in the field of visual language navigation, only contains 21,567 navigation instructions in 90 building scenes. Small-scale datasets are difficult to cover the diverse scenarios that may be encountered in indoor navigation, which makes the generalization ability of the trained model in the real world very limited. In addition, due to the diversity of indoor environments, a single dataset often cannot fully cover all navigation scenes of different styles, thus affecting the comprehensiveness and adaptability of the model. Summary of the invention

[0004] The purpose of the present invention is to overcome the defect of limited generalization ability of the model trained with a single data set in the above-mentioned prior art and to provide a visual language navigation data enhancement method, device and storage medium for indoor mobile service robots, which can provide richer and more diverse training data for deep learning models and effectively improve the comprehensiveness and adaptability of the model.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to a first aspect of the present invention, a method for enhancing visual language navigation data for an indoor mobile service robot is provided, comprising the following steps: S1, training a preset instruction generation model using a pre-acquired visual language navigation data set; S2, generating a new environment according to a preset simulation environment; S3, in the new environment, collecting a first navigation path; S4, generating a first navigation instruction using the instruction generation model trained in S1 according to the first navigation path, and obtaining an expanded simulation environment data set; S5, training a preset navigation model using the expanded simulation environment data set; S6, performing synchronous positioning and map construction in an actual deployment environment of the indoor mobile service robot, and constructing a topological navigation map according to the positioning and mapping results; S7, collecting a panoramic map in the actual deployment environment according to the topological navigation map; S8, collecting a second navigation path in the actual deployment environment according to the panoramic map, and generating a second navigation instruction using the instruction generation model trained in S1, and obtaining a deployment environment data set; S9, fine-tuning the navigation model trained in S5 using the deployment environment data set to complete the visual language navigation data enhancement.

[0007] As a preferred technical solution, in S4 and S8, the first navigation instruction and the second navigation instruction are automatically generated using a trained instruction generation model.

[0008] As a preferred technical solution, the instruction generation model is constructed using a sequence-to-sequence based Transformer structure, wherein the Transformer structure includes a connected spatial encoder and a temporal encoder, wherein the spatial encoder is used to fuse the spatial information of each time stage, and the temporal encoder is used to learn the intrinsic connection between different time stages after the spatial encoder completes the fusion.

[0009] As a preferred technical solution, the Transformer structure also includes a text decoder connected to the time encoder, and the text encoder includes an instruction generation word predictor for performing prediction tasks.

[0010] As a preferred technical solution, the instruction generation word predictor is optimized using a preset cross entropy loss, and the expression of the cross entropy loss is:

[0011]

[0012] In the formula, θ is the preset parameter, is a text instruction in the data set, l is the number of words contained in the text instruction, l≤L, f θ (·) is the probability that the target word appears in the i-th position of the instruction, predicted by the instruction generation model based on the action A, the environment observation E, and the first i-1 words in the instruction.

[0013] As a preferred technical solution, before the image to be processed is input into the instruction generation model, the image to be processed is encoded using a trained image encoder to extract corresponding image features, and the image encoder is trained using a contrastive learning method.

[0014] As a preferred technical solution, S2 specifically includes: using an environmental visual feature masking method to generate a new environment based on visual feature masking that is consistent with the view and viewpoint.

[0015] As a preferred technical solution, the view feature f observed from the new environment t ' ,i From the original feature f t,i and the ambient occlusion mask ξ E The Hadamard product is obtained as follows:

[0016] f t ' ,i =f t,i ⊙ξ E

[0017]

[0018] Where E represents the preset simulation environment, and the ambient occlusion mask ξ E Each element in are all samples of a random variable, Ber(·) represents Bernoulli Distribution, and p is the probability of covering the view feature.

[0019] According to a second aspect of the present invention, there is provided a visual language navigation data enhancement device for an indoor mobile service robot, comprising a memory, a processor, and a program stored in the memory, wherein the method described is implemented when the processor executes the program.

[0020] According to a third aspect of the present invention, there is provided a storage medium having a program stored thereon, wherein the program implements the method described above when executed.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. The data enhancement method provided by the present invention first trains the instruction generation model on the existing visual language navigation dataset in a simulation environment, trains the navigation model on the expanded simulation environment dataset, and then fine-tunes the trained navigation model on the deployment environment dataset. This method only consumes a small amount of cost to collect the dataset of the actual deployment environment, and can effectively improve the generalization of the navigation model in the deployment environment;

[0023] 2. In the present invention, the second navigation path collected in the actual deployment environment of the indoor mobile service robot uses the trained instruction generation model to automatically generate the second navigation instruction, which can effectively solve the problem that it takes a lot of manpower to collect visual language navigation data sets. It has been verified by experiments that the performance of fine-tuning the navigation model using the instructions generated by the instruction generation model in the present invention is close to that of fine-tuning the navigation model using manually labeled instructions;

[0024] 3. The present invention uses an image encoder trained by contrastive learning to encode the input image, which can represent the image and text in an embedding space, effectively integrate the information of the two modalities of image and text, and improve the accuracy of instruction generation;

[0025] 4. The present invention adopts an environmental visual feature masking method to generate a new environment based on visual feature shielding consistent with the view and viewpoint. The environmental visual feature enhancement method and navigation model require fewer computing resources, which is conducive to promotion and deployment in edge computing devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic flow chart of the method provided by the present invention;

[0027] Figure 2 Schematic diagram of the position, heading and elevation angle of an intelligent agent in the Matterport3D simulator in an embodiment of the present invention;

[0028] Figure 3 A schematic diagram of a panoramic image divided into 36 sub-images in an embodiment of the present invention;

[0029] Figure 4 The changes of the main performance indicators during the training of the instruction generation model in the embodiment of the present invention;

[0030] Figure 5 The definition of the heading angle in the Matterport3D coordinate system (left) and the ROS coordinate system (right) in an embodiment of the present invention (top view);

[0031] Figure 6 An example of a command-navigation path pair of a data set collected in a deployment environment in an embodiment of the present invention;

[0032] Figure 7The main performance indicator changes of the model process fine-tuning using the data set collected in the deployment environment in the embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0034] Example

[0035] Data enhancement method is a method of generating more training samples by transforming and expanding existing data. In the field of visual language navigation, the application of data enhancement method can alleviate the problems of difficult data set annotation and insufficient data volume, and provide richer and more diverse training data for deep learning models, thereby improving their performance. The method provided by the present invention is aimed at indoor mobile service robots and performs data enhancement on visual language navigation data. The method first trains an instruction generation model on an existing visual language navigation data set in a simulation environment, and trains a navigation model on an expanded simulation environment data set. Then, in a deployment environment, the trained navigation model is fine-tuned on the deployment environment data set. The method only consumes a small amount of cost to collect the data set of the actual deployment environment, which can improve the generalization of the visual language navigation model and improve the performance of the navigation model in the robot deployment environment. Figure 1 As shown, this embodiment provides a visual language navigation data enhancement method for an indoor mobile service robot.

[0036] This embodiment uses the Matterport 3D simulator as the simulation environment of the experiment, and uses the Room-to-Room (hereinafter referred to as R2R) dataset as the basic dataset (i.e., the pre-acquired visual language navigation dataset) to train the model.

[0037] In the Matterport3D simulator, a virtual agent is allowed to virtually "move" throughout the scene by adopting the same pose as the panoramic viewpoint. Figure 2 As shown in Figure 1, the pose of the agent consists of a 3D position v∈V, a heading ψ∈[0,2π) and an elevation angle α∈[-0.5π,0.5π], where V is a set of 3D points corresponding to the panoramic viewpoint in the scene. At each step t, the simulator outputs an RGB image observation o t , corresponding to the agent’s first-person camera view.

[0038] At each step t, the simulator also outputs a set of viewpoints that can be reached in the next step The agent selects a new viewpoint v among the reachable viewpoints t+1 ∈W t+1, and specify the camera heading change (Δψ t+1 ) and elevation change (Δα t+1 ) to interact with the simulator. Therefore, the total number of possible actions that the agent can take is determined.

[0039] In order to determine W t+1 , the simulator will generate a panoramic viewpoint weighted undirected graph for each scene, namely G =<V,E> ,The weight of the edge reflects the straight line distance between different viewpoints.

[0040] Given a navigation graph G, the set of viewpoints that can be reached in the next step is given by the following formula:

[0041] W t+1 = {v t}∪{v i ∈V| <v t ,v i >∈E∧v i ∈P t} (1)

[0042] In the formula, v t is the current viewpoint, v i is the i-th 3D point, P t It is the spatial area enclosed by the left and right sides of the camera viewing cone at the tth step.

[0043] The R2R task requires the agent to navigate from the starting position to the target position in the Matterport3D simulator according to natural language instructions. At the beginning of the task, the agent will receive a natural language instruction. Where L is the length of the instruction, x i is a word token. The agent first observes the initial RGB image o0, which is determined by the agent's initial posture and contains the three-dimensional position, heading, and elevation angle s0=<v0,ψ0,α0> Triplet. The agent needs to perform a series of actions <s0,a0,s1,a1,…,s T ,a T >To complete the task, the agent performs each action a t After that, a new position s is reached. t+1 = <v t+1 ,ψ t+1 ,α t+1 >. The task ends when the agent chooses a special stop action. If the action sequence brings the agent close to the predetermined target position v * , the task is completed successfully.

[0044] Assume that the task goal is to learn the mapping X→Y from X to Y through paired data {(X,Y)} and unpaired data {X′}. In this case, the back-translation method refers to first using paired data {(X,Y)} to train a forward model P X→Y and the backward model P Y→X , and then use the backward model P Y→X Generate additional data X′ from unpaired data Y′, and use the paired data {(X′,Y′)} as additional training data to fine-tune the forward model P X→Y .

[0045] In the visual language navigation task, the instruction generation-navigation path generation method uses the idea of ​​back translation. The forward model is a navigation model P E,d→r , it generates the correct navigation route r according to the given instruction d in the environment E. The backward model is an instruction generation model P E,r→d , which generates instructions d in the environment E according to the given navigation route r.

[0046] Based on this, the specific execution process of the method provided in this embodiment is as follows:

[0047] Step S1, using the pre-acquired visual language navigation data set to train a preset command generation model. The model can generate corresponding navigation commands based on visual input, providing a basis for the subsequent generation of navigation command data, specifically:

[0048] The connection graph G = {P, ξ} is generated using the Matterport3D simulator, where P represents the navigable viewpoints and ξ represents the connections between these viewpoints. In the R2R dataset, the data is represented as a pair of trajectories τ = {p1, p2, …, p N} and instruction I={w1,w2,…,w L}, where p i and w i Represent the visited nodes and words respectively, N and L represent the length of the path and instruction respectively.

[0049] At each step, the robot can observe a panoramic view of the environment, which includes three views, each of which is divided into 12 images of 30°, and the resolution of each image is 640×480. v and d o Represent the dimensions of image features and direction features respectively. Specifically, each step of the robot's path will be located on one of the images. This image and its offset direction are defined as the action feature A = {V a ; γ a},in represents N-dimensional image features, Represents the directional feature. Similarly, the environmental feature E={V e ; γ e}By Panorama Feature Collection and direction feature set Composition, where M represents the number of images each panorama is divided into.

[0050] In this embodiment, the panoramic image is divided into 36 sub-images, and image features are extracted respectively, such as Figure 3 shown.

[0051] Optionally, before feeding the image into the model, the image is encoded using an image encoder trained using contrastive learning. Specifically, the CLIP image encoder trained using contrastive learning is used to extract image features. CLIP jointly trains the image encoder and the text encoder to predict the matching degree of the image and the text, and projects the related image-text pairs into a shared embedding space. In this space, the distance between related image and text pairs will be very close, while the distance between unrelated image and text pairs will be far. The main part of the image encoder adopts the Transformer Encoder architecture with a normalization layer added before the Transformer; the text encoder adopts the Transformer architecture, and the output of the highest layer of the Transformer at the [EOS] marker is used as the feature representation of the text, which is linearly projected into the multimodal embedding space after layer normalization. During the training process, CLIP learns a joint representation by maximizing the similarity between the image and its corresponding text description, while minimizing the similarity between the image and unrelated text.

[0052] In this embodiment, the core of the instruction generation model is built based on a sequence-to-sequence Transformer structure, including a connected spatial encoder, a temporal encoder, and a text decoder.

[0053] In order to better utilize the action and environment information in both time and space during the entire navigation process, a spatiotemporal Transformer encoder structure designed based on the cross-modal attention mechanism module is used. Two encoders are used to fuse spatiotemporal information. Specifically, after using the spatial encoder to fuse the spatial information of each time stage, the temporal encoder is used to learn the intrinsic connection between different time stages.

[0054] (1) Spatial Encoder

[0055] The input of the instruction generation model consists of two kinds of observation data: a set of action features and a set of environmental observations The reasonable fusion of visual features is crucial, which affects the cognitive level of the model. Since the action features can represent the main changes in the navigation process, A is regarded as the query and E is regarded as the key and value. Using the multi-head attention mechanism, linear projection of the query, key and value through different linear transformations allows the model to focus on different representation subspaces at different positions. After splicing the results of the multi-head attention together, residual connection is applied, and then layer normalization is performed to obtain the result In order to improve the generalization ability, some additional feature masks are added to the model. The formula of the spatial encoder is as follows:

[0056]

[0057] MultiHead(Q,K,V)=Concat(head1,...,head H )W o (6)

[0058]

[0059] In the formula, After aggregating action features and environmental features, the output of the spatial encoder is The multi-head attention mechanism improves the model's ability to capture important semantic information in the environment and fuses them into image features in specific directions, reducing useless noise and interference.

[0060] (2) Time Encoder

[0061] After using the spatial encoder to fuse the spatial information of each time stage, the temporal encoder is used to learn the intrinsic connection between different time stages. Extracting the connection between different time periods is very important for the instruction generation model because the instructions are generated according to the entire forward process of the agent's navigation. Specifically, the spatial fusion features {z1,z2,...,z N} are input as independent tokens into the temporal encoder of layer L. The position encoding PE(·) is added to Z to retain the position information. Each encoder layer consists of a multi-head self-attention layer (MSA) and a small feedforward neural network (FFN). Since the trajectory instructions generated by the instruction generation model are closely related to the actual navigation sequence, the position encoding is added to the visual sequence encoding. Similar to the spatial encoder, residual connections are used between each sub-layer, followed by layer normalization. Let l represent the lth layer, where l=1…L, then the formula of the temporal encoder is:

[0062] Z0=PE(Z) (8)

[0063] Z l=LayerNorm(MSA(Z l-1 )+Z l-1 ) (9)

[0064] Z l =LayerNorm(FFN(Z l ')+Z l ') (10)

[0065] (3) Text decoder

[0066] The decoder part uses the Transformer decoder structure. Since language generation is an autoregressive process, it is necessary to ensure that each predicted word depends only on the previous predicted words. Therefore, word embeddings are marked with a special <bos>offset, and apply a masking function in the attention matrix to mask out illegal positions. Positional encoding is also added to the embedding vector to capture the relative position of tokens in the sequence. Assume the target word is d vocab , each instruction contains at most L words, and the output value of the last hidden layer is processed with a linear layer and softmax to obtain Convert to predicted probability The output head that performs this task is called the speaker word projector (SWP) and is optimized using the cross entropy loss as follows:

[0067]

[0068] In the formula, θ is the preset parameter, is a text instruction in the data set, l is the number of words contained in the text instruction, l≤L, f θ (·) is the probability that the target word appears in the i-th position of the instruction, predicted by the instruction generation model based on the action A, the environment observation E, and the first i-1 words in the instruction.

[0069] Figure 4 The figure shows the changes of main performance indicators during the instruction generation model training process. Figure 4 Part (a) is the change curve of the performance indicator BLEU on the seen environment verification set. Figure 4 Part (b) is the change curve of the performance indicator BLEU-4 on the seen environment verification set. Figure 4 Part (c) is the change curve of the performance indicator Loss on the seen environment verification set. Figure 4 Part (d) is the change curve of the performance indicator BLEU on the unseen environment verification set. Figure 4 Part (e) is the change curve of the performance indicator BLEU-4 on the unseen environment validation set. Figure 4 Part (f) is the change curve of the performance indicator Loss on the unseen environment validation set.

[0070] Step S2, generating a new environment according to the preset simulation environment.

[0071] This step simulates and generates a new environment by randomly covering some of the visual features of the environment in the simulation environment, achieving visual feature enhancement to increase the model's adaptability to different environments. Specifically:

[0072] The instruction generation-navigation path generation method fine-tunes the model by generating additional training data through back translation, which is usually implemented based on the instruction generation model, which synthesizes additional route instructions in the existing environment. However, the bottleneck of this semi-supervised learning method is that the changes in the given environment are limited, resulting in limited generalization ability of the model in unfamiliar environments. To overcome this problem, the environmental visual feature masking method is adopted to generate new environments based on view and viewpoint consistent visual feature masking, and then new navigation routes are collected from these new environments. Finally, the instruction generation model generates new navigation instructions based on these navigation routes, and uses these enhanced data to fine-tune the model.

[0073] Generate a new environment E′ by applying ambient occlusion to the existing environment E (the preset simulation environment):

[0074] E′=EnvDrop p (E) (12)

[0075] The view features f′ observed by the navigating agent from the new environment E′ t,i is the original feature f t,i and the ambient occlusion mask ξ E Finding the Hadamard product gives:

[0076] f′ t,i =f t,i ⊙ξ E (13)

[0077]

[0078] Where, the ambient occlusion mask ξ E Each element in are all samples of a random variable, Ber(·) represents Bernoulli Distribution, and p is the probability of covering the view feature.

[0079] In order to maintain the spatial relationship of the viewpoints, only the image features are covered when the environmental visual features are covered, while the directional features (cos(α t,i ),sin(α t,i ),cos(φ t,i ),sin(φ t,i ))remains unchanged.

[0080] Step S3, in the new environment, collect the first navigation path. This process can effectively expand the existing data set to cover more path changes. Specifically:

[0081] When the training data only covers a limited number of navigation instructions and route pairs, i.e., D = (d1, r1)…(d N ,r N ). In order to make the agent better adapt to the new route, the instruction generation model can be used to generate instructions on the new route samples in the training environment to expand the training data. To achieve this process, M routes are collected in the training environment using the same shortest path method as the original dataset. That is, the first navigation path is obtained.

[0082] Step S4, according to the first navigation path, using the instruction generation model trained in step S1 to generate the first navigation instruction, to obtain an expanded simulation environment data set. Specifically:

[0083] By performing greedy prediction in the instruction generation model, for each navigation path collected in step S3 Generate a text instruction similar to a human instruction:

[0084]

[0085] These M generated navigation paths and instructions It is combined with the original training dataset D to form an enhanced training set S∪D, which is the expanded simulation environment dataset. During the training process, the navigation model is first trained on this enhanced training set and then further fine-tuned on the original training set D. This data enhancement with the instruction generation model as the core is designed to overcome the data scarcity problem of the visual language navigation dataset and allow the navigation model to be trained on the collected new routes and synthetic instructions.

[0086] Step S5: Use the expanded simulation environment data set to train the preset navigation model. Specifically:

[0087] The basic navigation path generation model, also known as the navigation model, adopts an encoder-decoder structure.

[0088] The encoder is a bidirectional LSTM-RNN network with an embedding layer:

[0089]

[0090] In the formula, u j is the jth word in the instruction, and the instruction length is L.

[0091] The decoder is an LSTM-RNN network with an attention mechanism. In each decoding step t, the agent first performs an analysis on the view features {f t ,i} perform attention weighting to obtain the attention-weighted visual features

[0092]

[0093] The input of the decoder is the attention-weighted visual features With the previous action The output of the LSTM hidden layer and the instruction features after attention weighting After combination, the hidden layer output of instruction perception is obtained The probability p of moving to the kth navigation point t (a t,k ) is the navigation point feature g t,k Hidden layer output with instruction awareness Alignment and then softmax calculation:

[0094]

[0095] Step S6: In the actual deployment environment of the indoor mobile service robot, synchronous positioning and map construction are performed, and a topological navigation map is constructed based on the positioning and map construction results. Specifically:

[0096] At present, the accuracy of visual language navigation methods without map construction is still low, and it is easy to collide with environmental objects, so it is not suitable for deployment in real environments. Therefore, it is necessary to use the simultaneous localization and mapping (SLAM) method to build a map to limit the movement of the intelligent robot to a safe range where no collision occurs, and provide the robot with location information.

[0097] This embodiment uses a self-developed sampling mobile robot to complete scene mapping and image acquisition. The mobile robot is equipped with hardware equipment such as a mobile chassis, a high-precision laser radar, and a panoramic camera, which supports it to move on relatively flat ground and take panoramic pictures.

[0098] After building the map, select navigation points in the map. The navigation points should cover all areas that the robot wants to reach as much as possible, and the distance between adjacent navigation points should be about 2m. Based on the navigation points, the topological navigation map can be determined.

[0099] Step S7, based on the topology navigation map, collect a panoramic view in the actual deployment environment. Specifically:

[0100] After the navigation points are generated, the intelligent robot goes to each navigation point in the environment in turn and takes a panoramic view after arriving. If the positioning is inaccurate, the robot's position needs to be adjusted manually. When taking a panoramic view, the robot's position and posture information returned by the SLAM program needs to be recorded for adjusting the topology map and panoramic view preprocessing.

[0101] The image acquisition robot uses the ROS coordinate system for mapping and positioning. The definition of the heading angle of the ROS coordinate system is different from that of the Matterport3D simulator coordinate system. For example, Figure 5 As shown, in the Matterport3D coordinate system, the positive direction of the y-axis is the heading angle of 0, and the heading angle increases to the right, with a range of [0,2π]; while in the ROS coordinate system, the positive direction of the x-axis is the heading angle of 0, and the heading angle increases to the left, with a range of [-π,π]. When the Matterport3D simulator slices the sub-graph, it starts with the pitch angle of -π / 3 and the heading angle of 0 as the center, and increases by π / 3 to the right and upward in turn (the pitch angle increases again after the heading angle is traversed), and the horizontal and vertical spans of each sub-graph are π / 3. Therefore, after rotating the captured panoramic image to the image center of 0 in the ROS coordinate system, it is necessary to rotate it to the right again by π / 3, and then start slicing the sub-graph from the lower left. The slicing effect is as follows: Figure 3 After segmentation, each sub-graph is subjected to feature extraction in turn, and then the feature vectors are concatenated.

[0102] Step S8, based on the panoramic image, collect the second navigation path in the actual deployment environment, and use the instruction generation model trained in S1 to generate the second navigation instruction, so as to obtain the expanded deployment environment data set. Specifically:

[0103] First, the environment is divided into multiple areas. Generally, a room is divided into one area, and doors are used as channels to connect different areas. Then, each area is used as the starting and ending area of ​​the path, and all cross-area paths are traversed and paths with less than 4 or more than 10 navigation points and paths with a length of less than 3m are filtered out. The instruction generation model trained on the existing dataset is used to automatically generate navigation language instructions (i.e., the second navigation instructions) for the collected path (i.e., the second navigation path) to obtain the deployment environment dataset.

[0104] Figure 6 Generate navigation instruction examples for the navigation paths collected in the deployment environment using the instruction generation model in S1. Figure 6 In part (a) of

[0105] Generated instructions include: exit the office and turn right. walk down the hallway and turn right at the exit sign. wait in the doorway of the room on the right.

[0106] Human instructions include: Turn right, leave the room through the door closest to you, then turn right and walk down the corridor until you pass the black chair in the hallway, then turn right, enter the room with the basketball by the door and wait at the door.

[0107] Figure 6 In part (b) of:

[0108] Generated instructions include: walk around the glass table and turn right. Walk past the glass wall and turn left. Walk into the first door on the left and stop.

[0109] Human instructions include: Turn around the wooden counter facing the restaurant and go straight to the wall, then turn left and follow the wall to the restaurant door and turn right out of the restaurant, with the cardboard boxes in the storage room on the left and the corridor on the right. Turn left and go straight to the door of the storeroom and turn left into the storeroom.

[0110] Figure 6 In part (c) of:

[0111] Generated instructions include: walk forward and exit the room. Turn right and enter the first door on the right. Wait near the desk.

[0112] Human instructions include: Go ahead to the edge of the three vases, then turn right and follow the black counter. After passing the black sprinkler, you will see the open glass door on the left. Go through this door to the hallway, then turnright into the next door. Wait at the study desk with computers and books.

[0113] Advantages of the instruction generation model in generating instructions: ① Language instructions are automatically generated without manpower. Therefore, it can be used to automatically generate instructions in the collected image data set, so that path-instruction pairs can be automatically collected in the deployment environment, thereby fine-tuning the model and improving the performance of the model in the deployment environment. ② The navigation direction is roughly correct. The directional descriptors such as "turn left" and "turn right" and action instructions such as "leave the room" and "enter the room" generated by the instruction generation model are consistent with the actual navigation path. ③ It can identify the room type and common objects in the environment. For example Figure 6 The words "office", "hallway" and "exit sign" in part (a) of the instruction, Figure 6 In part (b) of the instruction, "glass table", "glass wall" and "door" are Figure 6 The words "door" and "desk" in part (c) of the instructions.

[0114] In the aforementioned step S4 and step S8, the trained instruction generation model is used to automatically generate the first navigation instruction and the second navigation instruction.

[0115] Step S9, fine-tune the navigation model trained in S5 using the expanded deployment environment dataset to enhance the visual language navigation data. This process can further optimize the model performance and make it perform better in a specific actual deployment environment. Specifically:

[0116] Fine-tune the model using the dataset collected in the deployment environment (i.e., the deployment environment dataset). Figure 7 The main performance indicators of the fine-tuning model process using the data set collected in the deployment environment have changed. It can be seen that the main performance indicators of the navigation model after fine-tuning have been significantly improved on the test set collected in the deployment environment. Figure 7 Part (a) is the performance indicator Loss change curve. Figure 7 Part (b) is the performance index NE change curve. Figure 7 Part (c) is the performance indicator PL change curve. Figure 7 Part (d) is the performance index SR change curve. Figure 7 Part (e) is the performance indicator SPL change curve.

[0117] Furthermore, the present embodiment also provides a visual language navigation data enhancement device for an indoor mobile service robot, including a memory, a processor, and a program stored in the memory. When the processor executes the program, one or more steps of the aforementioned method are implemented. The specific execution process is basically the same as the method execution process, which will not be repeated here. The device processor includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded from a storage unit to a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The CPU, ROM, and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus. Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks. The processing unit performs the various methods and processes described above, such as one or more steps of the aforementioned method. For example, in some embodiments, one or more steps of the aforementioned method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of the aforementioned method may be executed. Alternatively, in other embodiments, the CPU may be configured to execute one or more steps of the aforementioned method in any other appropriate manner (e.g., by means of firmware). The functions described above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0118] Furthermore, this embodiment also provides a storage medium on which a program is stored. When the program is executed, one or more steps of the aforementioned method are implemented. The specific execution process is basically the same as the method execution process, and will not be repeated here.

[0119] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.< / bos>

Claims

1. A visual language navigation data enhancement method for indoor mobile service robots, characterized in that: The following steps are involved: S1, trains a preset instruction generation model using a pre-acquired visual language navigation dataset; S2, generating a new environment according to the preset simulation environment; S3, collecting a first navigation path in the new environment; S4, generating a first navigation instruction according to the first navigation path using the instruction generation model trained in S1 to obtain an expanded simulation environment data set; S5, using the expanded simulation environment data set to train a preset navigation model; S6, in the actual deployment environment of the indoor mobile service robot, synchronous positioning and map construction are performed, and a topological navigation map is constructed based on the positioning and mapping results; S7, collecting a panoramic image in the actual deployment environment according to the topology navigation map; S8, collecting a second navigation path in the actual deployment environment according to the panoramic image, and generating a second navigation instruction using the instruction generation model trained in S1 to obtain a deployment environment data set; S9, using the deployment environment dataset to fine-tune the navigation model trained in S5 to complete visual language navigation data enhancement.

2. The visual language navigation data enhancement method for indoor mobile service robots according to claim 1 is characterized in that: In S4 and S8, the trained instruction generation model is used to automatically generate the first navigation instruction and the second navigation instruction.

3. The visual language navigation data enhancement method for indoor mobile service robots according to claim 1, characterized in that: The instruction generation model is constructed using a sequence-to-sequence based Transformer structure, which includes a connected spatial encoder and a temporal encoder. The spatial encoder is used to fuse the spatial information of each time stage, and the temporal encoder is used to learn the intrinsic connection between different time stages after the spatial encoder completes the fusion.

4. The visual language navigation data enhancement method for indoor mobile service robots according to claim 3 is characterized in that: The Transformer structure also includes a text decoder connected to the temporal encoder, and the text encoder includes an instruction generation word predictor for performing prediction tasks.

5. The visual language navigation data enhancement method for indoor mobile service robots according to claim 4, characterized in that: The instruction generation word predictor is optimized using a preset cross entropy loss, and the expression of the cross entropy loss is: In the formula, θ is the preset parameter, is a text instruction in the data set, l is the number of words contained in the text instruction, l≤L, f θ (·) is the probability that the target word appears in the i-th position of the instruction, predicted by the instruction generation model based on the action A, the environment observation E, and the first i-1 words in the instruction.

6. The visual language navigation data enhancement method for indoor mobile service robots according to claim 3, characterized in that: Before inputting the image to be processed into the instruction generation model, the image to be processed is encoded using a trained image encoder to extract corresponding image features, and the image encoder is trained using a contrastive learning method.

7. The visual language navigation data enhancement method for indoor mobile service robots according to claim 1, characterized in that: The S2 specifically includes: using an environmental visual feature masking method to generate a new environment based on visual feature masking consistent with the view and viewpoint.

8. The visual language navigation data enhancement method for indoor mobile service robots according to claim 7, characterized in that: The view features f′ observed from the new environment t,i From the original feature f t,i and the ambient occlusion mask ξ E The Hadamard product is obtained as follows: f’ t,i =f t,i ⊙ξ E Where E represents the preset simulation environment, and the ambient occlusion mask ξ E Each element in are all samples of a random variable, Ber(·) represents Bernoulli Distribution, and p is the probability of covering the view feature.

9. A visual language navigation data enhancement device for an indoor mobile service robot, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images