Multi-Granularity Instruction Generation Method and Device for Vision-Language Navigation

Abstract

The present invention discloses a multi-granularity instruction generation method and device for visual language navigation. The method includes: extracting coarse-grained panoramic image features and action features, and fine-grained region features and grid features; using Bi-LSTM and an attention mechanism to encode and fuse global features to obtain a coarse-grained trajectory representation; designing spatial self-attention based on time constraints, temporal self-attention, and cross-attention based on time and local constraints to encode and fuse local features to obtain a fine-grained trajectory representation; fusing the coarse-grained and fine-grained representations to generate a multi-granularity trajectory representation; decoding through LSTM-RNN to synthesize navigation instructions; the synthesized instructions and the sampled trajectories form enhanced data for training a navigation agent. Starting from the perspective of data augmentation, the present invention synthesizes high-quality natural language navigation instructions and improves the generalization ability of the agent, providing an innovative and practical solution.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual language navigation, and particularly relates to a multi-granularity instruction generation method and device for visual language navigation. Background Art

[0002] In the interaction of human society, natural language, as a basic tool for communication, makes the communication between people efficient and rich. Imagine if an intelligent system can understand and respond to human language instructions, such as performing specific actions or going to a specified location. This can not only replace manual operations but also significantly improve production efficiency and the level of social services. The applications of such technologies are extensive, covering domestic service robots, logistics automation equipment, and intelligent machinery in the construction field, etc. The research on visual language navigation technology aims to promote the development of embodied intelligence, that is, to enable an intelligent system to gradually possess the ability to understand and operate the physical world similar to humans through continuous learning and adaptation in the interaction with the actual or simulated environment. This progress is crucial for breaking through the limitations of the existing "narrow artificial intelligence" and enhancing the cognitive level of machines.

[0003] In 2018, Anderson et al. first proposed the concept of visual language navigation and constructed the Room-to-Room (R2R) dataset for this purpose. This dataset created a series of indoor scenes based on 3D reconstruction technology, and each scene consisted of multiple discrete nodes where an agent could move. In the experimental setup, the agent was first randomly placed at a certain point, and then, according to the received natural language instructions and combined with the panoramic view of the current position, it decided which direction to move next. As the position of the agent changed, the visual information it obtained was also updated. The agent needed to continuously repeat this process until it reached the target point or reached the preset maximum number of actions. When the final stopping position of the agent was within an acceptable range of the destination specified in the instruction, the task was considered completed.

[0004] However, visual language navigation faces multiple challenges. The primary problem lies in the effective integration and matching of multi-modal information in complex environments. Secondly, designing a reasonable reasoning mechanism to guide the navigation decision of the agent is also a major challenge. In addition, due to the high cost of collecting high-quality data, the existing dataset sizes are relatively small and are not sufficient to fully support such complex task requirements, resulting in the model being prone to overfitting. Especially when facing new scenes that have not been seen before, the performance of the model often fails to meet expectations. These factors jointly limit the development of visual language navigation technology and its actual application effects. Summary of the Invention

[0005] The main objective of the present invention is to overcome the drawbacks and deficiencies of the prior art, and to provide a multi-granularity instruction generation method and device for visual language navigation, which can automatically generate high-quality natural language navigation instructions based on the path data obtained in the simulation environment. The generated instructions are combined with the corresponding paths to form an enhanced dataset, which helps to alleviate the challenge of data sparsity, and further enhances the generalization performance of the visual language navigation agent, specifically manifested as improving the success rate of the agent navigating according to language instructions in an unknown environment.

[0006] To achieve the above objective, the present invention adopts the following technical solutions:

[0007] In the first aspect, the present invention provides a multi-granularity instruction generation method for visual language navigation, including the following steps:

[0008] Obtain the trajectory of the visual language navigation agent, and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include region features and grid features;

[0009] Encode and fuse the coarse-grained global features, specifically: encode the action features to capture the context dependencies of multiple time stages in the trajectory features, and then based on the attention mechanism, fuse the encoded action features and panoramic image features to obtain a coarse-grained navigation trajectory representation, which depicts the key visual cues on the navigation trajectory from a global dimension;

[0010] Encode and fuse the fine-grained local features, specifically: for the region features or grid features, input them into a preset spatio-temporal self-attention mechanism with time constraints. Within a given time period, this attention mechanism assigns higher weights to specific regions or grids, so that the model focuses on the key visual cues for navigation in the spatial dimension; for the region features or grid features output by the spatio-temporal self-attention mechanism with time constraints, input them into the temporal self-attention mechanism, so that the model focuses on the key visual cues in the temporal dimension; for the region features and grid features output by the temporal self-attention mechanism, input them into a preset cross-attention mechanism with time and local constraints to fuse the region features and grid features to obtain a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts the key visual cues on the navigation trajectory from a local dimension, representing fine-grained object semantics and color texture information;

[0011] Based on the attention mechanism, fuse the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation to obtain a multi-grained navigation trajectory representation; the multi-grained navigation trajectory representation integrates key visual cues in the global dimension and the local dimension, making the generated navigation instructions more comprehensive and accurate;

[0012] Decode the multi-grained navigation trajectory representation to obtain a synthesized navigation instruction;

[0013] Based on the synthesized navigation instruction and the sampled trajectory, form enhanced data to train the vision-language navigation agent, improving the generalization ability of the agent and the success rate of navigation.

[0014] As a preferred technical solution, the obtaining of the trajectory of the vision-language navigation agent and the extraction of the trajectory features of the agent are specifically as follows:

[0015] Extract panoramic image features, which are composed of the ResNet image features and the orientation features of the agent spliced together;

[0016] Extract action features. In each time stage, the agent selects an image from the panoramic image as the direction to go in the next time stage and moves to the corresponding path node. Therefore, the action feature corresponds to one of the panoramic image features;

[0017] Extract region features and grid features. Given an image, use Faster R-CNN to extract multiple regions and multiple grids, and splice each region or grid with the embedding function of its orientation feature to obtain region features and grid features.

[0018] As a preferred technical solution, the encoding and fusion of the coarse-grained global features are specifically as follows:

[0019] Use Bi-LSTM to encode the action features:

[0020]

[0021] In the formula, r1,...,r T respectively represent the trajectory encodings obtained by encoding the action features in the 1st to the Tth time periods, T represents the total number of time stages, that is, the number of nodes on the agent's trajectory, respectively represent the 1st to the Tth action feature encodings;

[0022] Based on the attention mechanism, fuse the trajectory encoding of the action features and the panoramic image features, where the trajectory encoding is used as the Query, and the panoramic image features are used as the Key and Value, to obtain the coarse-grained navigation trajectory representation, which is expressed as follows:

[0023]

[0024] f t g = ∑ i α t,i f t,i

[0025] where α t,i represents the attention weight of the i-th panoramic image at time t, f t,i represents the feature of the i-th panoramic image at time t, T represents the transpose symbol, W R represents the learnable parameter, r t represents the trajectory encoding of the t-th time period, f t g represents the coarse-grained navigation trajectory representation.

[0026] As a preferred technical solution, the encoding and fusion of the fine-grained local features are specifically as follows:

[0027] For the region feature, obtain its region spatial position encoding;

[0028] For the grid feature, obtain its grid spatial position encoding;

[0029] Design a time-constrained spatial self-attention mechanism, specifically:

[0030] MHTCSSA(Q, K, V) = Concat(head1,..., head h )W O

[0031] head i = TCSSA(QW i Q , KW i K , VW i V , spe q , spe k )

[0032]

[0033] where MHTCSSA represents the multi-head time-constrained spatial self-attention mechanism, Q represents Query, K represents Key, V represents Value, Concat represents the concatenation operation, W O , W Q , W K , W V are learnable parameters, head idenotes the i-th attention head, h represents the total number of attention heads, TCSSA is a single-head time-constrained spatial self-attention mechanism, spe q is the spatial position encoding of the Query, spe k is the spatial position encoding of the Key, d k is the dimension of the vector of the Key;

[0034] Input the region features and grid features into the multi-head time-constrained spatial self-attention mechanism:

[0035]

[0036]

[0037] In the formula, denotes the t-th time stage, the region features input at the l-th layer, RSPE is the region spatial position encoding,

[0038] denotes the grid features input at the t-th time stage, the l-th layer, GSPE is the grid spatial position encoding, denotes the region features output at the t-th time stage, the l-th layer, denotes the grid features output at the t-th time stage, the l-th layer;

[0039] The region features output by the time-constrained spatial self-attention mechanism or the grid features plus the position encoding PE of the time stage t :

[0040]

[0041]

[0042] Input into the time self-attention mechanism:

[0043]

[0044]

[0045] In the formula, MHTSA represents the multi-head time self-attention mechanism, and represent the region features and grid features of all time stages, represents the region features output by the time self-attention mechanism of the l-th layer,

[0046] represents the grid features output by the time self-attention mechanism of the l-th layer;

[0047] Input and into the feed - forward network respectively:

[0048]

[0049]

[0050] In the formula, FFN represents the feed - forward network, represents the region feature output by the temporal self - attention mechanism of the l - th layer, represents the grid feature output by the temporal self - attention mechanism of the l - th layer;

[0051] Design a cross - attention mechanism based on time and local constraints:

[0052] MHTLCCA(Q,K,V)=Concat(head1,...,head h )W O

[0053]

[0054]

[0055] In the formula, MHTLCCA represents the multi - head cross - attention mechanism based on time and local constraints, TLCCA represents the single - head cross - attention mechanism based on time and local constraints, represents the geometric alignment graph of regions and grids. In this geometric alignment graph, each region and grid corresponds to an independent node, jointly constituting the node set To ensure semantic consistency between regions and grids in the cross - attention mechanism, only when their respective bounding boxes intersect, the edges in the set will be established between the region nodes and the grid nodes. represents the graph - based softmax function, which assigns zero attention weights to non - adjacent nodes in and applies the softmax function to adjacent nodes in ; To ensure temporal consistency, for each image at time stage t, a dedicated geometric alignment graph

[0056] For the region features and grid features output by the temporal self - attention mechanism, input them into the designed cross - attention mechanism based on time and local constraints to fuse the region features and grid features:

[0057]

[0058]

[0059] In the formula, the region feature is used as the Query, the grid feature is used as the Key and Value, and the grid feature is embedded into the region feature to obtain the region feature output by the cross-attention mechanism based on time and local constraints at the l-th layer. In the formula, the grid feature is used as the Query, the region feature is used as the Key and Value, and the region feature is embedded into the grid feature to obtain the grid feature output by the cross-attention mechanism based on time and local constraints at the l-th layer.

[0060] Subsequently, and are respectively input into the feed-forward network FFN:

[0061]

[0062]

[0063] Due to the adoption of the multi-layer attention mechanism, and are input into the spatial self-attention mechanism based on time constraints in the next layer;

[0064] The region feature H r and the grid feature H g output by the last layer are concatenated to obtain the fine-grained navigation trajectory representation u:

[0065] u = [H r ; H g

[0066] In the formula, T is the total number of time stages of the agent's trajectory, N r is the number of extracted region features, N g is the number of extracted grid features, and d model represents the dimension of the hidden layer vector.

[0067] As a preferred technical solution, the region spatial position encoding for obtaining the region feature is specifically:

[0068] RSPE(i) = Emb(B i );

[0069] In the formula, i represents the i-th region in the image, and B i = (x min , y min , x max , y max ​The coordinates of the upper left and lower right corners of the bounding box of the representative region, Emb(·) represents the projection layer, and RSPE represents the spatial position encoding of the region.

[0070] As a preferred technical solution, the grid spatial position encoding for obtaining the grid features is specifically:

[0071] GSPE(i,j) = [PE i ; PE j

[0072] In the formula, the grid divides the image into i×j grids, i represents the row index of the grid, j represents the column index of the grid, PE i , PE j represent the position encodings of i and j respectively, and GSPE represents the grid spatial position encoding.

[0073] As a preferred technical solution, based on the attention mechanism, the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation are fused to obtain a multi-grained navigation trajectory representation, specifically:

[0074] Based on the attention mechanism, taking the coarse-grained navigation trajectory representation f t g as the Query, taking the fine-grained navigation trajectory representation u as the Key and Value, and fusing the coarse-grained and fine-grained navigation trajectory representations to obtain a multi-grained navigation trajectory representation:

[0075]

[0076] f t gl = ∑ i β t,i u t,i

[0077] In the formula, W F represents the learnable parameter, T represents the transpose symbol, u t,i represents the i-th region / grid feature at time t, β t,i represents the attention weight of the i-th region / grid feature at time t, and f t gl represents the multi-grained navigation trajectory representation;

[0078] Use Bi-LSTM to encode the multi-grained navigation trajectory representation to further capture its context dependence on time:

[0079]

[0080] ​Second aspect, the present invention provides a multi-granularity instruction generation system for visual language navigation, which is applied to the multi-granularity instruction generation method for visual language navigation, and includes a trajectory feature extraction module, a coarse-grained encoding module, a fine-grained encoding module, a navigation trajectory representation fusion module, a navigation instruction generation module, and an agent training module;

[0081] The trajectory feature extraction module is used to obtain the trajectory of the visual language navigation agent and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include region features and grid features;

[0082] The coarse-grained encoding module is used to encode and fuse the coarse-grained global features, specifically: encode the action features to capture the context dependencies of multiple time stages in the trajectory features, and then based on the attention mechanism, fuse the encoded action features and panoramic image features to obtain a coarse-grained navigation trajectory representation, and the coarse-grained navigation trajectory representation depicts the key visual cues on the navigation trajectory from a global dimension;

[0083] The fine-grained encoding module is used to encode and fuse the fine-grained local features, specifically: for the region features or grid features, input them into a preset spatio-temporal self-attention mechanism based on time constraints. During a given time period, this attention mechanism assigns higher weights to specific regions or grids, so that the model focuses on the key visual cues for navigation in the spatial dimension; for the region features or grid features output by the spatio-temporal self-attention mechanism based on time constraints, input them into the temporal self-attention mechanism, so that the model focuses on the key visual cues in the temporal dimension; for the region features and grid features output by the temporal self-attention mechanism, input them into a preset cross-attention mechanism based on time and local constraints to fuse the region features and grid features to obtain a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts the key visual cues on the navigation trajectory from a local dimension, representing fine-grained object semantics and color texture information;

[0084] The navigation trajectory representation fusion module, based on the attention mechanism, fuses the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation to obtain a multi-granularity navigation trajectory representation; the multi-granularity navigation trajectory representation integrates the key visual cues in the global dimension and the local dimension, making the generated navigation instructions more comprehensive and accurate;

[0085] The navigation instruction generation module is used to decode the multi-granularity navigation trajectory representation to obtain a synthesized navigation instruction;

[0086] The intelligent agent training module is used to train a vision-language navigation intelligent agent based on the enhanced data composed of the synthesized navigation instructions and the sampled trajectories, so as to improve the generalization ability of the intelligent agent and the success rate of navigation.

[0087] In a third aspect, the present invention provides an electronic device, which includes:

[0088] At least one processor; and, a memory communicatively connected to the at least one processor; wherein,

[0089] The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor, so that the at least one processor can execute the multi-granularity instruction generation method for vision-language navigation.

[0090] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which when executed by a processor, implements the multi-granularity instruction generation method for vision-language navigation.

[0091] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0092] 1. The existing vision-language navigation technology mainly focuses on how to improve the navigation intelligent agent, and relatively less attention is paid to synthesizing navigation instructions to provide detailed navigation path explanations. However, due to the problem of data sparsity, the generalization ability of the intelligent agent is limited, that is, the success rate of navigation in unfamiliar scenarios is not high. Starting from the perspective of data augmentation, the present invention can generate high-quality synthetic instructions, thereby improving the navigation success rate and generalization ability of the intelligent agent.

[0093] 2. Although the existing instruction synthesis technology for vision-language navigation can alleviate the problem of data sparsity to a certain extent, there is a problem of low quality of the synthesized instructions. This is because the existing technology only uses coarse-grained image-level representations to generate instructions, and it is difficult to capture the important details required to reproduce the navigation trajectory.

[0094] 3. The present invention integrates the coarse-grained image features and fine-grained key visual cues on the navigation trajectory, can cover the global environmental information and local object information required for navigation decisions, makes the generated navigation instructions more comprehensive and accurate, and solves the problem of poor quality of the synthesized instructions in the prior art.

[0095] In summary, starting from the perspective of data augmentation, the present invention can synthesize high-quality natural language navigation instructions, improve the generalization ability of the navigation intelligent agent, and provides an innovative and practical solution. Description of the Drawings

[0096] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for description in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0097] Figure 1 It is a flowchart of the multi-granularity instruction generation method for visual language navigation according to an embodiment of the present invention;

[0098] Figure 2 It is a structural schematic diagram of the geometric alignment graph according to an embodiment of the present invention;

[0099] Figure 3 It is a framework diagram of the multi-granularity instruction generation system for visual language navigation according to an embodiment of the present invention;

[0100] Figure 4 It is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed implementation manners

[0101] In order to enable those skilled in the art to better understand the solutions of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope protected by the present application.

[0102] Referring to "embodiments" in the present application means that specific features, structures, or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present application. The phrase appears in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments.

[0103] Please refer to Figure 1 , this embodiment provides a multi-granularity instruction generation method for visual language navigation, which fuses the coarse-grained image features and fine-grained key visual cues on the navigation trajectory to synthesize high-quality navigation instructions, and includes the following steps:

[0104] S1. Sampling is performed in a simulation environment to obtain a series of trajectories of visual language navigation agents, and the trajectory features of the agents are extracted; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include region features and grid features.

[0105] Furthermore, step S1 is specifically as follows:

[0106] S11. Extract panoramic image features, which are composed of the concatenation of ResNet image features and the orientation features of the agent: f

[0107] f t,i = [ResNet(v t,i ); E o (θ t,i , φ t,i )]

[0108] In the formula, t represents the time stage at which the agent is located, represents 36 panoramic images at the current time stage t, ResNet(·) represents the ResNet convolutional neural network, and E o (·) represents the embedding function for encoding the orientation features. θ t,i represents the orientation angle of the i-th panoramic image at the current time stage t, and φ t,i represents the elevation angle of the i-th panoramic image at the current time stage t.

[0109] S12. At each time stage, the agent selects one image from the 36 panoramic images as the direction to go to in its next time stage and moves to the corresponding path node. Therefore, the action feature corresponds to one of the panoramic image features and is formulated as

[0110] S13. Given an image, use Faster R-CNN to extract N r regions and N g grids; each region or grid is concatenated with the embedding function E o (θ t,i , φ t,i ) of its orientation feature to obtain the region feature and the grid feature

[0111] S2. Encode and fuse the coarse-grained global features, specifically: encode the action features to capture the context dependencies of multiple time stages in the trajectory features, and then based on the attention mechanism, fuse the encoded action features and the panoramic image features to obtain a coarse-grained navigation trajectory representation, which depicts the key visual cues on the navigation trajectory from a global dimension.

[0112] Furthermore, step S2 is specifically as follows:

[0113] S21. Encode the action features using Bi-LSTM:

[0114]

[0115] Where T represents the total number of time steps, i.e., the number of nodes on the agent's trajectory. r represents the trajectory encoding obtained by encoding the action features.

[0116] S22. Based on the attention mechanism, fuse the trajectory encoding of the action features and the panoramic image features; where the trajectory encoding serves as the Query, and the panoramic image features serve as the Key and Value, to obtain a coarse-grained navigation trajectory representation:

[0117]

[0118]

[0119] Where α t,i represents the attention weight of the i-th panoramic image at time t, f t,i represents the feature of the i-th panoramic image at time t, T represents the transpose symbol, W R represents the learnable parameter, r t represents the trajectory encoding of the t-th time period, f t g represents the coarse-grained navigation trajectory representation.

[0120] S3. Encode and fuse the fine-grained local features, specifically: for the region features or grid features, input them into a preset spatio-temporal self-attention mechanism based on time constraints. Within a given time period, this attention mechanism assigns higher weights to specific regions or grids, so that the model focuses on the key visual cues for navigation in the spatial dimension; for the region features or grid features output by the spatio-temporal self-attention mechanism based on time constraints, input them into the temporal self-attention mechanism, so that the model focuses on the key visual cues in the temporal dimension; for the region features and grid features output by the temporal self-attention mechanism, input them into a preset cross-attention mechanism based on time and local constraints to fuse the region features and grid features, and obtain a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts the key visual cues on the navigation trajectory from the local dimension, representing the fine-grained object semantics and color texture information.

[0121] Further, step S3 is specifically:

[0122] S31. For the region features, obtain their region spatial position encoding:

[0123] RSPE(i) = Emb(B i )

[0124] In the formula, i represents the i-th region in the image, and B i =(x min , y min , x max , y max ) represents the coordinates of the upper left corner and the lower right corner of the bounding box of the region, Emb(·) represents the projection layer, and RSPE represents the spatial position encoding of the region.

[0125] S32. For the grid features, obtain their grid spatial position encoding:

[0126] GSPE(i, j)=[PE i ; PE j

[0127] In the formula, since the grid divides the image into i×j grids, i represents the row index of the grid, j represents the column index of the grid, PE i , and PE j represent the position encodings of i and j respectively, and GSPE represents the grid spatial position encoding.

[0128] S33. Design a spatial self-attention mechanism based on time constraints:

[0129] MHTCSSA(Q, K, V)=Concat(head1,..., head h )W O

[0130] head i =TCSSA(QW i Q , KW i K , VW i V , spe q , spe k )

[0131]

[0132] In the formula, MHTCSSA represents the multi-head spatial self-attention mechanism based on time constraints, Q represents Query, K represents Key, V represents Value, Concat represents the concatenation operation, W O , W Q , W K , W V are learnable parameters, head i represents the i-th attention head, h represents the total number of attention heads, TCSSA is the single-head spatial self-attention mechanism based on time constraints, and spe q ​is the spatial location encoding of Query, spe k is the spatial location encoding of Key, d k is the dimension of the vector of Key.

[0133] S34. Input the region feature and the grid feature into the multi-head time-constrained spatial self-attention mechanism:

[0134]

[0135]

[0136] Note that the structure of the multi-layer attention mechanism is adopted here. In the formula, represents the region feature input at the l-th layer in the t-th time stage, RSPE is the region spatial location encoding, represents the grid feature input at the l-th layer in the t-th time stage, GSPE is the grid spatial location encoding. represents the region feature output at the l-th layer in the t-th time stage, represents the grid feature output at the l-th layer in the t-th time stage.

[0137] S35. The region feature or the grid feature output by the time-constrained spatial self-attention mechanism plus the position encoding PE of the time stage t :

[0138]

[0139]

[0140] are input into the time self-attention mechanism:

[0141]

[0142]

[0143] In the formula, MHTSA represents the multi-head time self-attention mechanism, and represent the region features and grid features of all time stages. represents the region feature output by the time self-attention mechanism at the l-th layer,

[0144] represents the grid feature output by the time self-attention mechanism at the l-th layer.

[0145] Subsequently, and are respectively input into the feed-forward network:

[0146]

[0147]

[0148] In the formula, FFN represents the feed-forward network, represents the regional feature output by the temporal self-attention mechanism of the l-th layer, represents the grid feature output by the temporal self-attention mechanism of the l-th layer.

[0149] S36. Design a cross-attention mechanism based on time and local constraints:

[0150] MHTLCCA(Q, K, V) = Concat(head1,..., head h )W O

[0151]

[0152]

[0153] In the formula, MHTLCCA represents the multi-head cross-attention mechanism based on time and local constraints, and TLCCA represents the single-head cross-attention mechanism based on time and local constraints. represents the geometric alignment graph of regions and grids. Please refer to Figure 2 , where each region and grid corresponds to an independent node, jointly forming the node set To ensure semantic consistency between regions and grids in the cross-attention mechanism, edges in the set are established between region nodes and grid nodes only when their respective bounding boxes intersect. represents the graph-based softmax function, which assigns zero attention weights to non-adjacent nodes in and applies the softmax function to adjacent nodes in . Note that to ensure temporal consistency, a dedicated geometric alignment graph

[0154] S37. For the regional features and grid features output by the temporal self-attention mechanism, input them into the designed cross-attention mechanism based on time and local constraints to fuse the regional features and grid features:

[0155]

[0156]

[0157] In the first formula, the region feature is used as the Query, and the grid feature is used as the Key and Value. The grid feature is embedded into the region feature to obtain the region feature output by the cross-attention mechanism based on temporal and local constraints at the l-th layer. In the second formula, the grid feature is used as the Query, and the region feature is used as the Key and Value. The region feature is embedded into the grid feature to obtain the grid feature output by the cross-attention mechanism based on temporal and local constraints at the l-th layer.

[0158] Subsequently, and are respectively input into the feed-forward network FFN:

[0159]

[0160]

[0161] Due to the adoption of the multi-layer attention mechanism, and are input into the spatial self-attention mechanism based on temporal constraints in the next layer in step S34, and steps S34 to S37 are repeatedly executed l - 1 times.

[0162] S38. Concatenate the region feature H r output by the last layer and the grid feature H g to obtain the fine-grained navigation trajectory representation u:

[0163] u = [H r ; H g

[0164] In the formula, T is the total number of time stages of the agent's trajectory, N r is the number of extracted region features, N g is the number of extracted grid features, and d model represents the dimension of the hidden layer vector.

[0165] S4. Based on the attention mechanism, fuse the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation to obtain a multi-grained navigation trajectory representation; the multi-grained navigation trajectory representation integrates key visual cues in the global dimension and the local dimension, making the generated navigation instructions more comprehensive and accurate.

[0166] Further, step S4 is specifically as follows:

[0167] S41. Based on the attention mechanism, the coarse-grained navigation trajectory representation f t g ​As the Query, the fine-grained navigation trajectory representation u obtained in step S3 is used as the Key and Value to fuse the coarse-grained and fine-grained navigation trajectory representations, obtaining a multi-grained navigation trajectory representation:

[0168]

[0169] f t gl = ∑ i β t,i u t,i

[0170] In the formula, W F represents learnable parameters, T represents the transpose symbol, and u t,i represents the i-th region / grid feature at time t. β t,i represents the attention weight of the i-th region / grid feature at time t, and f t gl represents the multi-grained navigation trajectory representation.

[0171] S42. Use Bi-LSTM to encode the multi-grained navigation trajectory representation to further capture its context dependence on time:

[0172]

[0173] S5. Use LSTM-RNN with an attention mechanism to decode the multi-grained navigation trajectory representation to obtain a synthesized navigation instruction.

[0174] S6. Based on the synthesized navigation instruction and the sampled trajectory, compose enhanced data to train the vision-language navigation agent to improve the generalization ability of the agent and the success rate of navigation.

[0175] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously.

[0176] Based on the same idea as the multi-granularity instruction generation method for vision-language navigation in the above embodiments, the present invention also provides a multi-granularity instruction generation system for vision-language navigation, which can be used to execute the above multi-granularity instruction generation method for vision-language navigation. For the convenience of description, in the structural schematic diagram of the embodiment of the multi-granularity instruction generation system for vision-language navigation, only the parts related to the embodiments of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than those illustrated, or combine some components, or arrange different components.

[0177] Please refer to Figure 3 , in another embodiment of the present application, a multi-granularity instruction generation system 100 for vision-language navigation is provided. The system includes a trajectory feature extraction module 101, a coarse-grained encoding module 102, a fine-grained encoding module 103, a navigation trajectory representation fusion module 104, a navigation instruction generation module 105, and an agent training module 106;

[0178] The trajectory feature extraction module 101 is used to obtain the trajectory of the vision-language navigation agent and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include region features and grid features;

[0179] The coarse-grained encoding module 102 is used to encode and fuse the coarse-grained global features. Specifically, it encodes the action features to capture the context dependencies of multiple time stages in the trajectory features, and then based on the attention mechanism, fuses the encoded action features and panoramic image features to obtain a coarse-grained navigation trajectory representation, and the coarse-grained navigation trajectory representation depicts the key visual cues on the navigation trajectory from a global dimension;

[0180] The fine-grained encoding module 103 is used to encode and fuse fine-grained local features. Specifically, for regional features or grid features, they are input into a preset spatio-temporal self-attention mechanism based on time constraints. Within a given time period, this attention mechanism assigns higher weights to specific regions or grids, enabling the model to focus on key visual cues for navigation in the spatial dimension. For the regional features or grid features output by the spatio-temporal self-attention mechanism based on time constraints, they are input into a temporal self-attention mechanism, enabling the model to focus on key visual cues in the temporal dimension. For the regional features and grid features output by the temporal self-attention mechanism, they are input into a preset cross-attention mechanism based on time and local constraints to fuse the regional features and grid features, obtaining a fine-grained navigation trajectory representation. The fine-grained navigation trajectory representation depicts the key visual cues on the navigation trajectory from the local dimension, representing fine-grained object semantics and color texture information.

[0181] The navigation trajectory representation fusion module 104 is used to fuse the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation based on the attention mechanism to obtain a multi-grained navigation trajectory representation. The multi-grained navigation trajectory representation integrates the key visual cues in the global dimension and the local dimension, making the generated navigation instructions more comprehensive and accurate.

[0182] The navigation instruction generation module 105 is used to decode the multi-grained navigation trajectory representation to obtain a synthesized navigation instruction.

[0183] The agent training module 106 is used to train the vision-language navigation agent based on the synthesized navigation instruction and the sampled trajectory to form enhanced data, improving the generalization ability of the agent and the success rate of navigation.

[0184] It should be noted that the multi-grained instruction generation system for vision-language navigation of the present invention corresponds one-to-one with the multi-grained instruction generation method for vision-language navigation of the present invention. The technical features and their beneficial effects described in the embodiments of the above-mentioned multi-grained instruction generation method for vision-language navigation are applicable to the embodiments of the multi-grained instruction generation for vision-language navigation. For specific content, reference can be made to the description in the method embodiments of the present invention, which will not be elaborated here. This is hereby declared.

[0185] In addition, in the implementation of the multi-granularity instruction generation system for visual language navigation in the above embodiments, the logical division of each program module is only an example. In practical applications, according to needs, for example, due to the configuration requirements of the corresponding hardware or the convenience of software implementation, the above functions can be assigned to different program modules to complete, that is, the internal structure of the multi-granularity instruction generation system for visual language navigation is divided into different program modules to complete all or part of the functions described above.

[0186] Please refer to Figure 4 , in one embodiment, an electronic device for implementing a multi-granularity instruction generation method for visual language navigation is provided. The electronic device 200 may include a first processor 201, a first memory 202, and a bus, and may further include a computer program stored in the first memory 202 and executable on the first processor 201, such as a multi-granularity instruction generation program 203 for visual language navigation.

[0187] Among them, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as: SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The first memory 202 may be an internal storage unit of the electronic device 200 in some embodiments, such as the mobile hard disk of the electronic device 200. The first memory 202 may also be an external storage device of the electronic device 200 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 200. Further, the first memory 202 may also include both an internal storage unit and an external storage device of the electronic device 200. The first memory 202 can be used not only to store application software installed in the electronic device 200 and various types of data, such as the code of the multi-granularity instruction generation program 203 for visual language navigation, but also to temporarily store data that has been output or will be output.

[0188] In some embodiments, the first processor 201 may be composed of integrated circuits. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting various components of the entire electronic device through various interfaces and circuits, and by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0189] Figure 4 Only the electronic device with components is shown. Those skilled in the art can understand that Figure 4 the shown structure does not constitute a limitation on the electronic device 200, and it may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0190] The multi-granularity instruction generation program 203 for visual language navigation stored in the first memory 202 in the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve:

[0191] Obtain the trajectory of the visual language navigation agent and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features. The global features include panoramic image features and action features, and the local features include region features and grid features;

[0192] Encode and fuse the coarse-grained global features. Specifically, encode the action features to capture the context dependencies of multiple time stages in the trajectory features, and then based on the attention mechanism, fuse the encoded action features and panoramic image features to obtain a coarse-grained navigation trajectory representation, which depicts the key visual cues on the navigation trajectory from a global dimension;

[0193] Encode and fuse fine-grained local features, specifically: for regional features or grid features, input them into a preset spatio-temporal self-attention mechanism with time constraints. During a given time period, this attention mechanism assigns higher weights to specific regions or grids, enabling the model to focus on key visual cues for navigation in the spatial dimension; for the regional features or grid features output by the spatio-temporal self-attention mechanism with time constraints, input them into a temporal self-attention mechanism, enabling the model to focus on key visual cues in the temporal dimension; for the regional features and grid features output by the temporal self-attention mechanism, input them into a preset cross-attention mechanism with time and local constraints to fuse the regional features and grid features, obtaining a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts key visual cues on the navigation trajectory from a local dimension, representing fine-grained object semantics and color texture information;

[0194] Based on the attention mechanism, fuse the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation to obtain a multi-grained navigation trajectory representation; the multi-grained navigation trajectory representation fuses key visual cues in both the global and local dimensions, making the generated navigation instructions more comprehensive and accurate;

[0195] Decode the multi-grained navigation trajectory representation to obtain a synthesized navigation instruction;

[0196] Based on the synthesized navigation instruction and the sampled trajectory, form enhanced data to train the vision-language navigation agent, improving the generalization ability of the agent and the success rate of navigation.

[0197] Furthermore, if the modules / units integrated in the electronic device 200 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory).

[0198] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0199] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0200] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A multi-granularity instruction generation method for visual language navigation, characterized in that, Including the following steps: Obtain the trajectory of the vision-language navigation agent and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include region features and grid features; Encode and fuse the coarse-grained global features, specifically: Encode the action features to capture the context dependencies of multiple time stages in the trajectory features, and then, based on the attention mechanism, fuse the encoded action features and panoramic image features to obtain a coarse-grained navigation trajectory representation, and the coarse-grained navigation trajectory representation depicts the key visual cues on the navigation trajectory from a global dimension; Encode and fuse the fine-grained local features, specifically: For the region features or grid features, input them into a preset spatio-temporal self-attention mechanism with time constraints. Within a given time period, this attention mechanism assigns higher weights to specific regions or grids, so that the model focuses on the key visual cues for navigation in the spatial dimension; for the region features or grid features output by the spatio-temporal self-attention mechanism with time constraints, input them into the temporal self-attention mechanism, so that the model focuses on the key visual cues in the temporal dimension; for the region features and grid features output by the temporal self-attention mechanism, input them into a preset cross-attention mechanism with time and local constraints to fuse the region features and grid features to obtain a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts the key visual cues on the navigation trajectory from a local dimension, representing fine-grained object semantics and color texture information; Based on the attention mechanism, fuse the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation to obtain a multi-granularity navigation trajectory representation; the multi-granularity navigation trajectory representation fuses the key visual cues in the global dimension and the local dimension, making the generated navigation instructions more comprehensive and accurate; Decode the multi-granularity navigation trajectory representation to obtain a synthesized navigation instruction; Based on the synthesized navigation instruction and the sampled trajectory, form enhanced data to train the vision-language navigation agent, improving the generalization ability of the agent and the success rate of navigation.

2. The multi-granularity instruction generation method for visual language navigation according to claim 1, wherein The obtaining of the trajectory of the vision-language navigation agent and the extraction of the trajectory features of the agent are specifically: Extract the panoramic image features, which are composed of the ResNet image features and the orientation features of the agent spliced together; Extract the action features. In each time stage, the agent selects an image from the panoramic image as the direction to go in the next time stage and moves to the corresponding path node. Therefore, the action feature corresponds to one of the panoramic image features; Extract the region features and grid features. Given an image, use Faster R-CNN to extract multiple regions and multiple grids, and splice each region or grid with the embedding function of its orientation feature to obtain the region features and grid features.

3. The multi-granularity instruction generation method for visual language navigation according to claim 1, characterized in that The encoding and fusion of the coarse-grained global features are specifically: Encode action features using Bi-LSTM: where r1,...,r T respectively represent the trajectory encodings obtained by encoding the action features in the 1st to the Tth time periods, where T represents the total number of time stages, i.e., the number of nodes on the agent's trajectory, respectively represent the 1st to the Tth action feature encodings; Based on the attention mechanism, fuse the trajectory encoding of action features and panoramic image features, where the trajectory encoding serves as the Query, and the panoramic image features serve as the Key and Value, to obtain a coarse-grained navigation trajectory representation, as shown below: f t g = ∑ i α t,i f t,i Where, α t,i represents the attention weight of the i-th panoramic image at time t, f t,i represents the feature of the i-th panoramic image at time t, T represents the transpose symbol, W R represents the learnable parameter, r t represents the trajectory encoding of the t-th time period, f t g represents the coarse-grained navigation trajectory representation.

4. The multi-granularity instruction generation method for visual language navigation according to claim 1, wherein The encoding and fusion of fine-grained local features are specifically as follows: For regional features, obtain their regional spatial position encoding; For grid features, obtain their grid spatial position encoding; Design a time-constrained spatial self-attention mechanism, specifically: MHTCSSA(Q, K, V) = Concat(head1,..., head h )W O head i = TCSSA(QW i Q ,KW i K ,VW i V ,spe q ,spe k ) Wherein, MHTCSSA represents the multi - head time - constrained spatial self - attention mechanism, Q represents Query, K represents Key, V represents Value, Concat represents the concatenation operation, W O , W Q , W K , W V are learnable parameters, head i represents the i - th attention head, h represents the total number of attention heads, TCSSA is the single - head time - constrained spatial self - attention mechanism, spe q is the spatial position encoding of Query, spe k is the spatial position encoding of Key, d k is the dimension of the vector of Key; Input regional features and grid features into a multi-head time-constrained spatial self-attention mechanism: In the formula, represents the regional feature of the input at the l-th layer in the t-th time stage, and RSPE is the regional spatial position encoding, represents the grid feature of the input at the l-th layer in the t-th time stage, and GSPE is the grid spatial position encoding, represents the regional feature of the output at the l-th layer in the t-th time stage, represents the grid feature of the output at the l-th layer in the t-th time stage; Regional features output by the spatial self-attention mechanism based on time constraints Or grid features Plus the position encoding PE of the time stage t : Input into the time self-attention mechanism: wherein, MHTSA represents the multi-head temporal self-attention mechanism, and represents the regional features and grid features of all time stages, represents the regional features output by the temporal self-attention mechanism of the l-th layer, represents the grid features output by the temporal self-attention mechanism of the l-th layer; Input and into the forward network respectively: where FFN represents the feed-forward network, represents the region feature output by the temporal self-attention mechanism of the l-th layer, represents the grid feature output by the temporal self-attention mechanism of the l-th layer; Design a cross-attention mechanism based on time and local constraints: MHTLCCA(Q, K, V) = Concat(head1,..., head h )W O Wherein, MHTLCCA represents the multi-head time- and locality-constrained cross-attention mechanism, and TLCCA represents the single-head time- and locality-constrained cross-attention mechanism. represents the geometric alignment graph of regions and grids, in which each region and grid corresponds to an independent node, jointly constituting the node set To ensure semantic consistency between regions and grids in the cross-attention mechanism, edges in the set will be established between the region nodes and grid nodes only when their respective bounding boxes intersect. represents the graph-based softmax function, which assigns zero attention weights to non-adjacent nodes in and applies the softmax function to adjacent nodes in To ensure temporal consistency, a dedicated geometric alignment graph is created for each image at each time stage t. For the regional features and grid features output by the time self-attention mechanism, input them into the designed cross-attention mechanism based on time and local constraints to fuse the regional features and grid features: In In the formula, the region feature serves as the Query, the grid feature serves as the Key and Value, and the grid feature is embedded into the region feature to obtain the region feature output by the cross-attention mechanism based on temporal and local constraints at the l-th layer In In the formula, the grid feature serves as the Query, the region feature serves as the Key and Value, and the region feature is embedded into the grid feature to obtain the grid feature output by the cross-attention mechanism based on temporal and local constraints at the l-th layer Subsequently, and are respectively input into the forward network FFN: Due to the adoption of a multi-layer attention mechanism, and it is input into the time-constrained spatio-self-attention mechanism of the next layer; Concatenate the regional feature H output by the last layer r and the grid feature H g to obtain the fine-grained navigation trajectory representation u: u = [H r ; H g ​ In the formula, T is the total number of time stages of the agent's trajectory, N r is the number of extracted regional features, N g is the number of extracted grid features, d model represents the dimension of the vector in the hidden layer.

5. The multi-granularity instruction generation method for visual language navigation according to claim 4, wherein Specifically, obtaining the regional spatial position encoding of regional features is as follows: RSPE(i) = Emb(B i ) where i represents the i-th region in the image, B i = (x min , y min , x max , y max ) represents the coordinates of the upper left and lower right corners of the bounding box of the region, Emb(·) represents the projection layer, and RSPE represents the spatial position encoding of the region.

6. The multi-granularity instruction generation method for visual language navigation according to claim 4, wherein Specifically, obtaining the grid spatial position encoding of grid features is as follows: GSPE(i,j) = [PE i ; PE j ​ wherein, the grid divides the image into i×j cells, i represents the row index of the grid, j represents the column index of the grid, PE i , PE j respectively represent the position encodings of i and j, and GSPE represents the grid spatial position encoding.

7. The multi-granularity instruction generation method for vision-language navigation according to claim 1, wherein Based on the attention mechanism, fuse the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation to obtain a multi-grained navigation trajectory representation, specifically: Based on the attention mechanism, the coarse-grained navigation trajectory representation f t g is used as the Query, and the fine-grained navigation trajectory representation u is used as the Key and Value to fuse the coarse-grained and fine-grained navigation trajectory representations, obtaining a multi-grained navigation trajectory representation: f t gl = ∑ i β t,i u t,i Where, W F represents learnable parameters, T represents the transpose symbol, and u t,i represents the i-th region / mesh feature at time t, and β t,i represents the attention weight of the i-th region / mesh feature at time t, and f t gl represents the multi-granularity navigation trajectory representation; Use Bi-LSTM to encode the multi-grained navigation trajectory representation to further capture its context dependence on time:

8. A multi-granularity instruction generation system for vision-language navigation, characterized in that Applied to the multi-grained instruction generation method for vision-language navigation according to any one of claims 1-7, including a trajectory feature extraction module, a coarse-grained encoding module, a fine-grained encoding module, a navigation trajectory representation fusion module, a navigation instruction generation module, and an agent training module; The trajectory feature extraction module is used to obtain the trajectory of the vision-language navigation agent and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include regional features and grid features; The coarse-grained encoding module is used to encode and fuse the coarse-grained global features, specifically: encode the action features to capture the context dependence of multiple time stages in the trajectory features, and then based on the attention mechanism, fuse the encoded action features and panoramic image features to obtain a coarse-grained navigation trajectory representation, and the coarse-grained navigation trajectory representation depicts the key visual cues on the navigation trajectory from a global dimension; The fine-grained encoding module is used to encode and fuse fine-grained local features. Specifically, for regional features or grid features, they are input into a preset spatio-temporal self-attention mechanism with time constraints. Within a given time period, this attention mechanism assigns higher weights to specific regions or grids, enabling the model to focus on key visual cues for navigation in the spatial dimension. For the regional features or grid features output by the spatio-temporal self-attention mechanism with time constraints, they are input into the temporal self-attention mechanism, enabling the model to focus on key visual cues in the temporal dimension. For the regional features and grid features output by the temporal self-attention mechanism, they are input into a preset cross-attention mechanism with time and local constraints to fuse the regional features and grid features, obtaining a fine-grained navigation trajectory representation. The fine-grained navigation trajectory representation depicts the key visual cues on the navigation trajectory from a local dimension, representing fine-grained object semantics and color texture information. The navigation trajectory representation fusion module, based on the attention mechanism, fuses the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation to obtain a multi-grained navigation trajectory representation. The multi-grained navigation trajectory representation integrates the key visual cues in the global dimension and the local dimension, making the generated navigation instructions more comprehensive and accurate. The navigation instruction generation module is used to decode the multi-grained navigation trajectory representation to obtain a synthesized navigation instruction. The agent training module is used to train a visual language navigation agent based on the synthesized navigation instruction and the sampled trajectory to form enhanced data, improving the generalization ability of the agent and the success rate of navigation.

9. An electronic device, characterized in that, The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the multi-grained instruction generation method for visual language navigation according to any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the multi-grained instruction generation method for visual language navigation according to any one of claims 1-7.

Images