Multi-granularity instruction generation method and device oriented to visual language navigation

Abstract

The invention discloses a multi-granularity instruction generation method and device oriented to visual language navigation. The method comprises the following steps: extracting panoramic image features and action features of coarse granularity and regional features and grid features of fine granularity; coding and fusing the global features by adopting Bi-LSTM and an attention mechanism to obtain coarse-grained trajectory representation; space self-attention and time self-attention based on time constraint and cross attention based on time and local constraint are designed, local features are coded and fused, and fine-grained trajectory representation is obtained; the characterization of the coarse granularity and the characterization of the fine granularity are fused, and multi-granularity track characterization is generated; decoding is carried out through the LSTM-RNN, and a navigation instruction is synthesized; and the synthesized instruction and the sampled trajectory form enhanced data which is used for training a navigation agent. From the perspective of data enhancement, a high-quality natural language navigation instruction is synthesized, the generalization ability of an intelligent agent is improved, and an innovative and practical solution is provided.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual language navigation, and in particular relates to a method and device for generating multi-granularity instructions for visual language navigation. Background Art

[0002] In the interaction of human society, natural language, as a basic tool for communication, makes communication between people efficient and rich. Imagine if intelligent systems can understand and respond to human language commands, such as performing specific actions or going to designated locations, which can not only replace manual operations, but also significantly improve production efficiency and social service levels. This type of technology has a wide range of applications, covering home service robots, logistics automation equipment, and intelligent machinery in the construction field. The research on visual language navigation technology aims to promote the development of embodied intelligence, that is, to enable intelligent systems to gradually acquire the ability to understand and operate the physical world similar to humans through continuous learning and adaptation in the interaction with the actual or simulated environment. This progress is crucial to breaking through the limitations of the existing "narrow artificial intelligence" and enhancing the cognitive level of machines.

[0003] In 2018, Anderson et al. first proposed the concept of visual language navigation and constructed a Room-to-Room (R2R) dataset for this purpose. This dataset creates a series of indoor scenes based on 3D reconstruction technology, each of which consists of multiple discrete nodes for the agent to move. In the experimental setting, the agent will first be randomly placed at a certain point, and then decide which direction to move next based on the natural language instructions received and the panoramic view of the current position. As the position of the agent changes, the visual information it obtains is also updated. The agent needs to repeat this process until it reaches the target point or reaches the preset maximum number of actions. When the distance between the agent's final stop position and the destination specified in the instruction is within an acceptable range, the task is considered completed.

[0004] However, visual language navigation faces multiple challenges. The first difficulty is to effectively integrate and match multimodal information in complex environments. Secondly, designing a reasonable reasoning mechanism to guide the navigation decision of the intelligent agent is also a major challenge. In addition, due to the high cost of collecting high-quality data, the existing datasets are relatively small and insufficient to fully support such complex task requirements, which makes the model prone to overfitting. Especially when faced with new scenes that have never been seen before, the performance of the model is often unsatisfactory. These factors together limit the development of visual language navigation technology and its practical application effects. Summary of the invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a multi-granularity instruction generation method and device for visual language navigation, which can automatically generate high-quality natural language navigation instructions based on path data obtained in a simulation environment. The generated instructions are combined with the corresponding paths to form an enhanced data set, which helps to alleviate the challenges of data sparsity and further enhance the generalization performance of the visual language navigation agent, specifically by improving the success rate of the agent's navigation based on language instructions in an unknown environment.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for generating multi-granularity instructions for visual language navigation, comprising the following steps:

[0008] Acquire the trajectory of the visual language navigation agent and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include regional features and grid features;

[0009] Encoding and fusing the coarse-grained global features, specifically: encoding the action features to capture the contextual dependencies of multiple time stages in the trajectory features, and then fusing the encoded action features and panoramic image features based on the attention mechanism to obtain a coarse-grained navigation trajectory representation, wherein the coarse-grained navigation trajectory representation depicts the key visual clues on the navigation trajectory from a global dimension;

[0010] The fine-grained local features are encoded and fused, specifically: for regional features or grid features, they are input into a preset spatial self-attention mechanism based on time constraints. Within a given time period, the attention mechanism assigns a higher weight to a specific region or grid, so that the model focuses on the key visual clues for navigation in the spatial dimension; for regional features or grid features output by the spatial self-attention mechanism based on time constraints, they are input into the temporal self-attention mechanism, so that the model focuses on the key visual clues in the temporal dimension; for regional features and grid features output by the temporal self-attention mechanism, they are input into a preset cross-attention mechanism based on time and local constraints, and the regional features and grid features are fused to obtain a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts the key visual clues on the navigation trajectory from a local dimension, and represents fine-grained object semantics and color texture information;

[0011] Based on the attention mechanism, the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation are fused to obtain a multi-grained navigation trajectory representation; the multi-grained navigation trajectory representation integrates key visual clues of the global dimension and the local dimension, making the generated navigation instructions more comprehensive and accurate;

[0012] Decoding the multi-granularity navigation trajectory representation to obtain a synthesized navigation instruction;

[0013] Based on the synthesized navigation instructions and the sampled trajectory composition enhanced data, a visual language navigation agent is trained to improve the agent's generalization ability and navigation success rate.

[0014] As a preferred technical solution, the acquisition of the trajectory of the visual language navigation agent and the extraction of the trajectory features of the agent are specifically as follows:

[0015] Extracting panoramic image features, where the panoramic image features are composed of ResNet image features and the orientation features of the agent;

[0016] Extracting action features, in each time stage, the agent selects an image from the panoramic image as the direction to go in the next time stage, and moves to the corresponding path node, so the action feature corresponds to one of the panoramic image features;

[0017] Extract regional features and grid features. Given an image, use Faster R-CNN to extract multiple regions and multiple grids. Each region or grid is concatenated with the embedding function of its orientation feature to obtain regional features and grid features.

[0018] As a preferred technical solution, the encoding and fusion of the coarse-grained global features are specifically as follows:

[0019] Use Bi-LSTM to encode action features:

[0020]

[0021] In the formula, r1,...,r T They represent the trajectory encodings obtained by encoding the action features from the 1st to the Tth time period, respectively. T represents the total time stage, that is, the number of nodes on the trajectory of the agent. Respectively represent the 1st to Tth action feature codes;

[0022] Based on the attention mechanism, the trajectory encoding of the action features and the panoramic image features are fused, where the trajectory encoding is used as the query and the panoramic image features are used as the key and value to obtain a coarse-grained navigation trajectory representation, which is expressed as follows:

[0023]

[0024] f t g =∑ i α t,i f t,i

[0025] In the formula, α t,i represents the attention weight of the i-th panoramic image at time t, f t,i represents the features of the i-th panoramic image at time t, T represents the transposition symbol, and W R represents the learnable parameters, r t represents the trajectory code of the tth time period, f t g Represents a coarse-grained representation of navigation trajectories.

[0026] As a preferred technical solution, the encoding and fusion of fine-grained local features are specifically as follows:

[0027] For regional features, obtain the regional spatial position code;

[0028] For the grid feature, obtain its grid space position encoding;

[0029] Design a spatial self-attention mechanism based on time constraints, specifically:

[0030] MHTCSSA(Q,K,V)=Concat(head1,...,head h )W O

[0031] head i =TCSSA(QW i Q ,KW i K ,VW i V ,spe q ,spe k )

[0032]

[0033] In the formula, MHTCSSA represents a multi-head spatial self-attention mechanism based on time constraints, Q represents Query, K represents Key, V represents Value, Concat represents concatenation operation, and W O , W Q , W K , W V is a learnable parameter, head irepresents the i-th attention head, h represents the total number of attention heads, TCSSA is a single-head spatial self-attention mechanism based on time constraints, spe q is the spatial position encoding of the Query, spe k is the spatial position encoding of the Key, d k is the dimension of the Key vector;

[0034] The regional features and grid features are input into the multi-head temporal constraint-based spatial self-attention mechanism:

[0035]

[0036]

[0037] In the formula, represents the regional features of the lth layer input at the tth time stage, RSPE is the regional spatial position encoding,

[0038] represents the grid feature of the lth layer input at the tth time stage. GSPE is the grid spatial position encoding. represents the regional features of the output of the lth layer at the tth time stage, Represents the grid features output by the lth layer at the tth time stage;

[0039] Regional features output by spatial self-attention mechanism based on time constraints Or mesh features Plus the position encoding PE of the time stage t :

[0040]

[0041]

[0042] Input to the temporal self-attention mechanism:

[0043]

[0044]

[0045] Where MHTSA stands for multi-head temporal self-attention mechanism, and Represents regional features and grid features for all time stages, Represents the regional features output by the temporal self-attention mechanism of layer l,

[0046] Represents the grid features output by the temporal self-attention mechanism of layer l;

[0047] Will and Input into the forward network respectively:

[0048]

[0049]

[0050] Where FFN stands for feed-forward network, Represents the regional features output by the temporal self-attention mechanism of layer l, Represents the grid features output by the temporal self-attention mechanism of layer l;

[0051] Design a criss-cross attention mechanism based on temporal and local constraints:

[0052] MHTLCCA(Q,K,V)=Concat(head1,...,head h )W O

[0053]

[0054]

[0055] Where MHTLCCA represents a multi-headed cross-attention mechanism based on time and local constraints, and TLCCA represents a single-headed cross-attention mechanism based on time and local constraints. A geometric alignment graph representing regions and grids, in which each region and grid corresponds to an independent node, which together constitute a node set To ensure semantic consistency between regions and grids in the criss-cross attention mechanism, the set The edges in will be established between the region nodes and the grid nodes. Represents the graph-based softmax function. The attention weight assigned to non-adjacent nodes in The softmax function is applied to adjacent nodes in ; to ensure temporal consistency, a dedicated geometric alignment graph is created for each image at time stage t

[0056] The regional features and grid features output by the temporal self-attention mechanism are input into the designed cross-attention mechanism based on temporal and local constraints to fuse the regional features and grid features:

[0057]

[0058]

[0059] exist In the formula, the regional feature is used as the query, the grid feature is used as the key and value, and the grid feature is embedded into the regional feature to obtain the regional feature output by the cross-attention mechanism based on time and local constraints at the lth layer. exist In the formula, the grid feature is used as the query, the regional feature is used as the key and value, and the regional feature is embedded into the grid feature to obtain the grid feature output by the cross-attention mechanism based on time and local constraints at the lth layer.

[0060] Then will and Input into the forward network FFN respectively:

[0061]

[0062]

[0063] Due to the multi-layer attention mechanism, and Input to the next layer’s spatial self-attention mechanism based on time constraints;

[0064] The regional features H output by the last layer r and the mesh feature H g Splicing to obtain a fine-grained navigation trajectory representation u:

[0065] u=[H r ;H g ]

[0066] In the formula, T is the total number of time stages of the agent’s trajectory, N r is the number of extracted regional features, N g is the number of mesh features extracted, d model The dimension of the vector representing the hidden layer.

[0067] As a preferred technical solution, the regional spatial position coding for obtaining regional features is specifically:

[0068] RSPE(i)=Emb(B i );

[0069] In the formula, i represents the i-th region in the image, B i =(x min ,y min ,x max ,y max) represents the coordinates of the upper left corner and the lower right corner of the bounding box of the region, Emb(·) represents the projection layer, and RSPE represents the spatial position encoding of the region.

[0070] As a preferred technical solution, the grid spatial position encoding for obtaining grid features is specifically:

[0071] GSPE(i,j)=[PE i PE j ]

[0072] In the formula, the grid divides the image into i×j grids, i represents the row index of the grid, j represents the column index of the grid, PE i , PE j They represent the position encoding of i and j respectively, and GSPE represents grid space position encoding.

[0073] As a preferred technical solution, the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation are fused based on the attention mechanism to obtain a multi-grained navigation trajectory representation, specifically:

[0074] Based on the attention mechanism, the coarse-grained navigation trajectory is represented as t g As Query, the fine-grained navigation trajectory representation u is used as Key and Value, and the coarse-grained and fine-grained navigation trajectory representations are fused to obtain a multi-grained navigation trajectory representation:

[0075]

[0076] f t gl =∑ i β t,i u t,i

[0077] Where W F represents the learnable parameters, T represents the transposed symbol, and u t,i represents the i-th region / grid feature at time t, β t,i represents the attention weight of the i-th region / grid feature at time t, f t gl Represents multi-granularity representation of navigation trajectories;

[0078] Bi-LSTM is used to encode the multi-granularity navigation trajectory representation to further capture its contextual dependency on time:

[0079]

[0080] In a second aspect, the present invention provides a multi-granularity instruction generation system for visual language navigation, which is applied to the multi-granularity instruction generation method for visual language navigation, including a trajectory feature extraction module, a coarse-grained encoding module, a fine-grained encoding module, a navigation trajectory representation fusion module, a navigation instruction generation module and an intelligent agent training module;

[0081] The trajectory feature extraction module is used to obtain the trajectory of the visual language navigation agent and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include regional features and grid features;

[0082] The coarse-grained encoding module is used to encode and fuse the coarse-grained global features, specifically: encode the action features to capture the contextual dependencies of multiple time stages in the trajectory features, and then fuse the encoded action features and panoramic image features based on the attention mechanism to obtain a coarse-grained navigation trajectory representation, which describes the key visual clues on the navigation trajectory from a global dimension;

[0083] The fine-grained encoding module is used to encode and fuse fine-grained local features, specifically: for regional features or grid features, they are input into a preset spatial self-attention mechanism based on time constraints. Within a given time period, the attention mechanism assigns a higher weight to a specific region or grid, so that the model focuses on the key visual clues for navigation in the spatial dimension; for regional features or grid features output by the spatial self-attention mechanism based on time constraints, they are input into the temporal self-attention mechanism, so that the model focuses on the key visual clues in the temporal dimension; for regional features and grid features output by the temporal self-attention mechanism, they are input into a preset cross-attention mechanism based on time and local constraints, and the regional features and grid features are fused to obtain a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts the key visual clues on the navigation trajectory from a local dimension, and represents fine-grained object semantics and color texture information;

[0084] The navigation trajectory representation fusion module, based on the attention mechanism, fuses the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation to obtain a multi-grained navigation trajectory representation; the multi-grained navigation trajectory representation fuses the key visual clues of the global dimension and the local dimension, so that the generated navigation instructions are more comprehensive and accurate;

[0085] The navigation instruction generation module is used to decode the multi-granularity navigation trajectory representation to obtain a synthesized navigation instruction;

[0086] The agent training module is used to train a visual language navigation agent based on the synthesized navigation instructions and the sampled trajectory composition enhanced data, thereby improving the generalization ability of the agent and the navigation success rate.

[0087] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0088] at least one processor; and a memory in communication with the at least one processor; wherein,

[0089] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the multi-granularity instruction generation method for visual language navigation.

[0090] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the multi-granularity instruction generation method for visual language navigation.

[0091] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0092] 1. Existing visual language navigation technology mainly focuses on how to improve the navigation agent, and pays relatively little attention to synthesizing navigation instructions to provide detailed navigation path explanations. However, due to the problem of data sparsity, the generalization ability of the agent is limited, that is, the navigation success rate in unfamiliar scenes is not high. From the perspective of data enhancement, the present invention can generate high-quality synthetic instructions, thereby improving the navigation success rate and generalization ability of the agent.

[0093] 2. Although the existing command synthesis technology for visual language navigation can alleviate the problem of data sparsity to a certain extent, the quality of synthesized commands is low. This is because the existing technology only uses coarse-grained image-level representation to generate commands, which makes it difficult to capture the important details required to reproduce the navigation trajectory.

[0094] 3. The present invention integrates coarse-grained image features and fine-grained key visual clues on the navigation trajectory, which can cover the global environmental information and local object information required for navigation decisions, making the generated navigation instructions more comprehensive and accurate, and solving the problem of poor quality of instructions synthesized by the prior art.

[0095] In summary, the present invention provides an innovative and practical solution from the perspective of data enhancement, which can synthesize high-quality natural language navigation instructions and improve the generalization ability of navigation agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0097] Figure 1 A flowchart of a method for generating multi-granularity instructions for visual language navigation according to an embodiment of the present invention;

[0098] Figure 2 It is a schematic diagram of the structure of a geometric alignment diagram according to an embodiment of the present invention;

[0099] Figure 3 A framework diagram of a multi-granularity instruction generation system for visual language navigation according to an embodiment of the present invention;

[0100] Figure 4 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0101] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0102] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0103] See also Figure 1 This embodiment provides a multi-granularity instruction generation method for visual language navigation, which combines coarse-grained image features on the navigation trajectory and fine-grained key visual clues to synthesize high-quality navigation instructions, and includes the following steps:

[0104] S1. Sampling is performed in a simulation environment to obtain a series of trajectories of a visual language navigation agent and extract trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include regional features and grid features.

[0105] Furthermore, step S1 is specifically as follows:

[0106] S11, extracting panoramic image features, panoramic image features It is composed of the concatenation of ResNet image features and the agent’s orientation features:

[0107] f t,i =[ResNet(v t,i );E o (θ t,i ,φ t,i )]

[0108] In the formula, t represents the time stage that the agent is in, represents the 36 panoramic images at the current time stage t, ResNet(·) represents the ResNet convolutional neural network, and E o (·) represents the embedding function that encodes the orientation feature, θ t,i represents the orientation angle of the i-th panoramic image at the current time stage t, φ t,i Represents the altitude angle of the i-th panoramic image at the current time stage t.

[0109] S12. At each time stage, the agent selects an image from 36 panoramic images As the direction to go in the next time stage, and move to the corresponding path node. Therefore, the action feature corresponds to the panoramic image feature One of them is formulated as

[0110] S13. Given an image, use Faster R-CNN to extract N r regions and N g grids; each region or grid has an embedding function E with its orientation feature o (θ t,i ,φ t,i ) to obtain regional features and mesh features

[0111] S2. Encode and fuse the coarse-grained global features, specifically: encode the action features to capture the contextual dependencies of multiple time stages in the trajectory features, and then fuse the encoded action features and panoramic image features based on the attention mechanism to obtain a coarse-grained navigation trajectory representation, which depicts the key visual clues on the navigation trajectory from a global dimension.

[0112] Furthermore, step S2 is specifically as follows:

[0113] S21. Use Bi-LSTM to encode action features:

[0114]

[0115] Where T represents the total time stage, that is, the number of nodes on the agent's trajectory. r represents the trajectory encoding obtained by encoding the action features.

[0116] S22. Based on the attention mechanism, the trajectory encoding of the action features and the panoramic image features are fused; the trajectory encoding is used as the query, and the panoramic image features are used as the key and value to obtain a coarse-grained navigation trajectory representation:

[0117]

[0118]

[0119] In the formula, α t,i represents the attention weight of the i-th panoramic image at time t, f t,i represents the features of the i-th panoramic image at time t, T represents the transposition symbol, and W R represents the learnable parameters, r t represents the trajectory code of the tth time period, f t g Represents a coarse-grained representation of navigation trajectories.

[0120] S3. Encode and fuse fine-grained local features, specifically: for regional features or grid features, input them into a preset spatial self-attention mechanism based on time constraints. Within a given time period, the attention mechanism assigns higher weights to specific regions or grids, so that the model focuses on the key visual cues for navigation in the spatial dimension; for regional features or grid features output by the spatial self-attention mechanism based on time constraints, input them into the temporal self-attention mechanism, so that the model focuses on the key visual cues in the temporal dimension; for regional features and grid features output by the temporal self-attention mechanism, input them into a preset cross-attention mechanism based on time and local constraints, fuse the regional features and grid features, and obtain a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts the key visual cues on the navigation trajectory from a local dimension, and represents fine-grained object semantics and color texture information.

[0121] Furthermore, step S3 is specifically as follows:

[0122] S31. For regional features, obtain the regional spatial position code:

[0123] RSPE(i)=Emb(B i )

[0124] In the formula, i represents the i-th region in the image, B i =(x min ,y min ,x max ,y max ) represents the coordinates of the upper left corner and the lower right corner of the bounding box of the region, Emb(·) represents the projection layer, and RSPE represents the spatial position encoding of the region.

[0125] S32. For the grid feature, obtain its grid space position code:

[0126] GSPE(i,j)=[PE i PE j ]

[0127] In the formula, since the grid divides the image into i×j grids, i represents the row index of the grid, j represents the column index of the grid, and PE i , PE j They represent the position encoding of i and j respectively, and GSPE represents grid space position encoding.

[0128] S33. Design of spatial self-attention mechanism based on time constraints:

[0129] MHTCSSA(Q,K,V)=Concat(head1,...,head h )W O

[0130] head i =TCSSA(QW i Q ,KW i K ,VW i V ,spe q ,spe k )

[0131]

[0132] In the formula, MHTCSSA represents a multi-head spatial self-attention mechanism based on time constraints, Q represents Query, K represents Key, V represents Value, Concat represents concatenation operation, and W O , W Q , W K , W V is a learnable parameter, head i represents the i-th attention head, h represents the total number of attention heads, TCSSA is a single-head spatial self-attention mechanism based on time constraints, spe qis the spatial position encoding of the Query, spe k is the spatial position encoding of the Key, d k Is the dimension of the Key vector.

[0133] S34. Input the regional features and grid features into the multi-headed time-constrained spatial self-attention mechanism:

[0134]

[0135]

[0136] Note that a multi-layer attention mechanism is used here, where: represents the regional features of the lth layer input at the tth time stage, RSPE is the regional spatial position encoding, Represents the grid features of the l-th layer input at the t-th time stage, and GSPE is the grid spatial position encoding. represents the regional features of the output of the lth layer at the tth time stage, Represents the grid features output by the lth layer at the tth time stage.

[0137] S35. Regional features output by the spatial self-attention mechanism based on time constraints Or mesh features Plus the position encoding PE of the time stage t :

[0138]

[0139]

[0140] Input to the temporal self-attention mechanism:

[0141]

[0142]

[0143] Where MHTSA stands for multi-head temporal self-attention mechanism, and Represents regional features and grid features for all time stages. Represents the regional features output by the temporal self-attention mechanism of layer l,

[0144] Represents the grid features output by the temporal self-attention mechanism at layer l.

[0145] Then, and Input into the forward network respectively:

[0146]

[0147]

[0148] Where FFN stands for feed-forward network, Represents the regional features output by the temporal self-attention mechanism of layer l, Represents the grid features output by the temporal self-attention mechanism at layer l.

[0149] S36. Design a cross-attention mechanism based on time and local constraints:

[0150] MHTLCCA(Q,K,V)=Concat(head1,...,head h )W O

[0151]

[0152]

[0153] Where MHTLCCA represents a multi-headed cross-attention mechanism based on time and local constraints, and TLCCA represents a single-headed cross-attention mechanism based on time and local constraints. Geometric alignment diagrams representing regions and grids, see Figure 2 , each area and grid corresponds to an independent node, together forming a node set To ensure semantic consistency between regions and grids in the criss-cross attention mechanism, the set The edges in will be established between the region nodes and the grid nodes. Represents the graph-based softmax function. The attention weight assigned to non-adjacent nodes in The softmax function is applied to adjacent nodes in . Note that to ensure temporal consistency, a dedicated geometric alignment graph is created for each image at time step t.

[0154] S37. The regional features and grid features output by the temporal self-attention mechanism are input into the designed cross-attention mechanism based on temporal and local constraints to fuse the regional features and grid features:

[0155]

[0156]

[0157] In the first formula, the regional feature is used as the query, the grid feature is used as the key and value, and the grid feature is embedded into the regional feature to obtain the regional feature output by the cross-attention mechanism based on time and local constraints at the lth layer. In the second formula, the grid feature is used as the query, the regional feature is used as the key and value, and the regional feature is embedded into the grid feature to obtain the grid feature output by the cross-attention mechanism based on time and local constraints at the lth layer.

[0158] Then, and Input into the forward network FFN respectively:

[0159]

[0160]

[0161] Due to the multi-layer attention mechanism, and Input into the time-constrained spatial self-attention mechanism of the next layer in step S34, and repeat steps S34 to S37 l-1 times.

[0162] S38, the regional feature H output by the last layer r and the mesh feature H g Splicing to obtain a fine-grained navigation trajectory representation u:

[0163] u=[H r ;H g ]

[0164] In the formula, T is the total number of time stages of the agent’s trajectory, N r is the number of extracted regional features, N g is the number of mesh features extracted, d model The dimension of the vector representing the hidden layer.

[0165] S4. Based on the attention mechanism, the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation are fused to obtain a multi-grained navigation trajectory representation; the multi-grained navigation trajectory representation integrates the key visual clues of the global dimension and the local dimension, so that the generated navigation instructions are more comprehensive and accurate.

[0166] Furthermore, step S4 is specifically as follows:

[0167] S41, based on the attention mechanism, the coarse-grained navigation trajectory representation f in step S2 t gAs Query, the fine-grained navigation trajectory representation u obtained in step S3 is used as Key and Value, and the coarse-grained and fine-grained navigation trajectory representations are fused to obtain a multi-grained navigation trajectory representation:

[0168]

[0169] f t gl =∑ i β t,i u t,i

[0170] Where W F represents the learnable parameters, T represents the transposed symbol, and u t,i Represents the i-th region / grid feature at time t. β t,i represents the attention weight of the i-th region / grid feature at time t, f t gl Represents a multi-granular representation of navigation trajectories.

[0171] S42. Use Bi-LSTM to encode the multi-granularity navigation trajectory representation to further capture its contextual dependency on time:

[0172]

[0173] S5. Use LSTM-RNN with attention mechanism to decode the multi-granularity navigation trajectory representation to obtain the synthesized navigation instructions.

[0174] S6. Based on the synthesized navigation instructions and the sampled trajectory composition enhanced data, a visual language navigation agent is trained to improve the generalization ability of the agent and the navigation success rate.

[0175] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0176] Based on the same idea as the multi-granularity instruction generation method for visual language navigation in the above embodiment, the present invention also provides a multi-granularity instruction generation system for visual language navigation, which can be used to execute the multi-granularity instruction generation method for visual language navigation. For ease of explanation, the structural diagram of the embodiment of the multi-granularity instruction generation system for visual language navigation only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than the illustrated one, or combine certain components, or arrange the components differently.

[0177] See also Figure 3 In another embodiment of the present application, a multi-granularity instruction generation system 100 for visual language navigation is provided, the system comprising a trajectory feature extraction module 101, a coarse-grained encoding module 102, a fine-grained encoding module 103, a navigation trajectory representation fusion module 104, a navigation instruction generation module 105, and an agent training module 106;

[0178] The trajectory feature extraction module 101 is used to obtain the trajectory of the visual language navigation agent and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include regional features and grid features;

[0179] The coarse-grained encoding module 102 is used to encode and fuse the coarse-grained global features, specifically: encode the action features to capture the contextual dependencies of multiple time stages in the trajectory features, and then fuse the encoded action features and panoramic image features based on the attention mechanism to obtain a coarse-grained navigation trajectory representation, wherein the coarse-grained navigation trajectory representation depicts the key visual clues on the navigation trajectory from a global dimension;

[0180] The fine-grained encoding module 103 is used to encode and fuse fine-grained local features, specifically: for regional features or grid features, they are input into a preset spatial self-attention mechanism based on time constraints. Within a given time period, the attention mechanism assigns a higher weight to a specific region or grid, so that the model focuses on the key visual clues for navigation in the spatial dimension; for regional features or grid features output by the spatial self-attention mechanism based on time constraints, they are input into the temporal self-attention mechanism, so that the model focuses on the key visual clues in the temporal dimension; for regional features and grid features output by the temporal self-attention mechanism, they are input into a preset cross-attention mechanism based on time and local constraints, and the regional features and grid features are fused to obtain a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts the key visual clues on the navigation trajectory from a local dimension, and represents fine-grained object semantics and color texture information;

[0181] The navigation trajectory representation fusion module 104 is used to fuse the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation based on the attention mechanism to obtain a multi-grained navigation trajectory representation; the multi-grained navigation trajectory representation fuses the key visual clues of the global dimension and the local dimension, so that the generated navigation instruction is more comprehensive and accurate;

[0182] The navigation instruction generating module 105 is used to decode the multi-granularity navigation trajectory representation to obtain a synthesized navigation instruction;

[0183] The agent training module 106 is used to train the visual language navigation agent based on the synthesized navigation instructions and the sampled trajectory composition enhanced data, so as to improve the generalization ability of the agent and the navigation success rate.

[0184] It should be noted that the multi-granularity instruction generation system for visual language navigation of the present invention corresponds one-to-one to the multi-granularity instruction generation method for visual language navigation of the present invention. The technical features and beneficial effects described in the embodiment of the multi-granularity instruction generation method for visual language navigation are all applicable to the embodiment of the multi-granularity instruction generation for visual language navigation. For specific contents, please refer to the description in the embodiment of the method of the present invention, and will not be repeated here. This is hereby declared.

[0185] In addition, in the implementation of the multi-granularity instruction generation system for visual language navigation in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the multi-granularity instruction generation system for visual language navigation is divided into different program modules to complete all or part of the functions described above.

[0186] See also Figure 4 In one embodiment, an electronic device for implementing a multi-granularity instruction generation method for visual language navigation is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a multi-granularity instruction generation program 203 for visual language navigation.

[0187] The first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Further, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed in the electronic device 200, such as the code of the multi-granularity instruction generation program 203 for visual language navigation, but also can be used to temporarily store data that has been output or is to be output.

[0188] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions and processes data of the electronic device 200 by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202.

[0189] Figure 4 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on the electronic device 200 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0190] The multi-granularity instruction generation program 203 for visual language navigation stored in the first memory 202 in the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve:

[0191] Acquire the trajectory of the visual language navigation agent and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include regional features and grid features;

[0192] Encoding and fusing the coarse-grained global features, specifically: encoding the action features to capture the contextual dependencies of multiple time stages in the trajectory features, and then fusing the encoded action features and panoramic image features based on the attention mechanism to obtain a coarse-grained navigation trajectory representation, wherein the coarse-grained navigation trajectory representation depicts the key visual clues on the navigation trajectory from a global dimension;

[0193] The fine-grained local features are encoded and fused, specifically: for regional features or grid features, they are input into a preset spatial self-attention mechanism based on time constraints. Within a given time period, the attention mechanism assigns a higher weight to a specific region or grid, so that the model focuses on the key visual clues for navigation in the spatial dimension; for regional features or grid features output by the spatial self-attention mechanism based on time constraints, they are input into the temporal self-attention mechanism, so that the model focuses on the key visual clues in the temporal dimension; for regional features and grid features output by the temporal self-attention mechanism, they are input into a preset cross-attention mechanism based on time and local constraints, and the regional features and grid features are fused to obtain a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts the key visual clues on the navigation trajectory from a local dimension, and represents fine-grained object semantics and color texture information;

[0194] Based on the attention mechanism, the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation are fused to obtain a multi-grained navigation trajectory representation; the multi-grained navigation trajectory representation integrates key visual clues of the global dimension and the local dimension, making the generated navigation instructions more comprehensive and accurate;

[0195] Decoding the multi-granularity navigation trajectory representation to obtain a synthesized navigation instruction;

[0196] Based on the synthesized navigation instructions and the sampled trajectory composition enhanced data, a visual language navigation agent is trained to improve the agent's generalization ability and navigation success rate.

[0197] Furthermore, if the module / unit integrated in the electronic device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0198] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0199] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0200] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A multi-granularity instruction generation method for visual language navigation, characterized in that: The steps include: Acquire the trajectory of the visual language navigation agent and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include regional features and grid features; Encoding and fusing the coarse-grained global features, specifically: encoding the action features to capture the contextual dependencies of multiple time stages in the trajectory features, and then fusing the encoded action features and panoramic image features based on the attention mechanism to obtain a coarse-grained navigation trajectory representation, wherein the coarse-grained navigation trajectory representation depicts the key visual clues on the navigation trajectory from a global dimension; The fine-grained local features are encoded and fused, specifically: for regional features or grid features, they are input into a preset spatial self-attention mechanism based on time constraints. Within a given time period, the attention mechanism assigns a higher weight to a specific region or grid, so that the model focuses on the key visual clues for navigation in the spatial dimension; for regional features or grid features output by the spatial self-attention mechanism based on time constraints, they are input into the temporal self-attention mechanism, so that the model focuses on the key visual clues in the temporal dimension; for regional features and grid features output by the temporal self-attention mechanism, they are input into a preset cross-attention mechanism based on time and local constraints, and the regional features and grid features are fused to obtain a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts the key visual clues on the navigation trajectory from a local dimension, and represents fine-grained object semantics and color texture information; Based on the attention mechanism, the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation are fused to obtain a multi-grained navigation trajectory representation; the multi-grained navigation trajectory representation integrates key visual clues of the global dimension and the local dimension, making the generated navigation instructions more comprehensive and accurate; Decoding the multi-granularity navigation trajectory representation to obtain a synthesized navigation instruction; Based on the synthesized navigation instructions and the sampled trajectory composition enhanced data, a visual language navigation agent is trained to improve the agent's generalization ability and navigation success rate.

2. The multi-granularity instruction generation method for visual language navigation according to claim 1, characterized in that: The step of obtaining the trajectory of the visual language navigation agent and extracting the trajectory features of the agent is as follows: Extracting panoramic image features, where the panoramic image features are composed of ResNet image features and the orientation features of the agent; Extracting action features, in each time stage, the agent selects an image from the panoramic image as the direction to go in the next time stage, and moves to the corresponding path node, so the action feature corresponds to one of the panoramic image features; Extract regional features and grid features. Given an image, use Faster R-CNN to extract multiple regions and multiple grids. Each region or grid is concatenated with the embedding function of its orientation feature to obtain regional features and grid features.

3. The multi-granularity instruction generation method for visual language navigation according to claim 1, characterized in that: The encoding and fusion of the coarse-grained global features are specifically as follows: Use Bi-LSTM to encode action features: In the formula, r1,...,r T They represent the trajectory encodings obtained by encoding the action features from the 1st to the Tth time period, respectively. T represents the total time stage, that is, the number of nodes on the trajectory of the agent. Respectively represent the 1st to Tth action feature codes; Based on the attention mechanism, the trajectory encoding of the action features and the panoramic image features are fused, where the trajectory encoding is used as the query and the panoramic image features are used as the key and value to obtain a coarse-grained navigation trajectory representation, which is expressed as follows: f t g =∑ i a t,i f t,i In the formula, α t,i represents the attention weight of the i-th panoramic image at time t, f t,i represents the features of the i-th panoramic image at time t, T represents the transposition symbol, and W R represents the learnable parameters, r t represents the trajectory code of the tth time period, f t g Represents a coarse-grained representation of navigation trajectories.

4. The multi-granularity instruction generation method for visual language navigation according to claim 1, characterized in that: The encoding and fusion of fine-grained local features are specifically as follows: For regional features, obtain the regional spatial position code; For the grid feature, obtain its grid space position encoding; Design a spatial self-attention mechanism based on time constraints, specifically: MHTCSSA(Q,K,V)=Concat(head1,...,head h )W O head i =TCSSA(QW i Q ,KW i K ,VW i V ,spe q ,spe k ) In the formula, MHTCSSA represents a multi-head spatial self-attention mechanism based on time constraints, Q represents Query, K represents Key, V represents Value, Concat represents concatenation operation, and W O , W Q , W K , W V is a learnable parameter, head i represents the i-th attention head, h represents the total number of attention heads, TCSSA is a single-head spatial self-attention mechanism based on time constraints, spe q is the spatial position encoding of the Query, spe k is the spatial position encoding of the Key, d k is the dimension of the Key vector; The regional features and grid features are input into the multi-head temporal constraint-based spatial self-attention mechanism: In the formula, represents the regional features of the lth layer input at the tth time stage, RSPE is the regional spatial position encoding, represents the grid feature of the lth layer input at the tth time stage. GSPE is the grid spatial position encoding. represents the regional features of the output of the lth layer at the tth time stage, Represents the grid features output by the lth layer at the tth time stage; Regional features output by spatial self-attention mechanism based on time constraints Or mesh features Plus the position encoding PE of the time stage t : Input to the temporal self-attention mechanism: Where MHTSA stands for multi-head temporal self-attention mechanism, and Represents the regional features and grid features of all time stages, Represents the regional features output by the temporal self-attention mechanism of layer l, Represents the grid features output by the temporal self-attention mechanism at layer l; Will and Input into the forward network respectively: Where FFN stands for feed-forward network, Represents the regional features output by the temporal self-attention mechanism of layer l, Represents the grid features output by the temporal self-attention mechanism at layer l; Design a criss-cross attention mechanism based on temporal and local constraints: MHTLCCA(Q,K,V)=Concat(head1,...,head h )W O Where MHTLCCA represents a multi-headed cross-attention mechanism based on time and local constraints, and TLCCA represents a single-headed cross-attention mechanism based on time and local constraints. A geometric alignment graph representing regions and grids, in which each region and grid corresponds to an independent node, which together constitute a node set To ensure semantic consistency between regions and grids in the criss-cross attention mechanism, the set The edges in will be established between the region nodes and the grid nodes. Represents the graph-based softmax function. The attention weight assigned to non-adjacent nodes in The softmax function is applied to adjacent nodes in ; to ensure temporal consistency, a dedicated geometric alignment graph is created for each image at time stage t The regional features and grid features output by the temporal self-attention mechanism are input into the designed cross-attention mechanism based on temporal and local constraints to fuse the regional features and grid features: exist In the formula, the regional feature is used as the query, the grid feature is used as the key and value, and the grid feature is embedded into the regional feature to obtain the regional feature output by the cross-attention mechanism based on time and local constraints at the lth layer. exist In the formula, the grid feature is used as the query, the regional feature is used as the key and value, and the regional feature is embedded into the grid feature to obtain the grid feature output by the cross-attention mechanism based on time and local constraints at the lth layer. Then will and Input into the forward network FFN respectively: Due to the multi-layer attention mechanism, and Input to the next layer’s spatial self-attention mechanism based on time constraints; The regional features H output by the last layer r and the mesh feature H g Splicing to obtain a fine-grained navigation trajectory representation u: u=[H r ;H g ] In the formula, T is the total number of time stages of the agent’s trajectory, N r is the number of extracted regional features, N g is the number of mesh features extracted, d model The dimension of the vector representing the hidden layer.

5. The multi-granularity instruction generation method for visual language navigation according to claim 4 is characterized in that: The specific regional spatial position encoding for obtaining regional features is: RSPE(i)=Emb(B i ); In the formula, i represents the i-th region in the image, B i =(x min ,y min ,x max ,y max ) represents the coordinates of the upper left corner and the lower right corner of the bounding box of the region, Emb(·) represents the projection layer, and RSPE represents the spatial position encoding of the region.

6. The multi-granularity instruction generation method for visual language navigation according to claim 4, characterized in that: The grid spatial position encoding for obtaining grid features is specifically: GSPE(i,j)=[PE i ;ON j ] In the formula, the grid divides the image into i×j grids, i represents the row index of the grid, j represents the column index of the grid, PE i , PE j They represent the position encoding of i and j respectively, and GSPE represents grid space position encoding.

7. The multi-granularity instruction generation method for visual language navigation according to claim 1, characterized in that: Based on the attention mechanism, the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation are fused to obtain a multi-grained navigation trajectory representation, specifically: Based on the attention mechanism, the coarse-grained navigation trajectory is represented as t g As Query, the fine-grained navigation trajectory representation u is used as Key and Value, and the coarse-grained and fine-grained navigation trajectory representations are fused to obtain a multi-grained navigation trajectory representation: f t gl =∑ i b t,i you t,i Where W F represents the learnable parameters, T represents the transposed symbol, and u t,i represents the i-th region / grid feature at time t, β t,i represents the attention weight of the i-th region / grid feature at time t, f t gl Represents multi-granularity representation of navigation trajectories; Bi-LSTM is used to encode the multi-granularity navigation trajectory representation to further capture its contextual dependency on time:

8. A multi-granularity instruction generation system for visual language navigation, characterized in that: A multi-granularity instruction generation method for visual language navigation applied to any one of claims 1-7, comprising a trajectory feature extraction module, a coarse-grained encoding module, a fine-grained encoding module, a navigation trajectory representation fusion module, a navigation instruction generation module, and an agent training module; The trajectory feature extraction module is used to obtain the trajectory of the visual language navigation agent and extract the trajectory features of the agent; the trajectory features include coarse-grained global features and fine-grained local features, the global features include panoramic image features and action features, and the local features include regional features and grid features; The coarse-grained encoding module is used to encode and fuse the coarse-grained global features, specifically: encode the action features to capture the contextual dependencies of multiple time stages in the trajectory features, and then fuse the encoded action features and panoramic image features based on the attention mechanism to obtain a coarse-grained navigation trajectory representation, which describes the key visual clues on the navigation trajectory from a global dimension; The fine-grained encoding module is used to encode and fuse fine-grained local features, specifically: for regional features or grid features, they are input into a preset spatial self-attention mechanism based on time constraints. Within a given time period, the attention mechanism assigns a higher weight to a specific region or grid, so that the model focuses on the key visual clues for navigation in the spatial dimension; for regional features or grid features output by the spatial self-attention mechanism based on time constraints, they are input into the temporal self-attention mechanism, so that the model focuses on the key visual clues in the temporal dimension; for regional features and grid features output by the temporal self-attention mechanism, they are input into a preset cross-attention mechanism based on time and local constraints, and the regional features and grid features are fused to obtain a fine-grained navigation trajectory representation; the fine-grained navigation trajectory representation depicts the key visual clues on the navigation trajectory from a local dimension, and represents fine-grained object semantics and color texture information; The navigation trajectory representation fusion module, based on the attention mechanism, fuses the coarse-grained navigation trajectory representation and the fine-grained navigation trajectory representation to obtain a multi-grained navigation trajectory representation; the multi-grained navigation trajectory representation fuses the key visual clues of the global dimension and the local dimension, so that the generated navigation instructions are more comprehensive and accurate; The navigation instruction generation module is used to decode the multi-granularity navigation trajectory representation to obtain a synthesized navigation instruction; The agent training module is used to train a visual language navigation agent based on the synthesized navigation instructions and the sampled trajectory composition enhanced data, thereby improving the generalization ability of the agent and the navigation success rate.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory in communication with the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the multi-granularity instruction generation method for visual language navigation as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the multi-granularity instruction generation method for visual language navigation described in any one of claims 1 to 7 is implemented.

Images