A visual language navigation method and device based on scene fusion knowledge and a medium

Abstract

The application discloses a visual language navigation method and device based on scene fusion knowledge and a medium, wherein the method comprises the following steps: acquiring a visual language navigation task; acquiring natural language instruction features, scene features and object features according to the visual language navigation task; iteratively updating the weight of the object features by using a graph convolution network based on semantic and position perception; and searching for knowledge-enhanced object features by using object labels in the scene; and using a multi-modal decision module based on scene and knowledge perception to fuse the natural language instruction features, the scene features and the object features, to perform action prediction and update the running state of an agent until the agent chooses to stop. By utilizing the semantic and position relationship between objects and knowledge in the scene, the application enables better alignment between the scene features and the natural language instruction features, and enables the agent to effectively navigate in a limited visual observation and an unobserved environment. The application can be widely applied to the technical field of visual language navigation.

Description

Technical Field

[0001] This invention relates to the field of visual language navigation technology, and in particular to a visual language navigation method, device and medium based on scene fusion knowledge. Background Technology

[0002] With the development and maturation of artificial intelligence technology in recent years, computer vision, natural language processing, and robotics have been widely applied in various fields. Endowing robots with human-like intelligence, enabling intelligent agents to "understand" human language, "read" visual information, and act autonomously to serve humanity, has been a long-term goal. The field of visual language navigation researches such methods, allowing intelligent agents to continuously explore the visual environment under the guidance of natural language instructions, ultimately completing designated tasks.

[0003] Existing visual-language navigation methods are mostly based on the fusion and alignment of visual features and natural language command features, enabling agents to navigate along paths guided by instructions. However, in real-world scenarios, object-finding navigation tasks are more practically valuable. Natural language commands in these tasks often only contain descriptive information about the target object and do not provide detailed path descriptions. Under existing models, agents are limited by the limited command content and insufficient perception of the overall scene layout, making it difficult to effectively explore the environment and find the target object. Summary of the Invention

[0004] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a visual language navigation method, device and medium based on scene fusion knowledge.

[0005] The technical solution adopted in this invention is:

[0006] A visual language navigation method based on scene fusion knowledge includes the following steps:

[0007] A visual language navigation task is acquired, which includes natural language instructions, as well as initial visual information and location information;

[0008] Natural language instructions are encoded into natural language instruction features and the initial operating state of the agent; visual information and location information are encoded and concatenated to obtain scene features;

[0009] Extract object labels from visual information, encode the semantic labels and location information of objects into object features, and update the node representations in the graph convolutional network.

[0010] The weights of object features are iteratively updated using a semantically and position-aware graph convolutional network, and knowledge-enhanced object features are retrieved using object labels in the scene.

[0011] Using a scene- and knowledge-aware multimodal decision-making module, the natural language instruction features, scene features, and knowledge-enhanced object features are fused to predict actions and update the agent's operating state until the agent chooses to stop.

[0012] Furthermore, the natural language instructions are encoded into natural language instruction features and the initial operating state of the agent; visual information and location information are encoded and concatenated to obtain scene features, including:

[0013] After the agent acquires the visual language navigation task, it receives natural language instructions. L represents the length of the instruction; where the agent is placed at the starting position;

[0014] During the initialization phase, [CLS], instruction sequence I, and segmentation identifier [SEP] are combined into a sequence input to the Transformer for encoding, resulting in the agent's initial running state s0 and the characteristics of the natural language instructions:

[0015] s0,X=Transformer([CLS],I,[SEP])

[0016] Navigation is a continuous iterative process. At each time step t, the agent acquires visual and positional information about the scene. The visual information includes a panoramic image of the agent's current location, divided into 36 discrete views. A CLIP-ViT-B-32 is used as a visual encoder to obtain the visual features of the panoramic image. The view's position information includes the view's rotation angle θ relative to its current position. i and elevation angle φ i By copying the vector (cosθ) 32 times to the position angle i sinθ i ,cosφ i ,sinφ i Construct a 128-dimensional directional code d i ;

[0017] For the current scenario, there exists N t A navigable direction, denoted as Obtain the corresponding location information features By combining the two, the corresponding scene features are obtained.

[0018] For each candidate direction i, the Faster-RCNN object extractor is used to extract the top m most salient objects in the scene, and the objects are labeled as follows:

[0019] Furthermore, the semantically and positionally aware graph convolutional network operates through the following mechanism:

[0020] Constructing objects and knowledge graphs: Objects and knowledge are combined into a graph structure. The relationship between objects and knowledge is the corresponding relationship in the knowledge base, while an implicit relationship is defined between objects.

[0021] Embedded edge relation representation: Using a set of learnable parameters as basis vectors, the edge relation embedding is obtained by ordered weight accumulation and normalization; different relation representations are obtained by embedding the relationship between different objects and knowledge, and the implicit relationship between objects is also a special kind of relation embedding;

[0022] Embedded node representation: Semantically encode all nodes into vectors, encode the positions of visited objects, and initialize the position encodings of unvisited objects and all knowledge entity nodes as all-zero vectors.

[0023] By combining graph convolutional networks and edge information representation, the features of nodes in the graph are updated to obtain the final graph node representation.

[0024] Input the object label type corresponding to the current scene, and output the object features after knowledge enhancement through graph convolution and representation update.

[0025] Furthermore, the feature update of nodes in the graph by combining graph convolutional networks and edge information representation to obtain the final graph node representation includes:

[0026] A1. Sum the neighboring nodes and add edge embedding representation to update the feature representation of the target node;

[0027] A2. In order to better represent the target node, the characteristics of the target node itself are also added during the final update. Then, the output result is passed through a non-linear activation function to obtain the updated representation of the node.

[0028] A3. Using a multi-layered graph convolutional model structure, iterate through steps A1-A2 to obtain the final graph node representation.

[0029] Furthermore, the construction of the objects and knowledge graph includes:

[0030] Using a pre-defined network model to detect and obtain a list of object labels as an index, the system retrieves the k knowledge items with the highest relevance weights from a pre-defined knowledge base.

[0031] Furthermore, the operating mechanism of the scene- and knowledge-aware multimodal decision-making module is as follows:

[0032] Using selective attention mechanisms to fuse multimodal representations;

[0033] Update the agent's running state: At each moment, concatenate the weighted sum of the state variables of the last layer of the multimodal decision module with respect to the natural language instruction features and the corresponding attention scores, as well as the weighted sum of the visual features and the attention scores, and obtain new state features through linear transformation;

[0034] Dynamic aggregation of scenes and knowledge: Based on state features, the attention scores of scene features and knowledge features in the same view are sorted, and the maximum attention score is selected as the score of the view;

[0035] Output the agent's decision: Perform a softmax operation on the final scores of all different views, and select the view corresponding to the maximum score as the direction the agent moves in; if the maximum score corresponds to the current view, the agent chooses to stop.

[0036] Furthermore, the use of selective attention mechanism to fuse multimodal representations includes:

[0037] The input consists of natural language instruction features, scene features, and knowledge-enhanced object features. Natural language instruction features and scene features are used only as keys and values ​​for the attention mechanism and are not updated, while knowledge-enhanced object features are updated with reference to natural language instruction features and scene features.

[0038] Furthermore, the graph convolutional network and the multimodal decision module are trained in the following manner:

[0039] The agent is trained using imitation learning (IL) and reinforcement learning (RL); in imitation learning, the agent takes the labeled action at each time point. To effectively follow the shortest path trajectory, such that the action probability p t Try to follow the trajectory of the shortest path as closely as possible;

[0040] Let T be the total length of the agent's trajectory, then the expression for the loss function of imitation learning is:

[0041]

[0042] In reinforcement learning, the agent learns from the action probability p t Action samples are selected through sampling, and learning is done from the rewards; the expression for the loss function of reinforcement learning is:

[0043]

[0044] The overall loss function is:

[0045] L = L IL +λL RL

[0046] Where λ represents the hyperparameter.

[0047] Another technical solution adopted in this invention is:

[0048] A visual language navigation device based on scene fusion knowledge, comprising:

[0049] At least one processor;

[0050] At least one memory for storing at least one program;

[0051] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0052] Another technical solution adopted in this invention is:

[0053] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.

[0054] The beneficial effects of this invention are: by utilizing the semantic and positional relationships of objects and knowledge in a scene, this invention enables better alignment between scene features and natural language instruction features, and allows intelligent agents to navigate effectively in limited visual observation and unseen environments. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a framework diagram of a visual language navigation method based on scene fusion knowledge in an embodiment of the present invention. Detailed Implementation

[0057] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0058] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0059] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0060] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0061] like Figure 1 As shown, this embodiment provides a visual language navigation method based on scene fusion knowledge. This method employs a semantically and positionally aware graph convolutional network module (OK-GCN) to infer the relationship between objects and knowledge, and fuses multimodal information through a scene- and knowledge-aware multimodal decision module (SK-Transformer) for decision-making. The aim is to improve the agent's ability to perceive the layout of the environment and its exploration ability in unseen environments. The specific steps of this method are as follows:

[0062] S1. Obtain visual language navigation task. The agent is placed at the starting position and obtains natural language instructions. Then, at each time t, it will acquire visual information and position information of the scene. The visual information includes panoramic image information of the agent's current position, which is divided into 36 discrete views. The position information includes the turning angle and elevation angle of the view relative to the current position.

[0063] S2. Construct a global object and knowledge relationship reasoning graph, OK-GCN. Using the most salient object labels detected by a pre-trained Faster-RCNN, retrieve the k knowledge items with the highest confidence from the knowledge graph. Treat all objects and knowledge entities as nodes, connecting each other with edges, and connecting related objects and knowledge entities with edges, thus constructing a global object and knowledge relationship graph. The nodes in the relationship graph use GloVe encoding for semantics, and positional encoding is initialized as zero-vector features; the two are concatenated to form the initial features of the nodes.

[0064] S3. Use Transformer to encode the input natural language instructions to obtain language feature vectors and initial state feature vectors.

[0065] S4. For the scene corresponding to the navigable direction at the current moment, CLIP-ViT-B-32 is used as the visual encoder to obtain the visual features of the scene. The visual features and the direction features obtained from the position encoding are concatenated to obtain the scene features. The pre-trained Faster-RCNN is used to detect the object labels observed in the current scene.

[0066] S5. For the detected object labels in the current scene, update their position features, and then use multi-step graph convolution to infer the object knowledge graph OK-GCN to obtain the updated object feature matrix. Use the object features of the current node to retrieve the corresponding knowledge-enhanced object features.

[0067] S6. The SK-Transformer (Scene and Knowledge AwareTransformer) module performs cross-modal encoding on the obtained language features, scene features, knowledge-enhanced object features, and the agent's state vector from the previous time step. It then obtains the current time step's state features and the aggregated attention score for the corresponding scene.

[0068] S7. Input the aggregated attention scores for the corresponding scenes into the Softmax classifier to obtain the probability distribution of the agent's actions. The agent selects the scene direction corresponding to the maximum score to move in. If the scene corresponding to the maximum score is the current scene, the agent chooses to stop.

[0069] S8. Repeat the above steps until the agent chooses to stop, and finally update the model weights, namely the semantic and location-aware relationship reasoning module OK-GCN and the scene and knowledge-aware fusion module SK-Transformer.

[0070] The above method will be explained in detail below with reference to specific embodiments.

[0071] like Figure 1 As shown, this invention provides a visual language navigation method that combines knowledge in dynamic topological map exploration. The specific steps are as follows:

[0072] (1) Extract multimodal features or labels: including natural language command features, scene features, and object labels corresponding to the scene.

[0073] After receiving the visual-language navigation task, the agent is placed at the starting position and receives natural language instructions. L represents the length of the instruction.

[0074] First, in order to obtain the features of natural language instructions, during the initialization phase, we input the sequence [CLS], instruction sequence I, and segmentation identifier [SEP] into the Transformer for encoding, thereby obtaining the initial running state s0 of the agent and the features of the natural language instructions.

[0075] s0,X=Transformer([CLS],I,[SEP])

[0076] The navigation process is then iterative. At each time step t, the agent acquires visual and positional information about the scene. The visual information includes a panoramic image of the agent's current position, divided into 36 discrete views. A CLIP-ViT-B-32 is used as a visual encoder to obtain the visual features of the panoramic image. The view's position information includes the view's rotation angle θ relative to its current position. i and elevation angle φ i This invention replicates the position angle 32 times (cosθ). i sinθ i ,cosφ i ,sinφ i Construct a 128-dimensional directional code d i .

[0077] Then, for the current scenario, there exists N. t A navigable direction, denoted as Similarly, we can also obtain the corresponding location information features. By combining the two, the corresponding scene features are obtained. For each candidate direction i, we use the Faster-RCNN object extractor to extract the top m most salient objects in the scene, and we label the objects as follows:

[0078] In summary, we obtain the initial operating state s0 of the agent, the features X of the natural language instructions, and the features of the adjacent scenes. and the corresponding object labels in adjacent scenes

[0079] (2) Construct a global object and knowledge relationship graph OK-GCN, use a location- and semantic-aware graph neural network to update node and neighbor features, and then retrieve the knowledge-enhanced object representations through the object label information of the scene.

[0080] We choose knowledge graphs, which are semantic representations of the real world. Their basic building blocks are entity-relation-entity triples. Common knowledge fusion methods involve acquiring knowledge from structured knowledge bases, such as ConceptNet, or from semi-structured knowledge bases, such as Visual Genome. Here, we choose ConceptNet as the source of external knowledge.

[0081] Construct an object knowledge graph and use the most salient object labels detected by Faster-RCNN pre-trained on Visual Genome. The most relevant top k pieces of knowledge are retrieved from ConceptNet using this as an index. Knowledge can be represented as (h... i r i,j , t j w i,j ). Where h i It refers to the objects detected in the scene, t j The object is retrieved from the knowledge base, w i,j The weight r represents the knowledge relevance. i,j It represents the relationship between objects and knowledge.

[0082] A graph representing the relationship between objects and knowledge can be represented as G. K =(H K E K ), where H K It is a node set, which includes all entities, namely the object labels in the scene and the knowledge entity labels retrieved from the knowledge base. E K Let N be the set of edges. Let the total number of nodes in the graph be N.

[0083] Nodes are encoded based on semantics and location. We use GloVe to encode all entities, obtaining a 300-dimensional semantic representation, and a 128-dimensional zero vector as the initial location representation for objects and knowledge. These two are concatenated to form the node representation of the graph, so h... i ,t j ∈R 428 Therefore, we obtain H. K ∈R N×428

[0084] Inference is performed using a graph neural network with edge embeddings, A K ∈R N×N This represents the adjacency matrix of the relationship graph, where each element represents the relationship between nodes. For relationships between objects and knowledge, where... The relation r is defined in the knowledge base i,j The relationships between objects are set up as a special implicit relationship, numbered sequentially from 1 to r.

[0085] First, the edges are embedded and represented using ordered accumulation and LayerNorm encoding, with a set of learnable parameters selected as the basis. Encode each relation as e r :

[0086]

[0087] Then, OK-GCN is updated using multi-layer graph convolution, including neighbor nodes. The features and edge features are aggregated together on the target node i to obtain the representation of the target node in the next layer.

[0088]

[0089] Where σ represents the activation function. Let W represent the set of neighboring nodes of node i. G This represents the learnable parameters in the model. This represents the embedding of the relational representation of the corresponding layer.

[0090] Finally, the features from different layers are concatenated to form the final features of the target node.

[0091]

[0092] Among them W k These are learnable parameters, where L is the number of layers in the model, and o i It is the feature output of the target object after knowledge enhancement.

[0093] (3) Multimodal feature reasoning and decision-making based on the SK-Transformer multimodal decision model with scene and knowledge awareness.

[0094] Now we have obtained the agent's initial operating state s0, the features X of the natural language instructions, and the features of the adjacent scenes. and knowledge-enhanced object features in adjacent scene i It can also be viewed as knowledge information about the scene.

[0095] Drawing on the successful experience of RecBERT (A Recurrent Vision-and-Language BERT for Navigation) in visual language navigation, we propose a novel scene- and knowledge-aware multimodal decision model, SK-Transformer, which incorporates knowledge fusion.

[0096] Selective attention mechanism:

[0097] During the fusion of multimodal features in SK-Transformer, a specific attention mask is set. This ensures that natural language instruction features and scene features are only input to SK-Transformer as keys and values. The instruction features and scene features remain unchanged during the update process; they are only provided as contextual information in the attention mechanism.

[0098] In particular, state feature s t and knowledge-enhanced object features Then, as a query, the key and value are input into SK-Transformer, which is the standard attention pattern in Transformer.

[0099] Intuitively, under this attention mask, the instructions and scene observed by the object do not change; they only provide contextual information. The object can select the parts of the scene features that need attention based on the features of the object and knowledge.

[0100] Dynamic aggregation of scenarios and knowledge:

[0101] For the aggregation of scene and knowledge, we select the object feature with the highest attention score among all scene and knowledge-enhanced features in a given view, and use the corresponding feature to represent that view. Intuitively, this approach allows the model to select a relevant object or knowledge or the entire scene to represent each navigable direction.

[0102] Agent action decision-making:

[0103] Let the function ψ(x) represent the output corresponding to the input variable x in the last layer of the SK-Transformer. ψ(s t ) represents the state feature s t The output of the corresponding last layer in the SK-Transformer, ψ(s) t )∈R d d represents the size of the hidden layers in the SK-Transformer model. Represents scene features, representing N t The directions of each adjacent scene and the direction corresponding to one stop marker. ψ(V) t ) represents scene feature V t In the SK-Transformer, the corresponding output after the last aggregation layer is used to calculate the state feature pair ψ(V). t Attention score α(V) t The attention score is obtained by standardizing it using a softmax layer. Represented as N t A navigable direction and a probability distribution of stopping.

[0104]

[0105]

[0106] Standardized values ​​of attention scores for scene features based on the agent's state features. Predict the action and select the direction corresponding to the value with the highest probability as the agent's decision direction.

[0107] Agent operational status update:

[0108] In SK-Transformer, the state history is derived from state features s t To maintain, of which s t ∈R d Specifically, s t+1 Through ψ(s) t The update is performed using the representation of the scene, the instruction, and the action, incorporating these features into the state history. We first calculate the weighted sum of the scene features and the attention score, and then the weighted sum of the instruction features and the attention score. The specific calculation process is shown in the following formula.

[0109]

[0110]

[0111]

[0112] Then the original state feature s t , and F v With F l The elements are multiplied element by element and then concatenated. After a linear transformation, this is combined with the position representation of the direction chosen by the agent. The data is spliced ​​together and then subjected to a linear transformation to obtain a new state s. t+1

[0113]

[0114] Where W1 and W2 are learnable parameters, ⊙ denotes element-wise multiplication, and [;] denotes concatenation. Positional features for selecting directions.

[0115] (4) Train OK-GCN and SK-Transformer models

[0116] During training, we used imitation learning (IL) and reinforcement learning (RL) to train the agent. In imitation learning, the agent takes the labeled action at each time point. To effectively follow the shortest path trajectory, such that the action probability pt The trajectory should be as close as possible to the shortest path. Let T be the total length of the agent's trajectory, then the expression for the loss function of imitation learning is:

[0117]

[0118] In reinforcement learning, the agent learns from the action probability p t By sampling action samples and learning from rewards, the agent can learn how to explore the environment and improve its generalization ability, drawing on the advantage variable A proposed in the A2C algorithm. t The expression for the loss function of reinforcement learning in this paper is as follows:

[0119]

[0120] L = L IL +λL RL

[0121] Here, λ represents a hyperparameter used to adjust the importance of the imitation learning and reinforcement learning loss functions, respectively. During training, AdamW is used to optimize the objective loss function and update the model's parameters.

[0122] (5) Visual language navigation using OK-GCN and SK-Transformer

[0123] When using this invention for navigation, OK-GCN and SK-Transformer are set to test mode, and gradient accumulation is not performed, thus speeding up computation. The model will also no longer use Dropout layers, and the global mean and variance of the batch normalization layers will no longer be updated.

[0124] When making navigation decisions, the agent will choose the direction with the highest score at each time t. The maximum number of steps the agent can take is set to T. max Repeat this navigation process until the number of moves exceeds T. max Alternatively, the agent may choose to remain at its current position.

[0125] As shown in Tables 1 and 2 below, Table 1 shows the experimental results of SK-Transformer on the R2R dataset, and Table 2 shows the experimental results of SK-Transformer on the REVERIE dataset. As can be seen from Tables 1 and 2, the SK-Transformer in this embodiment can achieve better results.

[0126] Table 1. Experimental results of the present invention (SK-Transformer) on the R2R dataset.

[0127]

[0128] Table 2. Experimental results of the present invention (SK-Transformer) on the REVERIE dataset.

[0129]

[0130] In summary, compared with the prior art, the present invention has at least the following advantages and beneficial effects:

[0131] (1) The present invention uses a semantic and location-aware relational reasoning module, which can effectively model the relationship between objects in the scene and knowledge in the knowledge graph, and output object representations with semantic and location information enhancement.

[0132] (2) Considering the relationship between scene, object and knowledge, this invention proposes a fusion module SK-Transformer based on scene and knowledge perception, which uses selective attention mechanism to update knowledge, then aggregates scenes and knowledge from different perspectives, scores the aggregation results and selects the final decision direction.

[0133] (3) The present invention can make inferences based on the relationship between objects in the scene and common sense, and has a stronger exploration ability under limited visual observation conditions or in unseen environments.

[0134] This embodiment also provides a visual language navigation device based on scene fusion knowledge, including:

[0135] At least one processor;

[0136] At least one memory for storing at least one program;

[0137] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.

[0138] This embodiment of a visual language navigation device based on scene fusion knowledge can execute a visual language navigation method based on scene fusion knowledge provided in the method embodiment of the present invention. It can execute any combination of implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0139] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0140] This embodiment also provides a storage medium storing instructions or programs that can execute the visual language navigation method based on scene fusion knowledge provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0141] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0142] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0143] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0145] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0146] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0147] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0148] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0149] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A visual language navigation method based on scene fusion knowledge, characterized in that, The method comprises the following steps: obtaining a visual language navigation task, the visual language navigation task comprising natural language instructions, and initial visual information and position information; encoding the natural language instructions into natural language instruction features and an initial running state of an agent, and encoding and splicing the visual information and the position information to obtain scene features; extracting object labels from the visual information, and encoding the semantic labels and position information of the objects into object features to update node representations in a graph convolution network; iteratively updating weights of the object features using a graph convolution network based on semantic and position perception, and retrieving knowledge-enhanced object features using object labels in the scene; using a multi-modal decision module based on scene and knowledge perception to fuse the natural language instruction features, the scene features and the knowledge-enhanced object features, to perform action prediction and update the running state of the agent until the agent chooses to stop; the encoding of the natural language instructions into natural language instruction features and the initial running state of the agent, and the encoding and splicing of the visual information and the position information to obtain scene features, comprises: After the agent obtains a visual language navigation task, a natural language instruction is obtained , represents the length of the instruction; wherein the agent is placed at a starting position; In the initialization phase, the , instruction sequence and split identifier , the sequence input Transformer is composed of the initial running state of the agent and the characteristics of the natural language instruction: Navigation is a continuous iterative process, at each moment , the agent will obtain the visual information and position information of the scene, the visual information includes the panoramic image information of the current position of the agent, which is divided into 36 discrete views, and the visual features of the panoramic image are obtained using CLIP-ViT-B-32 as the visual encoder ; the position information of the view includes the turning angle and the elevation angle of the view relative to the current position; by copying the position angle 32 times vector , a 128-dimensional direction encoding is constructed ; For the current scene, there are navigable directions, denoted as , get the corresponding position information feature , and splice them to get the corresponding scene feature ; For each candidate direction , the top most salient objects in the scene are extracted using a Faster-RCNN object extractor, and the labels of the objects are marked as .

2. The visual language navigation method based on scene fusion knowledge according to claim 1, characterized in that, the running mechanism of the graph convolution network based on semantic and position perception comprises: constructing an object and knowledge graph: the objects and the knowledge are structured into a graph, the relationship between the objects and the knowledge is the corresponding relationship in the knowledge base, and an implicit relationship is defined between the objects and the objects; embedding edge relationship representations: obtaining an embedding of the edge relationship; embedding different relationship representations between different objects and knowledge, and embedding the implicit relationship between the objects as a special relationship; embedding node representations: semantically encoding all nodes into vectors, positionally encoding the accessed objects, and initializing the position encoding of the unaccessed objects and all knowledge entity nodes into all-zero vectors; updating the features of the nodes in the graph by combining the graph convolution network and the edge information representation to obtain final graph node representations; inputting object label types corresponding to a current scene, and outputting object features enhanced by knowledge after graph convolution update. 3.The visual language navigation method based on scene fusion knowledge according to claim 2, characterized in that, the updating of the features of the nodes in the graph by combining the graph convolution network and the edge information representation to obtain final graph node representations comprises: A1, summing neighbor nodes and adding edge embedding representations to update the feature representation of a target node; A2, in order to better represent the target node, the feature of the target node itself is also added in the final update, and then the output result is subjected to a nonlinear activation function to obtain an updated representation of the node; A3, using a multi-layer graph convolution model structure to iteratively perform steps A1-A2 to obtain final graph node representations.

4. The visual language navigation method based on scene fusion knowledge according to claim 2, characterized in that, the construction of the object and knowledge graph comprises: The object label list is obtained by using a preset network model for detection, and the knowledge base with the highest relevance weight is searched from the preset knowledge base as an index. A knowledge base with the highest relevance weight is searched from a preset knowledge base as an index.

5. The method of claim 1, wherein the scene fusion knowledge is based on a visual language navigation. the running mechanism of the multi-modal decision module based on scene and knowledge perception comprises: using a selective attention mechanism to fuse multi-modal representations; updating the running state of the agent: at each time, the weighted sum value of the state variable of the last layer of the multi-modal decision module with respect to the natural language instruction features and the corresponding attention scores, and the weighted sum value of the visual features and the attention scores are spliced, and a new state feature is obtained through linear transformation; Dynamic aggregation of scene and knowledge: according to the state feature, the attention scores of the scene feature and the knowledge enhanced object feature in the same view are sorted, and the maximum attention score is selected as the score of the view; Output the decision of the agent: the last scores of all different views are subjected to Softmax, and the view corresponding to the maximum score is selected as the direction of the agent movement; if the maximum score corresponds to the current view, the agent chooses to stop.

6. The visual language navigation method based on scene fusion knowledge according to claim 5, characterized in that, The multi-modal representation is fused using the selective attention mechanism, including: The natural language instruction feature, the scene feature and the knowledge enhanced object feature are input, wherein the natural language instruction feature and the scene feature are only used as the key and value of the attention mechanism and are not updated, and the knowledge enhanced object feature is updated by referring to the natural language instruction feature and the scene feature.

7. The method of claim 1, wherein the scene fusion knowledge is based on a visual language. The graph convolution network and the multi-modal decision module are trained in the following manner: The agent is trained in a way that imitates learning IL and reinforcement learning RL; in the imitation learning, the agent takes the labeled action at each time point to effectively follow the trajectory of the shortest path, so that the action probability as close as possible to the trajectory of the shortest path; Let T be the total length of the agent trajectory, then the expression of the loss function of the imitation learning is: In reinforcement learning, an agent samples action samples from action probabilities and learns from rewards; The expression of the loss function of the reinforcement learning is: The total loss function is: wherein represents a hyperparameter.

8. A visual language navigation device based on scene fusion knowledge, characterized in that, It includes: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-7.

9. A computer readable storage medium having stored therein a program that is executable by a processor, characterized in that, The program executable by the processor is used to execute the method of any one of claims 1-7 when executed by the processor.

Images