Accelerating method, device and equipment of visual language navigation system and storage medium

Abstract

The application relates to the technical field of visual language navigation, in particular to an acceleration method and device of a visual language navigation system, equipment and a storage medium. The method comprises the following steps: acquiring image information of a target scene, generating a geometric map and a semantic map of the target scene according to the image information; generating a navigation map based on a preset navigation target and in combination with the geometric map and the semantic map; pre-encoding all direction nodes in a navigation path to obtain a plurality of pre-encoded nodes; calculating node scores of all the pre-encoded nodes based on all the position nodes and all the pre-encoded nodes; determining a plurality of high-priority nodes according to the node scores of all the pre-encoded nodes to prune the navigation path; and inputting the pruned navigation path into a preset language model to generate navigation information. The application can reduce the delay of the visual language navigation system and achieve the beneficial effect of optimizing the navigation speed of the visual language navigation system.

Description

Technical Field

[0001] The present application relates to the technical field of visual language navigation, and in particular to an acceleration method, apparatus, device and storage medium for a visual language navigation system. Background Art

[0002] In vision-language navigation (VLN) tasks, an agent needs to navigate according to natural language instructions to achieve smooth movement from a starting point to a target. Learning-based methods have been the primary solution for this task. However, these methods often suffer from high training costs and lack of versatility, limiting their widespread adoption in practical applications. In recent years, the development of large language models (LLMs) has made them an effective tool for VLN tasks with zero-shot learning due to their powerful generalization capabilities. By combining visual information with language understanding, LLMs can adapt to different navigation tasks in a variety of environments, improving not only accuracy but also system robustness. Furthermore, LLMs leverage the advantages of multi-round dialogue and contextual understanding, enabling them to adjust navigation strategies based on real-time feedback, enhancing navigation flexibility.

[0003] However, the current zero-shot learning visual language navigation method relies heavily on the reasoning ability of LLM, and the large latency of LLM has become a bottleneck of this method. Based on this, how to reduce the latency of the visual language navigation system and optimize the navigation speed of the visual language navigation system is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] In order to overcome the deficiencies of the prior art, the present application provides an acceleration method, apparatus, device and storage medium for a visual language navigation system to reduce the delay of the visual language navigation system, thereby achieving the beneficial effect of optimizing the navigation speed of the visual language navigation system.

[0005] The technical solution adopted by this application to solve its technical problems is:

[0006] In a first aspect, the present application provides an acceleration method for a visual language navigation system, applied to an agent, the method comprising:

[0007] Acquire image information of a target scene, and generate a geometric map and a semantic map of the target scene based on the image information; the semantic map includes a plurality of position nodes and a plurality of direction nodes; one position node corresponds to a plurality of direction nodes;

[0008] Based on a preset navigation target, generating a navigation map in combination with the geometric map and the semantic map, wherein the navigation map includes a navigation path, and the navigation path includes a plurality of target location nodes and direction nodes of all the target location nodes;

[0009] Precoding all direction nodes in the navigation path to obtain multiple precoded nodes;

[0010] Obtaining the current location node of the agent, and calculating the node scores of all the pre-coding nodes based on all the target location nodes and all the pre-coding nodes; all the target location nodes include the current location node;

[0011] determining a plurality of high-priority nodes according to the node scores of all the pre-coded nodes to prune the navigation path;

[0012] The pruned navigation path is input into a preset language model, and navigation information is generated by the preset language model.

[0013] Optionally, the step of acquiring image information of the target scene and generating a geometric map and a semantic map of the target scene according to the image information includes:

[0014] Acquire the image information and store the image information in a multi-level cache; the image information includes color image information and depth image information; the multi-level cache contains historical node information of all direction nodes in historical navigation;

[0015] The color image information and the depth image information are summarized and sorted according to the historical node information to obtain the semantic map.

[0016] Optionally, before the step of generating a navigation map based on a preset navigation target by combining the geometric map and the semantic map, the method includes:

[0017] Determine all adjacent position nodes according to the geometric map; a group of adjacent position nodes includes two adjacent position nodes;

[0018] Calculating semantic similarities between all pairs of semantic labels based on semantic labels of all direction nodes included in the adjacent position nodes in the same group;

[0019] Determining whether there is at least one group of similar semantic nodes based on the semantic similarity; the similar semantic nodes include directional nodes whose semantic similarity of at least two semantic tags exceeds a preset similarity threshold;

[0020] If there is at least one group of similar semantic nodes, all directional nodes in the same group of similar semantic nodes are merged into one directional node, and the semantic map is updated according to all the merged directional nodes.

[0021] Optionally, the step of calculating the node scores of all the precoding nodes based on all the target location nodes and all the precoding nodes includes:

[0022] Obtaining distance indicators and graph structure indicators of all the position nodes, and semantic indicators and state indicators of all the precoding nodes;

[0023] Based on the distance index, the graph structure index, the semantic index and the state index, a weighted average score of all the precoding nodes is calculated, and the weighted average score is used as the node score of the corresponding precoding node.

[0024] Optionally, the step of calculating the weighted average score of all the precoding nodes based on the distance index, the graph structure index, the semantic index, and the state index adopts the formula:

[0025] ;

[0026] in, is the preset weight, is the distance indicator, is a semantic indicator, is the graph structure indicator, is a status indicator;

[0027] Get the distance indicator The formula used is as follows:

[0028] ;

[0029] in, For all the target location nodes, is the current location node, is a function for calculating the geometric distance between any of the target location nodes and the current location node, is the offset;

[0030] Get the semantic index The formula used is as follows:

[0031] ;

[0032] in, For all the precoding nodes, For the target semantics corresponding to each of the pre-encoding nodes, For text encoder;

[0033] Get the graph structure index The formula used is as follows:

[0034] ;

[0035] in, For all the target location nodes, is the current location node, is a function for calculating the graph structure similarity between any of the target location nodes and the current location node;

[0036] Get the status indicator The formula used is as follows:

[0037] ;

[0038] in, For all the precoding nodes, To assign a function.

[0039] Optionally, the step of determining a plurality of high-priority nodes according to the node scores of all the precoding nodes includes:

[0040] Sorting all the node scores, and selecting the top k precoding nodes in the node score ranking; k is an integer not less than 1;

[0041] Each of the first k precoding nodes is regarded as a high priority node.

[0042] Optionally, the high-priority node is used as a high-frequency access node, and all precoding nodes except the high-frequency access node are used as low-frequency access nodes;

[0043] storing all the high-frequency access nodes and all the low-frequency access nodes as historical compilation nodes respectively;

[0044] The step of precoding all direction nodes in the navigation path to obtain a plurality of precoded nodes includes:

[0045] Traversing all the direction nodes and all the historical compilation nodes to filter out uncompiled nodes to be compiled;

[0046] Extracting features from all the nodes to be compiled, and vectorizing the extracted features to obtain a feature vector for each node to be compiled;

[0047] The historical coding node and the feature vector of each of the nodes to be coded are combined to obtain multiple precoding nodes.

[0048] In summary, this application has at least one of the following beneficial effects:

[0049] 1. This application first collects image information of the target scene and preliminarily generates a geometric map and a semantic map, wherein the semantic map includes multiple position nodes and multiple direction nodes; then, a navigation map is obtained based on the semantic map and the geometric map, wherein the navigation map includes the optimal navigation path for the agent to navigate from the starting point to the end point; then, all direction nodes in the navigation path are pre-encoded to obtain pre-encoded nodes; each pre-encoded node is scored using multiple indicators, and multiple high-priority nodes are determined based on the node scores for pruning; the pruned pre-encoded nodes are input into a preset language model. Since the key information of the current agent's position node has been filtered out in the above pruning step, the amount of data input into the preset language model is reduced while ensuring navigation accuracy, and noise information is reduced, thereby accelerating the reasoning process of the preset language model, thereby reducing the delay of the visual language navigation system and achieving the beneficial effect of optimizing the navigation speed of the visual language navigation system.

[0050] 2. Through a multi-level cache mechanism, the historical node information in the historical navigation information is recorded. The historical node information includes the historical compilation information of the node. In this way, when the historical node needs to be called, it can be pre-filled directly using the stored vector, which reduces the repeated calculation encoding and vectorization process each time and speeds up the calculation process.

[0051] 3. Completed historical compilation nodes are divided into high-frequency access nodes and low-frequency access nodes. Infrequently accessed low-frequency access nodes are stored in the CPU, while frequently accessed high-frequency access nodes are stored in the GPU, ensuring that high-frequency access nodes can be quickly retrieved when in use. By adopting the above technical solution and combining it with the least recently accessed cache strategy, the most frequently used encoding information is always stored in the GPU, while infrequently used encoding information can be quickly retrieved from the CPU when needed. This ensures minimal scheduling overhead for the encoding information stored between the CPU and GPU, thereby improving the inference speed of the preset language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 1 is a flow chart of an acceleration method of a visual language navigation system provided in an embodiment of the present application;

[0053] Figure 2 This is a schematic diagram of state updating and node merging in the acceleration method of the visual language navigation system provided by an embodiment of the present application;

[0054] Figure 3 Schematic diagram of the pruning strategy adopted by the acceleration method of the visual language navigation system provided in an embodiment of the present application;

[0055] Figure 4 Schematic diagram of modules used in the acceleration method of the visual language navigation system provided in an embodiment of the present application;

[0056] Figure 5 is a virtual structure schematic diagram of an acceleration device of a visual language navigation system provided by the present application;

[0057] Figure 6 is a structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] The present application will be further described below in conjunction with the drawings and embodiments.

[0059] The concept, specific structure and generated technical effects of the present application will be clearly and completely described below in conjunction with embodiments and drawings, so as to fully understand the purposes, features and effects of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application. In addition, all the coupling / connection relations involved in the patent do not mean that the components are directly connected, but means that a better coupling structure can be composed by adding or reducing coupling accessories according to the specific implementation. The technical features in the present application can be combined with each other without contradiction and conflict.

[0060] Reference Figure 1 , Figure 1 is a flow schematic diagram of an acceleration method of a visual language navigation system provided by an embodiment of the present application, and the method comprises the following steps, which will be described in detail below.

[0061] In step S1, image information of a target scene is acquired, and a geometric map and a semantic map of the target scene are generated according to the image information; the semantic map comprises a plurality of position nodes and a plurality of direction nodes; one position node corresponds to a plurality of direction nodes.

[0062] Specifically, the image information is simultaneously collected by an agent (for example, a robot) using an RGBD camera (color depth camera, RGB-D camera, where D represents depth information) to collect color images and depth images of a target scene, and to ensure that a panoramic image of a current position of the agent is acquired. Wherein, the image information is stored in a multi-level cache, which specifically comprises the following steps:

[0063] The image information is acquired, and the image information is stored in the multi-level cache.

[0064] Specifically, the image information comprises color image information and depth image information; the multi-level cache contains historical node information of all direction nodes in the historical navigation.

[0065] Furthermore, the environment perception module in the agent preliminarily generates a geometric map and a semantic map. The following describes the generation of the geometric map and the generation of the semantic map in detail:

[0066] Regarding the geometric map: Based on the depth image captured by the RGBD camera and the known camera intrinsic parameters, the depth information of each pixel in the image can be converted into 3D point coordinates, thereby generating a point cloud. The generated point cloud is then stitched together using point cloud stitching technology and combined with the calculation of the camera trajectory to generate the required 3D geometric map.

[0067] About semantic maps: By inputting the image information (i.e. panoramic image) of the agent in various directions into a multimodal large model or an open target detection algorithm, the semantic label of each object in the image can be obtained, and the corresponding semantic label can be added to the subsequently generated direction nodes to complete the initial creation of the semantic map. Among them, the semantic map is divided into position-level nodes and direction-level nodes. The position-level nodes are used to store the location information of the agent, and the direction-level nodes are used to describe the environmental information in a specific direction. Among them, the environmental information includes object semantics, status tags, etc. The status tags of each direction node include visited nodes (nodes that have been processed or walked through), nodes that do not need to be visited (nodes whose depth information is less than the corresponding threshold), and unvisited (nodes to be processed).

[0068] More specifically, a multi-level cache stores historical node information for all directional nodes in historical navigation. This information includes, but is not limited to, the status flags and encoding vectors for each directional node. This information is incorporated into the semantic map during its creation. Specifically, the color and depth image information is summarized and organized based on this historical node information to create the semantic map. This step facilitates the subsequent use of this historical information, avoids repeated calculations, and improves navigation speed.

[0069] More specifically, during the creation of the semantic map, there may be multiple redundant direction nodes with similar semantics. Therefore, in the embodiment of the present application, it is proposed to dynamically update the geometric map and the semantic map according to changes in the environment during the agent navigation process. The following is a detailed explanation:

[0070] All adjacent location nodes are determined according to the geometric map.

[0071] Specifically, the positional relationship between each location node and other location nodes is determined according to the distance index in the geometric map, thereby obtaining multiple groups of adjacent location nodes, wherein a group of adjacent location nodes includes two adjacent location nodes.

[0072] Furthermore, based on the semantic labels of all direction nodes included in the adjacent position nodes in the same group, the semantic similarity between all the semantic labels is calculated;

[0073] Furthermore, judging whether there is at least one group of similar semantic nodes according to the semantic similarity; the similar semantic nodes include direction nodes whose semantic similarity of at least two semantic tags exceeds a preset similarity threshold;

[0074] If there is at least one group of similar semantic nodes, all directional nodes in the same group of similar semantic nodes are merged into one directional node, and the semantic map is updated according to all the merged directional nodes.

[0075] Specifically, the semantic similarity of the semantic labels of all corresponding directional nodes is calculated for groups of adjacent position nodes. If the semantic similarity between the semantic labels of at least one directional node of a position node and at least one directional node of another position node exceeds a preset similarity threshold, these nodes are considered similar semantic nodes. Subsequently, all directional nodes in the same group of similar semantic nodes are merged into a single directional node to remove redundancy. The preset similarity threshold can be set by relevant personnel based on actual needs.

[0076] For example, assume there are location nodes A and B, where location node A corresponds to directional nodes A1 and A2, and location node B corresponds to directional nodes B1 and B2. Compare A1 and A2 with B1 and B2 to obtain the semantic similarity between each pair. If the semantic similarity of the semantic labels of A1 and B1 exceeds a preset similarity threshold, then A1 and B1 are merged into a single directional node. This merged directional node is both the directional node for location node A and the directional node for location node B.

[0077] Reference Figure 2 , Figure 2 This is a schematic diagram of state update and node merging in the acceleration method of the visual language navigation system provided in the embodiment of the present application. It can be seen that the initial state position node 1 and position node 2 each correspond to four direction nodes. After the node merging, two groups of similar semantic nodes are determined and merged into two direction nodes respectively. These two direction nodes are jointly associated by position node 1 and position node 2. Furthermore, it can be seen that in addition to the node merging, the state mark corresponding to each direction node is also synchronized to the direction node, that is, the above-mentioned visited nodes (nodes that have been processed or walked through), nodes that do not need to be visited (nodes whose depth information is less than the corresponding threshold), and unvisited (nodes to be processed), so as to facilitate the use of subsequent pruning and other steps.

[0078] Specifically, the semantic similarity can be calculated using the following formula:

[0079] ;

[0080] Among them, the direction nodes representing different front and back adjacent position nodes represent the text encoder.

[0081] More specifically, since each time the agent moves, its location node and the corresponding direction node must be updated, in this case, it is also necessary to perform the above-mentioned state synchronization of semantically similar nodes. In addition, since the state mark has been added to the semantic map in the above steps, those direction nodes that have been visited and are similar or redundant can be merged based on the distance between the nodes and semantic indicators (such as the actual direction represented by the node, the relationship with other nodes, etc.). In some possible embodiments, there are some direction nodes that do not need to be visited in the current task or analysis. However, the status of these nodes may still be affected by changes in the status of other nodes. Therefore, it is necessary to ensure that these direction nodes that do not need to be visited can also receive the necessary state synchronization information to maintain the integrity and accuracy of the entire system state.

[0082] Obviously, by adopting the above technical solution to merge semantically similar nodes, the storage space occupied can be reduced, thereby reducing the impact of redundant information on the reasoning navigation speed.

[0083] In step S2, based on a preset navigation target, a navigation map is generated in combination with the geometric map and the semantic map. The navigation map includes a navigation path. The navigation path includes a plurality of target location nodes and direction nodes of all the target location nodes.

[0084] Specifically, the target location node refers to the location node that the agent is currently located at, passing through, or ultimately reaches during the navigation path. The preset navigation target is the destination (i.e., the final target location node) that the agent is navigating to. Based on the final target location node, the distance metrics of the geometric map, and the semantic labels of the semantic map, path planning is used to provide the agent with a route from the current location node to the target location node, i.e., the navigation path. This navigation map is derived from the two cached maps (the geometric map and the semantic map) described above. This map can be reused after creation to avoid wasted computing power through repeated generation and improve inference speed.

[0085] More specifically, the navigation map consists of a high-level decision module and a low-level control module.

[0086] First, the high-level decision module, built based on the current semantic map, determines the agent's next direction. Specifically, this decision is made in two scenarios: First, if the semantic label of the final target location node exists in a previously stored semantic map, the location of that location node can be directly retrieved. Second, if the final target location node does not exist in the previously stored semantic map, the current semantic map is input into the LLM module to retrieve the locations of multiple target location nodes along the path and the semantic labels of the directional nodes.

[0087] Secondly, the underlying control module is built based on a geometric map to guide the agent on how to move. According to the positions of multiple path target nodes and the final target node obtained by the above-mentioned high-level decision module, the obstacle avoidance algorithm calculates the walking route in the geometric map, and then the agent can be guided to that route.

[0088] In step S3, all direction nodes in the navigation path are precoded to obtain a plurality of precoded nodes.

[0089] Specifically, all direction nodes of the navigation path obtained in the above steps are pre-encoded. Pre-encoding includes two stages: pre-filling and decoding. In the pre-filling stage, the encoder of the LLM module is used to extract features and vectorize the information contained in each direction node, and store it as historical node information. Specifically, it includes the following steps:

[0090] Traversing all the direction nodes and all the historical compilation nodes to filter out uncompiled nodes to be compiled;

[0091] Extracting features from all the nodes to be compiled, and vectorizing the extracted features to obtain a feature vector for each node to be compiled;

[0092] The historical coding node and the feature vector of each of the nodes to be coded are combined to obtain multiple precoding nodes.

[0093] Specifically, a historical compilation node refers to a node that has been compiled in the semantic map. The pre-filled vectors of the compiled nodes are stored in the cache, that is, the above-mentioned historical node information. By detecting whether there is a compiled historical compilation node in the direction node currently needing to be compiled, if there is, the corresponding stored vector of the historical compilation node is directly applied, thereby reducing the number of direction nodes that need to be recompiled and achieving the purpose of accelerating the calculation process.

[0094] In step S4, the current location node of the agent is obtained, and the node scores of all the pre-coding nodes are calculated based on all the target location nodes and all the pre-coding nodes; all the target location nodes include the current location node.

[0095] Specifically, the location node of the current agent, i.e., the current location node, is determined by measurement or other means, and then all path location nodes and all pre-coding nodes are calculated to obtain the node score of each pre-coding node. Specifically, the following steps are included:

[0096] The distance index and graph structure index of all the position nodes, and the semantic index and state index of all the precoding nodes are obtained.

[0097] About distance index: The distance index is calculated based on the distance between each location node and the current location node, and the distance weight is distributed in an inverse relationship, that is, the closer the distance, the greater the weight. The formula used is as follows:

[0098] ;

[0099] in, For all the precoding nodes, For the target semantics corresponding to each of the pre-encoding nodes, A text encoder.

[0100] About graph structure index: By calculating the number of edges between the current position node and other position nodes, as well as the number of similar nodes, if the node that is more relevant to the current position node is given a higher weight, the graph structure index is obtained. The formula used is as follows:

[0101] ;

[0102] in, For all the target location nodes, is the current location node, is a function for calculating the graph structure similarity between any of the target location nodes and the current location node.

[0103] Regarding semantic indicators: For each direction node of a position node, the text encoder TEXT calculates the similarity between its pre-encoded vector and the target semantics to obtain the semantic indicator The formula used is as follows:

[0104] ;

[0105] in, For all the precoding nodes, For the target semantics corresponding to each of the pre-encoding nodes, A text encoder.

[0106] Regarding the status index: In the above steps, the unvisited direction nodes and the visited direction nodes have been screened out. In the embodiment of the present application, the unvisited direction nodes are given a larger score, while the visited direction nodes are given a smaller score. The status index is obtained. The formula used is as follows:

[0107] ;

[0108] in, For all the precoding nodes, To assign a function.

[0109] Furthermore, based on the distance index, the graph structure index, the semantic index and the state index, a weighted average score of all the precoding nodes is calculated, and the weighted average score is used as the node score of the corresponding precoding node.

[0110] Specifically, a weighted average calculation is performed based on the four indicators obtained in the above steps to obtain the node scores corresponding to all pre-coding nodes. Specifically, the formula used in the step of calculating the weighted average score of all the pre-coding nodes based on the distance indicator, the graph structure indicator, the semantic indicator and the state indicator is:

[0111] ;

[0112] in, is the preset weight, is the distance indicator, is a semantic indicator, is the graph structure indicator, is a status indicator.

[0113] In step S5, a plurality of high-priority nodes are determined according to the node scores of all the pre-coded nodes to prune the navigation path.

[0114] Specifically, the node scores can be sorted from high to low, and nodes with lower node scores can be removed as noise and not input into the subsequent preset language model. In an embodiment of the present application, the step of determining multiple high-priority nodes based on the node scores of all the pre-coded nodes includes:

[0115] Sorting all the node scores, and selecting the top k precoding nodes in the node score ranking; k is an integer not less than 1;

[0116] Furthermore, each of the first k precoding nodes is regarded as a high-priority node.

[0117] Specifically, the first k precoding nodes with higher scores are historical information related to the current position node, and the other precoding nodes are regarded as noise information and are not input into the preset language model.

[0118] For example, refer to Figure 3 , Figure 3 This is a schematic diagram of the pruning strategy used in the acceleration method of the visual language navigation system provided in the embodiment of the present application. It can be seen that Figure 3 Including location node 1, location node 2, location node 3 and location node 4, where location node 4 is the target location node, location node 1, location node 2 and location node 3 are sub-target location nodes, in the above merger and state synchronization (refer to Figure 2 ) has determined the nodes that do not need to be visited and the nodes that have been visited for each position node and marked them accordingly. Figure 3 The visited direction nodes and unvisited direction nodes of each position node are calculated and sorted, and the first k direction nodes are selected as precoding nodes for subsequent input into the preset language module.

[0119] More specifically, the k pre-coding nodes may be saved as high-frequency called nodes, and the other nodes may be saved as low-frequency called nodes.

[0120] More specifically, the present application uses a GPU (Graphics Processing Unit) and a CPU (Central Processing Unit) to jointly store the encoding information calculated in the above pre-encoding stage. Since the encoding information needs to be in the GPU during LLM inference, the present application also includes the following steps:

[0121] The high-priority node is used as a high-frequency access node, and all precoding nodes except the high-frequency access node are used as low-frequency access nodes;

[0122] All the high-frequency access nodes and all the low-frequency access nodes are stored as historical compilation nodes respectively.

[0123] Specifically, frequently accessed nodes are stored in the GPU, while less frequently accessed nodes are stored in the CPU. A least recently used (LRU) cache strategy ensures that frequently accessed nodes can be quickly retrieved. It's worth noting that during agent movement, the required frequently accessed nodes may differ. In this case, pre-encoded node scheduling between the GPU and CPU is sufficient. This technical solution enables rapid access, accelerating navigation reasoning.

[0124] In step S6, the pruned navigation path is input into a preset language model, and navigation information is generated by the preset language model.

[0125] Specifically, the pruned navigation path includes the first k pre-coded nodes obtained in the above steps corresponding to the target location node and all sub-target location nodes. The preset language model can be a large language model LLM. The large language model is based on the received pre-coded nodes, and collaborative reasoning is performed to obtain the next navigation strategy, that is, navigation information.

[0126] For example, refer to Figure 4 , Figure 4 This is a schematic diagram of the modules used in the acceleration method of the visual language navigation system provided in the embodiment of the present application. Figure 4 It can be seen that this application first obtains environmental information by the environmental perception module and stores it in a multi-level cache; secondly, in the multi-level cache, the information is sorted and summarized to form a geometric map, a semantic map and a navigation map; finally, the semantic map is pre-encoded and pruned, and then the semantic map works together with the LLM model to complete navigation.

[0127] Reference Figure 5 , Figure 5 : is a virtual structural diagram of an acceleration device based on a visual language navigation system provided by the present application. In a second aspect, the present application provides an acceleration device based on a visual language navigation system, comprising:

[0128] The image information acquisition module 100 is used to acquire image information of a target scene and generate a geometric map and a semantic map of the target scene based on the image information; the semantic map includes a plurality of position nodes and a plurality of direction nodes; one position node corresponds to a plurality of direction nodes;

[0129] A navigation path acquisition module 200 is configured to generate a navigation map based on a preset navigation target and in combination with the geometric map and the semantic map, wherein the navigation map includes a navigation path, and the navigation path includes a plurality of target location nodes and direction nodes for all the target location nodes;

[0130] A precoding node module 300 is configured to precode all direction nodes in the navigation path to obtain a plurality of precoding nodes;

[0131] The node score acquisition module 400 is used to obtain the current location node of the agent, and calculate the node scores of all the pre-coded nodes based on all the target location nodes and all the pre-coded nodes; all the target location nodes include the current location node;

[0132] a navigation path pruning module 500, configured to determine a plurality of high-priority nodes according to the node scores of all the pre-coded nodes, so as to prune the navigation path;

[0133] The model collaborative decision module 600 is used to input the pruned navigation path into a preset language model and generate navigation information through the preset language model.

[0134] The acceleration device based on the visual language navigation system described in the embodiment of the present application can execute the acceleration method based on the visual language navigation system provided in the above embodiment. The acceleration device based on the visual language navigation system has the corresponding functional steps and beneficial effects of the acceleration method based on the visual language navigation system described in the above embodiment. Please refer to the above embodiment of the acceleration method based on the visual language navigation system for details. The embodiment of the present application will not be repeated here.

[0135] The present application also provides an electronic device. Figure 6 , Figure 6 : is a structural diagram of an electronic device provided in an embodiment of the present application, which electronic device may include a processor and a memory, wherein the processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned chips. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the acceleration method based on the visual language navigation system in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the acceleration method based on the visual language navigation system in the above-mentioned method embodiment.

[0136] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. The one or more modules are stored in the memory, and when executed by the processor, the acceleration method based on the visual language navigation system in the above-mentioned method embodiment is executed. The specific details of the above-mentioned electronic device can be understood by corresponding to the corresponding descriptions and effects in the above-mentioned method embodiment, and will not be repeated here. Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the embodiments of the above-mentioned methods. The storage medium may be a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above types of memory.

[0137] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0138] Similarly, it should be understood that in order to streamline the present disclosure and aid understanding of one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present application.

[0139] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims.

Claims

1. A method for accelerating a visual language navigation system, applied to an agent, characterized in that: The method comprises: Acquire image information of a target scene, and generate a geometric map and a semantic map of the target scene based on the image information; the semantic map includes a plurality of position nodes and a plurality of direction nodes; one position node corresponds to a plurality of direction nodes; Based on a preset navigation target, generating a navigation map in combination with the geometric map and the semantic map, wherein the navigation map includes a navigation path, and the navigation path includes a plurality of target location nodes and direction nodes of all the target location nodes; Precoding all direction nodes in the navigation path to obtain multiple precoded nodes; Obtaining the current location node of the agent, and calculating the node scores of all the pre-coding nodes based on all the target location nodes and all the pre-coding nodes; all the target location nodes include the current location node; determining a plurality of high-priority nodes according to the node scores of all the pre-coded nodes to prune the navigation path; The pruned navigation path is input into a preset language model, and navigation information is generated by the preset language model.

2. The acceleration method of the visual language navigation system according to claim 1, characterized in that: The steps of acquiring image information of the target scene and generating a geometric map and a semantic map of the target scene according to the image information include: Acquire the image information and store the image information in a multi-level cache; the image information includes color image information and depth image information; the multi-level cache contains historical node information of all direction nodes in historical navigation; The color image information and the depth image information are summarized and sorted according to the historical node information to obtain the semantic map.

3. The acceleration method of the visual language navigation system according to claim 2, characterized in that: Before the step of generating a navigation map based on a preset navigation target by combining the geometric map and the semantic map, the method includes: Determine all adjacent position nodes according to the geometric map; a group of adjacent position nodes includes two adjacent position nodes; Calculating semantic similarities between all pairs of semantic labels based on semantic labels of all direction nodes included in the adjacent position nodes in the same group; Determining whether there is at least one group of similar semantic nodes based on the semantic similarity; the similar semantic nodes include directional nodes whose semantic similarity of at least two semantic tags exceeds a preset similarity threshold; If there is at least one group of similar semantic nodes, all directional nodes in the same group of similar semantic nodes are merged into one directional node, and the semantic map is updated according to all the merged directional nodes.

4. The acceleration method of the visual language navigation system according to claim 1, characterized in that: The step of calculating the node scores of all the precoding nodes based on all the target location nodes and all the precoding nodes includes: Obtaining distance indicators and graph structure indicators of all the position nodes, and semantic indicators and state indicators of all the precoding nodes; Based on the distance index, the graph structure index, the semantic index and the state index, a weighted average score of all the precoding nodes is calculated, and the weighted average score is used as the node score of the corresponding precoding node.

5. The acceleration method of the visual language navigation system according to claim 4, characterized in that: The formula used in the step of calculating the weighted average score of all the precoding nodes based on the distance index, the graph structure index, the semantic index, and the state index is: ； in, is the preset weight, is the distance indicator, is a semantic indicator, is the graph structure indicator, is a status indicator; Get the distance indicator The formula used is as follows: ； in, For all the target location nodes, is the current location node, is a function for calculating the geometric distance between any of the target location nodes and the current location node, is the offset; Get the semantic index The formula used is as follows: ； in, For all the precoding nodes, For the target semantics corresponding to each of the pre-encoding nodes, For text encoder; Get the graph structure index The formula used is as follows: ； in, For all the target location nodes, is the current location node, is a function for calculating the graph structure similarity between any of the target location nodes and the current location node; Get the status indicator The formula used is as follows: ； in, For all the precoding nodes, To assign a function.

6. The acceleration method of the visual language navigation system according to claim 4, characterized in that: The step of determining a plurality of high-priority nodes according to the node scores of all the pre-coded nodes comprises: Sorting all the node scores, and selecting the top k precoding nodes in the node score ranking; k is an integer not less than 1; Each of the first k precoding nodes is regarded as a high priority node.

7. The acceleration method of the visual language navigation system according to claim 6, characterized in that: The high-priority node is used as a high-frequency access node, and all precoding nodes except the high-frequency access node are used as low-frequency access nodes; storing all the high-frequency access nodes and all the low-frequency access nodes as historical compilation nodes respectively; The step of precoding all direction nodes in the navigation path to obtain a plurality of precoded nodes includes: Traversing all the direction nodes and all the historical compilation nodes to filter out uncompiled nodes to be compiled; Extracting features from all the nodes to be compiled, and vectorizing the extracted features to obtain a feature vector for each node to be compiled; The historical coding node and the feature vector of each of the nodes to be coded are combined to obtain multiple precoding nodes.

8. An acceleration device for a visual language navigation system, characterized in that: Applies to proxies, including: An image information acquisition module is used to acquire image information of a target scene and generate a geometric map and a semantic map of the target scene based on the image information; the semantic map includes a plurality of position nodes and a plurality of direction nodes; one position node corresponds to a plurality of direction nodes; A navigation path acquisition module, configured to generate a navigation map based on a preset navigation target and in combination with the geometric map and the semantic map, wherein the navigation map includes a navigation path, and the navigation path includes a plurality of target location nodes and direction nodes of all the target location nodes; A precoding node module, configured to precode all direction nodes in the navigation path to obtain a plurality of precoding nodes; a node score acquisition module, configured to acquire the current location node of the agent, and calculate the node scores of all the pre-coded nodes based on all the target location nodes and all the pre-coded nodes; all the target location nodes include the current location node; a navigation path pruning module, configured to determine a plurality of high-priority nodes according to the node scores of all the pre-coded nodes, so as to prune the navigation path; The model collaborative decision-making module is used to input the pruned navigation path into a preset language model and generate navigation information through the preset language model.

9. An electronic device, characterized in that: include: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a program or instruction, and when the program or instruction is executed, the method according to any one of claims 1 to 7 is implemented.

Images