Acceleration method and device of visual language navigation system, equipment and storage medium

Abstract

The invention relates to the technical field of visual language navigation, in particular to an acceleration method and device of a visual language navigation system, equipment and a storage medium. The method comprises the steps of obtaining image information of a target scene, and generating a geometric map and a semantic map of the target scene according to the image information; generating a navigation map by combining the geometric map and the semantic map based on a preset navigation target; pre-coding all direction nodes in the navigation path to obtain a plurality of pre-coded nodes; calculating node scores of all the precoding nodes based on all the position nodes and all the precoding nodes; determining a plurality of high-priority nodes according to the node scores of all pre-coding nodes so as to prune a navigation path; and inputting the pruned navigation path into a preset language model to generate navigation information. According to the method, the delay of the visual language navigation system can be reduced, and the beneficial effect of optimizing the navigation speed of the visual language navigation system is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of visual language navigation, and in particular to an acceleration method, device, equipment and storage medium for a visual language navigation system. Background Art

[0002] In the visual-language navigation (VLN) task, the agent needs to navigate according to natural language instructions to achieve smooth movement from the starting point to the target. Learning-based methods have always been the main solution for this task. However, these methods usually face the problems of high training costs and lack of generality, which limits their widespread promotion in practical applications. In recent years, with the development of large language models (LLMs), their powerful generalization ability has become an effective tool for VLN tasks under zero-shot learning. By combining visual information with language understanding, LLM can adapt to different navigation tasks in a variety of environments, which not only improves accuracy but also enhances the robustness of the system. In addition, LLM also has the advantages of multi-round dialogue and context understanding, and can adjust navigation strategies according to real-time feedback to improve navigation flexibility.

[0003] However, the current zero-shot learning visual language navigation method strongly relies on the reasoning ability of LLM, and the large delay problem of LLM becomes the bottleneck of this method. Based on this, how to reduce the delay of the visual language navigation system and optimize the navigation speed of the visual language navigation system is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the present application provides an acceleration method, device, equipment and storage medium for a visual language navigation system to reduce the delay of the visual language navigation system, thereby achieving the beneficial effect of optimizing the navigation speed of the visual language navigation system.

[0005] The technical solution adopted by this application to solve its technical problem is: In a first aspect, the present application provides an acceleration method for a visual language navigation system, which is applied to an agent, and the method comprises: Acquire image information of a target scene, and generate a geometric map and a semantic map of the target scene according to the image information; the semantic map includes a plurality of position nodes and a plurality of direction nodes; one position node corresponds to a plurality of direction nodes; Based on a preset navigation target, a navigation map is generated in combination with the geometric map and the semantic map, wherein the navigation map includes a navigation path, and the navigation path includes a plurality of target location nodes and direction nodes of all the target location nodes; Precoding all direction nodes in the navigation path to obtain a plurality of precoded nodes; Obtaining the current location node where the proxy is located, and calculating the node scores of all the precoding nodes based on all the target location nodes and all the precoding nodes; all the target location nodes include the current location node; Determining a plurality of high priority nodes according to the node scores of all the pre-encoded nodes to prune the navigation path; The pruned navigation path is input into the preset language model, and navigation information is generated through the preset language model.

[0006] Optionally, the step of acquiring image information of the target scene and generating a geometric map and a semantic map of the target scene according to the image information includes: Acquire the image information, and store the image information in a multi-level cache; the image information includes color image information and depth image information; the multi-level cache includes historical node information of all direction nodes in historical navigation; The color image information and the depth image information are summarized and sorted according to the historical node information to obtain the semantic map.

[0007] Optionally, before the step of generating a navigation map based on a preset navigation target and combining the geometric map and the semantic map, the method includes: Determine all adjacent position nodes according to the geometric map; a group of adjacent position nodes includes two adjacent position nodes; Based on the semantic labels of all direction nodes included in the adjacent position nodes in the same group, calculating the semantic similarity between all the semantic labels; Judging whether there is at least one group of similar semantic nodes according to the semantic similarity; the similar semantic nodes include direction nodes whose semantic similarity of at least two semantic tags exceeds a preset similarity threshold; If there is at least one group of similar semantic nodes, all directional nodes in the similar semantic nodes of the same group are merged into one directional node, and the semantic map is updated according to all the merged directional nodes.

[0008] Optionally, the step of calculating the node scores of all the precoding nodes based on all the target location nodes and all the precoding nodes includes: Obtaining distance indicators and graph structure indicators of all the position nodes, and semantic indicators and state indicators of all the precoding nodes; Based on the distance index, the graph structure index, the semantic index and the state index, a weighted average score of all the precoding nodes is calculated, and the weighted average score is used as a node score of the corresponding precoding node.

[0009] Optionally, the step of calculating the weighted average score of all the precoding nodes based on the distance information, the graph structure information, the semantic information and the state information adopts the formula: ; in, is the preset weight, is the distance indicator, is a semantic indicator, is the graph structure indicator, is a status indicator; Get the distance indicator The formula used is as follows: ; in, For all the target location nodes, is the current location node, is a function for calculating the geometric distance between any of the target position nodes and the current position node, is the offset; Get the semantic index The formula used is as follows: ; in, For all the precoding nodes, For the target semantics corresponding to each of the pre-coding nodes, is a text encoder; Get the graph structure index The formula used is as follows: ; in, For all the target location nodes, is the current location node, is a function for calculating the graph structure similarity between any of the target position nodes and the current position node; Get the status indicator The formula used is as follows: ; in, For all the precoding nodes, To assign a function.

[0010] Optionally, the step of determining a plurality of high-priority nodes according to the node scores of all the precoding nodes includes: Sorting all the node scores, and selecting the first k precoding nodes in the node score ranking; k is an integer not less than 1; Each of the first k precoding nodes is regarded as a high priority node.

[0011] Optionally, the high priority node is used as a high frequency access node, and all precoding nodes except the high frequency access node are used as low frequency access nodes; storing all the high-frequency access nodes and all the low-frequency access nodes as historical compilation nodes respectively; The step of precoding all direction nodes in the navigation path to obtain a plurality of precoded nodes comprises: Traversing all the direction nodes and all the historical compilation nodes to filter out the uncompiled nodes to be compiled; Extracting features from all the nodes to be compiled, and vectorizing the extracted features to obtain a feature vector for each node to be compiled; The historical coding node and the feature vector of each of the nodes to be coded are combined to obtain a plurality of precoding nodes.

[0012] In summary, the present application has at least one of the following beneficial effects: 1. The present application first collects image information of the target scene, and preliminarily generates a geometric map and a semantic map, wherein the semantic map includes multiple position nodes and multiple direction nodes; then a navigation map is obtained based on the semantic map and the geometric map, wherein the navigation map includes the optimal navigation path for the agent to navigate from the starting point to the end point; then, all direction nodes in the navigation path are pre-coded to obtain pre-coded nodes; each pre-coded node is scored based on a plurality of indicators, and a plurality of high-priority nodes are determined according to the node scores, and pruning operations are performed; the pruned pre-coded nodes are input into a preset language model. Since the key information of the position node where the current agent is located has been screened in the above pruning step, the amount of data input into the preset language model is reduced while ensuring the navigation accuracy, and the noise information is reduced, thereby accelerating the reasoning process of the preset language model, thereby reducing the delay of the visual language navigation system and achieving the beneficial effect of optimizing the navigation speed of the visual language navigation system.

[0013] 2. Through a multi-level cache mechanism, the historical node information in the historical navigation information is recorded. The historical node information includes the historical compilation information of the node. In this way, when the historical node needs to be called, the stored vector can be directly used for pre-filling, which reduces the repeated calculation of encoding and vectorization process each time and speeds up the calculation process.

[0014] 3. The historical compilation nodes that have completed compilation are divided into high-frequency access nodes and low-frequency access nodes. The infrequently used low-frequency access nodes are stored in the CPU, while the frequently used high-frequency access nodes are stored in the GPU, thereby ensuring that the high-frequency access nodes can be quickly retrieved when in use; by adopting the above technical solution, combined with the least recent cache strategy, it is ensured that the most commonly used encoding information is always stored in the GPU, and the infrequently used encoding information can be quickly retrieved from the CPU when needed, thereby ensuring the minimum scheduling overhead of the encoding information stored between the CPU and the GPU, and improving the inference speed of the preset language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flowchart of an acceleration method of a visual language navigation system provided in an embodiment of the present application; Figure 2 It is a schematic diagram of state updating and node merging in the acceleration method of the visual language navigation system provided in an embodiment of the present application; Figure 3 is a schematic diagram of a pruning strategy adopted by an acceleration method of a visual language navigation system provided in an embodiment of the present application; Figure 4 Schematic diagram of modules used in the acceleration method of the visual language navigation system provided in the embodiment of the present application; Figure 5 It is a virtual structural schematic diagram of an acceleration device of a visual language navigation system provided by the present application; Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The present application is further described below in conjunction with the accompanying drawings and embodiments.

[0017] The following will clearly and completely describe the concept, specific structure and technical effects of the present application in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments of the present application, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present application. In addition, all the connection / connection relationships involved in the patent do not simply refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features created in this application can be combined interactively without conflicting with each other.

[0018] Reference Figure 1 , Figure 1: is a flow chart of an acceleration method of a visual language navigation system provided in an embodiment of the present application, and the method comprises the following steps, which are specifically described below: In step S1, image information of a target scene is acquired, and a geometric map and a semantic map of the target scene are generated according to the image information; the semantic map includes a plurality of position nodes and a plurality of direction nodes; one position node corresponds to a plurality of direction nodes.

[0019] Specifically, the image information is collected by the agent (such as a robot) using an RGBD camera (color depth camera, RGB-DCamera, where D stands for Depth) to simultaneously collect the color image and depth image of the target scene, and ensure that the panoramic image of the current position of the agent is obtained. The image information will be stored in a multi-level cache, which specifically includes the following steps: The image information is acquired, and the image information is stored in a multi-level cache.

[0020] Specifically, the image information includes color image information and depth image information; the multi-level cache contains historical node information of all direction nodes in the historical navigation.

[0021] Furthermore, the environment perception module in the agent preliminarily generates a geometric map and a semantic map. The generation of the geometric map and the generation of the semantic map are described in detail below: About geometric maps: Based on the depth image collected by the RGBD camera and the known camera internal parameters, the depth information of each pixel in the image can be converted into three-dimensional point coordinates to generate a point cloud. Then, the generated point cloud is stitched using point cloud stitching technology, and combined with the calculation of the camera trajectory to generate the required 3D geometric map.

[0022] About semantic maps: By inputting the image information (i.e. panoramic images) of the agent in all directions into a multimodal large model or an open target detection algorithm, the semantic labels of each object in the image can be obtained, and the corresponding semantic labels can be added to the subsequently generated direction nodes, thereby completing the preliminary creation of the semantic map. Among them, the semantic map is divided into position-level nodes and direction-level nodes. The position-level nodes are used to store the location information of the agent, and the direction-level nodes are used to describe the environmental information in a specific direction. Among them, the environmental information includes object semantics, status tags, etc. The status tags of each direction node include visited nodes (processed or walked nodes), nodes that do not need to be visited (nodes with depth information less than the corresponding threshold), and unvisited (nodes to be processed).

[0023] More specifically, the multi-level cache stores the historical node information of all direction nodes in the historical navigation, including but not limited to the status mark and encoding vector information of each direction node. The semantic map will be combined with this information during the creation process, that is, the color image information and the depth image information are summarized and sorted according to the historical node information to obtain the semantic map. Based on this step, the subsequent use of historical information is facilitated, repeated calculations are avoided, and the navigation speed is improved.

[0024] More specifically, in the process of creating a semantic map, there may be multiple redundant direction nodes with similar semantics. Therefore, in the embodiment of the present application, it is proposed to dynamically update the geometric map and the semantic map according to the changes in the environment during the agent navigation process, which is specifically described below: All adjacent location nodes are determined according to the geometric map.

[0025] Specifically, the positional relationship between each location node and other location nodes is determined according to the distance information in the geometric map, thereby obtaining multiple groups of adjacent location nodes, wherein a group of adjacent location nodes includes two adjacent location nodes.

[0026] Further, based on the semantic labels of all direction nodes included in the adjacent position nodes in the same group, the semantic similarities between all the semantic labels are calculated; Further, judging whether there is at least one group of similar semantic nodes according to the semantic similarity; the similar semantic nodes include direction nodes whose semantic similarity of at least two semantic tags exceeds a preset similarity threshold; If there is at least one group of similar semantic nodes, all directional nodes in the similar semantic nodes of the same group are merged into one directional node, and the semantic map is updated according to all the merged directional nodes.

[0027] Specifically, the semantic similarity of the semantic labels of all directional nodes corresponding to the two is calculated based on the group of adjacent position nodes. If the semantic similarity of the semantic labels of at least one directional node of a certain position node and at least one directional node of another position node exceeds the preset similarity threshold, they are regarded as similar semantic nodes, and then all directional nodes in the same group of similar semantic nodes are merged into one directional node to remove redundant parts. Among them, the preset similarity threshold can be set by relevant personnel according to actual needs.

[0028] For example, assume that there are location nodes A and location node B, where location node A corresponds to direction nodes A1 and A2, and location node B corresponds to direction nodes B1 and B2, and compare A1, A2 with B1, B2 to obtain the semantic similarity between them. Assuming that the semantic similarity of the semantic labels of A1 and B1 exceeds the preset similarity threshold, A1 and B1 are merged into one direction node, which is both the direction node of location node A and the direction node of location node B.

[0029] Reference Figure 2 , Figure 2 This is a schematic diagram of state updating and node merging in the acceleration method of the visual language navigation system provided in the embodiment of the present application. It can be seen that the initial state position node 1 and position node 2 each correspond to four direction nodes. After the node merging, two groups of similar semantic nodes are determined and merged into two direction nodes respectively. These two direction nodes are jointly associated by position node 1 and position node 2. Further, it can be seen that in addition to the node merging, the state mark corresponding to each direction node is also synchronized to the direction node, that is, the above-mentioned visited nodes (processed or walked nodes), nodes that do not need to be visited (nodes whose depth information is less than the corresponding threshold), and unvisited (nodes to be processed), so as to facilitate the use of subsequent pruning and other steps.

[0030] Specifically, the semantic similarity can be calculated using the following formula: ; Among them, each direction node representing different front and back adjacent position nodes represents a text encoder.

[0031] More specifically, since each time the agent moves, its location node and the corresponding direction node must be updated, in this case, it is also necessary to perform the above-mentioned state synchronization of semantically similar nodes. In addition, since the state mark has been added to the semantic map in the above steps, those direction nodes that have been visited and are similar or redundant can be merged according to the distance between the nodes and the semantic information (such as the actual direction represented by the node, the relationship with other nodes, etc.). In some possible embodiments, there are some direction nodes that do not need to be visited in the current task or analysis, but the status of these nodes may still be affected by changes in the status of other nodes. Therefore, it is necessary to ensure that these direction nodes that do not need to be visited can also receive the necessary state synchronization information to maintain the integrity and accuracy of the entire system state.

[0032] Obviously, by adopting the above technical solution to merge nodes with similar semantics, the storage space occupied can be reduced, thereby reducing the impact of redundant information on the reasoning navigation speed.

[0033] In step S2, based on a preset navigation target, a navigation map is generated in combination with the geometric map and the semantic map, wherein the navigation map includes a navigation path, and the navigation path includes a plurality of target location nodes and direction nodes of all the target location nodes.

[0034] Specifically, the target location node refers to the location node where the agent is located, passes through, or finally arrives in the navigation path. The preset navigation target is the end point to which the agent wants to navigate (i.e., the final target location node). Based on the final target location node, the distance information of the geometric map, and the semantic label of the semantic map, the route from the current location node to the target location node is provided to the agent in the form of path planning, i.e., the navigation path. The navigation map is summarized from the above two-level cache maps (geometric map and semantic map). The map can be reused after it is created to avoid repeated generation and waste of computing power, and to improve the reasoning speed.

[0035] More specifically, the navigation map consists of a high-level decision module and a low-level control module.

[0036] First, the high-level decision module is built based on the current semantic map to decide which direction the agent should go next. Specifically, the decision is divided into the following two situations: First, the semantic label of the final target location node exists in the previously stored semantic map, in which case the location of the location node can be directly obtained; second, the final target location node does not exist in the previously stored semantic map, then the current semantic map is input into the LLM module to obtain the locations of multiple target location nodes that need to be passed and the semantic labels on the direction nodes.

[0037] Secondly, the underlying control module is built based on the geometric map to guide the agent on how to move. According to the positions of multiple path target location nodes and the final target location nodes obtained by the above-mentioned high-level decision module, the obstacle avoidance algorithm calculates the walking route in the geometric map, and then the agent can be guided to move.

[0038] In step S3, all direction nodes in the navigation path are precoded to obtain a plurality of precoded nodes.

[0039] Specifically, all direction nodes of the navigation path obtained in the above steps are pre-encoded. The pre-encoding includes two stages: pre-filling and decoding. In the pre-filling stage, the encoder of the LLM module is used to extract features and vectorize the information contained in each direction node, and store it as historical node information. Specifically, it includes the following steps: Traversing all the direction nodes and all the historical compilation nodes to filter out the uncompiled nodes to be compiled; Extracting features from all the nodes to be compiled, and vectorizing the extracted features to obtain a feature vector for each node to be compiled; The historical coding node and the feature vector of each of the nodes to be coded are combined to obtain a plurality of precoding nodes.

[0040] Specifically, a historical compilation node refers to a node that has been compiled in the semantic map. The pre-filled vectors of the compiled nodes are stored in the cache, that is, the historical node information mentioned above. By detecting whether there is a compiled historical compilation node in the direction node currently required to be compiled, if there is, the corresponding stored vector of the historical compilation node is directly applied, thereby reducing the number of direction nodes that need to be recompiled, thereby achieving the purpose of accelerating the calculation process.

[0041] In step S4, the current location node of the proxy is obtained, and the node scores of all the precoding nodes are calculated based on all the target location nodes and all the precoding nodes; all the target location nodes include the current location node.

[0042] Specifically, the location node where the current agent is located, that is, the current location node, is determined by measurement or other means, and then all path location nodes and all pre-coding nodes are calculated, so as to calculate the node score of each pre-coding node, specifically including the following steps: The distance index and the graph structure index of all the position nodes, and the semantic index and the state index of all the precoding nodes are obtained.

[0043] About distance index: The distance index is to calculate the distance weight according to the distance between each location node and the current location node. The distribution of distance weight can be inversely proportional, that is, the closer the distance, the greater the weight. The formula used is as follows: ; in, For all the precoding nodes, For the target semantics corresponding to each of the pre-coding nodes, A text encoder.

[0044] About graph structure index: By calculating the number of edges between the current position node and other position nodes, as well as the number of similar nodes, if the node that is more relevant to the current position node is given a higher weight, the graph structure index is obtained The formula used is as follows: ; in, For all the target location nodes, is the current location node, is a function for calculating the graph structure similarity between any of the target location nodes and the current location node.

[0045] Regarding semantic indicators: For each direction node of the position node, the text encoder TEXT calculates the similarity between its pre-encoded vector and the target semantics to obtain the semantic indicators The formula used is as follows: ; in, For all the precoding nodes, For the target semantics corresponding to each of the pre-coding nodes, A text encoder.

[0046] Regarding the status indicator: In the above steps, the unvisited direction nodes and the visited direction nodes have been screened out. In the embodiment of the present application, the unvisited direction nodes are given a larger score, while the visited direction nodes are given a smaller score. The status indicator is obtained. The formula used is as follows: ; in, For all the precoding nodes, To assign a function.

[0047] Furthermore, based on the distance index, the graph structure index, the semantic index and the state index, a weighted average score of all the precoding nodes is calculated, and the weighted average score is used as the node score of the corresponding precoding node.

[0048] Specifically, a weighted average calculation is performed based on the four indicators obtained in the above steps to obtain the node scores corresponding to all precoding nodes. Specifically, the formula used in the step of calculating the weighted average score of all the precoding nodes based on the distance information, the graph structure information, the semantic information and the state information is: ; in, is the preset weight, is the distance indicator, is a semantic indicator, is the graph structure indicator, For status indicators.

[0049] In step S5, a plurality of high priority nodes are determined according to the node scores of all the pre-encoded nodes to prune the navigation path.

[0050] Specifically, the node scores can be sorted from high to low, and the nodes with lower node scores can be removed as noise and not input into the subsequent preset language model. In an embodiment of the present application, the step of determining a plurality of high priority nodes according to the node scores of all the pre-coded nodes includes: Sorting all the node scores, and selecting the first k precoding nodes in the node score ranking; k is an integer not less than 1; Further, each of the first k precoding nodes is taken as a high priority node.

[0051] Specifically, the first k pre-coding nodes with higher scores are the historical information related to the current position node, and the pre-coding nodes other than these are regarded as noise information and are not input into the preset language model.

[0052] For example, refer to Figure 3 , Figure 3 is a schematic diagram of the pruning strategy adopted by the acceleration method of the visual language navigation system provided in the embodiment of the present application. It can be seen that Figure 3 It includes location node 1, location node 2, location node 3 and location node 4, where location node 4 is the target location node, location node 1, location node 2 and location node 3 are sub-target location nodes, in the above merging and state synchronization (refer to Figure 2 ) has determined the nodes that do not need to be visited and the nodes that have been visited for each position node and marked them accordingly. Figure 3 The visited direction nodes and unvisited direction nodes of each position node are calculated and sorted, and the first k direction nodes are selected as pre-coding nodes for subsequent input into the preset language module.

[0053] More specifically, the k pre-coding nodes may be saved as high-frequency called nodes, and the other nodes may be saved as low-frequency called nodes.

[0054] More specifically, the present application uses a GPU (Graphics Processing Unit) and a CPU (Central Processing Unit) to jointly store the coding information calculated in the above pre-coding stage. Since the coding information needs to be in the GPU during LLM reasoning, the present application also includes the following steps: The high priority node is used as a high frequency access node, and all precoding nodes except the high frequency access node are used as low frequency access nodes; All the high-frequency access nodes and all the low-frequency access nodes are stored as historical compilation nodes respectively.

[0055] Specifically, high-frequency access nodes are stored in the GPU, and low-frequency access nodes are stored in the CPU. The Least Recently Used (LRU) cache strategy ensures that high-frequency access nodes can be quickly retrieved. It is worth noting that during the agent's movement, the required high-frequency access nodes may be different. In this case, pre-coding node scheduling between the GPU and the CPU is sufficient. By adopting the above technical solution, fast calls can be achieved, thereby accelerating the speed of navigation reasoning.

[0056] In step S6, the pruned navigation path is input into the preset language model, and navigation information is generated by the preset language model.

[0057] Specifically, the pruned navigation path includes the first k pre-coded nodes obtained in the above steps corresponding to the target position node and all sub-target position nodes. The preset language model can be a large language model LLM. The large language model is based on the received pre-coded nodes, and collaborative reasoning is performed to obtain the next navigation strategy, i.e., navigation information.

[0058] For example, refer to Figure 4 , Figure 4 Schematic diagram of the modules used in the acceleration method of the visual language navigation system provided in the embodiment of the present application. Figure 4 It can be seen that in this application, the environmental perception module first obtains environmental information and stores it in a multi-level cache; secondly, in the multi-level cache, the information is sorted and summarized to form a geometric map, a semantic map and a navigation map; finally, the semantic map is pre-encoded and pruned, and then the semantic map works together with the LLM model to complete navigation.

[0059] Reference Figure 5 , Figure 5 : is a virtual structural schematic diagram of an acceleration device based on a visual language navigation system provided by the present application. In a second aspect, the present application provides an acceleration device based on a visual language navigation system, comprising: The image information acquisition module 100 is used to acquire the image information of the target scene, and generate a geometric map and a semantic map of the target scene according to the image information; the semantic map includes a plurality of position nodes and a plurality of direction nodes; one position node corresponds to a plurality of direction nodes; A navigation path acquisition module 200, for generating a navigation map based on a preset navigation target in combination with the geometric map and the semantic map, wherein the navigation map includes a navigation path, and the navigation path includes a plurality of target location nodes and direction nodes of all the target location nodes; A precoding node module 300 is used to precode all direction nodes in the navigation path to obtain multiple precoding nodes; The node score acquisition module 400 is used to acquire the current location node of the agent, and calculate the node scores of all the pre-coded nodes based on all the target location nodes and all the pre-coded nodes; all the target location nodes include the current location node; A navigation path pruning module 500, configured to determine a plurality of high priority nodes according to the node scores of all the pre-coded nodes, so as to prune the navigation path; The model collaborative decision module 600 is used to input the pruned navigation path into the preset language model and generate navigation information through the preset language model.

[0060] The acceleration device based on the visual language navigation system described in the embodiment of the present application can execute the acceleration method based on the visual language navigation system provided in the above embodiment. The acceleration device based on the visual language navigation system has the corresponding functional steps and beneficial effects of the acceleration method based on the visual language navigation system described in the above embodiment. Please refer to the above embodiment of the acceleration method based on the visual language navigation system for details. The embodiment of the present application will not be repeated here.

[0061] The present application also provides an electronic device. Figure 6 , Figure 6 It is a structural diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include a processor and a memory, wherein the processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various types of chips. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the acceleration method based on the visual language navigation system in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the acceleration method based on the visual language navigation system in the above method embodiment is realized.

[0062] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. The one or more modules are stored in the memory, and when executed by the processor, the acceleration method based on the visual language navigation system in the above method embodiment is executed. The specific details of the above electronic device can be understood by corresponding to the corresponding descriptions and effects in the above method embodiment, and will not be repeated here. Those skilled in the art can understand that all or part of the processes in the above embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the processes of the embodiments of the above methods. Among them, the storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0063] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0064] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims, the inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present application.

[0065] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims.

Claims

1. A method for accelerating a visual language navigation system, applied to an agent, characterized in that: The method comprises: Acquire image information of a target scene, and generate a geometric map and a semantic map of the target scene according to the image information; the semantic map includes a plurality of position nodes and a plurality of direction nodes; one position node corresponds to a plurality of direction nodes; Based on a preset navigation target, a navigation map is generated in combination with the geometric map and the semantic map, wherein the navigation map includes a navigation path, and the navigation path includes a plurality of target location nodes and direction nodes of all the target location nodes; Precoding all direction nodes in the navigation path to obtain a plurality of precoded nodes; Obtaining the current location node where the proxy is located, and calculating the node scores of all the precoding nodes based on all the target location nodes and all the precoding nodes; all the target location nodes include the current location node; Determining a plurality of high priority nodes according to the node scores of all the pre-encoded nodes to prune the navigation path; The pruned navigation path is input into the preset language model, and navigation information is generated through the preset language model.

2. The acceleration method of the visual language navigation system according to claim 1, characterized in that: The step of acquiring image information of the target scene and generating a geometric map and a semantic map of the target scene according to the image information comprises: Acquire the image information, and store the image information in a multi-level cache; the image information includes color image information and depth image information; the multi-level cache includes historical node information of all direction nodes in historical navigation; The color image information and the depth image information are summarized and sorted according to the historical node information to obtain the semantic map.

3. The acceleration method of the visual language navigation system according to claim 2, characterized in that: Before the step of generating a navigation map based on a preset navigation target and combining the geometric map and the semantic map, the method includes: Determine all adjacent position nodes according to the geometric map; a group of adjacent position nodes includes two adjacent position nodes; Based on the semantic labels of all direction nodes included in the adjacent position nodes in the same group, calculating the semantic similarity between all the semantic labels; Judging whether there is at least one group of similar semantic nodes according to the semantic similarity; the similar semantic nodes include direction nodes whose semantic similarity of at least two semantic tags exceeds a preset similarity threshold; If there is at least one group of similar semantic nodes, all directional nodes in the similar semantic nodes of the same group are merged into one directional node, and the semantic map is updated according to all the merged directional nodes.

4. The acceleration method of the visual language navigation system according to claim 1, characterized in that: The step of calculating the node scores of all the precoding nodes based on all the target location nodes and all the precoding nodes comprises: Obtaining distance indicators and graph structure indicators of all the position nodes, and semantic indicators and state indicators of all the precoding nodes; Based on the distance index, the graph structure index, the semantic index and the state index, a weighted average score of all the precoding nodes is calculated, and the weighted average score is used as a node score of the corresponding precoding node.

5. The acceleration method of the visual language navigation system according to claim 4, characterized in that: The formula used in the step of calculating the weighted average score of all the precoding nodes based on the distance information, the graph structure information, the semantic information and the state information is: ； in, is the preset weight, is the distance indicator, is a semantic indicator, is the graph structure indicator, is a status indicator; Get the distance indicator The formula used is as follows: ； in, For all the target location nodes, is the current location node, is a function for calculating the geometric distance between any of the target position nodes and the current position node, is the offset; Get the semantic index The formula used is as follows: ； in, For all the precoding nodes, For the target semantics corresponding to each of the pre-coding nodes, is a text encoder; Get the graph structure index The formula used is as follows: ； in, For all the target location nodes, is the current location node, is a function for calculating the graph structure similarity between any of the target position nodes and the current position node; Get the status indicator The formula used is as follows: ； in, For all the precoding nodes, To assign a function.

6. The acceleration method of the visual language navigation system according to claim 4, characterized in that: The step of determining a plurality of high-priority nodes according to the node scores of all the pre-encoded nodes comprises: Sorting all the node scores, and selecting the first k precoding nodes in the node score ranking; k is an integer not less than 1; Each of the first k precoding nodes is regarded as a high priority node.

7. The acceleration method of the visual language navigation system according to claim 6, characterized in that: The high priority node is used as a high frequency access node, and all precoding nodes except the high frequency access node are used as low frequency access nodes; storing all the high-frequency access nodes and all the low-frequency access nodes as historical compilation nodes respectively; The step of precoding all direction nodes in the navigation path to obtain a plurality of precoded nodes comprises: Traversing all the direction nodes and all the historical compilation nodes to filter out the uncompiled nodes to be compiled; Extracting features from all the nodes to be compiled, and vectorizing the extracted features to obtain a feature vector for each node to be compiled; The historical coding node and the feature vector of each of the nodes to be coded are combined to obtain a plurality of precoding nodes.

8. An acceleration device for a visual language navigation system, characterized in that: include: An image information acquisition module, used to acquire image information of a target scene, and generate a geometric map and a semantic map of the target scene according to the image information; The semantic map includes multiple position nodes and multiple direction nodes; one position node corresponds to multiple direction nodes; A navigation path acquisition module, used to generate a navigation map based on a preset navigation target in combination with the geometric map and the semantic map, wherein the navigation map includes a navigation path, and the navigation path includes a plurality of target location nodes and direction nodes of all the target location nodes; A precoding node module, used for precoding all direction nodes in the navigation path to obtain a plurality of precoding nodes; A node score acquisition module, used to acquire the current location node of the agent, and calculate the node scores of all the pre-coded nodes based on all the target location nodes and all the pre-coded nodes; All of the target location nodes include the current location node; A navigation path pruning module, configured to determine a plurality of high priority nodes according to the node scores of all the pre-coded nodes, so as to prune the navigation path; The model collaborative decision-making module is used to input the pruned navigation path into the preset language model and generate navigation information through the preset language model.

9. An electronic device, characterized in that: include: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 7 is implemented.

Images