Long-short term memory target navigation method and device based on hierarchical graph, equipment and storage medium

Abstract

The invention relates to the technical field of visual language navigation, in particular to a long-short memory target navigation method and device based on a hierarchical graph, equipment and a storage medium. Comprising the following steps: generating logic chains according to a preset navigation target and a preset grading strategy, scoring each logic chain, and inserting the logic chains into a preset logic chain queue according to a high-low sequence of scores; taking the first logic chain in the preset logic chain queue as a first logic chain, obtaining a current panoramic image, and constructing a hierarchical map according to the first logic chain and the panoramic image; determining navigation sub-targets according to the first logic chain and the grading atlas; obtaining a navigation result to adjust the first logic chain, and updating the grading atlas according to the adjusted first logic chain; and returning to execute the step of determining the navigation sub-targets according to the first logic chain and the grading map until the navigation task of the preset navigation target is completed. The method can reduce the blindness of exploration, thereby achieving the beneficial effects of reducing the energy consumption of exploration and shortening the time delay of exploration.

Description

Technical Field

[0001] The present application relates to the technical field of visual language navigation, and in particular to a method, device, equipment and storage medium for navigating long-short memory targets based on a hierarchical graph. Background Art

[0002] In recent years, the rapid development of computer vision and natural language processing technologies has made up for the lack of semantic understanding and reasoning ability of robots without prior maps, making navigation in open vocabulary scenarios possible, and thus giving rise to an emerging task, namely visual language navigation (VLN) in open scenarios. This cross-modal task requires the integration of visual perception and natural language understanding, enabling agents to understand the surrounding environment in unfamiliar scenarios and navigate to the destination according to user instructions.

[0003] At present, methods based on reinforcement learning and imitation learning are limited to solving VLN tasks on a single or a set of specific data sets. For navigation in cross-dataset scenarios, their accuracy is difficult to meet actual needs. On this basis, the navigation strategy in open scenarios based on large language models (LLM) and VLM can solve the problem of scene migration, but the large language model is constantly called to explore the environment during the process, resulting in a certain degree of blindness in the exploration, which also results in higher energy consumption and longer latency.

[0004] Based on this, how to reduce the blindness of exploration, thereby reducing exploration energy consumption and shortening exploration delay, is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present application provides a long-short memory target navigation method, device, equipment and storage medium based on a hierarchical graph to reduce the blindness of exploration, thereby achieving the beneficial effects of reducing exploration energy consumption and shortening exploration delay.

[0006] The technical solution adopted by this application to solve its technical problem is: In a first aspect, the present application provides a long-short memory target navigation method based on a hierarchical graph, which is applied to an agent, and the method comprises: Generate at least one logic chain according to the preset navigation target and the preset classification strategy, and score each of the logic chains, and insert all the logic chains into the preset logic chain queue in order of the score of each logic chain; The first logical chain in the preset logical chain queue is used as a first logical chain, and a first panoramic image of the current scene is acquired, so as to construct a hierarchical map according to the first logical chain and the first panoramic image; Determine and guide the agent to the next navigation sub-goal according to the first logic chain and the hierarchical map; Obtaining a navigation result of the current navigation, adjusting the first logical chain according to the navigation result, and updating the hierarchical map according to the adjusted first logical chain; Return to the step of determining and guiding the agent to the next navigation sub-goal according to the first logical chain and the hierarchical map, until the navigation task for the preset navigation goal is completed.

[0007] Optionally, the step of acquiring a first panoramic image of the current scene to construct a hierarchical map according to the first logical chain and the first panoramic image includes: Acquire the first panoramic image and structure prompt information; Inputting the structure prompt information and the first panoramic image into a first preset model, parsing the first panoramic image through the first preset model, and converting the parsing result into a plurality of structured texts according to the structure prompt information; Acquire integration prompt information, input the integration prompt information and all the structured texts into a second preset model, so as to integrate all the structured texts according to the integration prompt information through the second preset model to obtain a hierarchical graph update statement; The hierarchical graph is constructed based on the hierarchical graph update statement and the first logical chain.

[0008] Optionally, the structured text includes room type information, location information, angle information and object description information; the step of integrating all the structured texts to obtain a hierarchical graph update statement includes: Acquire multiple room type sets, and add all structured texts with the same room type information into the same room type set; The position information, angle information and object description information of all the structured texts in each of the room type sets are summarized to obtain at least one of the hierarchical graph update statements.

[0009] Optionally, the step of scoring each of the logic chains includes: Get scoring tips; All the logic chains and the scoring prompt information are input into a second preset model, so that each logic chain is scored according to the scoring prompt information by the second preset model to obtain a score for each logic chain.

[0010] Optionally, the step of adjusting the first logical chain according to the navigation result and updating the hierarchical map according to the adjusted first logical chain includes: Determine whether the first logical chain is valid according to the navigation result; if the first logical chain is valid, update the first logical chain according to the navigation result; otherwise, move the first logical chain to a preset eliminated logical chain queue to update the preset logical chain queue; The first logical chain in the current preset logical chain queue is used as the first logical chain, and the hierarchical map is updated according to the first logical chain.

[0011] Optionally, the step of obtaining the navigation result of the current navigation includes: Obtain a second panoramic image and structural hint information of the new scene; Inputting the structure prompt information and the second panoramic image into a first preset model, parsing the second panoramic image through the first preset model, and converting the parsing result into a plurality of structured texts according to the structure prompt information; The integrated prompt information is obtained, and the integrated prompt information and all the structured texts are input into a second preset model, so that all the structured texts are integrated according to the integrated prompt information through the second preset model to obtain a hierarchical graph update statement for the new scene.

[0012] Optionally, the step of judging whether the first logical chain is valid according to the navigation result includes: Determining whether the navigation sub-goal exists in the hierarchical graph update statement of the new scene; If the navigation sub-target exists, it is determined that the first logical chain is valid; otherwise, it is determined that the first logical chain is invalid.

[0013] In a second aspect, the present application provides a long-short memory target navigation device based on a hierarchical graph, comprising: A logic chain determination module, used to generate at least one logic chain according to a preset navigation target and a preset classification strategy, and score each of the logic chains, and insert all the logic chains into a preset logic chain queue in order of the score of each logic chain; A hierarchical map construction module, used to take the first logical chain in the preset logical chain queue as the first logical chain, and obtain a first panoramic image of the current scene, so as to construct a hierarchical map according to the first logical chain and the first panoramic image; A sub-goal determination module, used to determine and guide the agent to the next navigation sub-goal according to the first logical chain and the hierarchical map; A dynamic adjustment module, used for obtaining a navigation result of the current navigation, adjusting the first logical chain according to the navigation result, and updating the hierarchical map according to the adjusted first logical chain; The continuous navigation module is used to return to the step of determining and guiding the agent to the next navigation sub-goal according to the first logical chain and the hierarchical map until the navigation task for the preset navigation goal is completed.

[0014] In a third aspect, the present application provides an electronic device, including: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions, which, when executed by the one or more processors, enable the electronic device to perform the above method.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, in which a program or instruction is stored. When the program or instruction is executed, the above method is implemented.

[0016] The beneficial effects of the present application are as follows: the present application generates multiple logic chains based on preset navigation targets and preset hierarchical strategies. Taking indoor navigation as an example, four hierarchies may be included: house level, room level, large object level and small object level; according to the hierarchical division, the inclusion relationship and connectivity relationship between the hierarchies may also be defined; then, scoring is performed based on the generated logic chains, and the scoring criteria may be the most logical and most likely to find the corresponding object logic chain, and after the scoring is completed, the sorting results are given in descending order; the logic chain with the highest score is selected as the first logic chain, and a panoramic image of the current scene is obtained, and a hierarchical map is constructed based on the panoramic image and the first logic chain, and then the next navigation sub-target to be headed to by the agent is determined based on the hierarchical map and the first logic chain based on the node at the highest level of the first logic chain, and the agent is guided to go there; After reaching the navigation sub-goal, the navigation result is fed back, and the priority of the logic chain is dynamically adjusted according to the navigation result. If the current first logic chain is determined to be invalid according to the navigation result, it will be eliminated, and the logic chain that was originally ranked second will be selected as the new first logic chain. Then, the hierarchical map is updated according to the changes in the logic chain and the navigation results; return to the step of determining the next navigation sub-goal based on the first logic chain and the hierarchical map until the preset navigation goal is found.

[0017] By adopting the above technical solution and the logic chain strategy, the validity of each logic chain is tried one by one according to the order of the logic chain from high to low, so that the navigation process is more purposeful, and the purpose of reducing the number of reasoning is achieved, thereby reducing the cost of reasoning. In addition, a hierarchical map that can record history is introduced in the navigation task, and subsequent navigation can refer to previous historical information, making the navigation process more reasonable, forming a navigation method that combines the short-term memory of the logic chain with the long-term memory of the hierarchical map. The short-term memory is responsible for inference, and the long-term history memory is responsible for feedback. In the navigation using each logic chain, each time a new scene is reached, the validity of the logic chain is verified, so as to stop the loss in time and avoid subsequent meaningless searches, further reducing the number of reasoning, reducing the cost of reasoning, and improving the efficiency of reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a long-short memory target navigation method based on a hierarchical graph provided in an embodiment of the present application; Figure 2 It is a schematic diagram of the framework of the long-short memory target navigation method based on the hierarchical graph provided in the embodiment of the present application; Figure 3 It is a schematic diagram of the long-short memory mechanism adopted by the long-short memory target navigation method based on the hierarchical graph provided in the embodiment of the present application; Figure 4 It is a virtual structural schematic diagram of a long-short memory target navigation device based on a hierarchical graph provided by the present application; Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The present application is further described below in conjunction with the accompanying drawings and embodiments.

[0020] The following will clearly and completely describe the concept, specific structure and technical effects of the present application in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments of the present application, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present application. In addition, all the connection / connection relationships involved in the patent do not simply refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features created in this application can be combined interactively without conflicting with each other.

[0021] Definitions of terms used in this application: VLN: The full English name is Visual language navigation, which means visual language navigation in Chinese; LLM: The full name of the English name is Large language model, which means large language model in Chinese; VLM: The full English name is Visual language model, and its Chinese meaning is Visual Language Large Model.

[0022] Reference Figure 1 , Figure 1 It is a flow chart of a long-short memory target navigation method based on a hierarchical graph provided in an embodiment of the present application, which specifically includes the following steps, which are specifically described below: In step S1, at least one logic chain is generated according to a preset navigation target and a preset classification strategy, and each of the logic chains is scored, and all the logic chains are inserted into a preset logic chain queue in order of the score of each logic chain.

[0023] Specifically, the preset navigation target is the final navigation destination in this navigation. In the embodiment of the present application, taking the indoor navigation scene as an example, the navigation target can be divided into four levels: house level, room level, large object level, and small object level. Among them, the house level is the highest level of the overall environment; the room level is the middle level of the environment, representing a specific room, such as a bedroom or bathroom; the large object level is the detail level of the environment, representing large objects in the room, such as a dining table and a bed; the small object level is the smallest level of the environment, representing small objects in the room, and can also be objects hidden in the space, such as fruits and cups. According to the above-mentioned level division, it is also necessary to define the inclusion relationship and connectivity relationship between each level. In the embodiment of the present application, the house level includes the relationship and connectivity between rooms; the room level includes large objects and small objects in the room; the large object level includes small objects, or it can only include itself.

[0024] Furthermore, the step of scoring each of the logic chains comprises: Get scoring tips; All the logic chains and the scoring prompt information are input into a second preset model, so that each logic chain is scored according to the scoring prompt information by the second preset model to obtain a score for each logic chain.

[0025] Specifically, the second preset model is LLM, and the scoring prompt information is a prompt word that guides LLM to generate a score for each logical chain. Specifically, the above preset navigation target and preset definition are input into LLM, and this step is completed in combination with the prompt words that can guide LLM to generate multiple candidate logical chains, wherein the prompt words can be variable. In the embodiment of the present application, the prompt words can be "Based on the four-level hierarchical strategy (house level, room level, large object level, small object level), design multiple candidate logical chains to reach the target object "cup". Each link should contain house-level and room-level descriptions, large objects and small objects along the way". Based on the above content, the following example logical chain can be generated: Candidate logic chain 1: living room - dining table - cup; Candidate logic chain 2: kitchen - cupboard - cup; Candidate logic chain 3: bedroom - desk - cup; Candidate logic chain 4: office - coffee table - cup.

[0026] Furthermore, appropriate guiding words can be added to each generated logical chain to further score the feasibility of each logical chain through LLM. The prompt words can be added as "Properly score each candidate logical chain, and the score takes into account logic and directness." to form the following scoring example: Candidate logic chain 1: living room - kitchen - dining table - cup; Rating: 6 / 10. The path is simple and in line with the general living environment layout.

[0027] Candidate logic chain 2: living room - kitchen - cupboard - cup; Rating: 5 / 10. The paths make sense and are consistent with common storage locations for items.

[0028] Candidate logic chain 3: living room - bedroom - desk - cup; Rating: 4 / 10. Logical but not common enough.

[0029] Candidate logic chain 4: living room - office - coffee table - cup; Rating: 5 / 10. Reasonable candidate chain, there are often cups on the coffee table.

[0030] Furthermore, the scored logical chains are arranged in descending order of the scores. In the embodiment of the present application, a preset logical chain queue is set, and the multiple candidate logical chains generated for the first time are inserted into the preset logical chain queue in order according to the scores, and the candidate logical chain ranked first in the preset logical chain queue is the highest-scoring candidate logical chain.

[0031] In step S2, the first logical chain in the preset logical chain queue is used as the first logical chain, and a first panoramic image of the current scene is obtained to construct a hierarchical map according to the first logical chain and the first panoramic image.

[0032] Specifically, the logical chain ranked first in the preset logical column queue is the logical chain with the highest score among all logical chains in the queue. In this step, it is selected as the logical chain used for the current reasoning, that is, the first logical chain. Then, the first panoramic image of the current scene is obtained. The difference between the first panoramic image and the second panoramic image described below is that the first panoramic image is a panoramic image obtained under the current scene, and the second panoramic image is a panoramic image obtained after the current scene is transformed into the next scene. For the reasoning of the next scene, the second panoramic image will be used as the new first panoramic image.

[0033] More specifically, for each scene in which it is located, the agent (including but not limited to a robot, a navigation car, etc.) captures a panoramic image in the scene through a series of rotations through its built-in environment perception module. Depending on the actual parameters used, four or six images with different perspectives can be taken for a 360-degree rotation, or images taken for a panoramic camera.

[0034] Furthermore, a hierarchical graph is constructed corresponding to the first logical chain based on the panoramic image. For example, assuming that the first logical chain is living room-kitchen-dining table-cup, four virtual nodes including the living room, kitchen, dining table, and cup are constructed, and according to the virtual relationship between them, a virtual edge of an adjacent relationship is added to the living room and the kitchen, a virtual edge of an inclusion relationship is added to the kitchen and the dining table, and a virtual edge of an inclusion relationship is added to the dining table and the cup.

[0035] In an embodiment of the present application, the step of acquiring a first panoramic image of the current scene to construct a hierarchical map according to the first logical chain and the first panoramic image specifically includes the following steps: The first panoramic image and structure prompt information are obtained.

[0036] Specifically, the structural prompt information is a prompt word used to guide the first preset model (i.e., VLM) used below to generate structured text in combination with the first panoramic image. In an embodiment of the present application, the structural prompt information can be "describe all the rooms in the picture, and describe all the objects with details in each room, and convert each image into a structured text representation."

[0037] Furthermore, the structural prompt information and the first panoramic image are input into a first preset model, so as to parse the first panoramic image through the first preset model, and convert the parsing result into a plurality of structured texts according to the structural prompt information.

[0038] Specifically, the first preset model is the environment perception tool VLM. In an embodiment of the present application, the structural prompt information and the first panoramic image are input into the first preset model together. The first preset model converts each image into a structured text representation, that is, multiple structured texts are obtained. The structured text can be: {Type of room: z, position: x, angle: y, object description: ...}, wherein z represents a room, such as a bedroom, bathroom, etc.; x is the three-dimensional space coordinate of the agent; y is the angle corresponding to different rotation parameters; k represents the description of a specific object, such as a bed with a pillow on it, a desk with several books on it, etc.

[0039] Furthermore, integration prompt information is obtained, and the integration prompt information and all the structured texts are input into a second preset model, so that all the structured texts are integrated according to the integration prompt information through the second preset model to obtain a hierarchical graph update statement.

[0040] Specifically, the integrated prompt information is used to guide the second preset model (i.e., LLM) to summarize the structured text to obtain prompt words for the hierarchical graph update sentence. The integrated prompt information and all the structured texts are input into the second preset model. The second preset model will classify the text information in all the structured texts to obtain sentences that can be used to update the following hierarchical graph.

[0041] More specifically, the step of integrating all the structured texts to obtain a hierarchical graph update statement specifically includes the following steps: Acquire multiple room type sets, and add all structured texts with the same room type information into the same room type set.

[0042] Specifically, in combination with the above steps, it can be known that the structured text includes room type information, location information, angle information and object description information. In the embodiment of the present application, classification is performed according to the room type information. For example, the room type information includes bedrooms and bathrooms. Then, all structured texts whose room type information is bedrooms can be added to a set (defined as a bedroom information set), and all structured texts whose room type information is bathrooms can be added to another set (defined as a bathroom information set). After all structured texts are classified, proceed to the following steps.

[0043] Furthermore, the position information, angle information and object description information of all the structured texts in each of the room type sets are summarized to obtain at least one of the hierarchical graph update statements.

[0044] Specifically, in the above steps, the structured texts with the same room type information have been classified to obtain multiple room type sets. In this step, the position information, angle information and object description information of the structured texts in the same group are summarized, wherein the object association relationship is supplemented for the object description information of each structured text. For example, two structured texts with the room type of z are merged {type room: z, location: ,angle: , object description: , , …} and {type room: z, location: ,angle: , object description: , , …}, the merged hierarchical map update statement is {Room type: z, location: ,angle: , object relationship: , , …}, where s is the relationship between the large object and the small objects it contains after merging, or the large object and the small objects themselves, such as a bed (pillow), a desk (book), a potted plant, a chair, etc.

[0045] Furthermore, the hierarchical graph is constructed based on the hierarchical graph update statement and the first logical chain. The construction of the hierarchical graph has been described in the above steps, that is, multiple virtual nodes are generated according to the information contained in the hierarchical graph update statement and the first logical chain, and virtual edges are constructed, which will not be repeated here.

[0046] In step S3, the agent is determined and guided to the next navigation sub-goal according to the first logical chain and the hierarchical map.

[0047] Specifically, refer to Figure 2 , Figure 2 is a schematic diagram of a framework of a long-short memory target navigation method based on a hierarchical graph provided in an embodiment of the present application, Figure 2 The process of finally obtaining the next navigation sub-goal (i.e., the target navigation in the figure) through collaborative reasoning of the current hierarchical graph and the adopted logic chain (i.e., the first logic chain) is shown. Specifically, in the navigation task corresponding to the first logic chain, the step-by-step strategy is: go to sub-goal 1, go to sub-goal 2, and so on to reach the final goal. According to the highest level node of the currently ranked highest logic chain and the current hierarchical graph, they are input into the LLM in the form of text, allowing it to reason and judge the next sub-goal that should be visited (the next navigation sub-goal), and use the current short-term memory to screen the most appropriate sub-goal.

[0048] In step S4, the navigation result of the current navigation is obtained, the first logical chain is adjusted according to the navigation result, and the hierarchical map is updated according to the adjusted first logical chain.

[0049] Specifically, first obtain the navigation result corresponding to the current navigation, and the navigation result includes finding the corresponding object and not finding the corresponding object, so as to evaluate the effectiveness of the first logical chain currently used. In the embodiment of the present application, each time the next sub-goal is inferred, the LLM is allowed to calculate the evaluation score of the current position and the target at the same time. If the corresponding object of the corresponding node is found, the effectiveness score is gradually improved. When this score gets higher and higher, it proves to be effective. On the contrary, if the object of the corresponding node is not found, it is invalid, so as to obtain the navigation result. It is worth noting that in order to record the navigation process, during the exploration process, once the corresponding navigation sub-goal is found in this reasoning, the status of the virtual link node in the logical chain node is updated, and this annotation is also included in the navigation result to facilitate the subsequent formation of long memory feedback.

[0050] Further, after obtaining the feedback navigation result, the priority of the first logical chain is dynamically adjusted according to the navigation result. In the embodiment of the present application, the step of adjusting the first logical chain according to the navigation result and updating the hierarchical map according to the adjusted first logical chain specifically includes the following steps: Determine whether the first logical chain is valid according to the navigation result; if the first logical chain is valid, update the first logical chain according to the navigation result; otherwise, move the first logical chain to a preset eliminated logical chain queue to update the preset logical chain queue.

[0051] Specifically, in addition to the preset logic column queue, a preset elimination logic chain queue is also set up to store logic chains that have been used and proved invalid. After verifying the validity of the first logic chain in the above manner, if the evaluation result is that the current first logic chain is valid, then along the first logic chain, only the object of the next node needs to be searched; if the evaluation result is that the current first logic chain is valid, the current first logic chain is moved to the preset elimination logic chain queue. Obviously, the original sorting is also adaptively adjusted, and the logic chain that was originally ranked second in the preset logic chain queue becomes the new first-ranked logic chain.

[0052] Furthermore, the first-ranked logical chain in the current preset logical chain queue is used as the first logical chain, and the hierarchical map is updated according to the first logical chain.

[0053] Specifically, the first logical chain is always the first logical chain in the current preset logical chain queue. After the original first logical chain is placed in the preset elimination logical chain queue, the new first logical chain is adjusted in order. Based on the new first logical chain, the hierarchical graph is also updated accordingly, that is, the above steps of generating virtual nodes and virtual edges are repeated.

[0054] For example, refer to Figure 3 , Figure 3 It is a schematic diagram of the long-short memory mechanism adopted by the long-short memory target navigation method based on the hierarchical graph provided in the embodiment of the present application. It can be seen that the upper side of the left figure includes the first logic chain and other spare logic chains in the preset logic chain queue. The nodes in each logic chain are divided into "actual nodes", "current search nodes" and "virtual nodes". The actual nodes are nodes that have been searched and the corresponding objects are found, so that "large object 2" is used as the next search node along the level of the first logic chain, and the virtual node is the node that has not yet confirmed whether the corresponding object actually exists. Based on the above-mentioned node status markings, a hierarchical graph is formed on the lower side of the left figure, in which "large object 5" and "large object 6" refer to the objects searched in "room 1", indicating that the first logic chain has not found the "large object 2" that should be found. In this case, the current first logic chain fails and the logic chain is switched.

[0055] Reference Figure 3 As shown in the right figure, the first logic chain in the left figure is eliminated as the invalid logic chain, and the backup logic chain is enabled as the new first logic chain. According to the new first logic chain, "Room 2" is searched and the "large object 4" is found therein, and the hierarchical atlas is updated based on the found object.

[0056] More specifically, the step of obtaining the navigation result of the current navigation includes the following steps: Obtain a second panoramic image and structural hint information of the new scene; Inputting the structure prompt information and the second panoramic image into a first preset model, parsing the second panoramic image through the first preset model, and converting the parsing result into a plurality of structured texts according to the structure prompt information; The integrated prompt information is obtained, and the integrated prompt information and all the structured texts are input into a second preset model, so that all the structured texts are integrated according to the integrated prompt information through the second preset model to obtain a hierarchical graph update statement for the new scene.

[0057] Specifically, similar to the above steps of combining the first panoramic image and integrating the prompt information, that is, when the corresponding search node is reached and the scene is converted, the above steps are re-executed. Based on this step, the agent continuously generates new hierarchical graph update statements when moving. The new hierarchical graph update statements are used to continuously update the hierarchical graph, and based on the new hierarchical graph update statements, it can be determined whether the navigated sub-target exists in the scene. If not, it is determined that the current first logical chain is invalid. That is: Determining whether the navigation sub-goal exists in the hierarchical graph update statement of the new scene; If the navigation sub-target exists, it is determined that the first logical chain is valid; otherwise, it is determined that the first logical chain is invalid.

[0058] In step S5, the process returns to the step of determining and guiding the agent to the next navigation sub-goal according to the first logical chain and the hierarchical map, until the navigation task for the preset navigation goal is completed.

[0059] Specifically, after dynamically updating the hierarchical map and the first logical chain, return to step S3, and iterate steps S3 to S5 until a logical chain with a preset navigation target is finally found, thus completing the navigation task.

[0060] Reference Figure 4 , Figure 4 It is a virtual structural schematic diagram of a long-short memory target navigation device based on a hierarchical graph provided by the present application. In a second aspect, the present application provides a long-short memory target navigation device based on a hierarchical graph, comprising: The logic chain determination module 100 is used to generate at least one logic chain according to the preset navigation target and the preset classification strategy, and score each of the logic chains, and insert all the logic chains into the preset logic chain queue in order of the score of each logic chain; A hierarchical map construction module 200 is used to use the first logical chain in the preset logical chain queue as a first logical chain, and obtain a first panoramic image of the current scene to construct a hierarchical map according to the first logical chain and the first panoramic image; A sub-goal determination module 300, for determining and guiding the agent to the next navigation sub-goal according to the first logical chain and the hierarchical map; The dynamic adjustment module 400 is used to obtain the navigation result of the current navigation, adjust the first logical chain according to the navigation result, and update the hierarchical map according to the adjusted first logical chain; The continuous navigation module 500 is used to return to the step of determining and guiding the agent to the next navigation sub-goal according to the first logical chain and the hierarchical map until the navigation task for the preset navigation goal is completed.

[0061] The long-short memory target navigation device based on a hierarchical graph described in the embodiment of the present application can execute the long-short memory target navigation method based on a hierarchical graph provided in the above embodiment. The long-short memory target navigation device based on a hierarchical graph has the corresponding functional steps and beneficial effects of the long-short memory target navigation method based on a hierarchical graph described in the above embodiment. Please refer to the embodiment of the long-short memory target navigation method based on a hierarchical graph for details. The embodiment of the present application will not be repeated here.

[0062] The present application also provides an electronic device. Figure 5 , Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include a processor and a memory, wherein the processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various types of chips. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the long-short memory target navigation method based on the hierarchical graph in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the long-short memory target navigation method based on the hierarchical graph in the above method embodiment is realized.

[0063] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. The one or more modules are stored in the memory, and when executed by the processor, the long-short memory target navigation method based on the hierarchical graph in the above-mentioned method embodiment is executed. The specific details of the above-mentioned electronic device can be understood by corresponding to the corresponding descriptions and effects in the above-mentioned method embodiment, and will not be repeated here. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0064] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0065] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims, the inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present application.

[0066] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims.

Claims

1. A hierarchical graph-based long-short memory target navigation method, applied to agents, characterized in that: The method comprises: Generate at least one logic chain according to the preset navigation target and the preset classification strategy, and score each of the logic chains, and insert all the logic chains into the preset logic chain queue in order of the score of each logic chain; The first logical chain in the preset logical chain queue is used as a first logical chain, and a first panoramic image of the current scene is acquired, so as to construct a hierarchical map according to the first logical chain and the first panoramic image; Determine and guide the agent to the next navigation sub-goal according to the first logic chain and the hierarchical map; Obtaining a navigation result of the current navigation, adjusting the first logical chain according to the navigation result, and updating the hierarchical map according to the adjusted first logical chain; Return to the step of determining and guiding the agent to the next navigation sub-goal according to the first logical chain and the hierarchical map, until the navigation task for the preset navigation goal is completed.

2. The long-short memory target navigation method based on hierarchical graph according to claim 1 is characterized in that: The step of acquiring a first panoramic image of the current scene to construct a hierarchical map according to the first logical chain and the first panoramic image includes: Acquire the first panoramic image and structure prompt information; Inputting the structure prompt information and the first panoramic image into a first preset model, parsing the first panoramic image through the first preset model, and converting the parsing result into a plurality of structured texts according to the structure prompt information; Acquire integration prompt information, input the integration prompt information and all the structured texts into a second preset model, so as to integrate all the structured texts according to the integration prompt information through the second preset model to obtain a hierarchical graph update statement; The hierarchical graph is constructed based on the hierarchical graph update statement and the first logical chain.

3. The long-short memory target navigation method based on hierarchical graph according to claim 2 is characterized in that: The structured text includes room type information, location information, angle information and object description information; The step of integrating all the structured texts to obtain a hierarchical graph update statement includes: Acquire multiple room type sets, and add all structured texts with the same room type information into the same room type set; The position information, angle information and object description information of all the structured texts in each of the room type sets are summarized to obtain at least one of the hierarchical graph update statements.

4. The long-short memory target navigation method based on hierarchical graph according to claim 1 is characterized in that: The step of scoring each of the logic chains comprises: Get scoring tips; All the logic chains and the scoring prompt information are input into a second preset model, so that each logic chain is scored according to the scoring prompt information by the second preset model to obtain a score for each logic chain.

5. The long-short memory target navigation method based on hierarchical graph according to claim 1 is characterized in that: The step of adjusting the first logical chain according to the navigation result and updating the hierarchical map according to the adjusted first logical chain includes: Determine whether the first logical chain is valid according to the navigation result; if the first logical chain is valid, update the first logical chain according to the navigation result; otherwise, move the first logical chain to a preset eliminated logical chain queue to update the preset logical chain queue; The first logical chain in the current preset logical chain queue is used as the first logical chain, and the hierarchical map is updated according to the first logical chain.

6. The long-short memory target navigation method based on hierarchical graph according to claim 5 is characterized in that: The step of obtaining the navigation result of the current navigation comprises: Obtain a second panoramic image and structural hint information of the new scene; Inputting the structure prompt information and the second panoramic image into a first preset model, parsing the second panoramic image through the first preset model, and converting the parsing result into a plurality of structured texts according to the structure prompt information; The integrated prompt information is obtained, and the integrated prompt information and all the structured texts are input into a second preset model, so that all the structured texts are integrated according to the integrated prompt information through the second preset model to obtain a hierarchical graph update statement for the new scene.

7. The long-short memory target navigation method based on hierarchical graph according to claim 6 is characterized in that: The step of judging whether the first logical chain is valid according to the navigation result comprises: Determining whether the navigation sub-goal exists in the hierarchical graph update statement of the new scene; If the navigation sub-target exists, it is determined that the first logical chain is valid; otherwise, it is determined that the first logical chain is invalid.

8. A long-short memory target navigation device based on a hierarchical graph, characterized in that: include: A logic chain determination module, used to generate at least one logic chain according to a preset navigation target and a preset classification strategy, and score each of the logic chains, and insert all the logic chains into a preset logic chain queue in order of the score of each logic chain; A hierarchical map construction module, used to take the first logical chain in the preset logical chain queue as the first logical chain, and obtain a first panoramic image of the current scene, so as to construct a hierarchical map according to the first logical chain and the first panoramic image; A sub-goal determination module, used to determine and guide the agent to the next navigation sub-goal according to the first logical chain and the hierarchical map; A dynamic adjustment module, used for obtaining a navigation result of the current navigation, adjusting the first logical chain according to the navigation result, and updating the hierarchical map according to the adjusted first logical chain; The continuous navigation module is used to return to the step of determining and guiding the agent to the next navigation sub-goal according to the first logical chain and the hierarchical map until the navigation task for the preset navigation goal is completed.

9. An electronic device, characterized in that: include: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 7 is implemented.

Images