Navigation method and device based on multi-modal knowledge enhancement, equipment and medium

Abstract

The invention relates to the technical field of intelligent navigation, financial science and technology and medical health, and discloses a navigation method, device and equipment based on multi-modal knowledge enhancement and a medium, and the method comprises the following steps: obtaining a navigation instruction, and analyzing the navigation instruction to obtain a target constraint, a spatial constraint and an entity constraint; generating a structured target description text based on the target constraint, the spatial constraint and the entity constraint; an input image is acquired, a semantic saliency map is generated based on the input image and the structured target description text, and the semantic saliency map comprises a plurality of candidate areas; calculating a score of each candidate region of the semantic saliency map based on a preset scene knowledge graph to obtain a fusion score; and performing path planning according to the fusion score of each candidate region to obtain a target path. Navigation can be realized without navigation data training, and meanwhile, a potential target can be focused according to semantics, so that the navigation precision is improved.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent navigation, financial technology, and medical health technology, and in particular to a navigation method, device, equipment, and medium based on multimodal knowledge enhancement. Background Art

[0002] Most current navigation systems rely on navigation scene data for supervised training. Especially in target navigation tasks, it is necessary to train the model through reinforcement learning (RL), imitation learning (IL) or path replay (Replay Memory) to learn target positioning and obstacle avoidance navigation in different environments.

[0003] The inventors recognized that the aforementioned training methods require a large amount of navigation data, such as scene image data, navigation trajectory data, and target annotation data, resulting in high training costs. Furthermore, they are unable to focus on potential target areas, leading to poor navigation accuracy. For example, intelligent customer service robots in the financial technology sector and nursing robots in the healthcare sector both require a large amount of navigation data for training. Furthermore, since neither can dynamically focus on potential target areas, they can easily lead users or patients to the wrong location. Summary of the Invention

[0004] The present invention provides a navigation method, apparatus, computer equipment and medium based on multimodal knowledge enhancement to solve the technical problems of high training cost and poor navigation accuracy of existing navigation systems.

[0005] In a first aspect, a navigation method based on multimodal knowledge enhancement is provided, comprising:

[0006] Obtaining navigation instructions, and parsing the navigation instructions to obtain target constraints, spatial constraints, and entity constraints;

[0007] generating a structured target description text based on the target constraint, the space constraint, and the entity constraint;

[0008] Acquire an input image, and generate a semantic saliency map based on the input image and the structured target description text, wherein the semantic saliency map includes a plurality of candidate regions;

[0009] Calculating the score of each candidate region of the semantic saliency map based on a preset scene knowledge graph to obtain a fusion score;

[0010] Path planning is performed according to the fusion score of each candidate region to obtain a target path.

[0011] In a second aspect, a navigation device based on multimodal knowledge enhancement is provided, comprising:

[0012] An acquisition module, configured to acquire navigation instructions and parse the navigation instructions to obtain target constraints, spatial constraints, and entity constraints;

[0013] A first generating module is configured to generate a structured target description text based on the target constraint, the space constraint, and the entity constraint;

[0014] a second generating module, configured to obtain an input image and generate a semantic saliency map based on the input image and the structured target description text, wherein the semantic saliency map includes a plurality of candidate regions;

[0015] A calculation module, configured to calculate a score for each candidate region of the semantic saliency map based on a preset scene knowledge graph to obtain a fusion score;

[0016] An execution module is used to perform path planning according to the fusion score of each candidate area to obtain a target path.

[0017] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned navigation method based on multimodal knowledge enhancement when executing the computer program.

[0018] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned navigation method based on multimodal knowledge enhancement are implemented.

[0019] In the scheme implemented by the above-mentioned navigation method, device, computer equipment and storage medium based on multimodal knowledge enhancement, navigation instructions are obtained, and the navigation instructions are parsed to obtain target constraints, spatial constraints and entity constraints; structured target description text is generated based on the target constraints, the spatial constraints and the entity constraints; an input image is obtained, and a semantic saliency map is generated based on the input image and the structured target description text, wherein the semantic saliency map includes multiple candidate areas; the score of each candidate area of ​​the semantic saliency map is calculated based on the preset scene knowledge graph to obtain a fusion score; path planning is performed based on the fusion score of each candidate area to obtain the target path. In the present invention, a structured target description text can be generated according to the user's navigation instructions, and the current input image is obtained. Then, a semantic saliency map is generated based on the input image and the structured target description text, and the fusion score of the candidate area of ​​each semantic saliency map is calculated based on the preset scene knowledge graph, and the target path is obtained based on the fusion score. Finally, movement is carried out according to the target path. Navigation can be achieved without training with navigation data. At the same time, the potential target can be focused on according to the semantics to improve navigation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 is a flow chart of a navigation method based on multimodal knowledge enhancement in one embodiment of the present invention;

[0022] Figure 2 yes Figure 1 A schematic flow chart of a specific implementation of step S30;

[0023] Figure 3 1 is a schematic diagram of a first sub-process of a navigation method based on multimodal knowledge enhancement in one embodiment of the present invention;

[0024] Figure 4 yes Figure 1 A schematic flow chart of a specific implementation of step S40;

[0025] Figure 5 yes Figure 1 A schematic flow chart of a specific implementation of step S50;

[0026] Figure 6 2 is a schematic diagram of a second sub-process of a navigation method based on multimodal knowledge enhancement in one embodiment of the present invention;

[0027] Figure 7 is a structural diagram of a navigation device based on multimodal knowledge enhancement in one embodiment of the present invention;

[0028] Figure 8 is a structural diagram of a computer device in one embodiment of the present invention;

[0029] Figure 9 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] The navigation method based on multimodal knowledge enhancement provided by the embodiment of the present invention can be applied to a navigation system, and the navigation system can be configured in an intelligent device with navigation capabilities, such as an intelligent customer service robot, an intelligent nursing robot, an intelligent housekeeper robot, etc. Among them, a navigation instruction is obtained, and the navigation instruction is parsed to obtain target constraints, spatial constraints and entity constraints; a structured target description text is generated based on the target constraints, the spatial constraints and the entity constraints; an input image is obtained, and a semantic saliency map is generated based on the input image and the structured target description text, wherein the semantic saliency map includes multiple candidate areas; the score of each candidate area of ​​the semantic saliency map is calculated based on a preset scene knowledge graph to obtain a fusion score; path planning is performed according to the fusion score of each candidate area to obtain the target path, and navigation can be achieved without training with navigation data, while also improving navigation accuracy. The present invention is described in detail below through specific embodiments.

[0032] See also Figure 1 As shown, Figure 1 A flowchart of a navigation method based on multimodal knowledge enhancement provided by an embodiment of the present invention includes the following steps:

[0033] S10: Obtain navigation instructions, and parse the navigation instructions to obtain target constraints, spatial constraints, and entity constraints.

[0034] The navigation method based on multimodal knowledge enhancement provided by this invention can be applied to navigation systems. This navigation system can be applied to intelligent robots in various technical fields, such as intelligent customer service robots in fintech and intelligent nursing robots in healthcare. Intelligent customer service robots can guide customers to designated locations to handle business based on their inquiries. Intelligent nursing robots can also navigate users to designated locations based on their needs or deliver items to patients at designated locations based on their needs. For example, if a patient needs water, they can move to the kitchen to bring a cup of water.

[0035] When a user needs navigation, they can express their needs through navigation commands. These commands can be text or voice messages. For example, a user can enter "I'm a little tired and want to rest" through the interactive interface, or express the same meaning through voice to express their need for sleep. After obtaining the navigation command, it can be parsed to obtain the target constraints, spatial constraints, and entity constraints.

[0036] Goal constraints constrain navigation targets, spatial constraints constrain navigation locations, and entity constraints constrain specific objects. For example, when an intelligent nursing robot receives a patient's utterance, "I'm a little tired and would like to rest near a window," it parses the sentence to obtain the goal constraints, spatial constraints, and entity constraints. The goal constraints include resting and sleeping, the spatial constraints include bedroom, living room, and window, and the entity constraints include sofa and bed. During parsing, a pre-trained large language model can be used to perform zero-shot language parsing of text or speech to identify keywords. These keywords are then used to identify the goal constraints, spatial constraints, and entity constraints.

[0037] When obtaining target constraints, spatial constraints, and entity constraints, in addition to directly obtaining them from user input, they can also be inferred from the context. For example, when an intelligent nursing robot receives a patient's "I'm thirsty," it can directly obtain the target constraint as drinking water. Based on drinking water, it can infer the entity constraint as a water cup. Furthermore, it can infer that the water cup is often placed in the kitchen, which in turn infers the spatial constraint as the kitchen. It is understandable that each constraint can have one or multiple targets. For example, places where water cups are often placed include the kitchen and the living room, and objects for drinking water include water cups and kettles.

[0038] S20: Generate a structured target description text based on the target constraint, the space constraint, and the entity constraint.

[0039] After obtaining the target constraints, spatial constraints, and entity constraints, a structured target description can be generated based on these constraints. This structured target description is used to refine the user's navigation needs. For example, in the sentence "I'm a little tired and want to find a place by the window to rest," the target constraints are "rest" and "sleep," the spatial constraints are "bedroom," "living room," and "windowside," and the entity constraints are "sofa" and "bed." Therefore, the structured target description is {'entity constraint':'sofa', 'spatial constraint':'windowside', 'target constraint':'rest'}.

[0040] S30: Acquire an input image, and generate a semantic saliency map based on the input image and the structured target description text, wherein the semantic saliency map includes a plurality of candidate regions.

[0041] The input image refers to the image of the current environment captured by the camera module. For example, an intelligent nursing robot can capture the current environment through a camera, that is, the input image is the field of view of the intelligent nursing robot. A semantic saliency map can be generated based on the input image and the structured target description text. The semantic saliency map is used to indicate the pixel-level mapping of the area in the image that is most relevant to a specific semantic target. Its core function is to guide the system to focus on high-value visual areas by quantifying the degree of match between each pixel and the target semantics, so that the area most relevant to the structured target description text can be located in the current visual frame. The semantic saliency map can be specifically a two-dimensional matrix of the same size as the original image, with pixel values ​​∈ [0, 1]. The higher the value, the higher the match between the area and the target semantics.

[0042] The semantic saliency map can include multiple candidate regions. For example, in the sentence "I am a little tired and want to find a place by the window to rest", if the intelligent nursing robot is in the living room, the candidate regions may include sofas and stools; if the intelligent nursing robot is in the bedroom, the candidate regions may include a bed.

[0043] It is important to understand that the semantic saliency map is used to quantify the degree of match between each pixel and the target semantics, guiding the system to focus on high-value visual areas. Figure 2 As shown, a method for generating a semantic saliency map is provided. In step S30, an input image is obtained, and a semantic saliency map is generated based on the input image and the structured target description text. Specifically, the method includes the following steps:

[0044] S31: performing image encoding on the input image to obtain an image feature map, and performing text encoding on the structured target description text to obtain a semantic vector;

[0045] S32: generating an initial saliency map based on the image feature map and the semantic vector;

[0046] S33: performing semantic enhancement on the initial saliency map according to the structured target description text to obtain a plurality of saliency activation maps;

[0047] S34: performing multi-view pose alignment and temporal consistency weighted fusion on the plurality of saliency activation maps to obtain the semantic saliency map.

[0048] The input image can be encoded using a visual encoder to obtain an image feature map. For example, the ViT-B / 32 model from CLIP (Contrastive Language-Image Pre-training) can be used to extract image features and generate a 14×14×512 feature map (which retains spatial location information). The structured object description text can then be encoded using a text encoder to obtain a semantic vector. For example, the structured object description text can be input into the CLIP text encoder to generate a 512-dimensional global semantic vector. The cosine similarity between each region of the image feature map and the text vector is then calculated, and the image is upsampled to the original image size to generate an initial saliency map with pixel values ​​∈ [0, 1]. For example, in the initial heat map of the "green sofa," the green area near the window appears bright, while other areas are darker.

[0049] After obtaining the initial saliency map, semantic enhancement is performed on it based on the structured target description text to generate multiple saliency activation maps. Specifically, using GradCAM (Gradient-weighted Class Activation Mapping) technology, the contribution of each image region to the target semantics is calculated, thereby locating the most discriminative local features. This is combined with spatial structural information (such as edges and colors) from the shallow network and semantic abstraction information (such as "sofa back" and "cup handle") from the deep network to improve localization accuracy. For example, the text semantic vector is concatenated or dot-producted with the initial saliency map to guide the model to focus on image regions related to the text. Backpropagation is then used to calculate weights. Using the target semantics as the "supervisory signal," backpropagation is used to calculate the gradient weights of each neuron in the last layer of the image encoder (such as the last attention layer of a visual Transformer or the convolutional layer of a CNN). The gradient weights reflect the importance of the neuron to the target semantics; regions with higher weights are more salient in the heat map. The gradient weights are weighted and summed with the feature map to generate an activation map. This is then normalized and fused with the initial saliency map (e.g., weighted averaging) to generate a saliency activation map. Furthermore, activation maps from different network layers (such as shallow Conv3 and deep Conv5) can be aggregated to balance spatial positioning accuracy and semantic abstraction capabilities. For example, in a "green sofa" scene, GradCAM will focus on key semantic features such as the "green leather surface" and "L-shaped backrest" in the image, rather than relying solely on color information, to avoid misidentifying green curtains as targets.

[0050] Using the continuous frame images and camera pose estimation during robot movement, saliency activation maps from different viewpoints are mapped to a unified global coordinate system to eliminate false activations caused by transient occlusion or viewpoint changes. A weighted average of the heat maps from multiple frames is performed using a sliding window to retain consistently salient regions and suppress short-term noise. Specifically, SLAM (such as ORB-SLAM) or visual odometry is used to obtain the robot's pose transformation (rotation matrix R, translation vector t) between consecutive frames. The saliency activation map of each frame is mapped to the global coordinate system using the pose transformation matrix to obtain a saliency distribution in a unified space. A weighted fusion is performed on the global heat maps of the most recent N frames (e.g., N = 5), with weights based on inter-frame similarity (e.g., the smaller the pose difference between the current frame and the reference frame, the higher the weight). A threshold is set to eliminate short-term activation regions (e.g., if a region is salient only in one frame and not in the remaining frames, it is considered noise) to generate a semantic saliency map. For example, when the robot moves, the sofa may be briefly blocked by the coffee table. Temporal fusion will retain the sofa area that is consistently significant in historical frames and eliminate the false activation of the coffee table caused by occlusion in the current frame, thus ensuring the stability of the semantic saliency map.

[0051] S40: Calculate the score of each candidate region of the semantic saliency map based on the preset scene knowledge graph to obtain a fusion score.

[0052] Different preset scene knowledge graphs can be selected according to different application scenarios. VisualGenome is used for home navigation, and the UMLS medical terminology library is used for medical navigation. For example, in the home scene, the "object-room" co-occurrence relationship is extracted from VisualGenome (such as "sofa → 85% appears in the living room"). The co-occurrence relationship of "room-object" or "object-relationship" can be constructed through the preset scene knowledge graph, and then the co-occurrence probability is confirmed based on the co-occurrence relationship, such as sofa → living room, co-occurrence probability 80%, coffee table → sofa, co-occurrence probability 90%, water cup → water, co-occurrence probability 95%. Then, an environmental map is constructed, and the co-occurrence relationship is mapped to the environmental map, thereby mapping the scene-object relationship into a spatial probability distribution.

[0053] Each candidate region in the semantic saliency map is mapped to the robot's map coordinate system to obtain the spatial coordinates of each candidate region. The spatial coordinates are then combined with the spatial probability distribution to obtain a fusion score for each candidate region. For example, if the candidate region is located near the window in the living room, the probability of the query "sofa" being in that location is 0.7.

[0054] It should be understood that the preset scene knowledge graph is used to construct the spatial probability distribution to facilitate scoring of candidate regions. Figure 3 As shown, a method for generating spatial probability distribution is provided, and a navigation method based on multimodal knowledge enhancement also includes the following steps:

[0055] S40a: Obtain a preset scene knowledge graph, and convert the preset scene knowledge graph into a low-dimensional vector based on a preset graph embedding algorithm;

[0056] S40b: Mapping the low-dimensional vector to the spatial coordinate system of the current environment to generate a knowledge prior probability map.

[0057] Pre-defined graph embedding algorithms (such as TransE and GraphSAGE) can be used to convert graph structures into low-dimensional vectors and map them to the spatial coordinates of the robot's current environment to generate a knowledge prior probability map. In a living room map, based on "sofa-co-occurs-in-living room," a probability distribution of sofas in the living room is generated, with a higher probability near the window. The pre-defined scene knowledge graph can be derived from existing public datasets (such as Visual Genome and WordNet) or from a small, manually constructed knowledge base.

[0058] Among them, such as Figure 4 As shown, a method for generating a fusion score is provided. In step S40, the score of each candidate region of the semantic saliency map is calculated based on the preset scene knowledge graph to obtain a fusion score, which specifically includes the following steps:

[0059] S41: Mapping the candidate area to the spatial coordinate system of the current environment to obtain the spatial coordinates of the candidate area;

[0060] S42: confirming the prior probability corresponding to the candidate region in the knowledge prior probability map according to the spatial coordinates;

[0061] S43: Substituting the prior probability into a preset fusion score calculation formula to calculate the fusion score, wherein the preset fusion score calculation formula is:

[0062] Score(i)=λ*SemHeat(i)+(1-λ)*KnowledgePrior(i)

[0063] SemHeat(i) is the average matching heat, KnowledgePrior(i) is the prior probability, and λ is the weight parameter. Map the candidate area in the semantic saliency map to the local map coordinate system constructed by the robot, obtain its spatial coordinates (x, y), query the knowledge prior probability map based on the coordinates (x, y), and obtain the prior probability of the target corresponding to that location. If the candidate area is located near the window in the living room, query the prior probability P=0.7 of "sofa" at that location, then confirm the average matching heat in the semantic saliency map, and then substitute the prior probability and average matching heat into the preset fusion score calculation formula to calculate the fusion score. In addition, λ can be adaptively adjusted according to environmental uncertainty. For example, when the visual information is clear (such as the target is unobstructed), λ→1, it is more dependent on visual saliency. When the visual information is blurred (such as low light), λ→0, it is more dependent on the prior probability. For example:

[0064] For candidate region A (the sofa), SemHeat = 0.7, KnowledgePrior = 0.8 (because it is located near the window in the living room), and λ = 0.6, Score = 0.6 × 0.7 + 0.4 × 0.8 = 0.74;

[0065] For candidate region B (coffee table location), SemHeat=0.5, KnowledgePrior=0.6, and Score=0.6×0.5+0.4×0.6=0.54.

[0066] The calculation results show that the fusion score of candidate region A is higher than that of candidate region B.

[0067] S50: Perform path planning according to the fusion score of each candidate region to obtain a target path.

[0068] Each candidate area corresponds to a fusion score, and planning rules can be set in advance to facilitate path planning according to the planning rules. For example, a fusion threshold can be set, and candidate areas with fusion scores higher than the fusion threshold are used as path points. Then, multiple path points are connected in series according to the points with higher scores to obtain the target path, so that candidate areas with lower fusion scores can be filtered out. Alternatively, the candidate areas can be sorted according to the fusion scores, and then each candidate area is connected in sequence to obtain the target path. For example, if the fusion score of the sofa is higher than that of the bed, and the fusion score of the bed is higher than that of the stool, then the target path is to go to the sofa first, then to the bed, and finally to the stool. It is understandable that if the location of the sofa meets the user's navigation needs after arriving at the sofa, there is no need to continue navigation.

[0069] Among them, such as Figure 5As shown, a method for confirming disturbance information is provided. In step S50, path planning is performed according to the fusion score of each candidate area to obtain a target path, which specifically includes the following steps:

[0070] S51: sorting all candidate regions in descending order according to the fusion score to obtain a first sequence;

[0071] S52: Taking the first N candidate regions in the first sequence as exploration targets, and generating the target path according to the spatial position of each exploration target.

[0072] Candidate regions are sorted by their joint scores, prioritizing exploration of high-scoring regions to reduce ineffective searches. Furthermore, the candidate set can be updated based on new visual information during navigation, achieving closed-loop optimization. Specifically, all candidate regions are sorted in descending order by their fusion scores to generate an exploration priority list. The N regions with the highest scores (e.g., N = 3) are selected as the current navigation targets to avoid falling into a local optimum (e.g., considering resting areas such as sofas and beds). As the robot moves, it acquires new visual frames. If a new salient region is discovered (with a fusion score exceeding the end of the current exploration priority list), it is added to the candidate set. Regions that have not been explored multiple times without confirming the target (e.g., no target was found after three consecutive attempts) have their score weights reduced or removed. For example, if the initial candidate regions are {sofa (0.74), bed (0.68), carpet (0.45)}, the robot first navigates to the sofa. If it finds that it is occluded and unrecognized, it updates the semantic saliency map, adding "side of sofa" as a candidate region with a score of 0.76, replacing the carpet as the new candidate.

[0073] Among them, such as Figure 6 As shown, a method for confirming disturbance information is provided, and a navigation method based on multimodal knowledge enhancement also includes the following steps:

[0074] S60: If the first candidate region is reached, continuously acquiring N frames of images in the first candidate region and determining the average matching heat of each frame of images;

[0075] S61: If all the average matching heats are greater than or equal to the preset target confidence, then the navigation is completed;

[0076] S62: If there is at least one average matching heat among all the average matching heats that is less than the preset target confidence, the semantic saliency map is updated according to the current input image, and the movement path is replanned to update the target path.

[0077] When the robot reaches the first candidate region output by the visual knowledge alignment module (e.g., the region with the highest joint score), it initiates a continuous image acquisition mechanism: The robot remains relatively stationary (or moves within a small range) in that region, continuously capturing N frames of imagery (N ≥ 3, configurable, e.g., N = 5 by default) through the camera. This prevents misjudgments of single frames due to factors such as occlusion and sudden changes in illumination. If the average matching heat of all N frames is ≥ a preset target confidence level (e.g., 0.7-0.9, adaptively adjustable), the target is considered stably located and the navigation task is complete. If at least one of the N frames has an average matching heat level less than the preset target confidence level, the robot re-performs multimodal alignment and semantic localization based on the latest input image, generating a new semantic saliency map. This map focuses on potentially occluded or newly appearing target regions. Combined with the updated saliency map, the joint scores of the candidate regions are recalculated, generating a new candidate list (possibly including unexplored regions, such as the side of the sofa). The robot then moves along the new path to the next candidate region, repeating steps S60-S62 until a termination condition is met or the task time limit is reached.

[0078] It can be seen that in the above scheme, a semantic saliency map can be generated through structured target description text. The semantic saliency map includes multiple candidate areas. The fusion score of each candidate area is calculated based on the preset scene knowledge graph, and then the path is planned based on the fusion score to obtain the target path. Navigation can be achieved without training with navigation data. At the same time, it can also focus on potential targets based on semantics to improve navigation accuracy.

[0079] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0080] In one embodiment, a navigation device based on multimodal knowledge enhancement is provided. The navigation device based on multimodal knowledge enhancement corresponds one-to-one with the navigation method based on multimodal knowledge enhancement in the above embodiment. Figure 7 As shown, the navigation device based on multimodal knowledge enhancement comprises an acquisition module 101, a first generation module 102, a second generation module 103, a calculation module 104 and an execution module 105. The functional modules are described in detail as follows:

[0081] An acquisition module 101 is configured to acquire navigation instructions and parse the navigation instructions to obtain target constraints, spatial constraints, and entity constraints;

[0082] A first generating module 102 is configured to generate a structured target description text based on the target constraint, the space constraint, and the entity constraint;

[0083] A second generating module 103 is configured to obtain an input image and generate a semantic saliency map based on the input image and the structured target description text, wherein the semantic saliency map includes a plurality of candidate regions;

[0084] A calculation module 104 is configured to calculate a score for each candidate region of the semantic saliency map based on a preset scene knowledge graph to obtain a fusion score;

[0085] The execution module 105 is configured to perform path planning according to the fusion score of each candidate region to obtain a target path.

[0086] In one embodiment, the second generating module 103 is specifically configured to:

[0087] Performing image encoding on the input image to obtain an image feature map, and performing text encoding on the structured target description text to obtain a semantic vector;

[0088] generating an initial saliency map based on the image feature map and the semantic vector;

[0089] Performing semantic enhancement on the initial saliency map according to the structured target description text to obtain a plurality of saliency activation maps;

[0090] The plurality of saliency activation maps are respectively subjected to multi-view pose alignment and temporal consistency weighted fusion to obtain the semantic saliency map.

[0091] In one embodiment, the navigation device based on multimodal knowledge enhancement further includes an acquisition submodule and a mapping module.

[0092] An acquisition submodule, configured to acquire a preset scene knowledge graph and convert the preset scene knowledge graph into a low-dimensional vector based on a preset graph embedding algorithm;

[0093] A mapping module is used to map the low-dimensional vector to a spatial coordinate system of the current environment to generate a knowledge prior probability map.

[0094] In one embodiment, the calculation module 104 is specifically configured to:

[0095] Mapping the candidate area to the spatial coordinate system of the current environment to obtain the spatial coordinates of the candidate area;

[0096] Confirming the prior probability corresponding to the candidate region in the knowledge prior probability map according to the spatial coordinates;

[0097] The prior probability is substituted into a preset fusion score calculation formula to calculate the fusion score.

[0098] In one embodiment, the execution module 105 is specifically configured to:

[0099] sorting all candidate regions in descending order according to the fusion score to obtain a first sequence;

[0100] The first N candidate regions in the first sequence are used as exploration targets, and the target path is generated according to the spatial position of each exploration target.

[0101] In one embodiment, the navigation device based on multimodal knowledge enhancement further includes a first confirmation module, a second confirmation module, and a third confirmation module.

[0102] A first confirmation module is configured to, upon arriving at a first candidate region, continuously acquire N frames of images in the first candidate region and confirm an average matching heat of each frame of images;

[0103] A second confirmation module is configured to complete the navigation if all the average matching heats are greater than or equal to a preset target confidence;

[0104] The third confirmation module is used to update the semantic saliency map according to the current input image and re-plan the movement path to update the target path if there is at least one average matching heat among all the average matching heats that is less than the preset target confidence.

[0105] The present invention provides a navigation device based on multimodal knowledge enhancement. It can generate a semantic saliency map through structured target description text. The semantic saliency map includes multiple candidate areas, calculates the fusion score of each candidate area based on the preset scene knowledge graph, and then plans the path based on the fusion score to obtain the target path. Navigation can be achieved without training with navigation data. At the same time, it can also focus on potential targets according to semantics, thereby improving navigation accuracy.

[0106] For the specific limitations of the navigation device based on multimodal knowledge enhancement, please refer to the limitations of the navigation method based on multimodal knowledge enhancement above, which will not be repeated here. The various modules in the above-mentioned navigation device based on multimodal knowledge enhancement can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0107] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a navigation method based on multimodal knowledge enhancement.

[0108] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the client side of a navigation method based on multimodal knowledge enhancement.

[0109] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0110] Obtaining navigation instructions, and parsing the navigation instructions to obtain target constraints, spatial constraints, and entity constraints;

[0111] generating a structured target description text based on the target constraint, the space constraint, and the entity constraint;

[0112] Acquire an input image, and generate a semantic saliency map based on the input image and the structured target description text, wherein the semantic saliency map includes a plurality of candidate regions;

[0113] Calculating the score of each candidate region of the semantic saliency map based on a preset scene knowledge graph to obtain a fusion score;

[0114] Path planning is performed according to the fusion score of each candidate region to obtain a target path.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0116] Obtaining navigation instructions, and parsing the navigation instructions to obtain target constraints, spatial constraints, and entity constraints;

[0117] generating a structured target description text based on the target constraint, the space constraint, and the entity constraint;

[0118] Acquire an input image, and generate a semantic saliency map based on the input image and the structured target description text, wherein the semantic saliency map includes a plurality of candidate regions;

[0119] Calculating the score of each candidate region of the semantic saliency map based on a preset scene knowledge graph to obtain a fusion score;

[0120] Path planning is performed according to the fusion score of each candidate region to obtain a target path.

[0121] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0122] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0123] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0124] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A navigation method based on multimodal knowledge enhancement, characterized in that: include: Obtaining navigation instructions, and parsing the navigation instructions to obtain target constraints, spatial constraints, and entity constraints; generating a structured target description text based on the target constraint, the space constraint, and the entity constraint; Acquire an input image, and generate a semantic saliency map based on the input image and the structured target description text, wherein the semantic saliency map includes a plurality of candidate regions; Calculating the score of each candidate region of the semantic saliency map based on a preset scene knowledge graph to obtain a fusion score; Path planning is performed according to the fusion score of each candidate region to obtain a target path.

2. The method according to claim 1, wherein Generating a semantic saliency map based on the input image and the structured target description text includes: Performing image encoding on the input image to obtain an image feature map, and performing text encoding on the structured target description text to obtain a semantic vector; generating an initial saliency map based on the image feature map and the semantic vector; Performing semantic enhancement on the initial saliency map according to the structured target description text to obtain a plurality of saliency activation maps; The plurality of saliency activation maps are respectively subjected to multi-view pose alignment and temporal consistency weighted fusion to obtain the semantic saliency map.

3. The method according to claim 1, wherein The method comprises: Obtain a preset scene knowledge graph, and convert the preset scene knowledge graph into a low-dimensional vector based on a preset graph embedding algorithm; The low-dimensional vector is mapped to the spatial coordinate system of the current environment to generate a knowledge prior probability map.

4. The method according to claim 3, wherein The step of calculating a score for each candidate region of the semantic saliency map based on a preset scene knowledge graph to obtain a fusion score includes: Mapping the candidate area to the spatial coordinate system of the current environment to obtain the spatial coordinates of the candidate area; Confirming the prior probability corresponding to the candidate region in the knowledge prior probability map according to the spatial coordinates; The prior probability is substituted into a preset fusion score calculation formula to calculate the fusion score.

5. The method according to claim 4, wherein The preset fusion score calculation formula is: Score(i)=λ*SemHeat(i)+(1-λ)*KnowledgePrior(i) Among them, SemHeat(i) is the average matching heat, KnowledgePrior(i) is the prior probability, and λ is the weight parameter.

6. The method according to claim 1, wherein The performing path planning according to the fusion score of each candidate region to obtain a target path includes: sorting all candidate regions in descending order according to the fusion score to obtain a first sequence; The first N candidate regions in the first sequence are used as exploration targets, and the target path is generated according to the spatial position of each exploration target.

7. The method according to claim 1, wherein The method comprises: If the first candidate area is reached, N frames of images are continuously acquired in the first candidate area, and the average matching heat of each frame of image is determined; If all the average matching heats are greater than or equal to the preset target confidence, the navigation is completed; If at least one of the average matching heats among all the average matching heats is less than the preset target confidence, the semantic saliency map is updated according to the current input image, and the movement path is replanned to update the target path.

8. A navigation device based on multimodal knowledge enhancement, characterized in that: include: An acquisition module, configured to acquire navigation instructions and parse the navigation instructions to obtain target constraints, spatial constraints, and entity constraints; A first generating module is configured to generate a structured target description text based on the target constraint, the space constraint, and the entity constraint; a second generating module, configured to obtain an input image and generate a semantic saliency map based on the input image and the structured target description text, wherein the semantic saliency map includes a plurality of candidate regions; A calculation module, configured to calculate a score for each candidate region of the semantic saliency map based on a preset scene knowledge graph to obtain a fusion score; An execution module is used to perform path planning according to the fusion score of each candidate area to obtain a target path.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the navigation method based on multimodal knowledge enhancement as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the navigation method based on multimodal knowledge enhancement as claimed in any one of claims 1 to 7 are implemented.

Images