Navigation method and device in unexplored environment, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the field of medical health, and discloses a navigation method, device and equipment in an unexplored environment and a medium. Analyzing a pre-trained large language model into a structured semantic representation at least comprising target entity information, spatial constraint information and contextual intention information; inputting the target entity into a pre-trained large language model to generate a standardized task description question, inputting the standardized task description question and real-time environment image data into a pre-trained visual language question and answer model, and outputting a semantic answer result about whether the target entity exists or not; and if yes, generating a navigation strategy through a pre-trained large language model based on the structured semantic representation, and controlling the robot to execute a navigation action through a preset path planning algorithm. The method does not need to depend on an environmental map and scene training, realizes navigation in an unexplored environment by utilizing cooperation of a large language model and a visual language question and answer model, solves the problem of strong dependency of a traditional scheme on prior data, and has good generalization ability and environmental adaptability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and can be applied to the field of medical and health. In particular, it relates to a navigation method, device, equipment and medium in an unexplored environment. Background Art

[0002] In medical scenarios, navigation robots are widely used in tasks such as medicine delivery, patient guidance, and transportation of medical equipment. However, traditional navigation systems highly rely on pre-constructed environmental maps and training data for specific scenarios, and have significant limitations. For example, in a dynamic and complex environment such as a hospital, the ward layout is frequently adjusted, medical equipment is temporarily added, or sudden personnel flow often causes the pre-stored map to become invalid. In addition, traditional methods need to be trained for a long time through techniques such as reinforcement learning or behavior cloning to meet the navigation requirements of specific scenarios, but such training processes are costly and difficult to cover all potential changes. Especially in highly dynamic areas such as the emergency department or operating room, robots need to quickly respond to the demand for temporary path planning. Due to the lack of generalization ability for unknown environments in existing technologies, task interruption or path planning failure often occurs. The above problems severely restrict the flexibility and reliability of navigation robots in medical scenarios, and there is an urgent need for a general navigation solution that does not require pre-training and can adapt to dynamic environments. Summary of the Invention

[0003] The present invention provides a navigation method, device, computer equipment and medium in an unexplored environment to solve the technical problem that existing navigation methods have strong dependence on environmental maps and scenario training.

[0004] In a first aspect, a navigation method in an unexplored environment is provided. The method includes:

[0005] Receiving a natural language navigation instruction input by a user, and parsing the natural language navigation instruction into a structured semantic representation through a pre-trained large language model. The structured semantic representation at least includes target entity information, spatial constraint information, and context intention information;

[0006] Inputting the structured semantic representation into a pre-trained large language model to generate a standardized task description question sentence matching the current navigation task, and inputting the standardized task description question sentence and real-time acquired environmental image data into a pre-trained visual language question-answering model to output a semantic answer result indicating whether the target entity exists;

[0007] If the semantic answer result is existence, generating a navigation strategy for the current navigation task through a pre-trained large language model based on the structured semantic representation, and controlling the robot to execute navigation actions according to the navigation strategy through a preset path planning algorithm.

[0008] In a second aspect, a navigation device in an unexplored environment is provided, including units for performing the above method.

[0009] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above method are implemented.

[0010] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0011] The present invention provides a navigation method, device, computer device, and medium in an unexplored environment. The method includes: receiving a natural language navigation instruction input by a user, and parsing the natural language navigation instruction into a structured semantic representation through a pre-trained large language model. The structured semantic representation at least includes target entity information, spatial constraint information, and contextual intention information; inputting the structured semantic representation into the pre-trained large language model to generate a standardized task description question sentence matching the current navigation task, and inputting the standardized task description question sentence and real-time acquired environmental image data into a pre-trained vision-language question-answering model to output a semantic answer result indicating whether the target entity exists; if the semantic answer result is existence, generating a navigation strategy for the current navigation task based on the structured semantic representation through the pre-trained large language model, and controlling a robot to execute navigation actions according to the navigation strategy through a preset path planning algorithm. The present invention significantly improves the logical understanding accuracy of the large model for target domain knowledge through multi-modal data fusion, knowledge graph-guided joint training, and adversarial optimization mechanisms. The present invention parses natural language instructions into structured semantic representations through a pre-trained large language model, performs cross-modal semantic matching in combination with a vision-language question-answering model, does not rely on environmental maps and scene training, generates navigation strategies and controls the robot to execute, solves the strong dependence of traditional navigation on prior data, and realizes zero-shot general navigation in an unexplored environment. Description of the Drawings

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0013] Figure 1 is a flowchart of the navigation method in an unexplored environment according to an embodiment of the present invention;

[0014] Figure 2 isFigure 1 The flowchart of the sub-steps of step S110 in

[0015] Figure 3 is Figure 1 The flowchart of the sub-steps of step S120 in

[0016] Figure 4 is Figure 1 The flowchart of the sub-steps of step S120 in

[0017] Figure 5 The flowchart of the navigation method in an unexplored environment according to another embodiment of the present invention;

[0018] Figure 6 The flowchart of the navigation method in an unexplored environment according to still another embodiment of the present invention;

[0019] Figure 7 is Figure 1 The flowchart of the sub-steps of step S130 in

[0020] Figure 8 The schematic block diagram of the navigation device in an unexplored environment according to an embodiment of the present invention;

[0021] Figure 9 The schematic structural diagram of a computer device in an embodiment of the present invention. Specific embodiments

[0022] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0023] Traditional robot navigation relies on pre-acquired environmental maps and scene-specific training, and has insufficient generalization ability in unexplored open environments (such as dynamically changing home and medical scenarios), and it is difficult to quickly respond to diverse natural language instructions. There is an urgent need for a zero-shot general navigation solution. The present invention proposes a navigation method based on a large language model and a visual question answering model, which realizes the generation of an autonomous navigation strategy without a pre-trained map through language driving and cross-modal verification, and significantly improves the adaptability in complex environments.

[0024] The navigation method in an unexplored environment provided by the embodiments of the present invention can be applied to a robot client equipped with sensors (such as cameras, lidar, etc.) and a computing module. The present invention will be described in detail below through specific embodiments.

[0025] Please refer to Figure 1 as shownFigure 1 This is a flowchart of the navigation method in an unexplored environment provided by an embodiment of the present invention, including the following steps: S110 - S130.

[0026] S110. Receive a natural language navigation instruction input by a user, and parse the natural language navigation instruction into a structured semantic representation through a pre-trained large language model, where the structured semantic representation at least includes target entity information, spatial constraint information, and context intention information;

[0027] In this embodiment, a natural language navigation instruction refers to a navigation task request issued by a user in a colloquial expression (such as "Go to the pharmacy to pick up antipyretics"). The structured semantic representation is to deconstruct the instruction into a machine-parsable standardized data format (including target entity information, spatial constraint information, and contextual intention information) through a pre-trained large language model. Among them, the target entity information is the navigation target object (such as "antipyretics"), the spatial constraint information is the orientation or path description (such as "the third shelf on the left side of the pharmacy"), and the contextual intention information refers to the implicit task requirements derived through context reasoning (such as "the patient has a high fever and needs urgent antipyretic treatment"). Due to the differences in parsing rules caused by the split of task types in traditional navigation systems, they can often only handle a certain type of task, resulting in a lack of generalization ability when facing diverse natural language instructions. For this reason, the robot in this embodiment receives the natural language text input by the user through a microphone or a human-computer interaction interface, and calls a pre-trained large language model (such as ChatGLM, Vicuna) for semantic parsing. First, it identifies the instruction type (object target navigation, vision-language navigation, or demand-driven navigation). For the object target navigation instruction, it directly extracts the target entity (such as "Find an AED defibrillator" extracts the "target entity information" as "AED defibrillator"); for the vision-language navigation instruction, it synchronously parses the spatial constraint (such as "Go straight and then turn right after reaching Ward 3" extracts the "spatial constraint information" as "Go straight → Ward 3 → Turn right"); for the demand-driven navigation instruction, it generates the contextual intention through common sense reasoning (such as "I need to take insulin" infers the "target entity information" as "insulin" and the "contextual intention information" as "medication needs of diabetic patients → find insulin"). Finally, it integrates the three types of information into a structured semantic representation in JSON or XML format, {target entity: xxx, spatial constraint: xxx, contextual intention: xxx → xxx}. In a specific example, taking a medical scenario as an example, when the patient inputs "I left my sphygmomanometer in the doctor's office", the large language model identifies it as an object target navigation instruction, extracts the "target entity information" as "sphygmomanometer", and the "spatial constraint information" as "doctor's office", and generates a structured semantic representation for subsequent module processing. This step realizes the unified parsing of multiple types of natural language instructions through the general semantic understanding ability of the large language model, provides a standardized input for subsequent cross-modal verification and navigation strategy generation, and effectively solves the problem of insufficient generalization ability caused by inconsistent instruction formats in traditional systems.

[0028] In one embodiment, as Figure 2 shown, step S110 includes: S111 - S115.

[0029] S111. Identify the instruction type of the natural language navigation instruction, where the instruction type includes: object target navigation instruction, vision-language navigation instruction, and demand-driven navigation instruction;

[0030] S112. If the natural language navigation instruction is an object target navigation instruction, extract the target entity information from the natural language navigation instruction through a pre-trained large language model;

[0031] S113. If the natural language navigation instruction is a vision-language navigation instruction, extract the target entity information and spatial constraint information from the natural language navigation instruction through a pre-trained large language model;

[0032] S114. If the natural language navigation instruction is a demand-driven navigation instruction, perform intention reasoning on the natural language navigation instruction through a pre-trained large language model to obtain context intention information, where the context intention information includes context intention and reasoning target;

[0033] S115. Integrate the target entity information, the spatial constraint information, and the context intention information into a structured semantic representation.

[0034] In this embodiment, the object target navigation instruction specifically refers to an instruction that clearly includes the target object (such as "find a stretcher"), the visual language navigation instruction includes a composite description instruction of path and orientation (such as "go through the corridor and turn right to the emergency room"), and the demand-driven navigation instruction refers to an instruction that implies an intention to be inferred (such as "I need to rest"). The instruction type refers to the semantic task category of natural language navigation instructions, including object target navigation instructions, visual language navigation instructions, and demand-driven navigation instructions. Specifically, the robot classifies the intention of the input instruction through a pre-trained large language model (such as ChatGLM): First, it determines the type attribution based on the instruction keyword and grammar structure. If the instruction contains an explicit target entity (such as "thermometer") and no path description, it is determined as an object target navigation instruction, and the entity extraction module is called to extract the target entity information (such as "target entity: thermometer"); if the instruction contains orientation prepositions (such as "turn left", "next to") or action sequences (such as "go straight to the pharmacy and then turn right"), it is recognized as a visual language navigation instruction, and the target entity (such as "medicine box") and spatial constraint information (such as "go straight → pharmacy → turn right") are extracted synchronously; if the instruction is expressed as a demand statement (such as "wound pain"), it is determined as a demand-driven navigation instruction, and the context intention (such as "pain relief demand") is mapped to an explicit inference target (such as "target entity: painkiller") through the common sense reasoning module of the large language model, generating "context intention information: wound pain → find painkiller"; finally, the system integrates the above information into a structured semantic representation in JSON format, such as "{target entity: wheelchair, spatial constraint: empty, context intention: empty}" (object target class), "{target entity: nurse station, spatial constraint: turn left on the second floor, context intention: empty}" (visual language class), "{target entity: ibuprofen, spatial constraint: pharmacy, context intention: wound pain → pain relief demand}" (demand-driven class). In a specific example, for example, in a medical scenario, in the face of the instruction "go straight to the pharmacy and then turn left, find a bandage next to the third shelf", the model recognizes it as a visual language navigation instruction, extracts the target entity "bandage" and the spatial constraint "go straight → pharmacy → turn left → next to the third shelf", and generates a standardized structure {target entity: bandage, spatial constraint: [go straight, pharmacy, turn left, next to the third shelf]}. This step realizes the accurate semantic deconstruction of multi-type instructions through classification parsing and structured integration, providing a unified data interface for subsequent cross-modal verification.

[0035] S120. Input the structured semantic representation into a pre-trained large language model to generate a standardized task description question sentence that matches the current navigation task, and input the standardized task description question sentence and the real-time acquired environmental image data into a pre-trained visual language question-answering model to output a semantic answer result indicating whether the target entity exists;

[0036] In this embodiment, the standardized task description question refers to a standardized natural language query generated based on a structured semantic representation (such as "Is there any painkiller on the second - layer shelf of the pharmacy?"). The vision - language question - answering model is a pre - trained multi - modal model (such as MiniGPT4, BLIP2 - VQA), which can jointly process text questions and image data and output semantic answer results (such as "Yes", "Not found"). Due to the separation of semantic description and visual verification in the traditional navigation scheme, the target positioning error rate is high, especially in unexplored environments where misjudgments cannot be dynamically corrected. Therefore, in this embodiment, a large - language model is called to convert the structured semantic representation into a standardized question suitable for visual question - answering: If there is spatial constraint information (such as "to the left of the nurse station"), a compound question with azimuth binding is generated (such as "Is there an electrocardiogram machine to the left of the nurse station?"); If it is a demand - driven instruction, its reasoning target (such as "painkiller") is used as the core entity of the question (such as "Is there any painkiller in the current area?"). Subsequently, the robot collects environmental images in real - time through a camera, inputs the question and the image into the vision - language question - answering model. The model aligns text and visual features through a cross - modal attention mechanism, outputs a natural - language answer and parses it into a boolean determination (exists / does not exist); Alternatively, the model analyzes whether there are visual elements (such as the color, shape, and spatial position of the target entity) in the image that match the question through a text - image feature alignment technology (such as CLIP image encoding + LLM text encoding). In a specific example, in a medical scenario, for the structured semantic representation {target entity: ventilator, spatial constraint: beside Bed No. 3 in the ICU ward}, the large - language model generates the question "Is there a ventilator beside Bed No. 3 in the ICU ward?" After analyzing the real - time image, the vision - question - answering model returns "Yes, on the bedside equipment rack"; If the target is not detected in the image, it returns "Ventilator not found". This step realizes the dynamic alignment of semantic goals and visual perception in an unknown environment through a cross - modal question - answering mechanism, significantly reducing the false - detection rate caused by single - modal deviation in the traditional scheme and improving the reliability of navigation decisions.

[0037] In one embodiment, as Figure 3 shown, the step S120 includes: S121 - S122.

[0038] S121. Extract the target entity information and the spatial constraint information from the structured semantic representation;

[0039] S122. Input the target entity information and the spatial constraint information into a pre - trained large - language model to construct a standardized task description question, where the standardized task description question is used to inquire whether the target entity information exists under the spatial constraint information.

[0040] In this embodiment, the system extracts key-value pairs from the structured semantic representation in JSON format. If the field contains "target entity" and "spatial constraint", it reads directly (e.g., extracting "electrocardiogram monitor" and "Bed No. 2 in the emergency room" from {target entity: electrocardiogram monitor, spatial constraint: Bed No. 2 in the emergency room}); if it only contains "target entity", the spatial constraint is left empty (e.g., the spatial constraint is "empty" after extracting {target entity: wheelchair}); subsequently, the extracted entity and constraint are input into a large language model (such as Vicuna), and a standardized task description question is generated through a preset Q&A template: if the spatial constraint is not empty, the model constructs a question according to the pattern "Is there a [target entity] in [spatial constraint]?" (e.g., "Is there an electrocardiogram monitor in Bed No. 2 in the emergency room?"); if the spatial constraint is empty, it generates "Is there a [target entity] in the current area?" (e.g., "Is there a wheelchair in the current area?"). In a specific example, for instance, in a medical scenario, for the structured representation {target entity: infusion pump, spatial constraint: beside the infusion stand in the pediatric ward}, after extraction, the large language model is called to generate the question "Is there an infusion pump beside the infusion stand in the pediatric ward?"; if the input is {target entity: tourniquet, spatial constraint: empty}, then "Is there a tourniquet in the current area?" is generated; this step converts the semantic representation into a query instruction that can be accurately understood by the visual question answering model through a rule-based question construction mechanism, eliminating natural language ambiguity and improving the efficiency and interpretability of cross-modal verification.

[0041] In one embodiment, as Figure 4 shown, step S120 includes: S123 - S124.

[0042] S123. Input the standardized task description question and the real-time acquired environmental image data into a pre-trained visual language question answering model;

[0043] S124. Perform semantic matching on the standardized task description question and the environmental image data through the visual language question answering model, and output a semantic answer result corresponding to the existence or non-existence of the target entity.

[0044] In this embodiment, the system first synchronously inputs the standardized question sentence and the environmental image data into the vision-language question-answering model: for questions with spatial constraints (such as "Is there insulin on the third-layer shelf of the pharmacy?"), the model extracts the visual features of each region in the image through the CLIP image encoder and parses the spatial constraints and target entities in the question sentence through the LLM text encoder; for questions without spatial constraints (such as "Is there a wheelchair in the current area?"), the model automatically focuses on the central area of the image for feature extraction. Subsequently, the model performs semantic matching: on the one hand, it identifies the objects in the image through the target detection module (such as YOLOv8), and on the other hand, it calculates the semantic similarity between the object features and the target entity through cross-modal contrast learning (such as the image-text alignment loss function). If the similarity exceeds the threshold (such as 0.8) and the spatial position meets the constraints, it is determined that "it exists"; if the relevant object is not detected in the image or the position does not match, it is determined that "it does not exist". In a medical scenario, when the input question is "Is there an electrocardiogram monitor on the 3rd bed in the ICU ward?", the model first locates the area of the 3rd hospital bed in the image, extracts the visual features of the equipment in this area, and through semantic matching, finds that the feature similarity with the "electrocardiogram monitor" is 0.92, and the position meets the "3rd bed" constraint, and outputs the semantic answer result of "it exists"; if only an empty bed is shown in the image, it outputs "it does not exist". This step realizes the semantic alignment between the natural language question sentence and the environmental image through the multi-modal understanding ability of the vision-language question-answering model, provides a real-time and accurate target existence judgment for navigation decision-making, and solves the problem of poor scene adaptability caused by the traditional solution relying on prior data.

[0045] In one embodiment, as Figure 5 shown, after the step S120, it further includes: S1201 - S1204.

[0046] S1201. Based on the standardized task description question sentence and the real-time environmental image, perform forward verification through the vision-language question-answering model to obtain a forward verification result;

[0047] S1202. Generate a natural language description of the environmental image data through the pre-trained vision-language question-answering model;

[0048] S1203. Input the natural language description into the pre-trained large language model, determine whether the natural language description contains content matching the target entity information and the spatial constraint information, and generate a reverse verification result;

[0049] S1204. When both the forward verification result and the reverse verification result are "it exists", determine that the semantic answer result is "it exists".

[0050] In this embodiment, "positive verification" refers to directly judging whether the target entity exists in the specified space through a vision-language question-answering model (such as the answer to the question "Is there a first aid kit beside the nurse station?"); natural language description refers to the text-based generalization of the image content by the model (such as "There are three insulin bottles on the third shelf of the pharmacy"); "negative verification" is to analyze whether the description text contains the target entity and constraint information through a large language model. Specifically, the system first extracts the image region features through the CLIP image encoder and combines with the YOLOv8 object detection module to identify candidate objects (such as insulin bottles, wheelchairs), and calculates the cross-modal similarity between the target entity and the candidate objects (such as the similarity of insulin bottles is 0.85). If the similarity exceeds the threshold (0.8) and the position conforms to the spatial constraint (the third shelf), the positive verification result is "exists"; at the same time, the vision-language question-answering model generates a natural language description of the environment (such as "The third shelf of the pharmacy is placed with medicines, including insulin and disinfectant"), and inputs the description into the large language model (such as ChatGLM) for negative verification to judge whether it contains the keywords of the target entity (insulin) and spatial constraint (the third shelf) (if it exists, it outputs "matched"); when the positive verification result ("exists") is consistent with the negative verification result ("matched"), the final semantic answer result is determined as "exists". In a specific example, taking the medical scenario as an example, for the question "Is there an electrocardiogram monitor beside the 3rd bed in the ICU ward?", after the positive verification confirms the existence through region feature matching, the negative verification checks whether the natural language description "There are electrocardiogram monitors and ventilators placed on the equipment rack beside the 3rd bed" contains the target entity and location constraint. If both verifications pass, "exists" is output; if the positive verification is "exists" but the target is missing in the negative description (such as "There is only a ventilator on the equipment rack"), the result is invalidated and re-verified. This step reduces the risk of misjudgment of a single modality through a two-way semantic alignment mechanism, improves the confidence and robustness of target detection in an unexplored environment, and solves the problem of high false detection rate caused by the environmental dynamics of traditional solutions.

[0051] In one embodiment, as Figure 6 shown, after the step S120, it further includes: S1205 - S1208.

[0052] S1205. Extract confidence vocabulary from the semantic answer result to generate a semantic matching intensity score;

[0053] S1206. Calculate the answer stability factor based on the answer consistency of the same standardized task description questions in the historical multi-frame image data;

[0054] S1207. Perform a weighted sum of the semantic matching intensity score and the stability factor according to a preset weight coefficient to obtain a comprehensive confidence score;

[0055] S1208. If the comprehensive confidence score exceeds a preset threshold, confirm that the semantic answer result indicating the existence of the target entity is valid.

[0056] In this embodiment, confidence words refer to deterministic descriptive words included in the semantic answer output by the vision-language question answering model (such as "certainly exists", "may not be found"). The semantic matching strength score (m_score) is generated by mapping according to the predefined confidence levels to which the words belong (for example, "certainly exists" is mapped to 0.9, and "may exist" is mapped to 0.6). The answer stability factor (s_score) is calculated by statistically analyzing the consistency ratio of the judgment results for the same question in the historical consecutive N frames (N≥5) (for example, if 4 out of 5 frames answer "exists", then s_score = 0.8). Specifically, the system extracts confidence words from the real-time semantic answer (for example, extracting "detected" from "detected an oxygen cylinder" and mapping m_score = 0.95), and looks back at the answers to the same question in the most recent 5 frames (such as "exists, exists, does not exist, exists, exists"), and calculates the consistency ratio s_score = (4 / 5) = 0.8. Subsequently, the comprehensive confidence score M is obtained by weighted summation according to the preset weight coefficients (α = 0.6, β = 0.4), M = 0.6×0.95 + 0.4×0.8 = 0.89. If M exceeds the threshold (such as 0.85), it is confirmed that the existence of the target entity is valid. In a specific example, in a medical scenario, for the question "Is there a defibrillator in the ICU ward?", the current frame answer is "exists"

[0057] (m_score = 0.9), the historical 5-frame answer consistency is 80% (s_score = 0.8), and the comprehensive score M = 0.6×0.9 + 0.4×0.8 = 0.86 > 0.85, so the judgment result is valid. If the current frame answer is "may exist"

[0058] (m_score = 0.7) and the historical consistency is 60% (s_score = 0.6), then M = 0.6×0.7 + 0.4×0.6 = 0.66 < 0.85, triggering re-verification. This step improves the reliability of the target existence judgment by fusing real-time semantic matching and historical stability through a dynamic confidence evaluation mechanism, and solves the problem of misjudgment caused by single-frame misdetection or instantaneous occlusion in an open environment.

[0059] S130. If the semantic answer result is "exists", generate a navigation strategy for the current navigation task based on the structured semantic representation, and control the robot to execute navigation actions according to the navigation strategy through a preset path planning algorithm.

[0060] In this embodiment, the "navigation strategy" refers to a sequence of navigation subtasks in the form of natural language generated by a large language model, and the path planning algorithm is a dynamic obstacle avoidance algorithm based on real-time environment perception (such as RRT or D Lite). Traditional navigation systems rely on pre-trained policy models or fixed map data, require a large amount of scenario data and are difficult to handle un-trained instruction types, resulting in insufficient zero-shot navigation ability and difficulty in quickly generating feasible paths suitable for multiple types of tasks in unexplored environments. For this reason, after confirming that the semantic answer result is "yes", this embodiment calls a large language model (such as Vicuna, ChatGLM) to parse the structured semantic representation: for object target navigation instructions (such as "find the electrocardiogram monitor"), the model infers high-probability areas (such as "ward equipment area, nurse station") based on the target entity information and generates an exploration path sorted by priority; for visual language navigation instructions (such as "go straight to the end of the corridor and turn right to find the AED device"), the model decomposes the spatial constraint information into an action chain (such as "go straight 10 meters → turn right → scan the right wall") and converts it into a waypoint sequence for the path planning module; for demand-driven instructions (such as "the patient urgently needs a hemostatic gauze"), the model infers the potential location of the target based on the contextual intention information ("hemostasis requirement → operating room or ambulance") and generates a region-focused search strategy. Subsequently, the path planning algorithm converts the natural language description in the navigation strategy into a motion instruction in the robot coordinate system (such as "move to coordinates (10,5) → rotate 90 degrees → grab the target") to control the robot to execute the navigation action. In a specific example, for example, in a medical scenario, when the semantic answer confirms that "there is a ventilator beside Bed No. 3 in the ICU ward", the large language model generates a navigation strategy "go straight along the current corridor to the entrance of the ICU ward → turn left and enter the area of Bed No. 3", and the path planning algorithm generates an obstacle avoidance path based on the real-time lidar data and controls the robot to move; if the target is behind a dynamic obstacle, the algorithm adjusts the path in real time (such as bypassing the hospital bed). This step seamlessly connects semantic understanding and path execution through a language-driven strategy generation mechanism, realizes real-time autonomous navigation of multiple types of tasks in unexplored environments, breaks through the strong dependence of traditional solutions on pre-stored maps and reinforcement learning training, and significantly improves the adaptability to complex scenarios.

[0061] In one embodiment, as Figure 7 shown, step S130 includes: S131 - S133.

[0062] S131. If the natural language navigation instruction is an object target navigation instruction, infer the areas where the target entity may exist through a pre-trained large language model and plan the exploration order, and generate a navigation strategy based on region priority;

[0063] S132: if the natural language navigation instruction is a visual language navigation instruction, decomposing the spatial constraint information into executable action chains through a pre-trained large language model, and generating a navigation strategy including a path sub-goal sequence;

[0064] S133: If the natural language navigation instruction is a demand-driven navigation instruction, the target object and its position are inferred based on the contextual intention information through a pre-trained large language model, and a navigation strategy converted into a navigation target is generated.

[0065] In this embodiment, the system first determines the type of instruction: if it is an object target navigation instruction (such as "find a blood glucose meter"), the large language model (such as GPT-4) calls the medical knowledge graph to infer the possible areas where the blood glucose meter may exist ("laboratory equipment area → ward nursing station → outpatient rapid testing room"), and generates a priority sequence according to the regional accessibility and frequency of use (such as laboratory equipment area priority 0.8, nursing station 0.6), and plans the navigation strategy of "explore the laboratory first → if not found, go to the nursing station"; if it is a visual language navigation instruction (such as "turn left and take the elevator to the third floor to find the rehabilitation department"), the model decomposes the spatial constraint into an action chain: "recognize the left turn sign → move to the elevator → press the 3rd floor button → look for the rehabilitation department sign after exiting the elevator", and each action corresponds to a path sub-goal (such as "coordinate point in front of the elevator door" and "3rd floor elevator exit"); if it is a demand-driven navigation instruction (such as "I need to measure blood pressure"), the model generates a navigation target through intent reasoning ("measure blood pressure → blood pressure monitor → nurse station"), and further plans the strategy of "go to the nurse station → ask the nurse to get a blood pressure monitor → return to the patient". In a specific example, for example, in a medical scenario, in the face of the object target instruction of "find antipyretic medicine", the model generates a priority sequence of "pharmacy→emergency pharmacy→fever clinic"; for the visual language instruction of "go to the treatment room to change the medicine", it is decomposed into an action chain of "go straight to the end of the corridor→turn left→swipe the access card→enter the treatment room"; for the demand-driven instruction of "I feel dizzy", the reverse target is "thermometer→nurse station" and the corresponding path is planned. This step realizes the precise mapping of multimodal instructions to executable paths through the task type adaptive strategy generation mechanism, solving the problem of insufficient scene generalization ability of traditional solutions due to the single strategy model.

[0066] The present invention uses a pre-trained large language model to parse natural language navigation instructions into a structured semantic representation that includes target entities, spatial constraints, and contextual intentions, combines a vision-language question answering model to achieve cross-modal semantic matching and two-way verification, and generates a semantic answer result regarding the existence of the target entity. For different types of instructions, it uses the large language model to infer and generate navigation strategies such as area priorities, action chains, or demand backtracking, and controls the robot to execute through traditional path planning algorithms. This solution does not rely on environmental maps and scene training data, solves the strong dependence problem of traditional navigation systems on prior information, realizes zero-shot general navigation in unexplored environments, significantly improves the instruction generalization ability, target recognition accuracy, and robustness of navigation strategy generation in complex dynamic scenarios, and is applicable to robot autonomous navigation tasks in multiple scenarios such as medical delivery and escort guidance.

[0067] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0068] The embodiment of the present invention also provides a navigation device 200 in an unexplored environment, and the navigation device in the unexplored environment corresponds one-to-one with the navigation method in the unexplored environment in the above embodiment. As Figure 8 shown, the navigation device 200 in the unexplored environment includes: a structured semantic unit 201, a question answering verification unit 202, and a navigation unit 203. The detailed description of each functional unit is as follows:

[0069] The structured semantic unit 201 is configured to receive a natural language navigation instruction input by a user, and parse the natural language navigation instruction into a structured semantic representation through a pre-trained large language model, where the structured semantic representation at least includes target entity information, spatial constraint information, and contextual intention information;

[0070] The question answering verification unit 202 is configured to input the structured semantic representation into a pre-trained large language model to generate a standardized task description question sentence that matches the current navigation task, and input the standardized task description question sentence and the real-time acquired environmental image data into a pre-trained vision-language question answering model to output a semantic answer result regarding whether the target entity exists;

[0071] The navigation unit 203 is configured to, if the semantic answer result is existence, generate a navigation strategy for the current navigation task based on the structured semantic representation through a pre-trained large language model, and control the robot to execute navigation actions according to the navigation strategy through a preset path planning algorithm.

[0072] In one embodiment, the structured semantic unit 201 is specifically configured to: identify the instruction type of the natural language navigation instruction, where the instruction type includes: object target navigation instruction, visual language navigation instruction, and demand-driven navigation instruction; if the natural language navigation instruction is an object target navigation instruction, extract the target entity information from the natural language navigation instruction through a pre-trained large language model; if the natural language navigation instruction is a visual language navigation instruction, extract the target entity information and spatial constraint information from the natural language navigation instruction through a pre-trained large language model; if the natural language navigation instruction is a demand-driven navigation instruction, perform intention reasoning on the natural language navigation instruction through a pre-trained large language model to obtain context intention information, where the context intention information includes context intention and reasoning target; integrate the target entity information, the spatial constraint information, and the context intention information into a structured semantic representation.

[0073] In one embodiment, the question-and-answer verification unit 202 is specifically configured to: extract the target entity information and the spatial constraint information from the structured semantic representation; input the target entity information and the spatial constraint information into a pre-trained large language model to construct a standardized task description question sentence, where the standardized task description question sentence is used to inquire whether the target entity information exists under the spatial constraint information; input the standardized task description question sentence and the real-time acquired environmental image data into a pre-trained visual language question-and-answer model; perform semantic matching on the standardized task description question sentence and the environmental image data through the visual language question-and-answer model, and output the semantic answer result corresponding to the existence or non-existence of the target entity.

[0074] In one embodiment, the navigation device 200 in an unexplored environment further includes: a forward and backward verification unit. The forward and backward verification unit is specifically configured to: perform forward verification through a visual language question-and-answer model based on the standardized task description question sentence and the real-time environmental image to obtain a forward verification result; generate a natural language description of the environmental image data through the pre-trained visual language question-and-answer model; input the natural language description into a pre-trained large language model, determine whether the natural language description contains content matching the target entity information and the spatial constraint information, and generate a backward verification result; when both the forward verification result and the backward verification result are existence, determine that the semantic answer result is existence.

[0075] In one embodiment, the navigation device 200 in an unexplored environment further includes: a scoring unit. The scoring unit is specifically configured to: extract confidence words from the semantic answer result to generate a semantic matching intensity score; calculate an answer stability factor based on the answer consistency of the same standardized task description question sentence in historical multi-frame image data; perform a weighted sum of the semantic matching intensity score and the stability factor according to a preset weight coefficient to obtain a comprehensive confidence score; and if the comprehensive confidence score exceeds a preset threshold, confirm that the semantic answer result indicating the existence of the target entity is valid.

[0076] In one embodiment, the navigation unit 203 is specifically configured to: if the natural language navigation instruction is an object target navigation instruction, infer the possible area where the target entity exists through a pre-trained large language model and plan the exploration order to generate a navigation strategy based on region priority; if the natural language navigation instruction is a vision language navigation instruction, decompose the spatial constraint information into an executable action chain through a pre-trained large language model to generate a navigation strategy including a path sub-goal sequence; and if the natural language navigation instruction is a demand-driven navigation instruction, infer the target object and its location based on the context intention information through a pre-trained large language model to generate a navigation strategy converted into a navigation target.

[0077] For the specific limitations of the navigation device in an unexplored environment, reference may be made to the limitations of the navigation method in an unexplored environment described above, which will not be elaborated here. Each module in the above-mentioned navigation device in an unexplored environment can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0078] In one embodiment, a computer device is provided. The computer device can be a client, and its internal structure diagram can be as Figure 9 shown. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the client side of a navigation method in an unexplored environment

[0079] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the navigation method in the above-mentioned unexplored environment are implemented.

[0080] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the navigation method in the above-mentioned unexplored environment are implemented.

[0081] It should be noted that for the functions or steps that the above computer-readable storage medium or computer device can achieve, reference can be made to the relevant descriptions on the server side and the client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0082] Those of ordinary skill in the art can understand that to implement all or part of the processes in the above method embodiments, it can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0083] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional unit and module is used for illustration. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0084] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A navigation method in an unexplored environment, characterized in that, The method includes: Receiving a natural language navigation instruction input by a user, and parsing the natural language navigation instruction into a structured semantic representation through a pre-trained large language model, where the structured semantic representation at least includes target entity information, spatial constraint information, and context intention information; Inputting the structured semantic representation into a pre-trained large language model to generate a standardized task description question matching the current navigation task, and inputting the standardized task description question and real-time acquired environmental image data into a pre-trained vision-language question answering model to output a semantic answer result indicating whether the target entity exists; If the semantic answer result is that it exists, generating a navigation strategy for the current navigation task based on the structured semantic representation through a pre-trained large language model, and controlling a robot to execute navigation actions according to the navigation strategy through a preset path planning algorithm.

2. The method according to claim 1, characterized in that, The step of receiving a natural language navigation instruction input by a user and parsing the natural language navigation instruction into a structured semantic representation through a pre-trained large language model, where the structured semantic representation at least includes target entity information, spatial constraint information, and context intention information, includes: Identifying the instruction type of the natural language navigation instruction, where the instruction type includes: object target navigation instruction, vision-language navigation instruction, and requirement-driven navigation instruction; If the natural language navigation instruction is an object target navigation instruction, extracting target entity information from the natural language navigation instruction through a pre-trained large language model; If the natural language navigation instruction is a vision-language navigation instruction, extracting target entity information and spatial constraint information from the natural language navigation instruction through a pre-trained large language model; If the natural language navigation instruction is a requirement-driven navigation instruction, performing intention reasoning on the natural language navigation instruction through a pre-trained large language model to obtain context intention information, where the context intention information includes context intention and reasoning target; Integrating the target entity information, the spatial constraint information, and the context intention information into a structured semantic representation.

3. The method according to claim 1, wherein The step of inputting the structured semantic representation into a pre-trained large language model to generate a standardized task description question matching the current navigation task includes: Extracting the target entity information and the spatial constraint information from the structured semantic representation; Inputting the target entity information and the spatial constraint information into a pre-trained large language model to construct a standardized task description question, where the standardized task description question is used to inquire whether the target entity information exists under the spatial constraint information.

4. The method according to claim 1, characterized in that, The step of inputting the standardized task description question and real-time acquired environmental image data into a pre-trained vision-language question answering model to output a semantic answer result indicating whether the target entity exists includes: Inputting the standardized task description question and real-time acquired environmental image data into a pre-trained vision-language question answering model; Performing semantic matching on the standardized task description question and the environmental image data through the vision-language question answering model, and outputting a semantic answer result corresponding to the existence or non-existence of the target entity.

5. The method according to claim 1, characterized in that, After the step of inputting the standardized task description question sentence and the real-time acquired environmental image data into the pre-trained vision-language question answering model to output the semantic answer result indicating whether the target entity exists, the following steps are further included: Based on the standardized task description question sentence and the real-time environmental image, perform forward verification through the vision-language question answering model to obtain a forward verification result; Generate a natural language description of the environmental image data through the pre-trained vision-language question answering model; Input the natural language description into the pre-trained large language model, determine whether the natural language description contains content matching the target entity information and the spatial constraint information, and generate a reverse verification result; When both the forward verification result and the reverse verification result are "exists", determine that the semantic answer result is "exists".

6. The method according to claim 1, wherein, After the step of inputting the standardized task description question sentence and the real-time acquired environmental image data into the pre-trained vision-language question answering model to output the semantic answer result indicating whether the target entity exists, the following steps are further included: Extract confidence words from the semantic answer result to generate a semantic matching intensity score; Calculate an answer stability factor based on the answer consistency of the same standardized task description question sentence in historical multi-frame image data; Perform weighted summation of the semantic matching intensity score and the stability factor according to a preset weight coefficient to obtain a comprehensive confidence score; If the comprehensive confidence score exceeds a preset threshold, confirm that the semantic answer result indicating the existence of the target entity is valid.

7. The method according to claim 1, wherein The step of, if the semantic answer result is "exists", generating a navigation strategy for the current navigation task based on the structured semantic representation through the pre-trained large language model, includes: If the natural language navigation instruction is an object target navigation instruction, infer the possible existing regions of the target entity through the pre-trained large language model and plan the exploration order to generate a navigation strategy based on region priorities; If the natural language navigation instruction is a vision-language navigation instruction, decompose the spatial constraint information into an executable action chain through the pre-trained large language model to generate a navigation strategy including a path sub-goal sequence; If the natural language navigation instruction is a demand-driven navigation instruction, infer the target object and its location based on the context intention information through the pre-trained large language model to generate a navigation strategy transformed into a navigation target.

8. A navigation device in an unexplored environment, characterized in that, It includes a unit for executing the method according to any one of claims 1-7 above.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

Images