Robotic autonomous exploration method fusing multimodal large model inference

Abstract

The present application relates to the field of artificial intelligence and robots, in particular to a robot autonomous exploration method fusing multi-modal large model reasoning, which acquires environment data collected by robot sensors and task instructions, then identifies semantic elements in the environment by using a multi-modal large model, generates a semantic feature set, calculates semantic value and information entropy, constructs a fusion gain field, including generating a semantic value map and an information entropy map, and dynamically adjusting the fusion weight of semantic value and information entropy, applying a prediction model based on context reasoning to predict semantics and structure in unobserved areas, updating the fusion gain field, selecting the optimal exploration target and path based on the fusion gain field under multi-dimensional resource constraints, controlling the robot to perform the exploration task, acquiring new environment data, comparing the exploration results with the expectations, performing semantic verification, and updating the exploration strategy parameters, the present application enhances the robot's understanding ability of complex natural language task instructions.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and robotics, specifically to a robot autonomous exploration method that integrates multimodal large model reasoning, belonging to the interdisciplinary technical fields of large model reasoning, robot navigation and control, intelligent perception and decision-making. Background Technology

[0002] With the rapid development of robotics technology, autonomous mobile robots are widely used in scenarios such as warehousing, security, and home environments. How to efficiently explore and navigate unknown environments has become a core issue. Existing exploration methods are mostly based on information gain (such as entropy minimization and Frontier expansion) or graph search strategies (such as A* and RRT*) to achieve path planning and action decision-making.

[0003] However, these methods have obvious limitations: First, traditional methods usually rely on geometric information or sensor data and lack the ability to understand the semantic information of the environment, resulting in a lack of intelligence in the exploration strategy; second, traditional robots only support structured command input and it is difficult to dynamically adjust the exploration strategy based on natural language; in addition, after the robot performs the task, it cannot determine whether the current state is consistent with the expected semantics, lacks a feedback mechanism, and is prone to the accumulation of erroneous actions.

[0004] With the rise of multimodal large models, they have demonstrated powerful capabilities in natural language understanding, image perception, and reasoning decision-making. Introducing them into robotic exploration tasks holds promise for breaking through existing technological bottlenecks and enabling more intelligent and semantically-based exploration methods. Summary of the Invention

[0005] This invention aims to provide a robot autonomous exploration method that integrates multimodal large-scale model reasoning capabilities, enabling robots to achieve the following capabilities in unknown environments: flexibly switching exploration modes and generating task strategies based on user natural language input; making action decisions by integrating visual semantic understanding and reasoning results on the basis of traditional information gain-driven approaches; verifying the semantic state through a multimodal large-scale model after action execution to ensure consistency between planning and execution; and constructing semantically enhanced maps and scene graphs to improve the intelligence of robot spatial perception and strategy planning.

[0006] This invention proposes a robot autonomous exploration method that integrates multimodal large model reasoning, including:

[0007] Acquire environmental data and task instructions gathered by the robot's sensors;

[0008] Based on the environmental data and task instructions, a multimodal large model is used to identify semantic elements in the environment and generate a semantic feature set.

[0009] Based on the semantic feature set and environmental data, semantic value and information entropy are calculated, and a fusion gain field is constructed, including:

[0010] The semantic elements in the semantic feature set are valued to generate a semantic value map;

[0011] Construct a multi-level information entropy representation of environmental data to generate an information entropy map;

[0012] Based on environmental complexity and task progress, the fusion weights of semantic value and information entropy are dynamically adjusted to generate a fusion gain field.

[0013] A context-based reasoning prediction model is applied to perform semantic and structural predictions on unobserved regions, and the fused gain field is updated accordingly.

[0014] Under multidimensional resource constraints, the optimal exploration target and path are selected based on the fusion gain field;

[0015] Control the robot to perform exploration tasks and acquire new environmental data;

[0016] Compare the exploration results with the expectations, perform semantic verification, and update the exploration strategy parameters.

[0017] Preferably, the step of quantifying the value of semantic elements in the semantic feature set to generate a semantic value map specifically includes:

[0018] The identified semantic elements are classified into functional areas, objects, structures, danger zones, and areas of interest.

[0019] Based on the task description, the value of each semantic element is evaluated from multiple dimensions, including exploration value, task value, security value, and structural value.

[0020] Considering the novelty and structural importance of semantic elements, the value of elements that have been observed multiple times is reduced, while the value of key structures is increased.

[0021] The value of discrete semantic elements is mapped to a continuous space to construct a semantic value map, in which the semantic value smoothly transitions from the center of the element to the surrounding area.

[0022] Preferably, the construction of a multi-level information entropy representation of environmental data and the generation of an information entropy map specifically includes:

[0023] Construct a multi-scale spatial representation that includes a high-precision layer, a standard layer, a mid-range layer, and a distant view layer;

[0024] The uncertainty of the occupied state is calculated for each spatial unit, and the observation history and time factors are taken into account and quantified into an information entropy value.

[0025] Simulate sensor noise characteristics to make the observation uncertainty higher the further away from the sensor;

[0026] The expected information gain for candidate observation locations is evaluated based on the cumulative entropy value within the visible area and the sensor perception model.

[0027] Preferably, the step of dynamically adjusting the fusion weights of semantic value and information entropy based on environmental complexity and task progress to generate a fusion gain field specifically includes:

[0028] Analyze the structural complexity, semantic diversity, and degree of dynamic change of the environment to assess environmental complexity;

[0029] Track the spatial coverage, key semantic discovery rate, and time consumption of the exploration task to determine the task progress stage;

[0030] Adjust the weights according to the complexity of the environment: increase the semantic weight in a simple but semantically rich environment, and increase the information weight in a complex but semantically simple environment.

[0031] Adjustments are made based on the task phase: in the early stages of the task, the focus is on information gain to expand coverage, while in the later stages of the task, the focus is on semantic value to deepen understanding.

[0032] A comprehensive gain evaluation function is constructed, which integrates the weighted semantic value and information gain to generate a fused gain field.

[0033] Preferably, the application uses a context-based reasoning prediction model to perform semantic and structural predictions on unobserved regions and update the fused gain field, specifically including:

[0034] Based on environmental data and semantic feature sets, the spatial structure characteristics of the explored areas are analyzed. Structural feature vectors, including spatial width, connectivity and semantic distribution, are extracted from each explored area. Based on the structural feature vectors, the structural similarity between the explored areas is calculated to identify repetitive patterns and spatial layout rules.

[0035] Construct a semantic co-occurrence relationship graph to analyze the correlation and spatial distribution patterns of different types of semantic elements;

[0036] The environmental boundary is divided into a frontier region. Based on the structural and semantic features of the surrounding known region, a multimodal large model is used for contextual reasoning to predict the possible properties of the frontier region.

[0037] By combining current observed gain and future predicted gain, a forward-looking gain assessment is constructed, and the fused gain field is updated.

[0038] Preferably, the step of selecting the optimal exploration target and path based on the fused gain field under multidimensional resource constraints specifically includes:

[0039] Establish energy consumption models, time consumption models, and safety risk assessments, and define the thresholds and tolerances for each constraint;

[0040] Construct a comprehensive objective function, integrate gain assessment and resource constraints into a unified decision-making framework, and set up a penalty mechanism for constraint violation;

[0041] A hierarchical search strategy is adopted, first roughly screening high-potential areas, and then conducting a fine search in local areas;

[0042] Construct a rolling time-domain planning framework, predict multi-step exploration sequences, select the action sequence with the largest cumulative gain, and actually execute the first decision in the plan.

[0043] Preferably, the comparison of the exploration results with the expectations, semantic verification, and updating of the exploration strategy parameters specifically include:

[0044] The exploration expectation is generated based on the decision model, including the expected semantic elements to be discovered, the expected spatial structure, and the expected information gain.

[0045] After conducting the exploration, new observational data is obtained, and the semantic elements and spatial structures that are actually discovered are extracted;

[0046] The degree of matching between the expected and the actual observation is calculated and evaluated from three dimensions: semantic matching, structural matching, and gain realization.

[0047] The parameters of the decision model are dynamically adjusted based on the verification results, and different adjustment strategies are adopted for mismatches caused by different reasons.

[0048] Establish an exploration experience database to record the correspondence between environmental characteristics, decision parameters, expected results and actual observations, and support experience transfer.

[0049] Preferably, the step of using a multimodal large model to identify semantic elements in the environment and generate a semantic feature set specifically includes:

[0050] Environmental data is preprocessed, including image enhancement and denoising;

[0051] Environmental data is input into a multimodal large model to identify objects, functional areas, and structural features in the environment;

[0052] Extract the attribute information of each semantic element, including category, location, size, and function;

[0053] Analyze the spatial and functional relationships between semantic elements to construct a semantic scene graph;

[0054] The stability of semantic recognition is improved by fusing information from multiple frames, and time consistency constraints are used to filter out noise during recognition.

[0055] Preferably, the robot sensors include an RGB-D camera, a 2D / 3D LiDAR, and an inertial measurement unit, and the environmental data includes RGB images, depth images, point cloud data, and pose information.

[0056] As a preferred option, deployment schemes adapted to different computing resources are also included:

[0057] A lightweight solution deploys a simplified multimodal model at the robot's edge, suitable for resource-constrained scenarios;

[0058] The standard approach involves deploying a medium-sized model at the edge of the robot, which is suitable for general application scenarios.

[0059] The enhanced solution adopts an edge-cloud collaborative deployment, with the edge responsible for real-time perception and basic decision-making, and the cloud responsible for complex reasoning;

[0060] The cluster solution enables multiple robots to collaborate in perception and decision-making, sharing environmental understanding and exploration experience.

[0061] This invention achieves the following beneficial effects by introducing multimodal large-model inference capabilities:

[0062] 1. It enhances the robot's ability to understand complex natural language task instructions, enabling it to extract exploration goals, preferences, and constraints from natural language and transform them into specific exploration strategies;

[0063] 2. It achieves an organic integration of semantic value and information gain, enabling robots to simultaneously consider what they see and what they understand, thereby improving exploration efficiency and target coverage;

[0064] 3. A semantic verification mechanism was established, enabling the robot to autonomously assess the consistency between the execution results and the expected goals, thereby reducing accumulated errors;

[0065] 4. A semantically enhanced environment representation was constructed, providing a foundation for environment understanding to support more advanced task planning;

[0066] 5. An adaptive exploration strategy adjustment mechanism has been introduced, enabling the system to dynamically optimize exploration decisions based on environmental characteristics and task progress. Attached Figure Description

[0067] Figure 1 This is a system architecture diagram of the multimodal task understanding and reasoning module of the present invention.

[0068] Figure 2 This is a flowchart illustrating the exploration planning process for the multimodal large model-assisted approach of this invention. Detailed Implementation

[0069] Please refer to the attached document. Figure 1-2The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The present invention provides a robot autonomous exploration method that integrates multimodal large model reasoning. This method integrates the semantic understanding and reasoning capabilities of the multimodal large model into all aspects of the robot's exploration decision-making, thereby realizing a paradigm shift from geometry-driven to semantic-geometric dual-driven exploration.

[0070] The system architecture of the robot autonomous exploration method integrating multimodal large model reasoning proposed in this invention is as follows: Figure 1 and Figure 2 As shown, it mainly includes a multimodal perception and semantic understanding module, a semantic-information fusion gain calculation module, a multi-objective exploration and planning module, and a semantic verification and adaptive adjustment module. These four modules work together to form a complete perception-understanding-decision-verification closed-loop system.

[0071] This step begins by acquiring environmental data and task instructions gathered by the robot's sensors. In a specific intelligent service robot application scenario, the robot is equipped with an RGB-D camera, a 2D LiDAR, and an inertial measurement unit. Upon entering a new office environment, it receives user instructions: explore this office area, paying particular attention to the printing area and the meeting room. These natural language instructions need to be understood by the system and translated into specific exploration strategies.

[0072] The robot collects environmental data of the currently visible area through sensors, including RGB images, depth images, point cloud data, and pose information. This raw data is then fed into a multimodal large model for processing to identify semantic elements in the environment and understand task requirements.

[0073] In a preferred embodiment of the present invention, a multimodal large model analyzes images of the office environment to identify semantic elements such as desks, chairs, computers, corridors, and doors, while simultaneously understanding the exploration preferences for printing areas and meeting rooms included in task instructions. The system associates these semantic elements with spatial locations to form an initial set of semantic features, providing a basis for subsequent exploration decisions.

[0074] In the office environment exploration scenario, the system quantifies the value of identified semantic elements (such as doors, corridors, desks, etc.) and generates a semantic value map. The system first classifies the identified elements into functional areas (such as doors, corridors, printing areas), objects (such as desks, chairs), and structures (such as walls, pillars), etc.

[0075] For tasks that specify exploring printing areas and meeting rooms, the system evaluates the value of each semantic element based on task relevance and defines a semantic element value function. , used to quantify the first The comprehensive value of each semantic element is calculated by weighted summation of the scores from each evaluation dimension:

[0076] ,

[0077] in: For the first A semantic element (such as a specific door or corridor); For the first Weighting coefficients for each evaluation dimension; For the first Evaluation functions in multiple dimensions (such as exploration value, task value, etc.); Describe the current task (i.e., explore the office area and find the printing area and meeting room).

[0078] In this specific task, the system assigns different weights to different elements. For example, doors and corridors leading to unexplored areas are given a higher exploration value (approximately 0.8) because they may lead to meeting rooms or printing areas; printers that have already been identified are given a higher task value (approximately 0.9) because they are directly related to the task; and densely populated areas with regular office work are given a higher security value (approximately 0.7) to avoid interference.

[0079] The system also considers the novelty of semantic elements, reducing the value of repeatedly observed elements through a time decay function, and defining a novelty decay factor. This is used to dynamically decay the value of an element based on the number of times it is observed repeatedly.

[0080] ,

[0081] in: The novelty decay factor is set to 0.2 in this embodiment; For elements The number of times a robot is observed. For example, when a robot passes through the same corridor multiple times, the exploration value of that corridor gradually decreases, prompting the robot to explore new areas.

[0082] Finally, the system maps the values ​​of discrete semantic elements to a continuous space, constructs a semantic value map, and defines a semantic value map function. Used to calculate the comprehensive semantic value of any point on a map:

[0083] ,

[0084] in: Represents a location point on the map (such as coordinates). ); Indicates the first The position of each semantic element; Indicates the first The radius of influence of a semantic element is proportional to the size of the element; This represents the total number of semantic elements currently identified. Indicates location point To the semantic element position The Euclidean distance is used. In this way, the system assigns semantic value to each location in the office environment, with higher value distributions around the printing area and along paths that may lead to the meeting room.

[0085] Simultaneously, the system constructs a multi-level information entropy representation of environmental data, generating an information entropy map. In the exploration of the office environment, the system constructs a spatial representation containing different resolution levels:

[0086] High-precision layer (0.05 m resolution): used to accurately represent close-range areas, such as details in the current room;

[0087] Standard layer (0.1 m resolution): Used to represent general working areas;

[0088] Mid-level (0.2 m resolution): Used to represent corridors and adjacent areas;

[0089] Distant view (0.5 m resolution): Used to roughly represent the layout of the entire office area.

[0090] The system divides the aforementioned multi-scale spatial representation into grid cells, calculates the occupancy state uncertainty for each grid cell, and quantifies it as an information entropy value. A grid cell entropy function is defined. This is used to quantify the occupancy uncertainty of the grid cell:

[0091] ,

[0092] in: This represents the probability that the grid cell is occupied, with a value ranging from 1 to 2. ; This represents the Shannon entropy value of the raster cell, with a maximum value of 1 (when...). (The information is obtained at that time). For example, when the robot first explores the office environment, the probability of occupying most areas is close to 0.5, indicating high uncertainty, and the information entropy is close to the maximum value of 1.

[0093] To reflect information aging, the entropy value of regions that have not been observed for a long time will increase over time. Therefore, a temporal entropy update function is defined. This is used to dynamically update the entropy value of the raster cell over time.

[0094] ,

[0095] in: Indicates time The entropy value of the raster cell; This represents the entropy value at the previous moment; This represents the maximum entropy increment, with a value of 0.1. This represents the time-related influence coefficient, with a value of 0.05. This indicates the time elapsed since the last observation (in seconds). This ensures that the robot will revisit areas that have not been observed for a long time, such as office areas where there may be personnel changes.

[0096] The system also simulates sensor noise characteristics, making the observation uncertainty higher the further away from the sensor, and defines an observation update weight function. Used to assess the reliability of sensor distance attenuation observations:

[0097] ,

[0098] in: Indicates the distance from the grid cell to the sensor (unit: m); This is the sensor distance attenuation coefficient, with a value of 0.01; The impact weights for observation updates have a range of values. For example, at a distance of 10 m, the observation weight is approximately 0.5, indicating a high degree of uncertainty in observing distant office areas.

[0099] Evaluate the expected information gain for candidate observation locations and define the information gain function. Used to quantify the robot's performance at candidate locations The information gains from observations at this location:

[0100] ,in: Indicates candidate observation locations; Indicates from position The set of visible grid cells; This represents the visibility coefficient, taking into account occlusion factors, and its value range is [value range missing]. ; This represents the reliability estimate of the sensor under this condition, with a value range of [value range missing]. In practical applications, when a robot is located at a corridor intersection, it typically gains a high information gain because it can observe unknown areas in multiple directions.

[0101] In exploring different areas of the office environment, the system needs to dynamically adjust the weights of semantic value and information gain. For example, semantic understanding is more important in highly structured meeting rooms, while geometric information may be more valuable in open-plan office areas.

[0102] The system first analyzes the complexity of the environment and defines the environment complexity function. This is used to comprehensively assess the overall complexity of the current exploration environment. .in: Represents the spatial structure complexity, with a value range of 1. Such as the degree of branching in the corridor, the connectivity between rooms, etc.; Indicates semantic diversity, with a value range of 100%. Such as the richness of semantic element types within a region; Indicates the degree of dynamic change in the environment, with a value range of [value range missing]. Such as the frequency of personnel movement; , and These are the weighting coefficients, which are set to 0.5, 0.3, and 0.2 in this embodiment. .

[0103] For example, in an office environment, meeting rooms typically have high semantic diversity but a relatively simple structure. The score might be 0.4; while complex corridor systems may have high structural complexity but relatively simple semantics, and the score might be 0.7.

[0104] The system tracks the progress of exploration tasks, dynamically adjusts strategies, and defines a task progress scoring function. This is used to comprehensively measure the progress of the current exploration task: .in: This represents the proportion of the total area that has been explored, with a value ranging from [value range missing]. ; This indicates the proportion of key semantic elements that have been discovered, with a value range of [value range missing]. For example, finding the printing area and meeting room information; This represents the proportion of time already consumed to the total expected time, with a value range of [value missing]. ; , and These are weighting coefficients, which in this embodiment take values ​​of 0.4, 0.4, and 0.2, respectively. .

[0105] For example, when the robot has explored 50% of the office area, found the printing area but has not yet found the meeting room, and has used 60% of the time, the progress score is about 0.4, indicating that it is in the mid-stage.

[0106] The system employs different exploration strategies depending on the stage of progress:

[0107] Initial stage ( ): biased towards information gain, rapidly expanding coverage;

[0108] Mid-term stage ( ): Balancing information gain and semantic value;

[0109] Later stage ( ): It leans towards semantic value, deepening the understanding of key areas.

[0110] Based on environmental complexity and task progress, the system dynamically adjusts the fusion weights of semantic value and information entropy. At the start of the exploration, the initial weights are set as: information gain weights. Semantic value weight .

[0111] The environmental adaptation adjustment is calculated as follows, defining the environmental weight adjustment amount. Used to offset the fusion weights based on environmental complexity: .in: The adjustment factor is set to 0.2. This represents the weight adjustment due to environmental factors; positive values ​​increase information weight, while negative values ​​increase semantic weight. For example, in a structurally complex corridor system (… )middle, Slightly increase the information weight.

[0112] The strategy for adjusting the mission phase is as follows:

[0113] Initial stage: (Information gain bias);

[0114] Mid-term stage: (Balanced allocation);

[0115] Later stage: (Semantic value bias).

[0116] The real-time feedback adjustment is calculated as follows, defining the feedback weight adjustment amount. This is used to dynamically adjust the weights based on recent differences in exploration efficiency. in: The feedback adjustment factor is set to 0.15. and These represent efficiency metrics for information gain and semantic discovery, respectively, with values ​​ranging from [value range missing]. For example, if recent explorations have more effectively discovered new regions than valuable semantic elements, It could be 0.045, slightly increasing the information weight.

[0117] Final weight calculation, defining the final fusion weight. and This is used to combine the adjustments from the three dimensions into the final information gain weight and semantic value weight: ; .

[0118] in: The function limits the weights to Within this scope, ensure that both factors have some influence. For example, in complex corridor systems during the initial exploration phase, it is possible to obtain... , The robots are guided to prioritize expanding their coverage area.

[0119] The system constructs a comprehensive gain evaluation function, fuses the weighted semantic value and information gain, generates a fused gain field, and defines the fused gain function. Used to calculate candidate positions The comprehensive exploratory value of the site: ,in: Indicates position The normalized semantic value, with a range of values ​​being ; Represents the normalized information gain, with a value range of 1. For example, in an office environment, corridor entrances leading to unexplored meeting areas often have both high semantic value and information gain, thus achieving high fusion gain.

[0120] To maintain the continuity of the gain field, the system applies spatial smoothing to avoid decision jitter, defining a smoothed gain function. Used for Gaussian kernel space smoothing of the fused gain field: ,in: Indicates location point The neighborhood is typically a region within a radius of 2 m; This represents the spatial smoothing coefficient, with a value of 2.0. Indicates location point To the location point The Euclidean distance. This smoothness ensures the continuity of the exploration path and avoids the robot frequently changing its target in an office environment.

[0121] In office environment exploration, the system analyzes the spatial structure characteristics of explored areas based on environmental data and semantic feature sets. It extracts structural feature vectors from each explored area, including spatial width, connectivity, and semantic distribution, and identifies patterns such as corridor-office layouts and meeting area features. These patterns help predict the structure of unobserved areas.

[0122] Structural similarity is calculated as follows, defining the structural similarity function. This is used to assess the similarity between two structural segments: ,in: and It can be two structural segments, such as two corridors or two office areas; Indicates the first A similarity function for each feature dimension, with values ​​ranging from... Examples include width similarity and connectivity similarity. This represents the weight of that dimension, satisfying... ; This represents the total number of feature dimensions.

[0123] For example, in a typical office building exploration, the system may identify a pattern where there are offices on both sides of a corridor. When a new corridor is explored, it can predict that there may also be offices on both sides, with a similarity of about 0.85.

[0124] The system constructs a semantic co-occurrence graph to analyze the correlation between different types of semantic elements. In an office environment, this helps predict the facilities that may exist in unseen areas.

[0125] The co-occurrence probability of semantic element pairs is calculated as follows, defining the co-occurrence probability function. This is used to quantify the statistical correlation between two semantic elements that appear simultaneously in space. ,in: and These can be two semantic elements, such as printer and copier; Represents element and The number of times they occur simultaneously; This indicates the number of times at least one element appears.

[0126] For example, during office environment exploration, the system might discover that a copier (co-occurrence probability 0.75) and a paper supply (co-occurrence probability 0.9) are usually located near the printing area. When a printer is detected, the system can predict that a copier and a paper supply area may be nearby.

[0127] The system divides the environmental boundary into a frontal region and predicts the nature of the frontal region based on the characteristics of the surrounding known areas. In office environment exploration, this helps determine whether an unexplored area is a meeting room, office area, or rest area.

[0128] Forward area its neighborhood It includes all observed areas within a distance of less than a threshold (typically 5 m). The system first calculates... Similarity with neighboring regions: ,in: Indicates the frontier area To the known area Spatial distance (unit: m); This is a distance-affected parameter, with a value of 3.0. For contextual similarity, spatial layout similarity is considered, and the value range is [value range missing]. Then, the system predicts the characteristics of the leading edge region and defines a leading edge region feature prediction function. It is used to infer the possible attributes of the frontier region based on the weighted features of the known neighboring regions: ,in: Represents a known region The feature vector contains geometric and semantic features; For the predicted frontier region The feature vector. For example, if the door leading to the front area is similar to the door to a conference room, the system may predict that the area is a conference room, and the feature vector will contain the expected elements such as the conference table and chairs.

[0129] Calculate the confidence score of the prediction results and define the prediction confidence function. This is used to assess the reliability of the predicted features in the frontier region. ,in: Indicates the number of known regions contained in the neighborhood; Indicates the consistency of neighborhood features, with a value range of 1. When consistency is high, the value is close to 1. For example, if the frontier area is surrounded by office areas, the consistency is high and the prediction confidence will be high; if there are office areas, meeting rooms and rest areas nearby, the consistency is low and the prediction confidence will be low.

[0130] The system combines current observed gain with future predicted gain to construct a forward-looking gain assessment. In office environment exploration, this enables the robot to plan more efficient paths, prioritizing paths that may lead to important areas (such as meeting rooms).

[0131] The prediction gain is calculated as follows, and the prediction gain function is defined. Used to quantize from candidate positions The expected exploration value of the unobserved areas reachable from the starting point: ,in: Indicates the candidate position. Indicates from position The set of reachable unobserved regions; Indicates the region The expected exploration value, based on predictive features Evaluation. For example, if the location If the area ahead of the predicted meeting room is accessible, and the task requires finding the meeting room, then... It will be relatively high.

[0132] The updated gain field calculation defines the total gain function. This is used to integrate directly observed gain and predicted gain into a unified exploratory decision-making basis: ,in: Indicates the candidate position Gain obtained directly from observation; The weighting coefficients for prediction are initially set at 0.3 and dynamically adjusted as the exploration progresses. In the initial stages of exploring the office environment... Smaller, focusing more on direct observation; with increased environmental awareness, Gradually increase, and give more consideration to predictive factors.

[0133] In exploring real-world office environments, the system needs to consider resource constraints such as energy, time, and security. For example, energy becomes a critical constraint for robots with limited battery life; and avoiding disturbing employees is an important consideration when exploring office areas during working hours.

[0134] Example of an energy consumption model, defining a path energy consumption function. This is used to estimate the total energy required for a robot to travel along a specified path. ,in: Indicates the first Length of the path segment (unit: m); Indicates the first The elevation change of a section of the path (unit: m), such as climbing a slope; Indicates the first Turning angle of the path segment (unit: rad); , and These are the base energy consumption coefficient, the climbing energy consumption coefficient, and the turning energy consumption coefficient, respectively, with values ​​of 0.1 kWh / m, 0.2 kWh / m, and 0.01 kWh / rad. For example, in an office environment, a 10-meter-long path including a 90-degree turn would have an energy consumption of approximately... kWh.

[0135] Safety constraints take into account human interaction and define a path safety scoring function. Used to assess the safety of a robot traveling along a specified path: ,in: Indicates the location on the path The population density, with a value range of [value missing]. ; This represents the risk factor, with a value of 0.8. For example, in a busy office area (density 0.7), the safety score is... A density of 0.1 indicates a higher risk; in an open corridor (density 0.1), the safety score is [missing value]. This indicates low risk.

[0136] The system constructs a comprehensive objective function, integrating gain evaluation with resource constraints, and defines the comprehensive objective function. This is used to unify exploration gains and multidimensional resource constraints into a single decision metric. ,in: For position Total gain; , , and These are the energy constraint function, time constraint function, safety constraint function, and coverage constraint function, respectively. , , and These are the corresponding weighting coefficients, with values ​​of 0.15, 0.2, 0.25, and 0.1.

[0137] In an office environment, if exploring a meeting area requires traversing a busy workspace, the security constraints will lower the target score, despite the high gain; conversely, navigating a longer but less crowded corridor may yield a higher overall score.

[0138] For constraints exceeding the threshold, the system adds a penalty term and defines a constraint violation penalty function. This is used to impose additional penalties on candidate locations where resource consumption exceeds a safe threshold. ,in: For the first The penalty coefficient for each constraint is typically 5 to 10 times the corresponding penalty factor. value; For the first A constraint threshold, such as 80% of the remaining battery power, is set. This ensures that the robot will not select targets that are too far away when the battery is low.

[0139] The system employs a hierarchical search strategy, first roughly screening high-potential areas before conducting a more refined search. In exploring an office environment, the system may first determine the direction of exploration (such as the east corridor), and then precisely locate the target point within that area (such as the door at the end of the corridor).

[0140] The global search employs adaptive sampling based on quadtrees, with a sample size of 20. The local fine-grained search performs dense sampling within the optimal region, with a sample size of 50, and applies gradient ascent for local optimization. This strategy balances computational efficiency and decision quality, enabling the robot to quickly find high-value exploration targets in complex office environments.

[0141] The system constructs a rolling time-domain planning framework, considers multi-step exploration sequences, and defines a rolling time-domain cumulative reward function. This is used to evaluate the overall benefit of a multi-step exploration sequence starting from the current state: ,in: The planned number of steps is set to 3. This is a discount factor, with a value of 0.8, representing the degree of importance attached to future rewards; In the state Next action The reward is calculated as the ratio of gain to cost.

[0142] For example, in exploring an office environment, the system might plan to first explore the east corridor, then check potential printing areas, and finally proceed to the predicted meeting room, rather than directly selecting the single point with the highest current gain. The system only executes the first step of the sequence and replans under new conditions to adapt to the dynamically changing environment.

[0143] In exploring the office environment, the system generates exploration expectations based on a decision model. For example, when the robot decides to explore an area at the end of a corridor, based on contextual reasoning, the system might generate the following expectations:

[0144] Expected semantic elements to be discovered: conference table (80% probability), projection equipment (70% probability), multiple chairs (90% probability);

[0145] Expected spatial structure: an enclosed room with an area of ​​approximately 20 to 30 square meters;

[0146] Expected information gain: approximately 0.75 (normalized value).

[0147] These expectations provide a benchmark for subsequent verification.

[0148] After conducting exploration, the robot acquires new observational data and extracts the semantic elements and spatial structures it discovers. For example, after actually exploring the area, the robot might discover:

[0149] Actual semantic elements: conference table, projector, 8 chairs, whiteboard;

[0150] Actual spatial structure: Enclosed room, area 25 square meters;

[0151] Actual information gain: 0.7.

[0152] The system calculates the degree of matching between the expected and actual observations, and defines a multidimensional degree of matching evaluation function. This function is used to comprehensively evaluate the consistency between the exploration results and the expectations from three dimensions: semantic matching, structural matching, and gain matching. It weights and sums the matching scores of the three dimensions to obtain a comprehensive matching score.

[0153] ,

[0154] ,

[0155] ,

[0156] ,

[0157] in: This function represents the similarity calculation between the expected set of semantic elements and the actual set of discovered semantic elements, with values ​​ranging from... ; This function represents the similarity calculation between the expected spatial structure and the actual observed spatial structure, and its value range is... ; The matching score represents the semantic dimension. The matching score is for the structural dimension; The matching score for the gain dimension is set to the upper limit of 1.0, which is the ratio of the actual gain to the expected gain. , and These are the weights for the semantic matching dimension, the structural matching dimension, and the gain matching dimension, respectively, with values ​​of 0.4, 0.3, and 0.3.

[0158] In the example above, the system may calculate (Most of the expected elements were found, and there were additional whiteboards); (The space type and size meet expectations); (Actual gain was slightly lower than expected); It was determined to meet expectations.

[0159] Based on the validation results, the system dynamically adjusts the parameters of the decision-making model. In exploring the office environment, this allows the robot to gradually adapt to the layout and characteristics of a specific office building.

[0160] For example, when the robot successfully predicts the location of the meeting room multiple times, the system may increase the weight of structural pattern recognition and semantic association analysis; when the prediction is incorrect, it may decrease the prediction weight. Alternatively, the model parameters can be adjusted. This adaptive mechanism improves exploration efficiency over time.

[0161] The system establishes an exploration experience database to record exploration experiences in different office environments. For example, it records patterns such as doors at the end of corridors usually leading to meeting rooms, and printing areas typically being located at the edge of open office areas.

[0162] When a robot enters a new office environment, the system searches for experience records of similar environments and selects a suitable parameter template as the initial configuration. This knowledge transfer significantly accelerates the adaptation process to the new environment, enabling the robot to efficiently explore the new office building in a short time.

[0163] This invention provides deployment schemes adaptable to different computing resources. In the scenario of an office building navigation robot, a suitable configuration can be selected based on hardware conditions:

[0164] For small reception robots, a lightweight solution can be adopted, deploying a simplified multimodal model, such as MobileVLM, at the edge, so that it can understand basic instructions and environment and meet simple tour needs;

[0165] For service robots in high-end commercial buildings, an enhanced solution can be adopted, with the edge device responsible for real-time navigation and basic decision-making, and the cloud server responsible for complex reasoning, providing deep semantic understanding and precise navigation services;

[0166] For large-scale park inspections, a cluster approach can be adopted, with multiple robots exploring collaboratively, sharing environmental understanding and exploration experience, and quickly generating a complete semantic map of the park.

[0167] The different deployment schemes described above enable this invention to adapt to various application scenarios and provide flexible and efficient solutions.

[0168] The robot autonomous exploration method provided by this invention, which integrates multimodal large model reasoning, introduces the semantic understanding and reasoning capabilities of multimodal large models into all aspects of robot exploration decision-making, realizing a paradigm shift from geometry-driven to semantic-geometric dual-driven exploration. This method achieves significant improvements in exploration efficiency, semantic understanding, adaptability, and resource optimization, opening up new avenues for the autonomous operation of intelligent robots in complex and unknown environments.

[0169] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robot autonomous exploration method integrating multimodal large-model reasoning, characterized in that, include: Acquire environmental data and task instructions gathered by the robot's sensors; Based on the environmental data and task instructions, a multimodal large model is used to identify semantic elements in the environment and generate a semantic feature set. Based on the semantic feature set and environmental data, semantic value and information entropy are calculated, and a fusion gain field is constructed, including: The semantic elements in the semantic feature set are valued to generate a semantic value map; Construct a multi-level information entropy representation of environmental data to generate an information entropy map; Based on the environmental complexity index and task progress score, the fusion weights of semantic value and information entropy are dynamically adjusted to generate a fusion gain field; wherein, the environmental complexity index is calculated based on the spatial structure complexity of the information entropy map, the semantic diversity of the semantic feature set, and the degree of dynamic change of the environment, and the task progress score is calculated based on the spatial coverage, key semantic discovery rate, and time consumption ratio. A context-based reasoning prediction model is applied to perform semantic and structural predictions on unobserved regions, and the fused gain field is updated accordingly. Under multidimensional resource constraints, the optimal exploration target and path are selected based on the fusion gain field; Control the robot to perform exploration tasks and acquire new environmental data; The exploration results are compared with the expectations, semantic verification is performed, and the exploration strategy parameters are updated; wherein, the exploration strategy parameters include the fusion weight, the semantic value assessment dimension weight, and the prediction model confidence parameter.

2. The method according to claim 1, characterized in that, The step of quantifying the value of semantic elements in the semantic feature set to generate a semantic value map specifically includes: The identified semantic elements are classified into functional areas, objects, structures, danger zones, and areas of interest. Based on the task description information in the task instructions, the value of each semantic element is evaluated from multiple dimensions, including exploration value, task value, security value, and structural value. Considering the novelty and structural importance of semantic elements, the value of elements that have been observed more than a preset observation threshold is reduced, while the value of structural nodes with connectivity greater than a preset connectivity threshold is increased. The discrete values ​​of each semantic element in the semantic feature set are mapped to a continuous space to construct a semantic value map, wherein the semantic value smoothly transitions from the center of the element to the surrounding area.

3. The method according to claim 1, characterized in that, The construction of a multi-level information entropy representation of environmental data and the generation of an information entropy map specifically includes: Construct a multi-scale spatial representation that includes a high-precision layer, a standard layer, a mid-range layer, and a distant view layer; The multi-scale spatial representation is divided into grid cells. The occupancy state uncertainty is calculated for each grid cell, and the observation history and time factors are taken into account, and the result is quantified as an information entropy value. Based on the sensor noise model, the observation uncertainty weights are set for the grid cells according to their distance from the sensor, and the observation uncertainty is higher for grid cells that are farther away. The expected information gain is evaluated for candidate observation locations, which is calculated based on the cumulative entropy values ​​of each grid cell within the visible area of ​​the candidate location and the sensor perception model.

4. The method according to claim 1, characterized in that, The process of dynamically adjusting the fusion weights of semantic value and information entropy based on environmental complexity and task progress to generate a fusion gain field specifically includes: Analyze the structural complexity, semantic diversity, and degree of dynamic change of the environment to assess environmental complexity; Track the spatial coverage, key semantic discovery rate, and time consumption of the exploration task to determine the task progress stage; The weights are adjusted according to the complexity of the environment. In an environment where the structural complexity is lower than the first preset threshold and the semantic diversity is higher than the second preset threshold, the semantic weights are increased. In an environment where the structural complexity is higher than the third preset threshold and the semantic diversity is lower than the fourth preset threshold, the information weights are increased. Adjustments are made based on the task progress score. When the task progress score is lower than the first progress threshold, information gain is preferred to expand coverage. When the task progress score is higher than the second progress threshold, semantic value is preferred to deepen understanding. A comprehensive gain evaluation function is constructed, which integrates the weighted semantic value and information gain to generate a fused gain field.

5. The method according to claim 1, characterized in that, The application uses a context-based reasoning prediction model to perform semantic and structural predictions on unobserved regions and update the fused gain field, specifically including: Based on the environmental data and the semantic feature set, the spatial structure features of the explored areas are analyzed, and structural feature vectors including spatial width, connectivity and semantic distribution are extracted from each explored area. Based on the structural feature vectors, the structural similarity between the explored areas is calculated, and repetitive patterns and spatial layout rules are identified. Construct a semantic co-occurrence relationship graph, count the co-occurrence frequency of different types of semantic elements in the semantic feature set, and analyze the correlation and spatial distribution patterns of the semantic elements; The environmental boundary is divided into frontier regions. Known regions within a preset distance range of each frontier region are obtained as neighborhoods. Based on the structural feature vectors and semantic co-occurrence relationships of the neighborhoods, a multimodal large model is used for contextual reasoning to predict the possible properties of the frontier regions. By combining the current observed gain and the predicted future gain, a forward-looking gain assessment is constructed, and the fused gain field is updated.

6. The method according to claim 1, characterized in that, The selection of the optimal exploration target and path based on the fused gain field under multidimensional resource constraints specifically includes: Establish energy consumption models, time consumption models, and safety risk assessment models, and define the thresholds and tolerances for each constraint; Construct a comprehensive objective function, integrate gain assessment and resource constraints into a unified decision-making framework, and set up a penalty mechanism for constraint violation; A hierarchical search strategy is adopted. First, coarse-grained sampling is performed according to the preset grid spacing to filter candidate regions whose fusion gain value is higher than the preset gain threshold. Then, dense sampling search is performed within the candidate regions according to the fine grid spacing. Construct a rolling time-domain planning framework, predict multi-step exploration sequences, select the action sequence with the largest cumulative gain, and actually execute the first decision in the plan.

7. The method according to claim 1, characterized in that, The process of comparing the exploration results with the expected results, performing semantic verification, and updating the exploration strategy parameters specifically includes: The exploration expectation is generated based on the decision model, including the expected semantic elements to be discovered, the expected spatial structure, and the expected information gain. After conducting the exploration, new observational data is obtained, and the semantic elements and spatial structures that are actually discovered are extracted; The degree of matching between the expected and the actual observation is calculated and evaluated from three dimensions: semantic matching, structural matching, and gain realization. The parameters of the decision model are dynamically adjusted based on the evaluation results of the matching degree, and different adjustment strategies are adopted for mismatches caused by different reasons. Establish an exploration experience database to record the correspondence between environmental characteristics, decision parameters, expected results and actual observations, and support experience transfer.

8. The method according to claim 1, characterized in that, The step of using a multimodal large model to identify semantic elements in the environment and generate a semantic feature set specifically includes: Environmental data is preprocessed, including image enhancement and denoising; Environmental data is input into a multimodal large model to identify objects, functional areas, and structural features in the environment; Extract the attribute information of each semantic element, including category, location, size, and function; Analyze the spatial and functional relationships between semantic elements to construct a semantic scene graph; The stability of semantic recognition is improved by fusing information from multiple frames, and time consistency constraints are used to filter out noise during recognition.

9. The method according to claim 1, characterized in that, The robot sensors include an RGB-D camera, a 2D / 3D LiDAR, and an inertial measurement unit. The environmental data includes RGB images, depth images, point cloud data, and pose information.

10. The method according to claim 1, characterized in that, It also includes deployment schemes adapted to different computing resources: A lightweight solution deploys a simplified multimodal model at the robot's edge, suitable for resource-constrained scenarios; The standard approach involves deploying a medium-sized model at the edge of the robot, which is suitable for general application scenarios. The enhanced solution adopts an edge-cloud collaborative deployment, with the edge responsible for real-time perception and basic decision-making, and the cloud responsible for complex reasoning; The cluster solution enables multiple robots to collaborate in perception and decision-making, sharing environmental understanding and exploration experience.

Images