Robot positioning control method, electronic device, and storage medium

Abstract

The application relates to a robot positioning control method, an electronic device and a storage medium, and belongs to the technical field of robot navigation, wherein the method comprises the following steps: acquiring environment observation data, motion state data and a natural language instruction of a robot, and respectively extracting corresponding visual feature vectors, ontology feature vectors and language instruction embedding vectors and structured semantic representations, and determining a current temperature parameter based on the same; adjusting the matching degree between the visual feature vectors and the language instruction embedding vectors and the structured semantic representations by using the temperature parameter, to obtain a confidence distribution; determining a positioning decision result based on the confidence distribution; and performing corresponding motion control operations according to the positioning decision result, to realize the positioning control of the robot. The adaptive calibration of visual and language confidence is realized through a course temperature scheduling mechanism, the problem that a fixed temperature parameter cannot adapt to the phased requirements of navigation is solved, and the positioning accuracy and navigation efficiency of the robot in a complex environment are improved.

Description

Technical Field

[0001] This application relates to the field of robot navigation technology, specifically to a robot positioning and control method, electronic device, and storage medium. Background Technology

[0002] Vision-language navigation (VLN), a core task in the field of embodied intelligence, aims to enable robots to autonomously navigate and locate themselves in unknown visual environments based on natural language commands. In recent years, with the rapid development of large-scale vision-language models, VLN technology has shown broad application prospects in service robots, human-robot collaboration, and home assistants.

[0003] In the field of visual-language alignment technology, the CLIP model laid the foundation for cross-modal representation learning. This model employs a contrastive learning framework, encoding images and text separately through a dual-tower structure, and utilizing a temperature scaling mechanism to transform the original similarity scores into a probability distribution. However, this fixed-temperature strategy has significant limitations: on the one hand, a single temperature is difficult to adapt to visual-language alignment tasks of varying complexity, resulting in overly smoothed confidence scores for simple samples and insufficient discriminative power for complex samples; on the other hand, a fixed temperature cannot reflect the dynamic uncertainties during navigation, leading to an overly conservative model during the exploration phase and potentially overconfidence during the precise localization phase.

[0004] To improve temperature scaling mechanisms, existing technologies have proposed various adaptive calibration methods. LiT predicts instance-level temperature through meta-learning networks, while CLIP-Adapter introduces a lightweight adaptation module to learn task-specific temperature offsets. However, these methods are primarily designed for static image-text retrieval tasks, neglecting the temporal dynamics and stage-specific features of navigation scenarios. In the field of humanoid robot navigation and localization, traditional methods mainly rely on SLAM technology, which suffers from drawbacks such as high map building costs, missing semantic information, and poor adaptability to dynamic environments.

[0005] Therefore, there is an urgent need for a robot positioning and control method that can dynamically adjust temperature parameters based on navigation stage, command complexity, historical confidence level, and robot status. Summary of the Invention

[0006] To overcome the aforementioned shortcomings, this application is proposed to address, or at least partially address, the shortcomings of existing vision and language navigation where temperature parameters remain constant and cannot adapt to the needs of different navigation stages. It provides a robot positioning and control technology that can dynamically adjust temperature parameters based on task execution progress, instruction complexity, historical confidence sequence, and robot stability.

[0007] According to a first aspect of the present invention, a robot positioning and control method is provided, comprising the following steps: Acquire the robot's environmental observation data, motion state data, and natural language commands, and extract the corresponding visual feature vectors, ontological feature vectors, language command embedding vectors, and structured semantic representations respectively; The current temperature parameter is determined based on the language instruction embedding vector and structured semantic representation, the ontology feature vector, the task execution progress, and the historical confidence sequence. Using the temperature parameter, the matching degree between the visual feature vector, the language instruction embedding vector, and the structured semantic representation is adjusted to obtain the calibrated confidence distribution; The location decision result is determined based on the confidence distribution; Based on the positioning decision result, corresponding motion control operations are executed to achieve robot positioning control.

[0008] This solves the problem of fixed temperature parameters in existing technologies, which cannot be dynamically adjusted according to the navigation stage. Specifically, existing methods (such as CLIP) use a globally fixed temperature, which leads to overconfidence in the early stages of navigation, causing the model to miss key landmarks, and the model to be too smooth in the later stages of navigation, resulting in inaccurate positioning.

[0009] A five-step closed-loop architecture of "perception → scheduling → calibration → decision → control" was constructed, which is the basic framework of this invention. By associating temperature parameters with task execution progress and historical confidence sequences, temperature is given dynamic adaptive capability: in the early stage of navigation, the temperature is higher and the probability distribution is smoother, so the model tends to explore multiple candidate targets and avoid locking onto the wrong target too early; in the later stage of navigation, the temperature is lower and the probability distribution is sharper, so the model confidently locks onto the best target and achieves accurate positioning.

[0010] Meanwhile, multimodal inputs (visual, ontological, and linguistic) provide a complete perceptual foundation for subsequent temperature scheduling and confidence calibration. Visual observations provide environmental information, motion state data provides the robot's own state, and natural language commands provide task semantics; the fusion of these three enables the robot to fully understand the current situation. The final motion control operation forms a closed-loop feedback, allowing the robot to adjust its movement in real time based on the localization decision results, achieving a complete link from perception to execution. This allows temperature parameters to adapt to changes during the navigation process, fundamentally resolving the contradiction of "early overconfidence and later inaccurate localization" caused by a fixed temperature. It also establishes a complete perception-decision-control closed loop, laying the architectural foundation for the functional implementation of subsequent modules.

[0011] In the above technical solution, the visual feature vector includes a visual feature vector, the ontology feature vector includes an ontology feature vector, the language instruction embedding vector and structured semantic representation include a language instruction embedding vector and structured semantic representation, and the task execution progress includes the navigation phase progress. The step of determining the current temperature parameter based on the language instruction embedding vector and structured semantic representation, the ontology feature vector, task execution progress, and historical confidence sequence includes: The navigation phase progress is calculated based on the structured semantic representation and the number of steps already executed. The instruction semantic complexity is calculated based on the language instruction embedding vector. The body stability is determined based on the ontology feature vector. The current temperature parameter is determined based on the navigation phase progress, the instruction semantic complexity, the historical confidence sequence, and the body stability. The historical confidence sequence is a sequence formed by confidence values ​​at K time points. The robot positioning and control method may further include: A neural network is constructed, with the navigation phase progress, the instruction semantic complexity, the historical confidence sequence, and the body stability as inputs to the neural network, and the output is an adaptive temperature correction amount. The adaptive temperature correction is coupled with the temperature parameter to obtain the final temperature, and the confidence distribution is obtained using the final temperature.

[0012] This solves the problem of "memoryless" temperature regulation in existing technologies. Existing instance-level adaptive methods (such as LiT and CLIP-Adapter) rely solely on current input features to estimate temperature, without incorporating temporal information such as navigation progress, instruction complexity, and historical confidence. This leads to decision oscillations in semantically ambiguous regions (such as corridor intersections)—the temperature adaptively increases to promote exploration, then suddenly drops to lock onto the wrong target after obtaining new observations, and then increases again to form an oscillating cycle.

[0013] The four core input variables of temperature parameters and their sources have been identified, enabling temperature regulation to have a "memory" time-series sensing capability.

[0014] First, the navigation phase progress is calculated from the structured semantic representation, enabling temperature to sense the degree of task completion. The temperature is maintained high in the early stages of navigation to facilitate exploration, and decreases in the later stages to achieve precise positioning, thus achieving a smooth transition between exploration and utilization.

[0015] Second, the semantic complexity of instructions is calculated from the language instruction embedding vector, enabling temperature to perceive the difficulty of the task. Complex instructions (such as those involving multiple spatial relationships and nested targets) correspond to higher temperatures, prompting the robot to conduct more thorough exploration when performing complex tasks; simple instructions correspond to lower temperatures, allowing the robot to quickly lock onto the target. This gives temperature regulation semantic awareness.

[0016] Third, the robot's stability is estimated from its own feature vector, enabling temperature to sense the robot's motion state. When the robot shakes violently, the temperature adjusts accordingly to avoid erroneous high confidence levels caused by decreased observation quality; when the robot is stable, the temperature remains normal to ensure positioning accuracy. This achieves dynamic matching between visual-verbal alignment confidence and the robot's motion capabilities.

[0017] Fourth, the historical confidence sequence records the confidence values ​​at multiple recent decision points, enabling temperature to perceive historical trends in confidence. When the confidence level remains low, temperature automatically triggers a re-exploration, achieving "predictive intervention"—actively adjusting before positioning fails, rather than passively correcting after failure.

[0018] Furthermore, a neural network alternative is provided, which takes navigation progress, command complexity, historical confidence sequence, and aircraft stability as inputs, and outputs an adaptive temperature correction coupled with the temperature calculated by the explicit formula. This scheme can serve as an enhanced implementation of the present invention, further optimizing the temperature prediction accuracy based on the explicit formula. It upgrades temperature regulation from "memoryless" single-frame estimation to "memory-based" temporal fusion, solving the decision oscillation problem in semantically ambiguous regions, achieving deep coupling between temperature and navigation progress, command complexity, aircraft state, and historical confidence, while providing an alternative implementation path for neural network enhancement.

[0019] In any of the above technical solutions, determining the current temperature parameter based on the navigation phase progress, the command semantic complexity, the historical confidence sequence, and the aircraft stability includes: The initial temperature value is calculated based on the preset base temperature, stage modulation factor, complexity modulation factor, trend modulation factor, and stability modulation factor. The initial temperature value is constrained to a preset effective range using a safety boundary function to obtain the temperature parameter; The phase modulation factor is determined based on the progress of the navigation phase, the complexity modulation factor is determined based on the semantic complexity of the instruction, the trend modulation factor is determined based on the historical confidence sequence, and the stability modulation factor is determined based on the stability of the organism.

[0020] This solves the problems of chaotic parameter coupling, lack of physical interpretability, and potential instability in numerical calculations. Designing the system to either mix all factors together, making it difficult to debug, or to lack boundary constraints on temperature, can lead to numerical overflow or overly smoothed distributions.

[0021] This invention decomposes the temperature parameter into a product of five independent modulation factors: τ = τ base × α stage ×α complex × α trend × α stability This multi-factor decoupling design has significant engineering value—each modulation factor responds independently to different environmental or state changes, and engineers can independently adjust the parameters of each factor without worrying about mutual interference.

[0022] Specifically, the stage modulation factor responds to task progress, the complexity modulation factor responds to instruction difficulty, the trend modulation factor responds to historical confidence changes, and the stability modulation factor responds to the organism's motion state. Each of these five factors has a clear physical meaning and collectively determines the final temperature value. This design makes the temperature regulation process interpretable: when the temperature changes, it is clear which factor caused the change.

[0023] Furthermore, a safety boundary function constrains the temperature parameter to a preset effective range. This design serves a dual purpose: firstly, it prevents excessively low temperatures from causing the exponential function in the softmax calculation to overflow (exp(large numbers) may exceed the range of floating-point representation); secondly, it prevents excessively high temperatures from causing the probability distribution to become overly smooth (all candidate views have probabilities approaching equality, losing their discriminatory power). The safety boundary ensures the numerical stability and decision effectiveness of the algorithm.

[0024] This ensures that the temperature parameters have clear physical interpretability and engineering controllability, that each modulation factor can be adjusted independently, and that the numerical stability of the algorithm is ensured through safety boundary constraints.

[0025] In any of the above technical solutions, the minimum value of the preset effective interval is 0.01, and the maximum value is 0.2; The formula for calculating the temperature parameter τ(t) is as follows: τ(t)=τ base ×α stage ×α complex ×α trend ×α stability Where, τ base The base temperature; α stage The stage modulation factor is determined by the navigation stage progress pt and adopts an exponential decay form α. stage=exp(-β·p t ), where β is the course decay coefficient; α complex The complexity modulation factor is determined by the instruction semantic complexity c. t It was decided to adopt the logarithmic compression form α. complex =1+γ·ln(1+c t ), where γ is the complexity sensitivity coefficient; α trend The trend modulation factor is calculated by combining the moving average and variance of the corresponding sequence of the historical confidence sequence; α stability The stability modulation factor is determined by the stability level *st* of the organism and is expressed as a linear inverse proportional form α. stability =1-κ·(1-s t ), where κ is the stability response coefficient.

[0026] This solution addresses the lack of a systematic mathematical framework in existing temperature control methods. If the design either uses a fixed temperature or a black-box neural network to predict the temperature, without a temperature calculation formula with a clear mathematical form and physical meaning, the temperature control process becomes difficult to predict, adjust, and transfer between different scenarios.

[0027] This solution provides a complete mathematical expression for the temperature parameter, formalizing temperature calculation into a repeatable, predictable, and debuggable mathematical formula.

[0028] The phase modulation factor adopts an exponential decay form, a classic strategy in course learning. The mathematical properties of exponential decay cause it to decrease rapidly in the early stages and then level off, which is highly consistent with the cognitive patterns of navigation tasks: in the early stages of navigation, the need for exploration is strong, requiring significantly higher temperatures; as the task progresses, the need for exploration gradually weakens, and the rate of temperature decrease slows down; in the later stages of navigation, the need for precise positioning dominates, and the temperature stabilizes at a low level. The exponential decay form makes the transition between exploration and utilization smooth and natural, without the need for manually designed phase switching rules.

[0029] The complexity modulation factor employs a logarithmic compression form. The logarithmic function exhibits a "sublinear growth" characteristic: as instructions increase from simple to moderately complex, the temperature rises significantly to increase exploration; as instructions increase from moderate to extremely complex, the rate of temperature increase slows to prevent overexploration. This design aligns with practical needs—simple instructions do not require excessive exploration, while extremely complex instructions do not require an infinitely increasing temperature.

[0030] The trend modulation factor is calculated based on the moving average and variance of the historical confidence series. The ingenuity of this design lies in the fact that a temperature correction is triggered only when the historical confidence mean is low and the variance is small. A low mean indicates poor overall confidence, and a small variance indicates a sustained downtrend (rather than oscillation). This judgment logic effectively distinguishes between "true, sustained downtrends" and "temporary, accidental fluctuations," avoiding the erroneous triggering of a temperature correction due to a single or brief instance of low confidence.

[0031] The stability modulation factor employs a linear relationship, making temperature positively correlated with the organism's stability: the temperature is normal when the organism is stable, and decreases when the organism shakes. This seems to contradict the intuition that "shaking requires conservative decision-making," but the actual logic is that: organism shaking leads to a decrease in observation quality, and lowering the temperature at this time makes the confidence distribution smoother. The model will not generate false high confidence for blurry and noisy observations, thus avoiding making incorrect decisions during shaking.

[0032] The safety boundary constraint limits the temperature to between 0.01 and 0.2, which covers the entire effective range from high exploration to high utilization. The upper limit ensures sufficient exploration, while the lower limit ensures numerical stability.

[0033] Therefore, this solution provides a complete set of mathematically rigorous temperature calculation formulas, with each factor having a clear physical meaning and mathematical form, making the temperature regulation process predictable, adjustable, and transferable.

[0034] In any of the above technical solutions, the navigation phase progress pt is calculated as follows: pt = completed_steps / total_steps, where completed_steps is the number of instruction fragments that have been executed, and total_steps is obtained by parsing from the structured semantic representation; The semantic complexity of the instruction is c t The calculation method is as follows: c t =w1·N spatial +w2·N nesting +w3·N condition , where N spatial N represents the number of spatial relation words extracted from the language instruction embedding vector corresponding to the instruction. nesting N represents the nesting level of the target object. condition The number of conditional constraints; The formula for calculating the stability level st of the organism is: s t =1 / (1+σ ω 2 +‖e joint ‖) Where, σ ω2 Let $\mathbf{ ...

[0035] This solution addresses the problem of the inability to accurately quantify navigation progress, command complexity, and aircraft stability. Using coarse estimates (such as estimating progress based on time ratios) or completely ignoring these factors would result in inaccurate data for temperature regulation.

[0036] This scheme provides a precise quantification method for the three core input variables of temperature scheduling.

[0037] The progress calculation during the navigation phase is based on the total number of instruction fragments and the number of completed instruction fragments parsed from the structured semantic representation. The advantage of this calculation method lies in "semantic alignment"—progress strictly corresponds to the semantic units of the instructions, rather than being a rough ratio of time or distance. For example, the instruction "cross the living room → go around the sofa → reach the kitchen refrigerator" is parsed into three sub-tasks, with progress increasing precisely by one-third for each completed sub-task. This ensures precise alignment between temperature decay and task completion, avoiding temperature mismatch caused by progress estimation errors.

[0038] The calculation of instruction semantic complexity considers three dimensions: the number of spatial relation words reflects the geometric complexity of the path (e.g., "through," "around," and "along" represent different spatial relationship processing difficulties); the target nesting level reflects the positioning depth (e.g., "the remote control on the sofa in the living room" requires hierarchical positioning); and the conditional constraints reflect the execution requirements (e.g., "avoid the middle chair" adds obstacle avoidance logic). The weighted combination of these three dimensions can quantitatively evaluate the complexity of any natural language instruction, enabling temperature regulation to have true semantic awareness.

[0039] The calculation of robot stability uses angular velocity variance and joint execution error norm as core variables. Angular velocity variance directly reflects the degree of robot swaying—the more violent the swaying, the greater the angular velocity fluctuation and the higher the variance. Joint execution error norm reflects joint tracking accuracy—the larger the error, the greater the deviation between the robot's actual and expected motion, potentially leading to instability. Both are included in the denominator and their reciprocals are taken to normalize the robot's stability to the (0,1) interval: a value of 1 for complete stability and close to 0 for severe instability. This design provides the robot's stability with clear physical interpretability and an intuitive comparative scale.

[0040] This scheme provides a precise, repeatable, and verifiable quantification method for the three core input variables of temperature regulation, ensuring that the basis for temperature control is accurate and reliable.

[0041] In any of the above technical solutions, the temperature parameter is adjusted in stages during task execution: When the task execution progress is less than the first progress threshold, the temperature parameter is maintained in the first temperature range so that the robot can explore the environment. When the progress of the task execution exceeds the second progress threshold, the temperature parameter is reduced to the second temperature range to enable the robot to perform precise positioning. When the historical confidence value is lower than the historical confidence threshold for K consecutive steps, or the stability of the organism is lower than the stability threshold, or the quality of the environmental observation data is lower than the visual quality threshold, the temperature parameter is increased to the third temperature range, and a conservative decision-making operation is performed. When the historical confidence value is higher than the historical confidence threshold, or the stability of the organism is higher than the stability threshold, or the quality of the environmental observation data is higher than the visual quality threshold, the temperature parameter is restored to the value before the increase, and the conservative decision-making operation is stopped. Wherein, the minimum value of the first temperature range is greater than the maximum value of the second temperature range; the third temperature range is between the first temperature range and the second temperature range, and the conservative decision operation includes at least one of the following: reducing movement speed, reducing stride length, increasing perception frequency, pausing confidence calibration, and performing pure ontology perception prediction based on the ontology feature vector and joint odometry through extended Kalman filtering.

[0042] This solves the problem of the lack of phased and safety constraints in temperature regulation. If the temperature is designed to be either fixed or adaptive but unable to distinguish between the three different behavior modes of "normal exploration", "precise positioning" and "abnormal conservatism", it will be even more impossible to automatically trigger safety operations in abnormal situations such as body shaking or visual degradation.

[0043] This scheme implements a three-segment temperature curve, which is an innovative extension of the exponentially decaying monotonic curve.

[0044] The high-temperature setting during the exploration phase allows the robot to explore fully in the early stages of navigation, preventing it from going astray by prematurely locking onto the wrong target. The low-temperature setting during the precise positioning phase allows the robot to approach the target with high confidence, achieving precise docking. These two phases constitute the main trend of temperature change, corresponding to the core idea of ​​course scheduling.

[0045] When historical confidence levels remain low, organism stability declines, or visual observation quality deteriorates, the temperature does not continue to decrease along the exponential decay curve; instead, it temporarily rises to the intermediate temperature range. This non-monotonic temperature change responds to sudden anomalies: low confidence levels indicate that the current positioning hypothesis may be incorrect, requiring re-exploration; organism swaying or visual degradation indicates a decline in observation quality, necessitating a reduction in confidence in the current observations. After the temperature rises to the intermediate temperature range, the probability distribution changes from sharp to moderately smooth, and the model switches from a "target locked" state to a "moderate exploration" state, avoiding erroneous decisions under abnormal conditions.

[0046] The specific actions taken in conservative decision-making operations further enhance safety: reducing movement speed allows more time for reaction, shortening stride improves the stability of bipedal walking, and increasing perception frequency allows for more intensive monitoring of environmental changes. Pure proprioceptive prediction is designed for extreme cases where visual observation completely fails—when the camera is blocked or the lighting changes abruptly, the robot pauses visual-verbal alignment, maintaining short-term localization solely based on data from the joint encoders and IMU, and then re-aligns once vision is restored.

[0047] The design of the temperature ranges has a clear physical meaning: the exploration range is the highest, the precise positioning range is the lowest, and the conservative decision-making range lies in between. This relationship ensures the accuracy of the relative temperature values ​​under different modes, making the temperature values ​​themselves implicitly contain information about the current behavior pattern.

[0048] The recovery logic ensures that conservative decision-making does not continue indefinitely: once the abnormal condition disappears, the temperature automatically returns to its value before the callback. This means that the robot will not remain in conservative mode for an extended period due to a brief shock, avoiding efficiency losses caused by excessive conservatism.

[0049] This solution upgrades temperature regulation from a simple, monotonic decay to a three-segment non-monotonic curve that can respond to anomalies, achieving a dynamic trade-off between "high efficiency under normal conditions and safety under abnormal conditions." At the same time, it directly ensures the physical safety of the robot through conservative decision-making operations.

[0050] In any of the above technical solutions, the matching degree between the visual feature vector and the language instruction embedding vector and structured semantic representation is adjusted using the temperature parameter to obtain a calibrated confidence distribution, including: The original similarity between the visual feature vector and the language embedding vector is scaled using the temperature parameter to obtain a calibrated confidence distribution. The entropy and confidence gap of the confidence distribution are then calculated to generate a localization decision based on the entropy and confidence gap of the confidence distribution.

[0051] This solution addresses the lack of quantitative analysis of confidence distributions. Currently, if only probability distributions are output without fully utilizing the uncertainty information contained within them, decision-making will be based on a single, incomplete approach.

[0052] The probability distribution scaled by temperature is used as the confidence distribution, and two key statistics are extracted from this distribution: entropy and confidence gap.

[0053] Temperature scaling is mathematically equivalent to division in the softmax function. This operation is differentiable, supports end-to-end neural network training, and allows the entire system to be optimized within a unified framework.

[0054] Entropy quantifies uncertainty from the perspective of the overall distribution. When the distribution is flat and the probabilities of each candidate view are similar, the entropy value is high, indicating that the model is highly uncertain; when the distribution is steep and the probability of one candidate view is significantly higher than others, the entropy value is low, indicating that the model is highly certain. As a continuous quantity, entropy provides a more refined measure of uncertainty than discrete confidence thresholds, capturing the "degree of uncertainty" rather than just "whether it is certain."

[0055] The confidence gap quantifies uncertainty from the perspective of local competition. It reflects the difference between the highest probability and the second-highest probability: a large gap indicates that the model has clearly distinguished the best candidate; a small gap indicates that the model hesitates between the best and second-best candidates. This metric complements entropy—entropy is sensitive to the overall distribution, while the gap is sensitive to local competition. Using both together can cover a wider range of decision-making scenarios.

[0056] Therefore, multidimensional uncertainty information was extracted from the confidence distribution, providing rich and complementary judgment basis for subsequent hierarchical decision-making.

[0057] In any of the above technical solutions, the confidence distribution is calculated as follows: Calculate the original similarity score between the visual feature vector vi and the language instruction embedding vector l: ; The calibrated probability distribution is calculated using a softmax function with the temperature parameter τ(t): in, Let be the confidence probability of the i-th candidate view, which constitutes the confidence distribution, zi be the i-th original similarity score between visual features and language embedding, τ(t) be the temperature parameter, and M be the total number of candidate views; Calculate the entropy value of the probability distribution: ; The confidence gap ΔP is obtained by calculating the difference between the maximum probability Pmax and the second-highest probability Psecond.

[0058] This scheme addresses the lack of a systematic mathematical expression for confidence score calculation, while clarifying the meaning of each parameter and ensuring the numerical stability of the calculation process. A complete mathematical expression for confidence score calculation is provided, making the process repeatable, verifiable, and optimizable.

[0059] The original similarity is expressed in inner product form z. i =v i •l, this is the simplest form of vector similarity calculation, with a computational complexity of only O(d) (d is the feature dimension, typically 768). This linear calculation supports high-frequency real-time reasoning and meets the time constraints of the robot control cycle.

[0060] The temperature scaling softmax formula is the standard form of probability normalization, ensuring that the output satisfies the probability axiom (the sum of all probabilities is 1, and each probability lies between 0 and 1). The temperature parameter τ(t) appears in the denominator, and its adjustment effect is mathematically as follows: the larger τ is, the smaller the absolute value of the independent variable in the exponential function, the smoother the change of the exp function, and the more uniform the output distribution; the smaller τ is, the larger the absolute value of the independent variable in the exponential function, the more drastic the change of the exp function, and the more concentrated the output distribution is near the maximum value.

[0061] The entropy calculation formula, as defined in information theory, physically represents the "average amount of information contained in a distribution" or the "expected value of uncertainty." When the distribution is perfectly deterministic (one term is 1, the rest are 0), the entropy is 0; when the distribution is perfectly uniform, the entropy reaches its maximum value, logM. This continuous measure allows uncertainty to be precisely quantified.

[0062] The confidence gap ΔP = max(P) - second_max(P) is the difference between the maximum probability and the second-highest probability, and its value ranges from [0,1]. This indicator intuitively reflects the model's confidence in the optimal choice: the larger the gap, the stronger the confidence; the smaller the gap, the more hesitant the model.

[0063] In numerical implementation, softmax calculation typically employs the technique of "subtracting the maximum value". This technique ensures that even when τ is extremely small (such as 0.01), the largest exponent term will not overflow, and other exponent terms will not become zero, thus guaranteeing the numerical stability of the calculation.

[0064] This scheme provides a complete and feasible mathematical framework for confidence calculation, ensuring the normalization, numerical stability and real-time performance of the calculation results, and providing a reliable data foundation for three-level decision-making.

[0065] In any of the above technical solutions, generating the location decision result based on the confidence distribution includes: When the entropy value is less than the first entropy threshold and the confidence gap is greater than the first gap threshold, it is determined that the hierarchical decision should be executed directly. When the entropy value is greater than or equal to the first entropy threshold and less than or equal to the second entropy threshold, or when the confidence gap is less than the second gap threshold, a local re-inference level decision is determined. When the entropy value is greater than the second entropy threshold and the historical confidence average is less than the historical confidence threshold, a global replanning level decision is determined.

[0066] This solution addresses the disconnect between temperature regulation and decision-making mechanisms. Treating temperature adaptation and decision-level switching as independent modules leads to mismatches such as "temperature is suitable for precise positioning but decision-making remains in exploratory mode" or "decision-making has triggered replanning but temperature has not been raised in time."

[0067] By directly using the temperature-scaled output (entropy value and confidence gap) as the basis for decision-making, a deep collaboration between temperature and decision-making is achieved.

[0068] The triggering condition for direct execution of the L1 level is low entropy and high confidence gap. When both conditions are met simultaneously, it indicates that the model is not only generally deterministic (low entropy) but also has a clear preference for the best candidate (high gap). In this case, executing L1 directly is the optimal choice—no additional verification is required; the target is directly locked and motion commands are issued, maximizing navigation efficiency.

[0069] The triggering condition for local re-inference is that the entropy value is in the medium range or the confidence gap is small. The former indicates that the model has a moderate degree of uncertainty, and the latter indicates that the model hesitates between the best and second-best candidates. In this case, direct execution may lead to errors, while global reprogramming is too aggressive. Therefore, L2 local re-inference is chosen—pausing the process and verifying the hypothesis through chain thinking, eliminating uncertainty at the lowest cost.

[0070] The trigger condition for global replanning is high entropy and persistently low historical confidence. High entropy indicates that the model is extremely uncertain under the current observations, and persistently low historical confidence indicates that this uncertainty is not a temporary fluctuation but a persistent state. At this point, neither L1 nor L2 can effectively solve the problem, and L3 global replanning must be triggered—updating the scene graph, performing active exploration, and re-establishing environmental cognition.

[0071] The core advantage of this temperature-decision coordination mechanism lies in the fact that temperature changes directly affect decision-making. When the temperature decreases due to progress, the probability distribution becomes sharper, the entropy value decreases, and the confidence gap increases, naturally leading the system to favor L1 direct execution. Conversely, when the temperature rises due to abnormal callbacks, the probability distribution becomes smoother, the entropy value increases, and the confidence gap decreases, naturally leading the system to favor L2 or L3. Temperature regulation and decision-making levels achieve seamless coordination through a shared mathematical foundation (probability distribution).

[0072] It achieves deep collaboration between temperature and decision-making, avoids information fragmentation and decision mismatch between modules, and enables the system to automatically select the most appropriate processing strategy under different levels of uncertainty.

[0073] In any of the above technical solutions, the direct execution of hierarchical decision-making includes: taking the candidate view corresponding to the highest probability in the confidence distribution as the navigation target, converting the three-dimensional spatial coordinates of the navigation target into target joint angles, and generating joint spatial trajectories based on the target joint angles using model prediction control and issuing them for execution. The local re-inference hierarchical decision includes: pausing the current motion, performing chained reasoning based on the language instruction embedding vector and structured semantic representation, generating intermediate reasoning steps to verify the rationality of the current positioning hypothesis, and if the verification is successful, executing the direct execution hierarchical decision in the manner described above; if the verification fails, re-extracting the visual feature vector from the current environmental observation data and re-calibrating the confidence level. The global replanning hierarchy includes: triggering an update of the scene map; performing an active exploration action based on the updated scene map; the active exploration action includes in-situ rotation scanning or moving to a new viewpoint in the direction of the confidence gradient; after the active exploration action is completed, re-acquiring the environmental observation data; extracting the visual feature vector based on the re-acquiring environmental observation data; and re-performing confidence calibration.

[0074] This solution addresses the issues of unclear implementation methods and lack of engineering implementation plans at each decision-making level. It provides complete engineering implementation plans for each of the three decision-making levels.

[0075] The direct execution level (L1) process translates high-level decisions into low-level motion commands. First, it identifies the candidate view with the highest probability from the confidence distribution, using the view's 3D spatial coordinates as the navigation target. Then, it solves the inverse kinematics problem to convert the target coordinates into target angles for each joint. This is a classic robot kinematics problem, where the existence and uniqueness of the solution depend on the robot configuration and target position. Finally, it uses model predictive control to optimize the control sequence in the time domain, generating a smooth joint spatial trajectory while satisfying physical constraints such as joint velocity and acceleration. After the trajectory is executed, extended Kalman filtering fuses visual localization and joint odometry in real time to form a closed-loop correction. When visual observation fails, the system can degrade to pure proprioception prediction, maintaining short-term localization capability until vision is restored.

[0076] The execution flow of the Local Re-inference Layer (L2) embodies the principle of "eliminating uncertainty at the lowest cost." First, the current movement is paused and the user maintains a standing posture, providing stable observation conditions for chained reasoning. Then, the chained thinking capability of the visual-language model is activated to generate intermediate reasoning steps, such as, "Currently, a window at the end of the corridor is observed, but the instruction requires turning left to enter the living room. In the previous frame, a door was seen on the left side in the middle of the corridor, therefore, a left turn should be made." After reasoning, verification is performed: if the reasoning result supports the current candidate view, the localization hypothesis is correct, and execution is downgraded to L1; if a contradiction is found, the current candidate view is abandoned, features are re-extracted, and realignment is performed. This design avoids blind action in uncertain states and avoids excessive consumption of computational resources for global replanning.

[0077] The execution flow of the global replanning layer (L3) embodies the concept of "proactive environment exploration." First, it triggers an online update of the scene graph, a semantic map representing environmental entities and their spatial relationships. The update process includes re-identifying entities, updating spatial locations, and reconstructing topological relationships. Then, based on the updated scene graph, proactive exploration actions are performed: a 360-degree in-situ rotation scan acquires comprehensive environmental observations, building a complete understanding of the local environment; moving towards the confidence gradient direction moves along the direction of increasing confidence, gradually approaching areas with greater information. After proactive exploration is complete, observation data is re-collected, features are re-extracted, and confidence calibration is re-performed, fundamentally resolving the localization failure problem.

[0078] This solution provides a complete engineering implementation plan for three-level decision-making, making the entire chain from high-level decision-making to low-level motion control operable and reproducible. L1 enables efficient positioning, L2 enables low-cost verification, and L3 enables autonomous recovery. The three work together to form a complete positioning failure handling system.

[0079] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the robot positioning and control method described in any of the above technical solutions.

[0080] According to another aspect of the present invention, an electronic device is also provided, including a processor and a memory, wherein a computer program is stored in the memory, and when the processor executes the computer program, it implements the robot positioning and control method described in any of the above technical solutions.

[0081] This invention achieves adaptive calibration of visual and linguistic confidence through a course temperature scheduling mechanism, solving the problem that fixed temperature parameters cannot adapt to the phased needs of navigation, and improving the robot's positioning accuracy and navigation efficiency in complex environments. Attached Figure Description

[0082] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein: Figure 1 This is a flowchart of a robot positioning and control method according to an embodiment of this application; Figure 2 This is a schematic diagram of the robot positioning and control principle according to an embodiment of this application. Detailed Implementation

[0083] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0084] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0085] like Figure 1 As shown, a robot positioning and control method according to an embodiment of the present invention may include the following steps: Step 102: Multimodal input perception, acquire the robot's environmental observation data, motion state data and natural language commands, and extract the corresponding visual feature vectors, ontological feature vectors and language command embedding vectors and structured semantic representations respectively.

[0086] The environmental observation data is collected by a camera mounted on the robot's head at a frequency of 30fps, with an image resolution of 640×480 pixels. After preprocessing by inertial measurement unit (IMU) electronic image stabilization, the data is then processed by a pre-trained VisionTransformer (V... i T) is encoded as a D1-dimensional visual feature vector vi, where the typical value of D1 is 768, and i is the candidate view index.

[0087] Specifically, V i T segments the input image into 16×16 pixel image blocks, extracts global visual semantic information through a multi-head self-attention mechanism, and outputs a feature vector. This encoding process can be represented as: v i = V i T(image i ) Among them, image i Let i be the RGB image of the i-th candidate view.

[0088] The motion state data includes joint angles, angular velocities, and actuation torques distributed throughout the robot's joints. For a robot platform with N degrees of freedom, a single frame of motion state data constitutes a 3N-dimensional vector.

[0089] Motion state data of the most recent T frames (T is typically 10) are acquired through a time-series sliding window. After concatenating the T-frame data, the data is compressed into a fixed-dimensional ontology feature vector f using a multilayer perceptron (MLP). proprio , which serves as the ontology feature vector.

[0090] The MLP contains three hidden layers, each with 512 neurons, and uses ReLU as the activation function. The compression process can be represented as: f propri =MLP([state {t-T+1} , state {t-T+2} ,...,state t ]) Where, state t Let be the 3N-dimensional motion state vector at time t.

[0091] The natural language instruction is expressed in natural language, such as "walk through the living room, go around the sofa, and stop in front of the refrigerator in the kitchen." This instruction is semantically parsed by a pre-trained language model (such as a text encoder from BERT, RoBERTa, or CLIP), outputting two key pieces of information: Language instruction embedding vector l: A D2-dimensional semantic feature vector, with a typical D2 value of 768, used for similarity calculation with the visual feature vector.

[0092] Structured semantic representation: An executable structured representation containing the following information: Target object sequence: List the target locations that need to be passed through or reached in order, such as [living room, sofa, kitchen, refrigerator]; Spatial constraints: describe the spatial relationships between targets, such as [passing through, bypassing, reaching]; Action sequence: describes the actions to be performed, such as [move, obstacle avoidance, docking]; The structured semantic representation can be extracted from natural language instructions through semantic role labeling or dependency parsing.

[0093] Step 104: Course temperature scheduling. Based on the task execution progress, instruction complexity, historical confidence sequence, and body stability, determine the current temperature parameter τ(t).

[0094] Task execution progress p t This indicates the proportion of currently completed instruction segments out of the total number of instructions. The calculation formula is: pt = completed_steps / total_steps in: `completed_steps` represents the number of instruction segments that have been executed, obtained by monitoring the execution status of the target object sequence in the structured semantic representation. For example, when the robot arrives at the "living room," `completed_steps` changes from 0 to 1.

[0095] `total_steps` represents the total number of instruction fragments obtained from the structured semantic representation. For example, the instruction "walk through the living room → go around the sofa → reach the refrigerator in the kitchen" contains 3 instruction fragments, so `total_steps=3`.

[0096] p t The value range of p is [0,1]. t =0 indicates that navigation has just begun, p t =1 indicates that the navigation task is complete.

[0097] Instruction semantic complexity c t It is a scalar value used to quantify the complexity of natural language instructions. The calculation formula is: c t =w1·N spatial +w2·N nesting +w3·N condition in: N spatial The number of spatial relation words extracted from the corresponding instructions in the language instruction embedding vector. Spatial relation words include "through", "around", "along", "passing", "next to", "in front of", etc. N nesting : The nesting level of the target object, indicating the depth of target positioning in the command. For example, "the remote control on the sofa in the living room" has a nesting level of 3 (living room → sofa → remote control); N condition : The number of conditional constraints, indicating the restrictive conditions in the instruction, such as "avoid the middle chair" or "stop at the red marker".

[0098] w1, w2, w3: corresponding weighting coefficients, typically with values ​​of w1=0.3, w2=0.4, and w3=0.3.

[0099] c t The typical value range is [0, 5]. The larger the value, the more complex the instruction and the more exploratory it requires.

[0100] Body stability s tIt is a scalar value used to quantify the robot's current motion stability and observation quality. The calculation formula is: s t =1 / (1+σ ω 2 +‖e joint ‖) in: σ ω 2 : The angular velocity variance extracted from the ontology feature vector. Specifically, from f proprio Extract the component related to angular velocity and calculate its variance within the time window. The angular velocity variance reflects the degree of robot body sway; the greater the sway, the higher the variance. ω 2 The larger.

[0101] ||e joint ‖: The joint execution error norm extracted from the ontology feature vector. Specifically, from f proprio Extract the deviation between the expected joint angle and the actual joint angle, and calculate its L2 norm. This value reflects the joint execution accuracy; the larger the error, the lower the L2 norm. joint The larger the ||, the better. Motion state data from the most recent T frames is acquired via a time-series sliding window... and compressed into a fixed-dimensional ontology feature vector f using a multilayer perceptron (MLP). proprio , which serves as the ontology feature vector.

[0102] s t The value range of s is [0,1]. t =1 indicates that the organism is completely stable (σ) ω 2 ≈0, ‖e joint ||≈0), s t A value approaching 0 indicates severe instability of the organism.

[0103] Historical confidence sequence H seq It is a time series formed by the confidence values ​​at K decision times, represented as: H seq = [c {t-K+1} , c {t-K+2} , ..., c t ] Among them, c t Let t be the confidence score at time t (i.e., the maximum probability value in the confidence distribution), and K be the sequence length, typically K=10.

[0104] This sequence is used to calculate the trend modulation factor α. trend The historical trend of confidence level is quantified by moving average and variance.

[0105] The formula for calculating the temperature parameter τ(t) is: τ(t) = τ base × α stage × α complex × α trend × α stability in: τ base : Baseline temperature reference value, typically 0.1.

[0106] α stage The stage modulation factor, determined by the task execution progress pt, adopts an exponential decay form. α stage = exp(-β·p t ) Where β is the course attenuation coefficient, typically ranging from 0.5 to 0.8. When p t When =0, α stage =1; when p t When α = 0.5, stage ≈0.60-0.67; when p t When α = 0.9, stage ≈0.41-0.45. This exponential decay pattern allows the temperature to remain at a higher level in the early stages of navigation to facilitate exploration, and then gradually decreases as the mission progresses to achieve precise positioning.

[0107] α complex Complexity modulation factor, determined by the instruction semantic complexity c t It has been decided to adopt a logarithmic compression form: α complex = 1 + γ·ln(1 + c t ) Here, γ is the complexity sensitivity coefficient, typically ranging from 0.2 to 0.5. Logarithmic compression causes the effect of complexity on temperature to increase sublinearly—the temperature rises significantly when instructions change from simple to medium, and the rate of increase slows down when they change from medium to complex.

[0108] α trend Trend modulation factor, derived from the historical confidence sequence H seq The moving average and variance are calculated together: Where μ H σ is the moving average of the historical confidence series. H Let θ be the variance. H ε is the confidence threshold (typically 0.5). Hα is the variance threshold (typically 0.05), δ is the trend response strength (typically 0.3-0.5), and [·] is the indicator function (1 if the condition is met, 0 otherwise). When historical confidence is persistently low (low mean and small variance), the indicator function makes α trend Increase the temperature to trigger enhanced exploration.

[0109] α stability The stability modulation factor is determined by the organism's stability level (st) and follows a linear proportional relationship. α stability =0.5+0.5·s t When s t When α = 1 (the organism is completely stable), stability =1, temperature remains at the baseline value; when st=0 (severe instability of the organism), α stability =0.5, temperature decreases. This design reduces the temperature when the robot is unstable, prompting it to be more cautious.

[0110] Temperature parameters are constrained to a preset effective range using a safety boundary function: τ clipped = max(τ min ,min(τ max , τ(t))) Where, τ min The minimum temperature value to prevent numerical overflow is typically taken as 0.01; τ max To prevent the temperature distribution from becoming overly smooth, a maximum temperature value is typically taken as 0.2. τ(t) is the initial temperature value calculated using the above formula. clipped This refers to the final temperature parameter used.

[0111] Phased course scheduling strategy: The temperature parameter is adjusted in stages during task execution: Exploration phase: When the progress of the task execution is less than the first progress threshold (e.g., p) t When the temperature parameter is less than 0.3, it is maintained within the first temperature range (a relatively high level, such as τ ∈ [0.12, 0.20]) to allow the robot to explore the environment. This stage maintains a high tolerance for potential targets to avoid prematurely locking onto incorrect targets.

[0112] Precise positioning phase: When the task execution progress is greater than the second progress threshold (e.g., p) t When the temperature parameter is greater than 0.7, the temperature parameter is reduced to a second temperature range (a lower level, such as τ ∈ [0.02, 0.06]) to enable the robot to perform precise positioning. During this stage, the confidence distribution is sharp, and the robot will directly lock onto the target with the highest probability.

[0113] Conservative decision-making phase: When any of the following conditions are met, increase the temperature parameter to the third temperature range (between the first and second temperature ranges, such as τ ∈ [0.08, 0.12]), and execute the conservative decision-making operation: The historical confidence value is lower than the historical confidence threshold for K consecutive steps (e.g., confidence < 0.4 for 5 consecutive steps). The organism's stability is below the stability threshold (e.g., s). t <0.6); The quality of environmental observation data is below the visual quality threshold (e.g., number of feature points <100 or ambiguity score >0.7).

[0114] Conservative decision-making practices include at least one of the following: Reduce the speed of motion (e.g., reduce the maximum linear velocity from 0.5 m / s to 0.2 m / s); Reduce stride length (e.g., reduce stride length from 0.3m to 0.15m). Increase the perception frequency (e.g., increase the visual sampling frequency from 15fps to 30fps). Confidence calibration is paused, and pure ontology-aware prediction is performed using extended Kalman filtering based on the ontology feature vector and joint odometry.

[0115] Recovery phase: When the abnormal conditions are eliminated (i.e., the historical confidence value is higher than the threshold, the stability of the organism is higher than the threshold, and the quality of environmental observation data is higher than the threshold), the temperature parameter is restored to the value before the increase, and the conservative decision-making operation is stopped.

[0116] The temperature ranges are related as follows: the minimum value of the first temperature range is greater than the maximum value of the second temperature range; the third temperature range lies between the first and second temperature ranges. Based on actual values: the first range is [0.12, 0.20], the third range is [0.08, 0.12], and the second range is [0.02, 0.06]. Therefore, the minimum value of the first temperature range (0.12) is greater than the maximum value of the second temperature range (0.06), and the third temperature range lies between them (0.08-0.12). Its minimum value (0.08) is greater than the maximum value of the second range (0.06), and its maximum value (0.12) is less than the minimum value of the first range (0.12). Therefore, the minimum value of the first temperature range is greater than the maximum value of the second temperature range; the third temperature range lies between the first and second temperature ranges.

[0117] Step 106: Temperature-based confidence calibration. Using the temperature parameter τ(t), the matching degree between the visual feature vector and the language instruction embedding vector and structured semantic representation is adjusted to obtain the calibrated confidence distribution.

[0118] Calculate the original similarity score (inner product) between the visual feature vector vi and the language instruction embedding vector l: z i = v i ·l Among them, v i Let be the visual feature vector (D1-dimensional) of the i-th candidate view, and l be the language instruction embedding vector (D2-dimensional), typically D1=D2=768. The inner product result z i This reflects the degree of semantic matching between the candidate view and the language instruction; a higher value indicates a higher degree of matching.

[0119] The calibrated probability distribution is calculated using a softmax function with the temperature parameter τ(t): in: P(v i | I, L; τ(t)) represents the confidence probability of the i-th candidate view, constituting the confidence distribution. I represents environmental observation data, L represents natural language commands, τ(t) is the temperature parameter, and M is the total number of candidate views. This probability distribution satisfies the normalization condition.

[0120] Calculate the entropy value of the probability distribution to quantify uncertainty: The entropy value ranges from [0, log M]. The entropy value is maximum (H=log M) when all candidate views have equal probabilities, indicating the model is most uncertain; the entropy value is minimum (H=0) when one candidate view has a probability of 1 and the others have a probability of 0, indicating the model is most certain.

[0121] The confidence gap is obtained by calculating the difference between the highest probability and the second highest probability. ΔP(t) = max(P) - second_max(P) Where max(P) is the maximum probability and second_max(P) is the second maximum probability. The confidence gap ranges from [0, 1]. A larger gap indicates that the model has a higher discrimination against the best choice; a smaller gap indicates that the model hesitates among multiple candidates.

[0122] Step 108: Hierarchical positioning decision, based on the confidence distribution, the positioning decision result is determined, including three levels.

[0123] When the entropy value is less than the first entropy threshold and the confidence gap is greater than the first gap threshold, it is determined that the hierarchical decision will be executed directly. Specific execution includes: The candidate view with the highest probability in the confidence distribution is used as the navigation target. This candidate view corresponds to a three-dimensional spatial coordinate (x, y, z) and orientation information.

[0124] The three-dimensional spatial coordinates of the navigation target are converted into target joint angles. Through inverse kinematics (IK), the target pose is converted into target angle values ​​q for each joint of the robot. target = [q1, q2,..., q N ] T , where N is the number of joint degrees of freedom.

[0125] Based on the target joint angle, Model Predictive Control (MPC) is used to generate a joint spatial trajectory and execute it. MPC optimizes the control sequence in the prediction time domain, so that the actual joint trajectory smoothly tracks the target angle while satisfying joint velocity and acceleration constraints.

[0126] Simultaneously, the current pose is estimated in real time by fusing visual localization results and joint odometry through an extended Kalman filter (EKF). The EKF state vector includes position, velocity, attitude, and IMU bias, while the observation vector includes visual localization results and joint odometry data.

[0127] When visual observation fails (e.g., due to occlusion or sudden changes in lighting), pure ontology-based prediction is performed based on the ontology feature vectors to maintain short-term localization. In this case, relying solely on IMU integration and joint odometry to predict pose will result in decreased localization accuracy over time, but basic localization capability can be maintained for several seconds.

[0128] When the entropy value is greater than or equal to the first entropy threshold and less than or equal to the second entropy threshold, or when the confidence gap is less than the second gap threshold, a local re-inference level decision is determined. Specific execution includes: Pause the current motion and maintain the robot's upright posture. This operation ensures that chained reasoning is performed under stable observation conditions, avoiding motion fuzziness from affecting the quality of reasoning.

[0129] Chain-of-Thought (CoT) is performed based on the language instruction embedding vectors and structured semantic representations to generate intermediate reasoning steps to verify the rationality of the current localization hypothesis. Chain-of-Thought generates a series of intermediate reasoning steps through a Visual-Language Model (VLM), for example: "The window at the end of the corridor is currently visible, but the instruction is to turn left into the living room. In the previous frame, a door was seen on the left side in the middle of the corridor. Therefore, you should rotate 90 degrees to the left to confirm whether the passage on the left leads to the living room." If the verification passes, the decision will be executed in the manner described above, directly executing the hierarchical decision. That is, if the current candidate view is confirmed to be correct, the decision will be downgraded to L1 for execution.

[0130] If the verification fails, the visual feature vector is extracted again from the current environmental observation data, and the confidence level is recalibrated. That is, the current candidate is discarded, and steps 102 to 106 are repeated.

[0131] When the entropy value is greater than the second entropy threshold and the historical confidence average is less than the historical confidence threshold, a global replanning level decision is determined. Specific execution includes: Triggering an update to the Scene Graph. A scene graph is a graph structure where nodes represent entities in the environment (such as rooms, objects, and doors), and edges represent spatial relationships between entities (such as "connected," "contained," or "next to"). Scene graph updates include: Re-identify entities in the current observation; Update the spatial location and attributes of the entity; Reconstruct the topological relationships between entities.

[0132] Based on the updated scene graph, perform an active exploration action. The active exploration action includes at least one of the following: In-situ rotation scanning: The robot rotates 360 degrees in place to collect omnidirectional visual observations and construct a local environment model. Move to the new viewpoint in the direction of the confidence gradient: Calculate the gradient of the confidence in space and move in the direction of increasing confidence. After the active exploration action is completed, the environmental observation data is re-acquired, the visual feature vector is extracted based on the re-acquired environmental observation data, and the confidence calibration is re-performed. That is, under the new viewpoint, steps S1 to S3 are re-executed.

[0133] Step 110: Motion control and closed-loop feedback. Based on the positioning decision result, execute the corresponding motion control operation and combine it with the body feature vector to perform closed-loop feedback, thereby realizing the robot's positioning control.

[0134] The specific motion control operation depends on the decision-making level: L1 direct execution: Executes the joint trajectory obtained by inverse kinematics solution, and achieves smooth motion through MPC; L2 Local Re-reasoning: Pause movement, remain standing, and wait for the reasoning to complete; L3 Global Replanning: Performs proactive exploration actions, including rotation or movement; Closed-loop feedback is achieved through EKF: the actual joint angle is compared with the expected angle, the tracking error is calculated, and the error is fed back to MPC for correction. At the same time, the pose estimated by EKF is fused with the position confidence output by the confidence calibration step to form a complete closed loop of perception-decision-control.

[0135] Compared with the prior art, the present invention has at least the following beneficial effects: Improving Long-Range Navigation and Positioning Accuracy: By employing a course temperature scheduling mechanism, early high temperatures promote exploration, while later low temperatures achieve precise positioning, significantly reducing positioning drift error compared to a fixed-temperature strategy. Experiments show that in R2R-CE continuous environment benchmark tests, the positioning drift error of this invention is reduced by 37% compared to a fixed-temperature strategy (τ=0.07) and by 21% compared to an instance-level adaptive strategy.

[0136] Reduce decision-making oscillations and path redundancy: By monitoring historical confidence sequences, temperature-triggered enhanced exploration is proactively reverted when confidence remains low, reducing unnecessary back-and-forth switching. Experimental data shows that the average number of decision-making switches in semantically ambiguous regions decreased from 2.3 to 0.4, and the redundancy ratio of the average navigation steps compared to the theoretical optimum decreased from 23% to 8%.

[0137] Enhancing the dynamic adaptability of the machine body: By using the machine body stability (angular velocity variance, joint execution error) as a temperature modulation factor, the system automatically reduces temperature smoothing noise during machine body swaying and maintains normal temperature to improve accuracy when the machine body is stationary. Actual test data shows that under swaying conditions with an angular velocity exceeding 0.3 rad / s, the positioning error rate of this invention is reduced by 4.8 times compared to a fixed low-temperature strategy and by 2.1 times compared to a fixed high-temperature strategy.

[0138] Reduced recovery time: Through a collaborative mechanism of three-level decision-making, rapid recovery is possible when localization fails. In scenarios where localization fails due to dynamic obstacle occlusion or sudden changes in lighting, the average recovery time is reduced from 8.7 seconds to 3.2 seconds, and the accuracy of replanning triggering is improved from 64% to 91%.

[0139] Improving the pass rate in safety-critical scenarios: Through a conservative decision-making mechanism, safety operations such as slowing down, reducing stride length, and increasing frequency are automatically triggered in high-risk terrain (stairs, slopes, soft ground). Experiments show that the navigation pass rate in high-risk terrain increased from 78% to 96%, while maintaining an average travel speed of no less than 85% of the conservative strategy, achieving a balance between safety and efficiency.

[0140] Enhanced system interpretability and engineering controllability: Temperature prediction uses explicit mathematical formulas, each modulation factor has a clear physical meaning, supports manual setting of safety boundaries and rapid parameter tuning, and the development iteration cycle is shortened by 40% compared with the black box adaptive method.

[0141] refer to Figure 2 This embodiment takes the execution of natural language navigation tasks by a humanoid robot in an indoor environment as a specific scenario, and fully describes the entire process from multimodal input perception to motion control execution.

[0142] This embodiment uses a humanoid robot with 28 degrees of freedom and bipedal locomotion as an example to illustrate the complete implementation process of the method of the present invention in indoor environment navigation and positioning.

[0143] Scenario: The robot is located at the entrance to the living room. The user inputs a natural language command via voice: "Walk through the living room, go around the sofa, and stop in front of the refrigerator in the kitchen." The robot needs to understand the command, navigate autonomously in the environment, and accurately locate the target position.

[0144] I. Multimodal Input Sensing Steps After the robot is started, each sensor subsystem begins to collect data synchronously.

[0145] Visual perception subsystem: The RGB-D camera on the robot's head begins acquiring environmental images at a sampling rate of 30 frames per second, with an image resolution of 640×480 pixels. The camera simultaneously outputs color and depth images.

[0146] Considering the periodic swaying that occurs during bipedal walking, the visual perception subsystem first performs IMU electronic image stabilization preprocessing on the acquired raw images. Specifically, the system reads the body's angular velocity measured by the IMU, calculates the rotation matrix between adjacent frames, and performs reverse compensation on the images to eliminate image shift caused by swaying. The image after image stabilization preprocessing is then sent to the Vision Transformer encoder.

[0147] The Vision Transformer segments the input image into 16×16 pixel patches. Each patch undergoes linear projection to obtain a patch embedding, which is then encoded with positional information to preserve spatial details. These patch embeddings are processed through a multi-layer multi-head self-attention mechanism, where each layer allows information exchange between patches, enabling the model to capture global visual semantics. The output of the final layer is aggregated to form a 768-dimensional visual feature vector.

[0148] For the current observation scenario, the visual perception subsystem may detect multiple candidate views simultaneously. For example, it may extract potential target regions from an image—such as a corner of a sofa, a coffee table in the living room, or a door frame in the kitchen—using a sliding window or object detection algorithm. Each candidate view corresponds to a 768-dimensional visual feature vector vi, where i is the candidate view index. These visual feature vectors will be used for subsequent matching calculations with language instructions.

[0149] Propriometry Subsystem: Encoders and torque sensors distributed across the robot's 28 joints collect motion state data in real time. The data for each joint includes: joint angle (in radians), joint angular velocity (in radians per second), and actuation torque (in Newton-meters). The motion state data of a single frame thus constitutes an 84-dimensional (3×28) original vector.

[0150] Considering the temporal continuity of motion states, the ontology perception subsystem maintains a 10-frame temporal sliding window. Whenever a new frame of motion state data arrives, the system pushes the new data into the sliding window and removes the oldest frame. Then, the 10 frames of data, totaling 840 dimensions, within the sliding window are concatenated into a large vector.

[0151] This 840-dimensional vector was then fed into a three-layer multilayer perceptron. Each of the three hidden layers of the multilayer perceptron has 512 neurons, and the activation function is ReLU. After compression by the multilayer perceptron, the original 840-dimensional data was reduced to a 512-dimensional ontology feature vector f. proprio The ontology feature vector contains a compressed representation of various motion state information, including the angular deviation of each joint, the variance of angular velocity, and the statistics of execution torque, which will be used as the ontology feature vector in subsequent calculations.

[0152] Language understanding subsystem: The user's voice command, "Walk through the living room, go around the sofa, and stop in front of the refrigerator in the kitchen," is first converted into text through speech recognition. This text is then fed into a pre-trained BERT language model for semantic parsing.

[0153] The BERT model first segments the text into a sequence of lexical units: ["cross", "living room", "around", "sofa", "to", "kitchen", "of", "refrigerator", "before", "stop"]. Then, it processes the text through multiple Transformer encoders, each performing self-attention computation, enabling the model to capture semantic dependencies between lexical units. For example, the model can learn that there is an action-target relationship between "cross" and "living room", and an obstacle avoidance relationship between "around" and "sofa".

[0154] The BERT model outputs two key results. The first result is a 768-dimensional language instruction embedding vector l, which is a dense semantic feature vector that captures the semantic information of the entire instruction and will be used to calculate similarity with the visual feature vector.

[0155] The second result is a structured semantic representation, which is an executable, structured representation. Through semantic role labeling and dependency parsing, the system extracts the following information from the instructions: Target object sequence: ["living room", "sofa", "kitchen", "refrigerator"], listing the target locations that need to be reached or passed through in order; Spatial constraint types: ["Pass through" (corresponding to the living room), "Go around" (corresponding to the sofa), "Arrive" (corresponding to the kitchen), "Stop" (corresponding to the refrigerator)]; Target nesting level: Analysis shows that "the refrigerator in the kitchen" contains two nesting levels, "the sofa" is a single level, and "the living room" is a single level. The overall maximum nesting depth is 2. Number of condition constraints: If the instruction contains the termination condition "stop", the condition constraint count is 1; This structured information will be used to calculate the progress of the navigation phase and the semantic complexity of the instructions.

[0156] II. Temperature Control Steps The course temperature scheduling network receives input signals from four dimensions and dynamically calculates the temperature parameters at the current moment.

[0157] Task execution progress calculation: Task execution progress p t This indicates the proportion of completed instruction fragments out of the total number of instructions. The system first parses the total number of instruction fragments, `total_steps`, from the structured semantic representation. Based on the target object sequence [living room, sofa, kitchen, refrigerator], the system divides the navigation task into four sub-tasks: reaching the living room, bypassing the sofa, reaching the kitchen, and stopping before the refrigerator; therefore, `total_steps` equals 4.

[0158] The system tracks the number of completed instruction segments (completed_steps) by monitoring the execution status of the target object sequence in the structured semantic representation. Initially, the robot has not completed any subtasks, so completed_steps equals 0.

[0159] Task execution progress p t The calculation formula is: p t = completed_steps / total_steps = 0 / 4 = 0 As navigation progresses, when the robot reaches the living room, completed_steps is updated to 1 and pt is updated to 0.25; after bypassing the sofa, pt is updated to 0.5; and so on.

[0160] Instruction semantic complexity calculation: The system extracts three dimensions of metrics from the original text corresponding to the language instruction embedding vectors and calculates the semantic complexity c of the instructions. t .

[0161] The first dimension is the number of spatial relation words, N. spatial The system matches commands using a predefined dictionary of spatial relation words (containing words like "through", "around", "along", "passing", "entering", and "leaving"). In the command "Walk through the living room, around the sofa, and stop in front of the refrigerator in the kitchen," the spatial relation words "through" and "around" are matched, therefore N... spatial It equals 2.

[0162] The second dimension is the number of nested levels N of the target. nesting The system calculates nesting depth by analyzing the sequence of target objects and their hierarchical relationships in the structured semantic representation. For example, "refrigerator in the kitchen" means the refrigerator belongs to the kitchen, with a nesting depth of 2; "sofa" has no hierarchical relationship, with a depth of 1; and "living room" has a depth of 1. The maximum nesting depth in the entire instruction is taken as N. esting Therefore N nesting It equals 2.

[0163] The third dimension is the number of constraints, N. condition The system calculates by recognizing conditional words in the instructions (such as "avoid," "before," "stop," "after," etc.). "Stop" is a termination condition constraint, therefore N condition It equals 1.

[0164] With weighting coefficients w1=0.3, w2=0.4, and w3=0.3, the instruction semantic complexity is c. t The calculation formula is: c t =0.3×2 +0.4×2 + 0.3×1= 0.6+0.8+0.3 = 1.7 Calculation of organism stability: The system starts from the ontology feature vector f proprio Extract the angular velocity component and joint angle deviation component, and calculate the body stability s t .

[0165] The angular velocity components are derived from IMU measurements. The system extracts the triaxial angular velocity sequence (ω) within a time window. x , ω y ,ω z ), calculate its variance σ ω 2 When the robot is in a static standing state and has not yet begun to walk, its angular velocity is close to zero, and its variance is also close to zero. Therefore, σ ω 2 ≈ 0.

[0166] Joint execution error norm ||e joint The deviation between the expected angle and the actual angle from each joint. The system extracts the deviation value e for all joints from the ontology feature vector. i(i=1 to 28), calculate its L2 norm: ||e joint || = sqrt(Σ e i 2 In the initial static standing state, the joint execution accuracy is high, and the deviation is very small. joint || ≈ 0.

[0167] Body stability s t The calculation formula is: s t = 1 / (1 + σ ω 2 + ‖e joint ||) ≈ 1 / (1 + 0 + 0) = 1 This means that the robot's body is completely stable in the initial state.

[0168] Historical confidence sequence maintenance: The system maintains a queue of length K=10 to store the confidence values ​​for the most recent 10 decision times. The queue update strategy is as follows: after each confidence calibration, the maximum probability value at the current time (i.e., the peak value in the confidence distribution) is pushed into the queue, and the oldest value is removed.

[0169] In the initial state, all values ​​in the queue are set to 0.5 (neutral confidence level). The historical mean confidence level at this time is μ. H =0.5, variance σ H Approaching zero.

[0170] Temperature parameter calculation: The system calculates the initial temperature value according to the temperature parameter formula based on the variables calculated above.

[0171] Let the base temperature τ base =0.1, course attenuation coefficient β=0.6, complexity sensitivity coefficient γ=0.3, trend response strength δ=0.4, confidence threshold θ H =0.5, variance threshold ε H =0.05.

[0172] Phase modulation factor α stage = exp(-β·p t ) = exp(0) = 1.

[0173] Complexity modulation factor α complex = 1 + γ·ln(1 + c t ) = 1 + 0.3 × ln(2.7) ≈ 1 +0.3 × 0.99 ≈ 1.30.

[0174] Trend modulation factor α trend The calculation involves conditional judgments. The current historical confidence level mean μ H =0.5, not less than θ H =0.5; variance σ H =0, less than ε H =0.05, but does not meet the condition of "mean less than threshold", therefore the indicator function value is 0, α trend =1.

[0175] Stability modulation factor α stability = 0.5 + 0.5·s t = 0.5 + 0.5×1 = 1.

[0176] Initial temperature value τ initial = 0.1 × 1 × 1.30 × 1 × 1 = 0.13.

[0177] The system then constrains the initial temperature value to a preset effective range [0.01, 0.2] using a safety boundary function. Since 0.13 falls within this range, the final temperature parameter τ(t) = 0.13.

[0178] This temperature value is at a high level, which means that the model will maintain a high degree of exploratoryness in the early stages of navigation and will not lock onto a single target too early.

[0179] III. Temperature-based Reliability Calibration Procedure The system uses the calculated temperature parameter τ(t)=0.13 to calibrate the matching degree between visual features and language instruction embeddings, and obtains the confidence distribution.

[0180] Original similarity calculation: For each candidate view's visual feature vector v i The system calculates the inner product of the language instruction embedding vector l with the original similarity score: z. i = v i · l Because of v i Both and l are 768-dimensional vectors, and the inner product calculation involves 768 multiplications and 767 additions. The computational complexity of this process is O(d), where d = 768, which is a lightweight operation that can be completed in milliseconds.

[0181] For example, suppose there are three candidate views in the current field of view: a corner of the sofa (v1), the coffee table in the living room (v2), and the door frame of the kitchen (v3). The system calculates the similarity between each of them and the language instruction embedding l: z1=0.65, z2=0.32, z3=0.48. These scores indicate that the sofa has the highest semantic matching degree with the instruction, followed by the kitchen door frame, and the coffee table has the lowest.

[0182] Temperature scaling probability calculation: The system converts the original similarity scores into a probability distribution using a softmax function with a temperature parameter. P(v i | I, L; τ) = exp(z i / τ) / Σ j exp(z j / τ) First, calculate the exponential similarity of each candidate view. Since τ=0.13, z i The value of / τ is relatively large: z1 / τ = 0.65 / 0.13 = 5.0, exp(5.0) ≈ 148.4 z2 / τ = 0.32 / 0.13 ≈ 2.46, exp(2.46) ≈ 11.7 z3 / τ = 0.48 / 0.13 ≈ 3.69, exp(3.69) ≈ 40.0 The denominator is the sum of the three: 148.4 + 11.7 + 40.0 = 200.1 The final probability distribution is: P(v1) = 148.4 / 200.1 ≈ 0.742 P(v2) = 11.7 / 200.1 ≈ 0.058 P(v3) = 40.0 / 200.1 ≈ 0.200 Calculation of the gap between entropy and confidence level: The system calculates the entropy value of the probability distribution to quantify uncertainty: H(p) = -Σ i P(v i log2P(v) i ) First term: -0.742 × log2(0.742) = -0.742 × (-0.430) = 0.319 The second term: -0.058 × log2(0.058) = -0.058 × (-4.107) = 0.238 The third term: -0.200 × log2(0.200) = -0.200 × (-2.322) = 0.464 H = 0.319 + 0.238 + 0.464 ≈ 1.021 (with a logarithmic base of 2, in bits) The system calculates the confidence gap, which is the difference between the highest probability and the second highest probability: ΔP(t) = max(P)-second_max(P) = 0.742-0.200 = 0.542 The large gap indicates that the model has a clear preference for the best candidate (the sofa).

[0183] IV. Decision-making steps for hierarchical positioning The system determines the decision level based on the entropy value H=1.021 and the confidence gap ΔP(t)=0.542, combined with a preset threshold.

[0184] The preset threshold is set as: low entropy threshold θ low =0.8 bits, high entropy threshold θ high =1.5 bits, confidence gap threshold θ gap =0.3.

[0185] Judgment process: H(t) = 1.021 > θ low =0.8, which does not satisfy the L1 condition (H < 0.8 is required). H(t) = 1.021 < θ high =1.5, which satisfies the entropy condition of L2 (θ). low ≤ H ≤ θ high ) Furthermore, ΔP(t) = 0.542 > θ gap =0.3, which does not meet the gap condition of L2, but the trigger condition of L2 is "OR", so only one condition needs to be met.

[0186] The system's judgment result is that the entropy value is in the medium range, and the model has a certain degree of uncertainty, thus triggering the L2-local re-inference level decision.

[0187] L2-Local Re-inference Hierarchical Execution: The system first pauses the robot's current movement and maintains its standing posture. This operation ensures that chained reasoning is performed under stable observation conditions, avoiding motion ambiguity from affecting the quality of reasoning.

[0188] Then, the system activates the chain-thinking ability of the visual-language model. The model receives the following inputs: Current visual observations: RGB image and depth map; Current candidate views and their confidence distribution: sofa (74.2%), kitchen door frame (20.0%), coffee table (5.8%); Historical trajectory information: Records of the most recent steps; Natural language command: "Walk through the living room, go around the sofa, and stop in front of the refrigerator in the kitchen."

[0189] Visual-language models validate the validity of the current localization hypothesis by generating intermediate inference steps. The model may generate the following inference chain: "The first step is to 'cross the living room' first. The candidate with the highest confidence is the sofa, but the sofa is not the target for 'crossing' the living room. The main characteristics of the living room are its open space, with a TV wall and a coffee table. The second step is to review the historical trajectory. The robot is currently at the entrance, with the living room space in front of it. The third step is to observe the current image. Although the entire living room is not shown, the TV wall on the left and the window on the right can be seen, which matches the characteristics of a living room. The fourth step is to determine: the robot should prioritize identifying the entire living room rather than a single sofa. The confidence distribution of the current candidate view may have an issue and needs adjustment." After the reasoning is completed, the system performs hypothesis verification. The reasoning finds that the current candidate view (sofa) does not match the first stage requirement of the instruction (passing through the living room), so the verification result is "fail".

[0190] Based on the failed verification results, the system re-extracts visual feature vectors from the current environmental observation data. Specifically, the system expands the scope of visual feature extraction, no longer limiting itself to candidate boxes for object detection, but performing more comprehensive semantic segmentation on the entire image to extract more environmental entities (such as TV walls, living room windows, coffee tables, carpets, etc.). Then, the system recalibrates the confidence level and updates the candidate view set.

[0191] V. Motion Control Execution and Closed-Loop Feedback After recalibration, the new confidence distribution identifies the entire living room area as a high-probability candidate. The system then determines that the entropy value has decreased to 0.65 and the confidence gap has increased to 0.68, satisfying the L1 condition.

[0192] L1 - Direct execution at the hierarchical level: The system identifies the candidate view with the highest probability from the confidence distribution—the spatial coordinates of the living room area (x=2.5m, y=0m, z=0.5m).

[0193] Then, inverse kinematics is used to convert the three-dimensional spatial coordinates into target angles for the 28 joints. The inverse kinematics solution process involves building a robot kinematic model and deriving the joint angles from the target pose. Since the robot has redundant degrees of freedom (28 joints, far more than the 6 degrees of freedom required for end-effector localization), there are multiple solutions for the inverse kinematics solution. The system selects the solution with the minimum energy consumption, i.e., the solution with the smallest change in joint angles.

[0194] The system utilizes model predictive control (MMCC) to generate smooth joint spatial trajectories. MMCC optimizes the control sequence within the prediction time domain (e.g., one second in the future) to ensure that the actual joint trajectory smoothly tracks the target angle while satisfying joint velocity, acceleration, and torque constraints. After trajectory generation, the system sends commands to the underlying joint controller for execution.

[0195] During motion, the system estimates the current pose in real time by fusing visual positioning results with joint odometry using an extended Kalman filter. The state vector of the extended Kalman filter includes 3D position, 3D velocity, attitude quaternions, and IMU bias. The prediction step updates the state using joint odometry integration, and the update step corrects accumulated errors using visual positioning results.

[0196] When the robot walks, if the body sways due to uneven ground, the system detects an increase in angular velocity variance and a decrease in body stability through body perception. It then automatically adjusts the temperature parameters and executes conservative decision-making operations (reducing walking speed and shortening stride) to ensure walking stability.

[0197] Once the robot reaches the center of the living room, the system updates the task execution progress. t At 0.25, the temperature parameter τ(t) is recalculated to a lower level, and preparations begin for the next stage of the task—bypassing the sofa.

[0198] VI. Brief Description of Subsequent Stages During the sofa-avoidance phase, the system uses the updated scene map (which now includes the sofa's precise location and outline) to plan a path from the center of the living room around the sofa to the kitchen entrance. The system calculates obstacle avoidance paths on both sides of the sofa and selects the side with the shorter distance and more open space to execute.

[0199] Upon reaching the kitchen, the system identifies the kitchen door through visual feature matching and locates the refrigerator after entering the kitchen.

[0200] Finally, during the docking phase with the refrigerator, the system temperature parameter is reduced to the lowest level (approximately 0.03), and the confidence distribution is extremely sharp, ensuring that the robot can accurately dock at the designated position in front of the refrigerator and complete the entire navigation task.

[0201] VII. Example of Exception Handling The system can respond automatically if any abnormal situation occurs during navigation.

[0202] For example, when the robot is walking around a sofa, its view may suddenly be obstructed by a dynamic obstacle (such as a moving pet), causing a significant decrease in visual observation quality. The proprioception subsystem detects a sharp decrease in the number of image feature points (from the normal 800 to less than 200), triggering a visual quality threshold judgment.

[0203] The system automatically increases the temperature parameter to the third temperature range (approximately 0.10) and performs conservative decision-making operations: reducing movement speed and increasing sensing frequency. At the same time, the system pauses confidence calibration and switches to a pure proprioception prediction mode, relying solely on joint odometry and IMU data for short-term localization (lasting approximately 3-5 seconds).

[0204] Once the dynamic obstacle leaves the field of view, the visual observation quality is restored, the system reactivates the confidence calibration, the temperature parameter is restored to its value before the callback, and the navigation task continues.

[0205] For example, when the robot wobbles due to slippery ground, the proprioception subsystem detects a significant increase in the variance of the angular velocity (σ). ω ²Increased from approximately 0.01 to 0.15), indicating a high degree of stability in the organism. t The probability distribution decreases from nearly 1 to approximately 0.87. The system automatically increases the temperature parameter to smooth the probability distribution, avoiding falsely high confidence levels for blurry images captured during shaking, thus preventing incorrect decisions based on erroneous observations.

[0206] This embodiment details the complete implementation process of the method of the present invention in indoor navigation and positioning of humanoid robots, including five core steps: multimodal input perception, course temperature scheduling, temperature shrinkage and confidence calibration, hierarchical positioning decision, and motion control execution, as well as the anomaly handling mechanism in the navigation process.

[0207] Through this embodiment, those skilled in the art can clearly understand the technical solution of the present invention and reproduce the present invention without creative effort, thereby realizing the function of autonomous navigation and positioning of the robot based on natural language instructions. The parameter settings described in this embodiment (such as temperature threshold 0.01-0.2, progress threshold 0.3 and 0.7, entropy threshold 0.8 and 1.5, confidence gap threshold 0.3, etc.) are merely illustrative examples. In practical applications, those skilled in the art can make appropriate adjustments to these parameters according to the specific robot platform, environmental conditions, and task requirements, and such adjustments do not depart from the protection scope of the present invention.

[0208] In addition to the preferred embodiments described above, those skilled in the art can also employ the following alternative solutions to achieve the same or similar technical objectives. Although these alternative solutions differ from the core solution of this invention in their technical path, they can all solve the technical problems to be addressed by this invention (fixed temperature parameters cannot meet the phased needs of navigation, decoupling of temperature and body perception, separation of temperature and decision-making mechanisms, etc.). Those skilled in the art can select appropriate implementation methods according to specific application scenarios.

[0209] I. Alternative solutions for determining temperature parameters Alternative Solution A: Temperature Policy Optimization Based on Reinforcement Learning This scheme models the selection of temperature parameters as a Markov Decision Process (MDP). The state space is defined as including current visual features, embedded language commands, historical trajectories, and the organism's motion state; the action space is either continuous temperature values ​​or discrete temperature levels; and the reward function is a weighted sum of navigation success rate, step efficiency, and organism stability. The policy network is trained using either Proximal Policy Optimization (PPO) or Soft Actor-Critic (SAC) algorithms, enabling the policy network to output the optimal temperature parameters based on the current state.

[0210] The advantage of this scheme is that it eliminates the need for manually designed modulation factors, allowing the policy network to automatically learn the optimal values ​​of temperature parameters in complex scenarios. Its limitations include: a large training sample requirement, necessitating the collection of substantial interactive data on actual robot platforms or in high-fidelity simulation environments; the trained policy network is a black-box model with poor interpretability, making it difficult to trace the basis for temperature decisions; and the need for retraining or domain adaptation when applied in new environments. In scenarios requiring high sample efficiency and interpretability, the preferred scheme of this invention (explicit mathematical formula) offers significant advantages.

[0211] Alternative Solution B: Temperature Adaptation Based on Bayesian Optimization This scheme treats the temperature parameter as a hyperparameter to be optimized, and uses a Bayesian optimization framework to dynamically update the posterior distribution of the temperature parameter based on the navigation performance observed online. The acquisition function can employ Expected Improvement (EI) or Upper Confidence Bound (UCB) to strike a balance between exploration and utilization. The core of this scheme lies in using a Gaussian process to model the objective function, selecting the next temperature value for evaluation in each iteration based on the acquisition function, and updating the posterior distribution based on the evaluation results.

[0212] The advantage of this scheme lies in its ability to quantify the uncertainty of temperature selection, exhibiting good sample efficiency in scenarios with few navigation tasks and high data costs. Its limitations include: significant computational overhead, requiring optimization of the acquisition function for each temperature update; the curse of dimensionality in Gaussian process regression in high-dimensional state spaces, making it difficult to handle complex state inputs; and real-time performance that cannot meet the 100-millisecond control cycle requirements of humanoid robots. In scenarios with high real-time requirements, the preferred scheme (explicit mathematical formula) of this invention can be calculated in milliseconds, offering a significant advantage.

[0213] Alternative Solution C: Expert System Based on Rule Engine This solution uses pre-defined temperature switching rules by domain experts to form a rule base. The rules are formatted as follows: "If navigation progress is less than 30% and the robot is stable, set the temperature to 0.15; if navigation progress is greater than 70%, set the temperature to 0.05; if the robot is unstable, set the temperature to 0.12." The rule engine monitors the robot's state in real time and outputs temperature parameters based on the matching rules. Rules can be implemented using decision trees, finite state machines, or generative systems.

[0214] The advantages of this scheme are its simplicity, high computational efficiency, and strong interpretability, allowing engineers to intuitively understand the meaning and triggering conditions of each rule. Its limitations include: the completeness of the rules heavily relies on expert experience, making it difficult to cover all combinations of complex scenarios; designing conflict resolution strategies is challenging when multiple rules are triggered simultaneously; long-term maintenance costs are high, and new scenarios require manual rule additions. In application scenarios with extremely high interpretability requirements and predictable scenarios, this scheme can serve as a supplement to the preferred solution of this invention.

[0215] II. Alternatives to Confidence Calibration Methods Alternative D: Alignment confidence estimation based on energy model This scheme abandons the temperature scaling mechanism and adopts an implicit probability estimation method based on an Energy-Based Model (EBM). Specifically, a joint energy function E(v, l) is constructed, mapping the matching degree between visual feature v and linguistic feature l to an energy value—the higher the matching degree, the lower the energy value. The confidence score is no longer calculated through softmax normalization, but rather by estimating the partition function through Langevin dynamics or contrastive divergence sampling, thereby calculating the normalized confidence score. P(v i | l) = exp(-E(v i , l)) / Σj exp(-E(v j , l)) The theoretical advantage of this scheme lies in its ability to model complex multimodal distributions using an energy model, capturing high-order dependencies in visual-linguistic alignment and exhibiting stronger expressive power in semantically ambiguous scenarios. Its limitations include: high computational overhead in the sampling process, making it difficult to meet real-time requirements; high estimation variance of the partition function, making it difficult to guarantee the stability of confidence calibration; and high engineering integration complexity with the humanoid robot control system. In scenarios with sufficient computational resources and highly semantically ambiguous scenarios, this scheme can serve as an alternative to the preferred scheme of this invention.

[0216] Alternative E: Cross-view relation reasoning based on graph neural networks This scheme constructs multiple candidate views into a graph structure G=(V, E), where node V is the visual feature vector of each candidate view, and edge E represents the spatial adjacency or semantic similarity between views. Through multi-layer message passing in a graph neural network (GNN), each node can aggregate information from its neighbors, thereby enhancing its feature representation by incorporating context. The enhanced view features are used to calculate the confidence score. P(v i ) = softmax(MLP(GNN_embedding(v i ))) The advantage of this scheme lies in its explicit modeling of the spatial and semantic relationships between views, enabling the use of structural priors to correct isolation errors caused by occlusion and viewpoint changes. For example, when a view's features are indistinct due to occlusion, information from its neighboring views can help correct the view's confidence level. Its limitations include: the graph construction and message passing processes introduce additional computational latency; the graph structure dynamically updates with the robot's movement and environmental changes, resulting in high complexity for online updates; and the fusion path with the humanoid robot's body perception signals is unclear. In scenarios where the relationships between views are strong and real-time requirements are relatively relaxed, this scheme can serve as an alternative to the preferred solution of this invention.

[0217] III. Alternative Solutions for Implementing the Decision-Making Mechanism Alternative Solution F: Proactive Perception Decision Based on Monte Carlo Tree Search This approach models navigation decisions as a Partially Observable Markov Decision Process (POMDP), where the robot cannot directly observe the complete environmental state but can only update its belief state—the distribution of beliefs about the environmental state—through observation. Monte Carlo Tree Search (MCTS) is used to plan within the belief space, and the long-term value of different decision paths is evaluated by simulating future observations and actions. When selecting an active exploration action, MCTS chooses the action that maximizes the expected information gain.

[0218] The theoretical advantage of this scheme lies in its ability to perform multi-step look-ahead planning, finding the globally optimal exploration path in complex mazes or scenarios requiring long-term strategies. Its limitations include: computational complexity increases exponentially with planning depth and branching factors, making it difficult to meet real-time requirements; the maintenance and updating of belief states incurs significant computational overhead; and the combination with the high-dimensional action space of humanoid robots faces the curse of dimensionality. In scenarios with sufficient computational resources and long-term task dependencies, this scheme can serve as an alternative to the global replanning layer (L3).

[0219] Alternative G: Hierarchical Reinforcement Learning Based on an Option Framework This scheme employs an options framework, decomposing navigation decisions into two levels: a high-level meta-policy responsible for selecting sub-goals or decision modes, and a low-level sub-policy responsible for executing specific action sequences. The decision space of the high-level policy includes abstract options such as direct execution, local re-inference, and global replanning, with its input being the current confidence distribution and ontology perception state. The low-level policies correspond to the specific implementations of each option; for example, the low-level policy under the "direct execution" option might be trajectory tracking based on MPC, while the low-level policy under the "global replanning" option might be active exploration based on MCTS.

[0220] The advantage of this scheme lies in its ability to automatically discover the optimal temperature-decision combination pattern without requiring manual design of the trigger thresholds for the three-level decision-making process. Through end-to-end reinforcement learning training, the high-level policy can learn when to choose which decision pattern. Its limitations include: option discovery is an open problem, typically requiring a predefined set of options; poor training stability and low sample efficiency, especially in high-dimensional state spaces; and difficulties in interpreting and verifying the security of the high-level policy, making it hard to ensure that the policy behaves as expected in all scenarios. In scenarios with relatively low interpretability requirements and sufficient training data, this scheme can serve as an alternative to the preferred scheme of this invention.

[0221] IV. Alternatives to Active Exploration Strategies Alternative H: Exploration strategy based on maximizing information gain This scheme does not employ in-situ rotation or movement towards the confidence gradient at the global replanning level. Instead, it explicitly calculates the expected information gain for each candidate exploration action and selects the action with the highest information gain for execution. The expected information gain is defined as the anticipated reduction in the confidence distribution entropy after the action is executed. IG(a) = H(current) - E[H(after action a)] The system evaluates the expected information gain for each candidate exploration action (such as rotating 30 degrees to the left, rotating 30 degrees to the right, moving forward 1 meter, or scanning 360 degrees in place). The evaluation process can be performed through rapid simulation using the current scene graph and environment model, or through direct prediction using a lightweight neural network. Executing the exploration action with the highest information gain allows for the acquisition of the most environmental information with the least action cost, thereby improving the efficiency and effectiveness of global replanning.

[0222] The advantage of this approach lies in its purposeful exploration actions—acquiring information rather than blindly moving or rotating. It is particularly advantageous in complex environments requiring efficient exploration (such as large warehouses or multi-story buildings). Its limitation is that accurate estimation of the expected information gain relies on relatively reliable prior environmental knowledge or a scene graph, while global replanning is triggered precisely when environmental cognition may have become ineffective, creating a certain contradiction. In scenarios where prior environmental information is relatively reliable, this approach can serve as an enhanced implementation of the L3 active exploration strategy.

[0223] Alternative Solution I: Active Exploration Strategy Based on Random Exploration This scheme represents the simplest proactive exploration strategy. When global replanning is triggered, the robot does not rely on scene graphs or confidence gradients, but instead executes exploration actions in a random manner, such as randomly selecting rotation direction, randomly selecting movement distance, and randomly generating exploration path points. After exploration is completed, observations are reacquired and confidence is recalibrated.

[0224] The advantage of this approach lies in its extreme simplicity, requiring no complex calculations or state estimations, and its high robustness—random exploration may still succeed even if the scene graph is completely flawed or the confidence gradient cannot be estimated. Its limitation is low exploration efficiency; in larger environments, multiple random explorations may be required to obtain valid observations, leading to long recovery times. In scenarios with relatively simple environments or extremely limited computational resources, this approach can serve as the simplest implementation of L3 active exploration.

[0225] V. Alternative Solutions to Ontology Perception Fusion Methods Alternative J: Implicit fusion based on end-to-end multimodal Transformer This approach eliminates the need for explicit temperature scheduling and proprioceptive feature extraction modules. Instead, it directly concatenates visual images, spoken commands, joint encoder readings, and IMU data into a sequence input, which is then processed end-to-end by a large-scale multimodal Transformer (such as RT-2, PaLM-E, and other basic robot models). This solution eliminates the need for manually designing temperature calculation formulas, feature extraction networks, and modulation factors. The model automatically learns through large-scale pre-training how to fuse multimodal information, adjust confidence and sharpness, and generate motion commands. Temperature parameters are no longer explicit intermediate variables but are implicitly included in the Transformer's self-attention calculations.

[0226] The advantage of this approach lies in its ability to leverage pre-trained knowledge at the internet scale, generalizing to unseen objects and tasks, and exhibiting strong transfer capabilities in zero-shot or few-shot scenarios. Its limitations include: massive model size (typically in the billions to hundreds of billions of parameters), high inference latency (seconds or even longer), making it difficult to meet the real-time control requirements of humanoid robots; uncontrollable vision-language alignment, making it impossible to explicitly adjust confidence levels; and difficulty in effectively encoding the ontological constraints unique to humanoid robots (such as bipedal balance and joint limitations). In scenarios with sufficient computing resources and lower real-time requirements (such as cloud robots and remote control), this approach can serve as a long-term evolution direction for this invention.

[0227] Alternative Solution K: State-Confidence Joint Estimation Based on Kalman Filtering This scheme jointly models the robot's localization state and visual-language alignment confidence as a state vector, which is then estimated online using either an Extended Kalman Filter (EKF) or an Unscented Kalman Filter (UKF). The state vector x includes position, velocity, pose, IMU bias, and confidence scores for each candidate view. The observation model includes visual-language matching scores (from CLIP or a similar model) and joint odometry measurements. The filter's prediction step updates the pose and confidence scores using the motion model (based on joint odometry), while the update step refines the state estimate using the observations.

[0228] The theoretical advantage of this scheme lies in its ability to provide optimal state estimation and explicitly quantify the uncertainty (covariance matrix) of linear or weakly nonlinear systems where noise is assumed to follow a Gaussian distribution. Jointly estimating the localization state and confidence level allows for mutual verification—when the localization state and confidence level are inconsistent, the covariance matrix naturally increases, triggering conservative decision-making. Its limitations include: the non-Gaussian and multimodal characteristics of visual-language matching are difficult to accurately describe using linear or Gaussian models; when the number of candidate views is large, the dimensionality of the state vector increases dramatically, and the computational complexity of maintaining and updating the covariance matrix grows quadratically; nonlinear observation models require the use of UKF or particle filtering, further increasing computational complexity. In scenarios with low state space dimensionality and approximately Gaussian noise, this scheme can serve as a supplementary implementation of the ontology perception fusion method of this invention.

[0229] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0230] According to an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the robot positioning and control method described in any of the above embodiments.

[0231] An electronic device according to an embodiment of the present invention includes a processor and a memory, wherein a computer program is stored in the memory, and when the processor executes the computer program, it implements the robot positioning and control method described in any of the above embodiments.

[0232] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0233] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A robot positioning and control method, characterized in that, Includes the following steps: Acquire the robot's environmental observation data, motion state data, and natural language commands, and extract the corresponding visual feature vectors, ontological feature vectors, language command embedding vectors, and structured semantic representations respectively; The current temperature parameter is determined based on the language instruction embedding vector and structured semantic representation, the ontology feature vector, the task execution progress, and the historical confidence sequence. Using the temperature parameter, the matching degree between the visual feature vector, the language instruction embedding vector, and the structured semantic representation is adjusted to obtain the calibrated confidence distribution; The location decision result is determined based on the confidence distribution; Based on the positioning decision result, corresponding motion control operations are executed to achieve robot positioning control.

2. The robot positioning and control method according to claim 1, characterized in that, The task execution progress includes the navigation phase progress; The step of determining the current temperature parameter based on the language instruction embedding vector and structured semantic representation, the ontology feature vector, task execution progress, and historical confidence sequence includes: The navigation phase progress is calculated based on the structured semantic representation and the number of executed instruction fragments. The instruction semantic complexity is calculated based on the language instruction embedding vector. The body stability is determined based on the ontology feature vector. The current temperature parameter is determined based on the navigation phase progress, the instruction semantic complexity, the historical confidence sequence, and the body stability. The historical confidence sequence is a sequence formed by confidence values ​​at K time points.

3. The robot positioning and control method according to claim 2, characterized in that, The step of determining the current temperature parameter based on the navigation phase progress, the command semantic complexity, the historical confidence sequence, and the aircraft stability includes: The initial temperature value is calculated based on the preset base temperature, stage modulation factor, complexity modulation factor, trend modulation factor, and stability modulation factor. The initial temperature value is constrained to a preset effective range using a safety boundary function to obtain the temperature parameter; The phase modulation factor is determined based on the progress of the navigation phase, the complexity modulation factor is determined based on the semantic complexity of the instruction, the trend modulation factor is determined based on the historical confidence sequence, and the stability modulation factor is determined based on the stability of the organism.

4. The robot positioning and control method according to claim 3, characterized in that, The minimum value of the preset valid interval is 0.01, and the maximum value is 0.2; The formula for calculating the temperature parameter τ(t) is as follows: τ(t)=τ base ×a stage ×a complex ×a trend ×a stability Where, τ base The base temperature; α stage The stage modulation factor is determined by the navigation stage progress p. t It was decided to adopt the exponential decay form α. stage =exp(-β·p t ), where β is the course decay coefficient; α complex The complexity modulation factor is determined by the instruction semantic complexity c. t It was decided to adopt the logarithmic compression form α. complex =1+γ·ln(1+c t ), where γ is the complexity sensitivity coefficient; α trend The trend modulation factor is calculated by combining the moving average and variance of the corresponding sequence of the historical confidence sequence; α stability The stability modulation factor is determined by the body's stability level s. t It is decided to adopt the linear inverse form α. stability =1-κ·(1-s t ), where κ is the stability response coefficient.

5. The robot positioning and control method according to claim 3, characterized in that, The navigation phase progress p t The calculation method is as follows: p t =completed_steps / total_steps, where completed_steps is the number of instruction fragments that have been executed, and total_steps is obtained by parsing from the structured semantic representation; The semantic complexity of the instruction is c t The calculation method is as follows: c t =w1·N spatial +w2·N nesting +w3·N condition , where N spatial N represents the number of spatial relation words extracted from the language instruction embedding vector corresponding to the instruction. nesting N represents the nesting level of the target object. condition The number of conditional constraints; The stability of the body s t The calculation formula is: s t =1 / (1+σ ω 2 +‖e joint ‖) Where, σ ω 2 Let ‖e be the angular velocity variance extracted from the body feature vector. joint ‖ represents the joint execution error norm extracted from the ontology feature vector.

6. The robot positioning and control method according to claim 2, characterized in that, The temperature parameter is adjusted in stages during task execution: When the task execution progress is less than the first progress threshold, the temperature parameter is maintained in the first temperature range so that the robot can explore the environment. When the progress of the task execution exceeds the second progress threshold, the temperature parameter is reduced to the second temperature range to enable the robot to perform precise positioning. When the historical confidence value is lower than the historical confidence threshold for K consecutive steps, or the stability of the organism is lower than the stability threshold, or the quality of the environmental observation data is lower than the visual quality threshold, the temperature parameter is increased to the third temperature range, and a conservative decision-making operation is performed. When the historical confidence value is higher than the historical confidence threshold, or the stability of the organism is higher than the stability threshold, or the quality of the environmental observation data is higher than the visual quality threshold, the temperature parameter is restored to the value before the increase, and the conservative decision-making operation is stopped. Wherein, the minimum value of the first temperature range is greater than the maximum value of the second temperature range; the third temperature range is between the first temperature range and the second temperature range, and the conservative decision operation includes at least one of the following: reducing movement speed, reducing stride length, increasing perception frequency, pausing confidence calibration, and performing pure ontology perception prediction based on the ontology feature vector and joint odometry through extended Kalman filtering.

7. The robot positioning and control method according to claim 2, characterized in that, Using the temperature parameter, the matching degree between the visual feature vector, the language instruction embedding vector, and the structured semantic representation is adjusted to obtain the calibrated confidence distribution, including: The original similarity between the visual feature vector and the language embedding vector is scaled using the temperature parameter to obtain a calibrated confidence distribution. The entropy and confidence gap of the confidence distribution are then calculated to generate a localization decision based on the entropy and confidence gap of the confidence distribution.

8. The robot positioning and control method according to claim 7, characterized in that, The confidence distribution is calculated as follows: Calculate the visual feature vector v i The original similarity score with the language instruction embedding vector l: ； The calibrated probability distribution is calculated using a softmax function with the temperature parameter τ(t): ； in, The confidence probability of the i-th candidate view constitutes the confidence distribution, z. i Let be the i-th original similarity score between visual features and language embedding, τ(t) be the temperature parameter, and M be the total number of candidate views; Calculate the entropy value of the probability distribution: ; Calculate the maximum probability P max With the second highest probability P second The difference is used to obtain the confidence gap ΔP.

9. The robot positioning and control method according to claim 7, characterized in that, The generation of location decision results based on the confidence distribution includes: When the entropy value is less than the first entropy threshold and the confidence gap is greater than the first gap threshold, it is determined that the hierarchical decision should be executed directly. When the entropy value is greater than or equal to the first entropy threshold and less than or equal to the second entropy threshold, or when the confidence gap is less than the second gap threshold, a local re-inference level decision is determined. When the entropy value is greater than the second entropy threshold and the historical confidence average is less than the historical confidence threshold, a global replanning level decision is determined.

10. The robot positioning and control method according to claim 9, characterized in that, The direct execution hierarchical decision includes: taking the candidate view corresponding to the highest probability in the confidence distribution as the navigation target, converting the three-dimensional spatial coordinates of the navigation target into target joint angles, and generating joint spatial trajectories based on the target joint angles using model prediction control and issuing them for execution. The local re-inference hierarchical decision includes: pausing the current motion, performing chained reasoning based on the language instruction embedding vector and structured semantic representation, generating intermediate reasoning steps to verify the rationality of the current positioning hypothesis, and if the verification is successful, executing the direct execution hierarchical decision in the manner described above; if the verification fails, re-extracting the visual feature vector from the current environmental observation data and re-calibrating the confidence level. The global replanning hierarchy includes: triggering an update of the scene map; performing an active exploration action based on the updated scene map; the active exploration action includes in-situ rotation scanning or moving to a new viewpoint in the direction of the confidence gradient; after the active exploration action is completed, re-acquiring the environmental observation data; extracting the visual feature vector based on the re-acquiring environmental observation data; and re-performing confidence calibration.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the robot positioning and control method according to any one of claims 1-10.

12. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the robot positioning and control method according to any one of claims 1-10.

Images