Robot vision positioning method and system based on causal inference
By separating interferences such as light and shadow through causal inference and combining meta-learning and physical constraint verification, the stability and adaptability problems of robot visual localization in complex environments are solved, achieving highly robust and safe robot visual localization.
Patent Information
- Application Number
- CN202511521345.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-23
- Publication Date
- 2026-01-23
AI Technical Summary
Existing robot visual localization technologies suffer from decreased feature matching stability in environments with drastic changes in lighting, interference from dynamic objects, or missing or repetitive textures. Furthermore, they lack environmental adaptability and common-sense judgment of the physical world, leading to unstable localization and error accumulation.
A causal inference-based approach is adopted, which separates essential environmental cues from apparent interferences by decoupling a causal feature network, combines a meta-learning modulator to adapt to environmental changes, integrates IMU data for state inference, verifies the optimal pose through physical constraints, and constructs a semantic memory map.
It improves the robustness and stability of robot visual positioning, reduces positioning drift, and enables cross-domain adaptation and safety in extreme environments.
Smart Images
Figure CN121391982A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of visual positioning, and particularly relates to a robot visual positioning method and system based on causal inference. BACKGROUND
[0002] Robot visual positioning technology is the cornerstone of realizing autonomous navigation and interaction. For a long time, it mainly relies on the following methods: traditional visual odometry and simultaneous localization and mapping technology based on geometric model, which estimates the pose by feature point extraction, matching and optimization, but in the case of severe light changes, dynamic object interference, texture missing or repetitive texture environment, the stability of feature matching will decrease sharply, resulting in tracking loss; another mainstream solution is visual inertial odometry, which compensates for the shortcomings of pure vision by fusing IMU data, improving robustness and accuracy, but its core still relies on short-term, data-driven geometric or filtering theory, and lacks the ability to fundamentally solve complex and long-tail confusion factors (such as appearance changes caused by seasonal changes, day and night alternation, and weather changes) in the environment. In recent years, end-to-end positioning models based on deep learning have shown great potential, but they usually rely heavily on a large amount of training data that covers all possible scenarios, and perform poorly in unfamiliar environments that deviate from the training distribution, and lack common sense judgment of the physical world, which may output trajectories that completely violate the laws of dynamics. In addition, most existing methods use static model parameters and cannot adjust themselves according to the environment context during the task, lacking the ability of lifelong learning and knowledge accumulation. SUMMARY
[0003] To solve the above problems in the prior art, the application provides a robot visual positioning method and system based on causal inference.
[0004] The object of the application can be achieved by the following technical solutions: A robot visual positioning method based on causal inference, the implementation of the robot visual positioning method based on causal inference comprises the following steps: Step S1: The robot shoots continuous color images, simultaneously obtains instantaneous acceleration and angular velocity, and obtains an original image sequence and an IMU original data stream; Step S2: Causal feature decoupling is performed through the original image sequence, and an essential semantic feature vector is output; Step S3: The essential semantic feature vector is modulated and reinforced through meta-learning, and an optimized essential semantic feature vector is obtained; Step S4: The optimized essential semantic feature vector and the IMU original data stream are fused, the robot state is calculated through continuous time state speculation, and a time-pose trajectory is output; Step S5: based on the time-pose trajectory, physical constraint verification is performed through a physical constraint verification formula, and an optimal pose is output; Step S6: the optimal pose and the corresponding optimized essential semantic feature vector are jointly stored in a semantic memory layer, and an environment semantic map is constructed.
[0005] Preferably, the causal feature decoupling in step S2 is specifically: The original image sequence is input into a causal feature decoupling network, and the original image is decomposed into an environmental essential clue and an environmental appearance interference through a causal feature purification formula, and the essential semantic feature vector is output for the original image, and the mathematical description of the causal feature purification formula is , wherein, is the total loss, is the expectation, and are two original images of the same scene under different conditions, is a function of extracting the environmental essential clue, is a function of extracting the environmental appearance interference, is the Euclidean distance, is a hyperparameter, is divergence, is a probability distribution.
[0006] Preferably, the meta-learning modulation in step S3 is specifically: The essential semantic feature vector is input into a meta-learning modulator, the current essential semantic feature vector is analyzed in real time, the strategy most suitable for processing the current environment is retrieved from an experience library, and the internal parameters of the meta-learning modulator are fine-tuned through a meta-learning modulation formula, and finally the optimized essential semantic feature vector is output, and the mathematical description of the meta-learning modulation formula is , wherein, is the current adaptive parameter, is the meta-learning initial parameter, is the learning rate, is a small neural network, is an environmental context vector.
[0007] Preferably, the time state speculation in step S4 is specifically: An input signal is constructed based on the optimized essential semantic feature vector and the IMU original data stream; Based on the input signal, a robot state is obtained through a continuous motion smoothing formula, and the time-pose trajectory is output, and the mathematical description of the continuous motion smoothing formula is , wherein, is the time The robot's state For neural networks, For a moment The input signal.
[0008] Preferably, the construction of the input signal specifically includes: Through the IMU raw data stream at time Directly read instantaneous acceleration and angular velocity; Based on the optimized essential semantic feature vector, the robot's rotation and translation within the time interval between two adjacent frames are estimated through geometric calculations and converted into estimated velocity and estimated angular velocity. The input signal is obtained by fusing the instantaneous acceleration, the angular velocity, the estimated velocity, and the estimated angular velocity.
[0009] Preferably, step S5 specifically includes: The actual acceleration is obtained based on the time-pose trajectory, and the estimated resultant force is obtained by multiplying the actual acceleration and the robot's mass. Based on the estimated resultant force, the physical constraint verification formula is constructed, mathematically described as follows: ,in, For the degree of unreasonableness, and For weight parameters, To estimate the resultant force, For robot quality, For instantaneous acceleration, The maximum acceleration of the robot as expected during system design. and These are the changes in kinetic energy and the changes in potential energy, respectively. To do work for the outside world This represents the maximum energy change of the robot expected during system design.
[0010] Preferably, step S6 specifically includes: The semantic memory layer is divided into a short-term working memory layer and a long-term structural memory layer. The environmental semantic map is constructed using memory consolidation and forgetting formulas. The mathematical description of these memory consolidation and forgetting formulas is... ,in, In order to be in Long-term memory of moments In order to be in Long-term memory of moments The global learning rate, ·) is the Sigmoid function. Assigning points based on importance, For short-term information to be memorized.
[0011] A robot vision localization system based on causal inference, used to execute the robot vision localization method based on causal inference described above, includes a data acquisition module, a causal inference module, a meta-learning modulation module, a state inference module, a constraint verification module, and an environmental semantic map construction module; The data acquisition module is used by the robot to capture continuous color images, and simultaneously acquire instantaneous acceleration and angular velocity to obtain the original image sequence and IMU raw data stream; The causal inference module is used to decouple causal features from the original image sequence and output an essential semantic feature vector. The meta-learning modulation module is used to enhance the essential semantic feature vector through meta-learning modulation to obtain an optimized essential semantic feature vector. The state inference module is used to fuse the optimized essential semantic feature vector and the IMU raw data stream, infer the robot state through continuous time state inference, and output the time-pose trajectory; The constraint verification module is used to perform physical constraint verification based on the time-pose trajectory using the physical constraint verification formula, and output the optimal pose. The environmental semantic map construction module is used to store the optimal pose and the corresponding optimized essential semantic feature vector together in the semantic memory layer, and construct the environmental semantic map.
[0012] The beneficial effects of this invention are as follows: (1) By decoupling causal features, the robot can eliminate the interference of appearances such as light and shadow and weather, directly capture the stable and unchanging essential structure in the environment, improve robustness and stability, and reduce positioning drift and failure.
[0013] (2) Through the meta-learning modulator, when the robot enters another environment from one environment, it does not need to recalibrate or switch models in a cumbersome way. Instead, it can intelligently call existing experience or quickly generate new strategies, fine-tune internal parameters, achieve cross-domain adaptation, and significantly improve the reliability of long-term operation in complex and unstable environments.
[0014] (3) By verifying physical constraints, it automatically identifies and corrects erroneous results that violate the basic laws of mechanics, thereby enhancing its safety and rationality in extreme or marginal situations. Attached Figure Description
[0015] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.
[0016] Figure 1 This is a flowchart of the steps of a robot visual localization method based on causal inference according to the present invention. Detailed Implementation
[0017] To better understand the invention, various aspects of the invention will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of the invention and are not intended to limit the scope of the invention in any way. Throughout the specification, the expression "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the terms "approximately," "about," and similar terms are used as expressions of approximation, not as expressions of degree, and are intended to describe inherent deviations in measured or calculated values that will be recognized by those skilled in the art. Furthermore, the order in which the steps are described in this invention does not necessarily indicate the order in which these steps occur in actual operation, unless otherwise expressly defined or deduced from the context.
[0018] It should also be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of the invention, the word "may" is used to mean "one or more embodiments of the invention." And the term "exemplary" is intended to refer to examples or illustrations.
[0019] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not in an idealized or overly formalized sense.
[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other. The invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0021] Example 1: Please see Figure 1 A robot visual localization method based on causal inference, comprising: Step S1: The robot captures continuous color images using a standard camera, and simultaneously obtains its instantaneous acceleration and angular velocity using an inertial measurement unit (IMU) to obtain the original image sequence and the IMU raw data stream; Step S2: Decouple causal features from the original image sequence and output essential semantic feature vectors; Step S3: Enhance the essential semantic feature vector through meta-learning modulation to obtain an optimized essential semantic feature vector; Step S4: Fuse the optimized essential semantic feature vector and the IMU raw data stream, infer the robot state through continuous time state, and output a smooth, continuous time-pose trajectory; Step S5: Based on the time-pose trajectory, perform physical constraint verification using the physical constraint verification formula, and output the verified optimal pose; Step S6: Store the optimal pose and the corresponding optimized essential semantic feature vector together in the semantic memory layer, and construct an environmental semantic map.
[0022] In this embodiment, the causal feature decoupling specifically refers to: The original image sequence is fed into a trained causal feature decoupling network. The causal feature purification formula decomposes the original images into essential environmental cues (permanent structures of objects, such as the permanent shape of a chair or the standard height of a door—properties inherent to the objects in the image that do not change with lighting or weather) and environmental appearance interference (temporary factors such as light, shadow, reflection, and seasonal changes, such as the color of a chair under lighting or reflections on floor tiles). For each original image, a set of pure, insensitive semantic feature vectors (i.e., a series of digital reports describing "what objects (such as doors), object coordinates, and object geometry) exist in the environment") is output. The mathematical description of the causal feature purification formula is... ,in, The total loss is defined as the objective of minimizing it. As expected, and Derived from training data, these are two original images of the same scene under different conditions (such as the same table under daytime and nighttime conditions). This is a function for extracting essential cues from the environment, with the goal of ensuring that the essential features of the same object remain consistent across different images. A function to extract environmental appearance interference. For Euclidean distance, These are hyperparameters used to balance the importance of the two tasks. for Divergence is used to maximize the difference in representational features between two original images. For probability distribution (in practical applications, the original image is input into the trained image). The function can be used to obtain the essential semantic feature vector.
[0023] The meta-learning modulation described in this embodiment is specifically as follows: The essential semantic feature vector is fed into a meta-learning modulator, which analyzes the current essential semantic feature vector in real time. It retrieves the most suitable strategy for the current environment from the experience base and fine-tunes the internal parameters of the meta-learning modulator using a meta-learning modulation formula. Finally, it outputs the optimized essential semantic feature vector (especially suitable for changing environments, such as a robot moving from a dimly lit warehouse to a bright outdoor environment; if the image analysis is performed using the same method as in the previous step, it is likely to lead to increased errors, requiring fine-tuning of the internal parameters to adapt to the new environment). The mathematical description of the meta-learning modulation formula is... ,in, For the current adaptation parameters, These are the initial parameters for meta-learning. The learning rate is used to control the magnitude of the adjustment. This is a small neural network that takes an environment description (environment context vector) as input and outputs suggestions for adjusting the meta-learning parameters. The environment context vector is a general description of the current environment type by the system. It is derived from the essential semantic feature vector (that is, inputting the complex essential semantic feature vector into another small neural network to generate a highly generalized, low-dimensional vector that only encodes the key attributes of the current environment, such as a vector representing "low light, high dynamics").
[0024] In this embodiment, the time state prediction specifically refers to: S401: Construct a continuous input signal using the optimized essential semantic feature vector and the IMU raw data stream; Through the IMU raw data stream at time Directly read instantaneous acceleration and angular velocity; Based on the optimized essential semantic feature vector, the rotation and translation of the camera (i.e., the robot) within the time interval between two adjacent frames are estimated through geometric calculations and converted into estimated velocity and estimated angular velocity. The input signal is obtained by fusing the instantaneous acceleration, the angular velocity, the estimated velocity, and the estimated angular velocity.
[0025] S402: Based on the input signal, the robot state is obtained through the continuous motion smoothing formula, and the time-pose trajectory is output. The mathematical description of the continuous motion smoothing formula is... ,in, For a moment The robot's state (including spatial coordinates, orientation, speed, and other multi-dimensional information). For neural networks, the goal is to predict how the state will change based on the current state and the input signal. For a moment The input signal.
[0026] In this embodiment, step S5 can be implemented through the following steps: The true acceleration is obtained based on the time-pose trajectory (the time-pose trajectory is continuous and smooth, containing both position and velocity; the true acceleration can be obtained through the derivative of the velocity), and the estimated resultant force is obtained by multiplying the true acceleration by the robot's mass. Based on the estimated resultant force, the physical constraint verification formula is constructed, mathematically described as follows: ,in, The value is considered unreasonable (the higher the value, the more the current time-pose trajectory may violate the laws of physics; the system needs to automatically reduce the weight of untrusted sensor data and provide a more reasonable correction value). and These are weighting parameters used to balance the importance of different constraints. To estimate the resultant force, For robot quality, For instantaneous acceleration, The maximum acceleration of the robot as expected during system design. and These represent changes in kinetic energy and potential energy, respectively, originating from changes in velocity and altitude during the time-pose trajectory. The amount of work done on the outside is usually estimated from the motor data. This is the maximum energy change (including kinetic and potential energy) expected in the system design of the robot. In this embodiment, step S6 can be implemented through the following steps: The semantic memory layer is divided into a short-term working memory layer (stores temporary information, such as pedestrians and moving vehicles, which will be automatically forgotten over time) and a long-term structural memory layer (stores only stable and important landmarks; the more times a location is visited, the deeper and stronger the memory of that location becomes). The environmental semantic map is constructed using memory consolidation and forgetting formulas. The mathematical description of these formulas is... ,in, In order to be in Long-term memory of moments In order to be in Long-term memory of moments The global learning rate, ·) is the Sigmoid function. The importance score is derived from factors such as the stability of the corresponding memory (whether it persists) and the frequency of its access. The short-term information to be memorized comes from the currently observed optimized essential semantic feature vector and optimal pose.
[0027] Example 2: A robot visual localization system based on causal inference includes a data acquisition module, a causal inference module, a meta-learning modulation module, a state inference module, a constraint verification module, and an environmental semantic map construction module. The data acquisition module is used by the robot to capture continuous color images, and simultaneously acquire instantaneous acceleration and angular velocity to obtain the original image sequence and IMU raw data stream; The causal inference module is used to decouple causal features from the original image sequence and output an essential semantic feature vector. The meta-learning modulation module is used to enhance the essential semantic feature vector through meta-learning modulation to obtain an optimized essential semantic feature vector. The state inference module is used to fuse the optimized essential semantic feature vector and the IMU raw data stream, infer the robot state through continuous time state inference, and output the time-pose trajectory; The constraint verification module is used to perform physical constraint verification based on the time-pose trajectory using the physical constraint verification formula, and output the optimal pose. The environmental semantic map construction module is used to store the optimal pose and the corresponding optimized essential semantic feature vector together in the semantic memory layer, and construct the environmental semantic map.
[0028] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. A robot visual localization method based on causal inference, characterized in that, The implementation of the robot visual localization method based on causal inference includes the following steps: Step S1: The robot captures continuous color images and simultaneously acquires instantaneous acceleration and angular velocity to obtain the original image sequence and the IMU raw data stream; Step S2: Decouple causal features from the original image sequence and output essential semantic feature vectors; Step S3: Enhance the essential semantic feature vector through meta-learning modulation to obtain an optimized essential semantic feature vector; Step S4: Fuse the optimized essential semantic feature vector and the IMU raw data stream, infer the robot state through continuous time state, and output the time-pose trajectory; Step S5: Based on the time-pose trajectory, perform physical constraint verification using the physical constraint verification formula, and output the optimal pose; Step S6: Store the optimal pose and the corresponding optimized essential semantic feature vector together in the semantic memory layer, and construct an environmental semantic map.
2. The robot visual localization method based on causal inference according to claim 1, characterized in that, The causal feature decoupling in step S2 specifically refers to: The original image sequence is fed into a causal feature decoupling network. The original image is decomposed into essential environmental cues and apparent environmental interference using a causal feature purification formula. The essential semantic feature vector is then output for the original image. The mathematical description of the causal feature purification formula is... ,in, For the total loss, As expected, and Two original images of the same scene under different conditions. A function to extract essential clues about the environment. A function to extract environmental appearance interference. For Euclidean distance, For hyperparameters, for divergence, It represents a probability distribution.
3. The robot visual localization method based on causal inference according to claim 1, characterized in that, The meta-learning modulation in step S3 specifically refers to: The essential semantic feature vector is fed into a meta-learning modulator, which analyzes the current essential semantic feature vector in real time, retrieves the most suitable strategy for handling the current environment from the experience base, and fine-tunes the internal parameters of the meta-learning modulator using a meta-learning modulation formula. Finally, the optimized essential semantic feature vector is output. The mathematical description of the meta-learning modulation formula is... ,in, For the current adaptation parameters, These are the initial parameters for meta-learning. For learning rate, For a small neural network, This is the environment context vector.
4. The robot visual localization method based on causal inference according to claim 1, characterized in that, The time state prediction in step S4 specifically refers to: The input signal is constructed using the optimized essential semantic feature vector and the IMU raw data stream; Based on the input signal, the robot state is obtained through a continuous motion smoothing formula, and the time-pose trajectory is output. The mathematical description of the continuous motion smoothing formula is as follows: ,in, For a moment The robot's state For neural networks, For a moment The input signal.
5. The robot visual localization method based on causal inference according to claim 4, characterized in that, The construction of the input signal specifically includes: Through the IMU raw data stream at time Directly read instantaneous acceleration and angular velocity; Based on the optimized essential semantic feature vector, the robot's rotation and translation within the time interval between two adjacent frames are estimated through geometric calculations and converted into estimated velocity and estimated angular velocity. The input signal is obtained by fusing the instantaneous acceleration, the angular velocity, the estimated velocity, and the estimated angular velocity.
6. The robot visual localization method based on causal inference according to claim 1, characterized in that, Step S5 specifically includes: The actual acceleration is obtained based on the time-pose trajectory, and the estimated resultant force is obtained by multiplying the actual acceleration and the robot's mass. Based on the estimated resultant force, the physical constraint verification formula is constructed, mathematically described as follows: ,in, For the degree of unreasonableness, and For weight parameters, To estimate the resultant force, For robot quality, For instantaneous acceleration, The maximum acceleration of the robot as expected during system design. and These are the changes in kinetic energy and the changes in potential energy, respectively. To do work for the outside world This represents the maximum energy change of the robot expected during system design.
7. The robot visual localization method based on causal inference according to claim 1, characterized in that, Step S6 specifically includes: The semantic memory layer is divided into a short-term working memory layer and a long-term structural memory layer. The environmental semantic map is constructed using memory consolidation and forgetting formulas. The mathematical description of these memory consolidation and forgetting formulas is... ,in, In order to be in Long-term memory of moments In order to be in Long-term memory of moments The global learning rate, ·) is the Sigmoid function. Assigning points based on importance, For short-term information to be memorized.
8. A robot visual localization system based on causal inference, characterized in that, The system is applied to the robot visual localization method based on causal inference as described in any one of claims 1-7, and includes a data acquisition module, a causal inference module, a meta-learning modulation module, a state inference module, a constraint verification module, and an environmental semantic map construction module; The data acquisition module is used by the robot to capture continuous color images, and simultaneously acquire instantaneous acceleration and angular velocity to obtain the original image sequence and IMU raw data stream; The causal inference module is used to decouple causal features from the original image sequence and output an essential semantic feature vector. The meta-learning modulation module is used to enhance the essential semantic feature vector through meta-learning modulation to obtain an optimized essential semantic feature vector. The state inference module is used to fuse the optimized essential semantic feature vector and the IMU raw data stream, infer the robot state through continuous time state inference, and output the time-pose trajectory; The constraint verification module is used to perform physical constraint verification based on the time-pose trajectory using the physical constraint verification formula, and output the optimal pose. The environmental semantic map construction module is used to store the optimal pose and the corresponding optimized essential semantic feature vector together in the semantic memory layer, and construct the environmental semantic map.