Robot positioning and navigation system and method based on multi-modal information fusion

Abstract

The invention discloses a robot positioning navigation system and method based on multi-modal information fusion, and belongs to the technical field of multi-source information fusion navigation. The invention discloses a robot positioning and navigation system based on multi-modal information fusion. The robot positioning and navigation system comprises a visual sensing module, an inertial measurement unit (IMU), a Beidou navigation receiver, a control processor and an electric control driver. The problems that in the prior art, the robot guiding mode is single, the positioning navigation precision and reliability are low, and the robot is difficult to adapt to the complex environment are solved; according to the invention, by fusing the positioning information of different guiding modes, such as vision, IMU and Beidou navigation information, the positioning precision and reliability of the robot in a complex environment are improved, and the robot can better adapt to low-texture scenes such as illumination change, large-angle curve motion and rapid motion.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-source information fusion navigation, and in particular to a robot positioning and navigation system and method based on multi-modal information fusion. Background Art

[0002] Simultaneous localization and mapping (SLAM) technology is a basic requirement for robots to operate in unknown environments. Existing single-sensor SLAM technologies are relatively mature, such as LOAM based on LiDAR and ORB-SLAM based on cameras, but they still cannot handle the degradation and errors caused by the limitations of the sensor's own capabilities. For example, LiDAR cannot handle glass surfaces and scenes lacking structured information, and cameras cannot extract effective information in the dark. Inertial measurement units (IMUs) can solve the problem of LiDAR and camera information degradation in a short time, but cannot avoid drift problems during long-term operation. Different sensors have their own advantages and disadvantages, and can compensate for each other, so fusing information from multiple sensors is very necessary in practical applications.

[0003] Autonomous robot landing requires very high positioning accuracy. Precise positioning is one of the core key technologies. Its main function is to provide accurate and highly reliable input information for the automatic control system, that is, the relative position relationship between the robot's current position and the ideal landing point. Common guidance methods include radar, satellite navigation, inertial navigation, and optoelectronic guidance. However, using any of these guidance methods alone is difficult to achieve very ideal results in terms of key technical performance such as guidance accuracy and reliability.

[0004] A Chinese patent with announcement number CN118149854A discloses an odometer method based on multi-sensor fusion, including: obtaining the event stream output by the event camera and extracting corner points, using the LK optical flow method to track and triangulate the extracted corner points, and obtaining the trajectory in the VIO local coordinate system as the VIO pose estimation result; initializing the GNSS, obtaining the coordinate trajectory under ECEF and aligning it with the trajectory in the VIO local coordinate system, and obtaining the GNSS pose estimation result; based on nonlinear joint optimization and based on the original measurement data obtained from the event camera, IMU and GNSS, constructing a joint optimization objective function, using the Ceres optimization library to solve the objective function to optimize the above two pose estimation results, and obtaining the optimized pose estimation result.

[0005] This invention overcomes the motion blur of traditional camera-based methods in high-speed motion scenarios by fusing data information from event cameras, IMU and GNSS, and improves the accuracy of pose estimation in dynamic environments. However, in robot positioning and navigation, in addition to high-speed motion, the influence of factors such as low-texture environment and lighting changes also need to be considered to ensure navigation accuracy. Summary of the invention

[0006] The purpose of the present invention is to provide a robot positioning and navigation system and method based on multimodal information fusion, which fuses the positioning information of different guidance methods to improve the reliability of input information and solve the problems raised in the above-mentioned background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A robot positioning and navigation system based on multimodal information fusion includes: a visual sensing module for collecting image information in an environment; an inertial measurement unit (IMU) for collecting motion information of the robot; a Beidou navigation receiver for receiving Beidou navigation signals; a control processor for performing fusion processing based on information provided by the visual sensing module, the inertial measurement unit and the Beidou navigation receiver to obtain accurate position and posture information of the robot; and an electric control driver for adjusting the speed of a driving motor according to a speed control instruction provided by the control processor so that the robot can move along a preset path.

[0009] Preferably, the control processor includes a speed estimation module and a PID controller. The speed estimation module is used to estimate the movement speed of the robot in real time, and adopts a motion state optimization estimation model to achieve optimized estimation of its motion state; the PID controller is used to adjust the control parameters of the electronic control drive according to the optimization variable information provided by the speed estimation module to ensure that the actual speed of the robot is consistent with the target speed.

[0010] Preferably, the PID controller adopts an auto-coupling PID collaborative control strategy to improve the accuracy and response speed of speed control.

[0011] Preferably, the autocoupled PID collaborative control strategy includes: calculating the speed control error of the left and right drive motors; introducing the speed factor Zc to tightly couple the proportional, integral and differential links to improve the speed control accuracy; compensating for the optimized change in the motion speed in the sliding window to control the speed of the drive wheel motor.

[0012] Preferably, the visual sensing module includes an RGB-D depth camera for providing three-dimensional environmental information.

[0013] Preferably, the motion state optimization estimation model obtains visual motion constraints, Beidou navigation motion constraints and IMU motion constraint input values ​​through an RGB-D depth camera, a Beidou navigation receiver and an inertial measurement unit, respectively, and obtains the optimization variable fit, key frames and visual inertial state variables through a motion observation residual nonlinear optimization function.

[0014] Preferably, the control processor also includes: a map construction module, used to construct an environment map according to the posture information of the robot; and a path planning module, used to plan the driving path of the robot according to the environment map.

[0015] Preferably, the inertial measurement unit comprises an accelerometer and a gyroscope for measuring and reporting the acceleration, angular velocity and / or orientation of the robot to determine the attitude and motion state of the device.

[0016] Preferably, the robot includes a suspension system, a body structure, a chassis structure and an electronically controlled drive module to achieve autonomous movement.

[0017] A method for implementing a robot positioning and navigation system based on multimodal information fusion is implemented based on a robot positioning and navigation system based on multimodal information fusion, comprising the following steps:

[0018] Step 1: Data collection: The all-terrain mobile robot collects visual and self-operation information in the environment through its own calibrated RGB-D depth camera, accelerometer and gyroscope, transmits the data to the web server, and obtains positioning information through the Beidou navigation receiver;

[0019] Step 2: Data processing: The all-terrain mobile robot creates a camera node through the ROS robot operating system, receives image information in real time through the RGB-D depth camera, and uses high- and low-dimensional feature nodes to aggregate feature information around image interest points to calculate the matching descriptor fit; it uses a tightly coupled method to fuse vision and IMU information, uses forward and reverse indexing strategies to select key frames, and combines Beidou navigation information to correct posture perception information;

[0020] Step 3: Map construction: Calculate the posture perception information of the all-terrain mobile robot to solve the current running speed and build a map;

[0021] Step 4. Speed ​​control: When the all-terrain mobile robot detects passive deceleration situations such as large-angle curve movement and obstacles, and its running speed does not meet the set speed, the control parameters can be calculated based on the deviation between the set value and the current value, and the control parameters can be fed back to the electronic control driver to change the motor input voltage to achieve the purpose of adjusting the motor speed. The auto-coupling PID collaborative control strategy is used to restore the current speed of the all-terrain mobile robot to the set value.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention improves the positioning accuracy and reliability of the robot in complex environments by integrating positioning information from different guidance methods, such as vision, IMU and Beidou navigation information.

[0024] 2. The present invention can better adapt to low-texture scenes such as lighting changes, large-angle curves and rapid movements, and shows stronger adaptability and robustness compared to traditional SLAM technology.

[0025] 3. The present invention utilizes an auto-coupling PID collaborative control strategy to optimize the variables in the estimation model according to the real-time updated motion state, thereby achieving real-time estimation and precise control of the motion speed of the all-terrain mobile robot.

[0026] 4. The present invention effectively reduces the trajectory error of the all-terrain mobile robot and has good composition capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A block diagram of a prototype of a vision-based all-terrain mobile robot according to the present invention;

[0028] Figure 2 It is a framework diagram of the speed control system of the all-terrain mobile robot of the present invention;

[0029] Figure 3 A schematic diagram of the speed control process of the all-terrain mobile robot of the present invention;

[0030] Figure 4 It is a comparative analysis diagram of the trajectory diagram of the present invention and ORB-SLAM2 and GCNv2-SLAM in KITTI and TUM data sets;

[0031] Figure 5 A comparative analysis diagram of the indoor and outdoor real scene trajectories of the all-terrain mobile robot of the present invention;

[0032] Figure 6 A comparison diagram of outdoor dynamic trajectory tracking of the all-terrain mobile robot of the present invention; DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] In order to solve the problem that the guidance method of the robot in the prior art is relatively single, resulting in low positioning and navigation accuracy and reliability, and difficulty in adapting to complex environments, this embodiment provides the following technical solutions:

[0035] See also Figure 1 The all-terrain mobile robot in this embodiment is mainly composed of modules such as a suspension system, a body structure, a chassis structure, and an electronic control drive.

[0036] The chassis frame is one of the important components of the all-terrain mobile robot. It forms the overall shape of the all-terrain mobile robot and receives the power of the electronic control drive. It not only ensures its normal driving, but also serves as the load basis for its own weight and the weight of the transported objects.

[0037] The body structure is installed on the chassis frame of the all-terrain mobile robot, connecting the various assembly parts of the mobile robot. It can withstand various loads from inside and outside the vehicle. Its performance directly affects the control performance and safety of the all-terrain mobile robot.

[0038] The suspension system is an important component for all-terrain mobile robots to cope with various complex road conditions. It is an elastic connection between the wheels and the body, which can reduce the impact force caused by uneven road surfaces, allowing the vehicle to maintain a stable posture during driving, and can effectively transfer loads, reduce impacts, and attenuate vibrations, etc., to ensure the safety and stability of the all-terrain mobile robot.

[0039] The electric drive is an important component that provides power for the all-terrain mobile robot to achieve autonomous movement. It can control the speed of the left and right drive motors through algorithms according to the tasks set by the user, coordinate and complete power distribution and output, and realize the movement of the all-terrain mobile robot in different scenarios.

[0040] A robot positioning and navigation system based on multimodal information fusion, comprising:

[0041] The visual sensing module is used to collect image information in the environment and provide three-dimensional environmental information, using the RGB-D depth camera Inter D435.

[0042] Inertial Measurement Unit (IMU), used to collect robot motion information, including accelerometer and gyroscope UM7, can be used to measure and report the robot's acceleration, angular velocity and / or direction to determine the device's attitude and motion state.

[0043] Beidou navigation receiver, used to receive Beidou navigation signals, uses K823W Beidou navigation terminal.

[0044] The control processor is used to perform fusion processing based on the information provided by the visual sensor module, inertial measurement unit and Beidou navigation receiver to obtain the precise position and posture information of the robot. The control processor includes a speed estimation module and a PID controller. The speed estimation module is used to estimate the movement speed of the robot in real time, and adopts a motion state optimization estimation model to achieve the optimization estimation of its motion state; the PID controller is used to adjust the control parameters of the electronic control drive according to the optimization variable information provided by the speed estimation module to ensure that the actual speed of the robot is consistent with the target speed; the PID controller adopts an auto-coupling PID collaborative control strategy to improve the accuracy and response speed of speed control.

[0045] The control processor also includes: a map construction module, which is used to construct an environmental map according to the posture information of the robot; and a path planning module, which is used to plan the driving path of the robot according to the environmental map.

[0046] The electronically controlled driver is used to adjust the speed of the drive motor according to the speed control instructions provided by the control processor, so that the robot can move along a preset path.

[0047] See also Figure 2 The all-terrain mobile robot achieves speed control through an electric drive to ensure stable movement at a set speed. To ensure the accuracy of the mobile robot's speed control, the motion state is optimized by fusing vision (collected by the depth camera Inter D435), IMU information (collected by the accelerometer and gyroscope UM7) and Beidou information (collected by the K823W Beidou terminal) and using the motion state optimization estimation model, and based on this, the drive motor speed control system is constructed.

[0048] The driving motor speed control system of the all-terrain mobile robot realizes the real-time estimation of the motion speed of the all-terrain mobile robot according to the real-time optimization variable consistency, key frames and visual inertial state variables in the motion state optimization estimation model, and completes the real-time control of its driving motor in combination with the auto-coupling PID collaborative control strategy.

[0049] See also Figure 3-6 , an implementation method of a robot positioning and navigation system based on multimodal information fusion, based on a robot positioning and navigation system based on multimodal information fusion, comprising the following steps:

[0050] Step 1: Data collection: The all-terrain mobile robot collects visual and self-operation information in the environment through its own calibrated RGB-D depth camera, accelerometer and gyroscope, transmits the data to the web server, and obtains positioning information through the Beidou navigation receiver;

[0051] Step 2: Data processing: The all-terrain mobile robot creates a camera node through the ROS robot operating system, receives image information in real time through the RGB-D depth camera, and uses high- and low-dimensional feature nodes to aggregate feature information around the image interest points to calculate the matching descriptor fit. The visual and IMU information are fused in a tightly coupled manner, and the forward and reverse indexing strategies are used to select the key frame kf j , and combined with Beidou navigation information to correct the posture perception information;

[0052] Step 3: Map construction: Calculate the posture perception information of the all-terrain mobile robot to solve the current running speed and build a map;

[0053] Step 4. Speed ​​control: When the all-terrain mobile robot detects passive deceleration situations such as large-angle curve movement and obstacles, and its running speed does not meet the set speed, the control parameters can be calculated based on the deviation between the set value and the current value, and the control parameters can be fed back to the electronic control driver to change the motor input voltage to achieve the purpose of adjusting the motor speed. The auto-coupling PID collaborative control strategy is used to restore the current speed of the all-terrain mobile robot to the set value.

[0054] The all-terrain mobile robot uses a self-coupling PID collaborative control strategy to adjust the robot's movement speed to ensure its safe and stable movement. The steps are as follows:

[0055] Step 1: When the all-terrain mobile robot detects passive deceleration conditions such as large-angle curve movement and obstacles, the all-terrain mobile robot calculates the longitudinal component v of the current speed v according to the distance d between the left and right driving wheels of the all-terrain mobile robot and the longitudinal component v of the current speed v. ex With the tangential component v ey and the expected speed v of the all-terrain mobile robot t and the desired angular velocity ω t , calculate the speed control error e of the left and right drive motors l and e r , respectively:

[0056] (1)

[0057] Step 2: In order to improve the speed control accuracy, the speed factor Zc is introduced. Zc tightly couples the three links of proportion, integration and differentiation to form an auto-coupling PID collaborative control, which overcomes the shortcomings of proportion, integration and differentiation being independent of each other in the control process and lacking coordination.

[0058] (2)

[0059] Where α=α1 / T t , 0<α1<100, β=1 / T t , T t is the transition time from dynamic transition to steady state. e1, e2, e3 are tracking error, integral error and differential error respectively, and b0 is the control channel gain adjustment parameter.

[0060] u is the coordinated output of the auto-coupling PID control:

[0061] (3)

[0062] Step 3: Optimize the change Δv of the motion speed in the sliding window s Compensation is performed to control the speed of the driving wheel motor of the all-terrain mobile robot. The speed control strategy of the all-terrain mobile robot is:

[0063] (4)

[0064] Among them, P h , P h-1 are the speed control outputs at the current speed control moment and the previous speed control moment (h-1), respectively. is the control error between the left and right drive motors at moment h and the previous moment (h-1) in formula (4).

[0065] Working principle: First, the image information in the environment and the motion information of the robot are collected through sensors such as RGB-D depth camera and inertial measurement unit, the positioning information is obtained by using Beidou navigation terminal, and the camera node is created through ROS robot operating system to receive the depth camera image information in real time, aggregate the feature information around the image interest point, and fuse the visual and IMU information in a tightly coupled manner; then, the key frame is selected through the forward and reverse indexing strategy, and the posture perception information is corrected in combination with Beidou navigation information; then, the variables in the model are optimized according to the real-time updated motion state, including visual inertial state variables, key frame information, etc., to achieve real-time estimation of the motion state of the all-terrain mobile robot; finally, the auto-coupling PID collaborative control strategy is adopted to calculate the speed control error of the left and right drive motors according to the distance between the left and right drive wheels of the all-terrain mobile robot, the longitudinal component and tangential component of the current speed, and the expected speed and expected angular velocity, and the speed factor Zc is introduced to tightly couple the three links of proportion, integration and differentiation to form an auto-coupling PID collaborative control to achieve real-time estimation and precise control of the current speed of the all-terrain mobile robot.

[0066] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0067] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.

Claims

1. A robot positioning and navigation system based on multimodal information fusion, characterized in that: include: Visual sensing module, used to collect image information in the environment; Inertial measurement unit IMU, used to collect robot motion information; Beidou navigation receiver, used to receive Beidou navigation signals; A control processor is used to perform fusion processing based on the information provided by the visual sensor module, the inertial measurement unit and the Beidou navigation receiver to obtain the precise position and attitude information of the robot; The electronically controlled driver is used to adjust the speed of the drive motor according to the speed control instructions provided by the control processor, so that the robot can move along a preset path.

2. The robot positioning and navigation system based on multimodal information fusion according to claim 1, characterized in that: The control processor includes a speed estimation module and a PID controller. The speed estimation module is used to estimate the movement speed of the robot in real time, and adopts a motion state optimization estimation model to achieve the optimization estimation of its motion state; the PID controller is used to adjust the control parameters of the electronic control drive according to the optimization variable information provided by the speed estimation module to ensure that the actual speed of the robot is consistent with the target speed.

3. The robot positioning and navigation system based on multimodal information fusion according to claim 2, characterized in that: The PID controller adopts an auto-coupling PID cooperative control strategy to improve the accuracy and response speed of speed control.

4. The robot positioning and navigation system based on multimodal information fusion according to claim 3, characterized in that: The auto-coupling PID collaborative control strategy includes: calculating the speed control error of the left and right driving motors; introducing the speed factor Zc to tightly couple the proportional, integral and differential links to improve the speed control accuracy; and compensating for the optimized change in the motion speed in the sliding window to control the speed of the driving wheel motor.

5. The robot positioning and navigation system based on multimodal information fusion according to claim 4, characterized in that: The visual sensing module includes an RGB-D depth camera for providing three-dimensional environmental information.

6. The robot positioning and navigation system based on multimodal information fusion according to claim 5, characterized in that: The motion state optimization estimation model obtains the visual motion constraints, Beidou navigation motion constraints and IMU motion constraints input values ​​through the RGB-D depth camera, Beidou navigation receiver and inertial measurement unit respectively, and obtains the optimization variable consistency, key frames and visual inertial state variables through the motion observation residual nonlinear optimization function.

7. The robot positioning and navigation system based on multimodal information fusion according to claim 6, characterized in that: The control processor also includes: a map construction module, which is used to construct an environment map according to the posture information of the robot; and a path planning module, which is used to plan the driving path of the robot according to the environment map.

8. The robot positioning and navigation system based on multimodal information fusion according to claim 7, characterized in that: The inertial measurement unit includes an accelerometer and a gyroscope to measure and report the acceleration, angular velocity and / or orientation of the robot to determine the attitude and motion state of the device.

9. The robot positioning and navigation system based on multimodal information fusion according to claim 8, characterized in that: The robot includes a suspension system, a body structure, a chassis structure and an electronically controlled drive module to achieve autonomous movement.

10. A method for implementing a robot positioning and navigation system based on multimodal information fusion, which is implemented based on the robot positioning and navigation system based on multimodal information fusion according to any one of claims 1 to 9, characterized in that: The following steps are involved: Step 1: Data collection: The all-terrain mobile robot collects visual and self-operation information in the environment through its own calibrated RGB-D depth camera, accelerometer and gyroscope, transmits the data to the web server, and obtains positioning information through the Beidou navigation receiver; Step 2: Data processing: The all-terrain mobile robot creates a camera node through the ROS robot operating system, receives image information in real time through the RGB-D depth camera, and uses high- and low-dimensional feature nodes to aggregate feature information around image interest points to calculate the matching descriptor fit; it uses a tightly coupled method to fuse vision and IMU information, uses forward and reverse indexing strategies to select key frames, and combines Beidou navigation information to correct posture perception information; Step 3: Map construction: Calculate the posture perception information of the all-terrain mobile robot to solve the current running speed and build a map; Step 4. Speed ​​control: When the all-terrain mobile robot detects passive deceleration situations such as large-angle curve movement and obstacles, and its running speed does not meet the set speed, the control parameters can be calculated based on the deviation between the set value and the current value, and the control parameters can be fed back to the electronic control driver to change the motor input voltage to achieve the purpose of adjusting the motor speed. The auto-coupling PID collaborative control strategy is used to restore the current speed of the all-terrain mobile robot to the set value.

Images