Robot positioning method, robot path planning method, device and robot

Abstract

This invention relates to the field of robotics, disclosing a robot localization method, a robot path planning method, an apparatus, and a robot. The localization method includes: acquiring first localization data of the robot in physical space; determining second localization data of the robot in a virtual space constructed in real time based on the surrounding environment (3D panoramic imaging); and correcting the first localization data based on the second localization data to obtain target localization data of the robot in physical space. This application introduces second localization data in a 3D panoramic imaging virtual space constructed in real time based on the surrounding environment. The 3D panoramic imaging virtual space can reflect environmental changes in real time, and the second localization data can capture these dynamic information in a timely manner, correcting the first localization data, enabling the robot to accurately adapt to environmental changes and improving the accuracy and stability of localization.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a robot localization method, a robot path planning method, an apparatus, and a robot. Background Technology

[0002] Mining environments are characterized by complex spatial structures, numerous dynamic interference factors, and high safety risks. Therefore, real-time monitoring of the operational process and the safety of vehicles and personnel is crucial for ensuring production safety and improving the efficiency of safety management. With technological advancements, inspection robots have replaced manual monitoring in some safety inspection fields, significantly improving safety monitoring efficiency.

[0003] However, the complexity of the mining environment places extremely stringent demands on the positioning technology of inspection robots. In terms of spatial structure, the underground tunnel network in mines is intricate and complex, resembling a giant maze. Numerous branches, intersections, and slope changes exist between tunnels, and the cross-sectional shapes of the tunnels are irregular, with significant variations in width and height. This complex spatial layout makes it difficult for existing inspection robots to accurately determine their position and orientation during movement, easily leading to them getting lost in the tunnel network. Summary of the Invention

[0004] The purpose of this invention is to provide at least one robot localization method, robot path planning method, device, and robot, which can at least solve the problem that existing robots have difficulty accurately determining their own position and direction during movement in complex spatial layouts, and can at least achieve a more accurate localization effect.

[0005] To address the aforementioned technical problems, at least one embodiment of this application provides a robot localization method, comprising:

[0006] Acquire the robot's initial positioning data in physical space;

[0007] Determine the second positioning data of the robot in a virtual space constructed in real time based on the three-dimensional panoramic imaging of its environment;

[0008] The first positioning data is corrected based on the second positioning data to obtain the target positioning data of the robot in physical space.

[0009] At least one embodiment of this application also provides a robot path planning method, including:

[0010] The robot's target location data in physical space is obtained using the above-described robot localization method;

[0011] The robot's current trajectory is determined based on the target positioning data, and the trajectory and trajectory probability of the moving target are determined based on the position and speed data of the moving target in the environment.

[0012] Based on the trajectory and trajectory probability of the moving target, and using the lowest probability of the robot's current trajectory encountering the moving target and the surrounding environment as a constraint, the robot's motion path is generated in real time.

[0013] At least one embodiment of this application also provides a robot positioning device, comprising:

[0014] The first positioning data acquisition module is used to acquire the robot's first positioning data in physical space;

[0015] The second positioning data acquisition module is used to determine the second positioning data of the robot in the virtual space of the three-dimensional panoramic imaging constructed in real time based on the environment.

[0016] The correction module is used to correct the first positioning data based on the second positioning data to obtain the target positioning data of the robot in physical space.

[0017] At least one embodiment of this application also provides a robot, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the robot localization method described above.

[0018] At least one embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot localization method described above.

[0019] The robot localization method, robot path planning method, device, and robot provided in the embodiments of this application are susceptible to interference from environmental factors, such as dust obscuring the laser beam or changes in lighting conditions, leading to deviations in the localization data. By introducing second localization data on a three-dimensional panoramic imaging virtual space constructed in real time based on the surrounding environment, the three-dimensional panoramic imaging virtual space can reflect environmental changes in real time. The second localization data can capture this dynamic information in a timely manner and correct the first localization data, enabling the robot to accurately adapt to environmental changes and improving the accuracy and stability of localization. This effectively reduces errors caused by environmental interference and improves localization accuracy.

[0020] In some optional embodiments, acquiring the robot's first positioning data in physical space includes:

[0021] Obtain wireless positioning data obtained by positioning the robot using a wireless positioning device;

[0022] Acquire inertial positioning data obtained by positioning the robot using an inertial navigation and positioning device;

[0023] The wireless positioning data and the inertial positioning data are fused to obtain the first positioning data.

[0024] In this embodiment, the fusion of wireless positioning data and inertial positioning data can fully leverage the advantages of both. Wireless positioning data can provide initial position reference and long-term position correction for inertial navigation, reducing the cumulative error caused by sensor drift. Inertial navigation data, on the other hand, can provide short-term, high-precision positioning information when wireless signals are unstable or lost, ensuring the continuity and stability of positioning. The first positioning data obtained by fusing the two more accurately reflects the actual physical space positioning.

[0025] In some optional embodiments, the step of fusing the wireless positioning data with the inertial positioning data to obtain the first positioning data includes:

[0026] Based on the position and velocity data in the wireless positioning data and the position and velocity data in the inertial positioning data, a first Kalman filter model for position and velocity is established. In the first Kalman filter model, the inertial navigation and positioning device provides the predicted position and velocity data for the next moment. When the wireless positioning data arrives, the wireless positioning data is used as the observation value.

[0027] The wireless positioning data is corrected by dynamically adjusting the weights of the wireless positioning data and the inertial positioning data using Kalman gain, thus obtaining the first positioning data.

[0028] In this embodiment, the weights of wireless positioning data and inertial positioning data are dynamically adjusted using Kalman gain, enabling the system to automatically allocate weights based on the real-time reliability of the two positioning methods. For example, when the wireless signal is stable, the weight of wireless positioning data is increased; when the wireless signal is interfered with, the weight of inertial positioning data is increased. This significantly improves positioning accuracy, stability, and real-time performance.

[0029] In some optional embodiments, determining the robot's second positioning data in a virtual space constructed in real time based on a 3D panoramic image of its surroundings includes:

[0030] Based on the orientation information of each structure in the virtual space, the position information of the robot in its environment is determined;

[0031] When the robot moves, the speed information of the robot in the virtual space is calculated based on the continuously changing position information and the corresponding timestamp;

[0032] The second positioning data is obtained based on the location information and the speed information.

[0033] In this embodiment, wireless positioning signals are easily affected in the mining environment, and traditional positioning methods are prone to accumulating errors. Positioning in a three-dimensional virtual space can provide structured information about the environment. Using the second positioning data as a reference point to correct the data from the physical sensors allows the short-term high-frequency data from the physical sensors to complement the long-term stable constraints of the virtual space, forming a balance mechanism similar to the human "visual-vestibular" system. Even when some sensors fail, the structural constraints of the virtual space can still provide basic navigation capabilities, achieving fault-tolerant positioning that maintains accuracy despite degradation.

[0034] In some optional embodiments, the step of correcting the first positioning data based on the second positioning data to obtain the target positioning data of the robot in physical space includes:

[0035] Based on the second positioning data and the first positioning data, a second Kalman filter model is established, wherein the second positioning data is used as the observed value and the first positioning data is used as the predicted value.

[0036] The target positioning data is obtained by dynamically adjusting the weights of the first positioning data and the second positioning data using Kalman gain to correct the first positioning data.

[0037] In this embodiment, a two-layer Kalman filter is constructed: the first-layer Kalman filter fuses wireless positioning and inertial positioning data to solve the positioning jump problem when the wireless signal is blocked or interfered with, while suppressing the drift error of the inertial sensor. The second-layer Kalman filter further fuses the output of the first layer with virtual space positioning to correct errors caused by dynamic changes in the environment. Through two recursive optimizations, the error is reduced layer by layer, and the final positioning accuracy is significantly improved.

[0038] In some optional embodiments, the information of the three-dimensional panoramic imaging is acquired by at least one of a visual acquisition device, a lidar module, and an infrared sensing device.

[0039] In this embodiment, the environmental perception capability is further enhanced by multimodal data complementarity, making the constructed three-dimensional panoramic imaging more closely resemble the real environment. Attached Figure Description

[0040] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0041] Figure 1 This is a flowchart of a robot localization method provided in one embodiment of this application;

[0042] Figure 2 This is a schematic diagram of a robot provided in one embodiment of this application;

[0043] Figure 3 This is a schematic diagram of a robot positioning device provided in another embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0045] To address the technical problem of existing robots struggling to accurately determine their position and orientation during movement in complex spatial layouts, this invention proposes a robot localization method. The implementation details of this robot localization method are described below. These details are provided for ease of understanding and are not essential for implementing this solution.

[0046] Example 1:

[0047] The robot localization method of this embodiment can be applied to electronic devices with communication, computing, and data storage capabilities. Its specific process can be as follows: Figure 1 As shown, it includes:

[0048] Step 110: Obtain the robot's first positioning data in physical space;

[0049] In this embodiment, the first positioning data is the preliminary positioning information acquired by the robot in the physical world through one or more sensors. The first positioning data may include position coordinates (such as latitude and longitude, Cartesian coordinates, etc.), direction, speed, etc. Sensors may include GPS (Global Positioning System), BeiDou system, IMU (Inertial Measurement Unit, including accelerometers and gyroscopes), lidar, and cameras, etc. In this embodiment, to achieve more accurate positioning in a mining environment, data from multiple sensors is fused to improve positioning accuracy.

[0050] Step 120: Determine the second positioning data of the robot in the virtual space of a three-dimensional panoramic imaging constructed in real time based on the surrounding environment;

[0051] In this embodiment, the second positioning data is the robot's position and posture information in a virtual space constructed in real time based on its surrounding environment, using a 3D panoramic imaging system. This virtual space is obtained by 3D modeling the robot's environment, and can more intuitively reflect the robot's relative position within the environment.

[0052] Specifically, environmental information can be captured using cameras or other sensors to construct a three-dimensional virtual space model. This 3D virtual space is built in real-time to adapt to changes in the environment. For example, multiple visual cameras can be used to collect image information of the surrounding environment. Simultaneous Localization and Mapping (SLAM) technology is employed to construct a 3D environment map in real time and determine the robot's position within that map. By extracting feature points (such as corners and edges) from the images, matching them between different images, and using triangulation principles to calculate the 3D coordinates of these feature points, a virtual 3D environment model is gradually constructed, simultaneously determining the robot's position within the model.

[0053] In some optional embodiments, a lidar system can be installed on the robot. The lidar emits a laser beam into the surrounding environment and measures the time it takes for the beam to travel from emission to reflection, thus obtaining distance information about surrounding objects. By rotating the lidar, multiple distance data points of the surrounding environment are acquired. These data points are then used for 3D reconstruction to construct a 3D panoramic imaging model of the mine environment. In this virtual space, the robot's position and orientation are determined by identifying its own features (such as specific reflective markings on the robot) or by utilizing the robot's motion trajectory, thus obtaining the second positioning data.

[0054] Step 130: Correct the first positioning data according to the second positioning data to obtain the target positioning data of the robot in physical space.

[0055] In this embodiment, the robot is used for mine patrol. Mine terrain is highly undulating, with numerous slopes, gullies, and steps. Traditional methods of acquiring primary positioning data are prone to errors in complex terrain. For example, on slopes, the IMU may cause inaccurate acceleration measurements due to gravity, and wheel speedometers may lead to incorrect distance calculations on bumpy roads due to wheel slippage. However, the virtual space constructed by 3D panoramic imaging accurately reflects the actual terrain conditions. By correcting the primary positioning data with secondary positioning data, errors caused by terrain factors can be eliminated.

[0056] Besides eliminating errors caused by terrain factors, correcting the first positioning data based on the second positioning data can also help eliminate obstacle interference. Mines contain numerous obstacles such as ore piles, equipment, and buildings, which can affect the performance of traditional positioning sensors. For example, GPS signals may be blocked by obstacles, leading to inaccurate positioning; lidar may generate multipath reflections when measuring obstacles, affecting the accuracy of distance measurements. Using the second positioning data obtained from 3D panoramic imaging can bypass these obstacle interferences, providing more reliable positioning information and effectively correcting the first positioning data.

[0057] Specifically, there are several ways to correct the first positioning data based on the second positioning data. These include methods such as Kalman filtering, particle filtering, neural network fusion, or covariance cross-validation. Taking particle filtering as an example, a set of particles represents the robot's state probability distribution, with each particle representing one possible state (including position and orientation). Over time, these particles are weighted and resampled based on observation data, gradually approximating the robot's true state.

[0058] In robot localization, an initial set of particles is randomly generated, each with a different position and orientation. Based on initial localization data, the possible states of the particles at the next moment are predicted. Then, using second localization data (such as localization data obtained from 3D panoramic imaging), the weight of each particle is calculated; the weight represents the degree of matching between the particle and the observed data. Particles with lower weights are eliminated, while particles with higher weights are replicated to generate a new particle set. After multiple iterations, the center position of the particle set becomes the corrected target localization data.

[0059] In summary, accurate positioning is fundamental to the safe operation of mine patrol robots. In complex mining environments, inaccurate positioning can lead to collisions with obstacles or trapping the robot in dangerous areas. By correcting the initial positioning data to obtain more accurate target positioning data, the robot can ensure timely detection and avoidance of obstacles, guaranteeing the safety of the patrol process. The robot positioning method provided in this embodiment introduces second positioning data based on a real-time constructed 3D panoramic imaging virtual space. This virtual space reflects environmental changes in real time, and the second positioning data captures this dynamic information promptly, intuitively reflecting the robot's relative position within the environment. This corrects the initial positioning data, enabling the robot to accurately adapt to environmental changes and improving positioning accuracy and stability. It effectively reduces errors caused by environmental interference, thereby improving positioning precision.

[0060] In some optional embodiments, obtaining the first positioning data of the robot in physical space includes: obtaining wireless positioning data obtained by positioning the robot using a wireless positioning device; obtaining inertial positioning data obtained by positioning the robot using an inertial navigation positioning device; and fusing the wireless positioning data and the inertial positioning data to obtain the first positioning data.

[0061] In this embodiment, the fusion of wireless positioning data and inertial positioning data can fully leverage the advantages of both. Wireless positioning data can provide initial position reference and long-term position correction for inertial navigation, reducing the cumulative error caused by sensor drift. Inertial navigation data, on the other hand, can provide short-term, high-precision positioning information when wireless signals are unstable or lost, ensuring the continuity and stability of positioning. The first positioning data obtained by fusing the two more accurately reflects the actual physical space positioning.

[0062] Specifically, a complementary filtering fusion algorithm can be used, defining the parameters of the complementary filters, including low-pass filter coefficients and high-pass filter coefficients. In this embodiment, the low-pass filter coefficients are used to process wireless positioning data, and the high-pass filter coefficients are used to process inertial positioning data. Fusion is performed according to the following formula:

[0063] First positioning data = α × low-pass filtered wireless positioning data + (1-α) × high-pass filtered inertial positioning data

[0064] Here, α is the complementary filtering coefficient, with a value ranging from 0 to 1. The value of α can be adjusted according to the actual situation to balance the contributions of the two types of data; no specific restrictions are imposed here.

[0065] In some optional embodiments, fusing the wireless positioning data with the inertial positioning data to obtain the first positioning data includes: establishing a first Kalman filter model for position and velocity based on the position and velocity data in the wireless positioning data and the position and velocity data in the inertial positioning data, wherein in the first Kalman filter model, the inertial navigation and positioning device provides predicted position and velocity data for the next moment, and when the wireless positioning data arrives, the wireless positioning data is used as an observation; and the weights of the wireless positioning data and the inertial positioning data are dynamically adjusted through Kalman gain to correct the wireless positioning data, thereby obtaining the first positioning data.

[0066] In this embodiment, the weights of wireless positioning data and inertial positioning data are dynamically adjusted using Kalman gain, enabling the system to automatically allocate weights based on the real-time reliability of the two positioning methods. For example, when the wireless signal is stable, the weight of wireless positioning data is increased; when the wireless signal is interfered with, the weight of inertial positioning data is increased. This significantly improves positioning accuracy, stability, and real-time performance.

[0067] Taking a two-dimensional plane as an example, the process of establishing the first Kalman filter model includes the following:

[0068] (1) Define state variables:

[0069]

[0070] Among them, P x and P y V represents the robot's position coordinates in physical space. x and V y Let x be the robot's velocity in the x and y directions;

[0071] (2) State prediction

[0072] First location data x k Equations of state:

[0073] x k|k-1 =F k x k-1 +w k ;

[0074] Among them, F k Let w be the state transition matrix. k This refers to process noise, used to reflect the acceleration error of inertial navigation and positioning devices.

[0075] Covariance prediction:

[0076]

[0077] Wherein, the covariance matrix Q k Noise characteristic calibration of inertial navigation positioning device or IMU is required;

[0078] (3) The Kalman gain is calculated as follows:

[0079]

[0080] Among them, H k Let R be the observation matrix and the covariance matrix. k It is determined by the error of the wireless positioning device.

[0081] (4) State and Covariance Update

[0082] x k =x k|k-1 +K k (z k -H k x k|k-1 )

[0083] P k = (1-K) k H k )P k|k-1

[0084] Among them, Z k =H k x k +v k v k For measuring noise.

[0085] In some optional embodiments, determining the second positioning data of the robot in a virtual space constructed in real time based on the three-dimensional panoramic imaging of the surrounding environment includes: determining the position information of the robot in the surrounding environment based on the orientation information of each structure in the virtual space; calculating the speed information of the robot in the virtual space according to the continuously changing position information and the corresponding timestamp when the robot moves; and obtaining the second positioning data based on the position information and the speed information.

[0086] In this embodiment, wireless positioning signals are easily affected in the mining environment, and traditional positioning methods are prone to accumulating errors. Positioning in a three-dimensional virtual space provides structured information about the environment. By matching structural features in the virtual space, velocity calculation based on absolute position changes helps eliminate the accumulated errors of inertial sensors. Using the second positioning data as a reference point to correct the data of the physical sensors allows the short-term high-frequency data of the physical sensors to complement the long-term stable constraints of the virtual space, forming a balance mechanism similar to the human "visual-vestibular" system. Even when some sensors fail, the structural constraints of the virtual space can still provide basic navigation capabilities, achieving fault-tolerant positioning that degrades without compromising accuracy.

[0087] Specifically, a 3D point cloud map of the environment can be generated in real time using visual SLAM algorithms (such as ORB-SLAM3). Each structure is extracted as a location reference for the virtual space. The virtual space map is stored in the form of a semantic topology graph, containing the 3D coordinates, dimensions, and semantic labels of each structure.

[0088] The steps to determine location information in virtual space include the following:

[0089] Feature matching localization: The robot collects surrounding environmental data in real time and extracts feature points (such as SIFT keypoints). The feature points of the current frame are matched with structural features in the virtual space to calculate the robot's pose (position + attitude) relative to the virtual space.

[0090] For example,

[0091] Time t1: The robot is located at virtual space coordinates (x1, y1, z1) = (2.3m, 5.1m, 0.0m);

[0092] Time t2: Moved to coordinates (x2, y2, z2) = (2.5m, 5.3m, 0.0m);

[0093] The timestamp interval Δt = t2 - t1 = 0.1s.

[0094] Based on the coordinate changes and timestamp intervals, the velocity vector is calculated to be decomposed into v. x =0.2m / s, v y =0.2m / s, v z =0, the virtual space velocity v is calculated based on each velocity vector component. virtual =0.28m / s, then the second positioning data is as follows:

[0095] P virtual = (2.5m, 5.3m, 0.0m), v virtual =0.28m / s.

[0096] In some optional embodiments, the step of correcting the first positioning data based on the second positioning data to obtain the target positioning data of the robot in physical space includes: establishing a second Kalman filter model based on the second positioning data and the first positioning data, wherein, in the second Kalman filter model, the second positioning data is used as the observation value and the first positioning data is used as the prediction value; and dynamically adjusting the weights of the first positioning data and the second positioning data through Kalman gain to correct the first positioning data to obtain the target positioning data.

[0097] In this embodiment, the construction process of the second Kalman filter model is similar to that of the first Kalman filter model. During the construction of the second Kalman filter model, the observation noise covariance can be set according to the positioning accuracy of visual SLAM.

[0098] In some alternative embodiments, a two-layer Kalman filter can be constructed: the first-layer Kalman filter fuses wireless and inertial positioning data to address positioning jumps caused by wireless signal obstruction or interference, while suppressing drift errors of the inertial sensor. The second-layer Kalman filter further fuses the first-layer output with virtual space positioning to correct errors caused by dynamic environmental changes. Through two recursive optimizations, errors are reduced layer by layer, ultimately resulting in a significant improvement in positioning accuracy.

[0099] In some optional embodiments, the information of the three-dimensional panoramic imaging is acquired by at least one of a visual acquisition device, a lidar module, and an infrared sensing device.

[0100] In this embodiment, multimodal data complementarity further enhances environmental perception capabilities, making the constructed 3D panoramic imaging more closely resemble the real environment. For example, a virtual space is generated by combining information from a visual acquisition device, a LiDAR module, and an infrared sensor. The visual acquisition device is equipped with a circular industrial camera array. Based on real-time panoramic video monitoring, the distance information of objects in the overlapping area captured by adjacent cameras can be obtained using the binocular vision principle of adjacent cameras. This allows for the direct construction of a 3D panoramic image of the robot's surroundings, achieving accurate environmental perception and stereo reconstruction. For images of non-overlapping areas from adjacent cameras, the distance information of surrounding environmental targets can be obtained by combining LiDAR, thereby completing the depth information and generating a 3D panoramic image. In situations with no light or weak light, the depth information obtained by the LiDAR module can be combined with thermal imaging obtained by the infrared sensor module to generate a 3D panoramic image, ensuring accurate environmental perception even under conditions of insufficient light and failure to acquire panoramic video images. This better adapts to mining scenarios.

[0101] Example 2:

[0102] Based on the above embodiments, this embodiment provides a robot path planning method, including:

[0103] Step 210: Obtain the target positioning data of the robot in physical space using the robot positioning method described above;

[0104] In this embodiment, target positioning data is obtained by utilizing the multi-layered Kalman filter fusion described in the previous embodiments. First, a first Kalman filter model is established using wireless positioning (e.g., GPS) and inertial positioning (e.g., IMU) data. The weights of the two types of data are dynamically adjusted to correct noise and errors in the wireless positioning data, resulting in first positioning data. Next, based on the orientation information of various structures in the virtual space, combined with the robot's continuous position changes and timestamps, velocity information is calculated to obtain second positioning data. Finally, the first and second positioning data are input into a second Kalman filter model for further fusion to obtain the target positioning data.

[0105] Step 220: Determine the current motion trajectory of the robot based on the target positioning data, and determine the trajectory and trajectory probability of the moving target based on the position and speed data of the moving target in the environment;

[0106] In this embodiment, after acquiring the target positioning data, the robot's current trajectory is analyzed, and combined with the position and velocity data of moving targets in the environment (such as other robots, pedestrians, etc.), the trajectory and trajectory probability of the moving target are determined. The trajectory probability can be predicted using statistical methods or machine learning algorithms (such as Hidden Markov Models), taking into account the moving target's historical behavior, environmental factors, etc. This step provides an important decision-making basis for subsequent path planning.

[0107] Step 230: Based on the trajectory and trajectory probability of the moving target, and using the lowest probability of the robot's current trajectory encountering the moving target and the surrounding environment as a constraint, generate the robot's motion path in real time.

[0108] In this embodiment, the constraint is the lowest probability that the robot's current trajectory will encounter the moving target and the surrounding environment. For example, an encounter probability threshold of 0.1 is set, and when the predicted encounter probability exceeds this threshold, path replanning is triggered.

[0109] The path planning algorithm can employ Dijkstra's algorithm or reinforcement learning, combined with a virtual map of the mine and constraints, to generate the robot's motion path in real time. During path planning, further consideration is given to the robot's dynamic constraints, such as maximum speed and acceleration, and the limitations of the mine environment, such as passage width and obstacle locations, to further improve the reliability of path planning.

[0110] Example 3:

[0111] Based on the above embodiments, this embodiment provides an application example. A robot is provided for use in mine patrol tasks. For example... Figure 2The diagram shows the overall structure of the robot. The robot includes a motion chassis 210 and an intelligent body 220. The motion chassis has independent control functions and consists of a mobile platform and a motor drive device, which are modularly connected to the intelligent body. The intelligent body is externally equipped with a panoramic acquisition device 230, a servo gimbal 240, a wireless positioning device 250, a wireless communication antenna 260, an audio acquisition device 270, and an audio playback device 280. Internally, the intelligent body is equipped with an inertial navigation and positioning device, a wireless communication device, and a data processing device. The inertial navigation and positioning device integrates an inertial sensor. The mobile platform is a tracked mobile platform.

[0112] The panoramic acquisition device consists of at least three rows of industrial cameras (visual acquisition devices), a LiDAR module, and an infrared sensor module. The areas covered by adjacent cameras, LiDAR, and infrared sensors overlap by at least 30%, and the cameras, LiDAR, and infrared sensors are vertically aligned. The panoramic acquisition device can generate video data, point cloud data, and infrared sensor data, stitching together the robot's surrounding environment in real time to form a distortion-free, continuous panoramic video monitoring image with 360 degrees horizontally and 180 degrees vertically. This addresses the problems of incomplete video information and unclear targets in mobile work environments.

[0113] The servo gimbal is primarily used to stabilize the panoramic vision acquisition device, ensuring stability during video stitching as the robot moves. A motor for pitch angle adjustment is mounted at the bottom of the servo gimbal. When capturing distant video footage in a wide field of view, the servo gimbal stabilizes the panoramic vision acquisition device, keeping it aligned with the mounting plane, enabling real-time panoramic monitoring of the robot's entire horizontal field of view. When the target object is too close, and the panoramic vision acquisition device can only capture a portion of it, the pitch angle of the servo gimbal can be adjusted in real-time as needed, allowing the panoramic vision acquisition device to tilt and deflect to capture and detect the entire object.

[0114] The audio acquisition device is used to collect audio information about the robot's environment. The audio playback device is used to play audio information into the robot's environment. The sound acquired by the audio acquisition device and the panoramic video monitoring information formed by stitching together the high-precision industrial camera module array in the panoramic vision acquisition device are synchronously synthesized by the data processing device and simultaneously displayed in real time on the user terminal. When the robot is working in the mining area, it can detect abnormal volume in real time. When the volume exceeds the preset normal operating volume amplitude, it performs secondary analysis based on the visual information to determine if any emergency has occurred. If the abnormal information is confirmed, a warning is issued to the operator through the playback device, and the information is simultaneously sent to the user terminal.

[0115] The wireless positioning device is used to achieve wireless communication positioning and to obtain real-time positioning information of the robot in complex mining scenarios.

[0116] The wireless communication antenna is used to enhance the ability of the wireless communication device to transmit and receive signals.

[0117] The inertial navigation and positioning device integrates an accelerometer sensor, a gyroscope, and a geomagnetic sensor.

[0118] The wireless communication device is used to transmit the robot's perception data. Through this device, real-time collected panoramic visual information, environmental sound information, positioning information, and inertial navigation sensing information are transmitted to the user terminal. At the same time, it can receive remote control command information issued by the user terminal to realize remote interactive control of the robot.

[0119] The data processing device integrates a high-performance processor, supports CUDA acceleration, and can run panoramic video stitching algorithms, 3D environment construction algorithms, intelligent navigation algorithms, chassis motor drive control algorithms, servo gimbal control algorithms, optimal path algorithms, target detection and recognition large models, and target tracking and marking algorithms.

[0120] The panoramic video stitching algorithm intelligently selects a panoramic video stitching method based on the real-time requirements of video stitching. The algorithm utilizes the specifications of the panoramic vision acquisition device, the structural features of the robot's environment, lighting characteristics, mining truck type characteristics, mining truck size characteristics, mining truck structural characteristics, and worker clothing characteristics to establish a threshold function. The calculated threshold is compared with a set threshold to improve the SURF registration algorithm and enhance registration accuracy. The specifications of the panoramic vision acquisition device include the number of industrial cameras and their parameters.

[0121] The panoramic video stitching algorithm employs a CUDA-accelerated intelligent image fusion algorithm. Based on the specifications of the panoramic vision acquisition device, the structural features of the robot's environment, lighting characteristics, mine card type characteristics, mine card size characteristics, mine card structure characteristics, and worker clothing characteristics, a fusion strategy classification function is established for the images on both sides of the stitching line, based on indicators such as color and brightness difference, texture difference, distance difference, and structural similarity. By comparing the calculated strategy value with the set threshold, a suitable stitching seam smoothing algorithm and the optimal stitching line fusion algorithm are autonomously selected to achieve real-time panoramic monitoring of the robot's surrounding environment at 360° horizontally and 180° vertically. Through keyframe processing, the panoramic video is stably stitched.

[0122] The panoramic video stitching algorithm mainly includes two steps: image registration and image fusion. In the image registration stage, the robot adaptively adjusts the detection threshold based on the characteristics of its environment. When overlapping images with obvious features appear, the threshold is lowered; when overlapping images with fewer feature points appear, the detection threshold is increased. The initial threshold is set to 30%, and subsequent detections do not fall below this threshold. A threshold-based SURF algorithm is used to extract and match features in overlapping areas of the images to improve the accuracy of image registration. Before image fusion, the panoramic vision acquisition device performs illumination compensation during its movement, considering the diversity of weather conditions and the differences in visibility within a 360-degree horizontal range. In the image fusion stage, the visual dynamic changes caused by the robot's movement are used to dynamically update the optimal stitching line on keyframes. A fusion strategy classification function is established using pixel information from images taken on both sides of the stitching line and feature information from the robot's environment. The obtained strategy value is compared with the initially set threshold, and the fusion algorithm is adaptively selected to achieve seamless stitching of the panoramic image with visual consistency.

[0123] The described 3D environmental reconstruction algorithm utilizes panoramic video images stitched together from an industrial camera array in a panoramic acquisition device to obtain real-time orientation information of the robot's surrounding environment. Based on real-time panoramic video monitoring, the distance information of objects within the overlapping areas captured by adjacent cameras can be obtained using the binocular vision principle of adjacent cameras. This allows for the direct construction of a 3D panoramic image of the robot's surroundings, achieving accurate environmental perception and stereo reconstruction. For images of non-overlapping areas from adjacent cameras, the distance information of surrounding environmental targets can be obtained by combining the LiDAR in the visual acquisition device, thereby completing the depth information and generating a 3D panoramic image. In the case of no light or weak light, the depth information obtained by the LiDAR module in the visual acquisition device can be combined with thermal imaging obtained by the infrared sensor module to generate a 3D panoramic image, ensuring accurate environmental perception even under conditions of insufficient light and failure of panoramic video image acquisition.

[0124] The intelligent navigation algorithm autonomously integrates data from a panoramic visual acquisition device, an inertial navigation positioning device, and a wireless positioning device to achieve autonomous navigation for the intelligent robot. First, it determines the robot's location using the latitude and longitude provided by the wireless positioning device. Then, it fuses the wireless positioning data with the inertial navigation positioning data to obtain the first positioning data. This improves the positioning accuracy of the intelligent robot during long-term navigation.

[0125] Then, using data from the panoramic vision acquisition device, visual imaging is performed on the environment in which the intelligent robot is located. Based on this, objects in the environment are identified, and the size and orientation of the objects are estimated according to their attributes. All identified objects are constructed according to their orientation and size to create a panoramic imaging virtual space, and the position of the intelligent robot in the virtual space is estimated to obtain the first positioning data and store the virtual space data.

[0126] Then, when the intelligent robot moves, the estimated position of the intelligent robot in the virtual space (first positioning data) is fused again with the position data (first positioning data) fused by wireless positioning and inertial navigation positioning to determine the precise position of the intelligent robot in the physical space.

[0127] 1) The fusion process of wireless positioning data and inertial navigation positioning data: The wireless positioning device provides position and velocity data, and the inertial navigation positioning data provides position and velocity data. A Kalman filter model for position and velocity is established for both devices to suppress the estimation error between them and improve the fusion accuracy of position and velocity. The specific calculation process is similar to the above embodiment, and will not be repeated here to avoid repetition.

[0128] 2) The position data of the intelligent robot in the virtual space is further fused with the position data obtained from the wireless positioning / inertial navigation positioning fusion: The virtual space is an environmental space built based on the robot's vision. Vision can model the structural dimensions of various targets in the environment with high accuracy. After the virtual space is established, the robot can determine its position and attitude in the environment based on the orientation information of various structures in the virtual space. Based on the position, the robot's velocity information in the virtual environment can be estimated. A Kalman filter is then established again based on the position and velocity information obtained in the virtual environment, as well as the position and velocity information after the fusion of wireless positioning / inertial navigation positioning, to suppress position and velocity errors and improve the position and velocity accuracy of the intelligent robot in both the actual physical space and the virtual space. The specific calculation process is similar to the above embodiment and will not be repeated here to avoid repetition.

[0129] The optimal path algorithm is based on the three-dimensional real-time detection and construction data of the environment. It combines the position of the intelligent robot itself with the position and speed data of the moving target in the environment to estimate the current trajectory of the intelligent robot and the trajectory of the moving target, estimate the trajectory probability of the moving target, and use the minimum probability of the intelligent robot's trajectory meeting the moving target and the surrounding environment as a constraint to generate the intelligent robot's trajectory in real time, thereby realizing the optimal path planning of the intelligent robot.

[0130] The target detection and recognition model utilizes data collected by a panoramic vision acquisition device to detect, identify, and classify different types of targets. The target tracking and labeling algorithm, based on 3D environmental detection and construction data, records the position and type of detected moving targets, extracts the features of each moving target, calculates the speed and orientation of the moving targets, and estimates their trajectories. When multiple moving targets meet and intersect, the algorithm matches the calculated target position, speed, orientation, type, and features with the parameters of each moving target after the intersection, and then labels them, thereby achieving real-time tracking and differentiation of multiple moving targets.

[0131] The robot in this embodiment has the following beneficial effects:

[0132] (1) Relying on the high-precision industrial camera module array in the panoramic vision acquisition device, a 360° panoramic monitoring video can be stitched together in real time. Simultaneously, the panoramic imagery is used to detect and identify targets of interest, enabling autonomous obstacle avoidance and navigation, providing precise visual information for task execution. Based on this, to further enhance environmental perception capabilities, the robot acquires its own position information through wireless positioning and inertial navigation devices, and combines this with an optimal path algorithm to perform efficient path planning in the mining environment. Finally, with the help of intelligent navigation algorithms, the robot achieves autonomous navigation and precise movement, ensuring stable operation and efficient work in complex environments.

[0133] (2) Relying on the high-precision industrial camera module array in the panoramic vision acquisition device, a 360° panoramic monitoring video can be stitched together in real time. However, in the absence of light or in low light conditions, the visibility of panoramic monitoring may be limited. To improve perception capabilities, the high-performance processor in the data processing device integrates a 3D construction algorithm. Combined with the lidar data and infrared sensor data acquired by the panoramic vision acquisition device, it achieves real-time fusion of multi-source sensor information through key steps such as data preprocessing, feature extraction and matching, graph optimization and SLAM, dynamic environment processing, and map post-processing. With the help of this technology, intelligent robots can accurately detect the surrounding environment and construct a high-precision 3D panoramic imaging virtual space, while presenting a realistic geographic space image to the user, thereby effectively compensating for the perception defects of panoramic monitoring in low light environments and significantly improving environmental understanding capabilities.

[0134] (3) By introducing second positioning data from a real-time constructed 3D panoramic imaging virtual space based on the surrounding environment, the 3D panoramic imaging virtual space can reflect environmental changes in real time. The second positioning data can capture these dynamic information in a timely manner and correct the first positioning data, enabling the robot to accurately adapt to environmental changes and improve the accuracy and stability of positioning. This can effectively reduce errors caused by environmental interference and improve positioning accuracy.

[0135] (4) A two-layer Kalman filter was constructed: The first-layer Kalman filter fuses wireless positioning and inertial positioning data to solve the positioning jump problem when the wireless signal is blocked or interfered with, while suppressing the drift error of the inertial sensor. The second-layer Kalman filter further fuses the output of the first layer with virtual space positioning to correct the error caused by dynamic changes in the environment. Through two recursive optimizations, the error is reduced layer by layer, and the final positioning accuracy is significantly improved.

[0136] (5) The sound acquired by the audio acquisition device and the panoramic video monitoring image information formed by splicing the high-precision industrial camera module array in the panoramic vision acquisition device are synchronously synthesized by the data processing device and simultaneously displayed in real time on the user terminal. When the intelligent robot is working in the mining area, it can detect abnormal volume in real time. When the volume exceeds the preset normal operating volume amplitude, it performs secondary analysis in combination with visual information to determine if there is an emergency. If it is confirmed to be abnormal information, it issues a warning to the workers through the playback device and sends the information to the user terminal at the same time.

[0137] Example 4:

[0138] Another embodiment of this application relates to a robot positioning device. The implementation details of the robot positioning device in this embodiment are described below. The following details are for ease of understanding and are not essential for implementing this solution. A schematic diagram of the robot positioning device in this embodiment can be seen as follows: Figure 3 As shown, it includes a first positioning data acquisition module 310, a second positioning data acquisition module 320, and a correction module 330.

[0139] The first positioning data acquisition module 310 is used to acquire the robot's first positioning data in physical space;

[0140] The second positioning data acquisition module 320 is used to determine the second positioning data of the robot in a virtual space of a three-dimensional panoramic imaging constructed in real time based on the environment.

[0141] The correction module 330 is used to correct the first positioning data based on the second positioning data to obtain the target positioning data of the robot in physical space.

[0142] The first positioning data acquisition module includes:

[0143] A wireless positioning data acquisition unit is used to acquire wireless positioning data obtained by positioning the robot using a wireless positioning device.

[0144] An inertial positioning data acquisition unit is used to acquire inertial positioning data obtained by positioning the robot using an inertial navigation and positioning device.

[0145] The first positioning data generation unit is used to fuse the wireless positioning data with the inertial positioning data to obtain the first positioning data.

[0146] In some optional embodiments, the first positioning data generation unit includes:

[0147] The first Kalman filter model building subunit is used to build a first Kalman filter model for position and velocity based on the position and velocity data in the wireless positioning data and the position and velocity data in the inertial positioning data. In the first Kalman filter model, the inertial navigation and positioning device provides the predicted position and velocity data for the next moment. When the wireless positioning data arrives, the wireless positioning data is used as the observation value.

[0148] The first correction subunit is used to dynamically adjust the weights of the wireless positioning data and the inertial positioning data through Kalman gain to correct the wireless positioning data and obtain the first positioning data.

[0149] In some optional embodiments, the second positioning data acquisition module includes:

[0150] A virtual location information determination unit is used to determine the robot's location information in its environment based on the orientation information of each structure in the virtual space.

[0151] A speed calculation unit is used to calculate the speed information of the robot in the virtual space based on the continuously changing position information and the corresponding timestamp when the robot moves.

[0152] The second positioning data generation unit is used to obtain the second positioning data based on the position information and the speed information.

[0153] In some optional embodiments, the correction module includes:

[0154] The second Kalman filter model building unit is used to build a second Kalman filter model based on the second positioning data and the first positioning data, wherein the second positioning data is used as the observation value and the first positioning data is used as the prediction value in the second Kalman filter model.

[0155] The second correction unit is used to dynamically adjust the weights of the first positioning data and the second positioning data through Kalman gain to correct the first positioning data and obtain the target positioning data.

[0156] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0157] Example 5:

[0158] Another embodiment of this application relates to an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the robot localization methods in the above embodiments.

[0159] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0160] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0161] Example 6:

[0162] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0163] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A robot localization method, characterized in that, include: Acquire the robot's initial positioning data in physical space; Determine the second positioning data of the robot in a virtual space constructed in real time based on the three-dimensional panoramic imaging of its environment; The first positioning data is corrected based on the second positioning data to obtain the target positioning data of the robot in physical space; The acquisition of the robot's first positioning data in physical space includes: Obtain wireless positioning data obtained by positioning the robot using a wireless positioning device; Acquire inertial positioning data obtained by positioning the robot using an inertial navigation and positioning device; The wireless positioning data and the inertial positioning data are fused to obtain the first positioning data; The step of fusing the wireless positioning data with the inertial positioning data to obtain the first positioning data includes: Based on the position and velocity data in the wireless positioning data and the position and velocity data in the inertial positioning data, a first Kalman filter model for position and velocity is established. In the first Kalman filter model, the inertial navigation and positioning device provides the predicted position and velocity data for the next moment. When the wireless positioning data arrives, the wireless positioning data is used as the observation value. The wireless positioning data is corrected by dynamically adjusting the weights of the wireless positioning data and the inertial positioning data using Kalman gain, thus obtaining the first positioning data. The determination of the second positioning data of the robot in a virtual space constructed in real time based on the surrounding environment in a three-dimensional panoramic imaging system includes: Based on the orientation information of each structure in the virtual space, the position information of the robot in its environment is determined; When the robot moves, the speed information of the robot in the virtual space is calculated based on the continuously changing position information and the corresponding timestamp; Based on the location information and the speed information, the second positioning data is obtained; The step of correcting the first positioning data based on the second positioning data to obtain the target positioning data of the robot in physical space includes: Based on the second positioning data and the first positioning data, a second Kalman filter model is established, wherein the second positioning data is used as the observed value and the first positioning data is used as the predicted value. By dynamically adjusting the weights of the first positioning data and the second positioning data using Kalman gain to correct the first positioning data, the short-term high-frequency data from the physical sensor and the long-term stable constraints of the virtual space complement each other, thus obtaining the target positioning data. The virtual space construction method for the three-dimensional panoramic imaging includes: using panoramic video images stitched together from an industrial camera array in a panoramic acquisition device to obtain the orientation information of the robot's surrounding environment in real time; based on real-time panoramic video monitoring, using the binocular vision principle of adjacent cameras to obtain the distance information of objects in the overlapping areas captured, so as to directly construct a three-dimensional panoramic image of the robot's surroundings, realizing environmental perception and stereo reconstruction; for the images of non-overlapping areas of adjacent cameras, combining the distance information of surrounding environmental targets obtained by the lidar in the visual acquisition device to complete the depth information and generate a three-dimensional panoramic image; in the case of no light or weak light, the depth information obtained by the lidar module in the visual acquisition device is combined with the thermal imaging obtained by the infrared sensor module to generate a three-dimensional panoramic image, so as to perceive the environment under the condition of insufficient light and failure of panoramic video image acquisition.

2. A robot path planning method, characterized in that, include: The robot localization method of claim 1 is used to obtain the target localization data of the robot in physical space; The robot's current trajectory is determined based on the target positioning data, and the trajectory and trajectory probability of the moving target are determined based on the position and speed data of the moving target in the environment. Based on the trajectory and trajectory probability of the moving target, and using the lowest probability of the robot's current trajectory encountering the moving target and the surrounding environment as a constraint, the robot's motion path is generated in real time.

3. A robot positioning device for implementing the robot positioning method of claim 1, characterized in that, include: The first positioning data acquisition module is used to acquire the robot's first positioning data in physical space; The second positioning data acquisition module is used to determine the second positioning data of the robot in the virtual space of the three-dimensional panoramic imaging constructed in real time based on the environment. The correction module is used to correct the first positioning data based on the second positioning data to obtain the target positioning data of the robot in physical space.

4. A robot, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the robot localization method as described in claim 1, or the robot path planning method as described in claim 2.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot localization method of claim 1 or the robot path planning method of claim 2.

Images