A Navigation and Positioning System for Patrol Robots Based on Multimodal Information Fusion

Abstract

This invention discloses a navigation and positioning system for a patrol robot based on multimodal information fusion, relating to the field of navigation and positioning. The system includes a data acquisition module that collects image data, 3D point cloud data, and acceleration data; a preprocessing module that preprocesses the image data and 3D point cloud data; a first positioning module that obtains a first determined position based on the acceleration data; a matrix module that determines the values ​​of each element in the matrix based on training data; a second positioning module that obtains a second determined position based on standard image data, standard point cloud data, and the matrix; a positioning confirmation module that determines the robot's actual position based on the first and second determined positions; and a navigation module that plans the route. This invention effectively improves the accuracy of robot navigation and positioning by fusing multi-source data.

Description

Technical Field

[0001] This invention relates to the field of navigation and positioning, specifically a navigation and positioning system for patrol robots based on multimodal information fusion. Background Technology

[0002] In modern industry, security and many other fields, patrol robots are being used more and more widely, and the accuracy and reliability of their navigation and positioning have become the key factors that determine their work efficiency.

[0003] Traditional navigation and positioning technologies for patrol robots have many limitations. In the application of single-modal sensors, navigation systems that rely solely on visual sensors can acquire rich environmental texture information, but in poor lighting conditions (such as direct sunlight, low light, or complete darkness), image feature extraction and matching may deviate or even fail, leading to inaccurate positioning. While lidar can accurately measure distances and build high-precision environmental geometric models, it lacks effective perception of semantic information such as texture and color in the environment, and is prone to positioning confusion when facing environments with similar geometric structures.

[0004] Multimodal information fusion can organically integrate data from different types of sensors (such as vision, lidar, inertial measurement units, etc.), give full play to the advantages of each modality of data, make up for the shortcomings of a single modality, and thus improve the comprehensiveness, accuracy and reliability of the system's environmental perception.

[0005] Improving the accuracy of navigation and positioning for patrol robots is a problem we need to solve. To this end, we present a patrol robot navigation and positioning system based on multimodal information fusion. Summary of the Invention

[0006] The purpose of this invention is to provide a navigation and positioning system for patrol robots based on multimodal information fusion.

[0007] The objective of this invention can be achieved through the following technical solution: a patrol robot navigation and positioning system based on multimodal information fusion, comprising:

[0008] Data acquisition module: Collects image data, 3D point cloud data, and acceleration data around the robot;

[0009] Preprocessing module: preprocesses image data and 3D point cloud data to obtain standard image data and standard point cloud data respectively;

[0010] The first positioning module processes the acceleration data to obtain the robot's initial position; the matrix module is used to determine the specific values ​​of each element in the matrix.

[0011] The second positioning module: obtains the robot's second judgment position based on standard image data, standard point cloud data, and using a matrix;

[0012] Positioning confirmation module: used to determine the robot's actual position based on the first and second determined positions;

[0013] Navigation module: Based on the robot's actual position and the target position, it uses dynamic programming algorithms to plan the robot's patrol route.

[0014] Preferably, the image data is an image of the robot's surrounding environment taken using a camera mounted on the robot;

[0015] The three-dimensional point cloud data is the three-dimensional coordinate data of spatial points around the robot obtained using LiDAR;

[0016] The acceleration data is a curve showing the change of the robot's acceleration along the three coordinate axes in a spatial rectangular coordinate system over time.

[0017] Preferably, the process of preprocessing image data and 3D point cloud data to obtain standard image data and standard point cloud data is as follows:

[0018] Gaussian filtering is used to remove noise from the image data, and histogram equalization is used to enhance the contrast of the image data, thereby obtaining standard image data.

[0019] The isolated points in the 3D point cloud data are removed by using a filter, and the 3D point cloud data is downsampled by voxel filtering to obtain standard point cloud data.

[0020] Preferably, the process of processing the acceleration data to obtain the robot's first determined position is as follows:

[0021] The acceleration data is processed to obtain velocity data, and the velocity data is integrated to obtain the robot's three-dimensional coordinates.

[0022] The first judgment position is obtained based on the robot's patrol starting point and three-dimensional coordinates.

[0023] Preferably, the process of determining the specific values ​​of each element in the matrix to be used is as follows:

[0024] Acquire training data from different locations and process the training data to obtain point cloud training matrices and image training matrices for different locations.

[0025] Initialize the adjustment matrix, and use the adjustment matrix to fuse the point cloud training matrix and the image training matrix to obtain the fused training matrix corresponding to different locations;

[0026] Calculate the matrix distance between the fusion training matrices corresponding to the two locations, and preset a matrix distance threshold. Based on the comparison between the matrix distance and the matrix distance threshold, adjust the element values ​​in the adjustment matrix to obtain the usable matrix.

[0027] The matrix distance is the absolute difference between the two fused training matrices.

[0028] Preferably, the process of obtaining the robot's second judgment position based on standard image data and standard point cloud data is as follows:

[0029] The SIFT algorithm is used to extract feature points from standard image data, and the feature points are converted into a matrix form as the image matrix.

[0030] Geometric features are extracted from standard point cloud data using a point cloud segmentation model, and these geometric features are converted into a matrix form, which is then used as a point cloud matrix.

[0031] By fusing the point cloud matrix and the image matrix using a matrix, the resulting fusion matrix is ​​compared with a comparison fusion matrix in a preset database to obtain the robot's second judgment position.

[0032] Preferably, the process of determining the robot's actual position based on the first and second determined positions is as follows:

[0033] Obtain the spatial distance between the first judgment position and the second judgment position, and preset a distance threshold. If the spatial distance is less than or equal to the distance threshold, connect the first judgment position and the second judgment position with a straight line as a position line, and take the midpoint of the position line as the actual position of the robot.

[0034] If the spatial distance is greater than the distance threshold, the third judgment position is obtained according to the satellite navigation system, and the actual position of the robot is determined by the first judgment position, the second judgment position and the third judgment position.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] This invention obtains a first judgment position by processing acceleration data, obtains a usage matrix by processing training data, fuses a point cloud matrix and an image matrix using the usage matrix to obtain a fusion matrix, and obtains a second judgment position of the robot by comparing the fusion matrix with a contrast fusion matrix. The actual position of the robot is determined based on the spatial distance between the first judgment position and the second judgment position, thereby improving the accuracy of robot navigation and positioning. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0038] Figure 1 This is a schematic diagram of the present invention. Detailed Implementation

[0039] like Figure 1 As shown, a navigation and positioning system for patrol robots based on multimodal information fusion includes...

[0040] Data acquisition module: Collects image data, 3D point cloud data, and acceleration data around the robot;

[0041] The image data is an image of the robot's surrounding environment taken using a camera mounted on the robot.

[0042] The three-dimensional point cloud data is the three-dimensional coordinate data of spatial points around the robot obtained using LiDAR;

[0043] In detail, lidar calculates distance by emitting a laser beam and measuring the time it takes for the reflected light to travel, thereby obtaining the three-dimensional coordinates of surrounding points in space. Using the three-dimensional coordinates of these points, the shape of objects around the robot can be constructed.

[0044] The acceleration data is a curve showing the change of the robot's acceleration along the three coordinate axes in a spatial rectangular coordinate system over time;

[0045] In detail, a spatial rectangular coordinate system is established with the robot's initial position point, i.e., the starting point where the robot begins its patrol, as the origin;

[0046] Preprocessing module: preprocesses image data and 3D point cloud data to obtain standard image data and standard point cloud data respectively;

[0047] Gaussian filtering is used to remove noise from the image data, and histogram equalization is used to enhance the contrast of the image data, thereby obtaining standard image data.

[0048] The isolated points in the 3D point cloud data are removed by using a filter, and the 3D point cloud data is downsampled by voxel filtering to obtain standard point cloud data.

[0049] The first positioning module processes the acceleration data to obtain the robot's initial position.

[0050] The acceleration data is processed to obtain velocity data, and the velocity data is integrated to obtain the robot's three-dimensional coordinates.

[0051] In detail, we take the curve of the robot's acceleration on the x-axis as an example, and the processing on the y-axis and z-axis is the same as that on the x-axis.

[0052] Specifically, the robot's initial velocity is set to 0. The curve of the robot's acceleration changing with time is integrated with respect to the current moment to obtain the robot's velocity value at the current moment. The velocity values ​​are then connected sequentially to form a velocity curve, which is used as the velocity data.

[0053] Specifically, by integrating the velocity curve at the current moment, we can obtain the x-coordinate corresponding to the current moment. Similarly, we can obtain the robot's y-coordinate and z-coordinate at the current moment, thus obtaining the robot's three-dimensional coordinates at the current moment.

[0054] The first judgment position is obtained based on the robot's patrol starting point and three-dimensional coordinates;

[0055] In detail, the patrol starting point is marked on the robot's 3D patrol map, the 3D coordinates of the robot's current position are obtained, and the 3D coordinates of the current position are used as the robot's first judgment position relative to the patrol starting point.

[0056] Using the matrix module: Determine the specific values ​​of each element in the matrix to use;

[0057] Acquire training data from different locations and process the training data to obtain point cloud training matrices and image training matrices for different locations.

[0058] Specifically, the acquisition of training data should include the robot's locations at various points within its patrol area;

[0059] Initialize the adjustment matrix, and use the adjustment matrix to train the point cloud matrix. and image training matrix By performing fusion, a fusion training matrix corresponding to different locations can be obtained. ;

[0060] In detail, initializing the adjustment matrix refers to randomly assigning values ​​to the elements of the adjustment matrix, with each element having a value between 0 and 1. The point cloud training matrix is ​​then used with the fusion formula. and image training matrix The fusion is performed, and the expression for the fusion formula is:

[0061] ;

[0062] in, Represents the adjustment matrix;

[0063] Calculate the matrix distance between the fusion training matrices corresponding to the two locations, and preset a matrix distance threshold. Based on the comparison between the matrix distance and the matrix distance threshold, adjust the element values ​​in the adjustment matrix to obtain the usable matrix.

[0064] In detail, matrix distance refers to subtracting the elements at corresponding positions in the two fusion matrices corresponding to two locations, then taking the square of the result as the squared difference of the corresponding positions, and summing the squared differences of all positions in the matrix to obtain the matrix distance.

[0065] In detail, if the matrix distance calculated between two locations is less than the preset matrix distance threshold, the elements in the adjustment matrix are adjusted through the backpropagation mechanism. The adjustment of the elements in the adjustment matrix is ​​stopped only when the matrix distance between any two locations is greater than the preset matrix distance threshold, and the adjusted adjustment matrix is ​​used as the matrix to be used.

[0066] In detail, the reason for ensuring that the matrix distance between any two locations is greater than the preset matrix distance threshold is to ensure that the fusion training matrix between different locations has a certain degree of difference. If the matrix distance between two locations is too small, the robot's location may be misjudged due to accidental factors when the fusion matrix is ​​used to determine the location later.

[0067] In detail, the point cloud training matrix and the image training matrix of each location are fused using a matrix, and the result is used as the comparison fusion matrix for each location and stored in the database.

[0068] The second positioning module: obtains the robot's second judgment position based on standard image data, standard point cloud data, and using a matrix;

[0069] The SIFT algorithm is used to extract feature points from standard image data, and the feature points are converted into a matrix form as the image matrix.

[0070] In detail, each row of the image matrix represents a vector represented by a feature point; the number of rows represents the number of feature points, and the number of columns represents the dimension of the vectors transformed from the feature points;

[0071] In detail, the SIFT algorithm can detect feature points of an image at different scales by constructing a scale space, and the extracted feature points have many characteristics such as scale invariance and rotation invariance.

[0072] Geometric features are extracted from standard point cloud data using a point cloud segmentation model, and these geometric features are converted into a matrix form, which is then used as a point cloud matrix.

[0073] In detail, each row of the point cloud matrix represents a vector representing a geometric feature; the number of rows represents the number of geometric features, and the number of columns represents the dimension of the vectors transformed from the geometric features;

[0074] In detail, planar features include planar features, corner features, etc.

[0075] In detail, the number of rows and columns of the image matrix and the point cloud matrix need to be consistent. If the number of rows of the two matrices is inconsistent, principal component analysis is used to reduce the number of rows of the matrix with more rows, so that the number of rows of the two matrices is consistent. If the number of columns of the two matrices is inconsistent, pooling can be used on the matrix with more columns to reduce the number of columns, thereby ensuring that the number of columns of the two matrices is consistent.

[0076] Specifically, the pooling method can be either max pooling or average pooling, and the pooling window is determined based on the difference in the number of columns of the two matrices.

[0077] By fusing the point cloud matrix and the image matrix using a matrix, the fusion matrix is ​​compared with a comparison fusion matrix in a preset database to obtain the robot's second judgment position.

[0078] In detail, comparison refers to calculating the matrix distance between the fusion matrix and each comparison fusion matrix, and taking the location corresponding to the fusion matrix with the smallest matrix distance as the robot's second judgment position.

[0079] Positioning confirmation module: Determines the robot's actual position based on the first and second determined positions;

[0080] Obtain the spatial distance between the first judgment position and the second judgment position, and preset a distance threshold. If the spatial distance is less than or equal to the distance threshold, connect the first judgment position and the second judgment position with a straight line as a position line, and take the midpoint of the position line as the actual position of the robot.

[0081] If the spatial distance is greater than the distance threshold, the third judgment position is obtained according to the satellite navigation system, and the actual position of the robot is determined by the first judgment position, the second judgment position and the third judgment position.

[0082] In detail, the first, second, and third judgment positions are connected by line segments to form a triangle, and the centroid of the triangle is taken as the fourth judgment position; the third judgment position is projected onto the position line as the fifth judgment position, and the center point of the fourth and fifth judgment positions is taken as the actual position of the robot.

[0083] Navigation module: Based on the robot's actual position and the target position, it uses dynamic programming algorithms to plan the robot's patrol route.

[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications or equivalent substitutions made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A navigation and positioning system for a patrol robot based on multimodal information fusion, characterized in that, include: Data acquisition module: Collects image data, 3D point cloud data, and acceleration data around the robot; Preprocessing module: preprocesses image data and 3D point cloud data to obtain standard image data and standard point cloud data respectively; The first positioning module processes the acceleration data to obtain the robot's initial position. Use the matrix module: Used to determine the specific values ​​of each element in the matrix; The second positioning module: obtains the robot's second judgment position based on standard image data, standard point cloud data, and using a matrix; Positioning confirmation module: used to determine the robot's actual position based on the first and second determined positions; Navigation module: Based on the robot's actual position and the target position, it uses dynamic programming algorithms to plan the robot's patrol route; The process of determining the robot's second position based on standard image data and standard point cloud data is as follows: The SI FT algorithm is used to extract feature points from standard image data, and the feature points are converted into a matrix form as the image matrix. Geometric features are extracted from standard point cloud data using a point cloud segmentation model, and these geometric features are converted into a matrix form, which is then used as a point cloud matrix. By fusing the point cloud matrix and the image matrix using a matrix, the resulting fusion matrix is ​​compared with a comparison fusion matrix in a preset database to obtain the robot's second judgment position.

2. The patrol robot navigation and positioning system based on multimodal information fusion according to claim 1, characterized in that, The image data is an image of the robot's surrounding environment taken using a camera mounted on the robot. The three-dimensional point cloud data is the three-dimensional coordinate data of spatial points around the robot obtained using LiDAR; The acceleration data is a curve showing the change of the robot's acceleration along the three coordinate axes in a spatial rectangular coordinate system over time.

3. The patrol robot navigation and positioning system based on multimodal information fusion according to claim 2, characterized in that, The process of preprocessing image data and 3D point cloud data to obtain standard image data and standard point cloud data is as follows: Gaussian filtering is used to remove noise from the image data, and histogram equalization is used to enhance the contrast of the image data, thereby obtaining standard image data. The isolated points in the 3D point cloud data are removed by using a filter, and the 3D point cloud data is downsampled by voxel filtering to obtain standard point cloud data.

4. The patrol robot navigation and positioning system based on multimodal information fusion according to claim 3, characterized in that, The process of processing acceleration data to obtain the robot's first determined position is as follows: The acceleration data is processed to obtain velocity data, and the velocity data is integrated to obtain the robot's three-dimensional coordinates. The first judgment position is obtained based on the robot's patrol starting point and three-dimensional coordinates.

5. A patrol robot navigation and positioning system based on multimodal information fusion according to claim 4, characterized in that, The process of determining the specific values ​​of each element in the matrix to use is as follows: Acquire training data from different locations and process the training data to obtain point cloud training matrices and image training matrices for different locations. Initialize the adjustment matrix, and use the adjustment matrix to fuse the point cloud training matrix and the image training matrix to obtain the fused training matrix corresponding to different locations; Calculate the matrix distance between the fusion training matrices corresponding to the two locations, and preset the matrix distance threshold. Based on the comparison between the matrix distance and the matrix distance threshold, adjust the element values ​​in the adjustment matrix to obtain the usable matrix.

6. A patrol robot navigation and positioning system based on multimodal information fusion according to claim 5, characterized in that, The process of determining the robot's actual position based on the first and second determined positions is as follows: Obtain the spatial distance between the first judgment position and the second judgment position, and preset a distance threshold. If the spatial distance is less than or equal to the distance threshold, connect the first judgment position and the second judgment position with a straight line as a position line, and take the midpoint of the position line as the actual position of the robot. If the spatial distance is greater than the distance threshold, the third judgment position is obtained according to the satellite navigation system, and the actual position of the robot is determined by the first judgment position, the second judgment position and the third judgment position.

Images