Robot positioning method for indoor and outdoor connection based on GNSS and vision

Abstract

The present invention provides a robot positioning method for indoor and outdoor connection based on GNSS and vision, and belongs to the field of positioning and navigation of mobile robots at distribution network engineering sites. The present invention is used on a robot operating system, which parses GNSS data and determines the robot's environment based on the GNSS status. If the robot is outdoors, GNSS and visual odometer fusion positioning is used. If the robot is indoors, the visual odometer estimated by the visual sensor is used as the robot's posture. In indoor and outdoor local environments, decisions are made for robot movement through the matching results of guide label templates, and during the movement, the robot operating system is used to detect positioning labels in the form of QR codes to provide accurate positioning information for the robot. The present invention achieves decimeter-level positioning accuracy for mobile robots in indoor and outdoor local environments at distribution network engineering sites, and realizes adaptive navigation in indoor and outdoor local environments at distribution network engineering sites.

Description

Technical Field

[0001] The invention belongs to the field of mobile robot positioning and navigation at distribution network engineering sites, and in particular relates to a robot positioning method for indoor and outdoor connection based on GNSS and vision. Background Art

[0002] Existing robot navigation methods include: visual navigation and satellite navigation.

[0003] Visual sensor equipment for visual navigation is simple, low-cost, economical, and has a wide range of applications. At the same time, the development of computer hardware provides support for high-throughput data processing. However, in visual navigation, factors such as illumination changes and target motion bring noise to image processing. At the same time, feature point mismatching generates errors and error accumulation, resulting in a gradual decrease in positioning accuracy during robot movement. It may even cause positioning loss during movement due to insufficient feature points in the image. Therefore, a single visual sensor cannot complete the positioning of a mobile robot. At the same time, there is often a large gap in the positioning effect of visual navigation indoors and outdoors.

[0004] Global satellite navigation technology (GNSS) can provide the highest centimeter-level positioning accuracy in open outdoor areas. However, in cities, due to the special artificial environment, such as flat roads and tall buildings, the receiver often receives satellite signals reflected by the surrounding environment while receiving satellite signals, resulting in multipath errors. There are many ways to deal with multipath errors, but there is currently no way to completely eliminate them. In the case of severe occlusion, especially in indoor environments, satellite signals cannot even be received, and satellite positioning is completely ineffective.

[0005] At present, the combined navigation method of GNSS and vision is mainly the fusion of GNSS and visual SLAM. For the fusion of GNSS and visual SLAM, the two are often fused to different degrees through two methods: loose coupling of the individual motion estimation of the two and tight coupling of merging the states of the two for overall state estimation. In addition, there are also methods for image processing-assisted GNSS and visual odometer fusion, which are generally carried out for structured roads. The image processing of path signs such as lane lines is used to determine the approximate error, and then different methods are used to fuse GNSS and visual odometer.

[0006] As for the fusion method of GNSS and visual SLAM, whether it is loose coupling or tight coupling, it can achieve good results for the navigation and positioning of mobile robots in indoor and outdoor situations. However, the visual SLAM framework includes four parts: front-end visual odometer, back-end optimization, loop detection and map construction. It is a computationally intensive algorithm with a large CPU load during operation, so it is highly dependent on the CPU. In addition, the scene similarity of the environment has a great influence on loop detection. When the scene similarity is too high, it may cause the system to close the loop incorrectly, affecting the navigation and positioning of the robot.

[0007] In the adaptive navigation of robots, it is generally necessary to build a global or local map. On the one hand, visual SLAM cannot directly build a grid map for robot autonomous navigation; on the other hand, visual construction of a grid map requires dense stereo matching, and then dimensionality reduction of dense spatial point clouds, and finally map construction through a binary Bayesian filter, which has high requirements for GPU and CPU, and it is often difficult to meet the real-time requirements in embedded applications. Summary of the invention

[0008] In order to overcome the above-mentioned deficiencies in the prior art, the present invention provides a robot positioning method for indoor and outdoor connection based on GNSS and vision.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] The robot positioning method for indoor and outdoor connection based on GNSS and vision includes the following steps:

[0011] When the true value of the robot position cannot be obtained, the robot's posture and speed are judged through the GNSS signal and the output image of the visual sensor, and the robot is positioned according to the robot's posture and speed; otherwise, the robot's true position is obtained by identifying the surrounding positioning tags in the form of QR codes through the robot operating system, and the estimated information is overwritten with the true position value for repositioning.

[0012] Preferably, the step of determining the position and speed of the robot by using the GNSS signal and the output image of the visual sensor, and positioning the robot according to the position and speed of the robot comprises:

[0013] Determine the robot's position and speed through GNSS signals;

[0014] Perform feature matching on the output image of the visual sensor to obtain the robot's posture;

[0015] The robot operating system synchronizes the timestamps of the GNSS signal and the output signal of the visual sensor;

[0016] The robot's position and speed determined by the robot operating system are integrated with the robot's position information obtained by the visual sensor to obtain the robot's position and speed estimation information, and the robot is positioned using the estimated information.

[0017] Preferably, the step of determining the position and speed of the robot by using GNSS signals comprises:

[0018] Parse the GPGGA statement information of GNSS to obtain the GNSS status and the number of satellites in use;

[0019] When the GNSS status is 0 and the number of satellites in use is 00, the robot is judged to be indoors; when the GNSS status is 1 or 2 and the number of satellites in use is not less than 04, the robot is judged to be outdoors. When the GNSS positioning data is received, it is judged that the robot's environment has a signal, otherwise it is judged that the robot's environment has no signal;

[0020] When the robot is outdoors and the signal is good, the robot's position and speed are calculated using GNSS information, including the following steps:

[0021] Set the starting point of the robot's motion as s0, and the two consecutive points in the robot's motion at any time as s n and n+1 ,s n and n+1 The time interval between n,n+1 ,s n The longitude and latitude of n ,lats n ), s n+1 The longitude and latitude of n+1 ,lats n+1 );

[0022] s n and n+1 The distance between n,n+1 It is expressed as follows,

[0023]

[0024] Where R is the average radius of the Earth;

[0025] s n+1 Relative to s n The azimuth angle θ n It is expressed as follows,

[0026]

[0027] Set s0 as the origin of the coordinate system, and establish a GNSS coordinate system that follows the right-hand rule with the true north direction as the x-axis of the coordinate system;

[0028] The robot's position at any time is estimated by the distance and azimuth between any two consecutive points as follows:

[0029]

[0030] The robot speed at any time can be estimated by the distance, azimuth and time interval between any two consecutive points as follows:

[0031]

[0032] Among them, v x is the robot's velocity in the x-axis direction of the GNSS coordinate system, v y is the robot's movement speed in the y-axis direction of the GNSS coordinate system, and ω is the robot's angular velocity in the GNSS coordinate system.

[0033] Preferably, the step of performing feature matching on the output image of the visual sensor to obtain the robot posture comprises:

[0034] Detect and obtain multiple ORB corner points; calculate the direction of each ORB corner point; calculate the descriptor of each ORB corner point;

[0035] The ORB corner points in two adjacent frames are violently matched through the descriptor to obtain the successfully matched 2D landmark points;

[0036] Use the SGBM algorithm in opencv to obtain the disparity map of the output image;

[0037] Perform depth calculation on the pixel points with non-zero pixel values ​​in the disparity map to obtain a depth map;

[0038] Get matching 3D landmark points through depth map and 2D landmark points;

[0039] The robot pose is obtained by decomposing 3D landmark points through SVD.

[0040] Preferably,

[0041] The step of detecting and obtaining ORB corner points comprises:

[0042] Set the threshold t, assuming that the brightness of any pixel p on the output image is I p ;

[0043] Draw a circle with each pixel as the center and r as the radius;

[0044] Take r+1 pixels evenly distributed on each circle, and set the brightness of any pixel in the r+1 pixels to 1;

[0045] When any circle satisfies |II pIf the number of pixels of <t is less than r, the pixel where the center of the circle is located is not an ORB corner point; otherwise, all pixels on the circle are judged. When any consecutive pixels on the circle satisfy |I - I p | The number of consecutive pixels of <t is greater than or equal to r 2 , the pixel where the center of the circle is located is an ORB corner point, otherwise, the pixel where the center of the circle is located is not an ORB corner point;

[0046] The steps for calculating the ORB corner point direction include:

[0047] For each ORB corner point, take an image patch A with the ORB corner point as the center and a radius of R, and calculate the moment of the image patch A,

[0048]

[0049] where 0 < p < 1, 0 < q < 1; x is the abscissa of a pixel on the image patch A, y is the z ordinate of a pixel on the image patch A, and I(x, y) is the gray value of the pixel at the coordinate (x, y);

[0050] Calculate the centroid of the image patch A through the moment of the image patch A,

[0051]

[0052] Connect the geometric center and the centroid of each image patch to obtain a vector, and define the direction of this vector as the direction of each ORB corner point, denoted as,

[0053] α = tan -1 2(c y , c x ) = tan -1 2(m 01 , m 10 )

[0054] where, c y represents the ordinate of the vector, and c x represents the abscissa of the vector;

[0055] The steps for calculating the descriptor of each ORB corner point include:

[0056] Set a neighborhood window with each ORB corner point as the center;

[0057] Randomly select 128 pairs of pixels in each neighborhood window, and calculate the feature vector corresponding to each pair of pixels through the following formula,

[0058]

[0059] where, p(x) < p(y) are the pixel values of the two pixels in a pair of pixels respectively;

[0060] The feature vector corresponding to the 128 pairs of pixels in the area window where each ORB corner point is located is the descriptor of the ORB corner point.

[0061] Preferably, the step of brute-force matching of ORB corner points in two adjacent image frames by using descriptors comprises:

[0062] In every two adjacent frames, calculate the Hamming distance between each ORB corner point in one image and all ORB corner points in the other image;

[0063] Determine the point with the shortest Hamming distance in two adjacent frames as the matching point in the two frames;

[0064] Eliminate all matching points whose Hamming distance is greater than twice the minimum Hamming distance;

[0065] Use the SGBM algorithm in opencv to obtain the disparity map of the output image.

[0066] Preferably, the depth of the pixel points with non-zero pixel values ​​in the disparity map is calculated by the following formula:

[0067]

[0068] Among them, Z is the pixel depth, d is the pixel disparity, b is the binocular baseline length, and f is the camera focal length.

[0069] Preferably, the step of obtaining the robot posture by decomposing 3D landmark points through SVD includes:

[0070] Get two sets of 3D landmark points that match at any two moments through the depth map and 2D landmark points.

[0071] P={p1,…,p n}

[0072] P′={p1′,…,p n ′}

[0073] The coordinate transformation between two sets of 3D landmark points is expressed as follows:

[0074]

[0075] Among them, R and t are the rotation matrix and translation vector of the camera at any two moments respectively;

[0076] Pair two sets of 3D landmark points, each pair of landmark points p i and p i The error between ′ is expressed as,

[0077] e i =pi -(Rp i ′+t)

[0078] Through each pair of 3D landmark points p i and p i The error of ′ is used to construct the least squares problem.

[0079]

[0080] The SVD algorithm is used to solve the least squares problem to obtain the optimal rotation matrix and translation vector, where the SVD algorithm is expressed as follows:

[0081]

[0082] The optimal rotation matrix is ​​expressed as follows:

[0083] R=UV T

[0084] The optimal translation vector is expressed as follows,

[0085] t=p-RP′

[0086] in,

[0087] q i =p i -p

[0088]

[0089] q i ′=p i ′-p′

[0090]

[0091] The robot pose is estimated by the rotation matrix and translation vector of the camera at each moment.

[0092] Preferably, the method further includes the following steps: guiding the movement of the robot using a guide label template of the robot according to the positioning information.

[0093] Preferably, the step of using the robot's guide label template to guide the robot's movement includes:

[0094] Obtain the ROI image of the output image;

[0095] Create image pyramids of ROI image and template image;

[0096] Using the image pyramid to obtain the similarity measure between the output image and the template image; judging the most matching template image according to the similarity measure;

[0097] Guiding the robot to move according to the guide label of the template image; wherein the guide label information includes: translation, rotation, stop and U-turn;

[0098] The steps of creating the image pyramid of the ROI image and the template image are as follows:

[0099] Create a 4-layer pyramid image corresponding to the ROI image and the template image;

[0100] Downsample the 4-layer pyramid image;

[0101] Take a 5×5 convolution kernel K, traverse each pixel of the 4-layer pyramid image after downsampling, and perform mean smoothing filtering, where the convolution kernel K is expressed as follows:

[0102]

[0103] The steps of obtaining a similarity measure between the output image and the template image by using the image pyramid and determining the most matching template image according to the similarity measure include:

[0104] Set thresholds;

[0105] The similarity measurement criterion of normalized correlation coefficient is used to calculate the similarity value between each template image and ROI image.

[0106]

[0107] Where I1 is the template image, I2 is the ROI image, u1 is the mean of image I1, u2 is the mean of image I2, and x is the value of a pixel in an image;

[0108] The template image with the highest similarity of the normalized correlation coefficient that is greater than the threshold is taken as the best matching template image.

[0109] The robot positioning method for indoor and outdoor connection based on GNSS and vision provided by the present invention has the following beneficial effects: the present invention realizes accurate repositioning through positioning tags, eliminates the phenomenon that the sensor estimation error increases over time, realizes high-precision and rapid positioning in different indoor and outdoor scenes, and provides a reliable foundation for the robot navigation system. In addition, the navigation system can be adaptively switched when turning from indoor to outdoor or from outdoor to indoor environment, and autonomous navigation can be realized by using guide tags, so as to plan a safe path for the robot in local safe areas indoors and outdoors, and realize local navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0111] Figure 1 This is a flow chart of a robot positioning method for indoor and outdoor connection based on GNSS and vision according to Example 1 of the present invention;

[0112] Figure 2 This is a flow chart of performing feature matching on the output image of the visual sensor to obtain the robot posture according to Embodiment 1 of the present invention;

[0113] Figure 3 This is a flow chart of using the robot's guide label template to guide the robot's movement according to Example 1 of the present invention. DETAILED DESCRIPTION

[0114] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0115] Example 1

[0116] The present invention applies the ROS framework and, based on the difference between indoor and outdoor GNSS signals, determines whether the robot is indoors or outdoors by detecting the number of satellite receptions. On this basis, the function of indoor and outdoor adaptive local repositioning is achieved through a loose coupling method of GNSS and vision.

[0117] See also Figure 1 The present invention provides a robot positioning method for indoor and outdoor connection based on GNSS and vision, including the following steps: when the true value of the robot position cannot be obtained, the robot's posture and speed are judged by the GNSS signal and the output image of the visual sensor, and the robot is positioned according to the robot's posture and speed; otherwise, the robot's true position is obtained by identifying the surrounding positioning tags in the form of QR codes through the robot operating system, and the estimated information is covered by the true position value to perform repositioning.

[0118] Specifically, the robot's position and speed are determined by GNSS signals and the output image of the visual sensor, and the steps of positioning the robot according to the robot's position and speed include: determining the robot's position and speed through GNSS signals; performing feature matching on the output image of the visual sensor to obtain the robot's position and speed; performing timestamp synchronization on the GNSS signal and the output signal of the visual sensor through the robot operating system; obtaining the robot's position and speed estimation information by fusing the robot's position and speed determined by the GNSS signal with the robot's position information obtained through the visual sensor through the robot operating system, and using the estimated information to position the robot; and guiding the robot's movement using the robot's guide label template according to the positioning information.

[0119] Specifically, the steps of judging the robot's position and speed by signals include: parsing the GPGGA statement information of GNSS, obtaining the GNSS status and the number of satellites used; when the GNSS status is 0 and the number of satellites used is 00, the robot is judged to be indoors; when the GNSS status is 1 or 2 and the number of satellites used is not less than 04, the robot is judged to be outdoors, and when the GNSS positioning data is received, it is judged that the robot's environment has a signal, otherwise it is judged that the robot's environment has no signal. When the robot is outdoors and the signal is good, the robot's position and speed are calculated using GNSS information.

[0120] Using GNSS information to calculate the robot's position and speed includes the following steps: setting the starting point of the robot's motion as s0, and setting the two consecutive points in the robot's motion at any time as s n and n+1 ,s n and n+1 The time interval between n,n+1 ,s n The longitude and latitude of n ,lats n ), s n+1 The longitude and latitude of n+1 ,lats n+1 ).

[0121] s n and n+1 The distance between n,n+1 It is expressed as follows,

[0122]

[0123] Where R is the average radius of the Earth.

[0124] s n+1 Relative to s n The azimuth angle θ n It is expressed as follows,

[0125]

[0126] Set s0 as the origin of the coordinate system, and establish a GNSS coordinate system that follows the right - hand rule with the due - north direction as the x - axis of the coordinate system; estimate the pose of the robot at any time through the distance and azimuth angle between any two consecutive points as follows,

[0127]

[0128] Estimate the speed of the robot at any time through the distance, azimuth angle and time interval between any two consecutive points as follows:

[0129]

[0130] where, v x is the moving speed of the robot in the x - axis direction of the GNSS coordinate system, v y is the moving speed of the robot in the y - axis direction of the GNSS coordinate system, and ω is the angular velocity of the robot's movement in the GNSS coordinate system.

[0131] Refer to Figure 2 , the steps of obtaining the robot pose through feature matching of the output image of the vision sensor include: detecting multiple ORB corner points; calculating the direction of each ORB corner point; calculating the descriptor of each ORB corner point; performing brute - force matching on the ORB corner points in two adjacent frames of images through the descriptor to obtain successfully - matched 2D landmark points; using the SGBM algorithm in opencv to obtain the disparity map of the output image; performing depth calculation on the pixel points with non - zero pixel values in the disparity map to obtain the depth map; obtaining the matched 3D landmark points through the depth map and 2D landmark points; obtaining the robot pose through SVD decomposition of the 3D landmark points.

[0132] Specifically, the steps of detecting ORB corner points include: setting a threshold t, assuming that the brightness of any pixel point p on the output image is I p ; drawing a circle with each pixel point as the center and r as the radius; taking r + 1 pixels evenly distributed on each circle, and assuming that the brightness of any one of the r + 1 pixels is I; when the number of pixel points satisfying |I - I p | < t on any circle is less than r, the pixel point where the center of the circle is located is not an ORB corner point; otherwise, judge all the pixel points on the circle. When the number of consecutive pixel points satisfying |I - I p | < t on any circle is greater than or equal to r 2 , the pixel point where the center of the circle is located is an ORB corner point, otherwise, the pixel point where the center of the circle is located is not an ORB corner point.

[0133] Specifically, the steps for calculating the ORB corner direction include: for each ORB corner, take an image patch A with the ORB corner as the center and a radius of R, and calculate the moments of the image patch A.

[0134]

[0135] where 0 < p < 1, 0 < q < 1; x is the abscissa of a pixel point on the image patch A, y is the z ordinate of a pixel point on the image patch A, and I(x, y) is the gray value of the pixel point at the coordinate (x, y).

[0136] Calculate the centroid of the image patch A through the moments of the image patch A.

[0137]

[0138] Connect the geometric center and the centroid of each image patch to obtain a vector, and define the direction of this vector as the direction of each ORB corner, denoted as

[0139] α = tan -1 2(c y , c x ) = tan -1 2(m 01 , m 10 )

[0140] where c y represents the ordinate of the vector, and c x represents the abscissa of the vector.

[0141] Specifically, the steps for calculating the descriptor of each ORB corner include: setting a neighborhood window centered on each ORB corner; randomly taking 128 pairs of pixel points in each neighborhood window, and calculating the feature vector corresponding to each pair of pixel points through the following formula

[0142]

[0143] where p(x) < p(y) are the pixel values of the two pixel points in a pair of pixel points respectively; the feature vectors corresponding to 128 pairs of pixel points in the neighborhood window where each ORB corner is located are the descriptors of the ORB corner.

[0144] Specifically, the steps for brute - force matching of ORB corners in two adjacent frames of images through descriptors include: in each pair of adjacent frames of images, calculate the Hamming distance between each ORB corner in one image and all ORB corners in the other image; determine that the point with the closest Hamming distance in the two adjacent frames of images is the matching point between the two frames of images; eliminate the matching points with a Hamming distance greater than twice the minimum Hamming distance among all matching points; use the SGBM algorithm in opencv to obtain the disparity map of the output image.

[0145] Specifically, the depth of the pixels with non-zero pixel values ​​in the disparity map is calculated using the following formula:

[0146]

[0147] Among them, Z is the pixel depth, d is the pixel disparity, b is the binocular baseline length, and f is the camera focal length.

[0148] In this embodiment, the step of obtaining the robot posture by decomposing 3D landmark points by SVD includes: obtaining two groups of landmark points at any two moments by using the depth map,

[0149] P={p1,…,p n}

[0150] P′={p1′,…,p n ′}

[0151] The coordinate transformation between two sets of landmark points is expressed as follows:

[0152]

[0153] Among them, R and t are the rotation matrix and translation vector of the camera at any two moments respectively.

[0154] Pair two sets of landmark points, each pair of landmark points p i and p i The error between ′ is expressed as,

[0155] e i =p i -(Rp i ′+t)

[0156] Through each pair of landmark points p i and p i The error of ′ is used to construct the least squares problem.

[0157]

[0158] The SVD algorithm is used to solve the least squares problem to obtain the optimal rotation matrix and translation vector, where the SVD algorithm is expressed as follows:

[0159]

[0160] The optimal rotation matrix is ​​expressed as follows:

[0161] R=UV T

[0162] The optimal translation vector is expressed as follows,

[0163] t=p-RP′

[0164] in,

[0165] q i =p i -p

[0166]

[0167] q i ′=p i ′-p′

[0168]

[0169] The robot pose is estimated by the rotation matrix and translation vector of the camera at each moment.

[0170] See also Figure 3 The steps of using the robot's guide label template to guide the robot's movement include: obtaining an ROI image of the output image; creating an image pyramid of the ROI image and the template image; using the image pyramid to obtain a similarity measure between the output image and the template image; judging the most matching template image according to the similarity measure; guiding the robot's movement according to the guide label of the template image; wherein the guide label information includes: translation, rotation, stop and U-turn.

[0171] Furthermore, the steps of creating an image pyramid of the ROI image and the template image are as follows: create a 4-layer pyramid image corresponding to the ROI image and the template image; downsample the 4-layer pyramid image; take a 5×5 convolution kernel K, traverse each pixel of the 4-layer pyramid image after downsampling, and perform mean smoothing filtering, where the convolution kernel K is expressed as follows,

[0172]

[0173] Specifically, the similarity measurement between the output image and the template image is obtained by using the image pyramid; the step of determining the most matching template image according to the similarity measurement includes: setting a threshold; calculating the similarity value between each template image and the ROI image by using the similarity measurement criterion of the normalized correlation coefficient,

[0174]

[0175] Among them, I1 is the template image, I2 is the ROI image, u1 is the mean of image I1, u2 is the mean of image I2, and x is the value of a pixel in an image; the template image with the highest similarity of the normalized correlation coefficient and greater than the threshold is taken as the most matching template image.

[0176] The robot is mainly composed of a motion chassis and a motion control system, and realizes autonomous positioning and navigation with the assistance of laser, vision, IMU, and GNSS. For example: the indoor scene is determined by judging satellite signals, and autonomous positioning and navigation is realized with the help of other sensors. When turning from indoors to outdoors, the robot receives satellite signals and automatically switches to outdoor navigation using the robot positioning method based on GNSS and vision for indoor and outdoor connection of the present invention, and realizes autonomous navigation under the guidance of the tag. The positioning is accurate in different scenes and the navigation accuracy is significantly improved.

[0177] The above embodiments are only preferred specific implementation methods of the present invention, and the protection scope of the present invention is not limited thereto. Any simple changes or equivalent replacements of the technical solutions that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belong to the protection scope of the present invention.

Claims

1. A robot positioning method for indoor and outdoor connection based on GNSS and vision, characterized in that: The following steps are involved: When the true value of the robot position cannot be obtained, the robot's position and speed are determined through the GNSS signal and the output image of the visual sensor, and the robot is positioned according to the robot's position and speed; Otherwise, the robot operating system identifies the surrounding positioning tags in the form of QR codes to obtain the true value of the robot position, and uses the true value of the position to overwrite the estimated information for repositioning; The step of determining the position and speed of the robot by using the GNSS signal and the output image of the visual sensor, and positioning the robot according to the position and speed of the robot comprises: Determine the robot's position and speed through GNSS signals; Perform feature matching on the output image of the visual sensor to obtain the robot's posture; The robot operating system synchronizes the timestamps of the GNSS signal and the output signal of the visual sensor; The robot's position and speed determined by the robot operating system are integrated with the GNSS signal and the robot's position information obtained by the visual sensor to obtain the robot's position and speed estimation information, and the robot is positioned using the estimated information; According to the positioning information, the robot's guide label template is used to guide the robot's movement, including: Obtain the ROI image of the output image; Create image pyramids of ROI image and template image; Using the image pyramid to obtain the similarity measure between the output image and the template image; judging the most matching template image according to the similarity measure; The robot is guided to move according to the guide label of the template image; wherein the guide label information includes: translation, rotation, stop and U-turn.

2. The robot positioning method for indoor and outdoor connection based on GNSS and vision according to claim 1, characterized in that: The step of determining the position and speed of the robot by using GNSS signals includes: Parse the GPGGA statement information of GNSS to obtain the GNSS status and the number of satellites in use; When the GNSS status is 0 and the number of satellites in use is 00, the robot is judged to be indoors; when the GNSS status is 1 or 2 and the number of satellites in use is not less than 04, the robot is judged to be outdoors. When the GNSS positioning data is received, it is judged that the robot's environment has a signal, otherwise it is judged that the robot's environment has no signal; When the robot is outdoors and the signal is good, the robot's position and speed are calculated using GNSS information, including the following steps: Set the starting point of the robot's motion as s0, and the two consecutive points in the robot's motion at any time as s n and S n+1 ,s n and n+1 The time interval between n，n+1 ,s n The longitude and latitude of n , lats n ), s n+1 The longitude and latitude of n+1 , lats n+1 ); s n and n+1 The distance between n，n+1 It is expressed as follows, Where R is the average radius of the Earth; s n+1 Relative to s n The azimuth angle θ n It is expressed as follows, Set s0 as the origin of the coordinate system, and establish a GNSS coordinate system that follows the right-hand rule with the true north direction as the x-axis of the coordinate system; The robot's position at any time is estimated by the distance and azimuth between any two consecutive points as follows: The robot speed at any time can be estimated by the distance, azimuth and time interval between any two consecutive points as follows: Among them, v x is the robot's velocity in the x-axis direction of the GNSS coordinate system, v y is the movement speed of the robot in the y-axis direction of the GNSS coordinate system, and ω is the angular velocity of the robot in the GNSS coordinate system.

3. The robot positioning method for indoor and outdoor connection based on GNSS and vision according to claim 1, characterized in that: The step of performing feature matching on the output image of the visual sensor to obtain the robot posture comprises: Detect and obtain multiple ORB corner points; calculate the direction of each ORB corner point; calculate the descriptor of each ORB corner point; The ORB corner points in two adjacent frames are violently matched through the descriptor to obtain the successfully matched 2D landmark points; Use the SGBM algorithm in opencv to obtain the disparity map of the output image; Perform depth calculation on the pixel points with non-zero pixel values ​​in the disparity map to obtain a depth map; Get matching 3D landmark points through depth map and 2D landmark points; Obtain the robot pose by SVD decomposition of 3D road markers.

4. The robot positioning method for indoor and outdoor connection based on GNSS and vision according to claim 3, characterized in that: The steps of the detection to obtain ORB corner points include: Set the threshold t, assuming that the brightness of any pixel p on the output image is I p ; Draw a circle with each pixel point as the center and r as the radius; Take r + 1 pixels evenly distributed on each circle, and set the brightness of any one of the r + 1 pixels as I; When any circle satisfies |II p |<t The number of pixels is less than r, then the pixel where the center of the circle is located is not an ORB corner point; otherwise, all pixels on the circle are judged, and when any pixel on the circle satisfies |II p |<tThe number of consecutive pixels is greater than or equal to r 2 , then the pixel point where the center of the circle is located is an ORB corner point, otherwise, the pixel point where the center of the circle is located is not an ORB corner point; The steps of calculating the ORB corner point direction include: For each ORB corner point, take an image patch A with the ORB corner point as the center and a radius of R, and calculate the moment of the image patch A, where 0 < p < 1, 0 < q < 1; x is the abscissa of a pixel point on the image patch A, y is the z ordinate of a pixel point on the image patch A, and I(x, y) is the gray value of the pixel point at the coordinate (x, y); Calculate the centroid of the image patch A through the moment of the image patch A, Connect the geometric center and the centroid of each image patch to obtain a vector, and define the direction of this vector as the direction of each ORB corner point, denoted as, α=tan -1 2(c y ,c x )=tan -1 2(m 01 ,m 10 ) Among them, c y represents the ordinate of the vector, c x represents the horizontal coordinate of the vector; The steps of calculating the descriptor of each ORB corner point include: Set a neighborhood window with each ORB corner point as the center; Arbitrarily take 128 pairs of pixel points in each neighborhood window, and calculate the feature vector corresponding to each pair of pixel points through the following formula, where p(x) < p(y) are the pixel values of the two pixel points in a pair of pixel points respectively; The feature vectors corresponding to 128 pairs of pixel points in the neighborhood window where each ORB corner point is located are the descriptors of this ORB corner point.

5. The robot positioning method for indoor and outdoor connection based on GNSS and vision according to claim 3, characterized in that: The steps of performing brute-force matching on the ORB corner points in two adjacent frames of images through descriptors include: In each two adjacent frames of images, calculate the Hamming distance between each ORB corner point in one image and all ORB corner points in the other image; Determine that the points with the closest Hamming distance in two adjacent frames of images are the matching points in the two frames of images; Eliminate the matching points with a Hamming distance greater than twice the minimum Hamming distance among all matching points; Use the SGBM algorithm in opencv to obtain the disparity map of the output image.

6. The robot positioning method for indoor and outdoor connection based on GNSS and vision according to claim 3, characterized in that: Perform depth calculation on the pixel points with non-zero pixel values in the disparity map through the following formula: where Z is the pixel point depth, d is the pixel point disparity, b is the binocular baseline length, and f is the camera focal length.

7. The robot positioning method for indoor and outdoor connection based on GNSS and vision according to claim 4, characterized in that: The steps of obtaining the robot pose by SVD decomposition of 3D road markers include: Obtain two sets of matching 3D road markers at any two moments through the depth map and 2D road markers, P={p1,…,p n } P′={p1′,…,p n ′} The coordinate transformation existing between the two sets of 3D road markers is expressed as follows, where R and t are the rotation matrix and translation vector of the camera at any two moments respectively; Pair two sets of 3D landmark points, each pair of landmark points p i and p i The error between ′ is expressed as, e i =p i -(Rp i ′+t) Through each pair of 3D landmark points p i and p i The error of ′ is used to construct the least squares problem. Use the SVD algorithm to solve the least squares problem to obtain the optimal rotation matrix and translation vector, where the SVD algorithm is expressed as follows, The optimal rotation matrix is expressed as follows, R=UV T The optimal translation vector is expressed as follows, t = p - RP’ where, q i =p i -p q i ’=p i ’-p’ Perform robot pose estimation through the rotation matrix and translation vector of the camera at each moment.

8. The robot positioning method for indoor and outdoor connection based on GNSS and vision according to claim 1, characterized in that: The steps of creating the image pyramids of the ROI image and the template image are as follows: Create 4-layer pyramid images corresponding to the ROI image and the template image; Downsample the 4-layer pyramid images; Take a 5×5 convolution kernel K, traverse each pixel of the 4-layer pyramid images after downsampling, and perform mean smoothing filtering, where the convolution kernel K is expressed as follows, The method comprises obtaining a similarity measure between the output image and the template image by using the image pyramid; The steps of determining the best matching template image based on similarity measurement include: Set thresholds; The similarity measurement criterion of normalized correlation coefficient is used to calculate the similarity value between each template image and ROI image. Where I1 is the template image, I2 is the ROI image, u1 is the mean of image I1, u2 is the mean of image I2, and x is the value of a pixel in an image; The template image with the highest similarity of the normalized correlation coefficient that is greater than the threshold is taken as the best matching template image.

Images