Global positioning method and device
By using image representation and descriptor matching technology in large hospital scenarios, combining lidar and camera sensors to build a frame-pose mapping relationship, the problem of difficulty in global positioning of mobile robots is solved, and the rapid and reliable in complex environments is achieved position.
Patent Information
- Application Number
- CN202411210412.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-30
AI Technical Summary
In hospital large scenarios, global positioning of mobile robots is difficult, especially in the absence of prior knowledge of initial poses, especially when only a limited number of lidar scans are available, especially in simple and repetitive with local geometric features In the environment, it is difficult to achieve effective global positioning.
In the offline SLAM stage, the lidar data is represented by the network method based on the network method, the visual information map of the keyframe and the Gist descriptor is obtained, the frame-pose mapping relationship is constructed, and the local two-dimensional-three-dimensional matching is performed through the ORB descriptor. , calculate the pose transformation matrix between the query image and the candidate keyframe, and use the Scan2Map score for positioning evaluation. If the relocation mechanism is triggered, the AMCL method is used for pose tracking.
By reducing the pose search space, using local descriptors to improve positioning accuracy, and combining with the relocation triggering mechanism, the problem of kidnapped robots in global positioning is effectively solved, and the fast and reliable global positioning of mobile robots in large hospital scenarios is achieved.
Smart Images

Figure CN119024351A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a large-scale indoor patient transport mobile robot AMR in a hospital, and in particular to a global positioning method and device for an indoor mobile robot, belonging to the technical field of robot positioning. Background Art
[0002] The problem of mobile robot localization refers to determining the posture of a robot relative to its surroundings. It is considered an important function for mobile robots to complete autonomous tasks. Generally speaking, localization can be divided into: position tracking and global localization.
[0003] Position tracking assumes that the initial pose of the robot is known, and the pose of the robot can be found by correcting the pose error and tracking the robot over time in a known environment. The global localization problem is more difficult than position tracking because it does not have an initial seed position to start the search for the pose of a given map.
[0004] The kidnapped robot problem refers to a well-localized robot that is kidnapped and teleported to another location, while the robot may believe its own location, which is also a global positioning. Image-based localization methods solve this problem by matching a query image with all image databases, and then estimating the pose of the image, which can be applied to global localization. However, feature matching relies on the invariance of feature viewpoints. In addition, when there are thousands of 3D points and their associated descriptors, it becomes challenging to search for accurate correspondences in the space. It is difficult for a robot to recognize a place that was previously only visible through a single lidar scan. This situation is more obvious in geometric structures. Although the accuracy of lidar sensors is higher than that of cameras, lidar reduces the dimensionality of perceived environmental information. In general, due to the rich visual information, visual methods have better performance in data association and loop closure detection than pure lidar methods.
[0005] Global localization is the problem of estimating the robot's pose without any prior knowledge of its initial pose. Global localization using LiDAR is difficult to achieve when only a limited number of LiDAR scans are available, especially in simple and repetitive environments with local geometric features. Cameras have the advantage of being rich in information, but lack reliable and efficient localization algorithms, as well as a relatively high computational burden. Therefore, the present application proposes a method and apparatus for achieving global localization of a mobile robot to at least partially solve the problems that may exist in the prior art. Summary of the invention
[0006] In order to overcome the above problems or at least partially solve the above problems, an embodiment of the present invention provides a method and device for realizing global positioning of a mobile robot in an indoor scene.
[0007] The embodiment of the present invention is achieved as follows:
[0008] The present application provides a method for realizing global positioning of a mobile robot, the method comprising:
[0009] Image representation step: In the offline SLAM stage, the lidar data is used to represent environmental parameters based on the network method, and a visual information map containing a series of key frames and their Gist descriptors is obtained. The top n key frames in the Gist distance between the key frames to be queried and clustered are determined, and a frame-pose mapping relationship is constructed;
[0010] Keyframe-based positioning step: Apply ORB descriptor for local 2D-3D matching, calculate the pose transformation matrix between the query image and the candidate keyframe, and obtain the pose information of the estimated query image;
[0011] Re-localization step: Based on the posture information, the Scan2Map score is used to evaluate the localization to further determine whether to re-localize in the relocalization trigger mechanism;
[0012] Posture tracking step: If the relocalization mechanism is triggered or not, the robot uses the AMCL method to perform posture tracking according to a preset global posture.
[0013] In some embodiments of the present invention, the image representation step comprises:
[0014] Based on the network method, the lidar data is used to represent the environmental parameters, and a visual information map containing a series of key frames and their Gist descriptors is obtained. The top n key frames in the Gist distance between the key frames to be queried and clustered are determined, and a frame-pose mapping relationship is constructed.
[0015] In the offline SLAM stage, the environment is discretized into two-dimensional square cells, each of which is associated with a value between [0, 1], where the value is the probability of the cell being occupied;
[0016] Obtain a series of keyframes through a preset RGB-D camera, and represent at least one set of keyframes as a scene on an occupied grid map to determine candidate poses that can be directly retrieved for tracking initialization;
[0017] Collect visual keyframes including RGB images and depth images of the odometry interval. When the actual motion offset is less than the set odometry interval, record n visual keyframes starting from the time interval.
[0018] Construct a one-to-one mapping relationship between each visual keyframe and posture.
[0019] In some embodiments of the present invention, the keyframe-based positioning step includes:
[0020] Apply the Gist global scene descriptor to quickly retrieve and track candidate poses during initialization and determine the KF of each keyframe i According to the preset rules, it is represented as a Gist vector g i The average value of
[0021] According to each key frame KF i and the corresponding relationship between the Gist vector, comparing the distance between the current image and the key frame in the preset database, and determining the key frame with the closest distance to the maximum value as the candidate frame, and estimating the position of the robot in the environment through the posture information of the map bound to the candidate frame;
[0022] The local 3D points of the camera in the global frame are calculated through the RGB image and the depth image, and the position of the given point in the local and global coordinates is determined, which is used to calculate the transformation matrix between the query image and the proposed key frame to obtain more accurate pose information of the estimated query image.
[0023] In some embodiments of the present invention, the Gist global scene descriptor is used to quickly retrieve and track candidate poses during the initialization process to determine each key frame KF i According to the preset rules, it is represented as a Gist vector g i The average values include:
[0024] Apply the Gist global scene descriptor to quickly retrieve candidate poses during tracking initialization;
[0025] Based on the context algorithm, global objects are identified and the keyframe image in each keyframe is divided into 4×4 blocks, where each block is represented by the average value of the filter in the [8,8,4] direction at each scale in the three color channels;
[0026] A Gist vector of length 960 is calculated for each keyframe.
[0027] In some embodiments of the present invention, the i The corresponding relationship between the current image and the Gist vector is compared, and the key frame with the closest distance to the maximum value is determined as the candidate frame. The position of the robot in the environment is estimated by the posture information of the map bound to the candidate frame, which specifically includes:
[0028] According to each key frame KF i The corresponding relationship with the Gist vector is used to compare the current image with the key frames in the preset database to measure the difference between the two feature vectors;
[0029] Through the Gist feature vector g r and corresponding to their respective images g s The distance between rs , the Minkowski distance of a metric on a Euclidean space with order p is defined as: Calculate the Gist vector distance L2 of length k = 960 between each key frame in the database and the query image;
[0030] If the distance d rs If the key frame is closest to the maximum value N, then the key frame is considered as a candidate frame, and the position of the robot in the environment is roughly estimated by the pose information of the map bound to the candidate frame.
[0031] In some embodiments of the present invention, the re-positioning step specifically includes:
[0032] The visual and lidar information are integrated to refine the robot's pose in a maximum likelihood framework, where the robot's pose at time t is determined as:
[0033] Among them, the probability represents the probability of predicting the robot's posture based on the image-based localization model, Modeling the impact of keyframe clustering on continuous pose evaluation, m l is the laser radar plane map, is a lidar scan containing n individual measurements (n corresponding lidar beams);
[0034] After global coordinate reference optimization, the range reading is converted into a lidar map based on the robot posture, where the lidar endpoints The grid cell hit is expressed in m hit(k) express;
[0035] When cell m hit(k) is occupied, the occupancy value of the cell is added to a voting score, where the likelihood of the sensor measurement is given by the final voting score:
[0036] express;
[0037] The keyframe clusters are updated based on the scan Scan2Map scores. If the new cluster has the highest score after Scan2Map reordering, it is confirmed that the reordering has occurred, and the trigger mechanism makes the corresponding position of the candidate frame a candidate pose estimate.
[0038] In some embodiments of the present invention, the step of integrating visual and lidar information to refine the robot's posture in a maximum likelihood framework may also include:
[0039] Depending on whether there are new keyframe clusters after k-means clustering, decide whether to enter the next session of the relocalization trigger mechanism or continue tracking the robot's posture for a period of time;
[0040] After calculating the distance between images in the process of calculating the similarity metric, the top candidate indexes of n key frames are obtained;
[0041] The indexes of the n candidate key frames are clustered using the k-means clustering method, and the center of the cluster is replaced by the index of the best matching candidate frame.
[0042] A device for realizing global positioning of a mobile robot, the device comprising:
[0043] The image representation module is used to represent the environmental parameters of the lidar data based on the network method in the offline SLAM stage, obtain a visual information map containing a series of key frames and their Gist descriptors, determine the top n key frames in the Gist distance between the key frames to be queried and clustered, and construct a frame-pose mapping relationship;
[0044] The first positioning module is used to apply the ORB descriptor to perform local 2D-3D matching, calculate the pose transformation matrix between the query image and the candidate key frame, and obtain the pose information of the estimated query image;
[0045] The second positioning module is used to perform positioning evaluation based on the posture information using Scan2Map score to further determine whether to perform positioning again in the re-localization trigger mechanism;
[0046] The posture tracking module is used to perform a posture tracking process using the AMCL method based on a preset global posture if the repositioning mechanism is triggered or not.
[0047] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned global positioning method of a mobile robot when executed by the processor.
[0048] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the global positioning method of a mobile robot is implemented.
[0049] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:
[0050] In the offline SLAM stage, the laser radar data is used to represent environmental parameters based on the network method, and a visual information map containing a series of key frames and their Gist descriptors is obtained. The top n key frames in the Gist distance between the key frames to be queried and clustered are determined, and a frame-pose mapping relationship is constructed; the positioning step based on the key frame: the ORB descriptor is used for local two-dimensional-three-dimensional matching, the pose transformation matrix between the query image and the candidate key frames is calculated, and the pose information of the estimated query image is obtained; based on the pose information, the Scan2Map score is used for positioning evaluation to further determine whether to re-position in the re-positioning trigger mechanism; if the re-positioning mechanism is triggered or not, the robot uses the AMCL method to track the pose according to a preset global pose. In view of the above problem of AMR being kidnapped in global positioning, the global descriptor is used to reduce the pose search space, the local descriptor is used to further improve the positioning accuracy, and the global positioning method of indoor mobile robots is effectively solved in combination with the re-positioning trigger mechanism. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0052] Figure 1 A flowchart of a global positioning method provided by an embodiment of the present invention;
[0053] Figure 2 A schematic diagram of a technical framework structure of a global positioning method provided by an embodiment of the present invention;
[0054] Figure 3 A flow chart of a relocation trigger mechanism of a global positioning method provided by an embodiment of the present invention;
[0055] Figure 4 A Gist distance between a sample query image randomly selected from a scene and a key frame of each data set in a global positioning method provided by an embodiment of the present invention;
[0056] Figure 5 A schematic diagram of the module structure of a global positioning device provided by one embodiment of the present invention;
[0057] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0059] Example 1
[0060] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0061] Please refer to Figure 1 , an embodiment of the present invention provides a flowchart of a method for realizing global positioning of a mobile robot, the method comprising:
[0062] S100 image representation step: In the offline SLAM stage, the lidar data is used to represent environmental parameters based on the network method, and a visual information map containing a series of key frames and their Gist descriptors is obtained. The top n key frames in the Gist distance between the key frames to be queried and clustered are determined, and a frame-pose mapping relationship is constructed;
[0063] S200 is a keyframe-based positioning step: applying the ORB descriptor to perform local 2D-3D matching, calculating the pose transformation matrix between the query image and the candidate keyframe, and obtaining the pose information of the estimated query image;
[0064] S300 performs a positioning step again: based on the posture information, a positioning evaluation is performed using a Scan2Map score to further determine whether to perform positioning again in a re-positioning trigger mechanism;
[0065] S400 posture tracking step: If the repositioning mechanism is triggered or not triggered, the robot uses the AMCL method to perform a posture tracking process according to a preset global posture.
[0066] The present invention proposes a fast and reliable global positioning method in which a laser radar and a camera sensor are integrated. In the SLAM process, a laser radar is used to introduce visual keyframes in a series of environmental maps. A global descriptor is used to reduce the posture search space, a local descriptor is used to further improve the positioning accuracy, and a re-localization trigger mechanism is used to effectively solve the kidnapped robot problem. The present invention extends the keyframe clustering to a two-dimensional or three-dimensional posture space instead of clustering the sequence numbers of candidate keyframes. In addition, the possibility of improving the re-localization performance is to consider automatically determining the number of clusters based on the clustering data characteristics, and the reliability of the algorithm will also be tested in a complex and large scene in a hospital.
[0067] Example 2
[0068] A global positioning method is used to achieve global positioning of a mobile robot in an indoor scene, wherein the “image representation step” as described in step S100 above includes:
[0069] Based on the network method, the lidar data is used to represent the environmental parameters, and a visual information map containing a series of key frames and their Gist descriptors is obtained. The top n key frames in the Gist distance between the key frames to be queried and clustered are determined, and a frame-pose mapping relationship is constructed.
[0070] In the offline SLAM stage, the environment is discretized into two-dimensional square cells, each of which is associated with a value between [0, 1], where the value is the probability of the cell being occupied;
[0071] Obtain a series of keyframes through a preset RGB-D camera, and represent at least one set of keyframes as a scene on an occupied grid map to determine candidate poses that can be directly retrieved for tracking initialization;
[0072] Collect visual keyframes including RGB images and depth images of the odometry interval. When the actual motion offset is less than the set odometry interval, record n visual keyframes starting from the time interval.
[0073] Construct a one-to-one mapping relationship between each visual keyframe and posture.
[0074] In this application, the above image representation step first requires creating a hybrid LiDAR and visual information map in the pre-processed offline SLAM stage. For the SLAM map, the LiDAR data represents the environment based on the grid method and contains a series of keyframes and their Gist descriptors without building a 3D point map. In the similarity metric, we only consider the top n keyframes in the Gist distance between the query and the keyframe to be clustered.
[0075] In the preprocessing stage, offline SLAM is performed to build a hybrid map combining LiDAR scans and images. As with traditional LiDAR SLAM methods such as GMapping or Karto SLAM, the environment is discretized into two-dimensional square cells, each of which is associated with a value between [0, 1], which refers to the probability that the cell is occupied. In order to introduce image-based localization into the LiDAR method, by equipping the robot with an RGB-D camera, a set of keyframes represents the scene on an occupied grid map, which allows direct retrieval of candidate poses for tracking initialization. The selection of keyframes is very useful for achieving effective feature matching because consecutive frames are almost identical in a short period of time. Changes in the robot's motion will result in changes in the landscape observed in the environment. Secondly, even if the robot is stationary, the surrounding objects may move over time, therefore, collecting visual keyframes KF i , including the odometer interval Δx i When the actual motion deviation is less than the set odometer interval Δx i When the time interval Δt is recorded i The starting visual keyframe. Finally, a frame-pose correspondence mapping is constructed.
[0076] As described in step S200 above, the “positioning step based on key frames” includes:
[0077] Apply the Gist global scene descriptor to quickly retrieve and track candidate poses during initialization and determine the KF of each keyframe i According to the preset rules, it is represented as a Gist vector g i The average value of
[0078] According to each key frame KF i and the corresponding relationship between the Gist vector, comparing the distance between the current image and the key frame in the preset database, and determining the key frame with the closest distance to the maximum value as the candidate frame, and estimating the position of the robot in the environment through the posture information of the map bound to the candidate frame;
[0079] The local 3D points of the camera in the global frame are calculated through the RGB image and the depth image, and the position of the given point in the local and global coordinates is determined, which is used to calculate the transformation matrix between the query image and the proposed key frame to obtain more accurate pose information of the estimated query image.
[0080] The global descriptor is used to reduce the posture search space, the local descriptor is used to further improve the positioning accuracy, and the relocalization trigger mechanism is combined to effectively solve the global positioning method of the indoor mobile robot.
[0081] Example 3
[0082] A method for realizing global positioning of a mobile robot, wherein the Gist global scene descriptor is used to quickly retrieve and track candidate postures during initialization, and determine the KF of each key frame. i According to the preset rules, it is represented as a Gist vector g i The average values include:
[0083] Apply the Gist global scene descriptor to quickly retrieve candidate poses during tracking initialization;
[0084] Based on the context algorithm, global objects are identified and the keyframe image in each keyframe is divided into 4×4 blocks, where each block is represented by the average value of the filter in the [8,8,4] direction at each scale in the three color channels;
[0085] A Gist vector of length 960 is calculated for each keyframe.
[0086] According to each of the key frames KF i The corresponding relationship between the current image and the Gist vector is compared, and the key frame with the closest distance to the maximum value is determined as the candidate frame. The position of the robot in the environment is estimated by the posture information of the map bound to the candidate frame, which specifically includes:
[0087] According to each key frame KF i The corresponding relationship with the Gist vector is used to compare the current image with the key frames in the preset database to measure the difference between the two feature vectors;
[0088] Through the Gist feature vector g r and the distance d between the corresponding images gs rs , the Minkowski distance of a metric on a Euclidean space with order p is defined as: Calculate the Gist vector distance L2 of length k = 960 between each key frame in the database and the query image;
[0089] If the distance d rs If the key frame is closest to the maximum value N, then the key frame is considered as a candidate frame, and the position of the robot in the environment is roughly estimated by the pose information of the map bound to the candidate frame.
[0090] As an example, based on the positioning of the keyframes, local two-dimensional-three-dimensional (2D-3D) matching is performed using the ORB descriptor to calculate the pose transformation matrix between the query image and the candidate keyframes.
[0091] Gist global scene descriptor is applied to quickly retrieve candidate poses during tracking initialization. This method uses the input image as a whole to describe the image with statistical or semantic low-dimensional features in order to improve the robustness of scene classification. Since natural images are easily mixed with random noise, this noise can have a catastrophic effect on local processing, while for global images, it can reduce this effect by averaging. Context-based methods solve the problem of segmentation and target recognition by identifying global objects rather than the precise boundaries of a small group of objects or regions in the scene, without having to deal with noise in isolated areas and changes in low-level images. From each keyframe KF i In , the key frame image is divided into 4 by 4 blocks, each block is represented by the average value of the filter in the [8,8,4] direction at each scale in the three color channels. Finally, a Gist vector g of length 960 is calculated for each key frame i (344(8+8+4)=960).
[0092] Similarity measure. Once each keyframe has been turned into a Gist vector, we can compare the query image with the keyframes in the database to measure the difference between the two feature vectors. rs is the Gist eigenvector g r and g s The distance between the respective images. The Minkowski distance of the metric on the Euclidean space with order p is defined as: Calculate the Gist vector L2 distance of length k = 960 between each key frame in the database and the query image. If the distance d rs The keyframe closest to the maximum value N is considered a candidate frame. The pose information of the map bound to the candidate frame can roughly estimate the robot's position in the environment. However, due to the existence of keyframe intervals and different camera perspectives, this position estimate is very rough.
[0093] Further pose extraction. ORB is a fast and robust local feature detector for multi-scale fast corners with 256-bit descriptors for each corner. It is very fast to compute and match and is highly invariant to viewpoints. If the number of intraframes after applying RANSAC is higher than 12, the proposed keyframe can be used for further pose refinement. Otherwise, the pose information bound to the keyframe is regarded as the rough pose of the query image to be published, which is a non-optimal but feasible option. In the offline SLAM preprocessing stage, the local 3D points of the camera in the global frame are calculated through the RGB image and the depth image. Given the position of the point in local and global coordinates, the transformation matrix between the query image and the proposed keyframe can be calculated to estimate a more accurate pose of the query image.
[0094] Example 4
[0095] A method for realizing global positioning of a mobile robot, wherein the "re-positioning step" as described in step S300 includes: based on the posture information, using Scan2Map score to perform positioning evaluation to further determine whether to re-position in the re-positioning trigger mechanism, specifically including:
[0096] The visual and lidar information are integrated to refine the robot's pose in a maximum likelihood framework, where the robot's pose at time t is determined as:
[0097] Among them, the probability represents the probability of predicting the robot's posture based on the image-based localization model, Modeling the impact of keyframe clustering on continuous pose evaluation, m l is the laser radar plane map, is a lidar scan containing n individual measurements (n corresponding lidar beams);
[0098] After global coordinate reference optimization, the range reading is converted into a lidar map based on the robot posture, where the lidar endpoints The grid cell hit is expressed in m hit(k) express;
[0099] When cell m hit(k) is occupied, the occupancy value of the cell is added to a voting score, where the likelihood of the sensor measurement is given by the final voting score:
[0100] express;
[0101] The keyframe clusters are updated based on the scan Scan2Map scores. If the new cluster has the highest score after Scan2Map reordering, it is confirmed that the reordering has occurred, and the trigger mechanism makes the corresponding position of the candidate frame a candidate pose estimate.
[0102] In one example, performing the positioning step again includes using the Scan2Map score to perform a positioning evaluation to further determine whether to perform positioning again in the re-positioning trigger mechanism. Figure 3 As shown in Figure 2, two strategies based on the motion continuity assumption are integrated into the entire algorithm. The first one is to use the current image retrieval result R t To predict the next image retrieval result R t+1 Another approach considers LiDAR data to further assess the likelihood that the robot is more likely to enter the distribution of candidate boxes provided by image retrieval and keyframe clustering.
[0103] (1) Keyframe clustering. Whether there is a new keyframe cluster after k-means clustering will determine whether to enter the next session of the relocalization trigger mechanism or continue to track the robot's posture for a period of time. After calculating the distance between images in the process of calculating the similarity metric, the top candidate indexes of n keyframes are obtained. Then the indexes of the n candidate keyframes are clustered using the k-means clustering method, and the center of the cluster is replaced by the index of the best matching candidate frame.
[0104] (1) Scan2Map score. The visual and lidar information are integrated to refine the robot's posture in a maximum likelihood framework. The robot's posture at time t is determined as: Probability represents the probability of predicting the robot's posture based on the image-based localization model, Modeling the impact of keyframe clustering on continuous pose evaluation is the first step in the relocalization mechanism. Let m l is the laser radar plane map, is a lidar scan containing n individual measurements (n corresponding lidar beams). The range readings are first converted into a lidar map according to the proposed robot pose after global coordinate reference optimization. The grid cell hit is expressed in m hit(k) If cell m hit(k) If a cell is occupied, the occupancy value of the cell is added to a voting score. The probability of the sensor measurement is determined by the final voting score. Represents. Update the keyframe clusters based on the Scan2Map scores. If the new cluster has the highest score after Scan2Map reordering, it is confirmed that reordering has occurred. It triggers a mechanism that makes the corresponding position of the candidate frame a candidate pose estimate.
[0105] If a global pose is given, the robot will use the AMCL method for the pose tracking process unless the relocalization mechanism is triggered.
[0106] The present invention proposes a fast and reliable global positioning method in which a laser radar and a camera sensor are integrated. In the SLAM process, a laser radar is used to introduce visual keyframes in a series of environmental maps. A global descriptor is used to reduce the posture search space, a local descriptor is used to further improve the positioning accuracy, and a re-localization trigger mechanism is used to effectively solve the kidnapped robot problem. The present invention extends the keyframe clustering to a two-dimensional or three-dimensional posture space instead of clustering the sequence numbers of candidate keyframes. In addition, the possibility of improving the re-localization performance is to consider automatically determining the number of clusters based on the clustering data characteristics, and the reliability of the algorithm will also be tested in a complex and large scene in a hospital.
[0107] Example 5
[0108] Reference Figure 5 As shown, a global positioning device is used to realize the global positioning of a mobile robot in an indoor scene, and the device includes:
[0109] The image representation module 100 is used to represent environmental parameters of the laser radar data based on the network method in the offline SLAM stage, obtain a visual information map containing a series of key frames and their Gist descriptors, determine the top n key frames in the Gist distance between the key frames to be queried and to be clustered, and construct a frame-pose mapping relationship;
[0110] The first positioning module 200 is used to apply the ORB descriptor to perform local 2D-3D matching, calculate the pose transformation matrix between the query image and the candidate key frame, and obtain the pose information of the estimated query image;
[0111] A second positioning module 300 is used to perform positioning evaluation based on the posture information using Scan2Map score to further determine whether to perform positioning again in the re-positioning trigger mechanism;
[0112] The posture tracking module 400 is used to, if the repositioning mechanism is triggered or not, perform a posture tracking process by the robot using the AMCL method according to a preset global posture.
[0113] Reference Figure 6 , shows a computer device for a control method for automatic control of the present invention, which may specifically include the following:
[0114] The computer device 12 is in the form of a general-purpose computing device, and the components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).
[0115] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor or a local bus 18 using any of a variety of bus 18 architectures. These architectures include, by way of example, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.
[0116] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.
[0117] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although Figure 6 Not shown, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42, which are configured to perform the functions of various embodiments of the present invention.
[0118] A program / utility 40 having a set (at least one) of program modules 42 may be stored in, for example, a memory, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0119] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, cameras, etc.), one or more devices that enable medical personnel to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs)), wide area networks (WANs), and / or public networks (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that although Figure 6Not shown, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to: microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34, etc.
[0120] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the global positioning method provided by the embodiment of the present invention.
[0121] That is, when the processing unit 16 executes the above program, it realizes: in the offline SLAM stage, based on the network method, the laser radar data is used to represent the environmental parameters, and a visual information map containing a series of key frames and their Gist descriptors is obtained, and the top n key frames in the Gist distance between the key frames to be queried and to be clustered are determined, and a frame-posture mapping relationship is constructed; the positioning step based on the key frame: the ORB descriptor is used for local two-dimensional-three-dimensional matching, and the posture transformation matrix between the query image and the candidate key frames is calculated to obtain the estimated posture information of the query image; based on the posture information, the Scan2Map score is used for positioning evaluation to further determine whether to re-position in the re-positioning trigger mechanism; if the re-positioning mechanism is triggered or not, the robot uses the AMCL method to perform posture tracking according to a preset global posture.
[0122] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the global positioning method provided in all embodiments of the present application is implemented:
[0123] That is, when the program is executed by the processor, it is implemented as follows: in the offline SLAM stage, the laser radar data is used to represent environmental parameters based on the network method, and a visual information map containing a series of key frames and their Gist descriptors is obtained, and the top n key frames in the Gist distance between the key frames to be queried and clustered are determined, and a frame-pose mapping relationship is constructed; the positioning step based on the key frame: the ORB descriptor is used for local two-dimensional-three-dimensional matching, and the pose transformation matrix between the query image and the candidate key frames is calculated to obtain the estimated pose information of the query image; based on the pose information, the Scan2Map score is used for positioning evaluation to further determine whether to re-position in the relocalization trigger mechanism; if the relocalization mechanism is triggered or not, the robot uses the AMCL method for pose tracking according to a preset global pose.
[0124] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPOM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device.
[0125] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0126] The computer program code for performing the operation of the present invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the medical staff computer, partially on the medical staff computer, as an independent software package, partially on the medical staff computer, partially on the remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the medical staff computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.
[0127] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0128] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0129] The above is a detailed introduction to a global positioning method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A global positioning method for realizing global positioning of a mobile robot in an indoor scene, characterized in that: The method comprises: Image representation step: In the offline SLAM stage, the lidar data is used to represent environmental parameters based on the network method, and a visual information map containing a series of key frames and their Gist descriptors is obtained. The top n key frames in the Gist distance between the key frames to be queried and clustered are determined, and a frame-pose mapping relationship is constructed; Keyframe-based positioning step: Apply ORB descriptor for local 2D-3D matching, calculate the pose transformation matrix between the query image and the candidate keyframe, and obtain the pose information of the estimated query image; Re-localization step: Based on the posture information, the Scan2Map score is used to evaluate the localization to further determine whether to re-localize in the relocalization trigger mechanism; Posture tracking step: If the relocalization mechanism is triggered or not, the robot uses the AMCL method to perform posture tracking according to a preset global posture.
2. The method according to claim 1, characterized in that The image representation step comprises: Based on the network method, the lidar data is used to represent the environmental parameters, and a visual information map containing a series of key frames and their Gist descriptors is obtained. The top n key frames in the Gist distance between the key frames to be queried and clustered are determined, and a frame-pose mapping relationship is constructed. In the offline SLAM stage, the environment is discretized into two-dimensional square cells, each of which is associated with a value between [0, 1], where the value is the probability of the cell being occupied; Obtain a series of keyframes through a preset RGB-D camera, and represent at least one set of keyframes as a scene on an occupied grid map to determine candidate poses that can be directly retrieved for tracking initialization; Collect visual keyframes including RGB images and depth images of the odometry interval. When the actual motion offset is less than the set odometry interval, record n visual keyframes starting from the time interval. Construct a one-to-one mapping relationship between each visual keyframe and posture.
3. The method according to claim 1, characterized in that The keyframe-based positioning step comprises: Apply the Gist global scene descriptor to quickly retrieve and track candidate poses during initialization and determine the KF of each keyframe i According to the preset rules, it is represented as a Gist vector g i The average value of According to each key frame KF i and the corresponding relationship between the Gist vector, comparing the distance between the current image and the key frame in the preset database, and determining the key frame with the closest distance to the maximum value as the candidate frame, and estimating the position of the robot in the environment through the posture information of the map bound to the candidate frame; The local 3D points of the camera in the global frame are calculated through the RGB image and the depth image, and the position of the given point in the local and global coordinates is determined, which is used to calculate the transformation matrix between the query image and the proposed key frame to obtain more accurate pose information of the estimated query image.
4. The method according to claim 3, characterized in that The Gist global scene descriptor is applied to quickly retrieve and track candidate poses during the initialization process and determine the KF of each key frame. i According to the preset rules, it is represented as a Gist vector g i The average values include: Apply the Gist global scene descriptor to quickly retrieve candidate poses during tracking initialization; Based on the context algorithm, global objects are identified and the keyframe image in each keyframe is segmented into 4×4 blocks, where each block is represented by the average value of the filters in the [8, 8, 4] direction at each scale in the three color channels; A Gist vector of length 960 is calculated for each keyframe.
5. The method according to claim 3, characterized in that: According to each of the key frames KF i The corresponding relationship between the current image and the Gist vector is compared, and the key frame with the closest distance to the maximum value is determined as the candidate frame. The position of the robot in the environment is estimated by the posture information of the map bound to the candidate frame, which specifically includes: According to each key frame KF i The corresponding relationship with the Gist vector is used to compare the current image with the key frames in the preset database to measure the difference between the two feature vectors; Through the Gist feature vector g r and corresponding to their respective images g s The distance between rs , the Minkowski distance of a metric on a Euclidean space with order p is defined as: Calculate the Gist vector distance L2 of length k = 960 between each key frame in the database and the query image; If the distance d rs If the key frame is closest to the maximum value N, then the key frame is considered as a candidate frame, and the position of the robot in the environment is roughly estimated by the pose information of the map bound to the candidate frame.
6. The method according to claim 1, characterized in that The re-positioning step specifically includes: The visual and lidar information are integrated to refine the robot's pose in a maximum likelihood framework, where the robot's pose at time t is determined as: Among them, the probability represents the probability of predicting the robot's posture based on the image-based localization model, Modeling the impact of keyframe clustering on continuous pose evaluation, m l is the laser radar plane map, is a lidar scan containing n individual measurements (n corresponding lidar beams); After global coordinate reference optimization, the range reading is converted into a lidar map based on the robot posture, where the lidar endpoints The grid cell hit is expressed in m hit(k) express; When cell m hit(k) is occupied, the occupancy value of the cell is added to a voting score, where the likelihood of the sensor measurement is given by the final voting score: m hit(k) Occupied} express; The keyframe clusters are updated based on the scan Scan2Map scores. If the new cluster has the highest score after Scan2Map reordering, it is confirmed that the reordering has occurred, and the trigger mechanism makes the corresponding position of the candidate frame a candidate pose estimate.
7. The method according to claim 6, characterized in that The described approach integrates visual and lidar information to refine the robot's pose in a maximum likelihood framework, which previously also included: Depending on whether there are new keyframe clusters after k-means clustering, decide whether to enter the next session of the relocalization trigger mechanism or continue tracking the robot's posture for a period of time; After calculating the distance between images in the process of calculating the similarity metric, the top candidate indexes of n key frames are obtained; The indexes of the n candidate key frames are clustered using the k-means clustering method, and the center of the cluster is replaced by the index of the best matching candidate frame.
8. A global positioning device for realizing global positioning of a mobile robot in an indoor scene, characterized in that: The device comprises: The image representation module is used to represent the environmental parameters of the lidar data based on the network method in the offline SLAM stage, obtain a visual information map containing a series of key frames and their Gist descriptors, determine the top n key frames in the Gist distance between the key frames to be queried and clustered, and construct a frame-pose mapping relationship; The first positioning module is used to apply the ORB descriptor to perform local 2D-3D matching, calculate the pose transformation matrix between the query image and the candidate key frame, and obtain the pose information of the estimated query image; The second positioning module is used to perform positioning evaluation based on the posture information using Scan2Map score to further determine whether to perform positioning again in the re-localization trigger mechanism; The posture tracking module is used to perform a posture tracking process using the AMCL method based on a preset global posture if the repositioning mechanism is triggered or not.
9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Robot positioning method and device
CN108256574A
Global self-positioning method and device for mobile robot
CN109144056A
Indoor positioning method and device based on WIFI and laser radar, and robot
CN112082553A
Method for acquiring robot laser odometer based on dynamic target tracking
CN116736330A
Multi-factor graph-based back-end optimization method for acquiring precise pose of robot
CN116758153A
