A global positioning method and device

Abstract

The present invention proposes a method and apparatus for achieving global positioning of a mobile robot, in which a laser radar and a camera sensor are integrated. During the SLAM process, a laser radar is used to introduce visual keyframes in a series of environmental maps. A global descriptor is used to reduce the pose search space, a local descriptor is used to further improve the positioning accuracy, and a re-positioning trigger mechanism is used to effectively solve the problem of the kidnapped robot. The present invention extends keyframe clustering to a two-dimensional or three-dimensional pose space, rather than clustering the sequence numbers of candidate keyframes. In addition, the possibility of improving the re-positioning performance is to consider automatically determining the number of clusters based on the characteristics of the clustering data. The reliability of the algorithm will also be tested in a complex and large hospital scene.

Description

Technical Field

[0001] The present invention relates to an AMR (Advanced Mobile Robot) for indoor patient transport in large hospital scenarios, and specifically to a global positioning method and device for an indoor mobile robot, belonging to the technical field of robot positioning. Background Art

[0002] Mobile robot localization involves determining a robot's posture relative to its surroundings. It's considered a crucial function for mobile robots to accomplish autonomous tasks. Generally speaking, localization can be categorized into two types: position tracking and global positioning.

[0003] Position tracking assumes that the initial pose of the robot is known. The pose of the robot can be found by correcting the pose error and tracking the robot over time in a known environment. The global localization problem is more difficult than position tracking because it does not have an initial seed position to start the search for the pose of a given map.

[0004] The kidnapped robot problem involves a well-localized robot being kidnapped and teleported to another location, where it may believe it is. This also constitutes global positioning. Image-based localization methods address this problem by matching a query image against a database of images, then estimating the image's pose, which can be applied to global positioning. However, feature matching relies on the invariance of feature viewpoints. Furthermore, when there are thousands of 3D points and their associated descriptors, searching for precise correspondences in the space becomes challenging. It is difficult for a robot to identify a location that was previously only visible through a single LiDAR scan. This situation is even more pronounced in geometric structures. Although LiDAR sensors offer higher accuracy than cameras, they reduce the dimensionality of the perceived environment. Generally speaking, due to their rich visual information, vision-based methods offer better performance than pure LiDAR methods in data association and loop closure detection.

[0005] Global localization is the problem of estimating a robot's pose without any prior knowledge of its initial pose. Global localization using LiDAR is difficult to achieve when only a limited number of LiDAR scans are available, particularly in simple and repetitive environments with localized geometric features. Cameras offer the advantage of rich information, but lack reliable and efficient localization algorithms and carry a relatively high computational burden. Therefore, this application proposes a method and apparatus for achieving global localization of a mobile robot to at least partially address the problems that may exist in the prior art. Summary of the Invention

[0006] In order to overcome the above problems or at least partially solve the above problems, an embodiment of the present invention provides a method and apparatus for achieving global positioning of a mobile robot in an indoor scene.

[0007] The embodiment of the present invention is achieved as follows:

[0008] An embodiment of the present application provides a method for achieving global positioning of a mobile robot, the method comprising:

[0009] Image representation step: In the offline SLAM stage, the lidar data is used to represent environmental parameters based on the grid method. A visual information map containing a series of keyframes and their Gist descriptors is obtained. The top n keyframes in the Gist distance between the keyframes to be queried and clustered are determined, and a frame-pose mapping relationship is constructed.

[0010] Keyframe-based positioning step: Apply ORB descriptors to perform local 2D-3D matching, calculate the pose transformation matrix between the query image and the candidate keyframes, and obtain the pose information of the estimated query image;

[0011] Re-localization step: Based on the posture information, the Scan2Map score is used to evaluate the localization to further determine whether to re-localize in the relocalization trigger mechanism;

[0012] Posture tracking step: If the relocalization mechanism is not triggered, the robot uses the AMCL method to perform posture tracking according to a preset global posture.

[0013] In some embodiments of the present invention, the image representation step includes:

[0014] Based on the grid method, the lidar data represents the environmental parameters, obtains a visual information map containing a series of key frames and their Gist descriptors, determines the top n key frames in the Gist distance between the key frames to be queried and clustered, and constructs a frame-pose mapping relationship;

[0015] In the offline SLAM stage, the environment is discretized into two-dimensional square cells, each of which is associated with a value between [0, 1], where the value is the probability of the cell being occupied;

[0016] Acquire a series of keyframes through a preset RGB-D camera, and represent at least one set of keyframes as a scene on an occupied grid map to determine candidate poses that can be directly retrieved for tracking initialization;

[0017] Collect visual keyframes including RGB images and depth images at the odometry interval. When the actual motion offset is less than the set odometry interval, record n visual keyframes starting from the time interval.

[0018] Construct a one-to-one mapping relationship between each visual keyframe and posture.

[0019] In some embodiments of the present invention, the keyframe-based positioning step includes:

[0020] Apply Gist global scene descriptor to quickly retrieve and track candidate poses during initialization and determine the KF of each keyframe i According to the preset rules, it is expressed as a Gist vector g i The average value of

[0021] According to each key frame KF i The corresponding relationship between the current image and the key frames in the preset database is compared, and the key frame with the closest distance to the maximum value is determined as the candidate frame, and the position of the robot in the environment is estimated by the posture information of the map bound to the candidate frame;

[0022] The local 3D points of the camera in the global frame are calculated through the RGB image and the depth image, and the position of the given point in the local and global coordinates is determined, which is used to calculate the transformation matrix between the query image and the proposed key frame to obtain more accurate pose information of the estimated query image.

[0023] In some embodiments of the present invention, the Gist global scene descriptor is used to quickly retrieve and track candidate poses during the initialization process to determine each key frame KF i According to the preset rules, it is expressed as a Gist vector g i The average value includes:

[0024] Apply Gist global scene descriptor to quickly retrieve candidate poses during tracking initialization;

[0025] Based on the context algorithm, global objects are identified and the key frame image in each key frame is divided into 4×4 blocks, where each block is represented by the average value of the filters in the [8, 8, 4] direction at each scale in the three color channels;

[0026] Calculate a Gist vector of length 960 for each keyframe.

[0027] In some embodiments of the present invention, the i The corresponding relationship between the current image and the Gist vector is compared, and the key frame with the closest distance to the maximum value is determined as the candidate frame. The position of the robot in the environment is estimated by the posture information of the map bound to the candidate frame, specifically including:

[0028] According to each key frame KF i The corresponding relationship between the Gist vector and the current image is compared with the key frames in the preset database to measure the difference between the two feature vectors;

[0029] Through Gist feature vector and corresponding to their respective images The distance between , the Minkowski distance of a metric on a Euclidean space with order p is defined as: , calculate the Gist vector distance L2 of length k = 960 between each key frame in the database and the query image;

[0030] If the distance The key frame closest to the maximum value N is considered as a candidate frame, and the position of the robot in the environment is roughly estimated by the pose information of the map bound to the candidate frame.

[0031] In some embodiments of the present invention, the re-positioning step specifically includes:

[0032] The visual and lidar information are integrated to refine the robot's pose in a maximum likelihood framework, where the robot's pose at time t is determined as: , where the probability represents the probability of predicting the robot's posture based on the image positioning model, Modeling the impact of keyframe clustering on continuous pose evaluation, is the laser radar plane map, is a lidar scan consisting of n individual measurements (n corresponding lidar beams);

[0033] After global coordinate reference optimization, the range reading is converted into a lidar map based on the robot pose, where the lidar endpoints The grid cell hit is express;

[0034] When the cell is occupied, the occupancy value of the cell is added to a voting score, where the likelihood of the sensor measurement is given by the final voting score: express;

[0035] The keyframe clusters are updated based on the Scan2Map scores. If the new cluster has the highest score after Scan2Map reordering, it is confirmed that reordering has occurred, and the trigger mechanism makes the corresponding position of the candidate frame a candidate pose estimate.

[0036] In some embodiments of the present invention, the step of integrating visual and lidar information to refine the robot's posture in a maximum likelihood framework may also include:

[0037] Based on whether there are new keyframe clusters after k-means clustering, it is decided whether to enter the next session of the relocalization trigger mechanism or continue tracking the robot's posture for a period of time;

[0038] After calculating the distance between images in the process of calculating the similarity metric, the top candidate indexes of n key frames are obtained;

[0039] The indexes of n candidate key frames are clustered using the k-means clustering method, and the center of the cluster is replaced by the index of the best matching candidate frame.

[0040] A device for realizing global positioning of a mobile robot, the device comprising:

[0041] The image representation module is used to represent environmental parameters from LiDAR data based on the grid method during the offline SLAM phase, obtain a visual information map containing a series of keyframes and their Gist descriptors, determine the top n keyframes in the Gist distance between the keyframes to be queried and clustered, and construct a frame-pose mapping relationship;

[0042] The first positioning module is used to apply the ORB descriptor to perform local 2D-3D matching, calculate the pose transformation matrix between the query image and the candidate keyframe, and obtain the pose information of the estimated query image;

[0043] The second positioning module is used to perform positioning evaluation based on the posture information using Scan2Map score to further determine whether to perform positioning again in the re-localization trigger mechanism;

[0044] The posture tracking module is used to perform posture tracking using the AMCL method based on a preset global posture if the relocalization mechanism is not triggered.

[0045] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned global positioning method of a mobile robot when executed by the processor.

[0046] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the global positioning method of a mobile robot is implemented.

[0047] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:

[0048] During the offline SLAM phase, the grid method is used to represent environmental parameters using LiDAR data. A visual information map containing a series of keyframes and their Gist descriptors is obtained. The top n keyframes in the Gist distance between the query and clustered keyframes are determined, and a frame-pose mapping relationship is constructed. The keyframe-based positioning step involves applying ORB descriptors for local 2D-3D matching, calculating the pose transformation matrix between the query image and the candidate keyframes, and obtaining the pose information of the estimated query image. Based on this pose information, the Scan2Map score is used for positioning evaluation to further determine whether to re-localize in the relocalization trigger mechanism. If the relocalization mechanism is not triggered, the robot uses the AMCL method to track the pose according to a preset global pose. To address the aforementioned AMR kidnapping problem in global positioning, a global descriptor is used to reduce the pose search space, local descriptors are used to further improve positioning accuracy, and combined with the relocalization trigger mechanism, an effective solution for the global positioning method of indoor mobile robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A flowchart of a global positioning method according to an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the technical framework structure of a global positioning method provided by one embodiment of the present invention;

[0052] Figure 3 A flow chart of a relocation trigger mechanism of a global positioning method provided by one embodiment of the present invention;

[0053] Figure 4 The Gist distance between a sample query image randomly selected from a scene and a key frame of each data set in a global positioning method provided by one embodiment of the present invention;

[0054] Figure 5 A schematic diagram of the module structure of a global positioning device provided by one embodiment of the present invention;

[0055] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0057] Example 1

[0058] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features thereof may be combined with each other.

[0059] Please refer to Figure 1 , an embodiment of the present invention provides a flowchart of a method for implementing global positioning of a mobile robot, the method comprising:

[0060] S100 image representation step: In the offline SLAM stage, the lidar data is represented as environmental parameters based on the grid method, a visual information map containing a series of keyframes and their Gist descriptors is obtained, and the top n keyframes ranked in the Gist distance between the keyframes to be queried and clustered are determined, and a frame-pose mapping relationship is constructed;

[0061] S200 is a keyframe-based positioning step: applying ORB descriptors to perform local 2D-3D matching, calculating the pose transformation matrix between the query image and the candidate keyframes, and obtaining the pose information of the estimated query image;

[0062] S300 performs a re-positioning step: based on the posture information, a positioning evaluation is performed using the Scan2Map score to further determine whether to perform re-positioning in the re-positioning trigger mechanism;

[0063] S400 posture tracking step: If the repositioning mechanism is not triggered, the robot uses the AMCL method to perform posture tracking according to a preset global posture.

[0064] The present invention proposes a fast and reliable global localization method in which a lidar and a camera sensor are integrated. In the SLAM process, a lidar is used to introduce visual keyframes in a series of environment maps. A global descriptor is used to reduce the pose search space, a local descriptor is used to further improve the positioning accuracy, and a relocalization trigger mechanism is used to effectively solve the kidnapped robot problem. The present invention extends keyframe clustering to a two-dimensional or three-dimensional pose space instead of clustering the sequence numbers of candidate keyframes. In addition, the possibility of improving the relocalization performance is to consider automatically determining the number of clusters based on the characteristics of the clustering data, and the reliability of the algorithm will also be tested in a complex and large scene in a hospital.

[0065] Example 2

[0066] A global positioning method is provided for achieving global positioning of a mobile robot in an indoor scene, wherein the "image representation step" described in step S100 includes:

[0067] Based on the grid method, the lidar data represents the environmental parameters, obtains a visual information map containing a series of key frames and their Gist descriptors, determines the top n key frames in the Gist distance between the key frames to be queried and clustered, and constructs a frame-pose mapping relationship;

[0068] In the offline SLAM stage, the environment is discretized into two-dimensional square cells, each of which is associated with a value between [0, 1], where the value is the probability of the cell being occupied;

[0069] Acquire a series of keyframes through a preset RGB-D camera, and represent at least one set of keyframes as a scene on an occupied grid map to determine candidate poses that can be directly retrieved for tracking initialization;

[0070] Collect visual keyframes including RGB images and depth images at the odometry interval. When the actual motion offset is less than the set odometry interval, record n visual keyframes starting from the time interval.

[0071] Construct a one-to-one mapping relationship between each visual keyframe and posture.

[0072] In this application, the image representation step described above first requires creating a hybrid LiDAR and visual map during the pre-processing offline SLAM phase. For this SLAM map, the LiDAR data is used to represent the environment using a grid-based approach, consisting of a series of keyframes and their Gist descriptors, rather than a 3D point map. For the similarity metric, only the top n keyframes in the Gist distance between the query and the keyframes to be clustered are considered.

[0073] In the preprocessing stage, offline SLAM is performed to build a hybrid map combining LiDAR scans and images. As with traditional LiDAR SLAM methods such as GMapping or Karto SLAM, the environment is discretized into two-dimensional square cells, each of which is associated with a value between [0, 1], which refers to the probability of the cell being occupied. In order to introduce image-based localization into the LiDAR method, by equipping the robot with an RGB-D camera, a set of keyframes represents the scene on an occupied grid map, which allows direct retrieval of candidate poses for tracking initialization. The selection of keyframes is very useful for achieving effective feature matching because consecutive frames are almost identical over a short period of time. Changes in the robot's motion will result in changes in the landscape observed in the environment. Second, even if the robot is stationary, the surrounding objects may move over time, so collecting visual keyframes is very important. , including odometer intervals When the actual motion offset is less than the set odometer interval When recording from the time interval Finally, a frame-to-pose mapping is constructed.

[0074] As described in step S200 above, the "keyframe-based positioning step" includes:

[0075] Apply Gist global scene descriptor to quickly retrieve and track candidate poses during initialization and determine the KF of each keyframe i According to the preset rules, it is expressed as a Gist vector g i The average value of

[0076] According to each key frame KF i The corresponding relationship between the current image and the key frames in the preset database is compared, and the key frame with the closest distance to the maximum value is determined as the candidate frame, and the position of the robot in the environment is estimated by the posture information of the map bound to the candidate frame;

[0077] The local 3D points of the camera in the global frame are calculated through the RGB image and the depth image, and the position of the given point in the local and global coordinates is determined, which is used to calculate the transformation matrix between the query image and the proposed key frame to obtain more accurate pose information of the estimated query image.

[0078] A global descriptor is used to reduce the pose search space, and a local descriptor is used to further improve the positioning accuracy. Combined with the relocalization trigger mechanism, the global positioning method of indoor mobile robots is effectively solved.

[0079] Example 3

[0080] A method for realizing global positioning of a mobile robot, wherein the Gist global scene descriptor is applied to quickly retrieve and track candidate poses during initialization, and determine the KF of each key frame. i According to the preset rules, it is expressed as a Gist vector g i The average value includes:

[0081] Apply Gist global scene descriptor to quickly retrieve candidate poses during tracking initialization;

[0082] Based on the context algorithm, global objects are identified and the key frame image in each key frame is divided into 4×4 blocks, where each block is represented by the average value of the filters in the [8, 8, 4] direction at each scale in the three color channels;

[0083] Calculate a Gist vector of length 960 for each keyframe.

[0084] According to each key frame KF i The corresponding relationship between the current image and the Gist vector is compared, and the key frame with the closest distance to the maximum value is determined as the candidate frame. The position of the robot in the environment is estimated by the posture information of the map bound to the candidate frame, specifically including:

[0085] According to each key frame KF i The corresponding relationship between the Gist vector and the current image is compared with the key frames in the preset database to measure the difference between the two feature vectors;

[0086] Through Gist feature vector and corresponding to their respective images The distance between , the Minkowski distance of a metric on a Euclidean space with order p is defined as: , calculate the Gist vector distance L2 of length k = 960 between each key frame in the database and the query image;

[0087] If the distance The key frame closest to the maximum value N is considered as a candidate frame, and the position of the robot in the environment is roughly estimated by the pose information of the map bound to the candidate frame.

[0088] As an example, based on the positioning of keyframes, ORB descriptors are used to perform local two-dimensional-three-dimensional (2D-3D) matching to calculate the pose transformation matrix between the query image and the candidate keyframes.

[0089] Gist global scene descriptor is applied to quickly retrieve candidate poses during tracking initialization. This method uses the input image as a whole feature to describe the image with statistical or semantic low-dimensional features in order to improve the robustness of scene classification. Since natural images are easily mixed with random noise, this noise will have a disastrous effect on local processing, while for global images, it can reduce this effect by averaging. Context-based methods solve the problem of segmentation and target recognition by identifying global objects rather than the precise boundaries of a small group of objects or regions in the scene, without having to deal with noise in isolated areas and changes in low-level images. From each key frame In , the key frame image is divided into 4 by 4 blocks, each block is represented by the average value of the filter in the [8,8,4] direction at each scale in the 3 color channels. Finally, a Gist vector of length 960 is calculated for each key frame (344(8+8+4)=960).

[0090] Similarity measure. Once each keyframe is turned into a Gist vector, we can compare the query image with the keyframes in the database to measure the difference between the two feature vectors. If is the Gist feature vector and The distance between the respective images. The Minkowski distance of the metric on the Euclidean space with order p is defined as: . Calculate the Gist vector L2 distance of length k = 960 between each key frame in the database and the query image. If the distance The keyframe closest to the maximum value N is considered a candidate frame. The pose information of the map bound to the candidate frame can roughly estimate the robot's position in the environment. However, due to the existence of keyframe intervals and the difference in camera perspective, this position estimate is very coarse.

[0091] Further pose extraction. ORB is a fast and robust local feature detector for multi-scale fast corners with 256-bit descriptors for each corner. It is very fast to compute and match, and is highly invariant to viewpoints. If the number of intra-frames after applying RANSAC is higher than 12, the proposed keyframe can be used for further pose refinement. Otherwise, the pose information bound to the keyframe is regarded as the rough pose of the query image to be published, which is a non-optimal but feasible option. In the offline SLAM preprocessing stage, the local 3D points of the camera in the global frame are calculated using the RGB image and the depth image. Given the position of the point in local and global coordinates, the transformation matrix between the query image and the proposed keyframe can be calculated to estimate a more accurate pose of the query image.

[0092] Example 4

[0093] A method for achieving global localization of a mobile robot, wherein the "re-localization step" described in step S300 includes: performing a localization evaluation based on posture information using a Scan2Map score to further determine whether to re-localize in a relocalization trigger mechanism, specifically including:

[0094] The visual and lidar information are integrated to refine the robot's pose in a maximum likelihood framework, where the robot's pose at time t is determined as: , where the probability represents the probability of predicting the robot's posture based on the image positioning model, Modeling the impact of keyframe clustering on continuous pose evaluation, is the laser radar plane map, is a lidar scan consisting of n individual measurements (n corresponding lidar beams);

[0095] After global coordinate reference optimization, the range reading is converted into a lidar map based on the robot pose, where the lidar endpoints The grid cell hit is express;

[0096] When the cell is occupied, the occupancy value of the cell is added to a voting score, where the likelihood of the sensor measurement is given by the final voting score: express;

[0097] The keyframe clusters are updated based on the Scan2Map scores. If the new cluster has the highest score after Scan2Map reordering, it is confirmed that reordering has occurred, and the trigger mechanism makes the corresponding position of the candidate frame a candidate pose estimate.

[0098] In one example, performing the positioning step again includes using the Scan2Map score to perform positioning evaluation to further determine whether to perform positioning again in the re-positioning trigger mechanism. Figure 3 As shown in Figure 2, two strategies based on the motion continuity assumption are integrated into the entire algorithm. The first one is to use the current image retrieval results To predict the next image retrieval result Another approach considers LiDAR data to further evaluate the likelihood that the robot is more likely to fall into the distribution of candidate boxes provided by image retrieval and keyframe clustering.

[0099] (1) Keyframe clustering. Whether a new keyframe cluster exists after k-means clustering determines whether to enter the next session of the relocalization trigger mechanism or continue tracking the robot's posture for a period of time. After calculating the distance between images in the similarity metric, the top candidate indexes of n keyframes are obtained. The indexes of the n candidate keyframes are then clustered using the k-means clustering method, and the center of the cluster is replaced by the index of the best matching candidate frame.

[0100] (1) Scan2Map score. The visual and lidar information are integrated to refine the robot’s pose in a maximum likelihood framework. The robot’s pose at time t is determined as: , probability represents the probability of predicting the robot's posture based on the image positioning model, Modeling the impact of keyframe clustering on continuous pose evaluation is the first step in the relocalization mechanism. is the laser radar plane map, is a lidar scan containing n individual measurements (n corresponding lidar beams). The range readings are first converted into a lidar map based on the proposed robot pose after global coordinate reference optimization. The grid cell hit is If the cell If the cell is occupied, the occupancy value of the cell will be added to a voting score. The probability of the sensor measurement is determined by the final voting score Represents. Updates the keyframe clusters based on the Scan2Map scores. If the new cluster has the highest score after Scan2Map reordering, then reordering is confirmed to have occurred. This triggers a mechanism that makes the corresponding position of the candidate frame a candidate pose estimate.

[0101] If a global pose is given, the robot will use the AMCL method for the pose tracking process unless the relocalization mechanism is triggered.

[0102] The present invention proposes a fast and reliable global localization method in which a lidar and a camera sensor are integrated. In the SLAM process, a lidar is used to introduce visual keyframes in a series of environment maps. A global descriptor is used to reduce the pose search space, a local descriptor is used to further improve the positioning accuracy, and a relocalization trigger mechanism is used to effectively solve the kidnapped robot problem. The present invention extends keyframe clustering to a two-dimensional or three-dimensional pose space instead of clustering the sequence numbers of candidate keyframes. In addition, the possibility of improving the relocalization performance is to consider automatically determining the number of clusters based on the characteristics of the clustering data, and the reliability of the algorithm will also be tested in a complex and large scene in a hospital.

[0103] Example 5

[0104] Reference Figure 5 As shown, a global positioning device is used to achieve global positioning of a mobile robot in an indoor scene, and the device includes:

[0105] The image representation module 100 is used to represent environmental parameters from the lidar data based on the grid method in the offline SLAM stage, obtain a visual information map containing a series of key frames and their Gist descriptors, determine the top n key frames in the Gist distance between the key frames to be queried and to be clustered, and construct a frame-pose mapping relationship;

[0106] The first positioning module 200 is used to perform local 2D-3D matching using ORB descriptors, calculate the pose transformation matrix between the query image and the candidate keyframes, and obtain the pose information of the estimated query image;

[0107] A second positioning module 300 is configured to perform positioning evaluation based on the posture information using the Scan2Map score to further determine whether to perform positioning again in a re-positioning trigger mechanism;

[0108] The posture tracking module 400 is used to, if the relocation mechanism is not triggered, enable the robot to perform a posture tracking process using the AMCL method according to a preset global posture.

[0109] Reference Figure 6 , shows a computer device for a control method for automated control of the present invention, which may specifically include the following:

[0110] The computer device 12 is a general-purpose computing device. The components of the computer device 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0111] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor, or a local bus 18 that utilizes any of a variety of bus 18 architectures. Examples of such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.

[0112] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0113] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read from and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although Figure 6 Although not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), as well as an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) can be provided. In these cases, each drive can be connected to bus 18 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 configured to perform the functions of various embodiments of the present invention.

[0114] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in a memory. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules 42, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0115] The computer device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, a camera, etc.), one or more devices that enable medical personnel to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN)), a wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 6Not shown, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34.

[0116] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the global positioning method provided by the embodiment of the present invention.

[0117] That is, when the processing unit 16 executes the above program, it realizes: in the offline SLAM stage, the laser radar data is represented as environmental parameters based on the grid method, a visual information map containing a series of key frames and their Gist descriptors is obtained, and the top n key frames in the Gist distance between the key frames to be queried and to be clustered are determined, and a frame-pose mapping relationship is constructed; the positioning step based on the key frame: the ORB descriptor is applied for local two-dimensional-three-dimensional matching, the pose transformation matrix between the query image and the candidate key frames is calculated, and the pose information of the estimated query image is obtained; based on the pose information, the Scan2Map score is used for positioning evaluation to further determine whether to re-position in the relocalization trigger mechanism; if the relocalization mechanism is not triggered, the robot uses the AMCL method to perform pose tracking according to a preset global pose.

[0118] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the global positioning method provided in all embodiments of the present application:

[0119] That is, when the program is executed by the processor, it is implemented as follows: in the offline SLAM stage, the lidar data is used to represent the environmental parameters based on the grid method, and a visual information map containing a series of key frames and their Gist descriptors is obtained, and the top n key frames in the Gist distance between the key frames to be queried and to be clustered are determined, and a frame-pose mapping relationship is constructed; the positioning step based on the key frame: the ORB descriptor is applied for local two-dimensional-three-dimensional matching, the pose transformation matrix between the query image and the candidate key frames is calculated, and the pose information of the estimated query image is obtained; based on the pose information, the Scan2Map score is used for positioning evaluation to further determine whether to re-position in the relocalization trigger mechanism; if the relocalization mechanism is not triggered, the robot uses the AMCL method for pose tracking according to a preset global pose.

[0120] Any combination of one or more computer-readable media may be employed. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0121] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0122] The computer program code for performing the operations of the present invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the medical staff computer, partially on the medical staff computer, as a stand-alone software package, partially on the medical staff computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the medical staff computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through the Internet using an Internet service provider). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced.

[0123] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0124] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0125] The above is a detailed introduction to a global positioning method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A global positioning method for realizing global positioning of a mobile robot in an indoor scene, characterized in that: The method comprises: Image representation step: In the offline SLAM stage, the lidar data is represented as environmental parameters based on the grid method, and a visual information map containing a series of key frames and their Gist descriptors is obtained. The top n key frames in the Gist distance between the key frames to be queried and clustered are determined, and a frame-pose mapping relationship is constructed, specifically including: in the offline SLAM stage, the environment is discretized into two-dimensional square units, each unit is associated with a value between [0, 1], where the value is the probability of the unit being occupied; a series of key frames are obtained through a preset RGB-D camera, and at least one group of key frames is represented as a scene on the occupied grid map, and candidate poses that can be directly retrieved for tracking initialization are determined; visual key frames including RGB images and depth images of the odometry interval are collected, and when the actual motion offset is less than the set odometry interval, n visual key frames starting from the time interval are recorded; a one-to-one mapping relationship between each visual key frame and the pose is constructed; Keyframe-based positioning step: Apply ORB descriptors to perform local 2D-3D matching, calculate the pose transformation matrix between the query image and the candidate keyframes, and obtain the pose information of the estimated query image; Re-localization step: Based on the posture information, the Scan2Map score is used to evaluate the localization to further determine whether to re-localize in the relocalization trigger mechanism; Posture tracking step: If the relocalization mechanism is not triggered, the robot uses the AMCL method to perform posture tracking according to a preset global posture.

2. The method according to claim 1, characterized in that The keyframe-based positioning step includes: Apply Gist global scene descriptor to quickly retrieve and track candidate poses during initialization and determine the KF of each keyframe i According to the preset rules, it is expressed as a Gist vector g i The average value of According to each key frame KF i The corresponding relationship between the current image and the key frames in the preset database is compared, and the key frame with the closest distance to the maximum value is determined as the candidate frame, and the position of the robot in the environment is estimated by the posture information of the map bound to the candidate frame; The local 3D points of the camera in the global frame are calculated through the RGB image and the depth image, and the position of the given point in the local and global coordinates is determined, which is used to calculate the transformation matrix between the query image and the proposed key frame to obtain more accurate pose information of the estimated query image.

3. The method according to claim 2, characterized in that The Gist global scene descriptor is applied to quickly retrieve and track candidate poses during the initialization process and determine the KF of each key frame. i According to the preset rules, it is expressed as a Gist vector g i The average value includes: Apply Gist global scene descriptor to quickly retrieve candidate poses during tracking initialization; Based on the context algorithm, global objects are identified and the key frame image in each key frame is divided into 4×4 blocks, where each block is represented by the average value of the filters in the [8, 8, 4] direction at each scale in the three color channels; Calculate a Gist vector of length 960 for each keyframe.

4. The method according to claim 2, characterized in that According to each key frame KF i The corresponding relationship between the current image and the Gist vector is compared, and the key frame with the closest distance to the maximum value is determined as the candidate frame. The position of the robot in the environment is estimated by the posture information of the map bound to the candidate frame, specifically including: According to each key frame KF i The corresponding relationship between the Gist vector and the current image is compared with the key frames in the preset database to measure the difference between the two feature vectors; Through the Gist feature vector g r and corresponding to their respective images g s The distance d rs , the Minkowski distance of a metric on a Euclidean space with order p is defined as: Calculate the Gist vector distance L2 of length k = 960 between each key frame in the database and the query image; If the distance d rs The key frame closest to the maximum value N is considered as a candidate frame, and the position of the robot in the environment is roughly estimated by the pose information of the map bound to the candidate frame.

5. The method according to claim 1, wherein The re-positioning step specifically includes: The visual and lidar information are integrated to refine the robot's pose in a maximum likelihood framework, where the robot's pose at time t is determined as: Among them, the probability represents the probability of predicting the robot's posture based on the image positioning model, Modeling the impact of keyframe clustering on continuous pose evaluation, m l is the laser radar plane map, is a lidar scan containing n individual measurements; After global coordinate reference optimization, the range reading is converted into a lidar map based on the robot pose, where the lidar endpoints The grid cell hit is expressed in m hit(k) express; When cell m hit(k) is occupied, the occupancy value of the cell is added to a voting score, where the likelihood of the sensor measurement is given by the final voting score: express; The keyframe clusters are updated based on the Scan2Map scores. If the new cluster has the highest score after Scan2Map reordering, it is confirmed that reordering has occurred, and the trigger mechanism makes the corresponding position of the candidate frame a candidate pose estimate.

6. The method according to claim 5, characterized in that The described approach integrates visual and lidar information to refine the robot's pose in a maximum likelihood framework, which previously also included: Based on whether there are new keyframe clusters after k-means clustering, it is decided whether to enter the next session of the relocalization trigger mechanism or continue tracking the robot's posture for a period of time; After calculating the distance between images in the process of calculating the similarity metric, the top candidate indexes of n key frames are obtained; The indexes of n candidate key frames are clustered using the k-means clustering method, and the center of the cluster is replaced by the index of the best matching candidate frame.

7. A global positioning device for realizing global positioning of a mobile robot in an indoor scene, characterized in that: The device comprises: The image representation module is used to represent environmental parameters with lidar data based on the grid method in the offline SLAM stage, obtain a visual information map containing a series of key frames and their Gist descriptors, determine the top n key frames in the Gist distance between the key frames to be queried and to be clustered, and construct a frame-pose mapping relationship, specifically including: in the offline SLAM stage, discretize the environment into two-dimensional square units, each unit is associated with a value between [0, 1], where the value is the probability of the unit being occupied; obtain a series of key frames through a preset RGB-D camera, and represent at least one group of key frames as a scene on the occupied grid map, and determine candidate poses that can be directly retrieved for tracking initialization; collect visual key frames including RGB images and depth images of the odometry interval, and when the actual motion offset is less than the set odometry interval, record n visual key frames starting from the time interval; and construct a one-to-one mapping relationship between each visual key frame and the pose; The first positioning module is used to apply the ORB descriptor to perform local 2D-3D matching, calculate the pose transformation matrix between the query image and the candidate keyframe, and obtain the pose information of the estimated query image; The second positioning module is used to perform positioning evaluation based on the posture information using Scan2Map score to further determine whether to perform positioning again in the re-localization trigger mechanism; The posture tracking module is used to perform posture tracking using the AMCL method based on a preset global posture if the relocalization mechanism is not triggered.

8. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 6 when executed by the processor.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images