Robot positioning method and device and computer readable storage medium

Abstract

The invention provides a robot positioning method and device and a computer readable storage medium, and the method comprises the steps: obtaining a first map corresponding to a target environment, and the first map comprises visual feature information and semantic information; generating a second map corresponding to the current position of the robot in the target environment, wherein the second map is a real-time local area map of the same type as the first map; and performing point cloud registration on boundary information of the second map and the first map based on the visual feature information and the semantic information to obtain position information of the robot. According to the embodiment of the invention, the positioning precision of the robot is improved.

Description

Technical Field

[0001] This application relates to the technical field of positioning and mapping, and particularly to a robot positioning method, device, and computer-readable storage medium. Background Art

[0002] Robot positioning is one of the key tasks in robot navigation and autonomous movement. For example, taking a lawn mowing robot as an example, through the positioning of the lawn mowing robot, it can autonomously plan a path and perform lawn mowing operations. In practical applications, the environments in which robots perform tasks are different. In complex environments, due to the influence of factors such as being easily blocked and changes in lighting, the positioning results are not accurate enough. Summary of the Invention

[0003] This application provides a robot positioning method, device, and computer-readable storage medium, aiming to improve the accuracy of robot positioning.

[0004] To achieve the above object, this application provides a robot positioning method, including:

[0005] Obtain a first map corresponding to a target environment, where the first map includes visual feature information and semantic information;

[0006] Generate a second map corresponding to the current position of the robot in the target environment, where the second map is a real-time local area map of the same type as the first map;

[0007] Based on the visual feature information and the semantic information, perform point cloud registration on the boundary information of the second map and the first map to obtain the position information of the robot.

[0008] In the robot positioning method according to an embodiment of this application, the obtaining of the first map corresponding to the target environment includes:

[0009] Obtain multiple frames of images of the target environment, and obtain target information based on the multiple frames of images;

[0010] Generate the first map according to the target information.

[0011] In the robot positioning method according to an embodiment of this application, the target information includes visual feature information, and the obtaining of the target information based on the multiple frames of images includes:

[0012] Extract key feature points from each frame of the multiple frames of images;

[0013] Obtain the visual feature information corresponding to the key feature points.

[0014] In the robot positioning method according to an embodiment of the present application, the target information includes key frame point clouds. Obtaining the target information based on the multi-frame images includes:

[0015] Select key frame images from the multi-frame images;

[0016] Based on the VIO visual-inertial odometry algorithm, perform feature point matching and three-dimensional reconstruction on the key frame images to obtain the key frame point clouds.

[0017] In the robot positioning method according to an embodiment of the present application, the target information includes semantic information. Obtaining the target information based on the multi-frame images includes:

[0018] Based on a semantic segmentation algorithm, perform semantic segmentation on each frame of the multi-frame images to obtain the semantic information.

[0019] Before performing semantic segmentation on each frame of the multi-frame images based on the semantic segmentation algorithm to obtain the semantic information in the robot positioning method according to an embodiment of the present application, it includes:

[0020] Perform loop closure detection on the current frame image and the key frame images before the current frame image;

[0021] When the current frame image and the key frame images before the current frame image form a loop, adjust the pose of the robot based on a loop closure correction algorithm.

[0022] In the robot positioning method according to an embodiment of the present application, the target information includes visual feature information, key frame point clouds, and semantic information. Generating the first map according to the target information includes:

[0023] Fuse the visual feature information, the semantic information, and the key frame point clouds to generate the first map.

[0024] In the robot positioning method according to an embodiment of the present application, performing point cloud registration on the boundary information between the second map and the first map based on the visual feature information and the semantic information to obtain the position information of the robot includes:

[0025] Based on the visual feature information and the semantic information, perform an initial alignment of the second map and the first map;

[0026] Based on the ICP iterative closest point algorithm, perform point cloud registration on the boundary information between the second map and the first map to obtain the position information of the robot; wherein, the semantic information corresponding to the matched boundary point pairs obtained by registration is the same.

[0027] In addition, to achieve the above object, the present application further provides a robot positioning device, including:

[0028] A first map acquisition module, configured to acquire a first map corresponding to a target environment, where the first map includes visual feature information and semantic information;

[0029] A second map acquisition module, configured to generate a second map corresponding to the current position of the robot in the target environment, where the second map is a real-time local area map of the same type as the first map;

[0030] A robot positioning module, configured to perform point cloud registration on the boundary information between the second map and the first map based on the visual feature information and the semantic information to obtain the position information of the robot.

[0031] In addition, to achieve the above object, the present application further provides a robot, where the robot includes a fuselage, a driving mechanism, a processor, and a memory; wherein, the driving mechanism is arranged on the fuselage and is configured to drive the robot to move; the memory stores a computer program executable by the processor, and when the computer program is executed by the processor, the steps of the robot positioning method as described above are implemented.

[0032] In addition, to achieve the above object, the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the robot positioning method as described above.

[0033] The robot positioning method, device, and computer-readable storage medium provided by the embodiments of the present application acquire a first map corresponding to a target environment, where the first map includes visual feature information and semantic information, generate a second map corresponding to the current position of the robot in the target environment, where the second map is a real-time local area map of the same type as the first map, perform point cloud registration on the boundary information between the second map and the first map based on the visual feature information and the semantic information to obtain the position information of the robot. Since the map information is rich, different boundary information can be recognized, and combined with point cloud registration, the accuracy of robot positioning is effectively improved.

[0034] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. Description of the Drawings

[0035] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0036] Figure 1 It is a schematic flowchart of a robot positioning method provided by an embodiment of the present application;

[0037] Figure 2 It is a schematic flowchart of a method for obtaining a first map corresponding to a target environment provided by an embodiment of the present application;

[0038] Figure 3 It is a schematic flowchart of a method for obtaining multiple frames of images of the target environment and obtaining target information based on the multiple frames of images provided by an embodiment of the present application;

[0039] Figure 4 It is a schematic flowchart of another method for obtaining multiple frames of images of the target environment and obtaining target information based on the multiple frames of images provided by an embodiment of the present application;

[0040] Figure 5 It is a schematic flowchart of a closed-loop detection and closed-loop correction provided by an embodiment of the present application;

[0041] Figure 6 It is a schematic flowchart of a point cloud registration of the boundary information between the second map and the first map based on the visual feature information and the semantic information provided by an embodiment of the present application;

[0042] Figure 7 It is a schematic flowchart of a lawn mowing robot positioning provided by an embodiment of the present application;

[0043] Figure 8 It is a schematic block diagram of a robot positioning device provided by an embodiment of the present application;

[0044] Figure 9 It is a schematic structural diagram of a robot provided by an embodiment of the present application. Detailed implementation manners

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0046] The flowcharts shown in the accompanying drawings are only illustrative examples, not necessarily including all content and operations / steps, nor necessarily executed in the order described. For example, some operations / steps can also be decomposed, combined, or partially merged, so the actual execution order may change according to the actual situation.

[0047] It should be understood that the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. As used in the specification of this application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0048] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0049] Embodiments of this application provide a robot positioning method, apparatus, and computer-readable storage medium for improving the accuracy of robot positioning.

[0050] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of the robot positioning method provided by an embodiment of this application. This method can be applied to a robot positioning device or other devices. The application scenarios of this method are not limited in this application.

[0051] As Figure 1 shown, the robot positioning method specifically includes steps S101 to S103.

[0052] S101. Obtain a first map corresponding to the target environment, where the first map includes visual feature information and semantic information.

[0053] Among them, the target environment can be the moving area of the robot. For example, taking a lawn mowing robot as an example, the target environment is the lawn mowing operation area. In this application, the robot includes but is not limited to lawn mowing robots, autonomous driving vehicles, indoor mobile robots, patrol robots, etc. It can be understood that different types of robots have different corresponding target environments.

[0054] What differentiates the first map from a normal map is that a normal map is usually an image map containing location information, while the first map is a map containing various types of information, including but not limited to geometric information, visual feature information, semantic information, etc. Among them, the visual feature information can include feature descriptors corresponding to feature parameters such as color, shape, texture, brightness, contrast, etc., and the feature descriptors usually include information such as the location, direction, and scale of feature points. Geometric information is composed of elements such as points, lines, and polygons, and these elements are used to represent geometric relationships such as positions, boundaries, and directions in space. For example, the positions and shapes of geographical entities such as grasslands, buildings, road surfaces, rivers, and shrubs can be expressed through geometric information. Semantic information refers to the actual meaning or function represented by the elements on the map, and each element on the map is assigned one or more semantic labels, which are used to describe the objects or concepts represented by the elements. For example, a red rectangular area may be labeled as a building, and a green circular area may be labeled as a grassland. Semantic information includes object category information, boundary information, etc., and the boundary information is used to represent the boundaries between different semantic regions.

[0055] It should be noted that the first map can be generated during robot positioning, or it can also be generated and saved in advance, and directly called when performing robot positioning. In this application, the method for obtaining the first map is not limited.

[0056] In some embodiments, as Figure 2 shown, step S101 may include sub-step S1011 and sub-step S1012.

[0057] S1011. Obtain multiple frames of images of the target environment, and obtain target information based on the multiple frames of images;

[0058] S1012. Generate the first map according to the target information.

[0059] Exemplarily, a robot is equipped with imaging devices such as cameras, and multiple frames of images of the target environment are obtained by shooting with the imaging devices, and then relevant processing is performed on the obtained images to obtain target information. Among them, the target information includes but not limited to visual feature information, key frame point clouds, semantic information, etc.

[0060] In some embodiments, as Figure 3 shown, step S1011 may include sub-step S10111 and sub-step S10112.

[0061] S10111. Extract key feature points from each frame of the multiple frames of images;

[0062] S10112. Obtain the visual feature information corresponding to the key feature points.

[0063] Exemplarily, a deep learning visual feature extraction algorithm is used to extract key feature points in the image and calculate feature descriptors. For example, the SuperPoint model is used to extract key feature points from the image. Another example is using the SuperGlue model to calculate feature descriptors. Among them, the SuperPoint model is a feature point detection and descriptor generation model based on deep learning, and the SuperGlue model is a deep learning-based model mainly used for natural language processing tasks. It should be noted that other deep learning visual feature extraction algorithms can also be used to obtain visual feature information, which is not limited in this application.

[0064] In some embodiments, as Figure 4 shown, step S1011 may include sub-step S10113 and sub-step S10114.

[0065] S10113. Select key frame images from the multi-frame images;

[0066] S10114. Perform feature point matching and 3D reconstruction on the key frame images based on the VIO algorithm to obtain the key frame point cloud.

[0067] In VIO (Visual-Inertial Odometry), a key frame image refers to an image that contains enough valid key feature points and can represent the motion state of the robot within a certain period of time.

[0068] For the selection of key frame images, appropriate key frame images can be selected according to factors such as the motion state of the robot, the number and quality of key feature points. Common selection strategies include selection based on time interval, selection based on the number of key feature points, selection based on the motion change of the robot, etc.

[0069] After determining the key frame images, between adjacent key frame images, by calculating the similarity between feature descriptors, such as Euclidean distance, Hamming distance, etc., feature point matching is performed to find matching feature point pairs. Then, according to the matching feature point pairs, combined with the inertial information of the robot such as accelerometer, gyroscope data, etc., the robot pose is estimated, and based on the robot pose, the pixel points in the key frame images are projected into the three-dimensional space for 3D reconstruction to generate the key frame point cloud. It should be noted that the algorithm for generating the key frame point cloud is not limited to the VIO algorithm, which is not limited in this application.

[0070] In some embodiments, obtaining the target information based on the multiple frames of the images includes: performing semantic segmentation on each frame of the multiple frames of images based on a semantic segmentation algorithm to obtain the semantic information.

[0071] For example, taking the target environment as the mowing operation area as an example, using the DeepLab model to perform semantic segmentation on the image to identify and obtain relevant semantic information such as grassland, road surface, water surface, buildings, shrubs, etc. in the target environment. Among them, the DeepLab model is a deep learning model for image segmentation and performs well in the image semantic segmentation task. It should be noted that other semantic segmentation algorithms can also be used to perform semantic segmentation on the image, which is not limited in this application.

[0072] Exemplarily, the semantic information is processed hierarchically. For example, in the first level, basic semantic extraction is performed to identify large-scale semantic regions, and in the second level, specific processing is performed on different types of semantic regions extracted in the first level.

[0073] For example, still taking the target environment as the mowing operation area as an example, in the first-level processing, a semantic segmentation model is first used to extract the corresponding semantic regions of grassland, road surface, water surface, buildings, shrubs, etc. in the image.

[0074] Then, specific second-level processing is performed on each semantic region respectively. For example:

[0075] For the grassland semantic region, the features of this semantic region are not significantly stable and are easily affected by environmental changes, which is not conducive to the positioning of the mowing robot. Therefore, the feature points of the grassland semantic region are removed to improve the positioning accuracy.

[0076] For the building semantic region, a straight-line extraction algorithm, such as the FSD (Fast Segment Detection) straight-line extraction algorithm, is used to extract the building structure lines and construct stable geometric features for tracking and matching.

[0077] For the road surface semantic region, only the texture feature points with larger gradients are extracted to avoid the problem of sparse feature points caused by a smooth road surface.

[0078] For the shrub semantic region, after using the data of the shrub semantic region to enhance the training of the SuperPoint model, the trained SuperPoint model is used to extract more stable feature points to enhance the matching ability.

[0079] In some embodiments, before performing semantic segmentation on the image using a semantic segmentation algorithm to obtain the semantic information, as Figure 5 shown, step S101 may include sub-step S1013 and sub-step S1014.

[0080] S1013. Perform closed-loop detection on the current frame image and the key frame image before the current frame image;

[0081] S1014. When a closed loop is formed between the current frame image and the key frame image before the current frame image, adjust the pose of the robot based on the closed-loop correction algorithm.

[0082] Use the feature matching algorithm to perform feature matching between the current frame image and the key frame image before the current frame image. Feature point matching can be performed by calculating the similarity between feature descriptors, such as Euclidean distance, Hamming distance, etc. For example, the SuperGlue model can be used to perform feature matching between the current frame image and the key frame image before the current frame image.

[0083] Verify the matching result to determine whether a valid closed loop is formed between the current frame image and the key frame image before the current frame image. For example, if the number of matched feature point pairs exceeds a certain threshold and other closed-loop detection conditions are met, such as PNP (Perspective-n-Point) geometric consistency, etc., it is determined that a closed loop is formed between the current frame image and the key frame image before the current frame image.

[0084] When a closed loop is formed between the current frame image and the key frame image before the current frame image, calculate the deviation between the robot pose corresponding to the current frame image and the robot pose corresponding to the key frame image before the current frame image based on the closed-loop correction algorithm. The obtained deviation is the pose correction amount, and the pose of the robot is adjusted based on the pose correction amount. For example, the Bundle Adjustment method is used to adjust the pose of the robot. Bundle Adjustment is an optimization algorithm widely used in the fields of computer vision and robotics, especially playing a key role in 3D reconstruction and SLAM (Simultaneous Localization and Mapping). It should be noted that different closed-loop correction algorithms can be used to adjust the pose of the robot, which is not limited in this application.

[0085] During the long-term movement of the robot, the pose of the robot will gradually deviate from the true value, forming cumulative errors. Adjusting the pose of the robot based on the closed-loop correction algorithm can effectively eliminate these cumulative errors, thereby improving the positioning accuracy.

[0086] In some embodiments, generating the first map according to the target information includes: fusing the visual feature information, the semantic information, and the key frame point cloud to generate the first map.

[0087] Combine semantic information, visual feature information, and key-frame point clouds to construct a complete first map. For example, fuse semantic information and visual feature information to form image features containing semantic labels, and construct the first map based on the fused image features and key-frame point clouds. It can be understood that the first map has richer information and can identify different boundary information.

[0088] Exemplarily, the semantic information is obtained through hierarchical processing. For example, in the above-listed examples, at the first level, basic semantic extraction is performed on the image to identify the large-scale semantic regions corresponding to the image, and at the second level, specific processing is performed on the different categories of semantic regions extracted at the first level. Compared with the semantic information obtained only through semantic segmentation, the semantic information obtained based on hierarchical processing is more reliable and effective, and can improve the accuracy of positioning.

[0089] S102. Generate a second map corresponding to the current position of the robot in the target environment, where the second map is a real-time local area map of the same type as the first map.

[0090] When the robot repeatedly runs to the position on the first map, perform closed-loop detection. Through feature matching algorithms, perform feature matching between the current frame image and the key-frame image, as well as perform depth feature matching and PNP geometric consistency verification to verify the closed-loop relationship. For details, reference can be made to the description in the previous embodiments, and it will not be elaborated here.

[0091] Even if the closed-loop detection is successful, there may still be matching errors, which may lead to inaccurate positioning. Therefore, based on the current frame image, obtain the corresponding semantic information in real time and construct a real-time local area second map.

[0092] S103. Based on the visual feature information and the semantic information, perform point cloud registration on the boundary information between the second map and the first map to obtain the position information of the robot.

[0093] Perform point cloud registration on the second map generated in real time and the first map. Generally speaking, it is to align the second map with the first map so as to determine the precise position information of the robot in the target environment and achieve robot positioning.

[0094] In some embodiments, as Figure 6 shown, step S103 may include sub-step S1031 and sub-step S1032.

[0095] S1031. Based on the visual feature information and the semantic information, perform an initial alignment on the second map and the first map;

[0096] S1032. Perform point cloud registration on the boundary information of the second map and the first map based on the ICP (Iterative Closest Point) algorithm to obtain the position information of the robot; wherein, the semantic information corresponding to the matched boundary point pairs obtained by registration is the same.

[0097] For example, use visual feature information for feature matching, perform an initial alignment of the second map and the first map once, and use semantic information for semantic boundary alignment of the second map and the first map to further optimize the initial alignment of the second map and the first map.

[0098] The ICP algorithm is an iterative algorithm used to find the best rigid body transformation (rotation and translation) in point cloud data. Exemplarily, the objective function formula corresponding to the ICP algorithm iteration is as follows:

[0099]

[0100] wherein, T is a transformation matrix containing rotation and translation, p i and p i ' are the corresponding points in the source point cloud and the target point cloud respectively, that is, the corresponding points in the first map and the second map, n is the number of corresponding points, and the objective function aims to find an optimal rigid body transformation matrix T k+1 , to minimize the sum of the squares of the Euclidean distances between p i and T·p i '. During the point cloud registration of the second map and the first map, the T matrix is used to perform an initial alignment of the second map and the first map.

[0101] In each iteration of the ICP algorithm, an optimal transformation matrix T will be calculated according to the objective function k+1 , and then the source point cloud will be transformed to be closer to the target point cloud. Then, search for the closest corresponding points in the target point cloud to each point in the transformed source point cloud for the next iteration. This process will be repeated until the convergence condition is met.

[0102] During the process of performing boundary point cloud registration on the boundary information of the second map and the first map based on the ICP algorithm, determine the normal vectors of each pair of registered boundary points, remove the boundary points with a large difference in normal vector directions, calculate the residuals of each pair of registered boundary points to the line / surface, set an error threshold to exclude misregistered boundary points, and perform boundary consistency checks on each pair of boundary points to ensure that the semantic information corresponding to the matched boundary point pairs is the same, that is, the matched boundary point pairs must belong to the same semantic category.

[0103] For example, still taking the target environment as the mowing operation area as an example, if one of the pair of registered boundary points belongs to the road boundary point and the other boundary point also belongs to the road boundary point, that is, they have the same semantic information, then this pair of boundary points is a matching boundary point pair. If one of the other pair of registered boundary points belongs to the grassland boundary point and the other boundary point belongs to the building boundary point, that is, they have different semantic information, then this pair of boundary points is not a matching boundary point pair.

[0104] Based on the ICP algorithm, the second map is registered with the first map to align the boundary information on the second map and the first map as much as possible, so as to determine the accurate position information of the robot in the target environment and achieve precise positioning of the robot.

[0105] Exemplarily, for the first map, if a certain boundary point is matched multiple times but the error is always large, it is considered that this boundary point may be invalid due to environmental changes, and this boundary point is removed to perform a local update on the first map. For the boundary points newly detected in the current frame image, if it is confirmed that this boundary point is stable (without significant drift) through multiple subsequent frame images, then this boundary point is added to the first map to perform a local update on the first map.

[0106] Taking a lawn mowing robot as an example below, as Figure 7 shown, the process of lawn mowing robot positioning is as follows:

[0107] The lawn mowing robot runs in the mowing operation area, extracts visual features based on the collected images of the operation area, performs feature point matching and 3D reconstruction based on the VIO algorithm to construct the VIO key frame point cloud, and performs loop detection on the current frame image and the previous key frame images. When a loop is formed between the current frame image and the key frame images, the pose of the lawn mowing robot is adjusted based on the loop correction algorithm, and the semantic segmentation algorithm is used to perform semantic segmentation on the image to obtain the corresponding semantic information. Based on the visual feature information, the VIO key frame point cloud, and the semantic information, the first map corresponding to the mowing operation area is constructed. When the lawn mowing robot runs to the position of the first map repeatedly, loop detection is performed, and depth feature matching and PNP geometric consistency verification are performed to verify the loop relationship. If the loop detection is successful, the second map is constructed, and the boundary information of the second map and the first map is subjected to ICP point cloud registration to achieve precise positioning of the lawn mowing robot.

[0108] In the above embodiments, by obtaining a first map corresponding to the target environment, the first map includes visual feature information and semantic information, generating a second map corresponding to the current location of the robot in the target environment, the second map being a real-time local area map of the same type as the first map, performing point cloud registration on the boundary information between the second map and the first map based on the visual feature information and the semantic information to obtain the position information of the robot. Since the map information is rich, different boundary information can be recognized, and combined with point cloud registration, the accuracy and robustness of robot positioning are effectively improved, and the robot can work stably in a complex environment.

[0109] Please refer to Figure 8 , Figure 8 which is a schematic block diagram of a robot positioning device provided by an embodiment of the present application. The robot positioning device can be configured in a robot to perform the aforementioned robot positioning method.

[0110] As Figure 8 shown, the robot positioning device 300 includes: a first map acquisition module 301, a second map acquisition module 302, and a robot positioning module 303.

[0111] The first map acquisition module 301 is configured to obtain a first map corresponding to the target environment, and the first map includes visual feature information and semantic information;

[0112] The second map acquisition module 302 is configured to generate a second map corresponding to the current location of the robot in the target environment, and the second map is a real-time local area map of the same type as the first map;

[0113] The robot positioning module 303 is configured to perform point cloud registration on the boundary information between the second map and the first map based on the visual feature information and the semantic information to obtain the position information of the robot.

[0114] In some embodiments, the first map acquisition module 301 is further configured to:

[0115] Obtain multiple frames of images of the target environment, and obtain target information based on the multiple frames of images;

[0116] Generate the first map according to the target information.

[0117] In some embodiments, the target information includes visual feature information, and the first map acquisition module 301 is further configured to:

[0118] Extract key feature points from each frame of the multiple frames of images;

[0119] Obtain the visual feature information corresponding to the key feature points.

[0120] In some embodiments, the target information includes key-frame point clouds, and the first map acquisition module 301 is further configured to:

[0121] Select key-frame images from the multiple frames of images;

[0122] Perform feature point matching and three-dimensional reconstruction on the key-frame images based on the VIO visual-inertial odometry algorithm to obtain the key-frame point clouds.

[0123] In some embodiments, the target information includes semantic information, and the first map acquisition module 301 is further configured to:

[0124] Perform semantic segmentation on each frame of the multiple frames of images based on a semantic segmentation algorithm to obtain the semantic information.

[0125] In some embodiments, the first map acquisition module 301 is further configured to:

[0126] Perform loop closure detection on the current frame image and the key-frame images before the current frame image;

[0127] When the current frame image and the key-frame images before the current frame image form a loop closure, adjust the pose of the robot based on a loop closure correction algorithm.

[0128] In some embodiments, the target information includes visual feature information, key-frame point clouds, and semantic information, and the first map acquisition module 301 is further configured to:

[0129] Fuse the visual feature information, the semantic information, and the key-frame point clouds to generate the first map.

[0130] In some embodiments, the robot positioning module 303 is further configured to:

[0131] Perform initial alignment of the second map and the first map based on the visual feature information and the semantic information;

[0132] Perform point cloud registration on the boundary information of the second map and the first map based on the ICP iterative closest point algorithm to obtain the position information of the robot; wherein, the semantic information corresponding to the paired matching boundary points obtained by registration is the same.

[0133] The robot positioning device 300 can execute the robot positioning method provided by the embodiments of the present application, and thus, can achieve the beneficial effects that the robot positioning method provided by the embodiments of the present application can achieve. For details, please refer to the previous embodiments and will not be elaborated herein.

[0134] Please refer to Figure 9 , Figure 9The following is a schematic structural diagram of a robot provided by an embodiment of the present application. As Figure 9 shown, the robot 1000 includes a fuselage 100, a driving mechanism (not shown in the figure), a processor (not shown in the figure), and a memory (not shown in the figure). Among them, the driving mechanism is provided on the fuselage 100. The driving mechanism includes, but is not limited to, a servo motor, a stepper motor, etc. The driving mechanism is used to drive the robot 1000 to move. For example, as Figure 9 shown, the robot 1000 further includes wheels 200. The driving mechanism is used to drive the wheels 200 to rotate. When the wheels 200 rotate, the robot 1000 is driven to move.

[0135] The processor can be a microcontroller unit (MCU), a central processing unit (CPU), a digital signal processor (DSP), etc.

[0136] The memory can be a Flash chip, a read-only memory (ROM), a magnetic disk, an optical disc, a USB flash drive, a mobile hard disk, etc. Various computer programs for the processor to execute are stored in the memory.

[0137] Among them, the processor is used to run the computer program stored in the memory and implement the robot positioning method provided by the embodiment of the present application when executing the computer program.

[0138] The robot 1000 can execute the robot positioning method provided by the embodiment of the present application. Therefore, the beneficial effects that can be achieved by the robot positioning method provided by the embodiment of the present application can be realized. For details, see the previous embodiments and will not be elaborated here.

[0139] The embodiment of the present application further provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the robot positioning method as described above are implemented.

[0140] Among them, the computer-readable storage medium can be an internal storage unit of the robot positioning device, the robot, or the computer device described in the foregoing embodiments, such as the hard disk or memory of the robot positioning device, the robot, or the computer device. The computer-readable storage medium can also be an external storage device of the robot positioning device, the robot, or the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital card (SD card), a flash card, etc. equipped on the robot positioning device, the robot, or the computer device.

[0141] Since the computer program stored in the storage medium can execute any one of the robot positioning methods provided by the embodiments of the present application, the beneficial effects achievable by any one of the robot positioning methods provided by the embodiments of the present application can be realized. For details, refer to the previous embodiments and will not be elaborated herein.

[0142] It should be noted that in this text, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or system including that element.

[0143] As described above, the above are only the specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should be covered within the protection scope of the present application.

Claims

1. A robot positioning method, characterized in that, Including: Obtain a first map corresponding to the target environment, where the first map includes visual feature information and semantic information; Generate a second map corresponding to the current location of the robot in the target environment, where the second map is a real-time local area map of the same type as the first map; Based on the visual feature information and the semantic information, perform point cloud registration on the boundary information between the second map and the first map to obtain the position information of the robot.

2. The robot positioning method according to claim 1, wherein The obtaining of the first map corresponding to the target environment includes: Obtain multiple frames of images of the target environment, and obtain target information based on the multiple frames of images; Generate the first map according to the target information.

3. The robot positioning method according to claim 2, wherein The target information includes visual feature information. The obtaining of the target information based on the multiple frames of images includes: Extract key feature points from each frame of the multiple frames of images; Obtain the visual feature information corresponding to the key feature points.

4. The robot positioning method according to claim 2, wherein The target information includes key frame point clouds. The obtaining of the target information based on the multiple frames of images includes: Select key frame images from the multiple frames of images; Based on the VIO visual inertial odometry algorithm, perform feature point matching and three-dimensional reconstruction on the key frame images to obtain the key frame point clouds.

5. The robot positioning method according to claim 2, characterized in that, The target information includes semantic information. The obtaining of the target information based on the multiple frames of images includes: Based on a semantic segmentation algorithm, perform semantic segmentation on each frame of the multiple frames of images to obtain the semantic information.

6. The robot positioning method according to claim 5, wherein Before performing semantic segmentation on each frame of the multiple frames of images based on the semantic segmentation algorithm to obtain the semantic information, it includes: Perform loop closure detection on the current frame image and the key frame image before the current frame image; When the current frame image and the key frame image before the current frame image form a loop, adjust the pose of the robot based on a loop closure correction algorithm.

7. The robot positioning method according to claim 2, characterized in that, The target information includes visual feature information, key frame point clouds, and semantic information. The generating of the first map according to the target information includes: Fuse the visual feature information, the semantic information, and the key frame point clouds to generate the first map.

8. The robot positioning method according to claim 1, characterized in that, The performing of point cloud registration on the boundary information between the second map and the first map based on the visual feature information and the semantic information to obtain the position information of the robot includes: Based on the visual feature information and the semantic information, perform an initial alignment between the second map and the first map; Based on the ICP iterative closest point algorithm, perform point cloud registration on the boundary information between the second map and the first map to obtain the position information of the robot; where the semantic information corresponding to the matched boundary point pairs obtained by the registration is the same.

9. A robot positioning device, characterized in that, Including: A first map acquisition module, configured to obtain a first map corresponding to the target environment, where the first map includes visual feature information and semantic information; A second map acquisition module, configured to generate a second map corresponding to the current location of the robot in the target environment, where the second map is a real-time local area map of the same type as the first map; A robot positioning module, which is used to perform point cloud registration on the boundary information between the second map and the first map based on the visual feature information and the semantic information, so as to obtain the position information of the robot.

10. A robot, characterized in that, The robot includes a fuselage, a driving mechanism, a processor, and a memory. Among them, the driving mechanism is arranged on the fuselage and is used to drive the robot to move. The memory stores a computer program that can be executed by the processor. When the computer program is executed by the processor, the steps of the robot positioning method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the robot positioning method according to any one of claims 1 to 8.

Images