Robot repositioning method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a robot repositioning method, electronic equipment and a computer readable storage medium. The robot repositioning method comprises the following steps: acquiring a texture map of a target scene; the texture map comprises a plurality of key frames collected by the robot, and the key frames are matched with poses and descriptors of triangles obtained based on texture feature points; a to-be-matched frame collected by the robot is obtained, a to-be-matched descriptor is determined, and all reference descriptors matched with the to-be-matched descriptor and candidate key frames associated with the reference descriptors are determined in the descriptors corresponding to all the key frames; the descriptor to be matched and each reference descriptor form a matching pair; based on each matching pair, determining a relative pose of the to-be-matched frame and the corresponding candidate key frame, and screening from all the relative poses by using all the matching pairs to obtain a reference relative pose; and based on the reference relative pose and the pose matched with the candidate key frame corresponding to the reference relative pose, obtaining the relocation pose of the robot. According to the scheme, the repositioning accuracy and convenience of the robot can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of robot positioning, and in particular, to a robot relocalization method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the increasingly wide application of robots, how to accurately relocalize a robot in a target scenario has become an important research direction. In the prior art, usually, QR codes are set in the target scenario as identifiers to assist the robot in relocalization. However, it is inconvenient to set QR codes in a relatively large target scenario, and when the QR codes are damaged, contaminated, or partially blocked, the recognition will be incorrect or even invalid. In view of this, how to improve the accuracy and convenience of robot relocalization has become an urgent problem to be solved. Summary of the Invention

[0003] The main technical problem to be solved by the present application is to provide a robot relocalization method, an electronic device, and a computer-readable storage medium, which can improve the accuracy and convenience of robot relocalization.

[0004] To solve the above technical problem, a first aspect of the present application provides a robot relocalization method, including: obtaining a texture map of a target scenario; wherein, the texture map includes a plurality of key frames collected by a robot, and the key frames are matched with poses and descriptors of triangles obtained based on texture feature points in the key frames; obtaining a to-be-matched frame collected by the robot and determining to-be-matched descriptors in the to-be-matched frame, and among the descriptors corresponding to all the key frames, determining all reference descriptors that match the to-be-matched descriptors and candidate key frames associated with the reference descriptors; wherein, each to-be-matched descriptor and each reference descriptor form a matching pair; based on each matching pair, determining a relative pose between the to-be-matched frame and the corresponding candidate key frame, and screening out a reference relative pose from all the relative poses by using all the matching pairs; based on the reference relative pose and the pose matched with the corresponding candidate key frame, obtaining a relocalization pose when the robot collects the to-be-matched frame.

[0005] To solve the above technical problem, a second aspect of the present application provides an electronic device, which includes: a memory and a processor coupled to each other, wherein, the memory stores program data, and the processor calls the program data to execute the method described in the first aspect above.

[0006] To solve the above technical problem, a third aspect of the present application provides a computer-readable storage medium, on which program data is stored, and when the program data is executed by a processor, the method described in the first aspect above is implemented.

[0007] In the above solution, a texture map matching the target scenario is obtained. The texture map includes multiple key frames collected by the robot. Each key frame is matched with a pose, and each key frame includes texture feature points and descriptors of triangles set based on the texture feature points, so as to construct a prior texture map and set the pose and descriptors that can be referenced. A to-be-matched frame collected by the robot from the target scenario is obtained, and the to-be-matched descriptors in the to-be-matched frame are determined. The to-be-matched descriptors are matched with the descriptors corresponding to all key frames, and all descriptors that match the to-be-matched descriptors are determined as reference descriptors. The key frames associated with the reference descriptors are determined as candidate key frames, so as to quickly find the reference descriptors through the matching of triangle descriptors, improve the matching efficiency and the accuracy of candidate key frames. Among them, the to-be-matched descriptors and each reference descriptor form a matching pair respectively. Based on the to-be-matched descriptors and reference descriptors in each matching pair, the relative pose between the to-be-matched frame and the candidate key frame is determined, and all matching pairs are used to filter the relative poses obtained based on each matching pair, so as to obtain the optimal reference relative pose and improve the accuracy of the relative pose. Based on the reference relative pose and the pose matched with the corresponding candidate key frame, the pose of the robot when collecting the to-be-matched frame is solved, and the relocalization pose of the robot when collecting the to-be-matched frame is obtained. Thus, through descriptor matching and filtering, and using the reference relative pose for solution, the robot relocalization can be completed, and the convenience of robot relocalization is improved. Description of the Drawings

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. Among them:

[0009] Figure 1 is a schematic flowchart of an embodiment of the robot relocalization method of the present application;

[0010] Figure 2 is a schematic flowchart of another embodiment of the robot relocalization method of the present application;

[0011] Figure 3 is a schematic structural diagram of an embodiment of the electronic device of the present application;

[0012] Figure 4 is a schematic structural diagram of an embodiment of the computer-readable storage medium of the present application. Detailed Embodiments

[0013] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments, and adaptive combinations can be made between different embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0014] The terms "system" and "network" are often used interchangeably in this document. The term "and / or" in this document is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects before and after. In addition, "multiple" in this document means two or more than two.

[0015] The robot relocalization method provided in the present application is used to relocalize a robot in a target scene, and its corresponding execution entity is a processing unit capable of data processing. Among them, the processing unit is integrated into the robot or independent of the robot and interacts with the robot.

[0016] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of an implementation manner of the robot relocalization method of the present application. The method includes:

[0017] S101: Obtain a texture map of the target scene; wherein, the texture map includes multiple key frames collected by the robot, and each key frame is matched with a pose and a descriptor of a triangle obtained based on the texture feature points in the key frame.

[0018] Specifically, obtain a texture map matching the target scene. Among them, the texture map includes multiple key frames collected by the robot. Each key frame is matched with a pose, and each key frame includes texture feature points and a descriptor of a triangle set based on the texture feature points, so as to construct a prior texture map and set the pose and descriptor that can be referred to.

[0019] It can be understood that the texture map is pre-constructed for the target scene. After the robot collects key frames in the target scene, the key frames are analyzed and processed to construct the texture map. When the texture map is constructed, it can be called.

[0020] In some implementation scenarios, a robot sequentially collects multiple key frames in a target scenario, determines the pose of a camera on the robot when collecting the key frames, and binds the pose to the corresponding key frames. Feature extraction is performed on the key frames to determine all texture feature points in the key frames. The texture feature points in the key frames are traversed to obtain three texture feature points that satisfy the distance condition between each other, thereby constructing a descriptor of a triangle. The texture feature points of all key frames and their matching descriptors are projected onto the coordinate system corresponding to the target scenario to obtain a texture map.

[0021] In some implementation scenarios, a robot sequentially collects multiple key frames in a target scenario, determines the pose of a sensor on the robot when the camera collects the key frames, and binds the pose to the corresponding key frames. Feature extraction is performed on the key frames to determine all texture feature points in the key frames. Taking each texture feature point as the center, other texture feature points within a preset distance range from the corresponding texture feature point are obtained to construct a descriptor of a triangle. The texture feature points of all key frames and their matching descriptors are projected onto the coordinate system corresponding to the target scenario to obtain a texture map.

[0022] It should be noted that the robot sequentially collects multiple key frames, the distance between the key frames collected at adjacent time points is less than a preset interval threshold, and all key frames cover an area exceeding an area threshold in the target scenario. Preferably, all areas of the target scenario are covered.

[0023] Optionally, the descriptor of the triangle includes three vertices, and the pixel distance between any two vertices is between a lower limit value and an upper limit value of the pixel distance.

[0024] S102: Obtain a to-be-matched frame collected by the robot and determine the to-be-matched descriptor in the to-be-matched frame. Among the descriptors corresponding to all key frames, determine all reference descriptors that match the to-be-matched descriptor and the candidate key frames associated with the reference descriptors; wherein, the to-be-matched descriptor and each reference descriptor form a matching pair.

[0025] Specifically, obtain the to-be-matched frame collected by the robot from the target scenario and determine the to-be-matched descriptor in the to-be-matched frame. Match the to-be-matched descriptor with the descriptors corresponding to all key frames, determine all descriptors that match the to-be-matched descriptor as reference descriptors, and determine the key frames associated with the reference descriptors as candidate key frames.

[0026] It can be understood that the reference descriptors are quickly found through the matching of the descriptors of the triangle, improving the matching efficiency and the accuracy of the candidate key frames. Among them, the to-be-matched descriptor and each reference descriptor form a matching pair.

[0027] In some implementation scenarios, the descriptors of a triangle correspond to the side length information of three sides and the angle information of three angles. The side length information and the angle information are converted into key - value pairs and stored in a hash list. Among them, similar side length information and angle information are mapped to the same key - value pair. Thus, each key - value pair in the hash list corresponds to at least one descriptor. Obtain the frame to be matched and extract texture feature points, thereby constructing the descriptor to be matched of the triangle, determine the key - value to be matched of the descriptor to be matched, match the key - value to be matched with the key - values in the hash list, obtain the matched key - values and extract the descriptors therein as reference descriptors, and determine the key frames associated with the reference descriptors as candidate key frames.

[0028] In some implementation scenarios, the descriptors in all key frames correspond to reference graphic features. Obtain the frame to be matched and extract texture feature points, thereby constructing the descriptor to be matched of the triangle, determine the to - be - matched graphic features of the descriptor to be matched, obtain the similarity between the to - be - matched image features and each reference graphic feature, take the descriptors with similarity exceeding the similarity threshold as reference descriptors, and determine the key frames associated with the reference descriptors as candidate key frames.

[0029] It can be understood that the number of side lengths and the graphic features corresponding to the descriptors of a triangle are unique. The matching efficiency between descriptors is better than that based on texture feature points for matching, and the processing resource consumption required for matching between descriptors is less, which is better than that based on the image features corresponding to the entire image for matching.

[0030] S103: Based on each matching pair, determine the relative pose between the frame to be matched and the corresponding candidate key frame, and use all matching pairs to filter out the reference relative pose from all relative poses.

[0031] Specifically, based on the descriptor to be matched and the reference descriptor in each matching pair, determine the relative pose between the frame to be matched and the candidate key frame, and use all matching pairs to filter the relative poses obtained based on each matching pair, so as to obtain the optimal reference relative pose and improve the accuracy of the relative pose.

[0032] It should be noted that each matching pair includes a descriptor to be matched and a reference descriptor. By solving based on the vertices of the descriptor to be matched and the reference descriptor, the relative pose between the frame to be matched and the corresponding candidate key frame can be determined. Among them, the algorithms used to solve the relative pose include, but are not limited to, the Iterative Closest Point (ICP) algorithm and the pose estimation algorithm based on deep learning.

[0033] In some implementation scenarios, perform ICP registration on the vertices of the descriptor to be matched and the reference descriptor in the matching pair to obtain the relative pose between the frame to be matched and the corresponding candidate key frame. Obtain other matching pairs in the candidate key frame, project the reference descriptors of other matching pairs in the candidate key frame to the frame to be matched using the relative pose to obtain projection points, and determine the statistical quantity of the matching pairs whose distance between the projection points and the vertices of the descriptors to be matched is less than the distance threshold. Traverse all the matching pairs, and based on the statistical quantity corresponding to each matching pair, filter out the matching pair with the highest statistical quantity, and use the relative pose solved by the matching pair with the highest statistical quantity as the reference relative pose.

[0034] In some implementation scenarios, use a deep learning model to perform pose analysis on the vertices of the descriptor to be matched and the reference descriptor to obtain the relative pose between the frame to be matched and the corresponding candidate key frame. Use the relative pose to transform the reference descriptors in other matching pairs into the frame to be matched, and obtain the matching deviation between the transformed reference descriptor and the descriptor to be matched. Traverse all the matching pairs, and based on the matching deviation corresponding to each matching pair, filter out the matching pair with the smallest matching deviation, and use the relative pose solved by the matching pair with the smallest matching deviation as the reference relative pose.

[0035] It can be understood that by screening the optimal reference relative pose, the probability of incorrect matching between the frame to be matched and the corresponding candidate key frame can be reduced.

[0036] S104: Obtain the relocalization pose of the robot when collecting the frame to be matched based on the reference relative pose and the pose matched with the corresponding candidate key frame.

[0037] Specifically, based on the reference relative pose and the pose matched with the corresponding candidate key frame, solve the pose of the robot when collecting the frame to be matched to obtain the relocalization pose of the robot when collecting the frame to be matched. Thus, through descriptor matching and screening, the robot relocalization can be completed by using the reference relative pose for solution, improving the convenience of robot relocalization.

[0038] In some implementation scenarios, based on the reference relative pose, filter the descriptors in the corresponding candidate key frame to obtain the descriptors whose projection points and the vertices of the descriptors to be matched satisfy the distance condition. Use the filtered descriptors to solve the relative pose between the frame to be matched and the corresponding candidate key frame to obtain the corrected relative pose after correction. Based on the corrected relative pose and the pose of the corresponding candidate key frame, obtain the relocalization pose of the robot when collecting the frame to be matched.

[0039] In some implementation scenarios, descriptors in corresponding candidate key frames are filtered based on a reference relative pose, and descriptors with a vertex matching deviation from the descriptor to be matched less than a deviation threshold are retained. The relative pose between the frame to be matched and the corresponding candidate key frame is solved using the retained descriptors, obtaining a corrected relative pose. Based on the corrected relative pose and the pose of the corresponding candidate key frame, the relocalization pose when the robot acquires the frame to be matched is obtained.

[0040] Optionally, the target scene also matches a point cloud map, where the point cloud map corresponds to the point cloud frames collected by the radar on the robot. The point cloud to be inspected when the robot acquires the frame to be matched is obtained, and the point cloud to be inspected is projected into the point cloud map using the relocalization pose. The inspection deviation is determined, and when the inspection deviation is greater than the inspection deviation threshold, the pose of the robot is adjusted and the frame to be matched is recollected in the target scene, thereby improving the accuracy of the relocalization pose.

[0041] In the above solution, a texture map matching the target scene is obtained, where the texture map includes multiple key frames collected by the robot. Each key frame is matched with a pose, and each key frame includes texture feature points and descriptors of triangles set based on the texture feature points, thereby constructing a prior texture map and setting reference poses and descriptors. The frame to be matched collected by the robot from the target scene is obtained and the descriptor to be matched in the frame to be matched is determined. The descriptor to be matched is matched with the descriptors corresponding to all key frames, and all descriptors that match the descriptor to be matched are determined as reference descriptors. The key frames associated with the reference descriptors are determined as candidate key frames, thereby quickly finding reference descriptors through the matching of triangle descriptors, improving the matching efficiency and the accuracy of candidate key frames. Among them, the descriptor to be matched and each reference descriptor form a matching pair respectively. Based on the descriptor to be matched and the reference descriptor in each matching pair, the relative pose between the frame to be matched and the candidate key frame is determined, and the relative poses obtained based on each matching pair are filtered using all matching pairs, thereby obtaining an optimal reference relative pose and improving the accuracy of the relative pose. Based on the reference relative pose and the pose matched with its corresponding candidate key frame, the pose of the robot when acquiring the frame to be matched is solved, obtaining the relocalization pose of the robot when acquiring the frame to be matched. Thus, through descriptor matching and filtering, the robot relocalization can be completed by solving using the reference relative pose, improving the convenience of robot relocalization.

[0042] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of another implementation manner of the robot relocalization method of this application. The method includes:

[0043] S201: Obtain the texture map and the point cloud map of the target scene; wherein, the texture map includes multiple key frames collected by the robot, and each key frame is matched with a pose and a descriptor of a triangle obtained based on the texture feature points in the key frame, and the point cloud map is obtained based on the pose matched with the key frame and the point cloud frames collected by the robot.

[0044] Specifically, obtain the texture map matched with the target scene and the point cloud map matched with the target scene. Among them, the texture map includes multiple key frames collected by the robot, and the point cloud map is obtained based on the pose matched with the key frame and the point cloud frames collected by the robot.

[0045] It should be noted that the texture map is obtained through the following steps: Obtain multiple key frames collected by the robot in the target scene, extract the texture feature points of the key frame in each key frame and perform homogenization processing; traverse all the texture feature points of each key frame, in the preset area range corresponding to the current texture feature point, obtain other texture feature points that meet the distance condition with the current texture feature point, and construct a descriptor of a triangle; project the texture feature points of all key frames and their matched descriptors onto the coordinate system corresponding to the target scene to obtain the texture map.

[0046] Specifically, obtain multiple key frames collected by the robot in the target scene, perform feature extraction in each key frame to obtain the texture feature points in the key frame and perform homogenization processing on the texture feature points. Among them, the homogenization processing method can be a quadtree algorithm or other homogenization processing algorithms.

[0047] Furthermore, traverse all the texture feature points of each key frame, respectively take each texture feature point as the current texture feature point, in the preset area range corresponding to the current texture feature point, obtain other texture feature points that meet the distance condition with the current texture feature point, so as to connect the other two texture feature points with the current texture feature point to construct a descriptor of a triangle. Among them, the distance condition corresponds to that the pixel distance between texture feature points is between the lower limit value and the upper limit value of the pixel distance.

[0048] Optionally, denote the point set after uniform distribution in the key frame as {S}, construct a k-dimensional tree for {S}, traverse the points in {S}, and find all the feature points within a certain preset area range (such as one-third or one-fourth of the key frame width) in Si to construct a descriptor of a triangle. Among them, when a descriptor with the same side length of three sides is obtained in the key frame, only the descriptor with the highest feature response value of the three vertex features will be retained to ensure the uniqueness of the descriptor in one frame of image. Among them, the feature response value is obtained based on the texture feature when extracting the texture feature points.

[0049] Further, the side lengths of the descriptors of the triangles are arranged in ascending order. Each descriptor is associated with a key frame index and records the pixel values of the vertices of each triangle. The extracted texture feature points together with the descriptors are projected onto the coordinate system corresponding to the target scene based on the pose matched by the key frames, obtaining a texture map, thereby ensuring the uniqueness and accuracy of the descriptors in the texture map.

[0050] It should be noted that based on the pose matched by the key frames in the texture map, the pose of the corresponding point cloud frame can be estimated, and thus a point cloud map aligned with the texture map can be constructed in a simple manner.

[0051] Optionally, the robot includes a camera, a radar, and an odometer. The point cloud map is obtained based on the following steps: obtaining the current time point corresponding to the current key frame collected by the camera, determining the historical time point corresponding to the previous key frame collected by the camera, obtaining the historical pose matched by the previous key frame at the historical time point, and the historical odometer pose corresponding to the odometer; determining the point cloud frames collected by the radar between the current time point and the historical time point and their corresponding detection time points, determining the current odometer pose corresponding to the odometer at the detection time point, and obtaining the current radar pose corresponding to the radar at the detection time point based on the historical pose, the historical odometer pose, and the current odometer pose; binding the point cloud frame at the detection time point to the key frame at the historical time point, and determining the pose change amount between the radar at the detection time point and the camera at the historical time point based on the current radar pose and the historical pose; after the pose optimization is performed in response to the texture map, determining the target radar pose of the radar based on the pose change amount and the pose matched by the optimized key frame, and constructing a point cloud map using the target radar pose and the point cloud frames collected by the radar.

[0052] Specifically, a point cloud map is constructed simultaneously during the construction of the texture map. When the camera collects the current key frame, there is a corresponding current time point, and when the camera collects the previous key frame, there is a corresponding historical time point. The point cloud frames collected by the radar between the current time point and the historical time point correspond to detection time points. Among them, the point cloud frame at the detection time point is bound to the key frame at the historical time point, and the pose of the point cloud frame at the detection time point is obtained based on the pose matched by the key frame at the historical time point.

[0053] It can be understood that the historical time point corresponding to the previous key frame collected by the camera is denoted as kt-1, and the current time point corresponding to the current key frame is denoted as kt. The pose of the point cloud frames between the key frames kt-1 and kt is derived from the pose matched by the key frame kt-1 and the odometer data between the point cloud frames and kt-1. Among them, the odometer collects the pose at a relatively high frequency and can be accurately matched with the moments when the camera or the radar performs data collection.

[0054] Further, the historical pose of the key frame at the moment kt-1 is denoted as The historical odometer pose of the odometer at time kt-1 is The detection time point of the point cloud frame collected by the radar between the current time point and the historical time point is denoted as ktli, and the current odometer pose of the odometer at this moment is Then the current radar pose of the point cloud frame at the detection time point is:

[0055]

[0056] Among them, is the current radar pose, is the historical pose of the key frame, is the historical odometer pose of the odometer, is the current odometer pose of the odometer, represents matrix inversion.

[0057] Furthermore, the point cloud frame will be bound to the index number of the key frame nearest to it. This index number represents the number of the key frame in the texture mapping process and is the index number for pose optimization. At the same time, the point cloud frame at time ktli will be associated with the pose change amount between the camera at time kt-1. Then when the texture mapping triggers loop detection for overall pose optimization, the pose of the key frame is optimized to Then the radar pose matching the operation of the points bound to it is updated to:

[0058]

[0059] Among them, is the updated target radar pose, is the pose matching the optimized key frame, is the pose change amount between the radar at the detection time point and the camera at the historical time point.

[0060] It can be understood that when the joint optimization of the key frames in the texture map is completed, all corresponding point cloud frames are triggered for synchronous update, and the point cloud in the point cloud frame is refreshed according to the corresponding target radar pose, and a point cloud map aligned with the texture map can be constructed. Therefore, by using the data collected by the radar to associate with the nearby camera data and using the odometer data between two frames to calculate the pose of the point cloud frame and bind this change amount, it is convenient to adjust the pose of the corresponding point cloud frame when the camera pose is adjusted, so as to ensure the construction of a point cloud map aligned with the texture map.

[0061] S202: Obtain the to-be-matched frame collected by the robot and determine the to-be-matched descriptors in the to-be-matched frame. Among the descriptors corresponding to all key frames, determine all reference descriptors that match the to-be-matched descriptors and the candidate key frames associated with the reference descriptors; wherein, each to-be-matched descriptor and each reference descriptor form a matching pair.

[0062] Specifically, obtain the to-be-matched frame collected by the robot from the target scene and determine the to-be-matched descriptors in the to-be-matched frame. Match the to-be-matched descriptors with the descriptors corresponding to all key frames, determine all the descriptors that match the to-be-matched descriptors as reference descriptors, and determine the key frames associated with the reference descriptors as candidate key frames. Wherein, each to-be-matched descriptor and each reference descriptor form a matching pair.

[0063] It should be noted that descriptor matching has pixel information, side length information, and angle information. The side length information and angle information of the descriptors corresponding to all key frames are converted into key values and stored in a hash list. The key values correspond to at least one descriptor. Among them, the side length information and angle information are converted into key values through a hash conversion function and stored in the hash list. The key value of the hash list is a six-dimensional array. The first three dimensions are the three sides of the descriptor, arranged in ascending order, and the last three dimensions are the three angles of the descriptor. The six-dimensional vector is used to map to the key value of the hash list. The key value in the hash list is an array container for storing descriptors with similar side length information and angle information, and the key values in the hash list are sorted numerically in sequence.

[0064] In some implementation scenarios, among the descriptors corresponding to all key frames, determining all reference descriptors that match the to-be-matched descriptors and the candidate key frames associated with the reference descriptors includes: determining the key value set corresponding to the to-be-matched descriptor from the hash list based on the to-be-matched side length and to-be-matched angle corresponding to the to-be-matched descriptor; wherein, the key value set includes the key values that match the to-be-matched key value and other key values within its preset range, and the descriptors corresponding to all the key values in the key value set are candidate descriptors; screening to obtain reference descriptors and determining the candidate key frames associated with the reference descriptors based on the pixel information, side length information, and angle information respectively matched by the candidate descriptors and the to-be-matched descriptors.

[0065] Specifically, convert the to-be-matched side length and to-be-matched angle of the to-be-matched descriptor into a to-be-matched key value, and obtain the key value set associated with the to-be-matched key value from the hash list. Among them, the key value set includes the key value that is exactly the same as the to-be-matched key value and other key values within the preset range of this key value, so as to obtain key values with relatively high similarity to the to-be-matched key value, and use the descriptors among them as candidate descriptors.

[0066] Further, compare the pixel information, side length information, and angle information between the candidate descriptor and the descriptor to be matched, filter out the descriptors with the deviation of each item of information less than the set value, and use the filtered descriptors to determine the candidate key frames. Thus, the descriptor in the candidate key frames that matches the descriptor to be matched is used as the reference descriptor, improving the matching degree between the reference descriptor and the descriptor to be matched.

[0067] Optionally, based on the pixel information, side length information, and angle information respectively matched by the candidate descriptor and the descriptor to be matched, filter to obtain the reference descriptor and determine the candidate key frames associated with the reference descriptor, including: determining the matching score corresponding to the key frames associated with the candidate descriptor based on the pixel information, side length information, and angle information respectively matched by the candidate descriptor and the descriptor to be matched; taking the key frames whose matching scores meet the matching conditions as candidate key frames, and obtaining the reference descriptors in the candidate key frames that match the descriptor to be matched.

[0068] Specifically, compare the pixel information, side length information, and angle information between the candidate descriptor and the descriptor to be matched, determine whether the candidate descriptor meets the condition that the deviation of each item of information is less than the set value. If it meets, add points to the key frames associated with the candidate descriptor, thereby finally determining the matching scores of the corresponding key frames.

[0069] Further, filter the key frames based on the matching scores, take the key frames whose matching scores meet the matching conditions as candidate key frames, and obtain the reference descriptors in the candidate key frames that match the descriptor to be matched, thereby improving the accuracy of the candidate key frames and the accuracy rate of the reference descriptors.

[0070] For ease of explanation, the present application gives a specific filtering method, but the present application does not limit the specific filtering conditions. Denote the i-th descriptor of the frame to be matched as Ti, and the j-th candidate descriptor in the key value as Pj. Judge whether the sum of the lengths of the three sides of Ti and the sum of the lengths of the three sides of Pj are less than the threshold σ li , where, σ li is β l times the sum of the lengths of the three sides of Ti. Judge whether the differences between the three included angles corresponding to Ti and the three included angles corresponding to Pj are all less than σ a . Judge whether the difference between the sum of the pixel values corresponding to the three vertices of Ti and the sum of the pixel values corresponding to the three vertices of Pj is less than σ pti , where, σ pti is β p times the sum of the pixel values of the three vertices of Ti. If all three conditions are met, it is considered that Ti and Pj are matching descriptors.

[0071] Further, traverse all the candidate descriptors that match the triangle descriptors in the frame to be matched. When the frame to be matched matches a certain descriptor, the score of the key frame associated with the candidate descriptor is incremented by 1. After the traversal, the top N frames with the highest scores and scores greater than the preset value are used as candidate key frames, and the descriptors matched by the candidate key frames are saved as the reference candidate frames.

[0072] S203: Based on each matching pair, determine the relative pose between the frame to be matched and the corresponding candidate key frame, and use all the matching pairs to filter out the reference relative pose from all the relative poses.

[0073] Specifically, based on the descriptor to be matched and the reference descriptor in each matching pair, determine the relative pose between the frame to be matched and the candidate key frame, and use all the matching pairs to filter the relative poses obtained based on each matching pair, so as to obtain the optimal reference relative pose and improve the accuracy of the relative pose.

[0074] In some implementation scenarios, based on each matching pair, determine the relative pose between the frame to be matched and the corresponding candidate key frame, and use all the matching pairs to filter out the reference relative pose from all the relative poses, including: for the current matching pair, based on the descriptor to be matched and the corresponding reference descriptor, determine the relative pose between the frame to be matched and the corresponding candidate key frame; use the relative pose to project the vertices matched by the reference descriptor in other matching pairs into the frame to be matched to obtain projection points, and based on the point-to-point distance between the projection points and the vertices of the descriptor to be matched, determine the transformation score of the current matching pair; traverse all the matching pairs, filter out the target matching pairs based on the transformation scores of each matching pair, and use the relative poses corresponding to the target matching pairs as the reference relative poses.

[0075] Specifically, solve based on the vertices of the descriptor to be matched and the reference descriptor to determine the relative pose between the frame to be matched and the corresponding candidate key frame.

[0076] Optionally, perform ICP registration on the vertices of the descriptor to be matched and the reference descriptor in the matching pair. Denote the i-th candidate key frame as Ci, and the j-th reference descriptor of this candidate key frame and the descriptor to be matched form a matching pair, denoted as Mj. The descriptor to be matched in Mj is p a , and the descriptor in the corresponding candidate key frame that matches is p b . Then the pose calculated using this matching pair is:

[0077]

[0078] [U, S, V] = SVD(H)(4)

[0079] ΔR j = UV T , Δtj =-R*q a +q b (5)

[0080] where q a is the mean of the three vertex coordinates of p a descriptor, and q b is the mean of the three vertex coordinates corresponding to p b descriptor, p ai and p bi are the coordinates of the i-th vertex corresponding to two descriptors, SVD represents singular value decomposition, [U, S, V] are the matrices obtained by decomposition, ΔR j and Δt j are the relative poses calculated using the j-th matching pair, including two parameters: rotation and translation.

[0081] Furthermore, using the relative pose, project the vertices of the reference descriptor in other matching pairs to the frame to be matched, obtaining the projected points. Among them, the projection formula can be:

[0082] p bii =ΔR j *p bi +Δt j (6)

[0083] where p bii represents the projected point corresponding to each vertex, and p bi represents the vertex of the reference descriptor in other matching pairs.

[0084] It can be understood that determine the point-to-point distance between the projected point and the corresponding vertex in the descriptor to be matched, and judge whether the point-to-point distance is less than the threshold σ tl , that is, judge whether the point-to-point distance between each pair of corresponding vertices of p ai and p bii is less than the threshold σ tl . If satisfied, determine that the relative pose calculated using the corresponding matching pair is accurate, and add 1 to the conversion score of the corresponding matching pair.

[0085] Furthermore, after traversing all matching pairs, count the conversion scores of each matching pair, select the matching pair with the highest score as the target matching pair, and use the relative pose calculated using the target matching pair as the reference relative pose to make the reference relative pose have higher accuracy.

[0086] S204: Based on the reference relative pose and the pose matched with the corresponding candidate key frame, obtain the relocalization pose when the robot acquires the frame to be matched.

[0087] Specifically, based on the reference relative pose and the pose corresponding to the candidate key frame match, the pose of the robot when collecting the frame to be matched is solved to obtain the relocalization pose of the robot when collecting the frame to be matched.

[0088] In some implementation scenarios, obtaining the relocalization pose of the robot when collecting the frame to be matched based on the reference relative pose and the pose corresponding to the candidate key frame match includes: obtaining the point distance obtained when projecting the vertices of the reference descriptor match using the reference relative pose; using the matching pairs that satisfy the distance condition, determining the corrected relative pose between the frame to be matched and the candidate key frame associated with the target matching pair; and obtaining the relocalization pose of the robot when collecting the frame to be matched based on the corrected relative pose and the pose corresponding to the candidate key frame associated with the target matching pair.

[0089] Specifically, when obtaining the point distance obtained when projecting the vertices of the reference descriptor match using the reference relative pose, the matching pairs with point distances not all less than the threshold σ tl are filtered out, the feature point set composed of the vertices of the reference descriptor among all the matching points obtained after screening is determined, and the relative pose between the frame to be matched and the candidate key frame is solved using the feature point set to obtain the corrected relative pose ΔT between the frame to be matched and the candidate key frame associated with the target matching pair. Using ΔT and the pose Ti of the candidate key frame, the relocalization pose Tc of the robot when collecting the frame to be matched can be obtained. The above process is expressed by the formula as follows:

[0090] Tc = Ti * ΔT (7)

[0091] where Tc represents the relocalization pose of the robot when collecting the frame to be matched, Ti represents the pose of the candidate key frame, and ΔT represents the corrected relative pose between the frame to be matched and the candidate key frame.

[0092] S205: Obtain the point cloud to be verified that matches the acquisition time point of the frame to be matched.

[0093] Specifically, determine the acquisition time point of the frame to be matched, and obtain the point cloud frame at the time point closest to the acquisition time point as the point cloud to be verified.

[0094] S206: Project the point cloud to be verified onto the point cloud map using the relocalization pose to obtain the verification score of the relocalization pose; among them, the relocalization pose that satisfies the verification condition is the final pose of the robot when collecting the frame to be matched.

[0095] Specifically, project the point cloud to be verified onto the point cloud map using the relocalization pose, determine the matching degree between the point cloud to be verified projected onto the point cloud map and the point cloud map, so as to obtain the verification score of the relocalization pose.

[0096] It can be understood that when there are multiple matching pairs with the highest conversion scores, multiple relocalization poses can be obtained. Then, based on the verification scores, the relocalization poses can be screened to obtain the final pose of the robot when collecting the frame to be matched. When the number of matching pairs with the highest conversion score is one, the relocalization pose is verified based on the verification score to determine whether the currently obtained relocalization pose is accurate and can be used as the final pose of the robot when collecting the frame to be matched. Therefore, the accuracy of the finally obtained relocalization pose is improved through verification.

[0097] Please refer to Figure 3 , Figure 3 FIG. is a schematic structural diagram of an embodiment of an electronic device according to the present application. The electronic device 30 includes a memory 301 and a processor 302 that are coupled to each other. Among them, the memory 301 stores program data (not shown in the figure), and the processor 302 calls the program data to implement the method in any of the above embodiments. For the description of related content, please refer to the detailed description of the above method embodiments and will not be repeated here.

[0098] Please refer to Figure 4 , Figure 4 FIG. is a schematic structural diagram of an embodiment of a computer-readable storage medium according to the present application. The computer-readable storage medium 40 stores program data 400, and when the program data 400 is executed by a processor, it implements the method in any of the above embodiments. For the description of related content, please refer to the detailed description of the above method embodiments and will not be repeated here.

[0099] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0101] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0102] The above are only the embodiments of this application, and do not limit the protection scope of this application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of this application, or directly or indirectly applied in other related technical fields, shall be included in the protection scope of this application by the same token.

Claims

1. A robot relocalization method, characterized in that, The method includes: Obtaining a texture map of a target scene; wherein, the texture map includes a plurality of key frames collected by a robot, and the key frames are matched with poses and descriptors of triangles obtained based on texture feature points in the key frames; Obtaining a frame to be matched collected by the robot and determining a descriptor to be matched in the frame to be matched, and determining all reference descriptors that match the descriptor to be matched and candidate key frames associated with the reference descriptors among the descriptors corresponding to all the key frames; wherein, the descriptor to be matched and each of the reference descriptors form a matching pair; Based on each matching pair, determining a relative pose between the frame to be matched and the corresponding candidate key frame, and screening a reference relative pose from all the relative poses by using all the matching pairs; Based on the reference relative pose and the pose matched with the corresponding candidate key frame, obtaining a relocalization pose when the robot collects the frame to be matched.

2. The robot relocalization method according to claim 1, wherein The target scene is further matched with a point cloud map, and the point cloud map is obtained based on the pose matched with the key frame and the point cloud frame collected by the robot; After obtaining the relocalization pose when the robot collects the frame to be matched based on the reference relative pose and the pose matched with the corresponding candidate key frame, it further includes: Obtaining a point cloud to be inspected that matches the acquisition time point of the frame to be matched; Projecting the point cloud to be inspected into the point cloud map by using the relocalization pose to obtain a verification score of the relocalization pose; wherein, the relocalization pose that satisfies the verification condition is the final pose when the robot collects the frame to be matched.

3. The robot relocalization method according to claim 2, wherein The robot includes a camera, a radar, and an odometer, and the point cloud map is obtained based on the following steps: Obtaining the current time point corresponding to the current key frame collected by the camera, determining the historical time point corresponding to the previous key frame collected by the camera, obtaining the historical pose matched with the previous key frame at the historical time point, and the historical odometer pose corresponding to the odometer; Determining the point cloud frames collected by the radar between the current time point and the historical time point and their corresponding detection time points, determining the current odometer pose corresponding to the odometer at the detection time point, and obtaining the current radar pose corresponding to the radar at the detection time point based on the historical pose, the historical odometer pose, and the current odometer pose; Binding the point cloud frame at the detection time point with the key frame at the historical time point, and determining the pose change amount of the radar relative to the camera at the historical time point based on the current radar pose and the historical pose; After the pose is optimized in response to the texture map, determining the target radar pose of the radar based on the pose change amount and the pose matched with the optimized key frame, and constructing the point cloud map by using the target radar pose and the point cloud frames collected by the radar.

4. The robot relocalization method according to claim 1, characterized in that, The descriptor is matched with pixel information, side length information, and angle information, and the side length information and angle information matched by the descriptors corresponding to all the key frames are converted into key values and stored in a hash list, and the key values correspond to at least one descriptor; Determining all reference descriptors that match the descriptor to be matched and candidate key frames associated with the reference descriptors among the descriptors corresponding to all the key frames includes: Determining a set of key values corresponding to the descriptor to be matched from the hash list based on the side length to be matched and the angle to be matched corresponding to the descriptor to be matched; wherein, the set of key values includes the key values matched by the descriptor to be matched and other key values within a preset range, and the descriptors corresponding to all the key values in the set of key values are candidate descriptors; Filtering to obtain the reference descriptors and determining candidate key frames associated with the reference descriptors based on the pixel information, side length information, and angle information respectively matched by the candidate descriptors and the descriptor to be matched.

5. The robot relocalization method according to claim 4, wherein The filtering to obtain the reference descriptors and determining candidate key frames associated with the reference descriptors based on the pixel information, side length information, and angle information respectively matched by the candidate descriptors and the descriptor to be matched includes: Determining a matching score corresponding to the key frame associated with the candidate descriptor based on the pixel information, side length information, and angle information respectively matched by the candidate descriptor and the descriptor to be matched; Taking the key frames whose matching scores meet the matching conditions as candidate key frames, and obtaining the reference descriptors that match the descriptor to be matched within the candidate key frames.

6. The robot relocalization method according to claim 1, wherein The determining the relative pose of the frame to be matched and the corresponding candidate key frame based on each matching pair, and filtering to obtain a reference relative pose from all the relative poses using all the matching pairs includes: For the current matching pair, determining the relative pose of the frame to be matched and the corresponding candidate key frame based on the descriptor to be matched and the corresponding reference descriptor; Projecting the vertices matched by the reference descriptor in other matching pairs into the frame to be matched using the relative pose to obtain projection points, and determining the transformation score of the current matching pair based on the point-to-point distance between the projection points and the vertices of the descriptor to be matched; Traversing all the matching pairs, filtering to obtain target matching pairs based on the transformation scores of each matching pair, and taking the relative poses corresponding to the target matching pairs as the reference relative pose.

7. The robot relocalization method according to claim 6, wherein The obtaining the relocalization pose when the robot acquires the frame to be matched based on the reference relative pose and the pose matched by the corresponding candidate key frame includes: Obtaining the point-to-point distance obtained when projecting the vertices matched by the reference descriptor using the reference relative pose; Using the matching pairs whose point-to-point distances meet the distance condition to determine the corrected relative pose between the frame to be matched and the candidate key frame associated with the target matching pair; Obtaining the relocalization pose when the robot acquires the frame to be matched based on the corrected relative pose and the pose matched by the candidate key frame associated with the target matching pair.

8. The robot repositioning method according to any one of claims 1-7, characterized in that The texture map is obtained based on the following steps: Obtaining a plurality of key frames acquired by the robot in the target scene, and extracting and uniformly processing the texture feature points in each key frame; Traverse all texture feature points of each of the key frames, and within the preset area range corresponding to the current texture feature point, obtain other texture feature points that satisfy the distance condition with the current texture feature point, and construct a descriptor of a triangle; Project the texture feature points of all the key frames and their matching descriptors onto the coordinate system corresponding to the target scene to obtain the texture map.

9. An electronic device, characterized in that, Comprising: A memory and a processor that are mutually coupled, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1-8.

10. A computer-readable storage medium having program data stored thereon, characterized in that, When the program data is executed by the processor, the method according to any one of claims 1-8 is implemented.

Images