A visual relocalization method based on object semantics

Abstract

The present invention discloses a visual relocalization method based on object semantics. While SLAM mapping is being built, the YOLOv5 algorithm is used to obtain the detection results of each object in the image. The three-dimensional ellipsoid representation of the object and the 6-Dof pose in three-dimensional space are estimated. The common view relationship of the object is recorded at the same time. The feature points of the image within the bbox of the detected object are extracted to build a bag-of-words database. When relocalization is required, a BOW vector is generated based on the object detected in the lost frame and compared with the database to find the candidate objects corresponding to each object. The constraint between the ellipsoid and the bbox is iterated and calculated multiple times, and the initial pose with the lowest cost is selected. Finally, the ICP problem is solved by the current frame feature points, the historical point cloud map, and the object pose to obtain the final precise positioning. This method effectively improves the speed, accuracy, robustness, and applicability of visual relocalization, and has just the right performance compared to the viewpoint feature-based method in a large parallax environment.

Description

Technical Field

[0001] The method of the present invention relates to the repositioning of a robot and provides a visual repositioning method based on object semantics in a large parallax environment. Background Art

[0002] Vision-based simultaneous localization and mapping (SLAM) using RGB, RGB-D, or stereo cameras is a key technology for autonomous robots and vehicles, as well as augmented reality (AR). Significant progress has been made, enabling systems to perform real-time 3D tracking and scene reconstruction over vast areas, both indoors and outdoors. However, developing fast and reliable relocalization—determining the 6Dof pose of a "missing" sensor, which may result from tracking failure or falling back to a previously constructed map—is crucial for practical robotic applications. However, vision-based relocalization suffers from limitations in accuracy, efficiency, and robustness in large-scale environments. Over the past decade, semantic SLAM has become a hot topic of research, with the introduction of semantics helping to overcome these limitations of current visual relocalization algorithms. We employ a semantic object-based approach, which, in contrast to view-dependent point features, ensures that the presence of visible objects is invariant to viewpoint. Currently, most relocalization methods are appearance-based, attempting to align the pose of a missing frame with the map by matching sets of 2D or 3D point features or by learning relationships between them. However, relocalization will fail in extreme cases where map features are not visible or the appearance of the lost frame and the keyframe are significantly different under large parallax environments.

[0003] Currently available solution technologies:

[0004] A keyframe-based visual relocalization method uses word vectors generated from orb feature points to search for adjacent keyframes of the current frame, and then solves the camera pose using a 2D-3D method. However, this method often uses differences between the query image and the database image, making it difficult to retrieve the true nearest neighbor image, thus affecting the accuracy of pose estimation.

[0005] A 6-DOF camera relocalization method based on a convolutional neural network (CNN) trains a convolutional neural network to regress 6DOF poses from RGB images end-to-end without requiring additional processing. However, relying solely on CNNs to extract image features leads to poor positioning accuracy, insufficient generalization, and difficulty in product implementation. Furthermore, training is difficult in small sample environments.

[0006] A method for achieving 6Dof camera localization through 3D surface regression proposes a method for densely fitting local block coordinates in an image using a fully connected neural network. The method then samples from a set of pose hypotheses, scores them using a softinlier counting method, and selects and optimizes the final estimate based on the scores. This method requires a sufficient dataset for long-term model training, requires high-performance computing equipment to run the algorithm, and performs poorly in large-scale scenes.

[0007] A robot relocalization method based on bag-of-visual words and laser matching uses a bag-of-words structured from visual feature points to find adjacent keyframes of the current frame to obtain an initial position. This position is then used to match the LiDAR point cloud with the map grid for localization. This method requires the use of LiDAR, which is relatively expensive. Summary of the Invention

[0008] The purpose of the present invention is to provide a visual relocalization method based on object semantics, aiming to solve the problems of existing relocalization algorithms in extreme cases such as large parallax changes, invisible map features, or significant differences in the appearance of lost frames and key frames, which are not robust and require high computing resources.

[0009] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0010] A visual relocalization method based on object semantics, comprising:

[0011] The robot uses an RGB-D camera to capture images of the environment, processes the images using a target detection algorithm, and obtains detection results for each object in the image. By modeling the detection boxes of the same object detected in multiple frames, the robot obtains a 3D ellipsoid representation of the object and its 6-DoF pose in 3D space.

[0012] Build a bag-of-words database for each object and a co-viewing adjacency list for all objects;

[0013] When the robot needs to relocalize, it first combines the target detection results of the lost frame image and determines the corresponding candidate objects of the object observed in the lost frame by querying the bag-of-words database and the common view relationship table, thereby obtaining the correspondence between the object and the detection box in the lost frame image;

[0014] Using the correspondence between the object and the detection frame, the initial camera pose is estimated through the detection frame and the corresponding object ellipsoid;

[0015] After obtaining the initial pose estimate, the pose is optimized using an iterative closest point registration algorithm to achieve global relocalization of the robot.

[0016] Furthermore, the object modeling is performed on the detection frames of the same object detected in multiple frames of images to obtain a three-dimensional ellipsoid representation of the object and a 6-DOF pose in three-dimensional space, including:

[0017] Using the monocular camera mounted on the robot, key frames are obtained from the camera's RGB image frame sequence by using the parallax between image frames. The key frame images are processed using the YOLOv5 algorithm to obtain the detection results of each object in the image.

[0018] Determine the data association between frames; whether the observed object is the same can be confirmed by whether the center point of the previous frame's bbox falls within the bbox of the current frame;

[0019] Determine the data association between frames and objects; an object refers to an ellipsoid representation obtained by solving multi-frame projection constraints for a specific object in the scene; obtain the current 6-Dof pose of the robot's monocular camera in three-dimensional space based on the RGB-D SLAM algorithm currently running on the robot, and calculate the projection of these objects on the current frame pixel image using the camera's 6-Dof pose and camera intrinsic parameters as well as successfully initialized objects; if the object projection center falls within a certain bbox and the object category of the bbox detected object is consistent with the object category of the object, then the object observed by the bbox is considered to be an object that has been successfully initialized; if the object observed by the bbox is not one of the successfully initialized objects, then the object is initialized;

[0020] For an object i that has been observed in more than three consecutive key frames and has not been successfully initialized, its ellipsoid representation and 6Dof pose are estimated.

[0021] Furthermore, for an object i that has been observed in three or more consecutive key frames and has not been successfully initialized, estimating its ellipsoid representation and 6Dof pose includes:

[0022] and The relationship between them is given by the projection matrix P f =K[R f |t f ]definition:

[0023]

[0024] Among them, R f is the rotation matrix of the camera in the f-th frame; t f is the translation vector of the camera in the f-th frame, is the ellipsoidal dual representation of object i; is the elliptical projection of object i in the fth frame; K is the camera internal parameter; β ifis the scale factor of the elliptical projection of object i on the f-th frame;

[0025] P f and The products are arranged in a single matrix by:

[0026]

[0027]

[0028]

[0029]

[0030] We can get:

[0031]

[0032] remember

[0033] in, is the Kronecker product; vech is the serialization of the lower triangular elements of the matrix; vec is the serialization of the matrix elements; D is the transformation matrix from vech to vec; E is the transformation matrix from vec to vech; Serialize the lower triangular elements of the matrix representing the ellipsoid of object i; Serialize the lower triangular elements of the matrix of the projection ellipse of ellipsoid i for object i in frame f;

[0034] Assume that there are F (F>=3) image frames stacked column by column:

[0035] M i w i =0 (6)

[0036]

[0037] Among them G1...G F is the G of each frame calculated according to formula (5) f ; Serialize the triangular elements of the matrix under the ellipsoidal projection of object i in the first frame, And so on; is the serialization of the lower triangular elements of the matrix representing the ellipsoid of object i; i =[β i1 ,β i2 ....β iF ] T , β i1 is the scale factor of the elliptical projection of object i on the first frame;

[0038] Solving the overdetermined equation (6) yields w i , due to the dual ellipse is a symmetric matrix, through The sequence of lower triangular elements of the matrix You can restore In this way, we get the ellipsoid expression of object i in the dual space;

[0039] General ellipsoid Q in dual space * can be rewritten as:

[0040]

[0041]

[0042] Where Z is a homogeneous transformation; is an ellipsoid centered at the origin and with its axes aligned with the 3D coordinates; R(θ) is the rotation matrix of the ellipsoid; t is the translation vector of the ellipsoid; a, b, c are the lengths of the ellipsoid in the xyz coordinate directions;

[0043] This rewrite can be done by * By performing eigenvalue decomposition, we can obtain The operation can obtain the three-dimensional ellipsoid representation of object i and its 6-Dof pose in three-dimensional space; at this time, a new object is initialized successfully, and an ID number is assigned to this object to distinguish different objects.

[0044] Furthermore, the step of constructing a bag-of-words database for each object and constructing a co-viewing adjacency list for all objects includes:

[0045] For each successfully initialized object, a BOW vector library is established. This vector library is a collection of BOW vectors. It is established when the object is successfully initialized and is initially an empty collection. The bboxes detected in this frame of image and the objects corresponding to these bboxes are calculated. For each detected bbox, the image feature points in the bbox are extracted using the orb feature point extraction algorithm. For all feature points in a specific bbox, the corresponding BOW vector is generated using the DBoW algorithm. The obtained BOW vector is matched with all BOW vectors in the BOW vector library of the object corresponding to the bbox. If the matching score is lower than the threshold, the BOW vector is added to the BOW vector library of the object.

[0046] If objects a and b can be observed simultaneously in a frame of image, they are considered to be in common view. All objects observed in the current frame can be considered to have a common view relationship between these objects, and these common view relationships are stored in the form of an adjacency list. The common view adjacency list is an n*n matrix, where n is the number of objects successfully initialized. The element in the i-th row and j-th column of the matrix is ​​1, indicating that objects i and j are in common view.

[0047] Furthermore, the target detection result of the lost frame image is combined with the query of the bag-of-words database and the common view relationship table to determine the corresponding candidate object of the object observed in the lost frame, thereby obtaining the correspondence between the object and the detection box in the lost frame image, including:

[0048] When the SLAM algorithm cannot calculate the current robot pose, the image at this time is defined as a lost frame image. The lost frame image is processed by the YOLOv5 algorithm to obtain the target detection results of each object in the image;

[0049] For a specific bbox, the image feature points in the bbox are extracted using the orb feature point extraction algorithm, and then the BOW vector v of the bbox is generated using the DBoW algorithm. The obtained BOW vector v is matched with the vectors in the BOW vector library of objects of the same object category as the bbox. For an object, if its BOW vector library has a matching score with the BOW vector v that is higher than the threshold, the object is added to the candidate object sequence of the bbox.

[0050] Candidate objects are screened by the co-viewing relationship of objects observed in the lost frame:

[0051] Assume that the lost frame observes a and an object of object category x at the same time. If A exists in the common view adjacency table and is common to the object classified as x, the common view score of the candidate object A is +1. The object corresponding to the bbox is determined to be the object with the highest common view score, and the correspondence between the object and the bbox is obtained.

[0052] Furthermore, using the correspondence between the object and the detection frame, the initial camera pose is estimated by using the detection frame and the corresponding object ellipsoid, including:

[0053] The four edges of a detection box bbox can recover four envelope sections, so only two or more bboxes are needed to solve the following equation P, that is, to obtain the camera's 6-Dof pose [R|t]; after obtaining the correspondence between the object and the bbox, the dual ellipse Q of the object corresponding to the bbox edge I in equation 9 can be obtained * :

[0054] (P T I) T Q* (P T I)=0 (9)

[0055] Where P = K[R|t], K is the camera intrinsic parameter, and I is the edge vector of the bbox;

[0056] When the number of observed objects is greater than or equal to 3, multiple iterations are performed to calculate the pose. In each iteration, three pairs of ellipse-ellipsoids are selected to solve Equation 9 to obtain a 6-Dof pose of the camera. For the pose calculated in each iteration, the ellipsoid is reprojected onto the camera imaging plane to obtain a projected ellipse. The circumscribed rectangle of the projected ellipse is calculated, and the sides of the rectangle are parallel to the sides of the bbox detection frame. The distances between the sides of the rectangle and the sides of the bbox are calculated as the cost of the pose. Finally, the pose with the smallest cost in multiple iterative calculations is selected as the calculated initial pose. If the number of observed objects in the lost frame is less than 3, the relocalization fails.

[0057] Furthermore, the robot should simultaneously run the RGB-D SLAM algorithm to provide camera pose data for the 3D ellipsoid generation algorithm and build a dense map.

[0058] Compared with the prior art, the present invention has the following technical features:

[0059] 1. The implementation process only uses YOLOv5 to obtain the bbox for target detection and solve two nonlinear optimization problems. The algorithm does not require high computing power and can run in real time on some low-computing power devices.

[0060] 2. Compared with view-dependent point features, the existence of visible objects is invariant to the viewpoint. It is more robust to large parallax environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is the overall flow chart of the visual relocalization method based on object semantics;

[0062] Figure 2 Schematic diagram of the ellipsoid representation and 6Dof pose of the observed object estimated by bbox;

[0063] Figure 3 Determine a flow chart for the observed object;

[0064] Figure 4 Schematic diagram of semantic-based coarse positioning. DETAILED DESCRIPTION

[0065] See also Figure 1 The present invention provides a method for visual relocalization based on object semantics, comprising the following steps:

[0066] In step 1, an RGB-D camera mounted on the robot is used to acquire an image of the environment. The image is processed using the YOLOv5 algorithm to obtain detection results for each object in the image, including the bounding box (bbox), object category, and detection confidence. Object modeling is performed on the bbox of the same object detected in multiple frames to obtain a 3D ellipsoid representation of the object and its 6-DOF pose in 3D space. The robot described in step 1 should also run the RGB-D SLAM algorithm to provide camera pose data for the 3D ellipsoid generation algorithm and build a dense map for use in subsequent steps.

[0067] See also Figure 2 , which is the detailed implementation process of step 1, is as follows:

[0068] In step 1.1, use the monocular camera mounted on the robot to obtain key frames from the camera's RGB image frame sequence based on the parallax between image frames. Then, process the key frame images using the YOLOv5 algorithm to obtain the detection results of each object in the image, namely, the bounding box, object category, and detection confidence.

[0069] Step 1.2: Determine the data association between frames. Specifically, for a bbox in the current frame, whether the center point of the bbox in the previous frame falls within the bbox in the current frame can be used to confirm whether the observed object is the same.

[0070] Step 1.3, determine the data association between the frame and the object; the object refers to the ellipsoid representation obtained by solving the multi-frame projection constraint for a specific object in the scene, which also has the object category of the corresponding object, the axis lengths of the ellipsoid, the 6-Dof pose of the ellipsoid, and the object ID number; the "object" in the following text has the same meaning; according to the RGB-DSLAM algorithm being run by the robot, the current 6-Dof pose of the monocular camera carried by the robot in the three-dimensional space is obtained, and the projection of these objects on the current frame pixel image is calculated through the 6-Dof pose of the camera and the camera intrinsic parameters and the objects that have been successfully initialized; if the object projection center falls in a certain bbox and the object category of the object detected by the bbox is consistent with the object category of the object, then the object observed by the bbox is considered to be an object that has been successfully initialized as an object; if the object observed by the bbox is not a member of the successfully initialized objects, execute step 1.4 to initialize the object.

[0071] Step 1.4: For objects i that have been observed in more than three consecutive keyframes and have not been successfully initialized, perform ellipsoid representation and 6Dof pose estimation:

[0072] In the dual space, the ellipse can be described as:

[0073] XT C * X=0 (1)

[0074] In the above formula, X is the envelope tangent of the ellipse, C * It is the dual representation of the ellipse. The superscript T represents the transpose. The same below.

[0075] Similarly, the ellipsoid is described as:

[0076] π T Q * π=0 (2)

[0077] In the above formula, π is the enveloping section of the ellipsoid, Q * is the dual representation of the ellipsoid.

[0078] and The relationship between them is given by the projection matrix P f =K[R f |t f ](R f is the rotation matrix of the camera in the f-th frame; t f is the translation vector of the camera in the f-th frame) is defined as:

[0079]

[0080] In the above formula is the ellipsoidal dual representation of object i; is the elliptical projection of object i in the fth frame; K is the camera internal parameter; β if is the scale factor of the elliptical projection of object i on the f-th frame.

[0081] P f and The products are arranged in a single matrix by:

[0082]

[0083]

[0084]

[0085]

[0086] We can get:

[0087]

[0088] remember

[0089] In the above formula is the Kronecker product; vech is the serialization of the lower triangular elements of the matrix; vec is the serialization of the matrix elements; D is the transformation matrix from vech to vec; E is the transformation matrix from vec to vech; Serialize the lower triangular elements of the matrix representing the ellipsoid of object i; Serializes the lower triangle elements of the matrix of the projection ellipse of object i onto ellipsoid i in frame f.

[0090] Assume that there are F (F>=3) image frames stacked column by column:

[0091] M i w i =0 (6)

[0092]

[0093] Among them G1...G F is the G of each frame calculated according to formula (5) f ; Serialize the triangular elements of the matrix under the ellipsoidal projection of object i in the first frame, And so on; is the serialization of the lower triangular elements of the matrix representing the ellipsoid of object i; i =[β i1 ,β i2 ....β iF ] T , β i1 The same formula (3) is the scale factor of the elliptical projection of object i on the first frame.

[0094] Solving the overdetermined equation (6) yields w i , due to the dual ellipse is a symmetric matrix, through The sequence of lower triangular elements of the matrix You can restore In this way, we get the ellipsoid expression of object i in the dual space.

[0095] General ellipsoid Q in dual space * can be rewritten as:

[0096]

[0097]

[0098] Where Z is a homogeneous transformation; It is an ellipsoid centered at the origin and with its axes aligned with the 3D coordinates; R(θ) is the rotation matrix of the ellipsoid; t is the translation vector of the ellipsoid; a, b, c are the lengths of the axes of the ellipsoid in the directions of the xyz coordinate axes.

[0099] This rewrite can be done by * By performing eigenvalue decomposition, we can obtain The operation can obtain the three-dimensional ellipsoid representation and 6-DOF pose of object i in three-dimensional space. At this time, a new object is initialized successfully and an ID number is assigned to this object to distinguish different objects. The ID number is given from 1 to n according to the object number, and the object category of the object is the object category of object i.

[0100] Step 2: Build a bag-of-words (BOW) database for each object and a co-viewing adjacency list for all objects.

[0101] The specific steps of step 2 are:

[0102] Step 2.1: For each successfully initialized object, a BOW vector library is established. The vector library is a collection that stores BOW vectors. It is established when the object is successfully initialized and is initially an empty collection. According to step 1, the bboxes detected in this frame image and the objects corresponding to these bboxes are calculated. For each detected bbox, the image feature points in the bbox are extracted using the orb feature point extraction algorithm. For all feature points in a specific bbox, the corresponding BOW vector is generated using the DBoW algorithm. The obtained BOW vector is matched with all BOW vectors in the BOW vector library of the object corresponding to the bbox. If the matching score is lower than the threshold, the BOW vector is added to the BOW vector library of the object.

[0103] In step 2.2, if objects a and b can be observed simultaneously in a frame of image, they are considered to be in common view. All objects observed in the current frame can be considered to have a common view relationship between these objects, and these common view relationships are stored in the form of an adjacency list. The common view adjacency list is an n*n matrix, where n is the number of objects successfully initialized. The element in the i-th row and j-th column of the matrix is ​​1, indicating that objects i and j are in common view.

[0104] Step 3: When the robot moves significantly or is transferred to other areas by external forces, SLAM tracking is lost, and the robot loses the ability to determine its own 6-DOF pose in the scene. At this time, the robot needs to perform relocalization to restore the robot's positioning in the map: first, combined with the target detection results of the lost frame image, by querying the BOW vector library and the common view relationship table, the corresponding candidate objects of the objects observed in the lost frame are determined, thereby obtaining the correspondence between the objects and the detection boxes in the lost frame image.

[0105] See also Figure 3 , which is the detailed implementation process of step 3, is as follows:

[0106] In step 3.1, when the robot tracking is detected to be lost, that is, the SLAM algorithm cannot calculate the current robot posture, the image at this time is defined as a lost frame image. The lost frame image is processed by the YOLOv5 algorithm to obtain the target detection results of each object in the image, that is, the bbox, object category, and detection confidence.

[0107] In step 3.2, for a specific bbox, the image feature points in the bbox are extracted using the orb feature point extraction algorithm, and then the BOW vector v of the bbox is generated using the DBoW algorithm. For the obtained BOW vector v, it is matched with the vectors in the BOW vector library of objects of the same object category as the bbox obtained in step 2. For an object, if its BOW vector library has a matching score with the BOW vector v that is higher than the threshold, the object is added to the candidate object sequence of the bbox.

[0108] Step 3.3, screen candidate objects based on the co-viewing relationship of objects observed in the lost frame:

[0109] Specifically, for one of the bboxes a, it is considered that a is co-visualized with other objects observed in the lost frame. For the candidate object A corresponding to a, it should have a similar co-visualized relationship with a; specifically:

[0110] Assume that the lost frame observes an object a and an object of category x (this classification is given by target detection) at the same time. If A in the common view adjacency table obtained in step 2 is common with the object classified as x, the common view score of the candidate object A is +1; the object corresponding to the bbox is determined to have the highest common view score, and the correspondence between the object and the bbox is obtained.

[0111] Step 4: Using the correspondence between the object and the detection box bbox obtained in step 3, the initial camera pose is estimated through the detection box bbox and the corresponding object ellipsoid.

[0112] See also Figure 2 , which is the detailed implementation process of step 4, is as follows:

[0113] As in step 2, the ellipsoid can be described in the dual space as π T Q * π=0.

[0114] A dual quadratic surface is a quadratic surface defined by a tangent plane. Ideally, the bbox should be the circumscribed rectangle of the projected ellipse of the ellipsoid on the imaging plane. That is, the plane formed by the optical center and any edge of the bbox on the imaging plane should be the tangent plane of the corresponding ellipsoid. Based on the projected line on the plane and the camera matrix, the spatial plane passing through the optical center can be recovered, that is, the envelope tangent plane of the ellipsoid:

[0115] π=PT I (8)

[0116] Among them, P T is the transpose of P; P = K[R|t]; R is the rotation matrix of the camera; t is the translation vector of the camera; K is the camera intrinsic parameter; I is the edge vector of the bbox.

[0117] From formula (2) and formula (8), we can get:

[0118] (P T I) T Q * (P T I)=0 (9)

[0119] The four edges of a bbox can recover four enveloping sections, that is, they can provide four plane constraints. Therefore, only two or more bboxes are needed to solve the above equation P, that is, to obtain the camera's 6-DoF pose [R|t]. After obtaining the correspondence between the object and the bbox in step 3, the dual ellipse Q of the object corresponding to the bbox edge I in equation 9 can be obtained. * .

[0120] Here, when the number of observed objects is greater than or equal to 3, multiple iterations are performed to calculate the pose. In each iteration, three pairs of ellipse-ellipsoids are selected to calculate and solve equation (9) in the above manner to obtain a 6-DoF pose of the camera. For the pose calculated in each iteration, the ellipsoid can be reprojected onto the camera imaging plane as equation (3) to obtain a projected ellipse. The circumscribed rectangle of the projected ellipse is calculated, and the sides of the rectangle are parallel to the sides of the bbox detection frame. The distances between the sides of the rectangle and the sides of the bbox are calculated as the cost of the pose. Finally, the pose with the smallest cost in multiple iterations is selected as the calculated initial pose. If the number of observed objects in the lost frame is less than 3, the relocalization fails.

[0121] In step 5, after obtaining the initial pose estimate, the pose is optimized using the iterative closest point registration algorithm (ICP algorithm) to achieve global relocalization of the robot.

[0122] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A visual relocalization method based on object semantics, characterized in that: include: The robot uses an RGB-D camera to capture images of the environment, processes the images using a target detection algorithm, and obtains detection results for each object in the image. By modeling the detection boxes of the same object detected in multiple frames, the robot obtains a 3D ellipsoid representation of the object and its 6-DoF pose in 3D space. Build a bag-of-words database for each object and a co-viewing adjacency list for all objects; When the robot needs to relocalize, it first combines the target detection results of the lost frame image and determines the corresponding candidate objects of the object observed in the lost frame by querying the bag-of-words database and the common view relationship table, thereby obtaining the correspondence between the object and the detection box in the lost frame image; Using the correspondence between the object and the detection frame, the initial camera pose is estimated through the detection frame and the corresponding object ellipsoid; After obtaining the initial pose estimate, the pose is optimized using the iterative closest point registration algorithm to achieve global relocalization of the robot. The step of constructing a bag-of-words database for each object and a co-viewing adjacency list for all objects includes: For each successfully initialized object, a BOW vector library is created. This vector library is a collection that stores BOW vectors. It is created when the object is successfully initialized and is initially an empty collection. Calculate the detection box bbox detected in this frame image and the objects corresponding to these detection box bboxes. For each detected detection box bbox, extract the image feature points in the detection box bbox using the orb feature point extraction algorithm. For all feature points in a specific detection box bbox, generate the corresponding BOW vector using the DBoW algorithm. Match the obtained BOW vector with all BOW vectors in the BOW vector library of the object corresponding to the detection box bbox. If the matching score is lower than the threshold, add the BOW vector to the BOW vector library of the object. If objects a and b are observed simultaneously in a frame of image, they are considered to be in common view. All objects observed in the current frame are considered to have a common view relationship between these objects, and these common view relationships are stored in the form of an adjacency list. The common view adjacency list is an n*n matrix, where n is the number of objects successfully initialized. The element in the i-th row and j-th column of the matrix is ​​1, indicating that objects i and j are in common view.

2. The object semantics-based visual relocalization method according to claim 1, characterized in that: The object modeling is performed on the detection frame of the same object detected in multiple frames of images to obtain a three-dimensional ellipsoid representation of the object and a 6-DoF pose in three-dimensional space, including: Using the monocular camera mounted on the robot, key frames are obtained from the camera's RGB image frame sequence by using the parallax between image frames. The key frame images are processed using the YOLOv5 algorithm to obtain the detection results of each object in the image. Determine the data association between frames; by checking whether the center point of the previous frame's detection box bbox falls within the current frame's detection box bbox, it can be confirmed whether the observed object is the same; Determine the data association between frames and objects; the object refers to the ellipsoid representation obtained by solving multi-frame projection constraints for a specific object in the scene; obtain the current 6-Dof pose of the robot's monocular camera in three-dimensional space based on the RGB-D SLAM algorithm currently running on the robot, and calculate the projection of these objects on the current frame pixel image through the camera's 6-Dof pose and camera intrinsic parameters and successfully initialized objects; if the object projection center falls within a certain detection frame bbox and the object category of the object detected by the detection frame bbox is consistent with the object category of the object, then the object observed by the detection frame bbox is considered to be an object that has been successfully initialized as an object; if the object observed by the detection frame bbox is not a member of the successfully initialized objects, then the object is initialized; For an object i that has been observed in more than three consecutive key frames and has not been successfully initialized, its ellipsoid representation and 6Dof pose are estimated.

3. The object semantics-based visual relocalization method according to claim 2, characterized in that: For an object i that has been observed in more than three consecutive key frames and has not been successfully initialized, an ellipsoid representation and a 6Dof pose estimation are performed, including: and The relationship between them is given by the projection matrix R f =K[R f |t f ]definition: Among them, R f is the rotation matrix of the camera in frame f; t f is the translation vector of the camera in the f-th frame, is the ellipsoidal dual representation of object i; is the elliptical projection of object i in the fth frame; K is the camera internal parameter; β if is the scale factor of the elliptical projection of object i on the f-th frame; P f and The products are arranged in a single matrix by: We can get: remember in, is the Kronecker product; vech is the serialization of the lower triangular elements of the matrix; vec is the serialization of the matrix elements; D is the transformation matrix from vech to vec; E is the transformation matrix from vec to vech; Serialize the lower triangular elements of the matrix representing the ellipsoid of object i; Serialize the lower triangular elements of the matrix of the projection ellipse of ellipsoid i for object i in frame f; Assume that there are F image frames stacked column by column, F>=3, then: M i w i =0 (6) Among them G1...G F is the G of each frame calculated according to formula (5) f ; Serialize the lower triangular elements of the matrix of the ellipsoidal projection of object i in the first frame; is the serialization of the lower triangular elements of the matrix representing the ellipsoid of object i; i =[β i1 ,β i2 ....β iF ] T , β i1 is the scale factor of the elliptical projection of object i on the first frame; Solving the overdetermined equation (6) yields w i , due to the dual ellipse is a symmetric matrix, through The sequence of lower triangular elements of the matrix You can restore In this way, we get the ellipsoid expression of object i in the dual space; General ellipsoid Q in dual space * can be rewritten as: Where Z is a homogeneous transformation; is an ellipsoid centered at the origin and with its axes aligned with the 3D coordinates; R(θ) is the rotation matrix of the ellipsoid; t is the translation vector of the ellipsoid; a, b, c are the lengths of the ellipsoid in the xyz coordinate directions; This rewrite can be done by * By performing eigenvalue decomposition, we can obtain The operation can obtain the three-dimensional ellipsoid representation of object i and its 6-Dof pose in three-dimensional space; at this time, a new object is initialized successfully, and an ID number is assigned to this object to distinguish different objects.

4. The object semantics-based visual relocalization method according to claim 1, characterized in that: The target detection result of the lost frame image is combined with querying the bag-of-words database and the common view relationship table to determine the corresponding candidate object of the object observed in the lost frame, thereby obtaining the correspondence between the object and the detection frame in the lost frame image, including: When the SLAM algorithm cannot calculate the current robot pose, the image at this time is defined as a lost frame image. The lost frame image is processed by the YOLOv5 algorithm to obtain the target detection results of each object in the image; For a specific detection box bbox, the image feature points in the detection box bbox are extracted using the orb feature point extraction algorithm, and then the BOW vector v of the detection box bbox is generated using the DBoW algorithm; for the obtained BOW vector v, it is matched with the vectors in the BOW vector library of objects of the same object category as the detection box bbox; for an object, if its BOW vector library has a matching score with the BOW vector v that is higher than the threshold, the object is added to the candidate object sequence of the detection box bbox; Candidate objects are screened by the co-viewing relationship of objects observed in the lost frame: Assume that the lost frame observes A and an object of object category x at the same time. If A is co-visible with the object classified as x in the co-visibility adjacency table, the co-visibility score of the candidate object A is +1; the object corresponding to the detection box bbox is determined to have the highest co-visibility score, so that the correspondence between the object and the detection box bbox is obtained.

5. The object semantics-based visual relocalization method according to claim 1, characterized in that: Using the correspondence between the object and the detection frame, the initial camera pose is estimated using the detection frame and the corresponding object ellipsoid, including: The four edges of a detection box bbox can recover four envelope sections, so only two or more detection boxes are needed to solve the following equation P, that is, to obtain the camera's 6-DoF pose [R|t]; after obtaining the correspondence between the object and the detection box bbox, the dual ellipse Q of the object corresponding to the detection box bbox edge I in equation (9) can be obtained * : (P T I) T Q * (P T I)=0 (9) Where P = K[p|t], K is the camera intrinsic parameter, and I is the edge vector of the detection box bbox; When the number of observed objects is greater than or equal to 3, multiple iterations are performed to calculate the pose. In each iteration, three pairs of ellipse-ellipsoids are selected to obtain a 6-DoF pose of the camera by solving equation (9). For the pose calculated in each iteration, the ellipsoid is reprojected onto the camera imaging plane to obtain a projected ellipse. The circumscribed rectangle of the projected ellipse is calculated, and the sides of the circumscribed rectangle are parallel to the sides of the detection frame bbox. The distances between the sides of the rectangle and the sides of the detection frame bbox are calculated as the cost of the pose. Finally, the pose with the smallest cost in multiple iterations is selected as the calculated initial pose. If the number of observed objects in the lost frame is less than 3, the relocalization fails.

6. The object semantics-based visual relocalization method according to claim 1, characterized in that: The robot should simultaneously run the RGB-D SLAM algorithm to provide camera pose data for the 3D ellipsoid generation algorithm and build a dense map.

Images