Object-level semantic relocation method and device, terminal and storage medium

Through the object-level semantic relocation method, the feature matching is performed using visual word vectors and object semantic information, which solves the problem of inaccurate positioning in the existing SLAM system under environmental changes and lighting conditions, and improves the success rate and accuracy of relocation. Enhanced system robustness.

CN118968514AActive Publication Date: 2024-11-15SOUTHERN UNIVERSITY OF SCIENCE AND TECHNOLOGY

Patent Information

Application Number
CN202410907116.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-15
Estimated Expiration
2044-07-08

AI Technical Summary

Technical Problem

During the relocation process, existing visual SLAM systems are prone to inaccurate positioning or failure due to noise, error, environmental changes or changes in lighting conditions, especially in scenarios where features are not rich or difficult to match.

Method used

Using object-level semantic relocation method, by extracting the visual word vectors and object semantic information of the current image, combining the pre-constructed object-level positioning map, the relocation candidate frames are selected, and feature point matching and object semantic matching are performed to calculate the current bit posture.

Benefits of technology

The success rate and accuracy of relocation are improved, especially in the case of changes in lighting conditions and perspective changes, stable relocation can be relocated by the semantic features of the object, which enhances the robustness and stability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968514A_ABST
    Figure CN118968514A_ABST
Patent Text Reader

Abstract

The invention provides an object-level semantic relocation method and device, a terminal and a storage medium. The method comprises the following steps: extracting a visual word vector of a current image and corresponding current object semantic information; obtaining a first key frame candidate set according to a pre-constructed object level positioning map and the visual word vector, and obtaining a second key frame candidate set according to the object level positioning map and the current object semantic information; determining common key frames in the first key frame candidate set and the second key frame candidate set as relocation candidate frames; performing feature point matching on the current image and the relocation candidate frame, and calculating a current pose; wherein the object level positioning map carries object semantic information. According to the method, the semantic information of the current object in the current image is extracted, and the semantic information of the object is added into the object-level positioning map, so that repositioning can still be performed by using the semantic features of the object even if the illumination condition is remarkably changed, and the success rate of repositioning is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of relocation technology, and in particular to an object-level semantic relocation method, device, terminal and storage medium. Background Art

[0002] SLAM (Simultaneous Localization and Mapping), that is, simultaneous positioning and map construction. The existing mature visual SLAM solutions are basically based on geometric feature relocation methods. This technology mainly relies on extracting and matching geometric features in images, such as corners, line and surface features, etc. Representatives of such systems include ORB-SLAM3, VINS-Fusion, etc. ORB-SLAM3 is the first SLAM system that has pure visual data processing, visual + inertial data processing, multi-map building functions, supports monocular, binocular and RGB-D cameras, and supports pinhole cameras and fisheye camera models. At the same time, ORB-SLAM3 is also the first visual and visual inertial system to associate short-term, medium-term and long-term data. Compared with other representative visual and visual inertial systems, it has obvious advantages in robustness and accuracy, and can run stably and in real time in small and large, indoor and outdoor environments. When the ORB-SLAM3 system is running, it first needs to extract feature points from the image. These feature points are usually locations with significant gradient changes or textures in the image. Common feature point extraction algorithms include Harris corner detection, FAST corner detection, ORB (Oriented FAST and Rotated BRIEF) features, etc. These algorithms are designed to extract unique features that can be robustly matched between different viewpoints. For each extracted feature point, the system generates a descriptor, which is an abstract representation of the area around the point. Common descriptors include BRIEF, ORB, SIFT (Scale-Invariant Feature Transform), etc.

[0003] When the camera needs to be relocated in a new scene, the system uses the feature points and descriptors extracted from the current frame to match the previously constructed map. This process is called feature matching. Classic matching methods include the Bag of Words (BOW) algorithm. During the matching process, the system calculates the current camera pose through the feature point pairs on the feature matching to ensure that the current frame correctly corresponds to a key frame in the previously constructed map. Through the matched camera poses, the system may obtain a preliminary global pose estimate. However, in practice, due to noise and errors, this estimate may not be accurate enough. Therefore, optimization algorithms (such as Bundle Adjustment) are usually used to globally optimize the camera trajectory and map to improve the accuracy of positioning.

[0004] However, in the relocalization process, most existing visual SLAMs use geometric features for positioning and map construction. If the features in the scene are not rich enough or difficult to match, relocalization will fail. In addition, if the features in the scene change, such as objects moving or the scene structure changing, or the brightness of the environment is very different, the system may fail during the relocalization process because the previously established map information may no longer be accurate; if relocalization is performed in different seasons or weather conditions, such as from summer to winter, the appearance of the scene may change significantly, which may make the previous map information unable to adapt to the new environment, thereby affecting the success rate of relocalization.

[0005] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide an object-level semantic relocation method, device, terminal and storage medium in view of the above-mentioned defects of the prior art, aiming to solve the problem that the relocation process of the prior art easily leads to relocation failure.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] An object-level semantic relocalization method, wherein the method comprises:

[0009] Extract the visual word vector of the current image and the corresponding semantic information of the current object;

[0010] Obtaining a first keyframe candidate set according to a pre-constructed object-level positioning map and the visual word vector, and obtaining a second keyframe candidate set according to the object-level positioning map and the current object semantic information;

[0011] Determine a key frame common to the first key frame candidate set and the second key frame candidate set as a repositioning candidate frame;

[0012] Perform feature point matching on the current image and the relocation candidate frame to calculate the current pose;

[0013] The object-level positioning map carries object semantic information.

[0014] In the embodiment of the present application, extracting the visual word vector of the current image and the corresponding current object semantic information includes:

[0015] Extract feature points of the current image, generate descriptors based on the feature points, and map the descriptors into visual word vectors;

[0016] The current object semantic information in the current image is extracted, wherein the current object semantic information includes object category, object size and object rotation angle.

[0017] In the embodiment of the present application, the object-level positioning map includes: a feature point correspondence relationship, wherein the feature point correspondence relationship is a correspondence relationship between a feature point and a key frame having the feature point;

[0018] The step of obtaining a first key frame candidate set according to the pre-constructed object-level positioning map and the visual word vector includes:

[0019] Using a pre-built bag-of-words model, searching for a target feature point that matches the visual word vector in the feature point correspondence;

[0020] A plurality of target key frames corresponding to the target feature points in the feature point correspondence are searched to obtain a first key frame candidate set.

[0021] In the embodiment of the present application, the object-level positioning map further includes: an object semantic correspondence, wherein the object semantic correspondence is a correspondence between object semantic information and a key frame having the object semantic information;

[0022] The obtaining a second key frame candidate set according to the object-level positioning map and the current object semantic information includes:

[0023] Matching the current object semantic information with the object semantic information in the object semantic correspondence to obtain target object semantic information that matches the current object semantic information;

[0024] A plurality of target key frames corresponding to the target object semantic information in the object semantic correspondence are searched to obtain a second key frame candidate set.

[0025] In an embodiment of the present application, after obtaining a first keyframe candidate set according to the pre-constructed object-level positioning map and the visual word vector, and obtaining a second keyframe candidate set according to the object-level positioning map and the current object semantic information, the method further includes:

[0026] If there is no common key frame in the first key frame candidate set and the second key frame candidate set, a repositioning candidate frame is determined in the second key frame candidate set according to a preset similarity threshold.

[0027] In an embodiment of the present application, feature point matching is performed on the current image and the relocation candidate frame to calculate the current posture, including:

[0028] Performing feature point matching between the current image and the relocation candidate frame;

[0029] If the number of feature point matches between the current image and the relocation candidate frame is greater than a preset threshold, the current pose is calculated using a PnP algorithm;

[0030] If the number of feature point matches between the current image and the repositioning candidate frame is less than or equal to a preset threshold, the pose of the ellipse and ellipsoid pair between the current image and the repositioning candidate frame is estimated to obtain an initial estimated pose, and the initial estimated pose is used to match feature points of the current image and the repositioning candidate frame, and the optimized pose is calculated by the PnP algorithm.

[0031] In the embodiment of the present application, the object semantic information in the object semantic correspondence is represented as ellipsoids corresponding to a number of objects, and the current object semantic information in the current image is represented as ellipses corresponding to a number of objects;

[0032] The step of estimating the pose of the ellipse-ellipsoid pair between the current image and the relocation candidate frame to obtain an initial estimated pose comprises:

[0033] Constructing a plurality of ellipse ellipsoid pairs according to the semantic information of the current object in the current image and the semantic information of the object in the relocation candidate frame;

[0034] The camera pose is calculated using the P3P algorithm on the ellipse and ellipsoid center to obtain four pose solutions, and the pose solution with the minimum cost is selected as the initial estimated pose.

[0035] The embodiment of the present application further discloses an object-level semantic relocation device, wherein the device comprises:

[0036] An extraction unit, used to extract the visual word vector of the current image and the corresponding current object semantic information;

[0037] An acquisition unit, configured to obtain a first keyframe candidate set according to a pre-constructed object-level positioning map and the visual word vector, and to obtain a second keyframe candidate set according to the object-level positioning map and the current object semantic information;

[0038] A determining unit, configured to determine a common key frame in the first key frame candidate set and the second key frame candidate set as a repositioning candidate frame;

[0039] A calculation unit, used for matching feature points between the current image and the relocation candidate frame to calculate the current posture;

[0040] The object-level positioning map carries object semantic information.

[0041] An embodiment of the present application also discloses a terminal, comprising: a memory, a processor, and an object-level semantic relocation program stored in the memory and executable on the processor, wherein the object-level semantic relocation program implements the steps of the object-level semantic relocation method as described above when executed by the processor.

[0042] The embodiment of the present application further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the object-level semantic relocation method as described above.

[0043] The present invention provides an object-level semantic relocalization method, device, terminal and storage medium, the method comprising: extracting the visual word vector of the current image and the corresponding current object semantic information; obtaining a first keyframe candidate set according to a pre-constructed object-level positioning map and the visual word vector, and obtaining a second keyframe candidate set according to the object-level positioning map and the current object semantic information; determining the keyframe common to the first keyframe candidate set and the second keyframe candidate set as a relocalization candidate frame; matching the feature points of the current image with the relocalization candidate frame to calculate the current posture; wherein the object-level positioning map carries the object semantic information. The present invention extracts the current object semantic information in the current image and adds the object semantic information to the object-level positioning map, so that the semantic features of the object can still be used for relocalization even when the lighting conditions change significantly, thereby improving the success rate of relocalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a preferred embodiment of the object-level semantic relocation method of the present invention;

[0045] Figure 2 It is a logical schematic block diagram of a preferred embodiment of the object-level semantic relocation method of the present invention;

[0046] Figure 3 It is a functional principle block diagram of a preferred embodiment of the object-level semantic relocation device of the present invention;

[0047] Figure 4 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] The relocalization process of the existing technology is prone to the problem of low relocalization accuracy, which is largely because the geometric features relied on in the existing visual SLAM system are too single and only geometric information is stored in the map.

[0050] In the embodiment of the present application, the object semantic information of unique landmarks in the environment (usually object landmarks that are not easy to change, such as tree trunks and public chairs) is added to the geometric sparse map to enrich the map's expressiveness. This approach can provide more semantic context and help the system better understand the environment. After adding such object semantic information to the map, the system can rely on more semantic features in addition to geometric features during relocation, thereby improving its relocation success rate and accuracy in the actual environment.

[0051] See also Figure 1 , Figure 1 Flowchart of the object-level semantic relocation method in the present invention. Figure 1 As shown, the object-level semantic relocation method described in the embodiment of the present invention includes:

[0052] Step S100: extracting the visual word vector of the current image and the corresponding semantic information of the current object.

[0053] In the embodiment of the present application, the step S100 specifically includes:

[0054] Step S110, extracting feature points of the current image, generating descriptors according to the feature points, and mapping the descriptors into visual word vectors;

[0055] Step S120: extracting semantic information of the current object in the current image, wherein the semantic information of the current object includes object category, object size and object rotation angle.

[0056] The embodiment of the present application takes into account both feature point matching and object semantic matching, so as to facilitate filtering key frames through feature point matching and filtering key frames through object semantic matching. When relocating, it can rely on more semantic features in addition to geometric features, thereby improving the success rate and accuracy of relocation.

[0057] like Figure 1 As shown, the object-level semantic relocation method described in this embodiment also includes:

[0058] Step S200: obtaining a first keyframe candidate set according to a pre-constructed object-level positioning map and the visual word vector, and obtaining a second keyframe candidate set according to the object-level positioning map and the current object semantic information.

[0059] Specifically, the embodiment of the present application first needs to create an object-level semantic visual SLAM system to obtain a more accurate object-level positioning map. The front end of the SLAM system relies on the camera pose and 3D points provided by the visual front end of ORB-SLAM2, and at the same time inserts the semantic information of the object detected in the picture into the map to form an object-level positioning map.

[0060] The main steps of the object creation process include object representation and initialization, object data association, etc.

[0061] For the object representation and initialization steps, specifically in the object detection module, the YOLOv8 object detection model is used to obtain the semantic information of the object on the image, and the output result is the object bounding box and the corresponding category probability. In the present invention, the object is represented as a parameterized ellipsoid and observed in an image with an ellipse. The ellipsoid is a rough but lightweight representation that requires only nine parameters: three parameters for the size of its representation axis, three for its direction, and three for its position. In addition, the ellipsoid is projected as an ellipse at any viewpoint, and its equation can be expressed in a closed form using a dual space. In this space, the ellipsoid is defined by a 4×4 matrix Q*, and the ellipse is defined by a 3×3 matrix C*, which are connected together by a projection matrix P:

[0062] C * =PQ * P T ;

[0063] Among them, P=KT is the camera projection matrix, which includes the intrinsic parameter matrix K and the camera extrinsic parameter matrix T.

[0064] The position of the ellipsoid is triangulated from the center of the associated box, and the size of the ellipsoid semi-axes is determined by the average bounding box size projected onto the central location:

[0065]

[0066] Among them, t iz is the depth of the ellipsoid center in the i-th camera coordinates, w i and h i is the pixel width and height of the 2D bounding box, f x and f y is the focal length of the camera and n is the number of observations associated with the object.

[0067] For the object data association step, the object is tracked on 2D frames based on bounding box overlap and label consistency before reconstructing the ellipsoid. Once 2D tracking of the object has been successfully performed on a sequence with sufficient baseline, longer-term tracking can be obtained by considering its 3D reconstruction. To this end, its elliptic curve model is projected in the current frame and its overlap with the object detection in that frame is used to establish the association. In both cases, the optimal association is found using the Hungarian algorithm, a well-known method for solving the assignment problem. The algorithm maximizes the total score of the matching in order to find the best possible assignment between N detections and M objects. Its score matrix at frame t can be defined as:

[0068]

[0069] in, is the ith detection in the current frame, is the latest bounding box associated with object j, is the ellipsoid corresponding to the jth object. The IoU, box, and proj operations represent the calculation of the intersection over union, closed bounding box, and projection in the current frame, respectively. The IoU term on the left refers to 2D bounding box tracking, and the IoU term on the right represents long-term tracking via ellipsoid projection, which is only possible when the initial reconstruction of the object is available.

[0070] In an embodiment of the present application, the object-level positioning map includes: a feature point correspondence relationship, wherein the feature point correspondence relationship is a correspondence relationship between a feature point and a key frame having the feature point. The first key frame candidate set is obtained according to the pre-constructed object-level positioning map and the visual word vector, including:

[0071] Using a pre-built bag-of-words model, searching for a target feature point that matches the visual word vector in the feature point correspondence;

[0072] A plurality of target key frames corresponding to the target feature points in the feature point correspondence are searched to obtain a first key frame candidate set.

[0073] Specifically, for the current frame, when matching feature points, extract the descriptor and map it to visual words, and then use the built bag-of-words model to quickly find the initial keyframe candidate set similar to the current frame. For the searched keyframe candidate set, you need to set the corresponding similarity threshold to select the most similar keyframe from it to form the first keyframe candidate set. Please refer to Figure 2 , the similarity threshold of feature point matching is N1. When the number of matching points is less than N1, the frame image is discarded.

[0074] The embodiment of the present application selects key frames by matching feature points to obtain a first key frame candidate set, which is then compared with the candidate set obtained by semantic matching of the object.

[0075] In an embodiment of the present application, the object-level positioning map further includes: an object semantic correspondence, wherein the object semantic correspondence is a correspondence between object semantic information and a key frame having the object semantic information. The second key frame candidate set is obtained according to the object-level positioning map and the current object semantic information, including:

[0076] Matching the current object semantic information with the object semantic information in the object semantic correspondence to obtain target object semantic information that matches the current object semantic information;

[0077] A plurality of target key frames corresponding to the target object semantic information in the object semantic correspondence are searched to obtain a second key frame candidate set.

[0078] Specifically, the current frame obtains the corresponding object semantic information through the object detection module. The semantic information includes category, size, rotation angle, etc. This information is used to search for the corresponding similar frame set in the object semantic correspondence. The similarity score between the two frames is obtained by calculating the overlapping area of ​​the ellipse in the current image detection frame and the corresponding ellipsoid projection of the similar frame. Among these similar frames, it is also necessary to set a similarity threshold to filter out the final candidate key frame set. Please refer to Figure 2 , the similarity threshold of object semantic matching is N2. When the number of matching points is less than N2, the frame image is discarded.

[0079] The embodiment of the present application combines object semantic information, making the system more stable when facing complex environments.

[0080] like Figure 1 As shown, the object-level semantic relocation method described in this embodiment also includes:

[0081] Step S300: Determine a common key frame in the first key frame candidate set and the second key frame candidate set as a repositioning candidate frame.

[0082] Specifically, the embodiment of the present application utilizes a relocation candidate keyframe screening strategy, implements feature point matching based on visual word screening, and implements object semantic matching based on object semantic information screening. In addition, the first keyframe candidate set is cross-matched with the second keyframe candidate set, and the common keyframes that appear in both sets are selected as the final relocation candidate frames.

[0083] The embodiment of the present application integrates object semantic information by utilizing the common key frames in the first key frame candidate set and the second key frame candidate set, and can more robustly restore the pose of the current frame in difficult scenes.

[0084] In the embodiment of the present application, the step S200 further includes: if there is no common key frame in the first key frame candidate set and the second key frame candidate set, determining a repositioning candidate frame in the second key frame candidate set according to a preset similarity threshold.

[0085] Specifically, if no common frames are found in the two sets, a higher threshold is set from the key frames filtered by semantic information to obtain the final candidate frames.

[0086] The embodiment of the present application selects a more accurate second key frame candidate set when no common frame is found in the two sets, thereby improving the accuracy of repositioning.

[0087] like Figure 1 As shown, the object-level semantic relocation method described in this embodiment also includes:

[0088] Step S400: performing feature point matching on the current image and the relocation candidate frame to calculate the current posture; wherein the object-level positioning map carries object semantic information.

[0089] In the embodiment of the present application, step S400 includes:

[0090] Step S410, performing feature point matching between the current image and the relocation candidate frame;

[0091] Step S420: If the number of feature point matches between the current image and the relocation candidate frame is greater than a preset threshold, the current pose is calculated using a PnP algorithm;

[0092] Step S430: If the number of feature point matches between the current image and the repositioning candidate frame is less than or equal to a preset threshold, the pose of the ellipse-ellipsoid pair between the current image and the repositioning candidate frame is estimated to obtain an initial estimated pose, and the initial estimated pose is used to match feature points between the current image and the repositioning candidate frame, and the optimized pose is calculated by the PnP algorithm.

[0093] Specifically, if the number of feature point matches between the current image and the relocation candidate frame is greater than a preset threshold, the relocation candidate frame needs to match feature points with the current frame to find point pairs with similar features between the two images. This process requires the use of the RANSAC (Random Sample Consensus) algorithm to remove bad matching points to increase the accuracy of subsequent pose estimation. Pose estimation requires the use of matching point pairs between the current frame and the corresponding relocation candidate frame through the PnP algorithm. After the pose is estimated, the BA (Bundle Adjustment) optimization algorithm is used to optimize the pose.

[0094] In an embodiment of the present application, the object semantic information in the object semantic correspondence is represented as an ellipsoid corresponding to a plurality of objects, and the current object semantic information in the current image is represented as an ellipse corresponding to a plurality of objects. The pose estimation of the ellipse-ellipsoid pair between the current image and the relocation candidate frame to obtain an initial estimated pose includes:

[0095] Constructing a plurality of ellipse ellipsoid pairs according to the semantic information of the current object in the current image and the semantic information of the object in the relocation candidate frame;

[0096] The camera pose is calculated using the P3P algorithm on the ellipse and ellipsoid center to obtain four pose solutions, and the pose solution with the minimum cost is selected as the initial estimated pose.

[0097] Specifically, ORB-SLAM2's original relocalization method provides good reliability, but often fails when the current frame is far from the past camera trajectory or when the illumination changes greatly. In this case, using a method based on object semantic information can enhance the performance of visual relocalization because it is more robust to viewpoint or illumination changes.

[0098] Ellipse-ellipsoid pairs are built based on their categories, and in each iteration, the smallest set of three pairs is selected and the camera pose is calculated using the P3P algorithm on the ellipse and ellipsoid centers. P3P provides four potential pose solutions. For each pose, the ellipsoid is projected and associated with the detection based on its overlap. The cost is calculated as the sum of 1-IoU for each associated pair, and the pose with the smallest cost among the four P3P solutions is selected as the pose calculation result.

[0099] Since the pose calculated by the PnP algorithm based on the feature point correspondence is usually more accurate than the pose solved based on the ellipsoid correspondence, the pose of the current frame obtained by the object semantic correspondence is only a rough pose of the current frame. This rough pose is enough to match the current frame with the relocation key frame to obtain more similar point pairs, and then calculate a more accurate pose through the PnP algorithm, and then the SLAM system can resume tracking.

[0100] In this way, during the relocalization process, when traditional feature point matching cannot provide enough information, the spatial relationship of objects is used to perform preliminary pose estimation. The relative position relationship of objects in space is relatively stable, so that a reliable initial pose estimation can be provided even in harsh environmental conditions.

[0101] The embodiment of the present application uses the initial estimated posture as a rough posture for feature point comparison, avoids using a random posture for feature point comparison, and improves the accuracy of repositioning.

[0102] The embodiments of the present application significantly improve the robustness and accuracy of the existing traditional SLAM relocalization solution. The specific advantages and effects are as follows:

[0103] First, it improves the robustness to lighting changes

[0104] In traditional SLAM relocation schemes, changes in lighting conditions can cause feature point detection and matching failures, which in turn affects the accuracy of pose estimation. The present invention adds object semantic information to the map, so that the semantic features of the object can still be used for relocation even when the lighting conditions change significantly. Because the semantic information of the object (such as the type and position of the object) is relatively stable and is not affected by lighting changes, the system can still perform reliable relocation under harsh lighting conditions.

[0105] Second, enhanced adaptability to perspective changes

[0106] The existing traditional SLAM relocalization technology based on geometric features may fail due to the inability to identify or match enough feature points when the viewing angle changes significantly. The present invention uses the semantic information of the object detected in the current frame to match the semantic information of the object in the map. Even if the viewing angle changes significantly, a preliminary pose estimation can still be performed based on the spatial relationship of the object. Since the relative position relationship of the object in space is relatively stable, this enables the system to maintain good relocalization performance under a wide range of viewing angle changes.

[0107] Third, the robustness and stability of the system are enhanced

[0108] The relocalization method combined with object-level semantic information makes the system more stable in complex environments. Whether it is illumination changes, perspective changes, or interference from other environmental factors, the method of the present invention can provide reliable preliminary estimates through object detection information, greatly reducing the probability of relocalization failure and improving the overall robustness of the system.

[0109] In one embodiment, if Figure 3As shown, based on the above-mentioned object-level semantic relocation method, the present invention also provides an object-level semantic relocation device, including:

[0110] An extraction unit 100 is used to extract the visual word vector of the current image and the corresponding current object semantic information;

[0111] An acquisition unit 200 is used to obtain a first key frame candidate set according to a pre-constructed object-level positioning map and the visual word vector, and to obtain a second key frame candidate set according to the object-level positioning map and the current object semantic information;

[0112] A determination unit 300, configured to determine a common key frame in the first key frame candidate set and the second key frame candidate set as a relocation candidate frame;

[0113] A calculation unit 400 is used to match feature points between the current image and the relocation candidate frame to calculate the current pose;

[0114] The object-level positioning map carries object semantic information.

[0115] Figure 4 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0116] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0117] When the processor 502 executes the program, the object-level semantic relocation method provided in the above embodiment is implemented.

[0118] Furthermore, the terminal further includes:

[0119] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0120] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0121] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0122] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0123] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0124] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0125] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the object-level semantic relocation method as described above is implemented.

[0126] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0127] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0128] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0129] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0130] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0131] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0132] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0133] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0134] In summary, the present invention discloses an object-level semantic relocalization method, device, terminal and storage medium, the method comprising: extracting the visual word vector of the current image and the corresponding current object semantic information; obtaining a first keyframe candidate set according to a pre-constructed object-level positioning map and the visual word vector, and obtaining a second keyframe candidate set according to the object-level positioning map and the current object semantic information; determining the keyframe common to the first keyframe candidate set and the second keyframe candidate set as a relocalization candidate frame; matching feature points between the current image and the relocalization candidate frame to calculate the current pose; wherein the object-level positioning map carries object semantic information. The present invention extracts the current object semantic information in the current image and adds the object semantic information to the object-level positioning map, so that even when the lighting conditions change significantly, the semantic features of the object can still be used for relocalization, thereby improving the success rate of relocalization.

[0135] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An object-level semantic relocalization method, characterized in that: The method comprises: Extract the visual word vector of the current image and the corresponding semantic information of the current object; Obtaining a first keyframe candidate set according to a pre-constructed object-level positioning map and the visual word vector, and obtaining a second keyframe candidate set according to the object-level positioning map and the current object semantic information; Determine a common key frame in the first key frame candidate set and the second key frame candidate set as a repositioning candidate frame; Perform feature point matching on the current image and the relocation candidate frame to calculate the current pose; The object-level positioning map carries object semantic information.

2. The object-level semantic relocalization method according to claim 1, characterized in that: The step of extracting the visual word vector of the current image and the corresponding semantic information of the current object includes: Extract feature points of the current image, generate descriptors based on the feature points, and map the descriptors into visual word vectors; The current object semantic information in the current image is extracted, wherein the current object semantic information includes object category, object size and object rotation angle.

3. The object-level semantic relocalization method according to claim 2, characterized in that: The object-level positioning map includes: a feature point correspondence relationship, wherein the feature point correspondence relationship is a correspondence relationship between a feature point and a key frame having the feature point; The step of obtaining a first key frame candidate set according to the pre-constructed object-level positioning map and the visual word vector includes: Using a pre-built bag-of-words model, searching for a target feature point that matches the visual word vector in the feature point correspondence; A plurality of target key frames corresponding to the target feature points in the feature point correspondence are searched to obtain a first key frame candidate set.

4. The object-level semantic relocalization method according to claim 2, characterized in that: The object-level positioning map also includes: object semantic correspondence, which is a correspondence between object semantic information and key frames having the object semantic information; The obtaining a second key frame candidate set according to the object-level positioning map and the current object semantic information includes: Matching the current object semantic information with the object semantic information in the object semantic correspondence to obtain target object semantic information that matches the current object semantic information; A plurality of target key frames corresponding to the target object semantic information in the object semantic correspondence are searched to obtain a second key frame candidate set.

5. The object-level semantic relocalization method according to claim 2, characterized in that: After obtaining the first keyframe candidate set according to the pre-constructed object-level positioning map and the visual word vector, and obtaining the second keyframe candidate set according to the object-level positioning map and the current object semantic information, the method further includes: If there is no common key frame in the first key frame candidate set and the second key frame candidate set, a repositioning candidate frame is determined in the second key frame candidate set according to a preset similarity threshold.

6. The object-level semantic relocalization method according to claim 4, characterized in that: Matching feature points between the current image and the relocation candidate frame to calculate the current pose includes: Performing feature point matching between the current image and the relocation candidate frame; If the number of feature point matches between the current image and the relocation candidate frame is greater than a preset threshold, the current pose is calculated using a PnP algorithm; If the number of feature point matches between the current image and the repositioning candidate frame is less than or equal to a preset threshold, the pose of the ellipse and ellipsoid pair between the current image and the repositioning candidate frame is estimated to obtain an initial estimated pose, and the initial estimated pose is used to match feature points of the current image and the repositioning candidate frame, and the optimized pose is calculated by the PnP algorithm.

7. The object-level semantic relocalization method according to claim 6, characterized in that: The object semantic information in the object semantic correspondence is represented as ellipsoids corresponding to a number of objects, and the current object semantic information in the current image is represented as ellipses corresponding to a number of objects; The step of estimating the pose of the ellipse-ellipsoid pair between the current image and the relocation candidate frame to obtain an initial estimated pose comprises: Constructing a plurality of ellipse ellipsoid pairs according to the semantic information of the current object in the current image and the semantic information of the object in the relocation candidate frame; The camera pose is calculated using the P3P algorithm on the ellipse and ellipsoid center, four pose solutions are obtained, and the pose solution with the minimum cost is selected as the initial estimated pose.

8. An object-level semantic relocalization device, characterized in that: The device comprises: An extraction unit, used to extract the visual word vector of the current image and the corresponding current object semantic information; An acquisition unit, configured to obtain a first keyframe candidate set according to a pre-constructed object-level positioning map and the visual word vector, and to obtain a second keyframe candidate set according to the object-level positioning map and the current object semantic information; A determining unit, configured to determine a common key frame in the first key frame candidate set and the second key frame candidate set as a repositioning candidate frame; A calculation unit, used for matching feature points between the current image and the relocation candidate frame to calculate the current posture; The object-level positioning map carries object semantic information.

9. A terminal, characterized in that: include: A memory, a processor, and an object-level semantic relocation program stored in the memory and executable on the processor, wherein the object-level semantic relocation program, when executed by the processor, implements the steps of the object-level semantic relocation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the object-level semantic relocalization method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Visual semantic database construction and global positioning method based on deep learning

    CN108830220A

  • State determination method and device for visual inertial odometer and electronic device

    CN109211277A

  • Visual repositioning method based on object semantics

    CN115830126A

  • Map relocation method and device combined with semantic information

    CN117830625A

  • Meat Loaf Cutting Machine for Meat Processed Food

    KR1020210147165A

Cited By

  • Multi-robot collaborative semantic SLAM and dynamic exploration method

    CN120765918A