An object-level semantic relocation method, device, terminal, and storage medium

Abstract

This invention provides an object-level semantic relocalization method, apparatus, terminal, and storage medium. The method includes: extracting visual word vectors and corresponding current object semantic information from the current image; obtaining a first keyframe candidate set based on a pre-constructed object-level localization map and visual word vectors, and obtaining a second keyframe candidate set based on the object-level localization map and current object semantic information; determining the common keyframes in the first and second keyframe candidate sets as relocalization candidate frames; performing feature point matching between the current image and the relocalization candidate frames to calculate the current pose; wherein the object-level localization map carries object semantic information. This invention, by extracting current object semantic information from the current image and adding object semantic information to the object-level localization map, can still utilize the semantic features of objects for relocalization even when lighting conditions change significantly, thus improving the success rate of relocalization.

Description

Technical Field

[0001] This invention relates to the field of relocation technology, and in particular to an object-level semantic relocation method, apparatus, terminal, and storage medium. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) is a system that simultaneously localizes and maps. Most existing mature visual SLAM solutions are based on geometric feature-based relocalization methods. This technology primarily relies on extracting and matching geometric features in images, such as corner points, line features, and polygon features. Representative systems in this category include ORB-SLAM3 and VINS-Fusion. ORB-SLAM3 is the first SLAM system to simultaneously support pure visual data processing, visual + inertial data processing, multi-map construction, and supports monocular, binocular, and RGB-D cameras, as well as pinhole and fisheye camera models. Furthermore, ORB-SLAM3 is the first visual and visual-inertial system to correlate short-term, medium-term, and long-term data. Compared to other representative visual and visual-inertial systems, it exhibits significant advantages in robustness and accuracy, and can operate stably in real-time in small and large, indoor and outdoor environments. When the ORB-SLAM3 system runs, it first needs to extract feature points from the image. These feature points are typically located in the image with significant gradient changes or textures. Commonly used feature point extraction algorithms include Harris corner detection, FAST corner detection, and ORB (Oriented FAST and Rotated BRIEF) features. These algorithms aim to extract unique features that can be robustly matched across different viewpoints. For each extracted feature point, the system generates a descriptor, which is an abstract representation of the region surrounding that point. Commonly used descriptors include BRIEF, ORB, and SIFT (Scale-Invariant Feature Transform).

[0003] When the camera needs to relocalize in a new scene, the system uses feature points and descriptors extracted from the current frame to match with the previously constructed map. This process is called feature matching, and classic matching methods include the Bag-of-Words (BOW) algorithm. During the matching process, the system calculates the current camera pose using the feature point pairs from the feature matching to ensure that the current frame correctly corresponds to a keyframe in the previously constructed map. Based on the matched camera pose, the system may obtain a preliminary global pose estimate. However, in practice, this estimate may not be accurate enough due to noise and errors. Therefore, optimization algorithms (such as Bundle Adjustment) are typically used to globally optimize the camera trajectory and map to improve localization accuracy.

[0004] However, during the relocalization process, most existing visual SLAM systems use geometric features for localization and map building. If the features in the scene are not rich enough or difficult to match, relocalization will fail. Furthermore, if the features in the scene change—for example, if objects move, the scene structure changes, or there is a significant difference in ambient brightness—the system may fail during relocalization because the previously built map information may no longer be accurate. If relocalization is performed under different seasons or weather conditions, such as from summer to winter, the appearance of the scene may change significantly, which may make the previous map information unsuitable for the new environment, thus affecting the success rate of relocalization.

[0005] Therefore, existing technologies have shortcomings and need to be improved and developed. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an object-level semantic relocation method, device, terminal and storage medium to address the above-mentioned defects of the prior art, and to solve the problem that the relocation process of the prior art is prone to relocation failure.

[0007] The technical solution adopted by this invention to solve the technical problem is as follows:

[0008] An object-level semantic relocation method, wherein the method includes:

[0009] Extract the visual word vectors of the current image and the corresponding semantic information of the current object;

[0010] A first keyframe candidate set is obtained based on the pre-constructed object-level localization map and the visual word vectors, and a second keyframe candidate set is obtained based on the object-level localization map and the semantic information of the current object.

[0011] The keyframes common to both the first keyframe candidate set and the second keyframe candidate set are identified as relocation candidate frames;

[0012] Feature point matching is performed between the current image and the relocation candidate frame to calculate the current pose;

[0013] The object-level localization map contains object semantic information.

[0014] In this embodiment of the application, the extraction of visual word vectors and corresponding semantic information of current objects from the current image includes:

[0015] Extract feature points from the current image, generate descriptors based on the feature points, and map the descriptors to visual word vectors;

[0016] Extract semantic information of the current object in the current image. The semantic information of the current object includes the object category, object size, and object rotation angle.

[0017] In this embodiment of the application, the object-level localization map includes: a feature point correspondence, wherein the feature point correspondence is the correspondence between a feature point and a keyframe containing the feature point;

[0018] The process of obtaining the first keyframe candidate set based on the pre-constructed object-level localization map and the visual word vectors includes:

[0019] The target feature points that match the visual word vectors are found in the feature point correspondence using a pre-built bag-of-words model.

[0020] Find several target keyframes that correspond to the target feature points in the feature point correspondence relationship to obtain the first keyframe candidate set.

[0021] In this embodiment of the application, the object-level localization map further includes: an object semantic correspondence, which is the correspondence between object semantic information and keyframes containing the object semantic information;

[0022] The step of obtaining the second keyframe candidate set based on the object-level localization map and the current object semantic information includes:

[0023] The semantic information of the current object is matched with the semantic information of the object in the semantic correspondence relationship to obtain the semantic information of the target object that matches the semantic information of the current object.

[0024] Find several target keyframes in the semantic correspondence of the object that correspond to the semantic information of the target object to obtain a second keyframe candidate set.

[0025] In this embodiment of the application, after obtaining the first keyframe candidate set based on the pre-constructed object-level localization map and the visual word vector, and obtaining the second keyframe candidate set based on the object-level localization map and the current object semantic information, the method further includes:

[0026] If there are no common keyframes in the first keyframe candidate set and the second keyframe candidate set, then a relocation candidate frame is determined in the second keyframe candidate set according to a preset similarity threshold.

[0027] In this embodiment of the application, feature point matching is performed between the current image and the relocation candidate frame to calculate the current pose, including:

[0028] Feature point matching is performed between the current image and the relocation candidate frame;

[0029] If the number of feature point matches between the current image and the relocation candidate frame is greater than a preset threshold, the current pose is calculated using the PnP algorithm.

[0030] If the number of feature point matches between the current image and the repositioning candidate frame is less than or equal to a preset threshold, then pose estimation is performed on the elliptic-ellipsoid pairs between the current image and the repositioning candidate frame to obtain an initial estimated pose. The initial estimated pose is then used to perform feature point matching between the current image and the repositioning candidate frame, and the optimized pose is calculated using the PnP algorithm.

[0031] In this embodiment of the application, the object semantic information in the object semantic correspondence is represented as an ellipsoid corresponding to several objects, and the current object semantic information in the current image is represented as an ellipse corresponding to several objects;

[0032] The step of performing pose estimation on the elliptic ellipsoid pair between the current image and the relocation candidate frame to obtain the initial estimated pose includes:

[0033] Several elliptic-ellipsoid pairs are constructed based on the semantic information of the current object in the current image and the semantic information of the object in the relocation candidate frame.

[0034] The camera pose is calculated using the P3P algorithm at the center of the ellipse and ellipsoid, resulting in four pose solutions. The pose solution with the lowest cost is selected as the initial estimated pose.

[0035] This application also discloses an object-level semantic relocation device, wherein the device includes:

[0036] The extraction unit is used to extract the visual word vectors of the current image and the corresponding semantic information of the current object;

[0037] The acquisition unit is used to obtain a first keyframe candidate set based on a pre-constructed object-level localization map and the visual word vector, and to obtain a second keyframe candidate set based on the object-level localization map and the semantic information of the current object.

[0038] The determining unit is used to determine the key frames common to the first key frame candidate set and the second key frame candidate set as relocation candidate frames;

[0039] The calculation unit is used to perform feature point matching between the current image and the relocation candidate frame to calculate the current pose;

[0040] The object-level localization map contains object semantic information.

[0041] This application also discloses a terminal, including: a memory, a processor, and an object-level semantic relocation program stored in the memory and executable on the processor. When the object-level semantic relocation program is executed by the processor, it implements the steps of the object-level semantic relocation method as described above.

[0042] This application also discloses a computer-readable storage medium storing a computer program that can be executed to implement the steps of the object-level semantic relocation method described above.

[0043] This invention provides an object-level semantic relocalization method, apparatus, terminal, and storage medium. The method includes: extracting visual word vectors and corresponding current object semantic information from a current image; obtaining a first keyframe candidate set based on a pre-constructed object-level localization map and the visual word vectors, and obtaining a second keyframe candidate set based on the object-level localization map and the current object semantic information; determining keyframes common to the first and second keyframe candidate sets as relocalization candidate frames; performing feature point matching between the current image and the relocalization candidate frames to calculate the current pose; wherein the object-level localization map carries object semantic information. This invention, by extracting current object semantic information from the current image and adding object semantic information to the object-level localization map, can still utilize the semantic features of objects for relocalization even when lighting conditions change significantly, thus improving the success rate of relocalization. Attached Figure Description

[0044] Figure 1 This is a flowchart of a preferred embodiment of the object-level semantic relocation method in this invention;

[0045] Figure 2 This is a schematic block diagram of a preferred embodiment of the object-level semantic relocation method of the present invention;

[0046] Figure 3 This is a functional principle block diagram of a preferred embodiment of the object-level semantic relocation device in this invention;

[0047] Figure 4 This is a functional principle block diagram of a preferred embodiment of the terminal in this invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0049] The problem of low accuracy in the relocalization process of existing technologies is largely due to the fact that existing visual SLAM systems rely on too few geometric features, and the map only stores geometric information.

[0050] In this embodiment, the semantic information of unique landmarks in the environment (usually tree trunks and public benches, which are not easily changed) is added to the geometrically sparse map to enrich the map's expressive power. This approach provides more semantic context, helping the system to better understand the environment. After adding such object semantic information to the map, the system can rely on more semantic features in addition to geometric features during relocalization, thereby improving its relocalization success rate and accuracy in real-world environments.

[0051] Please see Figure 1 , Figure 1 This is a flowchart of the object-level semantic relocation method in this invention. For example... Figure 1 As shown, the object-level semantic relocation method described in this embodiment of the invention includes:

[0052] Step S100: Extract the visual word vectors of the current image and the corresponding semantic information of the current object.

[0053] In this embodiment of the application, step S100 specifically includes:

[0054] Step S110: Extract feature points from the current image, generate descriptors based on the feature points, and map the descriptors to visual word vectors;

[0055] Step S120: Extract the semantic information of the current object in the current image. The semantic information of the current object includes the object category, object size, and object rotation angle.

[0056] This application embodiment considers both feature point matching and object semantic matching, so as to filter key frames by feature point matching and by object semantic matching. During relocalization, it can rely on more semantic features in addition to geometric features, thereby improving the success rate and accuracy of relocalization.

[0057] like Figure 1 As shown, the object-level semantic relocation method described in this embodiment further includes:

[0058] Step S200: Obtain a first keyframe candidate set based on the pre-constructed object-level localization map and the visual word vector, and obtain a second keyframe candidate set based on the object-level localization map and the semantic information of the current object.

[0059] Specifically, this embodiment first requires the creation of an object-level semantic visual SLAM system to obtain a relatively accurate object-level localization map. The front end of this SLAM system relies on the camera pose and 3D points provided by the visual front end of ORB-SLAM2, and simultaneously inserts the semantic information of objects detected in the image into the map to form an object-level localization map.

[0060] The main steps in the object creation process include object representation and initialization, and object data association.

[0061] The object representation and initialization steps specifically involve using the YOLOv8 object detection model in the object detection module to acquire semantic information about objects in the image. The output is the object's bounding box and its corresponding class probability. In this invention, objects are represented as parameterized ellipsoids, and observed in images with ellipses. The ellipsoid is a coarse but lightweight representation requiring only nine parameters: three for its axis size, three for its orientation, and three for its position. Furthermore, the ellipsoid projects as an ellipse at any viewpoint, and its equation can be represented in closed form using a dual space. In this space, the ellipsoid is defined by a 4×4 matrix Q*, and the ellipse by a 3×3 matrix C*, connected by a projection matrix P.

[0062] C * =PQ * P T ;

[0063] Where P = KT is the camera projection matrix, which contains the intrinsic parameter matrix K and the extrinsic parameter matrix T.

[0064] The position of the ellipsoid is obtained by triangulation from the center of the relevant bounding box, and the size of the ellipsoid's semi-axis is determined by the average size of the bounding box projected onto the center position.

[0065]

[0066] Among them, t iz w is the depth of the center of the ellipsoid in the i-th camera coordinate system. i and h i It is the pixel width and height of the 2D bounding box, f x and f y is the camera focal length, and n is the number of observations related to the object.

[0067] For the object data association step, before reconstructing the ellipsoid, the object is tracked on a 2D frame based on bounding box overlap and label consistency. Once 2D tracking of the object has been successfully performed on a sequence with sufficient baseline, longer-term tracking can be obtained by considering its 3D reconstruction. For this purpose, its elliptic curve model is projected into the current frame, and its overlap with object detections in that frame is used to establish associations. In both cases, the optimal association is found using the Hungarian algorithm, a well-known method for solving the assignment problem. This algorithm maximizes the total score of the matches to find the best possible assignment among N detections and M objects. Its score matrix at frame t can be defined as:

[0068]

[0069] in, It is the i-th detection in the current frame. It is the latest bounding box associated with object j. It is the ellipsoid corresponding to the j-th object. The IoU, box, and proj operations represent the computation of intersection-union, closed bounding box, and projection in the current frame, respectively. The IoU term on the left refers to 2D bounding box tracking, while the IoU term on the right represents long-term tracking via ellipsoid projection, which is only possible if the initial reconstruction of the object is available.

[0070] In this embodiment, the object-level localization map includes a feature point correspondence, which is the correspondence between a feature point and a keyframe containing that feature point. The step of obtaining a first keyframe candidate set based on the pre-constructed object-level localization map and the visual word vector includes:

[0071] The target feature points that match the visual word vectors are found in the feature point correspondence using a pre-built bag-of-words model.

[0072] Find several target keyframes that correspond to the target feature points in the feature point correspondence relationship to obtain the first keyframe candidate set.

[0073] Specifically, for the current frame, during feature point matching, descriptors are extracted and mapped to visual words. Then, the constructed bag-of-words model is used to quickly find an initial set of keyframe candidates similar to the current frame. For the searched keyframe candidate set, an appropriate similarity threshold needs to be set to select the most similar keyframes to form the first keyframe candidate set. Please refer to [link to relevant documentation]. Figure 2 The similarity threshold for feature point matching is N1. If the number of matching points is less than N1, the frame image is discarded.

[0074] In this application embodiment, key frames are filtered by feature point matching to obtain a first key frame candidate set, which is then compared with the candidate set obtained by object semantic matching.

[0075] In this embodiment, the object-level localization map further includes: an object semantic correspondence, wherein the object semantic correspondence is the correspondence between object semantic information and keyframes containing the object semantic information. The step of obtaining a second keyframe candidate set based on the object-level localization map and the current object semantic information includes:

[0076] The semantic information of the current object is matched with the semantic information of the object in the semantic correspondence relationship to obtain the semantic information of the target object that matches the semantic information of the current object.

[0077] Find several target keyframes in the semantic correspondence of the object that correspond to the semantic information of the target object to obtain a second keyframe candidate set.

[0078] Specifically, the current frame obtains corresponding object semantic information through the object detection module. This semantic information includes category, size, rotation angle, etc. This information is used to search for a set of similar frames within the object semantic correspondence. The similarity score between the two frames is obtained by calculating the overlapping area of ​​the ellipse projected onto the corresponding ellipsoid of the similar frame within the current image's detection bounding box. Among these similar frames, a similarity threshold is also set to filter out the final set of candidate keyframes. Please refer to [link to relevant documentation]. Figure 2 The similarity threshold for semantic matching of objects is N2. When the number of matching points is less than N2, the frame image is discarded.

[0079] The embodiments of this application incorporate object semantic information, making the system more stable when facing complex environments.

[0080] like Figure 1 As shown, the object-level semantic relocation method described in this embodiment further includes:

[0081] Step S300: Determine the common keyframes in the first keyframe candidate set and the second keyframe candidate set as relocation candidate frames.

[0082] Specifically, this application embodiment utilizes a relocation candidate keyframe filtering strategy, achieving feature point matching based on visual word filtering and object semantic matching based on object semantic information filtering. Furthermore, the first keyframe candidate set and the second keyframe candidate set are cross-matched, and common keyframes appearing in both sets are selected as the final relocation candidate frames.

[0083] This application embodiment utilizes keyframes shared by the first keyframe candidate set and the second keyframe candidate set to incorporate object semantic information, enabling more robust recovery of the pose of the current frame in difficult scenarios.

[0084] In this embodiment of the application, after step S200, the method further includes: if there are no common keyframes in the first keyframe candidate set and the second keyframe candidate set, then a relocation candidate frame is determined in the second keyframe candidate set according to a preset similarity threshold.

[0085] Specifically, if no common frame is found in the two sets, a higher threshold is set from the keyframes filtered by semantic information to obtain the final candidate frame.

[0086] In this application embodiment, when no common frame is found in the two sets, a more accurate second keyframe candidate set is selected, thereby improving the accuracy of relocation.

[0087] like Figure 1 As shown, the object-level semantic relocation method described in this embodiment further includes:

[0088] Step S400: Perform feature point matching on the current image and the relocation candidate frame to calculate the current pose; wherein, the object-level localization map carries object semantic information.

[0089] In this embodiment of the application, step S400 includes:

[0090] Step S410: Perform feature point matching between the current image and the relocation candidate frame;

[0091] Step S420: If the number of feature point matches between the current image and the relocation candidate frame is greater than a preset threshold, then the current pose is calculated using the PnP algorithm;

[0092] Step S430: If the number of feature point matches between the current image and the repositioning candidate frame is less than or equal to a preset threshold, then pose estimation is performed on the elliptic-ellipsoid pairs between the current image and the repositioning candidate frame to obtain an initial estimated pose, and feature point matching is performed between the current image and the repositioning candidate frame using the initial estimated pose, and the optimized pose is calculated using the PnP algorithm.

[0093] Specifically, if the number of feature point matches between the current image and the repositioning candidate frame exceeds a preset threshold, the repositioning candidate frame needs to be matched with the current frame to find point pairs with similar features between the two images. This process requires the use of the RANSAC (Random Sample Consensus) algorithm to remove bad matching points, thereby increasing the accuracy of subsequent pose estimation. Pose estimation requires using the matching point pairs between the current frame and the corresponding repositioning candidate frame to estimate the pose using the PnP algorithm. After estimating the pose, the BA (Bundle Adjustment) optimization algorithm is used to optimize the pose.

[0094] In this embodiment of the application, the object semantic information in the object semantic correspondence is represented as an ellipsoid corresponding to several objects, and the current object semantic information in the current image is represented as an ellipse corresponding to several objects. The step of performing pose estimation on the elliptic-ellipsoid pairs between the current image and the relocation candidate frames to obtain an initial estimated pose includes:

[0095] Several elliptic-ellipsoid pairs are constructed based on the semantic information of the current object in the current image and the semantic information of the object in the relocation candidate frame.

[0096] The camera pose is calculated using the P3P algorithm at the center of the ellipse and ellipsoid, resulting in four pose solutions. The pose solution with the lowest cost is selected as the initial estimated pose.

[0097] Specifically, the original ORB-SLAM2 relocalization method offers good reliability, but it often fails when the current frame deviates significantly from past camera trajectories or when there are large changes in lighting. In such cases, using a method based on object semantic information can enhance the performance of visual relocalization, as this method is more robust to changes in viewpoint or lighting.

[0098] Ellipse-ellipsoid pairs are established based on their categories. In each iteration, the set of three pairs with the smallest values ​​is selected, and the camera pose is calculated using the P3P algorithm at the centers of the ellipse and ellipsoid. P3P provides four potential pose solutions. For each pose, the ellipsoid is projected and associated with the detection based on its overlap relationship. Cost is calculated as the sum of 1-IoU for each associated pair, and the pose with the lowest cost among the four P3P solutions is selected as the pose calculation result.

[0099] Since the pose calculated using the PnP algorithm based on feature point correspondences is usually more accurate than the pose calculated based on ellipsoidal correspondences, the pose of the current frame obtained through object semantic correspondences is only a coarse pose. This coarse pose is sufficient to match the current frame with the relocalization keyframe to obtain more similar point pairs, and then the PnP algorithm is used to calculate a more accurate pose, allowing the SLAM system to resume tracking.

[0100] Thus, during the relocalization process, when traditional feature point matching cannot provide sufficient information, the spatial relationships of objects are used for preliminary pose estimation. The relative positional relationships of objects in space are relatively stable, enabling a reliable initial pose estimate even under harsh environmental conditions.

[0101] The embodiments of this application utilize the initial estimated pose as a coarse pose for feature point comparison, avoiding the use of random poses for feature point comparison and improving the accuracy of relocalization.

[0102] The embodiments of this application significantly improve the robustness and accuracy of existing traditional SLAM relocation schemes. The specific advantages and effects are as follows:

[0103] First, it improves robustness to changes in lighting conditions.

[0104] In traditional SLAM relocalization schemes, changes in lighting conditions can lead to the failure of feature point detection and matching, thus affecting the accuracy of pose estimation. This invention, however, incorporates object semantic information into the map, enabling relocalization even under significant changes in lighting conditions. Because object semantic information (such as object type and location) is relatively stable and unaffected by lighting changes, the system can still perform reliable relocalization under adverse lighting conditions.

[0105] Secondly, it enhances adaptability to changes in viewing angle.

[0106] Existing traditional SLAM relocalization techniques based on geometric features may fail when the viewpoint changes significantly due to the inability to identify or match sufficient feature points. This invention utilizes the semantic information of objects detected in the current frame and matches it with the semantic information of objects in the map. Even with significant viewpoint changes, preliminary pose estimation can still be performed based on the spatial relationships of objects. Since the relative positional relationships of objects in space are relatively stable, this allows the system to maintain good relocalization performance under large-scale viewpoint changes.

[0107] Third, it enhances the robustness and stability of the system.

[0108] By incorporating object-level semantic information into the relocalization method, the system becomes more stable in complex environments. Regardless of changes in lighting, perspective shifts, or interference from other environmental factors, the method of this invention can provide a reliable preliminary estimate using object detection information, significantly reducing the probability of relocalization failure and improving the overall robustness of the system.

[0109] In one embodiment, such as Figure 3As shown, based on the above-described object-level semantic relocation method, the present invention also provides an object-level semantic relocation device, comprising:

[0110] Extraction unit 100 is used to extract the visual word vectors of the current image and the corresponding semantic information of the current object;

[0111] The acquisition unit 200 is used to obtain a first keyframe candidate set based on a pre-constructed object-level localization map and the visual word vector, and to obtain a second keyframe candidate set based on the object-level localization map and the semantic information of the current object.

[0112] The determining unit 300 is used to determine the common key frames in the first key frame candidate set and the second key frame candidate set as relocation candidate frames;

[0113] The calculation unit 400 is used to perform feature point matching between the current image and the relocation candidate frame to calculate the current pose;

[0114] The object-level localization map contains object semantic information.

[0115] Figure 4 A schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal may include:

[0116] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0117] When the processor 502 executes the program, it implements the object-level semantic relocation method provided in the above embodiments.

[0118] Furthermore, the terminal also includes:

[0119] Communication interface 503 is used for communication between memory 501 and processor 502.

[0120] The memory 501 is used to store computer programs that can run on the processor 502.

[0121] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0122] If the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0123] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0124] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0125] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described object-level semantic relocation method.

[0126] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0127] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0128] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0129] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can read and execute instructions from or in conjunction with such an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically by optically scanning paper or other media, then editing, interpreting or otherwise processing them as necessary, and then storing them in computer memory.

[0130] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0131] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0133] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

[0134] In summary, this invention discloses an object-level semantic relocalization method, apparatus, terminal, and storage medium. The method includes: extracting visual word vectors and corresponding current object semantic information from a current image; obtaining a first keyframe candidate set based on a pre-constructed object-level localization map and the visual word vectors, and obtaining a second keyframe candidate set based on the object-level localization map and the current object semantic information; determining the common keyframes in the first and second keyframe candidate sets as relocalization candidate frames; performing feature point matching between the current image and the relocalization candidate frames to calculate the current pose; wherein the object-level localization map carries object semantic information. This invention, by extracting current object semantic information from the current image and adding object semantic information to the object-level localization map, can still utilize the semantic features of objects for relocalization even when lighting conditions change significantly, thus improving the success rate of relocalization.

[0135] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. An object-level semantic relocation method, characterized in that, The method includes: Extract the visual word vectors of the current image and the corresponding semantic information of the current object, wherein the semantic information of the current object includes the object category, object size and object rotation angle; A first keyframe candidate set is obtained based on the pre-constructed object-level localization map and the visual word vectors, and a second keyframe candidate set is obtained based on the object-level localization map and the semantic information of the current object. The keyframes common to both the first keyframe candidate set and the second keyframe candidate set are identified as relocation candidate frames; Feature point matching is performed between the current image and the relocation candidate frame to calculate the current pose; The object-level localization map contains object semantic information. The current image and the relocation candidate frame are matched for feature points to calculate the current pose, including: Feature point matching is performed between the current image and the relocation candidate frame; If the number of feature point matches between the current image and the relocation candidate frame is greater than a preset threshold, the current pose is calculated using the PnP algorithm. If the number of feature point matches between the current image and the repositioning candidate frame is less than or equal to a preset threshold, then pose estimation is performed on the elliptic-ellipsoid pairs between the current image and the repositioning candidate frame to obtain an initial estimated pose. The initial estimated pose is then used to perform feature point matching between the current image and the repositioning candidate frame, and the optimized pose is calculated using the PnP algorithm.

2. The object-level semantic relocation method according to claim 1, characterized in that, The extraction of visual word vectors and corresponding semantic information of current objects from the current image includes: Extract feature points from the current image, generate descriptors based on the feature points, and map the descriptors to visual word vectors; Extract the semantic information of the current object in the current image.

3. The object-level semantic relocation method according to claim 2, characterized in that, The object-level localization map includes: a feature point correspondence, which is the correspondence between a feature point and a keyframe containing the feature point; The process of obtaining the first keyframe candidate set based on the pre-constructed object-level localization map and the visual word vectors includes: The target feature points that match the visual word vectors are found in the feature point correspondence using a pre-built bag-of-words model. Find several target keyframes that correspond to the target feature points in the feature point correspondence relationship to obtain the first keyframe candidate set.

4. The object-level semantic relocation method according to claim 2, characterized in that, The object-level localization map also includes: object semantic correspondence, which is the correspondence between object semantic information and keyframes containing the object semantic information; The step of obtaining the second keyframe candidate set based on the object-level localization map and the current object semantic information includes: The semantic information of the current object is matched with the semantic information of the object in the semantic correspondence relationship to obtain the semantic information of the target object that matches the semantic information of the current object. Find several target keyframes in the semantic correspondence of the object that correspond to the semantic information of the target object to obtain a second keyframe candidate set.

5. The object-level semantic relocation method according to claim 2, characterized in that, After obtaining the first keyframe candidate set based on the pre-constructed object-level localization map and the visual word vector, and obtaining the second keyframe candidate set based on the object-level localization map and the current object semantic information, the method further includes: If there are no common keyframes in the first keyframe candidate set and the second keyframe candidate set, then a relocation candidate frame is determined in the second keyframe candidate set according to a preset similarity threshold.

6. The object-level semantic relocation method according to claim 1, characterized in that, The semantic information of the objects in the semantic correspondence is represented as an ellipsoid corresponding to several objects, and the semantic information of the current object in the current image is represented as an ellipse corresponding to several objects. The step of performing pose estimation on the elliptic ellipsoid pair between the current image and the relocation candidate frame to obtain the initial estimated pose includes: Several elliptic-ellipsoid pairs are constructed based on the semantic information of the current object in the current image and the semantic information of the object in the relocation candidate frame. The camera pose is calculated using the P3P algorithm at the center of the ellipse and ellipsoid, resulting in four pose solutions. The pose solution with the lowest cost is selected as the initial estimated pose.

7. An object-level semantic relocation device, characterized in that, The device includes: The extraction unit is used to extract the visual word vectors of the current image and the corresponding semantic information of the current object. The semantic information of the current object includes the object category, object size and object rotation angle. The acquisition unit is used to obtain a first keyframe candidate set based on a pre-constructed object-level localization map and the visual word vector, and to obtain a second keyframe candidate set based on the object-level localization map and the semantic information of the current object. The determining unit is used to determine the key frames common to the first key frame candidate set and the second key frame candidate set as relocation candidate frames; The calculation unit is used to perform feature point matching between the current image and the relocation candidate frame to calculate the current pose; The object-level localization map contains object semantic information. The current image and the relocation candidate frame are matched for feature points to calculate the current pose, including: Feature point matching is performed between the current image and the relocation candidate frame; If the number of feature point matches between the current image and the relocation candidate frame is greater than a preset threshold, the current pose is calculated using the PnP algorithm. If the number of feature point matches between the current image and the repositioning candidate frame is less than or equal to a preset threshold, then pose estimation is performed on the elliptic-ellipsoid pairs between the current image and the repositioning candidate frame to obtain an initial estimated pose. The initial estimated pose is then used to perform feature point matching between the current image and the repositioning candidate frame, and the optimized pose is calculated using the PnP algorithm.

8. A terminal, characterized in that, include: The device includes a memory, a processor, and an object-level semantic relocation program stored in the memory and executable on the processor, wherein the object-level semantic relocation program, when executed by the processor, implements the steps of the object-level semantic relocation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed to implement the steps of the object-level semantic relocation method as described in any one of claims 1 to 6.

Images