Map-free visual relocation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a map-free visual relocation method and device, electronic equipment, a storage medium and a program product. The map-free visual relocation method comprises the steps of obtaining a reference image and a query image; carrying out object extraction on the image based on instance segmentation to obtain instance mask information; feature point matching is carried out in the instance mask range, and high-precision matching information is obtained; calculating an essential matrix by using a matching result, and obtaining a rotation matrix and a scale-free translation vector of the camera; inputting a pre-trained depth estimation model, and performing depth estimation on the image to obtain depth information; recovering the scale of the translation vector in combination with the depth information to obtain an accurate camera attitude; and finally carrying out camera attitude estimation to obtain a visual relocation result. According to the method and the device, high-precision visual relocation can be realized under the condition that a three-dimensional map does not need to be constructed in advance, the demand on storage resources is reduced, and the location precision and robustness in a complex environment are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision and artificial intelligence technology, and in particular to a map-free visual repositioning method, device, electronic device, storage medium and program product. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Mapless relocalization technology is a technology that uses sensor data (such as visual information, lidar, IMU, etc.) to locate the object without pre-built maps. Traditional positioning methods usually rely on prior maps, such as vision-based SLAM technology, which has limitations in dynamic environments or places lacking map information. For example, in underground or indoor environments where GPS signals are unstable, traditional map-based positioning methods often fail.

[0004] Mapless relocalization technology can directly extract features from the environment for positioning, avoiding the complexity of relying on map updates, especially in unknown or changing environments. At present, vision-based mapless relocalization methods are widely used, but they still face challenges in environments with weak or uneven textures. To overcome this problem, researchers have tried to introduce multi-sensor fusion technology to improve positioning accuracy and robustness. However, how to deal with data inconsistency and noise problems and achieve high efficiency in real-time processing is still a difficulty in current technology. Summary of the invention

[0005] In view of this, the purpose of the present disclosure is to propose a map-free repositioning method, device, electronic device, storage medium and program product, which can at least solve some technical problems in the prior art to a certain extent.

[0006] Based on the above purpose, the first aspect of the exemplary embodiment of the present disclosure provides a map-free relocation method, the method comprising:

[0007] Determine sensor data, classify the data based on environmental characteristics, and obtain environmental classification information;

[0008] Extracting and restoring features of the sensor data according to the environmental classification information to obtain feature restoration information;

[0009] Inputting the feature recovery information into a pre-trained map-free relocalization model, obtaining position estimation information after processing, and optimizing the position estimation information to obtain a final positioning result;

[0010] The final positioning result is evaluated to obtain positioning accuracy and robustness evaluation results.

[0011] Based on the same inventive concept, a second aspect of the exemplary embodiment of the present disclosure provides a map-free repositioning device, including:

[0012] an environmental classification information determination module, configured to determine sensor data, classify the data based on environmental characteristics, and obtain environmental classification information;

[0013] A feature extraction and recovery information determination module, configured to extract and recover features of the sensor data according to the environmental classification information to obtain feature recovery information;

[0014] A positioning information optimization module is configured to input the feature recovery information into a pre-trained map-free repositioning model, obtain position estimation information after processing, and optimize the position estimation information to obtain a final positioning result;

[0015] The positioning result evaluation module is configured to evaluate the final positioning result to obtain positioning accuracy and robustness evaluation results.

[0016] Based on the same inventive concept, a third aspect of the exemplary embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0017] Based on the same inventive concept, a fourth aspect of the exemplary embodiments of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0018] Based on the same inventive concept, a fifth aspect of the exemplary embodiments of the present disclosure provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer executes the method described in the first aspect.

[0019] From the above description, it can be seen that the map-free relocalization method proposed in the embodiment of the present disclosure solves the limitations of traditional map-dependent relocalization technology through steps such as environmental feature classification, feature recovery, model optimization and positioning evaluation, and improves the positioning accuracy and robustness in dynamic and complex environments through map-free processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A schematic diagram of an application scenario of the map-free relocation method provided in an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a flow chart of a map-free relocation method provided by an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of the structure of a map-free repositioning device provided in an embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure; DETAILED DESCRIPTION

[0025] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0026] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present application according to the prompt message.

[0027] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0028] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of the present application. Other methods that meet the relevant laws and regulations may also be applied to the implementation method of the present application.

[0029] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0030] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, the principles and spirit of the present disclosure will be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0031] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0032] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "one" or "a" before an element does not exclude the existence of multiple such elements.

[0033] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0034] Existing visual relocalization technologies often face many challenges in low common view conditions, especially in harsh environments. Segmentation and precise positioning of target objects in images are key steps to improve relocalization accuracy. However, traditional methods usually rely on feature extraction and matching of the entire image, which can lead to reduced matching accuracy when there is a lot of background noise or severe object occlusion. Especially when the image resolution is low, the environment is complex, or the field of view is limited, the feature points of the target object are difficult to extract reliably, which greatly affects the accuracy of the relocalization task.

[0035] The inventors of the present disclosure have found that based on the existing deep learning segmentation model, the accuracy of target segmentation can be effectively improved by generating accurate instance masks, thereby providing strong support for subsequent feature point matching. Through instance-level segmentation, the interference of background noise and irrelevant objects can be reduced, ensuring that feature points are only extracted within the target object area, significantly improving the accuracy of the relocation task.

[0036] In order to overcome the limitations of the prior art, the present invention provides a new map-free visual relocalization method, which accurately extracts target instance objects by adopting instance segmentation technology based on the SAM2 model, provides accurate prior information for subsequent feature point matching, and thus improves the relocalization accuracy in a low common view environment. A pair of weight-sharing encoders and decoders are used to output point maps of the reference image and the query image, and an improved reciprocal matching method is further used for feature point matching. The five-point method is used to restore the pose matrix, and finally, combined with depth estimation technology, the positioning result is further optimized. The method has strong practical application value, and has broad application prospects, especially in the fields of autonomous driving and robot navigation.

[0037] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0038] refer to Figure 1 , which is an application scenario of the mapless relocation method provided by the exemplary embodiment of the present disclosure. In this application scenario, a terminal device 101, a server 102, a data storage system 103 and a client 104 are included. Among them, the terminal device 101, the server 102, the data storage system 103 and the client 104 can be connected through a wired or wireless communication network to achieve data interaction.

[0039] The terminal device 101 may be an electronic device with data transmission and multimedia input / output functions close to the user side, including but not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA) or other electronic devices capable of realizing the above functions. The electronic device may include a processor and a display screen with a touch input function, the display screen is used to present a graphical user interface, the graphical user interface can display an application interface, and the processor is used to process application data, generate a graphical user interface, and control the display of the graphical user interface on the display screen.

[0040] Server 102 and data storage system 103 can both be independent physical servers, or a server cluster or distributed system composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0041] In some exemplary embodiments, the map-free relocation system may be run on the terminal device 101 or the server 102 .

[0042] When the map-free relocation method is run on the server 102 , the server 102 is used to provide a map-free relocation service to a user of the client 104 .

[0043] The terminal device 101 receives sensor data of the current environment (such as image information, IMU data, etc.) and sends it to the server 102;

[0044] The server 102 receives the sensor data of the current environment sent by the terminal device 101, performs feature extraction and matching based on the data, and obtains scene feature matching information;

[0045] The server 102 performs posture estimation and optimization on the environment data according to the scene feature matching information to obtain posture estimation optimization information;

[0046] The server 102 inputs the posture estimation optimization information into the mapless relocalization model pre-trained in the data storage system 103, obtains accurate position coordinate information through model processing, and then optimizes the position coordinate information to obtain a mapless relocalization result;

[0047] The server 102 sends the map-free relocation result to the client 104;

[0048] The user can obtain the results of map-free relocation in different scenarios through the client 104 .

[0049] A large amount of training data is stored in the data storage system 103 for pre-training the mapless relocation model. These training data include various types of environmental sensor data and corresponding real location coordinates, so that the model learns how to map environmental features to accurate location coordinate information. The sources of training data include but are not limited to existing databases, data crawled from the Internet, or data uploaded by users when using the client. When the difference between the location information output by the mapless relocation model and the real coordinates reaches a certain requirement, the server 102 can provide users with more accurate mapless relocation services based on the model.

[0050] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0051] refer to Figure 2 , without map relocation, applied to the server, the method comprises the following steps:

[0052] Step S210: Use the segmentation model to extract the same instance objects in the current image and the reference image in the database.

[0053] In this embodiment, the SAM2 model is used to perform instance-level segmentation on the query image and the reference image to obtain mask information of the instance object, thereby providing an accurate target area for subsequent feature point matching. The specific steps are as follows:

[0054] Generate instance masks: In the query and reference images, SAM2 generates instance-level object masks through prompt input. The model supports multiple prompt methods, including click, bounding box, and mask. Through these input prompts, SAM2 can accurately segment the target object instance from the image and obtain the instance-level mask of the object.

[0055] Region limitation using instance mask: Based on the instance mask generated by SAM2, feature points are extracted only in the target region. This can effectively eliminate the interference of background and noise, ensure that the feature point extraction process is limited to the target instance object, and thus improve the accuracy and robustness of feature point matching.

[0056] Enhanced matching accuracy: Through instance-level segmentation information, the mask provided by SAM2 helps exclude irrelevant background information, making subsequent feature point extraction and matching more focused on the target area. This technology significantly improves global matching accuracy and provides accurate input for subsequent matching algorithms.

[0057] Step S220: extract feature points from the current image and the reference image and perform reciprocal matching.

[0058] Feature point matching method enhanced by instance knowledge

[0059] In the mapless visual relocalization task, the accuracy of feature point matching is crucial for pose estimation. In order to reduce errors in the matching process and improve matching accuracy, this study introduces a feature point matching method enhanced by instance knowledge, which limits the scope of feature point matching through instance-level segmentation information, thereby ensuring that matching only occurs within the same instance and avoiding errors in cross-instance matching.

[0060] Specifically, for the reference image I1 and the query image I2 and their corresponding feature point graphs X 1,1and X 2,1 , using a matching mechanism based on the nearest neighbor and reciprocity principle. First, the DUSt3R model is used to generate a three-dimensional point map of the two images and extract feature points from them. At each feature point (i, j), find the point map X 2,1 Zhong and X 1,1 (i,j) The closest matching point X 2,1 (k, l), and perform the initial pairing by nearest neighbor matching. After obtaining the initial matching relationship, the reciprocal matching principle is further applied to ensure the accuracy of the matching. That is, find the matching point (k, l) in I2 from the pixel (i, j) in image I1, and vice versa. Only when The matching results are consistent in two-way verification, and the matching is considered valid. The global matching relationship is defined as follows:

[0061] Map g =M(I1,I2,X 1,1 ,X 2,1 )

[0062] To further improve the accuracy of feature point matching, we introduce instance segmentation technology to limit the matching range. Specifically, for the same instance object o, the corresponding instance mask Mask is obtained from the reference image and the query image. o1 and Mask o2 Inside the instance mask, construct a local feature point set X o1,1 and X o2,1 And match them, the local matching relationship is as follows:

[0063] Map o =M(I o1 ,I o2 ,X o1,1 ,X o2,1 )

[0064] The overall representation of the final matching relationship is as follows, where Map is used in the instance area o For matching, non-instance areas use Map g :

[0065]

[0066] This instance-enhanced feature point matching method significantly improves the accuracy and robustness of matching by limiting the matching range and precise instance marking, so that feature point matching is only performed within relevant instances, providing reliable support for subsequent pose estimation and depth calculation.

[0067] In order to further improve the efficiency and accuracy of feature point matching, this study proposed an improved fast reciprocal nearest neighbor matching method. This method combines the reciprocal matching principle of two-way verification, the confidence screening mechanism and the flexible distance metric selection, and optimizes the calculation process through block calculation and parallel processing, ensuring that the matching accuracy is improved while the matching speed is significantly improved.

[0068] Mutually beneficial matching principle: The core of mutually beneficial matching lies in two-way verification. and If point a i The nearest neighbor of ∈A is b j ∈B, and at the same time satisfy b j The nearest neighbor of i , it is considered a valid match. Specifically, we define the distance matrix from point set A to B as D AB :

[0069] D AB (i,j)=dist(a i ,b j )

[0070] Where dist represents the distance measurement method (such as Euclidean distance or dot product similarity). The two-way reciprocal matching relationship can be expressed as:

[0071]

[0072] When the two-way matching condition is met, it is considered that (a i ,b j ) are valid matching pairs.

[0073] Confidence screening: To improve the accuracy of matching, a confidence screening mechanism is added on the basis of reciprocal matching. The confidence function C(x) is set to represent the confidence of feature point x, and only high-confidence matching pairs are retained, so that the filtered matching pairs meet the condition C(a i )≥C thresh and C(b j )≥C thresh Among them C thresh is the confidence threshold. This mechanism can be expressed as:

[0074]

[0075] Confidence screening can effectively eliminate noise points and improve matching reliability.

[0076] Flexible distance metric selection: According to different feature point types, flexibly select the appropriate distance metric. In geometric matching tasks, use Euclidean distance The formula is:

[0077]

[0078] In feature vector matching, dot product similarity is used

[0079]

[0080] This flexible metric selection can better adapt to different types of data and thus improve the matching accuracy.

[0081] In large-scale point set matching, in order to improve the speed, this study adopts block calculation and parallel processing. Assume that point sets A and B are divided into several small blocks, denoted by A = {A1, A2, …, A k} and B={B1,B2,…,B m}. In each pair of blocks (A i ,B j ) and calculate their distance matrix And perform reciprocal nearest neighbor matching in each subset. The block strategy not only reduces memory usage but also improves matching speed. Finally, the global matching relationship can be expressed as

[0082]

[0083] This method greatly improves the computational efficiency while maintaining high matching accuracy, ensuring the application effect of the map-free visual relocalization framework under real-time requirements.

[0084] Step S230, using the five-point method to calculate the pose of the matched feature points, and obtain a rotation matrix and a scale-free translation vector;

[0085] Next, the method for calculating the pose will be introduced;

[0086] In the pose solver, we chose the 5-point algorithm, combined with the essential matrix method, to robustly calculate the essential matrix E through matching point pairs:

[0087] E=UΣV T

[0088] Among them, U and V are orthogonal matrices representing rotation or reflection, and Σ is a diagonal matrix representing the scaling factor of the original matrix. After obtaining E, it is decomposed into the rotation matrix R and the scale-free translation vector After decomposition, additional geometric constraints are used to select the correct R and

[0089]

[0090] in, is the scale-free translation vector, and R is the rotation matrix. To ensure that most 3D points have positive depth, we use the depth estimation of a single image to back-project the 2D matching points into 3D points and verify which R and To provide a reasonable and physically consistent configuration.

[0091] Step S240: Estimate the depth using a depth estimation method that uses a normal and depth co-optimization method, and obtain a final translation matrix in combination with the scale-free translation vector.

[0092] Next, we introduce how to get the depth estimation result:

[0093] In scale recovery, we use Metric3D as the benchmark. The input image I is first transformed into image I by normalizing the camera space c , using standardized camera model parameters, including focal length f c and the principal point (u c ,v c ). The internal parameters of image I, such as focal length f and principal point (u, v), are expressed in terms of ratio The image is scaled. This transformation facilitates encoding and decoding in the DINO2reg-ViT-giant2 model and RAFT-8iter framework, and refines the scale-recovered depth estimate through iterative optimization.

[0094] I c =T(I,ω r )

[0095] Where T(·) represents the image scaling operation. c Input the encoder-decoder network and first get the depth prediction d c :

[0096] d c =Decoder(Encoder(I c ))

[0097] Initial depth prediction and input image I c It is fed into a joint optimization module, which iteratively optimizes the depth-normal consistency constraints and pseudo-normal labels for weakly supervised learning. The depth prediction is updated in each iteration to improve accuracy and reliability:

[0098]

[0099] in represents the depth prediction in the tth iteration, Δd c is the depth increment of this iteration. This iterative process gradually refines the depth prediction and enhances its accuracy and reliability. Finally, the normalized depth prediction d c Proportional Scale back to the actual depth d in the original camera space:

[0100] d=ω d d c

[0101] This transformation ensures that the depth prediction is consistent across different camera models, adapting to different imaging conditions and maintaining the accuracy of depth measurement. The depth values ​​of I1 and I2 are denoted as d1 and d2, respectively.

[0102] In order to incorporate depth information into the matching process, we combine the depth estimation results d1 and d2 with the instance-enhanced matching result Map. Assuming that the coordinates of the matching point pair in the reference image are (i1, j1) and in the query image are (k1, l1), and the corresponding depths are d1 and d2 respectively, their three-dimensional coordinates can be expressed as

[0103] and

[0104] Where K is the camera parameter matrix, p1 and p2 are the corresponding 3D coordinates. For each pair of 3D-3D correspondences, calculate the scale factor s of the translation vector:

[0105]

[0106] Among them, ∥P1-RP2∥ represents the 3D point spacing difference between the two images, is the length of the translation vector. Calculate the scale factor for each pair of 3D-3D correspondences, and use the RANSAC algorithm to select the one with the highest consistency as the final scale factor. Finally, the unscaled translation vector Scaling by s to get the final translation vector Combined with the rotation matrix R, the final relative pose of the query image relative to the reference image is obtained.

[0107] refer to Figure 3 , the map-free repositioning device comprises:

[0108] The image acquisition module 210 is configured to receive traffic scene image data from different types of sensors or camera systems. The module first performs image preprocessing operations, such as denoising, grayscale, normalization, etc., in order to improve image quality and prepare for subsequent processing. During the image acquisition process, the image acquisition module can automatically detect and mark key areas in the image, such as roads, pedestrians, vehicles and other traffic elements, to ensure that subsequent analysis can be focused on the most relevant areas. In addition, the module also has a certain scene recognition capability, which can identify different weather or lighting conditions so as to adjust the image processing method according to different environments. Finally, the preprocessed image is passed to the downstream module for further analysis and processing.

[0109] The instance information extraction module 220 is configured to perform instance-level segmentation on the received traffic scene image based on advanced image segmentation technology (such as the SAM2 model). The module uses input prompts (such as clicks, borders, or masks, etc.) to accurately identify and segment object instances in the image through a deep learning model. Specifically, the SAM2 model analyzes each instance in the image and generates a corresponding instance mask that can accurately mark the boundaries of the target object and eliminate the interference of background noise on subsequent analysis. In this process, the instance information extraction module not only pays attention to the spatial position of the object, but also pays special attention to the shape and features of the object to ensure that the significant instances in the image can be accurately captured. The obtained instance mask will be used as prior information for subsequent feature point extraction and matching, thereby improving the accuracy and robustness of image recognition in the subsequent processing process.

[0110] The feature point extraction and reciprocal matching module 230 is one of the key modules in the whole system, which is responsible for extracting significant feature points in the image and performing the matching task of feature points. Specifically, after the instance information extraction module 220 generates the instance mask, the feature point extraction and reciprocal matching module 230 will only extract feature points in the mask area of ​​the target object, while ignoring the interference of the background area. This module uses advanced feature point detection algorithms, such as SIFT, ORB or SURF, to extract significant feature points in the target area. Through feature point extraction, the module can find a set of preliminary matching point pairs between the query image and the reference image. Subsequently, the reciprocal matching algorithm is used to perform accurate bidirectional verification on these matching points to ensure the accuracy and consistency of the matching results. This module also includes the calculation and optimization of feature point descriptors. By calculating the similarity of point pairs, the unmatched point pairs are eliminated and the most accurate matching points are retained. These optimized matching points will provide reliable input for subsequent pose estimation and target positioning.

[0111] The pose calculation module 240 is configured to calculate the pose of the target object in space based on the matching results from the feature point extraction and reciprocal matching module 230. The module uses a modern pose estimation algorithm (such as the essential matrix + five-point method, etc.) to solve the relative pose of the object based on the position of the feature points in the two-dimensional image and their known coordinates in the three-dimensional space. In order to improve the accuracy of pose estimation, the pose calculation module 240 also combines multi-view geometric constraints and optimization methods to optimize the pose using multiple perspectives of the image to reduce the impact of errors. In actual operation, pose calculation not only relies on the information of a single image, but also comprehensively considers the motion trajectory and relative changes of the object in continuous image frames. This multi-view and time series pose estimation method greatly improves the robustness and accuracy of the system in dynamic scenes, can adapt to rapidly changing traffic environments, and accurately calculate the spatial position information of objects.

[0112] The depth estimation and scale recovery module 250 is mainly responsible for recovering the three-dimensional depth information of the target object and solving the problem of scale uncertainty. In traffic scenes, depth information is the key to achieving object positioning and navigation, and recovering accurate depth information requires combining image data from multiple perspectives and their relative positional relationships. This module estimates the three-dimensional depth of the target object through a deep learning method or a traditional stereoscopic vision method. Specifically, the depth estimation and scale recovery module 250 combines the feature point information obtained from the feature point extraction and matching module 230 with the image data from multiple perspectives, and uses a depth estimation model (convolutional neural network Metric3D) to recover the depth of the object. In addition, considering the scale ambiguity problem that often occurs in real environments, this module adopts a multi-perspective and scale compensation algorithm, which can restore the true three-dimensional scale of the object according to the geometric relationship between images. The depth estimation and scale recovery module can not only improve the accuracy of the depth information, but also eliminate the scale differences that may occur under different camera perspectives through the scale recovery method, ensuring that the final depth estimation conforms to the actual physical scale.

[0113] Figure 4 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0114] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0115] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0116] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0117] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0118] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0119] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0120] The electronic device of the above embodiment is used to implement the corresponding map-free repositioning method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0121] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the map-free repositioning method as described in any of the above embodiments.

[0122] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0123] The above-mentioned non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0124] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the map-free repositioning method described in any embodiment in the above exemplary method part, and have the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0125] Based on the same inventive concept, corresponding to the mapless relocation method described in any of the above embodiments, the present disclosure further provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the mapless relocation method. Corresponding to the execution subject corresponding to each step in each embodiment of the mapless relocation method, the processor that executes the corresponding step can belong to the corresponding execution subject.

[0126] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the map-free repositioning method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0127] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." In addition, in some embodiments, the present disclosure may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0128] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0129] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0130] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0131] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Python, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the functions / operations specified in the boxes in the flowchart and / or block diagram.

[0133] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0134] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0135] In addition, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0136] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0137] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0138] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0139] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0140] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0141] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

[0142] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. A map-free relocation method, characterized in that: include: Obtaining a reference image and a query image; Extracting objects from the image based on instance segmentation to obtain instance mask information; Performing feature point matching within the instance mask range to obtain high-precision matching information; Calculate the essential matrix using the matching information to obtain the rotation matrix and scale-free translation vector of the camera; Inputting the image into a pre-trained depth estimation model to obtain depth information; Recovering the scale of the translation vector in combination with the depth information to obtain an accurate camera posture; Finally, the camera pose is estimated and the visual relocalization result is obtained.

2. The method according to claim 1, characterized in that The extracting objects from the image based on instance segmentation to obtain instance mask information includes: Preprocessing the received reference image and query image to obtain images of uniform size; Processing the image using a pre-trained instance segmentation model to obtain main object instances in the image and their corresponding mask information; The instance mask is applied to the image to extract the instance object region and obtain instance mask information.

3. The method according to claim 2, characterized in that The instance segmentation model includes: an encoder, a decoder and a prediction head; The using a pre-trained instance segmentation model to process the image to obtain instance mask information includes: Extracting features from the image based on the encoder to obtain a high-level feature representation; Upsampling the feature representation using the decoder to restore spatial resolution; Classifying each pixel through the prediction head to generate an instance mask; The instance mask is post-processed to obtain instance mask information.

4. The method according to claim 1, characterized in that: The instance mask information includes at least one of the following: mask information of a main instance object, mask information of a background area; The matching of feature points within the instance mask range to obtain high-precision matching information includes the following steps: Extracting feature points in the mask area of ​​the instance object to obtain feature point information in the instance; Matching feature points in the instance to obtain high-precision matching pairs; Use the global feature matching method to perform feature matching in the non-masked area to obtain global matching information; The final matching information is obtained by combining the in-instance matching information and the global matching information.

5. The method according to claim 1, characterized in that Inputting the image into a pre-trained depth estimation model to obtain depth information also includes training the depth estimation model by the following method: Determine image sample information with depth labels; Inputting the image sample information into the depth estimation model to obtain depth prediction information output by the model; determining a difference between the depth label information and the depth prediction information; Based on the difference, the model parameters are updated through a loss function until the difference between the depth label information and the depth prediction information is minimized, thereby obtaining the pre-trained depth estimation model.

6. The method according to claim 1, characterized in that The method of restoring the scale of the translation vector by combining the depth information to obtain the accurate camera posture includes: Combining the depth information with the matched feature points, back-projecting the two-dimensional feature points into three-dimensional space to obtain three-dimensional point coordinates; Calculating a scale factor of a translation vector using the three-dimensional point coordinates; Multiplying the scale-free translation vector by the scale factor to obtain a scaled translation vector; The rotation matrix is ​​combined with the scaled translation vector to obtain the precise camera pose.

7. A map-free visual relocation device, characterized in that: include: An image acquisition module, configured to acquire a reference image and a query image; An instance mask determination module is configured to extract objects from the image based on instance segmentation to obtain instance mask information; A feature matching module is configured to perform feature point matching within the instance mask range to obtain high-precision matching information; A posture calculation module is configured to calculate an essential matrix using the matching information to obtain a rotation matrix and a scale-free translation vector of the camera; A depth estimation module is configured to input the image into a pre-trained depth estimation model to obtain depth information; A scale recovery module, configured to recover the scale of the translation vector in combination with the depth information to obtain an accurate camera pose; The relocalization result output module is configured to finally perform camera pose estimation and obtain the visual relocalization result.

8. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

Images