Image positioning method and device, depth estimation model training method and device and electronic equipment

Abstract

The invention relates to an image positioning method, a depth estimation model training method, a depth estimation model training device and electronic equipment. The image positioning method comprises the following steps: inputting a query image of a target scene into a reference image library for matching so as to find a reference image close to the view angle of the query image; performing depth estimation on the reference image to obtain a pixel-level depth predicted value and a pixel-level depth confidence coefficient of the reference image; performing scale alignment based on the pixel-level depth predicted value, the pixel-level depth confidence coefficient and a point cloud depth value constructed based on a three-dimensional reconstruction technology to obtain an alignment parameter; and obtaining 2D-3D matching point pairs corresponding to the 2D-2D matching point pairs of the query image and the reference image based on the alignment parameter, the pixel-level depth prediction value and the camera pose of the reference image, and determining pose information of the query image based on the 2D-3D matching point pairs. The positioning success rate can be improved, and the camera pose estimation quality can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image localization method, a depth estimation model training method, an apparatus, and an electronic device, which can be applied to the field of augmented reality (AR). Background Technology

[0002] Camera pose estimation plays a crucial role in augmented reality (AR) applications. It refers to determining the position and orientation (including translation and rotation) of the camera in three-dimensional space, enabling the system to accurately blend virtual content with real-world scenes.

[0003] Traditional localization algorithms mainly include steps such as SFM (Structure from Motion) map construction, keyframe retrieval, feature matching, and camera pose calculation. However, because traditional localization algorithms reconstruct sparse point clouds, the 2D-3D keypoint matching information stored in the reference image is relatively limited, resulting in a lack of sufficient 2D-3D matching point pairs for the final PnP (Perspective-n-Point) pose calculation. Summary of the Invention

[0004] This application provides an image localization method, a depth estimation model training method, a device, and an electronic device, which can solve the problem of insufficient 2D-3D matching point pairs for PnP pose solving in traditional localization algorithms.

[0005] In a first aspect, embodiments of this application provide an image localization method, including:

[0006] The query image of the target scene is input into the reference image library for matching in order to find a reference image with a similar viewpoint to the query image;

[0007] Depth estimation is performed on the reference image to obtain the pixel-level depth prediction value and pixel-level depth confidence of the reference image;

[0008] Scale alignment is performed based on the pixel-level depth prediction value, the pixel-level depth confidence value, and the point cloud depth value constructed based on 3D reconstruction technology to obtain alignment parameters;

[0009] Based on the alignment parameters, the pixel-level depth prediction value, and the camera pose of the reference image, a 2D-3D matching point pair corresponding to the 2D-2D matching point pair between the query image and the reference image is obtained, and the pose information of the query image is determined based on the 2D-3D matching point pair.

[0010] Secondly, embodiments of this application provide a method for training a depth estimation model, including:

[0011] The training samples are input into the depth estimation model to be trained to perform depth estimation, and the pixel-level depth prediction value and pixel-level depth confidence value of the training samples are obtained.

[0012] Based on the pixel-level depth prediction values ​​and pixel-level depth labels of the training samples, a depth loss function is determined;

[0013] Based on the pixel-level depth confidence, pixel-level depth prediction, and pixel-level depth label of the training samples, the binary cross-entropy loss function is used to determine the confidence loss function.

[0014] Based on the depth loss function and the confidence loss function, the loss function is determined, and the first learnable parameter and the second learnable parameter of the depth estimation model are adjusted according to the loss function; the first learnable parameter is used to predict pixel-level depth confidence, and the second learnable parameter is used to predict pixel-level depth value;

[0015] The depth estimation model is applied to an image localization task to obtain pixel-level depth prediction values ​​and pixel-level depth confidence scores of a reference image. The pixel-level depth prediction values ​​and pixel-level depth confidence scores of the reference image are used to scale-align with the depth values ​​of a point cloud constructed based on 3D reconstruction technology to obtain alignment parameters. The alignment parameters are used to combine the pixel-level depth prediction values ​​and the camera pose of the reference image to determine the pose information of the query image.

[0016] Thirdly, embodiments of this application provide an image positioning device, comprising:

[0017] The matching module is used to input the query image of the target scene into the reference image library for matching, so as to find a reference image with a similar viewpoint to the query image;

[0018] The depth estimation module is used to perform depth estimation on the reference image to obtain the pixel-level depth prediction value and pixel-level depth confidence of the reference image;

[0019] The scale alignment module is used to perform scale alignment based on the pixel-level depth prediction value, the pixel-level depth confidence value, and the point cloud depth value constructed based on 3D reconstruction technology to obtain alignment parameters.

[0020] The pose estimation module is used to obtain 2D-3D matching point pairs corresponding to the 2D-2D matching point pairs between the query image and the reference image based on the alignment parameters, the pixel-level depth prediction values ​​and the camera pose of the reference image, and to determine the pose information of the query image based on the 2D-3D matching point pairs.

[0021] Fourthly, embodiments of this application provide a depth estimation model training apparatus, comprising:

[0022] The depth estimation module is used to input training samples into the depth estimation model to be trained for depth estimation, and obtain pixel-level depth prediction values ​​and pixel-level depth confidence values ​​of the training samples.

[0023] The first determining module is used to determine the depth loss function based on the pixel-level depth prediction value and pixel-level depth label of the training samples;

[0024] The second determining module is used to determine the confidence loss function based on the pixel-level depth confidence, pixel-level depth prediction value and pixel-level depth label of the training samples, using the binary cross-entropy loss function.

[0025] The third determining module is used to determine the loss function based on the depth loss function and the confidence loss function;

[0026] The parameter adjustment module is used to adjust the first learnable parameter and the second learnable parameter of the depth estimation model according to the loss function; the first learnable parameter is used to predict pixel-level depth confidence, and the second learnable parameter is used to predict pixel-level depth value;

[0027] The depth estimation model is applied to an image localization task to obtain pixel-level depth prediction values ​​and pixel-level depth confidence scores of a reference image. The pixel-level depth prediction values ​​and pixel-level depth confidence scores of the reference image are used to scale-align with the depth values ​​of a point cloud constructed based on 3D reconstruction technology to obtain alignment parameters. The alignment parameters are used to combine the pixel-level depth prediction values ​​and the camera pose of the reference image to determine the pose information of the query image.

[0028] Fifthly, embodiments of this application provide an electronic device, including:

[0029] One or more processors;

[0030] The processor is configured to invoke instructions to cause the electronic device to perform the methods described in the first and second aspects above.

[0031] In a sixth aspect, embodiments of this application provide a storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the first and second aspects described above.

[0032] In a seventh aspect, embodiments of this application provide a program product, including at least one of a program and instructions, wherein when the program and instructions are executed by an electronic device, they implement the steps of the methods described in the first and second aspects.

[0033] According to the technical solution of this application, by obtaining depth estimation results including pixel-level depth confidence, and by using scale alignment to ensure that each matched 2D-2D point pair can obtain a corresponding 2D-3D matching point pair, the problem of inaccurate PnP solution caused by insufficient 2D-3D key point matching information stored in the reference image can be solved. This improves the success rate of PnP solution and the localization success rate, enabling the image localization system to accurately integrate virtual content with the real-world scene.

[0034] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0035] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0036] Figure 1 This is a flowchart illustrating an image localization method according to an exemplary embodiment;

[0037] Figure 2 This is a flowchart illustrating an image localization method according to an exemplary embodiment;

[0038] Figure 3 This is a flowchart illustrating a depth estimation model training method according to an exemplary embodiment;

[0039] Figure 4 This is a schematic diagram illustrating the processing flow of a depth estimation model according to an exemplary embodiment;

[0040] Figure 5 This is a block diagram of an image positioning device according to an exemplary embodiment;

[0041] Figure 6 This is a block diagram of an image positioning device according to an exemplary embodiment;

[0042] Figure 7 This is a block diagram illustrating a depth estimation model training apparatus according to an exemplary embodiment;

[0043] Figure 8 This is a block diagram of an electronic device 800 according to an exemplary embodiment. Detailed Implementation

[0044] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0045] The embodiments described in this application are not exhaustive, but merely illustrative of some embodiments, and are not intended to limit the scope of protection of this application. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined with each other. For example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0046] In each embodiment of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0047] It should be noted that the acquisition, transmission, storage, use, and processing of data in this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0048] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0049] It is worth noting that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0050] The embodiments of this application relate to fields such as computer vision, 3D reconstruction technology, and camera pose estimation.

[0051] Computer vision uses various imaging systems to replace visual organs as the means of input sensing, allowing computers to process and interpret information instead of the brain. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision, just like humans, and to have the ability to autonomously adapt to their environment.

[0052] 3D reconstruction refers to establishing a mathematical model of a 3D object suitable for computer representation and processing. It forms the basis for processing, manipulating, and analyzing the object's properties in a computer environment and is a key technology for creating virtual reality representations of the objective world within a computer. In computer vision, 3D reconstruction refers to the process of reconstructing 3D information from single-view or multi-view images. Because single-view information is incomplete, 3D reconstruction requires the use of empirical knowledge. Multi-view 3D reconstruction (similar to human binocular positioning) is relatively easier. The method involves first calibrating the camera, i.e., calculating the relationship between the camera's image coordinate system and the world coordinate system, and then reconstructing the 3D information using information from multiple 2D images.

[0053] Structure from Motion (SfM), also known as structure from motion or structure from motion, is a technique that recovers camera parameters and 3D scene structure by analyzing image sequences. This technique is suitable for situations where the camera is moving within a static scene. The core process includes feature point extraction, image matching, camera pose estimation, and 3D point cloud reconstruction. It has wide applications in fields such as robot navigation, augmented reality, and industrial inspection.

[0054] Camera pose estimation plays a crucial role in augmented reality (AR) applications. It refers to determining the position and orientation (including translation and rotation) of the camera in three-dimensional space, enabling the system to accurately blend virtual content with real-world scenes.

[0055] Traditional localization algorithms mainly include steps such as SFM (Structure from Motion) map construction, keyframe retrieval, feature matching, and camera pose calculation. The SFM map construction step involves using a series of images to create a sparse point cloud map of the scene through 3D reconstruction techniques (such as SFM). The keyframe retrieval step involves finding the keyframes in a reference image library that have the closest viewpoint to the query image, achieved using image retrieval techniques. The feature matching step uses methods such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF, where FAST represents feature point detection and BRIEF represents feature description) to match 2D feature points between keyframes and the query image. Camera pose calculation steps: Using the 2D-3D key point matching information stored in the keyframes, we can obtain the 2D-3D matching point pairs between the query image and the scene. We then apply the PnP (Perspective-n-Point) algorithm to solve for the position and pose of the camera corresponding to the query image in the 3D point cloud map.

[0056] However, since traditional localization algorithms reconstruct sparse point clouds, the reference image contains relatively little 2D-3D keypoint matching information, which leads to a lack of sufficient 2D-3D matching point pairs for the final PnP pose solution.

[0057] Based on this, embodiments of this application provide an image localization method, a depth estimation model training method, an apparatus, and an electronic device. By obtaining depth estimation results including pixel-level depth confidence, and through scale alignment, each matched 2D-2D point pair can obtain a corresponding 2D-3D matching point pair, thereby improving the success rate of PNP solution and localization success rate.

[0058] The image localization method, depth estimation model training method, apparatus, and electronic device of this application are described below with reference to the accompanying drawings.

[0059] It should be noted that the execution subject of the image positioning method in this application embodiment can be an image positioning device, which can be implemented by software and / or hardware, and can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc.

[0060] It should also be noted that the execution subject of the depth estimation model training method in the embodiments of this application can be a depth estimation model training device. The device can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc.

[0061] Figure 1 This is a flowchart illustrating an image localization method according to an exemplary embodiment. Figure 1 As shown, the image localization method may include, but is not limited to, the following steps.

[0062] In step 101, the query image of the target scene is input into the reference image library for matching in order to find a reference image with a similar viewpoint to the query image.

[0063] In some embodiments, the query image can be understood as the image to be located of the target scene. For example, when locating an image of a target scene, the image to be located (i.e., the query image) can be obtained, and the query image can be input into a reference image library for matching to find a reference image with a viewpoint close to that of the query image. For instance, image retrieval technology can be used to find the keyframe in the reference image library that has the closest viewpoint to the query image as the reference image. Here, the image to be located can refer to an image of the target scene captured by a camera.

[0064] In step 102, depth estimation is performed on the reference image to obtain the pixel-level depth prediction value and pixel-level depth confidence of the reference image.

[0065] In some embodiments, monocular depth estimation techniques can be used to perform monocular depth estimation on a reference image to obtain pixel-level depth prediction values ​​and pixel-level depth confidence scores for the reference image.

[0066] In some embodiments, a pre-trained depth estimation model (such as a monocular depth estimation model) can be used to estimate the depth of a reference image. For example, the reference image can be input into the pre-trained depth estimation model to obtain pixel-level depth prediction values ​​and pixel-level depth confidence scores for the reference image. The depth estimation model may include a first learnable parameter for predicting the depth confidence score and a second learnable parameter for predicting the depth value. These first and second learnable parameters are optimized using a loss function. Optionally, the training of the depth estimation model can be implemented using any of the optional implementation methods in the depth estimation model training method embodiments described below, and will not be repeated here.

[0067] In one possible implementation, the network structure of an existing monocular depth estimation model can be modified by adding an additional branch after the last decoder of the monocular depth estimation model to predict depth confidence. This confidence is some learnable parameters (i.e., the first learnable parameter mentioned above), which can be optimized through a loss function during the model training phase.

[0068] In some embodiments, the pixel-level depth prediction value of the reference image may refer to the depth prediction value of a pixel in the reference image. The pixel-level depth confidence of the reference image may refer to the depth confidence of a pixel in the reference image, which can be understood as a quantitative representation of the degree of certainty of the depth prediction value for that pixel. In embodiments of this application, the depth prediction value and its depth confidence of each pixel in the reference image can be obtained by performing depth estimation on the reference image.

[0069] In step 103, scale alignment is performed based on pixel-level depth prediction values, pixel-level depth confidence values, and point cloud depth values ​​constructed based on 3D reconstruction technology to obtain alignment parameters.

[0070] In some embodiments, the 3D reconstruction technique can be SFM (Surface View Mechanism), but is not limited to it; for example, the MVS (Multi-View Stereo) algorithm. For instance, using SFM, multiple keyframe images of the target scene can be used to construct a point cloud map of the target scene based on SFM. The point cloud map data includes point cloud depth values. After obtaining the pixel-level depth prediction value and pixel-level depth confidence of the reference image, the pixel-level depth prediction value of the reference image can be aligned with the point cloud depth values ​​constructed based on SFM to obtain alignment parameters. For instance, these alignment parameters can be used to combine the pixel-level depth prediction value and the camera pose of the reference image to determine the pose information of the query image.

[0071] In step 104, based on the alignment parameters, pixel-level depth prediction values, and camera pose of the reference image, 2D-3D matching point pairs corresponding to the 2D-2D matching point pairs of the query image and the reference image are obtained, and the pose information of the query image is determined based on the 2D-3D matching point pairs.

[0072] In some embodiments, the pixel-level depth prediction values ​​of the reference image can be adjusted based on the alignment parameter. This can resolve the scale uncertainty problem of the depth prediction values ​​for each pixel in the reference image, ensuring that each pixel in the reference image has a deterministic depth value. When the pixel-level depth value adjusted by the alignment parameter is obtained, the corresponding 2D-3D matching point pair can be obtained based on the camera pose of the reference image, the 2D coordinates in the 2D-2D matching point pair between the query image and the reference image, and the adjusted pixel-level depth value. In other words, scale alignment can ensure that each matched 2D-2D point pair yields a corresponding 2D-3D matching point pair. Furthermore, based on the 2D-3D matching point pair, the PnP algorithm can be applied to solve for the position and pose of the query image.

[0073] In the above embodiments, by obtaining depth estimation results including pixel-level depth confidence and by using scale alignment to ensure that each matched 2D-2D point pair can obtain a corresponding 2D-3D matching point pair, the problem of inaccurate PnP solution caused by insufficient 2D-3D key point matching information stored in the reference image can be solved. This improves the success rate of PnP solution and the localization success rate, enabling the image localization system to accurately integrate virtual content with the real-world scene.

[0074] Figure 2 This is a flowchart illustrating an image localization method according to an exemplary embodiment. Figure 2 As shown, the image localization method may include, but is not limited to, the following steps.

[0075] In step 201, the query image of the target scene is input into the reference image library for matching in order to find a reference image with a similar viewpoint to the query image.

[0076] Optionally, step 201 can be implemented using any of the implementation methods in the various embodiments of this application. The embodiments of this application do not limit this, nor will they be described in detail.

[0077] In step 202, depth estimation is performed on the reference image to obtain the pixel-level depth prediction value and pixel-level depth confidence of the reference image.

[0078] Optionally, step 202 can be implemented using any of the implementation methods in the various embodiments of this application. The embodiments of this application do not limit this, nor will they be described in detail.

[0079] In step 203, a minimum objective function is constructed based on the pixel-level depth prediction value, alignment parameter, point cloud depth value constructed based on 3D reconstruction technology, and pixel-level depth confidence. The optimal solution of the minimum objective function is then sought to obtain the value of the alignment parameter.

[0080] In some embodiments, the 3D reconstruction technique can be SFM (Surface View Mechanism), but is not limited to this, such as the MVS (Multi-View Stereo) algorithm. For example, using SFM, multiple keyframe images of the target scene can be used to construct a point cloud map of the target scene based on SFM. The data of this point cloud map includes point cloud depth values. After obtaining the pixel-level depth prediction value and pixel-level depth confidence of the reference image, the pixel-level depth prediction value of the reference image can be aligned with the point cloud depth value constructed based on SFM to obtain alignment parameters.

[0081] In one possible implementation, suppose D(u,v) represents the depth prediction value obtained by depth estimation at coordinates (u,v) on the reference image, D... s (u,v) represents the point cloud depth value constructed using SFM technology at coordinates (u,v) on the reference image. The predicted depth value at (u,v) can be scaled with the point cloud depth value at coordinates (v,v) based on an alignment parameter. For example, the formula for this scale alignment is as follows:

[0082] D s (u,v)=a*D(u,v)+b

[0083] Here, a and b are alignment parameters, and their values ​​can be obtained by finding the optimal solution that minimizes the objective function. As an example, the formula for minimizing the objective function is expressed as follows:

[0084] minimize∑c i *(D s (u,v)-(a*D(u,v)+b)) 2

[0085] Where, minimize is the objective function, which can be expressed as minimizing the values ​​of a and b through gradient optimization. i Let represent the depth confidence of the i-th pixel, and let the coordinates of the i-th pixel be (u, v). In other words, the purpose of minimizing the objective function is: for i pixels, calculate the difference between the point cloud depth value of the pixel (e.g., obtained based on SFM reconstruction) and the depth prediction value of the pixel after scale alignment; then multiply the depth confidence of the pixel by the square of the difference to obtain the squared error of the pixel; sum the average errors of all pixels to obtain the total average error; and optimize the values ​​of a and b using the gradient method to minimize the total average error, thereby obtaining the values ​​of the alignment parameters a and b.

[0086] It should be noted that the above formula for minimizing the objective function is merely an example. That is, it is provided to facilitate those skilled in the art in understanding how to construct a minimization objective function based on pixel-level depth prediction values, alignment parameters, point cloud depth values ​​constructed using 3D reconstruction technology, and pixel-level depth confidence. This example should not be construed as limiting this application; that is, the minimization objective function can also be expressed by other formulas, such as minmize∑c i *Dsu,v-(a*Du,v+b), but not limited to this.

[0087] In step 204, the pixel-level depth prediction value is adjusted based on the alignment parameter to obtain the adjusted pixel-level depth value.

[0088] In the embodiments of this application, after obtaining the value of the alignment parameter, the depth prediction value of each pixel in the reference image can be adjusted using the value of the alignment parameter to obtain the adjusted depth value. For example, assuming D(u,v) represents the depth prediction value of a pixel (coordinates (u,v)), the alignment parameters include a and b, and the adjusted depth value of the pixel is a*D(u,v)+b. Each pixel is adjusted in this way so that each pixel in the reference image has a corresponding 3D depth value.

[0089] In step 205, based on the camera pose of the reference image, the 2D coordinates in the 2D-2D matching point pairs between the query image and the reference image, and the adjusted pixel-level depth value, the corresponding 2D-3D matching point pairs are obtained.

[0090] In some embodiments, the 2D-3D matching point pair corresponding to the 2D-2D matching point pair can be obtained using the camera pose of the reference image, camera intrinsic parameters, 2D coordinates in the 2D-2D matching point pair between the query image and the reference image, and adjusted pixel-level depth values. As an example, the camera pose, camera intrinsic parameters, 2D coordinates, and adjusted pixel-level depth values ​​can be multiplied to obtain the 3D coordinates corresponding to the 2D coordinates, thus yielding the 2D-3D matching point pair corresponding to the 2D-2D matching point pair.

[0091] For example, the camera pose T of the known reference image cw Camera intrinsic parameters k, 2D coordinates p of pixel (u,v) 2D The depth prediction value D(u,v) of pixel (u,v) can be obtained using the following formula to obtain the corresponding 3D coordinates P. 3D :

[0092] P 3D =T cw -1*k -1 *p 2D (u,v)*(a*D(u,v)+b);

[0093] The reference images are keyframes saved during scene modeling, and the camera pose T... cw The camera intrinsic parameters k and p are known. 2D (u,v) is the feature matching result obtained when the reference image and the query image are matched. The depth prediction value D(u,v) is obtained by estimating the depth of the pixel (u,v) in the reference image. a*D(u,v)+b is the depth value after adjustment by the alignment parameter.

[0094] In step 206, the pose information of the query image is determined based on the 2D-3D matching point pairs.

[0095] In the embodiments of this application, when each matched 2D-2D point pair can obtain a corresponding 2D-3D matching point pair through scale alignment, the PnP algorithm can be applied based on the 2D-3D matching point pair to solve for the position and pose of the query image.

[0096] In the above embodiments, a minimum objective function can be constructed based on pixel-level depth prediction values, alignment parameters, point cloud depth values ​​constructed based on 3D reconstruction technology, and pixel-level depth confidence. By finding the optimal solution of the minimum objective function, the value of the alignment parameter can be obtained. Based on the alignment parameter, 3D coordinates can be calculated more accurately, so that each matched 2D-2D point pair can obtain a more accurate 2D-3D matching point pair, which can further improve the success rate of PnP solution and further improve the localization success rate.

[0097] Figure 3 This is a flowchart illustrating a depth estimation model training method according to an exemplary embodiment. Figure 3 As shown, the training method for this depth estimation model may include, but is not limited to, the following steps.

[0098] In step 301, the training samples are input into the depth estimation model to be trained for depth estimation, and the pixel-level depth prediction value and pixel-level depth confidence of the training samples are obtained.

[0099] In the embodiments of this application, the depth estimation model can be a monocular depth estimation model. For example, the network structure of an existing monocular depth estimation model can be modified, and an additional branch can be added after the last decoder of the monocular depth estimation model to predict depth confidence. The depth estimation model can be trained using a publicly available dataset.

[0100] For example, such as Figure 4As shown, training samples can be input into the depth estimation model to be trained for depth estimation. The depth estimation model can encode the training samples to obtain encoded feature vectors. After passing the encoded feature vectors through cross-layer skip connections, a rich contextual information feature map can be obtained. The contextual information feature map is then passed through the first convolutional layer (such as a 1×1 convolutional layer) to obtain a depth map (i.e., a pixel-level depth map). The contextual information feature map is then passed through the second convolutional layer (such as a 1×1 convolutional layer) to obtain the depth confidence of each pixel. Thus, the pixel-level depth prediction value and pixel-level depth confidence of the training samples can be obtained.

[0101] In step 302, a depth loss function is determined based on the pixel-level depth prediction values ​​and pixel-level depth labels of the training samples.

[0102] In one possible implementation, the depth loss function can be determined using the L1 loss function based on the pixel-level depth predictions and pixel-level depth labels of the training samples. For example, for a pixel, the L1 loss function can be used to calculate the absolute difference between the pixel's depth prediction and its depth label (i.e., the true depth value), and the loss function can be obtained by summing the absolute differences of all pixels.

[0103] In another possible implementation, the depth loss function can be determined using the L2 loss function based on the pixel-level depth predictions and pixel-level depth labels of the training samples. For example, for a pixel, the L2 loss function can be used to calculate the squared difference between the pixel's depth prediction and its depth label (i.e., the true depth value), and the loss function can be obtained by summing the squared differences of all pixels.

[0104] It should be noted that the two methods for calculating the depth loss function given above are merely examples for those skilled in the art to understand, and do not constitute a specific limitation on this application. In other words, other loss functions can also be used, combining the pixel-level depth prediction values ​​and pixel-level depth labels of the training samples to determine the depth loss function, such as gradient loss or scale-invariant loss, etc., which will not be elaborated here.

[0105] In step 303, based on the pixel-level depth confidence, pixel-level depth prediction value, and pixel-level depth label of the training samples, the binary cross-entropy loss function is used to determine the confidence loss function.

[0106] In some embodiments, the confidence label of the training sample can be determined based on the pixel-level depth prediction value and pixel-level depth label of the training sample; the confidence loss function can be determined based on the pixel-level depth confidence and confidence label of the training sample using the binary cross-entropy loss function.

[0107] For example, the confidence label can be determined using the difference between the pixel-level depth prediction value and the pixel-level depth label. For instance, for a pixel in the training sample, the difference between the pixel's depth prediction value and its depth label (i.e., the actual depth value or true depth value) can be compared to a depth difference threshold. If the difference is greater than the threshold, the model's predicted depth value for that pixel is considered to have low confidence, and the confidence label for that pixel can be set to 0. If the difference is less than or equal to the threshold, the model's predicted depth value for that pixel is considered to have high confidence, and the confidence label for that pixel can be set to 1. This method can be used to determine the confidence labels for other pixels, thus obtaining the confidence labels for the training samples. As an example, suppose c... g Confidence label of a pixel, c g The possible values ​​are as follows:

[0108]

[0109] Among them, D ′ This represents the depth value predicted by the depth estimation model for a pixel, where D represents the depth label of the pixel, and τ represents the depth difference threshold.

[0110] After obtaining the confidence label for each pixel in the training samples, the confidence loss function can be calculated using the binary cross-entropy loss function based on the confidence label and depth confidence of each pixel in the training samples. For example, the formula for calculating this confidence loss function is as follows:

[0111]

[0112] Where M represents the number of pixels in the training samples; c i c represents the depth confidence score predicted by the depth estimation model for the i-th pixel in the training samples, and can be a real number between 0 and 1; g It can represent the confidence label of the i-th pixel; L C This represents the confidence loss function.

[0113] In step 304, a loss function is determined based on the depth loss function and the confidence loss function, and the first and second learnable parameters of the depth estimation model are adjusted according to the loss function. The first learnable parameter can be used to predict pixel-level depth confidence, and the second learnable parameter can be used to predict pixel-level depth values.

[0114] In some embodiments, the depth loss function and the confidence loss function can be directly summed to obtain the loss function.

[0115] In some embodiments, the depth loss function and the confidence loss function can be weighted and summed to obtain the loss function. As an example, the formula for this loss function is as follows:

[0116] L = L D +λL C

[0117] Where L represents the loss function; L D L represents the depth loss function; C Let represent the confidence loss function; λ is a hyperparameter.

[0118] It should be noted that the two methods for calculating the loss function given above are merely examples for those skilled in the art to understand, and do not constitute a specific limitation on this application. In other words, other methods can also be used to combine the depth loss function and the confidence loss function to determine the loss function, which will not be elaborated here.

[0119] It should be noted that, in the embodiments of this application, the depth estimation model can be applied to image localization tasks to obtain pixel-level depth prediction values ​​and pixel-level depth confidence scores of a reference image. These pixel-level depth prediction values ​​and confidence scores can be used to scale-align with point cloud depth values ​​constructed based on 3D reconstruction (such as SFM) techniques to obtain alignment parameters. These alignment parameters can be used to combine the pixel-level depth prediction values ​​and the camera pose of the reference image to determine the pose information of the query image. Optional implementations can be found in the optional implementations of the image localization method in the above method embodiments, and will not be repeated here.

[0120] In the above embodiments, by adding a branch for predicting depth confidence after the last decoder block of an existing depth estimation network (such as a monocular depth estimation network) and modifying the loss function accordingly, the trained depth estimation model can output a depth estimation result including pixel confidence. When applied to image localization tasks, this depth estimation result including pixel confidence can be combined with the point cloud depth values ​​constructed based on 3D reconstruction technology for scale alignment. Scale alignment ensures that each matched 2D-2D point pair can obtain a corresponding 2D-3D matching point pair, thereby improving the success rate of PnP solution and localization success rate.

[0121] Figure 5 This is a block diagram illustrating an image positioning device according to an exemplary embodiment. Figure 5 As shown, the image localization device may include: a matching module 501, a depth estimation module 502, a scale alignment module 503, and a pose estimation module 504.

[0122] The matching module 501 is used to input the query image of the target scene into the reference image library for matching, so as to find a reference image with a similar viewpoint to the query image.

[0123] The depth estimation module 502 is used to perform depth estimation on the reference image to obtain the pixel-level depth prediction value and pixel-level depth confidence of the reference image.

[0124] The scale alignment module 503 is used to perform scale alignment based on pixel-level depth prediction values, pixel-level depth confidence values, and point cloud depth values ​​constructed based on 3D reconstruction technology to obtain alignment parameters.

[0125] The pose estimation module 504 is used to obtain 2D-3D matching point pairs corresponding to 2D-2D matching point pairs between the query image and the reference image based on alignment parameters, pixel-level depth prediction values ​​and camera pose of the reference image, and to determine the pose information of the query image based on the 2D-3D matching point pairs.

[0126] In some embodiments, the depth estimation module 502 is used to: input a reference image into a pre-trained depth estimation model to perform depth estimation, and obtain a pixel-level depth prediction value and a pixel-level depth confidence value of the reference image; wherein, the depth estimation model includes a first learnable parameter for predicting the depth confidence value and a second learnable parameter for predicting the depth value, and the first learnable parameter and the second learnable parameter are optimized by a loss function.

[0127] In some embodiments, the scale alignment module 503 is used to: construct a minimum objective function based on pixel-level depth prediction values, alignment parameters, point cloud depth values ​​constructed based on 3D reconstruction technology, and pixel-level depth confidence, and find the optimal solution of minimizing the objective function to obtain the value of the alignment parameters.

[0128] In some embodiments, the pose estimation module 504 is used to: adjust the pixel-level depth prediction value based on the alignment parameter to obtain the adjusted pixel-level depth value; and obtain the 2D-3D matching point pair corresponding to the 2D-2D matching point pair based on the camera pose of the reference image, the 2D coordinates in the 2D-2D matching point pair and the adjusted pixel-level depth value.

[0129] Optionally, in some embodiments, such as Figure 6 As shown, in Figure 5Based on the illustrated embodiment, the image localization device may further include a model training module 605. The model training module 605 is configured to: input training samples into a depth estimation model to be trained for depth estimation, obtaining pixel-level depth prediction values ​​and pixel-level depth confidence scores for the training samples; determine a depth loss function based on the pixel-level depth prediction values ​​and pixel-level depth labels of the training samples; determine a confidence loss function using a binary cross-entropy loss function based on the pixel-level depth confidence scores, pixel-level depth prediction values, and pixel-level depth labels of the training samples; determine a loss function based on the depth loss function and the confidence loss function, and adjust the first and second learnable parameters of the depth estimation model according to the loss function.

[0130] In some embodiments, the model training module 605 is used to: determine the confidence label of the training samples based on the pixel-level depth prediction values ​​and pixel-level depth labels of the training samples; and determine the confidence loss function using a binary cross-entropy loss function based on the pixel-level depth confidence and confidence labels of the training samples. Figure 6 601-604 and Figure 5 The 501-504 series have the same function and structure.

[0131] Regarding the image positioning device in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0132] Figure 7 This is a block diagram illustrating a depth estimation model training apparatus according to an exemplary embodiment. Figure 7 As shown, the depth estimation model training device may include: a depth estimation module 701, a first determination module 702, a second determination module 703, a third determination module 704, and a parameter adjustment module 705.

[0133] The depth estimation module 701 is used to input training samples into the depth estimation model to be trained for depth estimation, and obtain pixel-level depth prediction values ​​and pixel-level depth confidence values ​​of the training samples.

[0134] The first determining module 702 is used to determine the depth loss function based on the pixel-level depth prediction values ​​and pixel-level depth labels of the training samples.

[0135] The second determination module 703 is used to determine the confidence loss function based on the pixel-level depth confidence, pixel-level depth prediction value and pixel-level depth label of the training samples, using the binary cross-entropy loss function.

[0136] The third determination module 704 is used to determine the loss function based on the depth loss function and the confidence loss function.

[0137] The parameter adjustment module 705 is used to adjust the first and second learnable parameters of the depth estimation model according to the loss function. The first learnable parameter can be used to predict pixel-level depth confidence, and the second learnable parameter can be used to predict pixel-level depth values.

[0138] In some embodiments, the second determining module 703 is configured to: determine the confidence label of the training sample based on the pixel-level depth prediction value and pixel-level depth label of the training sample; and determine the confidence loss function based on the pixel-level depth confidence and confidence label of the training sample using the binary cross-entropy loss function.

[0139] In the embodiments of this application, the depth estimation model is applied to the image localization task to obtain the pixel-level depth prediction value and pixel-level depth confidence of the reference image. The pixel-level depth prediction value and pixel-level depth confidence of the reference image are used to scale-align with the point cloud depth value constructed based on 3D reconstruction technology to obtain alignment parameters. The alignment parameters are used to combine the pixel-level depth prediction value and the camera pose of the reference image to determine the pose information of the query image.

[0140] Regarding the depth estimation model training apparatus in the above embodiments, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0141] Figure 8 This is a block diagram illustrating an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0142] Reference Figure 8 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0143] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0144] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0145] Power component 806 provides power to various components of electronic device 800. Power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0146] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0147] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0148] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0149] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0150] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0151] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0152] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0153] In an exemplary embodiment, a program product is also provided, including at least one of a program and instructions, which is executed by a processor 820 of an electronic device 800 to implement the steps of the above method.

[0154] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0155] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0156] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0157] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An image localization method, characterized in that, include: The query image of the target scene is input into the reference image library for matching in order to find a reference image with a similar viewpoint to the query image; Depth estimation is performed on the reference image to obtain the pixel-level depth prediction value and pixel-level depth confidence of the reference image; Scale alignment is performed based on the pixel-level depth prediction value, the pixel-level depth confidence value, and the point cloud depth value constructed based on 3D reconstruction technology to obtain alignment parameters; Based on the alignment parameters, the pixel-level depth prediction value, and the camera pose of the reference image, a 2D-3D matching point pair corresponding to the 2D-2D matching point pair between the query image and the reference image is obtained, and the pose information of the query image is determined based on the 2D-3D matching point pair.

2. The method as described in claim 1, characterized in that, The step of performing depth estimation on the reference image to obtain pixel-level depth prediction values ​​and pixel-level depth confidence scores for the reference image includes: The reference image is input into a pre-trained depth estimation model to perform depth estimation, thereby obtaining the pixel-level depth prediction value and pixel-level depth confidence of the reference image; The depth estimation model includes a first learnable parameter for predicting depth confidence and a second learnable parameter for predicting depth values, which are optimized using a loss function.

3. The method as described in claim 2, characterized in that, The depth estimation model is pre-trained in the following manner: The training samples are input into the depth estimation model to be trained to perform depth estimation, and the pixel-level depth prediction value and pixel-level depth confidence value of the training samples are obtained. Based on the pixel-level depth prediction values ​​and pixel-level depth labels of the training samples, a depth loss function is determined; Based on the pixel-level depth confidence, pixel-level depth prediction, and pixel-level depth label of the training samples, the binary cross-entropy loss function is used to determine the confidence loss function. Based on the depth loss function and the confidence loss function, the loss function is determined, and the first learnable parameter and the second learnable parameter of the depth estimation model are adjusted according to the loss function.

4. The method as described in claim 3, characterized in that, The confidence loss function is determined using the binary cross-entropy loss function based on the pixel-level depth confidence, pixel-level depth prediction, and pixel-level depth label of the training samples, including: Based on the pixel-level depth prediction value and pixel-level depth label of the training sample, determine the confidence label of the training sample; Based on the pixel-level depth confidence of the training samples and the confidence labels, the binary cross-entropy loss function is used to determine the confidence loss function.

5. The method as described in claim 1, characterized in that, The process of scaling based on the pixel-level depth prediction value, the pixel-level depth confidence score, and the point cloud depth value constructed based on 3D reconstruction technology yields alignment parameters, including: Based on the pixel-level depth prediction value, the alignment parameter, the point cloud depth value constructed based on 3D reconstruction technology, and the pixel-level depth confidence, a minimization objective function is constructed, and the optimal solution of the minimization objective function is found to obtain the value of the alignment parameter.

6. The method according to any one of claims 1-5, characterized in that, The step of obtaining 2D-3D matching point pairs corresponding to the 2D-2D matching point pairs between the query image and the reference image based on the alignment parameters, the pixel-level depth prediction values, and the camera pose of the reference image includes: The pixel-level depth prediction value is adjusted based on the alignment parameters to obtain the adjusted pixel-level depth value. Based on the camera pose of the reference image, the 2D coordinates in the 2D-2D matching point pair, and the adjusted pixel-level depth value, the corresponding 2D-3D matching point pair is obtained.

7. A method for training a depth estimation model, characterized in that, include: The training samples are input into the depth estimation model to be trained to perform depth estimation, and the pixel-level depth prediction value and pixel-level depth confidence value of the training samples are obtained. Based on the pixel-level depth prediction values ​​and pixel-level depth labels of the training samples, a depth loss function is determined; Based on the pixel-level depth confidence, pixel-level depth prediction, and pixel-level depth label of the training samples, the binary cross-entropy loss function is used to determine the confidence loss function. Based on the depth loss function and the confidence loss function, the loss function is determined, and the first learnable parameter and the second learnable parameter of the depth estimation model are adjusted according to the loss function; the first learnable parameter is used to predict pixel-level depth confidence, and the second learnable parameter is used to predict pixel-level depth value; The depth estimation model is applied to an image localization task to obtain pixel-level depth prediction values ​​and pixel-level depth confidence scores of a reference image. The pixel-level depth prediction values ​​and pixel-level depth confidence scores of the reference image are used to scale-align with the depth values ​​of a point cloud constructed based on 3D reconstruction technology to obtain alignment parameters. The alignment parameters are used to combine the pixel-level depth prediction values ​​and the camera pose of the reference image to determine the pose information of the query image.

8. The method as described in claim 7, characterized in that, The confidence loss function is determined using the binary cross-entropy loss function based on the pixel-level depth confidence, pixel-level depth prediction, and pixel-level depth label of the training samples, including: Based on the pixel-level depth prediction value and pixel-level depth label of the training sample, determine the confidence label of the training sample; Based on the pixel-level depth confidence of the training samples and the confidence labels, the binary cross-entropy loss function is used to determine the confidence loss function.

9. An image positioning device, characterized in that, include: The matching module is used to input the query image of the target scene into the reference image library for matching, so as to find a reference image with a similar viewpoint to the query image; The depth estimation module is used to perform depth estimation on the reference image to obtain the pixel-level depth prediction value and pixel-level depth confidence of the reference image; The scale alignment module is used to perform scale alignment based on the pixel-level depth prediction value, the pixel-level depth confidence value, and the point cloud depth value constructed based on 3D reconstruction technology to obtain alignment parameters. The pose estimation module is used to obtain 2D-3D matching point pairs corresponding to the 2D-2D matching point pairs between the query image and the reference image based on the alignment parameters, the pixel-level depth prediction values ​​and the camera pose of the reference image, and to determine the pose information of the query image based on the 2D-3D matching point pairs.

10. A depth estimation model training device, characterized in that, include: The depth estimation module is used to input training samples into the depth estimation model to be trained for depth estimation, and obtain pixel-level depth prediction values ​​and pixel-level depth confidence values ​​of the training samples. The first determining module is used to determine the depth loss function based on the pixel-level depth prediction value and pixel-level depth label of the training samples; The second determining module is used to determine the confidence loss function based on the pixel-level depth confidence, pixel-level depth prediction value and pixel-level depth label of the training samples, using the binary cross-entropy loss function. The third determining module is used to determine the loss function based on the depth loss function and the confidence loss function; The parameter adjustment module is used to adjust the first learnable parameter and the second learnable parameter of the depth estimation model according to the loss function; the first learnable parameter is used to predict pixel-level depth confidence, and the second learnable parameter is used to predict pixel-level depth value; The depth estimation model is applied to an image localization task to obtain pixel-level depth prediction values ​​and pixel-level depth confidence scores of a reference image. The pixel-level depth prediction values ​​and pixel-level depth confidence scores of the reference image are used to scale-align with the depth values ​​of a point cloud constructed based on 3D reconstruction technology to obtain alignment parameters. The alignment parameters are used to combine the pixel-level depth prediction values ​​and the camera pose of the reference image to determine the pose information of the query image.

11. An electronic device, characterized in that, include: One or more processors; The processor is configured to invoke instructions to cause the electronic device to perform the method of any one of claims 1-6 and 7-8.

12. A storage medium storing instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method of any one of claims 1-6 and 7-8.

13. A program product comprising at least one of a program and instructions, characterized in that, When at least one of the programs or instructions is executed by an electronic device, it implements the steps of the method according to any one of claims 1-6 and 7-8.

Images