Pose determination method and apparatus, electronic device, and storage medium

Abstract

The application discloses a pose determination method and device, electronic equipment and a storage medium, and is applied to electronic equipment including a first camera and a second camera. The first camera collects color images at a first frequency, and the second camera collects depth images at a second frequency smaller than the first frequency. The pose determination method comprises the following steps: obtaining three-dimensional space points corresponding to two-dimensional feature points in multiple color images as to-be-corrected space points according to the obtained multiple color images; obtaining three-dimensional space points corresponding to two-dimensional feature points in a target color image as reference space points according to the obtained depth images, wherein the target color image is a color image matched with the depth image in the multiple color images; correcting the to-be-corrected space points according to the reference space points to obtain corrected target space points; and determining the pose information of the electronic equipment according to the target space points. The application can realize accurate positioning by using depth images with a lower frequency.

Description

Technical Field

[0001] This application relates to the field of positioning technology, and more specifically, to a pose determination method, apparatus, electronic device, and storage medium. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a method that uses a terminal device to move within a scene and capture images of that scene, simultaneously determining the device's own pose and modeling the scene. It is a fundamental technology in fields such as indoor navigation, AR (Augmented Reality), and robotics. Most existing SLAM methods extract features from scene images captured by a camera to determine the camera's pose in real time. This means that the localization process largely considers only the two-dimensional features of the visual image, resulting in insufficient degrees of freedom in localization and inaccurate pose estimation. Summary of the Invention

[0003] In view of the above problems, this application proposes a pose determination method, apparatus, electronic device and storage medium.

[0004] In a first aspect, embodiments of this application provide a pose determination method applied to an electronic device. The electronic device includes a first camera and a second camera. The first camera acquires color images at a first frequency, and the second camera acquires depth images at a second frequency, wherein the second frequency is less than the first frequency. The method includes: acquiring three-dimensional spatial points corresponding to two-dimensional feature points in the acquired multiple color images as spatial points to be corrected; acquiring three-dimensional spatial points corresponding to two-dimensional feature points in a target color image as reference spatial points, wherein the target color image is a color image that matches the depth image among the multiple color images; correcting the spatial points to be corrected based on the reference spatial points to obtain corrected target spatial points; and determining the pose information of the electronic device based on the target spatial points.

[0005] Secondly, embodiments of this application provide a pose determination device applied to an electronic device. The electronic device includes a first camera and a second camera. The first camera acquires color images at a first frequency, and the second camera acquires depth images at a second frequency, wherein the second frequency is less than the first frequency. The device includes: a correction point acquisition module, configured to acquire three-dimensional spatial points corresponding to two-dimensional feature points in the acquired multi-frame color images as spatial points to be corrected; a reference point acquisition module, configured to acquire three-dimensional spatial points corresponding to two-dimensional feature points in a target color image as reference spatial points based on the acquired depth image, wherein the target color image is a color image that matches the depth image among the multi-frame color images; a correction processing module, configured to perform correction processing on the spatial points to be corrected based on the reference spatial points to obtain corrected target spatial points; and a pose determination module, configured to determine the pose information of the electronic device based on the target spatial points.

[0006] Thirdly, embodiments of this application provide an electronic device, including: a first camera and a second camera; one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to perform the pose determination method provided in the first aspect above.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be called by a processor to execute the pose determination method provided in the first aspect above.

[0008] The solution provided in this application allows an electronic device to include a first camera and a second camera. The first camera acquires color images at a first frequency, and the second camera acquires depth images at a second frequency, where the second frequency is lower than the first frequency. This allows for the acquisition of three-dimensional spatial points corresponding to two-dimensional feature points in multiple color images as points to be corrected. Then, based on the acquired depth image, three-dimensional spatial points corresponding to two-dimensional feature points in a target color image are acquired as reference spatial points. The target color image is a single color image that matches the depth image among the multiple color images. The points to be corrected are then corrected based on the reference spatial points to obtain the corrected target spatial points, which are then used to determine the pose information of the electronic device. Since the acquisition of depth and color images in this application does not require maintaining the same frequency, the system's hardware and software power consumption is significantly reduced. Furthermore, spatial information determined solely by two-dimensional features of the image can be corrected and adjusted using spatial information determined by a low-frequency depth image, resulting in more accurate three-dimensional spatial information for precise pose acquisition. This increases the degrees of freedom in positioning and improves positioning accuracy, thereby achieving precise positioning using low-frequency depth images. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart of a pose determination method according to an embodiment of this application is shown.

[0011] Figure 2 A flowchart of a pose determination method according to another embodiment of this application is shown.

[0012] Figure 3 A flowchart of step S210 in a pose determination method according to another embodiment of this application is shown.

[0013] Figure 4 A flowchart of step S220 in a pose determination method according to another embodiment of this application is shown.

[0014] Figure 5 A flowchart of step S221 in a pose determination method according to another embodiment of this application is shown.

[0015] Figure 6 A schematic diagram of an effective depth value filtering method provided in this application is shown.

[0016] Figure 7 A schematic diagram of a method for determining correction parameters provided in this application is shown.

[0017] Figure 8 A schematic diagram of another method for determining correction parameters provided in this application is shown.

[0018] Figure 9 A flowchart of a pose determination method according to yet another embodiment of this application is shown.

[0019] Figure 10 A flowchart of a pose determination method according to another embodiment of this application is shown.

[0020] Figure 11 A schematic diagram of an overall process provided in this application is shown.

[0021] Figure 12 A block diagram of a pose determination apparatus according to an embodiment of this application is shown.

[0022] Figure 13 This is a block diagram of an electronic device for performing a pose determination method according to an embodiment of this application.

[0023] Figure 14 This is a storage unit in this application embodiment for storing or carrying program code that implements the pose determination method according to this application embodiment. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0025] With the rapid development of technology for self-localization using camera sensors, the theory of visual SLAM has gradually matured. However, the drawbacks of using only the camera as the sole sensor for localization have begun to emerge. Specifically, image frames acquired using cameras often lose depth information of the environment. This means that the localization process mostly considers only the two-dimensional features of the visual image, resulting in scale uncertainty in the calculated pose and leading to inaccurate pose estimation.

[0026] To address the aforementioned issues, Visual-IMU Odometry (VIO) technology has been continuously developing. VIO is an algorithm that fuses information from cameras and inertial measurement units (IMUs) to calculate a device's pose in space. By aligning the pose sequences estimated by the IMU with those estimated by the camera, it can estimate the true scale of the camera trajectory. This approach leverages the different characteristics of various sensor types to compensate for the inherent limitations of a single sensor, significantly improving the device's localization capabilities and accelerating the deployment of applications such as indoor navigation, augmented reality, and robotics. However, this method is limited by IMU accuracy and latency. While it can obtain relatively accurate pose information in a short time, prolonged use leads to significant drift, resulting in inaccurate camera tracking and ultimately, unsatisfactory localization performance.

[0027] Although a VIO technique that integrates a depth camera and a color RGB (Red, Green, Blue) camera has been proposed, which can use the absolute scene depth information provided by the depth camera to achieve more accurate pose estimation, it requires that the acquisition frequency of the RGB camera and the depth camera be consistent. Each frame of RGB image needs to correspond to a depth image, and the RGB image and the depth image need to be perfectly aligned in time and space. Otherwise, the algorithm will not work properly, resulting in very high system power consumption. At the same time, it also places high demands on the depth camera hardware, requiring it to ensure high-frequency and stable output.

[0028] To address the aforementioned problems, the inventors have proposed a pose determination method, apparatus, electronic device, and storage medium as described in the embodiments of this application. For spatial information determined solely based on two-dimensional features of an image, spatial information determined from low-frequency depth images can be used for correction and adjustment, thereby obtaining more accurate three-dimensional spatial information for precise pose acquisition. This increases the degrees of freedom in positioning and improves positioning accuracy, thus achieving precise positioning from low-frequency depth images. The specific pose determination method will be described in detail in subsequent embodiments.

[0029] The following describes the application environment in which the pose determination method provided in the embodiments of this application can be used.

[0030] In embodiments of this application, the electronic device may include a first camera and a second camera. The first camera may be a camera for acquiring color images, such as a color RGB (Red, Green, Blue) camera, and the second camera may be a camera for acquiring depth images, such as a depth camera based on structured light principles or a depth camera based on Time of Flight (TOF) principles. The TOF-based depth camera may be based on direct Time of Flight (dTOF) principles or indirect Time of Flight (iTOF) principles; no limitation is made in this application.

[0031] In this embodiment, the first camera may acquire color images at a first frequency, and the second camera may acquire depth images at a second frequency, wherein the second frequency is lower than the first frequency. Compared with the prior art, this application abandons high-frequency depth image acquisition and adopts a strategy of low-frequency control for depth image acquisition, which greatly reduces the system's hardware and software power consumption. Specifically, in terms of hardware power consumption, since high-frequency depth image acquisition is no longer required, a high-performance depth camera is not needed, reducing the requirements for depth camera hardware. It is not necessary to ensure stable high-frequency output from the depth camera, resulting in reduced hardware power consumption. In terms of software power consumption, it is not necessary to acquire depth images and perform time alignment for every frame of color image, thus reducing the system's software power consumption.

[0032] In some embodiments, the first frequency may be a fixed multiple of the second frequency. The specific multiple is not limited here, as long as the second frequency is less than the first frequency. For example, when the first frequency for the first camera to acquire color images is 30Hz, the first frequency may be three times the second frequency, that is, the second frequency is one-third of the first frequency. In other words, the second frequency for the second camera to acquire depth images is 10Hz, so that one depth image can be acquired for every three color images, thereby reducing the hardware power consumption to one-third of the original.

[0033] In other embodiments, the first frequency may not be fixedly related to the second frequency; that is, it may not strictly adhere to the strategy of acquiring one depth image frame for every few frames of color images acquired. The second frequency only needs to be lower than the first frequency. For example, depth image quality can be assessed so that the second camera acquires images at a fixed frequency only when the depth image acquisition quality is good, and does not acquire images when the depth image acquisition quality is poor.

[0034] In some embodiments, the first camera and the second camera can be set independently or as an integrated unit. For example, a color camera-depth camera (RGB-D, RGB-D) can be formed, which can provide both color RGB images and depth images.

[0035] In some embodiments, multiple first cameras and second cameras may be provided on the electronic device (e.g., four or three first cameras), and other devices may be added, such as inertial measurement units (IMUs) that are paired with the first and second cameras, to further help determine the pose of the camera or electronic device.

[0036] Please see Figure 1 , Figure 1 A flowchart illustrating a pose determination method according to an embodiment of this application is shown. In a specific embodiment, the pose determination method is applied to, for example... Figure 12 The pose determination device 600 shown and the electronic device 100 equipped with the pose determination device 600 are also shown. Figure 13 The following will address... Figure 1 The process shown will be described in detail. The video playback method may specifically include the following steps:

[0037] Step S110: Based on the acquired multi-frame color images, obtain the three-dimensional spatial points corresponding to the two-dimensional feature points in the multi-frame color images as spatial points to be corrected.

[0038] In this embodiment, since the second frequency for acquiring depth images is lower than the first frequency for acquiring color images, not every frame of color image corresponds to a frame of depth image. Therefore, for color images without a corresponding depth image, the pose of the camera or device cannot be accurately estimated based on the depth information. Thus, in this embodiment, when acquiring a color image from the first camera, the three-dimensional spatial points corresponding to the two-dimensional feature points in the color image can be obtained without relying on the depth image.

[0039] Understandably, since the pose obtained from only the two-dimensional features of the image will have scale uncertainty, when obtaining the corresponding three-dimensional spatial points from only the two-dimensional features of the color image, it is necessary to correct the three-dimensional space. That is, the three-dimensional spatial points obtained from only the two-dimensional features of the color image are taken as spatial points to be corrected, so that scale correction can be performed later.

[0040] Because the image points of the same 3D spatial point in a 3D scene have epipolar constraints under different viewpoints, in some embodiments, the 3D spatial points corresponding to the 2D feature points of the color images can be obtained based on the epipolar constraints between color images from different viewpoints. Specifically, 2D feature points can be extracted from color images from different viewpoints, and feature point matching can be performed to obtain a set of 2D feature points that are matched one-to-one. Then, the 3D spatial points corresponding to the 2D feature points in the color images from different viewpoints can be obtained based on the set of 2D feature points that are matched one-to-one. The color images from different viewpoints can be color images from adjacent frames or color images separated by several frames; this is not limited here.

[0041] In some embodiments, the two-dimensional feature points in the color image can be pixels in the color image, or they can be extracted using a feature extraction algorithm. These feature extraction algorithms may include, but are not limited to, FAST feature point detection algorithm, DOG feature point detection algorithm, Harris feature point detection algorithm, SIFT feature point detection algorithm, SURF feature point detection algorithm, etc.

[0042] In some embodiments, to facilitate optimization of the current camera or device pose, or optimization of recently observed 3D spatial points, a sliding window can be set up. This window stores the most recent few frames of color images, enabling local optimization and thus achieving stable pose tracking. The sliding window can be a pre-constructed fixed-length sliding window containing a fixed number of image frames. Like a queue, the sliding window continuously adds new color image frames and removes old ones. The sliding window can record the pose and 3D spatial point data of the current color image frame and previous color image frames.

[0043] Understandably, at the initial moment, the first camera has not yet captured any color images of the scene. Only after positioning and tracking begins will the first camera follow the electronic device as it moves within the scene, capturing color images as it moves, forming a continuous stream of color images. After obtaining a certain number of color images, the electronic device can initialize a 3D map model of the scene, typically including only a small portion of the scene or differing from the actual scene. Subsequently, for each color image captured by the first camera, the electronic device can update and optimize the 3D map model based on that image. This involves calculating the 3D spatial points corresponding to the 2D feature points in the color image to add 3D spatial points (also called 3D map points) that are not present in the 3D map model. Since these 3D spatial points are obtained only from the 2D features of the color image, they have scale uncertainties. Therefore, these 3D spatial points need to be treated as points to be corrected, awaiting subsequent optimization and correction.

[0044] Step S120: Based on the acquired depth image, obtain the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image as reference spatial points, wherein the target color image is a color image that matches the depth image among the multiple color images.

[0045] In this embodiment, since not every frame of color image corresponds to a frame of depth image, for a target color image that has a corresponding depth image, the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image can be calculated again based on its corresponding depth image. The depth image can contain distance information in the depth direction between multiple points on the surface of the target object in the scene and the depth camera, similar to a grayscale image, except that a single pixel value represents the actual distance between the depth camera and a point on the surface of the target object.

[0046] Since the depth information of the scene in the target color image can be clearly obtained from the corresponding depth image, the 3D spatial points corresponding to the 2D feature points in the target color image obtained from the depth information in the depth image are relatively accurate, and the subsequent pose estimation will not introduce scale uncertainty. Therefore, the 3D spatial points corresponding to the 2D feature points in the target color image obtained from the depth image can be used as reference spatial points to correct the aforementioned space to be corrected. The reference spatial points can be understood as relatively accurate 3D spatial points that do not require correction.

[0047] Since the second frequency for acquiring depth images is inconsistent with the first frequency for acquiring color images, it is necessary to align the depth image acquired by the second camera with the color image acquired by the first camera when obtaining the depth image.

[0048] Alignment includes time alignment. Specifically, when acquiring a depth image, its timestamp can be matched with the timestamps of multiple acquired color images to obtain a color image whose timestamp is time-aligned with the depth image. Optionally, since depth and color images may not exist at exactly the same time in reality, the depth image and color image can be used as a time-aligned pair if the time difference between their timestamps is less than the error tolerance value.

[0049] The alignment process also includes spatial alignment between two-dimensional pixels. Specifically, this can be achieved by spatially aligning pixels in the color image and the corresponding depth image based on the extrinsic parameters between the first and second cameras. The extrinsic parameters refer to the rotation and translation matrices between the camera coordinate systems of the first and second cameras, including rotation and translation parameters along three axes. Since the first and second cameras are typically calibrated beforehand to obtain the extrinsic parameter calibration parameters, these parameters are usually known. Therefore, through this alignment, a pair of depth and color images that are aligned and matched both temporally and spatially can be obtained.

[0050] In this embodiment of the application, since the target color image is a color image that matches and aligns with the depth image among multiple color images, the depth value corresponding to the two-dimensional feature point in the target color image can be found on the depth image. Then, the depth value can be converted into three-dimensional coordinates through the intrinsic parameter matrix of the depth camera, thereby obtaining the three-dimensional spatial point corresponding to the two-dimensional feature point in the target color image.

[0051] Step S130: Based on the reference spatial point, perform correction processing on the spatial point to be corrected to obtain the corrected target spatial point.

[0052] Since the three-dimensional spatial points obtained using depth information are relatively accurate, in this embodiment of the application, after obtaining the reference spatial points obtained from the depth image, the aforementioned obtained spatial points to be corrected can be corrected based on the reference spatial points, thereby obtaining more accurate three-dimensional spatial points after correction, which can be used as target spatial points for calculating pose.

[0053] It is understandable that, since each time a new color image frame is acquired, a spatial point with scale uncertainty is calculated based on step S110, and when a depth image is acquired, a reference spatial point without scale uncertainty is calculated again for the target color image corresponding to the depth image based on step S120, there will inevitably be two-dimensional feature points that use two methods to calculate three-dimensional spatial points at the same time. Since the reference spatial point corresponding to the two-dimensional feature point is relatively accurate, the difference between the reference spatial point corresponding to the two-dimensional feature point and the spatial point to be corrected can reflect the degree of correction required to adjust the spatial point to be corrected to a more accurate position.

[0054] In some embodiments, the reference space and the space points to be corrected may be in different coordinate systems. Therefore, before correcting the space points to be corrected based on the reference space points, the reference space points and the space points to be corrected can be transformed to the same coordinate system. As one approach, since the coordinate system of the first frame of the color image is generally used as the world coordinate system, both the reference space points and the space points to be corrected can be transformed to the coordinate system of the first frame of the color image. Then, based on the difference between the reference space points and the space points to be corrected corresponding to the same two-dimensional feature point, all the previously obtained space points to be corrected can be corrected to obtain the corrected target space points.

[0055] Step S140: Determine the pose information of the electronic device based on the target spatial point.

[0056] In this embodiment of the application, after obtaining a relatively accurate target spatial point, the pose information of the electronic device can be determined based on that target spatial point. The pose information includes position information and attitude information. The position information is the translation information in the x, y, and z directions of the coordinate system, and the attitude information is the rotation information in the x, y, and z directions of the coordinate system.

[0057] In some embodiments, the pose information of an electronic device can be calculated using the Perspective-n-Point (PnP) algorithm. PnP is an algorithm that solves for the motion between two frames of color images using a set of matched 3D spatial points and 2D feature points. It describes how to estimate the camera pose when n 3D spatial points and their projected positions are known. Therefore, the pose information of the electronic device can be determined based on the target spatial point and its corresponding 2D feature points.

[0058] In some embodiments, pose optimization can also be performed jointly using IMU data. Specifically, when acquiring color images, multiple sets of corresponding IMU data can be acquired, and then the acquired IMU data is pre-integrated to obtain inertial data. Simultaneously, the target spatial point is registered with the inertial data, and VIO optimization is performed based on the registration result to solve for a more accurate pose. VIO optimization can be performed using a first constraint (i.e., a cost function) constructed using the target spatial point and a second constraint constructed using the inertial data. Then, a nonlinear optimization method is used to optimize the cost function, continuously reducing its value to determine the pose optimization result. This application does not specifically limit the nonlinear optimization method; for example, it can be the Gauss-Newton algorithm, the Levenberg-Marquardt algorithm, etc.

[0059] The pose determination method provided in this application embodiment includes an electronic device that may include a first camera and a second camera. The first camera acquires color images at a first frequency, and the second camera acquires depth images at a second frequency, where the second frequency is less than the first frequency. This allows for the first acquisition of three-dimensional spatial points corresponding to two-dimensional feature points in multiple color images as points to be corrected. Then, based on the acquired depth image, three-dimensional spatial points corresponding to two-dimensional feature points in a target color image are acquired as reference spatial points. The target color image is a single color image that matches the depth image among the multiple color images. Finally, based on the reference spatial points, the points to be corrected are corrected to obtain the corrected target spatial points, which are then used to determine the pose information of the electronic device. Since the acquisition of depth images and color images in this application does not need to be at the same frequency, it greatly reduces the power consumption of the system's hardware and software. Furthermore, for spatial information determined solely by the two-dimensional features of the image, the spatial information determined by the low-frequency depth image can be used for correction and adjustment, thereby obtaining more accurate three-dimensional spatial information for precise pose acquisition. This increases the degree of freedom in positioning and improves positioning accuracy, thus achieving precise positioning of low-frequency depth images, which can be applied to low-performance devices.

[0060] Please see Figure 2 , Figure 2 A flowchart illustrating a pose determination method according to another embodiment of this application is shown. This pose determination method is applied to the aforementioned electronic device, and will be discussed below. Figure 2 The process shown will be described in detail. The pose determination method may specifically include the following steps:

[0061] Step S210: Based on the acquired multi-frame color images, obtain the three-dimensional spatial points corresponding to the two-dimensional feature points in the multi-frame color images as spatial points to be corrected.

[0062] In some embodiments, the two-dimensional feature points of adjacent color images may be triangulated to determine the three-dimensional spatial point information of adjacent color images based on the two-dimensional feature points. For details, please refer to... Figure 3 Step S210 may include:

[0063] Step S211: Based on the acquired multi-frame color images, match the two-dimensional feature points in adjacent color images to obtain matching two-dimensional feature points.

[0064] In some embodiments, each time a new color image is acquired, the two-dimensional feature points of the new color image can be matched with the two-dimensional feature points of the previous frame of the color image to obtain a one-to-one matching set of two-dimensional feature points. In other embodiments, after matching feature points with the previous frame of the color image, feature point matching can also be performed with the frame before that, or feature point matching can be performed on any two frames of color images existing in the sliding window to recover the three-dimensional spatial points corresponding to each two-dimensional feature point in the color image to the greatest extent possible.

[0065] Step S212: Triangulate the matching two-dimensional feature points to obtain the three-dimensional spatial points corresponding to the matching two-dimensional feature points in the adjacent color images as spatial points to be corrected.

[0066] In this embodiment, after obtaining matching two-dimensional feature points in adjacent color images, triangulation can be performed on the matching two-dimensional feature points to obtain the corresponding three-dimensional spatial points in each frame of the adjacent color images. Specifically, triangulation, also known as triangulation, refers to determining the distance of a point by observing the angle between two points.

[0067] Step S220: Based on the acquired depth image, obtain the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image as reference spatial points, wherein the target color image is a color image that matches the depth image among the multiple color images.

[0068] Since each pixel in a depth image reflects the actual distance between the depth camera and the target point, that is, it reflects the depth information of the environment, the three-dimensional spatial points corresponding to each two-dimensional feature point in the color image can be obtained by combining the two-dimensional features of the color image with the environmental depth. For details, please refer to [link to relevant documentation]. Figure 4 Step S220 may include:

[0069] Step S221: Based on the acquired depth image, obtain the effective depth value corresponding to the two-dimensional feature point in the target color image.

[0070] Step S222: Convert the effective depth value into three-dimensional coordinates to obtain the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image as reference spatial points.

[0071] Since the value of each pixel in the depth image represents the actual distance between the depth camera and the target point, after spatially aligning the pixels of the depth image with those in the target color image, the effective depth value corresponding to each pixel (i.e., two-dimensional feature point) in the target color image can be obtained based on the depth image and this alignment relationship. This effective depth value can then be converted into three-dimensional coordinates to obtain the three-dimensional spatial point corresponding to each two-dimensional feature point in the target color image.

[0072] Since not all depth information in a depth image is accurate—for example, when the second camera is an iTOF-based depth camera—the measurement principle of iTOF cameras can lead to large depth errors at scene edges and in low-reflectivity areas. Therefore, in some embodiments, valid depth information can be filtered out before acquiring 3D spatial points based on the depth information. For details, please refer to… Figure 5 Step S221 may include:

[0073] Step S2211: Based on the acquired depth image, obtain the target depth value corresponding to the target two-dimensional feature point in the target color image, wherein the target two-dimensional feature point is any two-dimensional feature point in the target color image.

[0074] In some embodiments, feature points in the depth image whose depth values ​​exceed the range of the second camera can be discarded first. For example, if the effective range of the second camera is 0.1-6m, then feature points with depth values ​​less than 0.1m and greater than 6m can be discarded.

[0075] When all depth values ​​in the depth image are within the range of the second camera, it is still necessary to determine whether the depth values ​​of the feature points are reliable and accurate. Generally, if the depth values ​​within a certain neighborhood of a feature point change very little or linearly, the feature point can be considered to be located on a small plane, and the depth value corresponding to the feature point is of high accuracy and is a reliable value. Otherwise, it is unreliable and can be discarded.

[0076] In this embodiment, a two-dimensional feature point can be randomly selected from the target color image as the target two-dimensional feature point, and the reliability of the target depth value corresponding to the target two-dimensional feature point can be tested. Specifically, after spatially aligning the pixels of the depth image with the pixels of the target color image, the target depth value corresponding to the target two-dimensional feature point in the target color image can be obtained based on the depth image and the alignment relationship, so as to compare the difference between the target depth value and the corresponding depth values ​​around the feature point.

[0077] Step S2212: Based on the depth image, obtain the average depth value of all neighboring two-dimensional feature points within a specified area surrounding the target two-dimensional feature point.

[0078] In this embodiment, after spatially aligning the pixels of the depth image with the pixels of the target color image, the depth values ​​corresponding to all neighboring two-dimensional feature points within a specified area surrounding the target two-dimensional feature point can be obtained based on the depth image and the alignment relationship. Then, a matching process is performed to calculate the average depth value. The specified area can be a fixed-size area, such as a 7px*7px square, or an area with a fixed number of feature points, such as the area containing 30 surrounding two-dimensional feature points; no limitation is made here.

[0079] Step S2213: When the difference between the target depth value and the average depth value is less than a specified threshold, the target depth value is taken as the effective depth value corresponding to the target two-dimensional feature point in the target color image.

[0080] In this embodiment of the application, when the target depth value corresponding to the target two-dimensional feature point is obtained, the target depth value can be compared with the average depth value of all neighboring two-dimensional feature points in a specified area around the target two-dimensional feature point to determine whether the depth values ​​are not significantly different, thereby determining whether the target depth value corresponding to the target two-dimensional feature point is a valid depth value.

[0081] Specifically, when the difference between the target depth value and the average depth value is less than a specified threshold, the target depth value corresponding to the target two-dimensional feature point is determined as a valid depth value. This target depth value can then be used as the valid depth value corresponding to the target two-dimensional feature point in the target color image for subsequent three-dimensional spatial point reconstruction. The specified threshold can be pre-stored in an electronic device and can be reasonably set according to actual conditions; no limitation is made here.

[0082] Understandably, when the target depth value is not significantly different from the average depth value, it can be assumed that the target 2D feature point and its neighboring 2D feature points lie on a small plane. Therefore, the target depth value is likely reliable and accurate, and can be retained. Conversely, if the target depth value differs significantly from the average depth value, it can be assumed that the target 2D feature point and its neighboring 2D feature points are not on a small plane. The target depth value of this 2D feature point has undergone a sudden change, and this target depth value is not necessarily reliable; therefore, it can be discarded. In this way, most inaccurate depth values ​​can be filtered out, and the retained depth values ​​are generally accurate and reliable.

[0083] For example, please refer to Figure 6Centered on the extracted feature point (hollow circle), move 3 pixels up, down, left, and right to obtain a 7x7 neighborhood square (dashed box). Extract the depth values ​​(solid triangles, also 48) of all pixels within the square (excluding the center point) and sum them to calculate the average depth. If the difference between the depth value of the center feature point (hollow circle) and the average depth value within the square exceeds a specified threshold, the depth value of the center feature point is considered unreliable and must be discarded. This filters out most inaccurate depth values, retaining only accurate and reliable ones.

[0084] After filtering unreliable depth values ​​in the depth image, the accurate effective depth values ​​corresponding to the two-dimensional feature points in the target color image can be obtained from the filtered depth image. These effective depth values ​​can then be converted into three-dimensional coordinates, thereby obtaining the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image more accurately, which can then be used as reference spatial points.

[0085] Step S230: Match the two-dimensional feature point corresponding to the reference spatial point with the two-dimensional feature point corresponding to the spatial point to be corrected to obtain the reference spatial point and the spatial point to be corrected corresponding to the same two-dimensional feature point.

[0086] In this embodiment, after obtaining the three-dimensional spatial points corresponding to two-dimensional feature points in a color image using triangulation and depth information, two-dimensional feature points obtained using both methods can be identified. The differences in the three-dimensional spatial points obtained using different methods for the same two-dimensional feature point are used to determine the deviation of the triangulation method, thereby reasonably correcting the three-dimensional spatial points obtained by triangulation. Specifically, the two-dimensional feature points corresponding to the obtained reference spatial points can be matched with the two-dimensional feature points corresponding to the obtained spatial points to be corrected. This yields a matching pair of reference and spatial points corresponding to the same two-dimensional feature point, which can then be used to correct the spatial points to be corrected.

[0087] Step S240: Determine the correction parameters based on the reference spatial point corresponding to the same two-dimensional feature point and the spatial point to be corrected.

[0088] In this embodiment, after obtaining a matching pair consisting of a reference spatial point and a spatial point to be corrected corresponding to the same two-dimensional feature point, the correction parameter can be determined based on the reference spatial point and the spatial point to be corrected. One approach is to calculate the correction parameter to be adjusted for the spatial point to be corrected using the least squares method, with the following formula: S0M1-M2=0, where M1 is the set of spatial points to be corrected obtained from the color image, M2 is the set of reference spatial points obtained from the depth image, and S0 is the correction parameter.

[0089] Since the coordinate system of the first frame color image is usually used as the world coordinate system, in some embodiments, the reference space point and the space point to be corrected corresponding to the same two-dimensional feature point can be transformed to the coordinate system of the first frame (i0 frame) color image before the correction parameter is calculated, so as to obtain the correction parameter S0 in the coordinate system of the first frame color image.

[0090] For example, when the two-dimensional feature point is a feature point in the fourth frame (i3 frame) color image, the coordinate transformation can be performed using the following formula: M3 = R 03 M2+t 03 Where R03 and t03 are the rotation and translation of frame i3 relative to frame i0, respectively, and M3 is the set of reference points M in the i3 coordinate system after transformation to the i0 coordinate system.

[0091] In this context, R03 and T03 are known values. It can be understood that during pose calculation, the poses of all color image frames within a sliding window are calculated one at a time. Then, in the next calculation, the first color image frame in the window is removed, and a new color image frame is added to the window, and so on, continuously removing older color image frames and adding new ones to optimize the pose. That is, when there are 10 color image frames in a sliding window, before pose optimization begins, only the newly added frame has not undergone optimization calculation (the other 9 frames were already in the window; only the newly added frame is being added to the optimization window for the first time). Even so, the pose of the newly added frame is known; this pose comes from the IMU's integration calculation. This IMU integration result is used as the initial pose value for the newly added frame, thus yielding R03 and T03.

[0092] For example, please refer to Figure 7 , Figure 7A schematic diagram illustrating the determination of correction parameters is shown. When acquiring three color images (i0, i1, i2), since a depth image is not available, the three-dimensional spatial points corresponding to the two-dimensional feature points in these three color images can only be calculated using triangulation. Due to the continuous scale drift during tracking, the triangulated three-dimensional spatial points become increasingly inaccurate; therefore, they need to be used as spatial points to be corrected in subsequent adjustments. Figure 7 As can be seen, the corresponding depth image is only obtained when the i3-frame color image is acquired. Since this depth image corresponds to the i3-frame color image, the two-dimensional feature points in the i3-frame color image have depth information. At this point, the three-dimensional spatial points corresponding to each two-dimensional feature point in the i3-frame color image can be calculated based on this depth image. Since obtaining the three-dimensional spatial points based on the depth information is more accurate, it can be used as a reference spatial point to correct the aforementioned spatial points to be corrected.

[0093] Furthermore, since the coordinate system of the reference spatial point obtained from the i3 frame color image is the same as the coordinate system of the i3 frame color image, it needs to be transformed to the coordinate system of the i0 frame color image (i.e., the world coordinate system). Similarly, the spatial points to be corrected obtained above also need to be transformed to the coordinate system of the i0 frame color image (i.e., the world coordinate system). This allows us to find matching pairs between the spatial points to be corrected and the reference spatial points corresponding to the same two-dimensional feature point, and to calculate the correction parameter s0 in the i0 coordinate system based on all matching pairs and the least squares method.

[0094] Step S250: Based on the correction parameters, perform correction processing on the spatial point to be corrected to obtain the corrected target spatial point.

[0095] Since not all color images have corresponding depth images, meaning not all two-dimensional feature points can be accurately reconstructed into three-dimensional spatial points, most two-dimensional feature points result in spatial points with scale uncertainties that require correction. Therefore, in this embodiment, after obtaining the correction parameter, all spatial points to be corrected can be corrected according to this parameter to obtain a more accurate target spatial point. Specifically, correcting all spatial points to be corrected can be achieved by multiplying all corrected spatial points by the correction parameter; the result is the corrected target spatial point.

[0096] In some embodiments, once the spatial point to be corrected has been modified by the correction parameters to obtain a more accurate target spatial point, even if a depth image is acquired later, it is not necessary to perform correction again. That is, it is only necessary to correct the previously acquired spatial point to be corrected using the most recently acquired depth image, and only once is required, thus reducing system power consumption.

[0097] For example, please refer to Figure 8 , Figure 8 A schematic diagram of a spatial point correction method is shown. Specifically, as the sliding window continuously slides, new color image frames are added to the window. These color image frames bring new three-dimensional spatial points, which are recovered using triangulation. For example, when color images i4, i5, and i6 are added to the sliding window, these frames, through triangulation, will recover some new three-dimensional spatial points M4 corresponding to two-dimensional feature points that were not observed in previous frames. After coordinate system transformation, these map points are added to the three-dimensional spatial point M1 in the i0 coordinate system. At this time, M1 contains some new three-dimensional spatial points. These three-dimensional spatial points have not undergone correction, so these new three-dimensional spatial points can be used as spatial points to be corrected.

[0098] A new depth image is only acquired when the i7 frame color image arrives. At this point, the 3D spatial point M5 corresponding to some relatively accurate 2D feature points in the i7 frame color image can be recovered from the depth image and used as a reference spatial point. Since some of the 2D feature points may have already had their corresponding 3D spatial points recovered using triangulation, these points can be matched with the 3D spatial points recovered from the depth image corresponding to the i7 frame color image, forming matching point pairs of the same 2D feature point. Using these matching point pairs, new correction parameters can be calculated. These correction parameters can then be used to correct all uncorrected spatial points that need correction. This process is repeated, and the spatial points to be corrected within the sliding window are continuously obtained as reference spatial points from low-frequency depth images, resulting in intermittent correction. The calculation of the correction parameters also requires conversion to the coordinate system of the i0 frame color image (i.e., the world coordinate system), which will not be elaborated here.

[0099] Step S260: Determine the pose information of the electronic device based on the target spatial point.

[0100] In this embodiment, step S260 can be referred to the foregoing embodiments, and will not be repeated here.

[0101] The pose determination method provided in this application embodiment includes an electronic device that may include a first camera and a second camera. The first camera acquires color images at a first frequency, and the second camera acquires depth images at a second frequency, where the second frequency is less than the first frequency. This allows for the first acquisition of three-dimensional spatial points corresponding to two-dimensional feature points in multiple color images as points to be corrected. Then, based on the acquired depth image, three-dimensional spatial points corresponding to two-dimensional feature points in a target color image are acquired as reference spatial points. The target color image is a single color image that matches the depth image among the multiple color images. Finally, based on the reference spatial points, the points to be corrected are corrected to obtain the corrected target spatial points, which are then used to determine the pose information of the electronic device. Since the acquisition of depth and color images in this application does not need to be at the same frequency, it greatly reduces the system's hardware and software power consumption. Furthermore, for spatial information determined solely by the two-dimensional features of the image, spatial information determined by the low-frequency depth image can be used for correction and adjustment, thereby obtaining more accurate three-dimensional spatial information for precise pose acquisition. This increases the degrees of freedom in positioning and improves positioning accuracy, thus achieving precise positioning using low-frequency depth images, applicable to low-performance devices. Moreover, the algorithm for scaling landmark points corresponding to high-frequency color images using low-frequency depth images avoids the use of a large amount of redundant depth information and also avoids introducing excessive inherent noise from depth images into the VIO system through frequent use of depth information. This enables more accurate and efficient positioning.

[0102] Please see Figure 9 , Figure 9 A schematic flowchart of a pose determination method according to another embodiment of this application is shown. This pose determination method is applied to the above-mentioned electronic device, and will be discussed below. Figure 7 The process shown will be described in detail. The pose determination method may specifically include the following steps:

[0103] Step S310: Based on the acquired multi-frame color images, obtain the three-dimensional spatial points corresponding to the two-dimensional feature points in the multi-frame color images as spatial points to be corrected.

[0104] Step S320: Determine whether a depth image has been acquired. If yes, proceed to steps S330 to S350. If no, proceed to step S360.

[0105] In some embodiments, when the frequency of depth image acquisition by the second camera is irregular or the quality of the acquired images is poor, the strategy of acquiring one depth image only after acquiring a preset number of color images may not be strictly followed. Instead, the decision to perform subsequent correction processing may be determined by detecting whether a depth image has been acquired. Specifically, when a depth image is acquired, reference spatial points can be obtained based on the depth image, and correction processing can be performed on the spatial points to be corrected based on the reference spatial points, so as to determine the pose information of the electronic device based on the corrected target spatial points. When a depth image is not acquired, the pose information of the electronic device can be roughly calculated based on the spatial points to be corrected to roughly determine the current pose.

[0106] In some embodiments, determining whether a depth image has been acquired can be done by determining whether a depth image with a quality greater than a specified quality has been acquired. This allows for selective use of the depth image for pose optimization only when the input depth image is good, resulting in greater adaptability and lower performance requirements for the depth camera.

[0107] Step S330: Based on the depth image, obtain the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image as reference spatial points, wherein the target color image is a color image that matches the depth image among the multiple color images.

[0108] Step S340: Based on the reference spatial point, perform correction processing on the spatial point to be corrected to obtain the corrected target spatial point.

[0109] Step S350: Determine the pose information of the electronic device based on the target spatial point.

[0110] In the embodiments of this application, steps S330 to S350 can be referred to the content of the foregoing embodiments, and will not be repeated here.

[0111] Step S360: Determine the pose information of the electronic device based on the spatial point to be corrected.

[0112] In this embodiment of the application, when a depth image is not obtained, the pose information of the electronic device can be roughly calculated based on the spatial points to be corrected in order to roughly determine the current pose.

[0113] In some embodiments, to improve accuracy, after a preliminary calculation of the pose information of the electronic device based on the spatial points to be corrected, the pose can be jointly optimized using IMU data. Specifically, this can involve constructing a first constraint (i.e., a cost function) using the spatial points to be corrected and a second constraint using the IMU data, and then using a nonlinear optimization method to optimize the cost function, thereby continuously reducing the value of the cost function to obtain a more accurate pose.

[0114] The pose determination method provided in this application embodiment includes an electronic device that may include a first camera and a second camera. The first camera acquires color images at a first frequency, and the second camera acquires depth images at a second frequency, where the second frequency is less than the first frequency. This allows for the initial acquisition of three-dimensional spatial points corresponding to two-dimensional feature points in multiple color images as points to be corrected. Then, when a depth image is acquired, three-dimensional spatial points corresponding to two-dimensional feature points in a target color image can be acquired as reference spatial points. The target color image is a single color image that matches the depth image among the multiple color images. Based on these reference spatial points, the points to be corrected are then corrected to obtain the corrected target spatial points. The pose information of the electronic device is then determined based on these target spatial points. Since the acquisition of depth images and color images in this application does not need to be at the same frequency, it greatly reduces the power consumption of the system's hardware and software. Furthermore, for spatial information determined solely by the two-dimensional features of the image, the spatial information determined by the low-frequency depth image can be used for correction and adjustment, thereby obtaining more accurate three-dimensional spatial information for precise pose acquisition. This increases the degree of freedom in positioning and improves positioning accuracy, thus achieving precise positioning of low-frequency depth images, which can be applied to low-performance devices.

[0115] Please see Figure 10 , Figure 10 A schematic flowchart of a pose determination method according to another embodiment of this application is shown. This pose determination method is applied to the above-mentioned electronic device, and will be discussed below. Figure 10 The process shown will be described in detail. The pose determination method may specifically include the following steps:

[0116] Step S410: Based on the acquired multi-frame color images, obtain the three-dimensional spatial points corresponding to the two-dimensional feature points in the multi-frame color images as spatial points to be corrected.

[0117] Step S420: Determine whether a depth image has been acquired. If yes, proceed to steps S430 to S470. If no, proceed to steps S480 to S500.

[0118] Step S430: When the depth image is obtained, the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image are obtained as reference spatial points based on the depth image, wherein the target color image is a color image that matches the depth image among the multiple color images.

[0119] Step S440: Based on the reference spatial point, perform correction processing on the spatial point to be corrected to obtain the corrected target spatial point.

[0120] Step S450: Determine the pose information of the electronic device based on the target spatial point.

[0121] Step S460: Construct the first projection error based on the target spatial point.

[0122] In this embodiment, after determining the pose information of the electronic device based on the target spatial point, the pose can be optimized. Specifically, after obtaining a more accurate target spatial point, a first reprojection error can be constructed using the target spatial point. The reprojection error refers to the error between the projected point (theoretical value) and the measured point on the image. Specifically, the first reprojection error is obtained by acquiring the two-dimensional projection point of the target spatial point in the color image and calculating the error between the two-dimensional projection point and the corresponding two-dimensional feature point of the target spatial point.

[0123] Step S470: Optimize the pose information of the electronic device based on the first projection error.

[0124] In this embodiment of the application, after obtaining the first projection error, the first constraint condition can be constructed using the first projection error, and then the cost function can be optimized using a nonlinear optimization method, so that the value of the cost function continuously decreases, thereby minimizing the distance between all two-dimensional projection points and the corresponding two-dimensional feature points, so as to gradually optimize a more accurate pose.

[0125] In some embodiments, after constructing the first constraint using the first projection error, a second constraint can also be constructed using IMU data, and then a nonlinear optimization method can be used to perform tightly coupled joint optimization on the cost function, so that the value of the cost function continuously decreases.

[0126] In some embodiments, since the target spatial point is a corrected and relatively accurate 3D spatial point, its value can be fixed during the optimization process and no longer used as an optimization variable, thus eliminating the need for optimization along with the pose. In this way, a large number of map points are fixed, significantly saving optimization time.

[0127] Step S480: Determine the pose information of the electronic device based on the spatial point to be corrected.

[0128] Step S490: Construct a second projection error based on the spatial point to be corrected.

[0129] In this embodiment, after determining the pose information of the electronic device based on the spatial point to be corrected, the pose can be optimized. Specifically, a second projection error can be constructed using the spatial point to be corrected. Specifically, the second projection error is obtained by acquiring the two-dimensional projection point of the spatial point to be corrected in the color image and calculating the error between the two-dimensional projection point and the corresponding two-dimensional feature point of the spatial point to be corrected.

[0130] Step S500: Optimize the pose information of the electronic device and the spatial point to be corrected based on the second projection error.

[0131] In this embodiment of the application, after obtaining the second projection error, the first constraint condition can be constructed using the second projection error, and then the cost function can be optimized using a nonlinear optimization method, so that the value of the cost function continuously decreases, thereby minimizing the distance between all two-dimensional projection points and the corresponding two-dimensional feature points, so as to gradually optimize a more accurate pose.

[0132] In some embodiments, after constructing the first constraint using the second projection error, the second constraint can also be constructed using IMU data, and then the cost function can be tightly coupled and jointly optimized using a nonlinear optimization method, so that the value of the cost function continuously decreases.

[0133] In some embodiments, since the spatial point to be corrected is not corrected and is not accurate enough, its value can be used as an optimization variable during the optimization process, and optimized together with the pose. This results in a more accurate pose and 3D spatial point.

[0134] For example, please refer to Figure 11 , Figure 11A flowchart illustrating the pose determination method of this application is shown. The electronic device includes a first camera, a second camera, and an inertial measurement unit (IMU). The first camera is a color camera that acquires color images at a first frequency, and the second camera is a depth camera that acquires depth images at a second frequency. After the VIO system initialization is complete and the tracking process begins, color images can be continuously acquired through the color camera. The electronic device can perform calculations on the input color images to determine whether the number of input color images is a specified number, thus determining whether depth image detection is necessary. For example, if the first frequency is three times the second frequency, it can be determined whether the number of input color images is an integer multiple of 3. If so, a depth image is likely to be input, therefore depth image detection must be performed. When the number of input color images is a specified number, it can be checked whether there is a matching depth image. If so, the depth image can be filtered for valid depth values ​​to remove inaccurate depth values. Then, the reference space points recovered from the depth images are used to correct the triangulated space points to be corrected. Visual constraints are constructed based on the corrected target space points and jointly optimized with IMU constraints to obtain a more accurate pose. Meanwhile, during the optimization process, the corrected spatial points are fixed and are no longer optimized as optimization variables along with the pose.

[0135] The pose determination method provided in this application embodiment includes an electronic device that may include a first camera and a second camera. The first camera acquires color images at a first frequency, and the second camera acquires depth images at a second frequency, where the second frequency is less than the first frequency. This allows for the initial acquisition of three-dimensional spatial points corresponding to two-dimensional feature points in multiple color images as points to be corrected. Then, when a depth image is acquired, three-dimensional spatial points corresponding to two-dimensional feature points in a target color image can be acquired as reference spatial points. The target color image is a single color image that matches the depth image among the multiple color images. Based on these reference spatial points, the points to be corrected are then corrected to obtain the corrected target spatial points. The pose information of the electronic device is then determined based on these target spatial points. Since the acquisition of depth images and color images in this application does not need to be at the same frequency, it greatly reduces the power consumption of the system's hardware and software. Furthermore, for spatial information determined solely by the two-dimensional features of the image, the spatial information determined by the low-frequency depth image can be used for correction and adjustment, thereby obtaining more accurate three-dimensional spatial information for precise pose acquisition. This increases the degree of freedom in positioning and improves positioning accuracy, thus achieving precise positioning of low-frequency depth images, which can be applied to low-performance devices.

[0136] Please see Figure 12This diagram illustrates a structural block diagram of a pose determination device 600 provided in an embodiment of this application. The pose determination device 600 utilizes the aforementioned electronic device, which includes a first camera and a second camera. The first camera acquires color images at a first frequency, and the second camera acquires depth images at a second frequency, which is lower than the first frequency. The pose determination device 600 includes: a correction point acquisition module 610, a reference point acquisition module 620, a correction processing module 630, and a pose determination module 640. The correction point acquisition module 610 is used to acquire, based on the acquired multi-frame color images, the three-dimensional spatial points corresponding to the two-dimensional feature points in the multi-frame color images as the spatial points to be corrected; the reference point acquisition module 620 is used to acquire, based on the acquired depth image, the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image as the reference spatial points, wherein the target color image is a color image in the multi-frame color images that matches the depth image; the correction processing module 630 is used to perform correction processing on the spatial points to be corrected based on the reference spatial points to obtain the corrected target spatial points; and the pose determination module 640 is used to determine the pose information of the electronic device based on the target spatial points.

[0137] In some embodiments, the reference point acquisition module 620 includes: a depth determination unit, used to acquire effective depth values ​​corresponding to two-dimensional feature points in the target color image based on the acquired depth image; and a coordinate transformation unit, used to convert the effective depth values ​​into three-dimensional coordinates to obtain three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image as reference spatial points.

[0138] In some embodiments, the depth determination unit described above may be specifically used to: obtain a target depth value corresponding to a target two-dimensional feature point in a target color image based on the acquired depth image, wherein the target two-dimensional feature point is any two-dimensional feature point in the target color image; obtain the average depth value of all neighboring two-dimensional feature points in a specified area around the target two-dimensional feature point based on the depth image; and when the difference between the target depth value and the average depth value is less than a specified threshold, use the target depth value as the effective depth value corresponding to the target two-dimensional feature point in the target color image.

[0139] In some implementations, the correction point acquisition module 610 may be specifically used to: match two-dimensional feature points in adjacent color images based on the acquired multi-frame color images to obtain matching two-dimensional feature points; and perform triangulation processing on the matching two-dimensional feature points to obtain the three-dimensional spatial points corresponding to the matching two-dimensional feature points in the adjacent color images as spatial points to be corrected.

[0140] In some implementations, the correction processing module 630 may be specifically used to: match the two-dimensional feature point corresponding to the reference spatial point with the two-dimensional feature point corresponding to the spatial point to be corrected, to obtain the reference spatial point and the spatial point to be corrected corresponding to the same two-dimensional feature point; determine correction parameters based on the reference spatial point and the spatial point to be corrected corresponding to the same two-dimensional feature point; and perform correction processing on the spatial point to be corrected based on the correction parameters to obtain the corrected target spatial point.

[0141] In some embodiments, the pose determination device 600 may further include: a first construction module, configured to construct a first projection error based on the target spatial point; and a first optimization module, configured to optimize the pose information of the electronic device based on the first projection error.

[0142] In some embodiments, the pose determination device 600 may further include: a pose estimation module, used to determine the pose information of the electronic device based on the spatial point to be corrected when the depth image is not acquired.

[0143] In some embodiments, the pose determination device 600 may further include: a second construction module for constructing a first projection error based on the target spatial point; and a second optimization module for optimizing the pose information of the electronic device based on the first projection error.

[0144] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0145] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0146] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0147] Please refer to Figure 13This document illustrates a structural block diagram of an electronic device according to an embodiment of this application. The electronic device 100 can be a smartphone, tablet, smartwatch, or other electronic device capable of running applications. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, a first camera 130, a second camera 140, and one or more applications. The one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110. The one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0148] Processor 110 may include one or more processing cores. Processor 110 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0149] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0150] Please refer to Figure 14 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0151] The computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A pose determination method, characterized in that, Applied to an electronic device, the electronic device including a first camera and a second camera, the first camera acquiring color images at a first frequency, and the second camera acquiring depth images at a second frequency less than the first frequency, the method includes: Based on the acquired multi-frame color images, the three-dimensional spatial points corresponding to the two-dimensional feature points in the multi-frame color images are obtained as spatial points to be corrected. Based on the acquired depth image, the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image are obtained as reference spatial points, wherein the target color image is a color image that matches the depth image among the multiple color images; The two-dimensional feature points corresponding to the reference spatial point are matched with the two-dimensional feature points corresponding to the spatial point to be corrected, so as to obtain the reference spatial point and the spatial point to be corrected corresponding to the same two-dimensional feature point. Based on the reference spatial point corresponding to the same two-dimensional feature point and the spatial point to be corrected, the correction parameters are determined; Based on the correction parameters, the spatial point to be corrected is corrected to obtain the corrected target spatial point; The pose information of the electronic device is determined based on the target spatial point.

2. The method according to claim 1, characterized in that, The step of obtaining the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image as reference spatial points based on the acquired depth image includes: Based on the acquired depth image, obtain the effective depth value corresponding to the two-dimensional feature points in the target color image; The effective depth value is converted into three-dimensional coordinates, and the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image are obtained as reference spatial points.

3. The method according to claim 2, characterized in that, The step of obtaining the effective depth value corresponding to the two-dimensional feature point in the target color image based on the acquired depth image includes: Based on the acquired depth image, the target depth value corresponding to the target two-dimensional feature point in the target color image is obtained, wherein the target two-dimensional feature point is any two-dimensional feature point in the target color image; Based on the depth image, obtain the average depth value of all neighboring two-dimensional feature points within a specified area surrounding the target two-dimensional feature point; When the difference between the target depth value and the average depth value is less than a specified threshold, the target depth value is taken as the effective depth value corresponding to the target two-dimensional feature point in the target color image.

4. The method according to claim 1, characterized in that, The step of obtaining the three-dimensional spatial points corresponding to the two-dimensional feature points in the acquired multi-frame color images as spatial points to be corrected includes: Based on the acquired multi-frame color images, the two-dimensional feature points in adjacent color images are matched to obtain the matching two-dimensional feature points; The matched two-dimensional feature points are triangulated to obtain the three-dimensional spatial points corresponding to the matched two-dimensional feature points in the adjacent color images as spatial points to be corrected.

5. The method according to any one of claims 1-4, characterized in that, After determining the pose information of the electronic device based on the target spatial point, the method further includes: Based on the target spatial points, a first projection error is constructed; The pose information of the electronic device is optimized based on the first projection error.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: When the depth image is not acquired, the pose information of the electronic device is determined based on the spatial point to be corrected.

7. The method according to claim 6, characterized in that, After determining the pose information of the electronic device based on the spatial point to be corrected, the method further includes: Based on the spatial points to be corrected, a second projection error is constructed; Based on the second projection error, the pose information of the electronic device and the spatial point to be corrected are optimized.

8. A pose determination device, characterized in that, An electronic device comprising a first camera and a second camera, wherein the first camera acquires color images at a first frequency, and the second camera acquires depth images at a second frequency less than the first frequency, the device comprising: The correction point acquisition module is used to acquire the three-dimensional spatial points corresponding to the two-dimensional feature points in the acquired multi-frame color images as spatial points to be corrected. The reference point acquisition module is used to acquire, based on the acquired depth image, the three-dimensional spatial points corresponding to the two-dimensional feature points in the target color image as reference spatial points, wherein the target color image is a color image that matches the depth image among the multiple color images; The correction processing module is used to match the two-dimensional feature points corresponding to the reference spatial point with the two-dimensional feature points corresponding to the spatial point to be corrected, to obtain the reference spatial point and the spatial point to be corrected corresponding to the same two-dimensional feature point; to determine correction parameters based on the reference spatial point and the spatial point to be corrected corresponding to the same two-dimensional feature point; and to perform correction processing on the spatial point to be corrected based on the correction parameters to obtain the corrected target spatial point. The pose determination module is used to determine the pose information of the electronic device based on the target spatial point.

9. An electronic device, characterized in that, include: First camera and second camera; One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-7.

Images