Target positioning method and device, storage medium and terminal

Abstract

A target positioning method, device, storage medium and terminal, the method comprises the following steps: acquiring a plurality of color images obtained by shooting a to-be-positioned feature point at different positions and a depth image corresponding to each color image; performing error screening on the plurality of color images to obtain a plurality of target images; extracting a depth value of the to-be-positioned feature point in a depth image corresponding to each target image, and projecting and converting the depth value based on a selected image in the plurality of target images to obtain a plurality of target depth values; calculating an average depth value of the plurality of target depth values, and determining a coordinate value of the to-be-positioned feature point in the selected image; and positioning the to-be-positioned feature point according to the selected image, the average depth value and the coordinate value. The method can effectively improve the positioning efficiency of positioning the feature point, and further improve the positioning efficiency of positioning the object itself.

Description

Technical Field

[0001] This application relates to the field of robot localization and mapping technology, specifically to a target localization method, device, storage medium, and terminal. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a technology that uses the fusion of data from multiple sensors to map the environment and locate itself. This technology combines image data acquired by an inertial measurement unit (IMU) and a camera to achieve precise object localization.

[0003] In recent years, the localization speed and accuracy of SLAM technology have been significantly improved. Among them, the open-source visual inertial system (OpenVINS) has been widely used due to its high speed, strong compatibility, and powerful modular functions.

[0004] However, in some scenarios, object localization in OpenVINS suffers from insufficient accuracy. Summary of the Invention

[0005] This application provides a target positioning method, apparatus, storage medium, and terminal, which can improve positioning accuracy.

[0006] Firstly, this application provides a target localization method, the method comprising:

[0007] Acquire multiple color images of the feature points to be located taken at different locations, as well as a depth image corresponding to each color image;

[0008] Error filtering is performed on the multiple color images to obtain multiple target images;

[0009] Extract the depth value of the feature point to be located in the depth image corresponding to each target image, and perform projection transformation on the depth value based on the selected image among the multiple target images to obtain multiple target depth values;

[0010] Calculate the average depth value of the plurality of target depth values, and determine the coordinate value of the feature point to be located in the selected image;

[0011] The feature point to be located is located based on the selected image, the average depth value, and the coordinate value.

[0012] Accordingly, a second aspect of this application provides a target positioning device, the device comprising:

[0013] The acquisition module is used to acquire multiple color images of the feature points to be located taken at different locations, as well as the depth image corresponding to each color image;

[0014] The filtering module is used to perform error filtering on the multiple color images to obtain multiple target images;

[0015] An extraction module is used to extract the depth value of the feature point to be located in the depth image corresponding to each target image, and to perform projection transformation on the depth value based on the selected image among the multiple target images to obtain multiple target depth values;

[0016] The calculation module is used to calculate the average depth value of the multiple target depth values, and to determine the coordinate value of the feature point to be located in the selected image;

[0017] The first positioning module is used to locate the feature point to be located based on the selected image, the average depth value, and the coordinate value.

[0018] Thirdly, this application also provides a target localization method, the method comprising:

[0019] Sample multiple feature points to be located in the object to be located;

[0020] The feature points to be located that have more than a preset threshold of observed image frames are identified as target feature points to be located.

[0021] The target feature points to be located are located according to the target localization method provided in the first aspect, and the localization information of each target feature point to be located is obtained.

[0022] The object to be located is located based on the location information of each target feature point.

[0023] Accordingly, a fourth aspect of this application provides a target positioning device, the device comprising:

[0024] The sampling module is used to sample multiple feature points to be located in the object to be located;

[0025] The determination module is used to determine the feature points to be located that have a number of observed image frames greater than a preset threshold as target feature points to be located.

[0026] The second positioning module is used to locate the target feature points to be located according to the target positioning method provided in the first aspect, and to obtain the positioning information of each target feature point to be located.

[0027] The third positioning module is used to locate the object to be located based on the positioning information of each target feature point to be located.

[0028] Fifthly, this application provides a storage medium having a computer program stored thereon, which, when loaded by a processor of an electronic device, executes the steps of any of the target positioning methods provided in this application.

[0029] Sixthly, this application also provides an electronic device including a processor and a memory, the memory storing a computer program, the processor executing steps in any of the target positioning methods provided in this application by loading the computer program stored in the memory.

[0030] The technical solution provided in this application involves acquiring multiple color images of the feature point to be located at different locations, along with a corresponding depth image for each color image; performing error filtering on the multiple color images to obtain multiple target images; extracting the depth value of the feature point to be located in the depth image corresponding to each target image; and projecting the depth value onto a selected image from the multiple target images to obtain multiple target depth values; calculating the average depth value of the multiple target depth values ​​and determining the coordinate value of the feature point to be located in the selected image; and locating the feature point to be located based on the selected image, the average depth value, and the coordinate value. In this way, by acquiring the depth image of the feature point to be located, this application can directly read the depth value of the feature point to be located in the observation frame. Furthermore, by converting the pose transformation to the depth in the selected image, compared to the method of solving a system of linear equations using multiple frames, more accurate depth information can be obtained, thereby improving the accuracy of target localization. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating the target localization method provided in an embodiment of this application.

[0033] Figure 2 This is a schematic diagram of multiple observation cameras observing feature points.

[0034] Figure 3 This is a comparison chart of the positioning effects.

[0035] Figure 4 The structural block diagram of the target positioning device provided in the embodiments of this application.

[0036] Figure 5 This is a structural block diagram of the terminal provided in the embodiments of this application. Detailed Implementation

[0037] It should be noted that the terms "first," "second," and "third," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules not listed, or some embodiments also include other steps or modules inherent to these processes, methods, products, or devices.

[0038] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0039] This application provides a target positioning method, apparatus, storage medium, and terminal. The target positioning method can be executed by the target positioning device provided in this application, or by an electronic device integrating the target positioning device. The target positioning device can be implemented in hardware or software. The electronic device can be a mobile terminal, such as a smartphone, tablet, vehicle-mounted terminal, or smart wearable device.

[0040] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the target localization method provided in the embodiments of this application, such as... Figure 1 As shown, the flow of the target localization method provided in this application embodiment can be as follows:

[0041] In step 110, multiple color images of the feature points to be located and the corresponding depth images are obtained.

[0042] In this embodiment, the target positioning method can be applied to a target positioning device, which can be specifically installed in a terminal. The terminal can be a smartphone, tablet, vehicle terminal, or other smart wearable device.

[0043] In related technologies, a crucial part of the localization task is obtaining the 3D depth of observed feature points. Typically, a feature point is observed by multiple cameras from different viewpoints. By analyzing the relative positions of the cameras, the 3D depth of the feature point can be calculated through triangulation. In SLAM (Simultaneous Localization and Mapping), the 3D spatial information of multiple feature points can be used to localize an object. Therefore, the accuracy of the 3D spatial information of feature points is crucial to the accuracy of object localization in SLAM.

[0044] Among current SLAM localization schemes, OpenVINS is widely used in the industry due to its fast localization speed and strong adaptability. In OpenVINS, triangulation employs a 3D Cartesian coordinate system triangulation method, triangulating feature points within the anchor frame. This process is described in detail below.

[0045] like Figure 2 The figure shows a schematic diagram of multiple observation cameras observing a feature point. As shown, the feature point is observed by m+1 cameras. The image frame observed by camera A can be selected as the anchor frame, or the first observation frame corresponding to the feature point (sorted according to the time order of the observation of the feature point) can be selected as the anchor frame.

[0046] Step 1: Calculate the initial value of the feature point depth.

[0047] Feature point p f In addition to being observed by camera A, it can also be observed by other observation cameras, whose poses are denoted as C. i Where i = 1, 2, ..., m. Feature point p f The coordinates in the anchor frame are A p f At the observation camera C i The coordinates in are Ci p f Camera frame C i The rotation to the anchor frame is Camera frame C i Translation to the anchor frame is A p Ci Camera frames and anchor frames for feature point p f The observations are related as follows:

[0048]

[0049]

[0050] In the absence of noise, the normalized image coordinates in the camera frame are: Ci b, depth is Ciz, the normalized coordinates of the feature points in the anchor frame. A b f ,depth A z f The mapping relationship between feature points in an image frame and normalized image coordinates in the anchor frame is as follows:

[0051]

[0052] in, Ci z f For feature points and camera frame C i The corresponding depth, Ci b f For feature points in camera frame C i Normalized coordinates in the matrix. A b Ci→f These are the coordinates of the feature points mapped from the camera frame to the anchor frame.

[0053] To eliminate irrelevant degrees of freedom, define with A b Ci→f orthogonal vectors All three rows of this vector are... A b Ci→f Vertical, that is Multiply the transformation relation of formula (3) by the transformation matrix on the left. A N i Construct two only depths A z f Relevant equations:

[0054]

[0055] By combining the constraints from multiple camera observations, the following depth constraint equation is constructed:

[0056]

[0057] The following transformation is performed on equation (3):

[0058]

[0059] A 1m*1m system can be obtained, which can be solved quickly using scalar division. A z f ,Right now:

[0060]

[0061] OpenVINS also adds a validity check for triangulated features, meaning that feature points must be directly in front of the camera and not too far away from the camera.

[0062] The second step is inverse depth nonlinear optimization of feature points.

[0063] After obtaining the initial depth estimate of the feature points, nonlinear least squares is used to further optimize the depth estimate. Using the inverse depth representation of the feature points yields better numerical stability and aids convergence. In most cases, indoor scenes converge after 2 to 3 iterations. The least squares problem is constructed as follows: Define the normalized coordinates u of the anchor frame image. A = A X f / A z f v A = A y f / A z f Inverse depth ρ A =1 / A z f The feature mapping is rewritten in the following form:

[0064]

[0065] Then, the measurement equation is constructed:

[0066]

[0067] Since feature points can be observed from multiple camera poses, the observation equation is as follows:

[0068]

[0069] Therefore, the least squares equation and the Jacobian can be constructed:

[0070]

[0071]

[0072]

[0073] Then, the least squares problem is solved iteratively using the Gauss-Newton method or the Levenberg-Marquardt method to finally obtain the optimization depth.

[0074] As described above, feature point localization in OpenVINS requires complex calculations. Each feature point needs at least five observation frames for computation. In rapidly changing scenes such as large rotations, the same feature point may have fewer than five observation frames, rendering these feature points unusable during triangulation and resulting in poor localization performance in these scenarios. In such cases, the extracted feature points do not contribute to pose estimation, reducing the system's localization accuracy and robustness. Furthermore, the initial depth of each feature point requires solving complex linear equations. Optimizing the 3D feature point depth using the Gauss-Newton method requires first constructing a least-squares problem, then the Jacobian matrix, followed by incremental iterations until convergence or the maximum number of iterations is reached. This process is slow and the results are easily affected by noise; a large error in one observation can negatively impact the final optimization result. In short, feature point localization in OpenVINS is both inaccurate and inefficient, leading to its poor performance in SLAM.

[0075] To address the aforementioned issues of low accuracy and efficiency in feature point localization, this application provides a target localization method aimed at improving the accuracy and efficiency of feature point localization, thereby further enhancing the accuracy and efficiency of object localization in SLAM. The solution provided in this application is described in detail below.

[0076] In this embodiment, it is also necessary to acquire observation frames from multiple camera perspectives for the same feature point. However, at each observation perspective, both the color image and the depth image corresponding to the feature point are acquired simultaneously. The color image is an RGB image, and the camera acquiring the RGB image can be a monocular camera or a binocular camera. The camera acquiring the depth image can specifically be a time-of-flight (TOF) depth camera or a structured light depth camera.

[0077] When locating a feature point, the target localization device can acquire multiple RGB images and multiple depth images of the feature point from different locations. Each RGB image corresponds one-to-one with a depth image, meaning the RGB image and its corresponding depth image were captured at the same location and at the same time. To ensure accuracy, at least three images can be used.

[0078] In step 120, the depth values ​​of the feature points to be located are extracted from the depth images corresponding to the color images, and the depth values ​​are projected and transformed based on the selected images from multiple color images to obtain multiple target depth values.

[0079] In this embodiment of the application, after obtaining the color image and the corresponding depth image of the feature point to be located, the depth value of the feature point to be located can be calculated based on the color image and the depth image. Specifically, the depth value of the feature point to be located in each depth image can be extracted first, and then the depth value can be projected onto the selected observation frame (such as the aforementioned anchor frame) to obtain the projected depth of each depth value in the anchor frame.

[0080] To further ensure the accuracy of target localization and avoid interference from abnormal images, this embodiment of the application, after the target localization device acquires multiple RGB images of the feature point to be located, can further filter these RGB images to remove some interfering images. Specifically, this can involve removing reprojection errors to obtain the remaining multiple target images. Then, accurate target localization is performed based on these target images. That is, reprojection error filtering can be performed on multiple observation frames of the same feature point, retaining the depth values ​​of observations with smaller reprojection errors.

[0081] In some embodiments, error filtering is performed on multiple color images to obtain multiple target images, including:

[0082] 1. Select the desired image from multiple color images;

[0083] 2. Based on the selected image, perform pose transformation on the images other than the selected image from multiple color images to obtain multiple transformed images;

[0084] 3. Identify multiple target conversion images from among multiple conversion images whose error with the selected image is less than a preset threshold;

[0085] 4. Determine multiple target images based on the color images corresponding to the multiple target transformed images and the selected image.

[0086] In this embodiment, multiple RGB images are filtered. Specifically, a selected image can be determined from multiple RGB image frames, where the selected image can be the aforementioned anchor frame. The anchor frame can also be understood as a reference frame. Specifically, the first observation frame of the feature point to be located can be selected as the anchor frame. Then, pose transformation can be performed on the other RGB images based on the anchor frame to obtain the transformed image corresponding to each image. Then, images whose pose transformation error with the anchor frame is less than a preset threshold are further determined to be qualified images, while images whose transformation error with the anchor frame is greater than the preset threshold are unqualified images. Further, the qualified images and the anchor frame are determined to form the target image.

[0087] In some embodiments, determining multiple target transformed images from a plurality of transformed images whose error with the selected image is less than a preset threshold includes:

[0088] 3.1 Calculate the reprojection error between each transformed image and the selected image;

[0089] 3.2. Determine the transformed image whose reprojection error is less than the preset threshold as the target transformed image.

[0090] In this embodiment, the specific method for determining the error between the transformed image and the anchor frame after pose transformation can be to calculate the reprojection error between the transformed image and the anchor frame. The reprojection error refers to the difference between the projection of a real 3D point onto the image plane (i.e., a pixel in the image) and the reprojection (which is actually a virtual pixel obtained using our calculated value). Due to various reasons, the calculated value will not perfectly match the actual situation; that is, this difference cannot be exactly 0. Therefore, it is necessary to minimize the sum of these differences to obtain the optimal camera pose parameters and the coordinates of the 3D point.

[0091] After calculating the reprojection error between the transformed image and the selected image, transformed images with a reprojection error less than a preset threshold are identified as target transformed images. The RGB images corresponding to these target transformed images and the anchor frames constitute the target images. Then, the depth values ​​of the feature points to be located in the depth images corresponding to these target images can be extracted to calculate the depth of the feature points. That is, based on the pose transformation relationship between the observation frame and the anchor frame, the feature points observed in the observation frame are transformed to the anchor frame coordinate system to obtain the projected pixel coordinate positions. The error value (i.e., reprojection error) is calculated by comparing the projected position with the observed position of the feature point in the anchor frame. Since pose information cannot be 100% accurate, and there are errors in feature point extraction, the projected position and the position of the feature point in the anchor frame cannot completely coincide, thus resulting in a reprojection error.

[0092] In some embodiments, calculating the reprojection error between each transformed image and the selected image includes:

[0093] 3.1.1 Obtain the first coordinate value of the feature point to be located in the color image corresponding to each transformed image, and obtain the first depth value of the feature point to be located in the depth image corresponding to each transformed image;

[0094] 3.1.2 Obtain the rotation and translation parameters corresponding to the pose transformation of each transformed image;

[0095] 3.1.3 Calculate the representation parameters of each transformed image based on the first coordinate value, the first depth value, the rotation parameter, and the translation parameter;

[0096] 3.1.4 Calculate the difference between the representation parameters of each transformed image and the target representation parameters of the selected image to obtain the reprojection error between each transformed image and the selected image.

[0097] In this embodiment, the calculation of the reprojection error between the transformed image and the anchor frame can specifically involve first obtaining the first coordinate value of the feature point to be located in the color image corresponding to each transformed image and the first depth value of the feature point to be located in the depth image corresponding to each transformed image. The depth value in the depth image corresponding to the transformed image is the depth value obtained by projecting the depth value of the feature point to be located in the depth image matching the RGB image corresponding to the transformed image onto the anchor frame.

[0098] Then, the rotation and translation parameters corresponding to the pose transformation of each transformed image are obtained. Then, based on the first coordinate value, the first depth value, the rotation and translation parameters, the representation parameters of each transformed image are calculated. Then, the difference between the representation parameters of each transformed image and the target representation parameters corresponding to the anchor frame is calculated to obtain the reprojection error between each transformed image and the selected image.

[0099] Specifically, determining the reprojection error between each observation and the anchor frame observation includes: first, extracting feature points in the observed image; matching feature points across different frames to determine the correspondence between them; obtaining rotation and translation parameters between multiple frames; and based on the correspondence between feature points and the rotation and translation relationships between frames, projecting the observation points in the observed image onto the anchor frame to obtain the projected coordinates. Finally, calculating the difference between the coordinates of the observation points projected onto the anchor frame and the coordinates of the observation points in the anchor frame, thus obtaining the reprojection error from the observation points in the observed frame to the observation points in the anchor frame.

[0100] In some embodiments, the difference between the representation parameters of each transformed image and the target representation parameters of the selected image is calculated to obtain the reprojection error between each transformed image and the selected image, including:

[0101] 3.1.4.1. Obtain the second coordinate value of the feature point to be located in the selected image, and obtain the second depth value of the feature point to be located in the depth image corresponding to the selected image;

[0102] 3.1.4.2 Calculate the product of the second coordinate value and the second depth value to obtain the target representation parameters corresponding to the selected image;

[0103] 3.1.4.3 Calculate the difference between the representation parameters of each transformed image and the target representation parameters to obtain the reprojection error between each transformed image and the selected image.

[0104] Among them, in the embodiments of the present application, the target characterization parameter of the anchor frame can also be calculated from the coordinate value of the feature point to be located in the anchor frame and the depth value in the depth image corresponding to the anchor frame. Specifically, the target positioning device can first obtain the second coordinate value of the feature point to be located in the anchor frame and the second depth value of the feature point to be located in the depth image corresponding to the anchor frame, and then calculate the product of the second coordinate value and the second depth value to obtain the target characterization parameter corresponding to the anchor frame. Then, the difference between the characterization parameter of each transformed image and the target characterization parameter can be calculated based on this, and the reprojection error between each transformed image and the selected image can be obtained.

[0105] Specifically, assume that there are m frames of RGB images of the feature point to be located, where m≥3. Select the first observation frame where the feature point is observed as the anchor frame, and then read the depth value d of the feature point in the anchor frame. A The depth value d of the feature point is also read in other observation frames. i And through the conversion relationship between poses, the feature point is transformed to the anchor frame, and the reprojection error is calculated. If the reprojection error e is greater than a certain threshold, the observation in this frame image is discarded. If the reprojection error e is within the threshold range, the projected depth A d i is recorded, a total of n frames (n < m). The calculation formula of the reprojection error is as follows:

[0106]

[0107] e = A d i A b f -d A A b f Formula (15)

[0108] Among them, after determining multiple target images (that is, the RGB images corresponding to the anchor frame and the transformed images with a reprojection error less than the preset threshold between the anchor frame), the depth value of the feature point to be located in the depth image corresponding to each target image can be further extracted. Then, the depth value corresponding to the target image other than the anchor frame is projected and transformed based on the anchor frame to obtain the projected depth, that is, the aforementioned projected depth A d i . It can be understood that the projected depth A d i can also be obtained by extracting the depth value of the feature point to be located from the depth image corresponding to each transformed image. Multiple (n - 1) projected depths A d i and the depth value d of the feature point to be located in the anchor frame A constitute multiple target depth values.

[0109] In step 130, the average depth value of multiple target depth values ​​is calculated, and the coordinates of the feature point to be located in the selected image are determined.

[0110] After determining multiple target depth values, the average depth value d of these multiple target depth values ​​can be further calculated. avg This is used as the depth value of the feature point to be located. As follows:

[0111]

[0112] Furthermore, the coordinates of the feature point to be located in the anchor frame can also be obtained.

[0113] In step 140, the feature points to be located are located based on the selected image, average depth value, and coordinate values.

[0114] Once the anchor frame, the depth value of the feature point to be located relative to the anchor frame (i.e., the aforementioned average depth value), and the coordinates of the feature point within the anchor frame are determined, the feature point can be accurately located based on the anchor frame, the average depth value, and the coordinates of the feature point within the anchor frame, thus determining the accurate 3D spatial location information of the feature point. As follows:

[0115] A p f =d avg A b f Formula (17)

[0116] In some embodiments, the target localization method provided in this application further includes:

[0117] 1. Obtain the target pose of the selected image in the world coordinate system;

[0118] 2. Based on the target pose, average depth value, and coordinate value, perform coordinate transformation to obtain the positioning information of the feature point to be located in the world coordinate system.

[0119] In this embodiment, after determining the 3D position of the feature point to be located relative to the anchor frame, the position can be further transformed to the world coordinate system to obtain the positioning information of the feature point in the world coordinate system. Specifically, the pose of the anchor frame in the world coordinate system can be obtained first, and then coordinate transformation can be performed based on the target pose, the depth of the feature point to be located relative to the anchor frame, and the coordinates of the feature point to be located in the anchor frame to obtain the positioning information of the feature point to be located in the world coordinate system. As follows:

[0120]

[0121] in, Gp represents the 3D information of the feature point to be located in the world coordinate system. For the rotation parameters of the anchor frame, G p A The translation parameters for the anchor frame.

[0122] The target localization method provided in this application allows for the direct reading of a feature point's depth from the depth map of fewer than five observation frames. This means that relatively accurate depth information can be obtained from a limited number of observations, improving feature point utilization, enhancing robustness in scenarios involving large rotations, and increasing the system's localization accuracy. Furthermore, when the depth map at the feature point is valid, the depth value of the 3D feature point can be directly read from the observation frame using depth images acquired by a depth sensor. Converting the pose transformation to depth at the anchor frame is faster than solving a system of linear equations using multiple frames. After verifying reprojection errors, frames with large depth errors can be removed, ensuring the accuracy of the final 3D feature point depth calculation. In addition, statistical averaging of depth information from multiple frames yields the final 3D feature point depth, and measurement noise can be filtered out, ensuring the accuracy of the observation point depth values. This approach guarantees both speed and accuracy, eliminating the need for complex least-squares problem construction and solution processes.

[0123] OpenVINS is sensitive to IMU noise; under high IMU noise, pose drift is likely. During large rotations, the number of observation frames for feature points is limited, making it difficult to guarantee a sufficient number of 3D feature points for initialization using triangulation methods. Adding depth information provides additional constraints to 3D feature points, enabling the initialization of a sufficient number of feature points even with limited observation frames during large rotations. This reduces drift caused by insufficient feature points and increases localization accuracy and stability. By introducing a depth map, the method avoids solving for depth through linear equations and constructing a least-squares problem, optimizing 3D feature point initialization and improving speed. Furthermore, the introduction of additional reprojection error constraints and statistical averaging optimization methods improves the accuracy of depth calculation.

[0124] In some embodiments, after performing coordinate transformation based on the target pose, average depth value, and coordinate values ​​to obtain the positioning information of the feature point to be located in the world coordinate system, the method further includes:

[0125] Spatial positioning of the object to be located is performed based on the positioning information of multiple feature points in the world coordinate system, thereby obtaining the position information of the object to be located.

[0126] Since SLAM's task is to locate and map an object itself, accurate 3D information of feature points is required for this process. The target localization method provided in this application can quickly and accurately locate feature points, meaning it can rapidly obtain accurate 3D information of feature points. Furthermore, by traversing every feature point in the localization space, the accurate 3D information of each feature point can be determined, allowing for accurate localization and mapping of objects in the localization space based on this accurate 3D information. For example, when locating an autonomous vehicle, it is only necessary to obtain the 3D information of multiple feature points associated with it, and then accurately locate the autonomous vehicle based on the relative positional relationship between the autonomous vehicle and these feature points and the 3D information of the feature points.

[0127] The target localization method provided in this application can also greatly improve the accuracy of object localization in SLAM tasks.

[0128] In addition, as shown in Table 1 and Figure 3 The table and diagram shown are a comparison between the method for collecting data from a robot dog based on the target localization method provided in this application and the traditional OpenVINS.

[0129] algorithm Average time elapsed (ms) The trajectory showed a significant drift ratio OpenVINS 13.031 4 / 5 This application method 12.188 1 / 5

[0130] Table 1 Comparison of Positioning Effects

[0131] like Figure 3 As shown in the figure, when the robot dog walks around the conference table and returns to its original position, the trajectory drift caused by the method provided in this application for locating the robot dog's position is significantly reduced. Moreover, as shown in Table 1, the positioning time is also significantly reduced using the method provided in this application, and the positioning speed is effectively improved.

[0132] As described above, the target localization method provided in this application acquires multiple color images of the feature point to be located and corresponding depth images; extracts the depth value of the feature point in the depth image corresponding to each color image, and projects and transforms the depth value based on a selected image among the multiple color images to obtain multiple target depth values; calculates the average depth value of the multiple target depth values, and determines the coordinate value of the feature point to be located in the selected image; and locates the feature point to be located based on the selected image, the average depth value, and the coordinate value. Therefore, this application, based on the depth value of the depth image and the pose constraints and feature point mapping relationship between the color images, can obtain the depth value of the feature point to be determined in the observation frame more quickly and accurately. Compared with the method of solving a system of linear equations using multiple frames, more accurate depth information can be obtained, thereby improving the accuracy of target localization.

[0133] Please refer to Figure 4 , Figure 4 This is a schematic diagram of a target positioning device 200 provided in an embodiment of this application. The target positioning device 200 is applied to the electronic device provided in this application. Figure 4 As shown, the target positioning device 200 may include:

[0134] The acquisition module 210 is used to acquire multiple color images of the feature points to be located and the depth images corresponding to the color images;

[0135] The extraction module 220 is used to extract the depth value of the feature point to be located in the depth image corresponding to each color image, and to perform projection transformation on the depth value based on the selected image in multiple target images to obtain multiple target depth values.

[0136] The calculation module 230 is used to calculate the average depth value of multiple target depth values ​​and to determine the coordinate values ​​of the feature points to be located in the selected image;

[0137] The first positioning module 240 is used to locate the feature point to be located based on the selected image, the average depth value, and the coordinate value.

[0138] Optionally, in some embodiments, the extraction module includes:

[0139] The filtering submodule is used to filter the multiple color images to obtain multiple target images;

[0140] The extraction submodule is used to extract the depth values ​​of the feature points to be located in the depth images corresponding to the multiple target images, and to perform projection transformation on the depth values ​​based on the selected image in the multiple target images to obtain multiple target depth values.

[0141] Optionally, in some embodiments, the filtering submodule includes:

[0142] The first determining unit is used to determine the selected image among multiple color images;

[0143] The transformation unit is used to perform pose transformation on images other than the selected image from multiple color images based on the selected image, so as to obtain multiple transformed images;

[0144] The second determining unit is used to determine multiple target transformed images from multiple transformed images whose error with the selected image is less than a preset threshold.

[0145] The third determining unit is used to determine multiple target images based on the color images corresponding to the multiple target transformed images and the selected image.

[0146] Optionally, in some embodiments, the second determining subunit includes:

[0147] The calculation subunit is used to calculate the reprojection error between each transformed image and the selected image;

[0148] A sub-unit is defined to identify the transformed image whose reprojection error is less than a preset threshold as the target transformed image.

[0149] Optionally, in some embodiments, the computation subunit is further used for:

[0150] Obtain the first coordinate value of the feature point to be located in the color image corresponding to each transformed image, and obtain the first depth value of the feature point to be located in the depth image corresponding to each transformed image;

[0151] Obtain the rotation and translation parameters corresponding to the pose transformation of each transformed image;

[0152] The representation parameters of each transformed image are calculated based on the first coordinate value, the first depth value, the rotation parameter, and the translation parameter;

[0153] The difference between the representation parameters of each transformed image and the target representation parameters of the selected image is calculated to obtain the reprojection error between each transformed image and the selected image.

[0154] Optionally, in some embodiments, the computation subunit is further used for:

[0155] Obtain the second coordinate value of the feature point to be located in the selected image, and obtain the second depth value of the feature point to be located in the depth image corresponding to the selected image;

[0156] Calculate the product of the second coordinate value and the second depth value to obtain the target representation parameters corresponding to the selected image;

[0157] Calculate the difference between the representation parameters of each transformed image and the target representation parameters to obtain the reprojection error between each transformed image and the selected image.

[0158] Optionally, in some embodiments, the target positioning device provided in this application further includes:

[0159] The acquisition submodule is used to acquire the target pose of the selected image in the world coordinate system;

[0160] The transformation submodule is used to perform coordinate transformation based on the target pose, average depth value, and coordinate value to obtain the positioning information of the feature point to be located in the world coordinate system.

[0161] In some embodiments, the target positioning device provided in this application further includes:

[0162] The positioning submodule is used to spatially locate the object to be located based on the positioning information of multiple feature points to be located in the world coordinate system, and obtain the position information of the object to be located.

[0163] It should be noted that the target positioning device 200 provided in this application embodiment is different from that in the above embodiment. Figure 1 The target localization methods shown belong to the same concept, and their specific implementation process can be found in the relevant embodiments above, which will not be repeated here.

[0164] As described above, the target localization device provided in this application acquires multiple color images of the feature point to be located and corresponding depth images through the acquisition module 210; the extraction module 220 extracts the depth value of the feature point to be located in the depth image corresponding to each color image, and performs projection transformation on the depth value based on the selected image among the multiple color images to obtain multiple target depth values; the calculation module 230 calculates the average depth value of the multiple target depth values ​​and determines the coordinate value of the feature point to be located in the selected image; and the localization module 240 locates the feature point to be located based on the selected image, the average depth value, and the coordinate value. Therefore, based on the depth value of the depth image and the pose constraints and feature point mapping relationship between the color images, this application can obtain the depth value of the feature point to be located in the observation frame more quickly and accurately. Compared with the method of solving a system of linear equations using multiple frames, more accurate depth information can be obtained, thereby improving the accuracy of target localization.

[0165] This application embodiment also provides a storage medium storing a computer program thereon. When the computer program stored thereon is executed on the processor of the electronic device provided in this application embodiment, the processor of the electronic device performs any of the steps in the target positioning method suitable for the electronic device described above. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0166] This application also provides a terminal, please refer to... Figure 5 The terminal 300 includes a processor 310 and a memory 320.

[0167] The processor 310 in this embodiment can be a general-purpose processor, such as an ARM architecture processor. The memory 320 stores a computer program; it can be a high-speed random access memory or a non-volatile memory, such as at least one disk storage device, flash memory, or other volatile solid-state storage device. Accordingly, the memory 320 may also include a memory controller to provide the processor 301 with access to the memory 320. The processor 310 executes the computer program in the memory 320 to perform any of the above target location methods, such as:

[0168] Acquire multiple color images of the feature point to be located and the corresponding depth images of the color images; extract the depth value of the feature point to be located in the depth image corresponding to each color image, and perform projection transformation on the depth value based on the selected image among the multiple color images to obtain multiple target depth values; calculate the average depth value of the multiple target depth values, and determine the coordinate value of the feature point to be located in the selected image; locate the feature point to be located according to the selected image, the average depth value, and the coordinate value.

[0169] The above provides a detailed description of a target positioning method, apparatus, storage medium, and terminal provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A target localization method, characterized in that, The method includes: Acquire multiple color images of the feature points to be located, as well as the corresponding depth images; Extracting the depth value of the feature point to be located in the depth image corresponding to the color image, and performing a projection transformation on the depth value based on the selected image among the multiple color images to obtain multiple target depth values, includes: performing error filtering on the multiple color images to obtain multiple target images; extracting the depth value of the feature point to be located in the depth image corresponding to the multiple target images, and performing a projection transformation on the depth value based on the selected image among the multiple target images to obtain multiple target depth values; Calculate the average depth value of the plurality of target depth values, and determine the coordinate value of the feature point to be located in the selected image; The feature point to be located is located based on the selected image, the average depth value, and the coordinate value.

2. The method according to claim 1, characterized in that, The step of performing error filtering on the multiple color images to obtain multiple target images includes: Select the chosen image from the plurality of color images; Based on the selected image, pose transformation is performed on the images other than the selected image among the multiple color images to obtain multiple transformed images; Among the multiple transformed images, identify multiple target transformed images whose error with the selected image is less than a preset threshold; Multiple target images are determined based on the color images corresponding to the multiple target transformed images and the selected image.

3. The method according to claim 2, characterized in that, The step of determining multiple target converted images from the multiple converted images whose error with the selected image is less than a preset threshold includes: Calculate the reprojection error between each transformed image and the selected image; The transformed image whose reprojection error is less than a preset threshold is determined as the target transformed image.

4. The method according to claim 3, characterized in that, The calculation of the reprojection error between each transformed image and the selected image includes: Obtain the first coordinate value of the feature point to be located in the color image corresponding to each transformed image, and obtain the first depth value of the feature point to be located in the depth image corresponding to each transformed image; Obtain the rotation and translation parameters corresponding to the pose transformation of each transformed image; The characterization parameters of each transformed image are calculated based on the first coordinate value, the first depth value, the rotation parameter, and the translation parameter. The difference between the representation parameters of each transformed image and the target representation parameters of the selected image is calculated to obtain the reprojection error between each transformed image and the selected image.

5. The method according to claim 4, characterized in that, The calculation of the difference between the representation parameters of each transformed image and the target representation parameters of the selected image to obtain the reprojection error between each transformed image and the selected image includes: Obtain the second coordinate value of the feature point to be located in the selected image, and obtain the second depth value of the feature point to be located in the depth image corresponding to the selected image; Calculate the product of the second coordinate value and the second depth value to obtain the target representation parameters corresponding to the selected image; The difference between the representation parameters of each transformed image and the target representation parameters is calculated to obtain the reprojection error between each transformed image and the selected image.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the target pose of the selected image in the world coordinate system; Based on the target pose, the average depth value, and the coordinate value, coordinate transformation is performed to obtain the positioning information of the feature point to be located in the world coordinate system.

7. The method according to claim 6, characterized in that, After performing coordinate transformation based on the target pose, the average depth value, and the coordinate values ​​to obtain the positioning information of the feature point to be located in the world coordinate system, the method further includes: Based on the positioning information of multiple feature points to be located in the world coordinate system, the object to be located is spatially located to obtain the position information of the object to be located.

8. A target positioning device, characterized in that, The device includes: The acquisition module is used to acquire multiple color images of the feature points to be located, as well as the depth images corresponding to the color images; An extraction module is used to extract the depth value of the feature point to be located in the depth image corresponding to the color image, and to perform projection transformation on the depth value based on a selected image of the multiple color images to obtain multiple target depth values. This includes: performing error filtering on the multiple color images to obtain multiple target images; extracting the depth value of the feature point to be located in the depth image corresponding to the multiple target images, and performing projection transformation on the depth value based on a selected image of the multiple target images to obtain multiple target depth values. The calculation module is used to calculate the average depth value of the multiple target depth values, and to determine the coordinate value of the feature point to be located in the selected image; The first positioning module is used to locate the feature point to be located based on the selected image, the average depth value, and the coordinate value.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor of the electronic device, it performs the steps in the target localization method as described in any one of claims 1-7.

10. A terminal, the terminal comprising a processor and a memory, the memory storing a computer program, characterized in that, The processor executes the steps of the target localization method as described in any one of claims 1-7 by loading the computer program.

Images