Computer vision positioning method and system based on depth information and deep learning

Abstract

The application discloses a computer vision positioning method and system based on depth information and deep learning, innovatively arranges AprilTag marks on each face of a polyhedron, combines accurate depth information of a depth camera and a deep learning model, and realizes high-precision, high-robustness and low-computing-complexity pose positioning, and is particularly suitable for medical operation navigation, industrial robot positioning, unmanned aerial vehicle positioning and augmented reality application scenes, and can still maintain stable positioning performance in a shielding environment.

Description

Technical Field

[0001] This invention relates to the field of visual positioning technology, and specifically to a computer vision positioning method and system based on depth information and deep learning. Background Technology

[0002] Visual positioning technology has been widely applied in fields such as robot navigation, augmented reality, and autonomous driving. Traditional visual positioning methods are mainly based on monocular cameras and feature matching techniques, such as SIFT and SURF (feature matching techniques for handicrafts), or deep learning-based methods. However, these methods face many challenges in practical applications, as follows:

[0003] 1) Poor environmental adaptability. Traditional methods are extremely sensitive to changes in lighting, occlusion, and differences in viewing angle, and are prone to feature matching failures or decreased positioning accuracy in complex environments. For example, the robot vision positioning method described in patent CN201910620329.0 uses different calibration strategies for long and short distances, but it still struggles to solve positioning problems in variable environments.

[0004] 2) Reliance on external positioning systems. Many systems (such as the UAV positioning scheme described in patent CN202210418712.X) rely on external positioning systems such as GNSS, but these systems cannot work properly indoors or in obstructed environments, resulting in positioning failure.

[0005] 3) High computational cost. Scene coordinate regression (SCR) methods based on deep learning require the prediction of a large number of pixel-level coordinates, which limits real-time performance; while methods such as NeRF have high overhead due to pixel-level training and rendering, making it difficult to meet the needs of real-time applications.

[0006] 4) High calibration complexity. Existing technologies typically rely on multi-step hardware adjustments for camera calibration, which is prone to introducing errors. For example, the line detection calibration method mentioned in patent CN201710560044.3 requires complex geometric analysis and manual adjustments.

[0007] 5) Insufficient adaptation to medical scenarios. The medical field (such as surgical navigation) has extremely high requirements for the accuracy and real-time performance of visual positioning, which current technologies struggle to meet. It is difficult to achieve sub-millimeter-level positioning accuracy for surgical instruments. Summary of the Invention

[0008] This invention provides a computer vision localization method and system based on depth information and deep learning to solve the above-mentioned technical problems existing in existing vision localization methods.

[0009] According to a first aspect, one embodiment provides a computer vision localization method based on depth information and deep learning, the method comprising:

[0010] Start the depth camera, use the depth camera to acquire RGB images containing a polyhedral calibration tool and obtain the corresponding depth information and point cloud data. The polyhedral calibration tool is designed with AprilTag markers on multiple faces to ensure that at least one marker can be detected from any viewpoint.

[0011] Data acquired from a depth camera is input into a pre-trained deep learning model to predict the pose estimate of the AprilTag detection surface.

[0012] By combining the obtained depth information, the pose estimation of the AprilTag detection surface predicted by the deep learning model is optimized.

[0013] Based on the optimized pose information of each AprilTag detection surface, the optimized pose data of multiple AprilTag detection surfaces are fused and calculated to obtain the global pose of the polyhedron calibration tool.

[0014] Furthermore, the AprilTag markers on the polyhedron calibration tool all adopt a uniform size and ID allocation rule, and the AprilTag markers on each face have an assigned ID.

[0015] Furthermore, the data acquired from the depth camera is input into a pre-trained deep learning model to predict the pose estimate of the AprilTag detection surface, specifically including:

[0016] In Unity, build a 3D scene containing the target object with Apriltag, use Unity's Camera to simulate a depth camera, set the object's motion conditions, and allow it to move freely within the defined field of view and distance. Record the target object's motion trajectory and obtain the target object's pose, timestamp, and point cloud data.

[0017] Construct a deep learning model, including a feature extraction module, a pose and temporal encoding module, a feature fusion module, and a pose prediction module;

[0018] Model training: During the training phase, a training set is constructed using data simulated in Unity to train the deep learning model;

[0019] Model prediction: In the prediction phase, data acquired from the depth camera is input into the trained deep learning model, and the model outputs a pose estimate of the AprilTag detection surface. The pose consists of a translation vector t = (x, y, z) and a rotation matrix R.

[0020] Furthermore, model training specifically includes:

[0021] Inputs include: attitude Point cloud o, time step t;

[0022] Feature extraction: The point cloud o is input into the feature extraction module for feature extraction to obtain global features. ;

[0023] Attitude and Timing Coding: The attitude and timing coding modules are used to process the attitude separately. Encode the feature at time step t. and ;

[0024] Feature fusion: The feature fusion module is used to fuse the obtained features and By fusing the data, we can obtain global features. This forms a comprehensive contextual feature vector;

[0025] Pose prediction: Using the pose prediction module, the global features are first processed by the solver. The precise changes in attitude parameters at each time step are calculated using numerical integration methods, thereby generating a series of intermediate candidate attitude estimates. The optimal pose estimate is obtained by selecting or fusing candidate pose estimates. .

[0026] Furthermore, by combining the obtained depth information, the pose estimation of the AprilTag detection surface predicted by the deep learning model is optimized, specifically including:

[0027] First, the corrected depth value h of the center point of the AprilTag detection surface is obtained through weighted fusion:

[0028]

[0029] in, The depth value predicted by the deep learning model, i.e., the Z-axis coordinate; The mean depth measured by the depth camera; As weight;

[0030] Replace the original center point's Z-axis coordinates with the corrected depth value h to obtain the corrected center point coordinates (x, y, h) of the AprilTag detection surface, and obtain the corrected pose_matrix = [R | t], with the translation vector t = (x, y, h), where R is the rotation matrix and remains unchanged before and after correction.

[0031] Furthermore, based on the optimized pose information of each AprilTag detection surface, the optimized pose data of multiple AprilTag detection surfaces are fused and calculated to obtain the global pose of the polyhedron calibration tool, specifically including:

[0032] Based on the corrected coordinates of the center point of each AprilTag detection surface, estimate the volume center coordinates of each surface. :

[0033]

[0034] in, The position is the center of the i-th face; This represents the displacement from the center of the surface to the geometric center of the polyhedron. The normal vector from the center of the face to the geometric center of the polyhedron; Let be the side length of the polyhedron.

[0035] Furthermore, based on the optimized pose information of each AprilTag detection surface, the optimized pose data of multiple AprilTag detection surfaces are fused and calculated to obtain the global pose of the polyhedron calibration tool, specifically including:

[0036] Based on the volume center coordinates corresponding to each face The volume center coordinates of the polyhedron calibration tool are obtained through weighted fusion calculation. :

[0037]

[0038] in, Weights for each face; Here are the estimated volume center coordinates for each face; N is the total number of faces;

[0039] Polyhedral center rotation matrix The calculation is as follows:

[0040]

[0041] in, A pre-calculated transformation matrix is ​​defined based on the polyhedral geometry and the coordinate system of the polyhedral calibration tool, which is used to uniformly transform the local rotations of each face to the coordinate system of the polyhedral calibration tool; It is obtained by detecting the single-sided rotation matrix through AprilTag markers and uniformly transforming it to the coordinate system of the polyhedral calibration tool; yes The transpose matrix of ; where the coordinate system of the polyhedron calibration tool is defined with the center of the polyhedron as the origin.

[0042] According to a second aspect, one embodiment provides a computer vision localization system based on depth information and deep learning, the system comprising:

[0043] The image acquisition module is used to start the depth camera, acquire RGB images containing a polyhedral calibration tool, and obtain corresponding depth information and point cloud data. Each face of the polyhedral calibration tool is designed with an AprilTag mark to ensure that at least one mark can be detected from any viewpoint.

[0044] The single-face pose estimation module is used to input data acquired by a depth camera into a pre-trained deep learning model to predict the pose estimate of the AprilTag detection surface.

[0045] The pose optimization module is used to optimize the pose estimation of the AprilTag detection surface predicted by the deep learning model by combining the obtained depth information.

[0046] The global pose estimation module is used to fuse and calculate the optimized pose data of multiple AprilTag detection surfaces based on the optimized pose information of each AprilTag detection surface to obtain the global pose of the polyhedron calibration tool.

[0047] According to three aspects, one embodiment provides an electronic device, the device comprising: a processor and a memory;

[0048] The memory is used to store one or more program instructions;

[0049] The processor is configured to run one or more program instructions to perform the steps of a computer vision localization method based on depth information and deep learning as described in any of the preceding claims.

[0050] According to a fourth aspect, one embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a computer vision localization method based on depth information and deep learning as described in any of the preceding claims.

[0051] This invention provides a computer vision localization method based on depth information and deep learning, which has the following beneficial effects:

[0052] 1) Positioning accuracy in high occlusion environments: By designing a polyhedral AprilTag layout on the tool, at least one tag can be detected from any viewpoint, solving the problem that single-sided tags are easily occluded.

[0053] 2) Enhance environmental adaptability: Combine depth information from depth cameras with the robust detection of AprilTag by deep learning models to improve the system's localization stability in environments with changing lighting and weak textures.

[0054] 3) Reduce computational costs: Employ lightweight pose calculation methods to improve the system's usability in resource-constrained environments. Attached Figure Description

[0055] Figure 1 A flowchart illustrating a computer vision localization method based on depth information and deep learning, as provided in one embodiment of the present invention;

[0056] Figure 2 A flowchart illustrating the overall implementation of a computer vision localization method based on depth information and deep learning, as provided in one embodiment of the present invention.

[0057] Figure 3 A schematic diagram of the polyhedron AprilTag marking in a computer vision localization method based on depth information and deep learning, provided in an embodiment of the present invention;

[0058] Figure 4 A flowchart illustrating the prediction output of a deep learning model in a computer vision localization method based on depth information and deep learning, provided as an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram illustrating the construction of a scene and target detection object in Unity in a computer vision localization method based on depth information and deep learning, as provided in one embodiment of the present invention.

[0060] Figure 6 A diagram illustrating the architecture of a deep learning model in a computer vision localization method based on deep information and deep learning, provided as an embodiment of the present invention.

[0061] Figure 7 A schematic diagram of nmODE in a computer vision localization method based on depth information and deep learning provided in an embodiment of the present invention;

[0062] Figure 8 This invention provides a computer vision localization method based on depth information and deep learning, which includes predefined transformation matrices for each facet. Detailed Implementation

[0063] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to the present invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0064] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0065] The first embodiment of this invention provides a computer vision localization method based on depth information and deep learning. By innovatively arranging AprilTag markers on each face of a polyhedron, and combining precise depth information from a depth camera with a deep learning model, it achieves high-precision, highly robust, and computationally low pose localization. It is particularly suitable for scenarios such as medical surgical navigation, industrial robot localization, drone localization, and augmented reality applications, maintaining stable localization performance even in occluded environments. The following section will further elaborate on this method. Figure 1 and Figure 2 Please provide a detailed explanation.

[0066] like Figure 1 As shown, in step S100, the depth camera is activated, and the depth camera is used to acquire an RGB image containing a polyhedral calibration tool and obtain the corresponding depth information and point cloud data. Each face of the polyhedral calibration tool is designed with an AprilTag marker to ensure that at least one marker can be detected from any viewpoint.

[0067] The above steps specifically include:

[0068] S110, Polyhedral AprilTag Marking Design

[0069] like Figure 3As shown, taking a cube as an example, the polyhedral AprilTag marker uses a design where the markers are evenly distributed across the five faces of the cube, ensuring that at least one marker can be detected from any viewing angle. The markers use a uniform size (e.g., 50mm) and an ID allocation rule (e.g., 0-4, meaning the ID allocation starts from 0 and proceeds sequentially as 0, 1, 2, 3, 4) to facilitate subsequent processing. Specific design parameters are as follows:

[0070]

[0071] Depth Information Acquisition: A depth camera acquires color images and corresponding depth maps. The depth map provides depth information for each pixel, which can be directly used to verify and calculate the Z-axis coordinates. This embodiment uses a depth camera such as the Intel RealSense D456 or Intel RealSense D457. The depth camera directly acquires the Z-axis depth value using infrared structured light or ToF technology, and combines it with the camera intrinsic parameter matrix to convert the pixel coordinates (u,v) into three-dimensional coordinates in the camera coordinate system.

[0072] Figure 1 As shown, in step S200, the data acquired by the depth camera is input into the pre-trained deep learning model to predict the pose estimate of the AprilTag detection surface.

[0073] The above steps specifically include:

[0074] Deep learning models: Based on the diffusion model for object pose estimation, the core idea is to restore random noise to realistic data through progressive denoising, thereby achieving high-precision pose estimation. The overall processing flow is as follows... Figure 4 As shown.

[0075] 1) Training Data Construction: Construct a 3D scene in Unity containing Apriltag labels for detected objects, such as... Figure 5 As shown, after setting the motion conditions of the object, it can move freely within the defined field of view and distance. When recording the motion trajectory of the detected target object, data such as the target object's pose, timestamp, point cloud data, and camera intrinsic parameters can be obtained.

[0076] 2) To further improve the robustness of the model to different lighting, occlusion and sensor noise, this embodiment adds random Gaussian noise, color jitter and random downsampling strategies in the Unity rendering stage, so that the training set size can be expanded without additional collection.

[0077] 3) Model Construction: Construct a deep learning model network to implement the pose estimation network. The overall network architecture is as follows: Figure 6As shown, the architecture consists of a feature extraction module, a pose and temporal encoding module, a feature fusion module, and a pose prediction module. In this embodiment, the feature extraction module uses the PointNet++ network, and the pose and temporal encoding module uses a multilayer perceptron (MLP).

[0078] 4) Model training: Input pose during the training phase Point cloud (o), time step (t). All input data are standardized to ensure that the data in each dimension has the same scale, facilitating subsequent calculations and optimization.

[0079] 5) Model prediction: The prediction data is provided by the depth camera, which provides the raw output data at 30fps. RGB frames, 30fps The model uses depth frames and 16-bit depth maps for point cloud data. Before feeding the point cloud data into the model, it also needs to undergo standardization to ensure data consistency. Furthermore, denoising algorithms are applied to reduce noise interference and improve the accuracy of pose estimation.

[0080] 6) The point cloud data input during the training and inference phase contains partial geometric information of the target object. To extract global features from this point cloud data, this embodiment uses the PointNet++ network. PointNet++ is a powerful point cloud processing network capable of automatically learning local and global structural features in point clouds. The specific global extraction formula is as follows: .

[0081] The point cloud is input into the PointNet++ network, and after a series of convolution and pooling operations, global features are extracted. PointNet++ effectively captures the global geometric properties of point clouds through multi-level feature aggregation and upsampling, maintaining good robustness even when the point cloud is sparse or partially occluded.

[0082] 7) To transform the pose parameters and time steps into a form that can be processed by a neural network, this embodiment uses an MLP (Multilayer Perceptron) to process the pose. Encode the feature at time step t. and .in .

[0083] The multilayer perceptron (MLP) will determine the attitude. The time step t is input into the MLP, and the pose parameters are mapped into high-dimensional features through multiple fully connected layers and activation functions. and characteristics The MLP Encoder can capture the non-linear relationships between attitude parameters, providing rich contextual information for subsequent attitude adjustments.

[0084] The multilayer perceptron (MLP) has two branches responsible for predicting rotational and temporal components. After each layer, the MLP applies the ReLU activation function to process the input data, resulting in pose and temporal feature data.

[0085] 8) After processing by a multilayer perceptron encoder, the obtained features are... and Fusion yields global features This forms a comprehensive contextual feature vector. This feature vector contains all the information about the current point cloud observation, the initial attitude assumption, and the stage of the diffusion process, providing comprehensive input for subsequent attitude prediction.

[0086] 9) In this embodiment, the information obtained from the above processing is input into the nmODE (Neural Memory Ordinary Differential Equation) solver. The nmODE solver updates the attitude parameters step by step, causing them to gradually converge from the initial noisy state to the true attitude.

[0087] Specifically, the nmODE solver will use global features The nmODE solver uses numerical integration to calculate the precise changes in attitude parameters at each time step, thus generating a series of intermediate attitude estimates. The nmODE calculation method is as follows: Figure 7 As shown.

[0088] Through iterative optimization of the nmODE solver, this embodiment ultimately yields a series of attitude candidates. These pose candidates contain multiple intermediate results that progressively approximate the true pose from the initial noisy state. To obtain the final, optimal pose estimate, an optimal value needs to be selected or fused from these candidates.

[0089] 10) This embodiment employs an average pooling strategy to fuse all candidate poses to obtain the final pose estimation result. Specifically, for a series of pose candidates generated iteratively by the nmODE solver... In this embodiment, its translation component is... and rotational components Value pooling is performed separately. To ensure the rationality of the rotation average, all rotation matrices... First, convert the vector to the corresponding rotation vector or unit quaternion form, perform linear averaging in the vector space, and then convert it back to the rotation matrix to obtain the final average pose.

[0090] 11) After nmODE solving and mean pooling, this embodiment obtains an accurate object pose estimate. The pose includes a translation vector: t = (x, y, z) and a rotation matrix: R = Pose matrix: pose_matrix = [R | t].

[0091] In this embodiment, The input pose is obtained from the depth camera; The posture is derived through reasoning by a deep learning model. The accurate pose matrix needs to be obtained through subsequent processes such as deep information fusion, polyhedral position fusion, and polyhedral rotation fusion.

[0092] like Figure 1 As shown, in step S300, the pose estimation of the AprilTag detection surface predicted by the deep learning model is optimized by combining the obtained depth information.

[0093] The above steps specifically include:

[0094] S310, firstly, the corrected depth value h of the center point of the AprilTag detection surface is obtained through weighted fusion:

[0095]

[0096] Where h is the corrected depth value after fusion; The depth value predicted by the deep learning model, i.e., the Z-axis coordinate; The mean depth measured by the depth camera; The value is the weight, and its range is [0,1].

[0097] S320, replace the original center point's Z-axis coordinate with the corrected depth value h to obtain the corrected coordinates (x, y, h) of the center point of the single-sided AprilTag marker, and obtain the corrected pose_matrix = [R | t], with the translation vector t = (x, y, h), where R is the rotation matrix and remains unchanged before and after the correction.

[0098] like Figure 1 As shown, in step S400, based on the optimized pose information of each AprilTag detection surface, the optimized pose data of multiple AprilTag detection surfaces are fused and calculated to obtain the global pose of the polyhedron calibration tool.

[0099] The above steps specifically include:

[0100] S410, based on polyhedral geometric constraints, uses the mean method to fuse and calculate the pose data of multiple faces after AprilTag marking optimization to obtain the global pose of the polyhedral calibration tool.

[0101] like Figure 5 As shown, in this embodiment, the three-dimensional geometric center of the entire polyhedron is estimated based on the detected positions of the AprilTag faces. The details are as follows:

[0102] 1) Define coordinates and parameters

[0103] : The first in camera coordinate system The coordinates of the center point of the detected AprilTag, i.e., the corresponding coordinates (x, y, h).

[0104] : Corresponding to the normal vector of the AprilTag surface;

[0105] : Side length of a polyhedron.

[0106] 2) Calculation of the center of a single-sided polyhedron

[0107] For each detected AprilTag surface, the volume center coordinates corresponding to each surface are estimated as follows:

[0108]

[0109] in:

[0110] : The center position of the i-th face in the camera coordinate system;

[0111] : Tag normal vector, pointing to the interior of the polyhedron, the direction is given by the Tag orientation;

[0112] : Side length of the polyhedron;

[0113] Displacement from the center of the surface to the geometric center of the polyhedron.

[0114] 3) Polyhedron Fusion Algorithm

[0115] Formula for calculating the center pose of a polyhedron:

[0116]

[0117] The weight of each face can be set to 1 or based on the confidence level;

[0118] : Estimated volume center coordinates for each face.

[0119] Polyhedral center rotation matrix The calculation is as follows:

[0120]

[0121] in:

[0122] Define pre-calculated transformation matrices based on polyhedral geometry and the coordinate system of polyhedral calibration tools, such as... Figure 8 As shown, this is used to uniformly transform the local rotations of each face to the coordinate system of the polyhedron calibration tool;

[0123] It is obtained by detecting the single-sided rotation matrix through AprilTag markers and uniformly transforming it to the coordinate system of the polyhedral calibration tool;

[0124] yes The transpose of .

[0125] Coordinate system determination method for polyhedron calibration tools:

[0126] Origin: The geometric center of the polyhedron;

[0127] Z-axis: Direction of the normal vector of plane 0 (top);

[0128] X-axis: Direction of the normal vector of plane 1 (front);

[0129] Y-axis: Direction of the normal vector of plane 2 (right side).

[0130] Using a polyhedron fusion algorithm and a polyhedron center pose rotation calculation method, the position and rotation matrix of the polyhedron's center point are obtained. These position and rotation matrices are collectively referred to as the global pose. At this point, the localization system can continuously and in real-time detect and calculate the pose of the polyhedron's center point in the world coordinate system, and subsequent calibration uses only a unified pose matrix.

[0131] The visual positioning method of this invention has broad application prospects, as detailed below:

[0132] 1) Medical surgical navigation: In scenarios such as minimally invasive surgery and robot-assisted surgery, it can achieve sub-millimeter-level positioning accuracy of surgical instruments, significantly improving the success rate and safety of surgery.

[0133] 2) Industrial robot positioning: In scenarios such as AGV navigation and robot assembly, it can provide high-precision, low-latency positioning services to improve production efficiency and quality.

[0134] 3) Augmented Reality (AR) Applications: In scenarios such as AR games and AR navigation, it can provide stable and accurate positioning information, thereby improving the user experience.

[0135] 4) Drone positioning: In drone applications, it can provide high-precision, interference-resistant positioning solutions, especially in complex environments or indoor scenarios.

[0136] Corresponding to the aforementioned computer vision localization method based on depth information and deep learning, this invention also discloses a computer vision localization system based on depth information and deep learning, which specifically includes:

[0137] The image acquisition module is used to start the depth camera, acquire RGB images containing a polyhedral calibration tool, and obtain corresponding depth information and point cloud data. Each face of the polyhedral calibration tool is designed with an AprilTag mark to ensure that at least one mark can be detected from any viewpoint.

[0138] The single-face pose estimation module is used to input data acquired by a depth camera into a pre-trained deep learning model to predict the pose estimate of the AprilTag detection surface.

[0139] The pose optimization module is used to optimize the pose estimation of the AprilTag detection surface predicted by the deep learning model by combining the obtained depth information.

[0140] The global pose estimation module is used to fuse and calculate the optimized pose data of multiple AprilTag detection surfaces based on the optimized pose information of each AprilTag detection surface to obtain the global pose of the polyhedron calibration tool.

[0141] It should be noted that for a detailed description of a computer vision localization system based on depth information and deep learning provided in the embodiments of the present invention, please refer to the relevant description of a computer vision localization method based on depth information and deep learning provided in the embodiments of the present invention, which will not be repeated here.

[0142] In addition, embodiments of the present invention also provide an electronic device, the device comprising: a processor and a memory; the memory being used to store one or more program instructions; the processor being used to execute one or more program instructions to perform the steps of a computer vision localization method based on depth information and deep learning as described in any of the preceding embodiments.

[0143] It should be noted that for a detailed description of an electronic device provided in the embodiments of the present invention, please refer to the relevant description of a computer vision localization method based on depth information and deep learning provided in the embodiments of this application, which will not be repeated here.

[0144] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a computer vision localization method based on depth information and deep learning as described in any of the preceding claims.

[0145] It should be noted that for a detailed description of the computer-readable storage medium provided in the embodiments of the present invention, please refer to the relevant description of the computer vision localization method based on depth information and deep learning provided in the embodiments of this application, which will not be repeated here.

[0146] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A computer vision localization method based on depth information and deep learning, characterized in that, The method includes: Start the depth camera, use the depth camera to acquire RGB images containing a polyhedral calibration tool and obtain the corresponding depth information and point cloud data. The polyhedral calibration tool is designed with AprilTag markers on multiple faces to ensure that at least one marker can be detected from any viewpoint. Data acquired from a depth camera is input into a pre-trained deep learning model to predict the pose estimate of the AprilTag detection surface. Data acquired from a depth camera is input into a pre-trained deep learning model to predict the pose estimate of the AprilTag detection surface, specifically including: In Unity, build a 3D scene containing the target object with Apriltag, use Unity's Camera to simulate a depth camera, set the object's motion conditions, and allow it to move freely within the defined field of view and distance. Record the target object's motion trajectory and obtain the target object's pose, timestamp, and point cloud data. Construct a deep learning model, including a feature extraction module, a pose and temporal encoding module, a feature fusion module, and a pose prediction module; Model training: During the training phase, a training set is constructed using data simulated in Unity to train the deep learning model; Model prediction: In the prediction phase, data acquired from the depth camera is input into the trained deep learning model, and the model outputs a pose estimate of the AprilTag detection surface. The pose includes a translation vector t = (x, y, z) and a rotation matrix R; Model training specifically includes: Inputs include: attitude Point cloud o, time step t; Feature extraction: The point cloud o is input into the feature extraction module for feature extraction to obtain global features. ; Attitude and Timing Coding: The attitude and timing coding modules are used to process the attitude separately. Encode the feature at time step t. and ; Feature fusion: The feature fusion module is used to fuse the obtained features and By fusing the data, we can obtain global features. This forms a comprehensive contextual feature vector; Pose prediction: Using the pose prediction module, the global features are first processed by the solver. The precise changes in attitude parameters at each time step are calculated using numerical integration methods, thereby generating a series of intermediate candidate attitude estimates. The optimal pose estimate is obtained by selecting from or fusing candidate pose estimates. ; By combining the obtained depth information, the pose estimation of the AprilTag detection surface predicted by the deep learning model is optimized. Based on the optimized pose information of each AprilTag detection surface, the optimized pose data of multiple AprilTag detection surfaces are fused and calculated to obtain the global pose of the polyhedron calibration tool.

2. The computer vision localization method based on depth information and deep learning as described in claim 1, characterized in that, The AprilTag markers on the polyhedron calibration tool all adopt a uniform size and ID allocation rule, and each AprilTag marker on each face has an assigned ID.

3. The computer vision localization method based on depth information and deep learning as described in claim 1, characterized in that, Combining the obtained depth information, the pose estimation of the AprilTag detection surface predicted by the deep learning model is optimized, specifically including: First, the corrected depth value h of the center point of the AprilTag detection surface is obtained through weighted fusion: in, The depth value predicted by the deep learning model, i.e., the Z-axis coordinate; The mean depth measured by the depth camera; As weight; Replace the original center point's Z-axis coordinates with the corrected depth value h to obtain the corrected center point coordinates (x, y, h) of the AprilTag detection surface, and obtain the corrected pose_matrix = [R | t], with the translation vector t = (x, y, h), where R is the rotation matrix and remains unchanged before and after correction.

4. The computer vision localization method based on depth information and deep learning as described in claim 3, characterized in that, Based on the optimized pose information of each AprilTag detection surface, the optimized pose data of multiple AprilTag detection surfaces are fused and calculated to obtain the global pose of the polyhedron calibration tool, specifically including: Based on the corrected coordinates of the center point of each AprilTag detection surface, estimate the volume center coordinates of each surface. : in, The position is the center of the i-th face; This represents the displacement from the center of the surface to the geometric center of the polyhedron. The normal vector from the center of the face to the geometric center of the polyhedron; Let be the side length of the polyhedron.

5. The computer vision localization method based on depth information and deep learning as described in claim 4, characterized in that, Based on the optimized pose information of each AprilTag detection surface, the optimized pose data of multiple AprilTag detection surfaces are fused and calculated to obtain the global pose of the polyhedron calibration tool, specifically including: Based on the volume center coordinates corresponding to each face The volume center coordinates of the polyhedron calibration tool are obtained through weighted fusion calculation. : in, Weights for each face; Here are the estimated volume center coordinates for each face; N is the total number of faces; Polyhedral center rotation matrix The calculation is as follows: in, A pre-calculated transformation matrix is ​​defined based on the polyhedral geometry and the coordinate system of the polyhedral calibration tool, which is used to uniformly transform the local rotations of each face to the coordinate system of the polyhedral calibration tool; It is obtained by detecting the single-sided rotation matrix through AprilTag markers and then uniformly transforming it to the coordinate system of the polyhedral calibration tool. yes The transpose matrix of ; where the coordinate system of the polyhedron calibration tool is defined with the center of the polyhedron as the origin.

6. A computer vision localization system based on depth information and deep learning, characterized in that, The system includes: The image acquisition module is used to start the depth camera, acquire RGB images containing a polyhedral calibration tool, and obtain corresponding depth information and point cloud data. Each face of the polyhedral calibration tool is designed with an AprilTag mark to ensure that at least one mark can be detected from any viewpoint. The single-face pose estimation module is used to input data acquired by a depth camera into a pre-trained deep learning model to predict the pose estimate of the AprilTag detection surface. Data acquired from a depth camera is input into a pre-trained deep learning model to predict the pose estimate of the AprilTag detection surface, specifically including: In Unity, build a 3D scene containing the target object with Apriltag, use Unity's Camera to simulate a depth camera, set the object's motion conditions, and allow it to move freely within the defined field of view and distance. Record the target object's motion trajectory and obtain the target object's pose, timestamp, and point cloud data. Construct a deep learning model, including a feature extraction module, a pose and temporal encoding module, a feature fusion module, and a pose prediction module; Model training: During the training phase, a training set is constructed using data simulated in Unity to train the deep learning model; Model prediction: In the prediction phase, data acquired from the depth camera is input into the trained deep learning model, and the model outputs a pose estimate of the AprilTag detection surface. The pose includes a translation vector t = (x, y, z) and a rotation matrix R; Model training specifically includes: Inputs include: attitude Point cloud o, time step t; Feature extraction: The point cloud o is input into the feature extraction module for feature extraction to obtain global features. ; Attitude and Timing Coding: The attitude and timing coding modules are used to process the attitude separately. Encode the feature at time step t. and ; Feature fusion: The feature fusion module is used to fuse the obtained features and By fusing the data, we can obtain global features. This forms a comprehensive contextual feature vector; Pose prediction: Using the pose prediction module, the global features are first processed by the solver. The precise changes in attitude parameters at each time step are calculated using numerical integration methods, thereby generating a series of intermediate candidate attitude estimates. The optimal pose estimate is obtained by selecting from or fusing candidate pose estimates. ; The pose optimization module is used to optimize the pose estimation of the AprilTag detection surface predicted by the deep learning model by combining the obtained depth information. The global pose estimation module is used to fuse and calculate the optimized pose data of multiple AprilTag detection surfaces based on the optimized pose information of each AprilTag detection surface to obtain the global pose of the polyhedron calibration tool.

7. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store one or more program instructions; The processor is configured to run one or more program instructions to perform the steps of a computer vision localization method based on depth information and deep learning as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a computer vision localization method based on depth information and deep learning as described in any one of claims 1 to 5.

Images