A monocular camera-based pose recognition method for any object and an application system

Abstract

The application provides a monocular camera pose recognition method for any object and an application system, and the method comprises the following steps: S1, constructing a target three-dimensional model, collecting multi-view ring shooting images of the target, and generating a dense grid model through feature extraction, matching, pose solving and multi-view geometry method; S2, generating an image depth map, and predicting the depth information of the target in the movement process based on the monocular image sequence; S3, extracting a target image mask, and generating a target region mask image through an image encoder, a prompt encoder and a mask decoder; S4, performing pose estimation, and jointly performing pose initialization, correction and screening on the three-dimensional model, the depth map and the mask image, and outputting a target six-degree-of-freedom pose result. The application realizes ring shooting modeling of the target based on the multi-view geometry method, generates a three-dimensional model of the target, and realizes pose estimation in combination with the image mask and the depth map, so that the generalization ability and the application range of the pose estimation algorithm in the actual scene are improved.

Description

Technical Field

[0001] This invention relates to the field of target posture recognition technology, and in particular to a method and application system for posture recognition of arbitrary objects using a monocular camera. Background Technology

[0002] Target pose estimation is one of the core research tasks in the field of computer vision. Its goal is to recover the position and orientation (i.e. translation and rotation information) of an object in three-dimensional space from an image or video sequence. This task has wide application value in scenarios such as augmented reality, smart manufacturing, autonomous driving, and medical image navigation.

[0003] Traditional pose estimation methods typically rely on hand-designed image feature extraction algorithms, such as SIFT and HOG, combined with PnP (Perspective-n-Point) models to estimate the pose of objects in three-dimensional space. These methods can achieve high accuracy under simple structural or controlled lighting conditions, but they are unstable and have poor robustness in real-world scenarios such as complex backgrounds, occlusion interference, scale changes, or textureless objects.

[0004] With the development of deep learning, end-to-end pose estimation methods based on neural networks have gradually emerged in recent years, mainly including two categories: one is based on 2D keypoint detection and 3D regression, such as OpenPose and PoseNet; the other is to directly predict six degrees of freedom (6DoF) pose through deep networks, such as PVNet, GDR-Net, and DenseFusion. However, most of these methods are based on class-specific model training, requiring a large number of labeled samples for supervised learning, resulting in weak generalization and difficulty in adapting to the rapid deployment and recognition of arbitrary objects in real-world scenes.

[0005] Traditional technology 1 [Application No.: 202411576347.0] discloses a 3D pose estimation method based on a monocular camera. By constructing a synthetic and real mixed dataset, the YOLOPose model is trained to extract 2D key points and fit a GHUM 3D human body model. Although this method has a certain accuracy in human pose recognition, it cannot estimate the pose of other types of targets, let alone unknown types of targets. At the same time, this method still requires special model training before it can be used for human pose estimation.

[0006] Traditional technique 2 [Application No.: 202411221949.4] proposed a "monocular six-DOF pose estimation method for spatial targets", which obtains the relative pose between adjacent frames through image masking and depth estimation; However, this method only applies to spacecraft objects moving in space and cannot estimate the attitude of other types of targets or unknown types of targets. In addition, this method only applies image input and estimated depth data, and does not apply information such as the target's three-dimensional data, resulting in low accuracy.

[0007] To address this, a method and application system for pose recognition of arbitrary objects using a monocular camera are proposed. Summary of the Invention

[0008] In view of this, the present invention provides a method and application system for pose recognition of arbitrary objects using a monocular camera, in order to solve or alleviate the technical problems existing in the prior art, and at least provide a beneficial option.

[0009] The technical solution of this invention is implemented as follows: a method for pose recognition of any object using a monocular camera, the method comprising the following steps: Step S1: Construct a 3D model of the target: Collect panoramic images of the target and construct a dense mesh model through feature extraction, image registration, camera pose calculation and point cloud reconstruction. Step S2: Generate image depth map: Estimate the depth of the image sequence during the target's motion process, and complete the scale transformation and 3D mapping; Step S3: Extract the target image mask: Perform interactive point selection in the target image, and generate a target area mask map through image encoding and prompt information; Step S4: Perform attitude estimation: Combine the target 3D model, image depth map and target region mask map to construct an initial attitude hypothesis set, and perform attitude initialization, attitude correction and attitude filtering to output the target's six-degree-of-freedom attitude result.

[0010] More preferably, in step S1, the construction of the target 3D model includes: using the ORB feature extraction algorithm to detect key points from the panoramic image and generate descriptors, matching feature points between adjacent images through descriptors, using the RANSAC method for geometric consistency screening, and obtaining the rotation and translation parameters of the image pair by solving the fundamental matrix based on the feature matching and combined with the camera intrinsic parameters. The sparse point cloud is reconstructed using the Structure from Motion method, then a dense point cloud is generated using a multi-view stereo matching algorithm, and finally a triangular mesh is constructed using Delaunay triangulation, and a texture model is generated by combining the original image texture.

[0011] More preferably, in step S2, generating the image depth map includes: generating a relative depth map for each frame of the input image using a monocular image depth estimation network, performing scale calibration by combining known structures or calibration reference points in the selected frame, and back-projecting each pixel position to the three-dimensional spatial coordinate system through camera intrinsic parameters to complete the point cloud mapping from image to space for subsequent target position initialization.

[0012] More preferably, in step S3, the extraction of the target image mask includes: using an image encoder based on a visual model to extract high-dimensional semantic features from the entire image; inputting foreground and background points obtained by user interaction to a prompt encoder to generate spatial guidance information; jointly inputting the image encoding and prompt encoding into a mask decoder; and outputting a target region mask map with consistent resolution. The mask map is directly associated with the image cropping region required for pose estimation in step S4.

[0013] More preferably, in the attitude initialization process in step S4, the pixel set corresponding to the depth map is extracted in the target area identified by the mask image, and the target depth position is obtained by median calculation. The initial translation vector of the target in the world coordinate system is obtained by combining the monocular camera intrinsic back projection. The initial estimation of the rotation matrix is ​​to uniformly sample multiple observation directions on an icosphere constructed with the target center as the center, forming a rotation hypothesis set as a candidate for attitude initialization.

[0014] In a further preferred embodiment, each set of initial pose parameters in the initial hypothesis set described in step S4 is applied to the target 3D model to perform image rendering, generating a corresponding simulated image. Then, the simulated image is compared with the texture of the masked cropping region in the real observed image to construct a pose loss function based on image error. The rotation and translation parameters are then optimized by gradient backpropagation using an implicit neural network representation to obtain a converged high-confidence pose estimation set.

[0015] In a further preferred embodiment, during the pose selection process in step S4, target image rendering is performed on each set of pose results output by the pose correction, the image semantic coding features are extracted, and they are compared one by one with the coding features of the input image mask region. The pose set is sorted by the indicators of contour overlap, region alignment and texture matching, and finally the pose with the highest score is output as the estimation result of the current frame.

[0016] This invention also provides an application system for a pose recognition method for arbitrary objects using a monocular camera, the system comprising: The 3D model building module is used to perform the target 3D modeling operation in step S1; The depth map generation module is used to perform the image depth map construction operation in step S2; The image mask generation module is used to perform the target region mask extraction operation in step S3; The attitude estimation module includes an initialization submodule, a correction submodule, and a filtering submodule, which are used to execute the attitude estimation process in step S4; Each module in the system corresponds to a step in the method. The control program running the processor calls the modules to work together to complete the attitude recognition task for any target.

[0017] More preferably, the system completes the caching of the execution results of the 3D model construction module during the initialization phase, the depth map generation module and the image mask generation module support dynamic processing of real-time input image sequences, and the pose estimation module optimizes multiple sets of parameters of the initial pose hypothesis set constructed in step S4 based on a parallel computing architecture, and outputs high-confidence pose estimation results.

[0018] The embodiments of the present invention have the following advantages due to the adoption of the above technical solutions: I. This invention uses a multi-view geometry-based method to perform panoramic modeling of the target, generating a 3D model of the target, and combines image masking and depth map to achieve pose estimation. Unlike traditional deep learning methods, it does not rely on large-scale training data for the target object or specific scene, nor does it require model fine-tuning or transfer learning. It can be applied to any type of target object without prior knowledge, significantly improving the generalization ability and applicability of the pose estimation algorithm in real-world scenarios, and overcoming the limitation of traditional methods that can only identify trained objects.

[0019] Second, compared with traditional methods that require the deployment of multiple cameras or the use of complex sensor devices such as structured light and depth cameras to collect multi-view motion data, the technical solution provided by this invention only requires a single ordinary monocular camera to collect image sequences of the target during its motion to complete the attitude estimation process. This method significantly reduces the difficulty of system deployment and the cost of data acquisition, and is suitable for use in resource-constrained, rapid deployment or large-scale application scenarios.

[0020] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the invention will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0023] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0024] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0025] like Figure 1 As shown, this embodiment of the invention provides a method for pose recognition of any object using a monocular camera. The method includes the following steps: Step S1: Construct the target 3D model This step involves capturing panoramic images of the target while it is stationary, and then generating a complete 3D surface model of the target through image processing and reconstruction techniques. Specifically... Image acquisition employs a fixed-focal-length monocular camera to capture images from multiple angles around the target, with the angles evenly distributed to cover 360° and the viewing angle covering at least 20 frames. All images maintain consistent exposure and resolution during acquisition. For each image, the ORB (Oriented Fast and Rotated BRIEF) algorithm is used to extract corner points and descriptor information. This ORB algorithm possesses both scale and rotation invariance and fast computation characteristics, making it suitable for high-frequency recognition scenarios in industry. For feature points between adjacent images, coarse matching is performed using descriptor distance, and the RANSAC algorithm is introduced for geometric consistency filtering to eliminate mismatched point pairs and improve registration accuracy. By matching feature point pairs and camera intrinsic parameter matrices, the fundamental matrix between image pairs is calculated, and then the essential matrix is ​​solved. The camera extrinsic parameters, namely the rotation matrix and translation vector, are obtained through SVD decomposition, realizing a unified coordinate representation of multi-view images. Based on the acquired pose and feature points, all images are uniformly input into the Structure from Motion (SfM) process to reconstruct sparse point clouds. Subsequently, Multi-view Stereo (MVS) technology is used to expand them into dense point cloud data. The point cloud surface is reconstructed using Poisson or Delaunay triangulation to obtain a continuous target triangular mesh model.

[0026] The RGB color data of the corresponding grid area is extracted from the original image and mapped onto the triangular face to form a texture map, so that the model has both geometric integrity and retains the target texture information.

[0027] Output: The completed target 3D model includes geometry and color texture, which will be used as a 3D reference in the subsequent pose recognition process.

[0028] Step S2: Generate image depth map This step is performed during the target's motion. It involves reconstructing the depth information of each frame in the scene using a monocular image sequence, which is then used for pose initialization and localization. Specifically... Image sequences of the target moving continuously are acquired using a monocular camera at a frame rate of ≥10fps to ensure smooth pose changes and good data continuity. The input image frames are fed into a depth estimation model, such as a depth estimation network based on a U-Net or Transformer structure, and output a relative depth map. Before generating the depth map, the target object region needs to be identified from the input image through a mask or image classification module. The prediction range is limited during the depth estimation process to reduce the impact of background interference on depth accuracy. Each pixel in the depth prediction result is combined with the target region index and mapped to a unified three-dimensional coordinate system through a back-projection algorithm with the camera intrinsic parameters to generate a dense point cloud structure.

[0029] Since monocular estimation only has relative depth information, by identifying known physical reference objects in the image or using the model size in the 3D modeling stage, the mapping from relative depth to actual scale can be achieved. The depth value of each pixel in the depth map is combined with the camera intrinsic parameters for back projection to obtain the corresponding dense point cloud in the frame image, that is, to establish the mapping relationship from the image pixel plane to the three-dimensional coordinates. Output: Depth map (Z channel) and dense point cloud structure for each frame of image.

[0030] Step S3: Extract the target image mask This step is used to segment the target image region before pose estimation, serving as a crucial foundation for cropping region and image alignment calculations. Specifically... The original image is input to an image encoder (optionally SAM, DINO, or Mask2Former model) to obtain a global semantic representation. Through user interaction or rule recognition, several pixel coordinates inside and outside the target area are selected as foreground and background points. The prompt encoder converts these into guiding vectors for mask inference. The user manually selects foreground and background points in the target area image, and the system records the spatial coordinates of the selected pixels, forming a list of selected coordinates. This list drives the prompt encoder to generate local guiding information, forming a prompt vector. The image encoding result and the prompt encoding result are input into the mask decoder. After multi-scale fusion, a mask image of the target area is output. The mask image is a binary image with the same resolution as the input image, where the foreground target area is 1 and the other areas are 0.

[0031] Output: A target mask image that can be used for image cropping and pose optimization.

[0032] Step S4: Perform attitude estimation This step involves comprehensively using the target's 3D model, depth map, and mask map to estimate the target's pose for each frame in the image sequence. Specifically, In the masked region, all valid pixels in the depth map are extracted. The target depth center is obtained by calculating the median. Combined with camera intrinsic parameters, the image is back-projected to world coordinates to obtain the initial translation vector. An icosphere is constructed at the target center, and several pose directions are uniformly sampled on its surface. Each direction, together with the camera viewpoint, forms a set of rotation matrices as the initial rotation hypothesis set. The original image is then cropped using the masked image to generate a cropped image, which is paired with the simulated rendering image for pose error calculation. The cropped image can be considered as the effective target region in the observed image and is used to calculate pixel alignment. Each initial pose is input into the 3D model, viewpoint rendering is performed to obtain a simulated image (RGB rendering), the simulated image is compared with the cropped image in terms of texture, and a loss function is constructed based on the contour overlap and texture error between the images; a hidden neural network or a deep pose regression network is used for gradient backpropagation, and the rotation and translation parameters are iteratively updated until the loss converges.

[0033] For each optimized set of pose rendering images, input the encoder to extract image features, compare them with the mask region features of the real image, score all pose candidates based on multiple indicators such as contour overlap, region alignment rate, and texture residual, and sum the scores according to weights. Output the pose result with the highest score as the final estimate of the image in this frame and store it in the pose sequence.

[0034] In addition, this invention also provides an application system for a pose recognition method for arbitrary objects using a monocular camera. The system includes the following modules, which are interconnected via a control bus or connected via a logic call interface, and work with a processor and memory to implement the algorithm's execution flow: The 3D model building module is used to implement the target 3D modeling operation in step S1, including target image acquisition, ORB feature extraction, feature point matching, RANSAC geometric filtering, fundamental matrix and essential matrix calculation, Structure from Motion point cloud recovery, multi-view stereo matching (MVS) reconstruction of dense point cloud, and Delaunay triangulation to realize mesh reconstruction and texture map generation.

[0035] Preferably, the 3D model building module is pre-running during the system initialization phase and caches the output target 3D model in the memory for subsequent recognition processes, avoiding repeated calculation of modeling steps in each recognition task and improving system response efficiency.

[0036] The depth map generation module, corresponding to step S2, is used to generate the depth map and corresponding point cloud of each frame of the image based on the image sequence during the target's motion. The depth map generation module integrates a monocular depth estimation network, generates a relative depth map through the input sequence of images, and performs back projection operation in combination with camera intrinsic parameters to map the pixels to a three-dimensional coordinate system and construct a dense point cloud structure. In actual operation, this module can process continuous input images in real time to adapt to the continuous changes in the target's motion.

[0037] The image mask generation module, corresponding to step S3, is used to extract the foreground mask region of the target in the image, which serves as the basis for image cropping in subsequent pose estimation. The image mask generation module integrates an image encoder, a cue encoder, and a mask decoder. It collects the positions of foreground and background points in the image through user selection or rule recognition, and generates a cue vector. The global semantic information output by the image encoder and the cue vector are input into the mask decoder together, and the target mask image is generated through fusion inference.

[0038] The attitude estimation module, corresponding to step S4, executes the core process of attitude estimation. This attitude estimation module includes the following sub-modules: The initialization submodule is used to calculate the translation vector of the target based on the depth map and mask region, and to construct a uniformly distributed initial rotation hypothesis set based on the icosphere; The correction submodule applies the initial pose to the target 3D model for image rendering, compares the texture of the simulated image with the cropped image, constructs the pose residual loss, and optimizes the pose parameters through an implicit neural network. The filtering submodule sorts all pose candidates based on metrics such as image semantic feature similarity, contour overlap, and texture alignment, and selects the optimal pose as the output.

[0039] In this embodiment, the attitude estimation module adopts a parallel computing architecture during system operation, which can simultaneously optimize multiple initial attitude assumptions in parallel, improve the attitude calculation speed and stability, and is especially suitable for attitude update tasks in high frame rate real-time scenarios.

[0040] The system is uniformly scheduled by the control program running on the processor, calling each module in sequence according to steps S1 to S4, and working together to complete the entire process of recognition from input image to pose result. All modules transmit 3D model, depth map, mask map and intermediate rendering image through a unified data interface to ensure data consistency and process traceability.

[0041] In this embodiment, the English abbreviations of the following technical terms are explained as follows: ORB (Oriented FAST and Rotated BRIEF) is an image feature extraction algorithm that combines FAST corner detection with BRIEF descriptors. It has rotation and scale invariance and is suitable for quickly extracting key points and their descriptive information in images, and can be used for matching between multi-view images.

[0042] RANSAC (Random Sample Consensus): This algorithm is used to robustly estimate model parameters in sample sets containing errors or outliers. In this invention, it is used for geometric consistency screening in the image feature matching process to eliminate incorrect matching point pairs.

[0043] SfM (Structure from Motion) is a method that uses feature points extracted from image sequences and camera viewpoint information to gradually reconstruct the 3D structure of a scene and the camera pose, and is used to construct sparse point clouds.

[0044] MVS (Multi-view Stereo): This technology uses multi-view stereo matching to generate denser point clouds from sparse point clouds through stereo matching of images from multiple angles, enabling more detailed modeling of 3D structures.

[0045] U-Net is a typical encoder-decoder convolutional neural network that is widely used in tasks such as image segmentation and depth estimation. In this invention, U-Net can be used as the foundation for building a monocular depth estimation network.

[0046] RGB: Represents the three-channel color composition of an image, corresponding to the red, green, and blue color channels respectively. It is the basic image data format for image rendering and comparison.

[0047] Crop image: refers to a sub-region of the image obtained by cropping from the original image based on the target region mask. This image retains the pixel information of the target region and is used for pose comparison with the rendered image.

[0048] icosphere: An approximately spherical three-dimensional mesh structure composed of uniformly distributed triangles, used to sample rotation directions at equal intervals during the attitude initialization phase.

[0049] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in the present invention, and these should all be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for pose recognition of arbitrary objects using a monocular camera, characterized in that: The method steps include: Step S1: Construct a 3D model of the target: Collect panoramic images of the target and construct a dense mesh model through feature extraction, image registration, camera pose calculation and point cloud reconstruction. Step S2: Generate image depth map: Estimate the depth of the image sequence during the target's motion process, and complete the scale transformation and 3D mapping; Step S3: Extract the target image mask: Perform interactive point selection in the target image, and generate a target area mask map through image encoding and prompt information; Step S4: Perform attitude estimation: Combine the target 3D model, image depth map and target region mask map to construct an initial attitude hypothesis set, and perform attitude initialization, attitude correction and attitude filtering to output the target's six-degree-of-freedom attitude result.

2. The pose recognition method for any object using a monocular camera according to claim 1, characterized in that: In step S1, the construction of the target 3D model includes: using the ORB feature extraction algorithm to detect key points from the panoramic image and generate descriptors; matching feature points between adjacent images through descriptors; using the RANSAC method for geometric consistency screening; and, based on feature matching, obtaining the rotation and translation parameters of the image pair by solving the fundamental matrix in combination with camera intrinsic parameters. The sparse point cloud is reconstructed using the Structure from Motion method, then a dense point cloud is generated using a multi-view stereo matching algorithm, and finally a triangular mesh is constructed using Delaunay triangulation, and a texture model is generated by combining the original image texture.

3. The pose recognition method for any object using a monocular camera according to claim 1, characterized in that: In step S2, generating the image depth map includes: generating a relative depth map for each frame of the input image using a monocular image depth estimation network, performing scale calibration by combining known structures or calibration reference points in the selected frame, and back-projecting each pixel position to the three-dimensional spatial coordinate system through camera intrinsic parameters to complete the point cloud mapping from image to space for subsequent target position initialization.

4. The pose recognition method for any object using a monocular camera according to claim 1, characterized in that: In step S3, the extraction of the target image mask includes: using an image encoder based on a visual model to extract high-dimensional semantic features from the entire image; inputting foreground and background points obtained by user interaction to a prompt encoder to generate spatial guidance information; inputting the image encoding and prompt encoding together into a mask decoder; and outputting a target region mask map with consistent resolution. The mask map is directly associated with the image cropping region required for pose estimation in step S4.

5. The pose recognition method for any object using a monocular camera according to claim 1, characterized in that: In the attitude initialization process in step S4, the set of pixels corresponding to the depth map is extracted in the target area identified by the mask image, and the target depth position is obtained by median calculation. The initial translation vector of the target in the world coordinate system is obtained by combining the monocular camera intrinsic back projection. The initial estimation of the rotation matrix is ​​to uniformly sample multiple observation directions on the icosphere constructed with the target center as the center, forming a rotation hypothesis set as a candidate for attitude initialization.

6. The pose recognition method for any object using a monocular camera according to claim 5, characterized in that: Each set of initial pose parameters in the initial hypothesis set described in step S4 is applied to the target 3D model to perform image rendering, generating a corresponding simulated image. Then, the simulated image is compared with the texture of the masked cropping region in the real observed image to construct a pose loss function based on image error. The rotation and translation parameters are optimized by gradient backpropagation through an implicit neural network representation to obtain a converged high-confidence pose estimation set.

7. The pose recognition method for any object using a monocular camera according to claim 6, characterized in that: In the pose selection process in step S4, target image rendering is performed on each set of pose results output by pose correction, and its image semantic coding features are extracted and compared with the coding features of the input image mask region one by one. The pose set is sorted by the indicators of contour overlap, region alignment and texture matching, and finally the pose with the highest score is output as the current frame estimation result.

8. An application system for a pose recognition method for arbitrary objects using a monocular camera according to any one of claims 1-7, characterized in that, The system includes: The 3D model building module is used to perform the target 3D modeling operation in step S1; The depth map generation module is used to perform the image depth map construction operation in step S2; The image mask generation module is used to perform the target region mask extraction operation in step S3; The attitude estimation module includes an initialization submodule, a correction submodule, and a filtering submodule, which are used to execute the attitude estimation process in step S4; Each module in the system corresponds to a step in the method. The control program running the processor calls the modules to work together to complete the attitude recognition task for any target.

9. An application system for a pose recognition method for arbitrary objects using a monocular camera according to claim 8, characterized in that: The system completes the execution result caching of the 3D model construction module during the initialization phase. The depth map generation module and the image mask generation module support dynamic processing of real-time input image sequences. The pose estimation module optimizes multiple sets of parameters of the initial pose hypothesis set constructed in step S4 based on a parallel computing architecture and outputs high-confidence pose estimation results.

Images