Monocular Vision 6D Pose Measurement Method and Device for Segmented Zoom Scenes
Through the joint optimization method of the object detection network and sparse key point set, the robustness and accuracy of single-eye pose measurement in the zoom-distance scenario is solved, and efficient and accurate pose measurement effect is achieved.
Patent Information
- Application Number
- CN202510628763.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2025-07-25
- Estimated Expiration
- 2045-05-15
AI Technical Summary
The existing single-eye position measurement methods have problems of insufficient robustness and accuracy in zoom-distance scenarios, especially the low imaging resolution of fixed focal length cameras, complex calibration of parameters in zoom-distance cameras, and poor accuracy. The application range of existing methods is limited or cumbersome.
The target bounding box is identified by the target detection network, and the preset target 3D sparse key point set is used for key point recognition. The camera internal parameters and pose initial values are jointly optimized through perspective reprojection residual and single-stroke reprojection residual functions, and the camera internal parameters and pose initial values are combined with multi-view geometric constraint information.
The robustness and accuracy of monocular visual 6D posture measurement is improved, and efficient and accurate posture measurement is achieved in segmented zoom scenarios.
Smart Images
Figure CN120147429B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of vision technology, and particularly to a monocular vision 6D pose measurement method and device for segmented variable focal length scenarios. Background Art
[0002] Real-time, robust, and high-precision inter-platform monocular pose measurement is one of the core key technologies in fields such as aircraft automatic landing, autonomous driving, high-precision navigation of robots, and augmented reality and virtual reality. Due to complex environmental interference and application conditions, monocular pose measurement remains a challenging task.
[0003] Existing monocular pose measurement methods can be divided into cooperative methods and non-cooperative methods according to whether cooperative markers are used. Cooperative methods measure the pose by extracting cooperative markers arranged on the target. Such methods are simple and efficient, but their application scope is relatively limited. Non-cooperative methods only use the characteristics of the target itself to achieve pose estimation, and have advantages such as simplicity, low cost, and wide application scenarios. Non-cooperative methods can be further divided into traditional methods and deep learning-related methods. Traditional methods perform pose estimation based on feature points, templates, etc., and it is difficult to handle problems such as textureless objects and cluttered backgrounds. Benefiting from the powerful feature extraction and expression capabilities of neural networks, deep learning-based methods have achieved excellent performance in monocular pose estimation. Such methods can be further divided into direct methods and indirect methods. The direct method regresses the 6D pose end-to-end through the network, with a simple process, but the pose estimation accuracy is insufficient. The indirect method outputs the detected intermediate representation through the network, and combines the high-precision three-dimensional information of the target, and uses methods such as Iterative Closest Point (ICP) and Perspective-n-Point (PnP) to solve the pose.
[0004] Currently, visual pose measurement is mostly based on fixed focal length cameras, which have simple camera internal parameter calibration and convenient application. However, when a fixed focal length camera images a distant target, the short focal length results in low imaging resolution and large pose measurement errors. In contrast, variable focal length cameras can keep the target in high-resolution imaging all the time, improving the pose measurement accuracy, but their internal parameter calibration is complex. Some scholars use a look-up table to model the internal parameters of the zoom camera. Due to not considering the interaction between parameters, the calibration accuracy is poor, the process is cumbersome, and each camera needs to be processed one by one. Self-calibration methods based on absolute dual quadric surfaces, etc., determine the camera internal parameters by means of multiple uncalibrated images. Although flexible, their accuracy and robustness are insufficient, and they mostly focus on focal length calibration, with high requirements for the accuracy of the principal point. The active vision calibration method requires the camera to perform specific movements, which limits the application scope. In addition, some studies expand the PnP problem and solve the camera focal length and distortion coefficients in a single frame, but these methods assume that the camera principal point is at the center of the image, which does not conform to the actual situation. Summary of the Invention
[0005] Based on this, in view of the above technical problems, it is necessary to provide a monocular vision 6D pose measurement method and device for segmented variable focal length scenarios that can improve robustness and accuracy.
[0006] A monocular vision 6D pose measurement method for segmented variable focal length scenarios, the method is applied to a non-cooperative system, the system includes a tracking target and a tracking party that perform relative motion, and the method is implemented in the tracking party:
[0007] Obtain an image sequence set of the tracking target in the segmented variable focal length scenario, where the image sequence set includes multiple frames of target images sorted by time;
[0008] Use a target detection network to identify the target area in each frame of the target image to obtain a target bounding box, and use a preset target 3D sparse key point set to perform key point identification within the target bounding box to obtain the target key points in each frame of the image;
[0009] For each frame of the target image, if it is the first frame of the target image, obtain the camera internal parameters and the initial pose value of the first frame based on the initial value of the optical center. If it is a non-first frame of the target image, obtain the initial pose value of the current frame according to the optimized camera internal parameters of the previous frame of the image;
[0010] Use the target 3D sparse key point set and planar feature points to perform reprojection respectively to obtain three-dimensional key point reprojection and planar feature point reprojection. For each frame of the target image, use the corresponding target key points and the three-dimensional key point reprojection to construct a perspective reprojection residual function. At the same time, use the corresponding planar feature points in the current frame of the target image and the previous frame of the target image to construct a homography reprojection residual function;
[0011] Jointly optimize the camera internal parameters and the initial pose value of the current frame by minimizing the perspective reprojection residual function and the homography reprojection residual function to obtain the measurement values of the 6D pose in each frame of the target image.
[0012] In one embodiment, when constructing the target 3D sparse key point set:
[0013] Generate a target three-dimensional dense model based on the target entity;
[0014] Use a human pose estimation method to extract multiple target key points with significant structural features on the target three-dimensional dense model;
[0015] Generate the target 3D sparse key point set according to the multiple target key points.
[0016] In one embodiment, when using a target detection network to identify the target area in each frame of the target image to obtain a target bounding box:
[0017] Generate a first-frame target bounding box using the target detection network based on the first-frame target image;
[0018] Based on the first-frame target bounding box, use the STAPLE algorithm to detect the target bounding boxes in the subsequent-frame target images.
[0019] In one embodiment, the target detection network uses a YOLOX neural network.
[0020] In one embodiment, when acquiring each frame of target image in the image sequence set:
[0021] Real-time monitor the imaging size of the tracking target in each frame of target image. When it is detected that the imaging size exceeds a preset threshold, adjust the focal length of the imaging system.
[0022] In one embodiment, the perspective reprojection residual function is expressed as:
[0023] ;
[0024] In the above formula, represents the two-dimensional projection of the detected th predefined three-dimensional key point on the th image, represents the camera internal parameters to be optimized, represents the pose to be optimized of the th frame of target image, represents the homogeneous coordinates of the th predefined three-dimensional key point, and respectively represent the number of selected images and key points.
[0025] In one embodiment, the homography reprojection residual function is constructed by reprojecting the planar feature points of the previous frame of target image to the current frame of target image through a homography relationship.
[0026] In one embodiment, the homography reprojection residual function is expressed as:
[0027] ;
[0028] In the above formula, represents the number of feature point matches between the th frame and the th frame of target image, and respectively represent the th and the th feature points on the th frame of target image, , respectively represent the intrinsic camera parameters of the target images of the -th and -th frames. , respectively represent the rotation values of the poses of the target images of the -th and -th frames. , respectively represent the translation components of the target images of the -th and -th frames. and respectively represent the plane normal vector and the plane equation intercept in the object coordinate system.
[0029] In one embodiment, when jointly optimizing the intrinsic camera parameters and the initial pose values of the current frame by using the perspective reprojection residual function and the homography reprojection residual function:
[0030] If the intrinsic camera parameters of the current frame and the target image of the previous frame are the same in the homography reprojection residual function, the intrinsic camera parameters in the homography reprojection residual function are fixed, and only the pose is optimized;
[0031] If the intrinsic camera parameters of the current frame and the target image of the previous frame are different, the intrinsic camera parameters of the target image of the previous frame in the homography reprojection residual function are fixed, and only the intrinsic camera parameters and the pose of the target image of the current frame are optimized.
[0032] This application also provides a monocular vision 6D pose measurement device for a segmented variable focal length scenario. The device includes:
[0033] An image sequence set acquisition module, configured to acquire an image sequence set of a tracking target in a segmented variable focal length scenario, where the image sequence set includes multiple frames of target images sorted by time;
[0034] A target key point extraction module, configured to use a target detection network to identify the region where the target is located in each frame of the target image to obtain a target bounding box, and perform key point identification within the target bounding box by using a preset set of 3D sparse key points of the target to obtain the target key points in each frame of the image;
[0035] An intrinsic camera parameter and initial pose value obtaining module, configured to, for each frame of the target image, if it is the first frame of the target image, obtain the intrinsic camera parameters and the initial pose value of the first frame based on the initial optical center value, and if it is not the first frame of the target image, obtain the initial pose value of the current frame according to the optimized intrinsic camera parameters of the previous frame of the image;
[0036] A joint optimization module, which is used to respectively perform reprojection by using the target 3D sparse key point set and planar feature points to obtain 3D key point reprojection and planar feature point reprojection. For each frame of target image, a perspective reprojection residual function is constructed by using the corresponding target key points and the 3D key point reprojection. At the same time, a homography reprojection residual function is constructed by using the corresponding planar feature points in the current frame of target image and the previous frame of target image.
[0037] A pose measurement value obtaining module, which is used to jointly optimize the camera internal parameters and the initial pose values of the current frame by minimizing the perspective reprojection residual function and the homography reprojection residual function, so as to obtain the measurement values of the 6D poses in each frame of target image.
[0038] A computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0039] Obtain an image sequence set of the tracking target in a segmented variable focal length scene, where the image sequence set includes multiple frames of target images sorted by time;
[0040] Use a target detection network to identify the target area in each frame of target image to obtain a target bounding box, and use a preset target 3D sparse key point set to perform key point identification within the target bounding box to obtain the target key points in each frame of image;
[0041] For each frame of target image, if it is the first frame of target image, the camera internal parameters and the initial pose value of the first frame are obtained based on the initial optical center value. If it is a non-first frame of target image, the current frame pose initial value is obtained according to the optimized camera internal parameters of the previous frame of image;
[0042] Use the target 3D sparse key point set and planar feature points to respectively perform reprojection to obtain 3D key point reprojection and planar feature point reprojection. For each frame of target image, a perspective reprojection residual function is constructed by using the corresponding target key points and the 3D key point reprojection. At the same time, a homography reprojection residual function is constructed by using the corresponding planar feature points in the current frame of target image and the previous frame of target image;
[0043] Jointly optimize the camera internal parameters and the initial pose values of the current frame by minimizing the perspective reprojection residual function and the homography reprojection residual function to obtain the measurement values of the 6D poses in each frame of target image.
[0044] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0045] Obtain an image sequence set of the tracking target in a segmented variable focal length scenario, where the image sequence set includes multiple frames of target images sorted by time;
[0046] Use a target detection network to identify the region where the target is located in each frame of the target image to obtain a target bounding box, and use a preset set of 3D sparse key points of the target to perform key point identification within the target bounding box to obtain the target key points in each frame of the image;
[0047] For each frame of the target image, if it is the first frame of the target image, obtain the camera internal parameters and the initial pose value of the first frame based on the initial value of the optical center. If it is not the first frame of the target image, obtain the initial pose value of the current frame according to the optimized camera internal parameters of the previous frame;
[0048] Use the set of 3D sparse key points of the target and planar feature points to perform reprojection respectively to obtain the 3D key point reprojection and the planar feature point reprojection. For each frame of the target image, use the corresponding target key points and the 3D key point reprojection to construct a perspective reprojection residual function. At the same time, use the corresponding planar feature points in the current frame of the target image and the previous frame of the target image to construct a homography reprojection residual function;
[0049] Jointly optimize the camera internal parameters and the initial pose value of the current frame by minimizing the perspective reprojection residual function and the homography reprojection residual function to obtain the measured values of the 6D pose in each frame of the target image.
[0050] The above monocular vision 6D pose measurement method and device for a segmented variable focal length scenario first use a target detection network to identify the region where the target is located in each frame of the target image to obtain a target bounding box, and use a preset set of 3D sparse key points of the target to perform key point identification within the target bounding box to obtain the target key points in each frame of the image. For each frame of the target image, obtain the initial pose value of the current frame according to the optimized camera internal parameters of the previous frame. Use the set of 3D sparse key points of the target to perform perspective reprojection on the image to obtain the 3D key point reprojection. For each frame of the target image, use the corresponding target key points and the 3D key point reprojection to construct a perspective reprojection residual function. At the same time, use the matching feature points in the current frame of the target image and the previous frame of the target image to construct a homography reprojection residual function. Finally, jointly optimize the camera internal parameters and the initial pose value of the current frame by minimizing the perspective reprojection residual function and the homography reprojection residual function to obtain the measured values of the 6D pose in each frame of the target image. Using this method can effectively improve the robustness and the pose measurement accuracy. Description of the Drawings
[0051] Figure 1 It is a schematic flowchart of the monocular vision 6D pose measurement method for a segmented variable focal length scenario in an embodiment;
[0052] Figure 2 Schematic diagram of a target 3D sparse model in an embodiment, where Figure 2 (a) represents the actual target model, Figure 2 (b) represents the 3D model obtained by scanning the target in the left figure;
[0053] Figure 3 Schematic diagram of the relative pose measurement of the double-acting platform rendezvous visual guidance in an embodiment;
[0054] Figure 4 Schematic diagram of the framework of a monocular vision 6D pose measurement method for a segmented variable focal length scene in an embodiment;
[0055] Figure 5 Schematic diagram of the process of a monocular vision 6D pose measurement method for a segmented variable focal length scene in another embodiment;
[0056] Figure 6 Structural block diagram of a monocular vision 6D pose measurement device for a segmented variable focal length scene in an embodiment;
[0057] Figure 7 Internal structure diagram of a computer device in an embodiment. Detailed implementation manners
[0058] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0059] There are many problems in the existing monocular pose measurement and visual pose measurement technologies. For example, in monocular pose measurement, the application scope of cooperative methods is limited, non-cooperative traditional methods are difficult to handle scenes without texture and with cluttered backgrounds, the accuracy of deep learning direct methods is poor, and indirect methods rely on specific conditions. In the use of cameras for visual pose measurement, fixed focal length cameras have low imaging resolution and large measurement errors for distant targets, and the internal parameters of variable focal length cameras are complex to calibrate. Existing methods such as lookup table modeling, self-calibration, active vision calibration, and extended PnP problem solving have problems such as low calibration accuracy, poor robustness, limited application, and non-conformity to reality respectively.
[0060] In view of the above problems, in the present application, as Figure 1 shown, a monocular vision 6D pose measurement method for a segmented variable focal length scene is provided. This method is applied to a non-cooperative system, which includes a tracking target and a tracking party that are in relative motion. The method is implemented in the tracking party and specifically includes the following steps:
[0061] Step S100: Obtain an image sequence set of the tracking target in a segmented variable focal length scenario, where the image sequence set includes multiple frames of target images sorted by time.
[0062] Step S110: Use a target detection network to identify the region where the target is located in each frame of the target image, obtain a target bounding box, and perform key point identification within the target bounding box using a preset set of 3D sparse key points of the target to obtain the target key points in each frame of the image.
[0063] Step S120: For each frame of the target image, if it is the first frame of the target image, obtain the camera internal parameters and the initial pose value of the first frame based on the initial value of the optical center. If it is a non-first frame of the target image, obtain the initial pose value of the current frame according to the optimized camera internal parameters of the previous frame image.
[0064] Step S130: Use the set of 3D sparse key points of the target and planar feature points to perform reprojection respectively to obtain 3D key point reprojection and planar feature point reprojection. For each frame of the target image, use the corresponding target key points and the 3D key point reprojection to construct a perspective reprojection residual function. At the same time, use the corresponding planar feature points in the current frame target image and the previous frame target image to construct a homography reprojection residual function.
[0065] Step S140: Jointly optimize the camera internal parameters and the initial pose value of the current frame by minimizing the perspective reprojection residual function and the homography reprojection residual function to obtain the measured values of the 6D pose in each frame of the target image.
[0066] In this embodiment, the non-cooperative system includes a tracking target and a tracking party, and both of them are moving. This system can refer to a system with a ship as the tracking target and a drone as the tracking party, and its task is for the drone to land accurately at a preset position through the measured relative pose. It can also be in the city, with a moving vehicle as the tracking target and a drone as the tracking party, and the task is also for the drone to land on the top of the vehicle. This method is implemented on the tracking party, and the tracking target is photographed by a camera system set on the tracking party, and the relative pose is measured through the photographed images. In this article, an example of a ship as the tracking target and a drone as the tracking party is used for illustration.
[0067] In step S100, first, the tracking target is continuously photographed by a camera set on the drone, so as to obtain multiple frames of target images sorted by time. Among them, when each frame of the target image is photographed, the focal length of the camera changes, so that when the tracking target moves in the tracking direction, its target always maintains a high resolution on the image.
[0068] Specifically, when obtaining each frame of the target image in the image sequence set: the imaging size of the tracking target in each frame of the target image is monitored in real time. When it is detected that the imaging size exceeds the preset threshold, the focal length of the imaging system is adjusted. This enables the target imaging to be stably within the optimal size range, thereby ensuring the accuracy and stability of visual measurement.
[0069] In step S110, when constructing the target 3D sparse key point set: first, a target three-dimensional dense model is generated based on the target entity. Multiple target key points with significant structural features are extracted from the target three-dimensional dense model using the human pose estimation method. Finally, a target 3D sparse key point set is generated according to multiple target key points.
[0070] Considering that the target pose measurement method based on the three-dimensional dense model is usually slow and difficult to be directly used in subsequent optimization steps, in this embodiment, referring to the idea of setting key points of human body parts in the human pose estimation method, a reasonable object three-dimensional coordinate system is first established in the target three-dimensional dense model , and a sparse 3D key point set with significant structural features is proposed to represent the 3D sparse model of the target, as Figure 2 shown. Using to represent the predefined 3D sparse key point set, which is expressed as:
[0071] (1)
[0072] In formula (1), represents the number of three-dimensional key points used. To simulate the actual scene, the 3D sparse key point coordinates of the target are scaled to the actual size.
[0073] In step S110, when using the target detection network to identify the target area in each frame of the target image and obtain the target bounding box: the target detection network generates the first-frame target bounding box according to the first-frame target image, and then based on the first-frame target bounding box, the STAPLE algorithm is used to track the target bounding box in the subsequent frame target images.
[0074] In this embodiment, the target detection network uses the YOLOX neural network.
[0075] Specifically, after selecting the predefined 3D key points , the YOLOX algorithm is used to detect the target area. The YOLOX outputs the bounding box of the target in the image, denoted as . The target detection of YOLOX on a single-frame image takes a long time. Therefore, to improve the efficiency, after obtaining the target area in the first frame, the STAPLE algorithm is used to track the target in the subsequent input images. To keep the symbols unified, the target bounding box output by the STAPLE algorithm is also denoted as .
[0076] In this embodiment, to ensure the reliability of target tracking, the intersection ratio between the target detection box and the pose reprojection box is further calculated, expressed as:
[0077] (2)
[0078] In formula (2), if , it is considered that , and at this time the target tracking result is unreliable, and YOLOX is used again to detect the target.
[0079] In one embodiment, when using the preset 3D sparse key point set of the target to perform key point recognition within the target bounding box, the RTMPose algorithm is used to achieve efficient and accurate detection of 2D key points in the input image. The RTMPose algorithm performs key point detection as a classification task and has the characteristics of simplicity, high efficiency, and easy deployment. In fact, in this method, other various efficient and accurate key point detection methods can also be used, not limited to RTMPose.
[0080] Specifically, after the key point detection in the target image, the 3D-2D key point matching can actually be obtained, which can be applied to the subsequent pose initial value solution.
[0081] In step S120, when obtaining the camera internal parameters and the initial pose of each frame of the target image, considering that the zoom camera is usually set to a relatively small focal length before working, and the internal parameters of the camera at this focal length can be accurately calibrated in advance using the Zhang Zhengyou calibration method. When the system starts to work, the camera first takes an image at this focal length and calculates the current pose based on solving the PnP problem , and at this time the coordinates of the camera optical center in the object coordinate system are . Subsequently, the zoom controller is driven to zoom in and a high-resolution image of the target is taken.
[0082] Furthermore, for the first frame of the target image, the calculated camera optical center is substituted into the following formula and further transformed to obtain:
[0083] (3)
[0084] In formula (3), is the depth of the three-dimensional key point, is the homogeneous form of , and are the initial values of the camera internal parameters and the rotation amount for the first target image after zooming to be determined. is called the infinite homography. Further, writing Equation (3) in the form of a homogeneous linear equation system gives:
[0085] (4)
[0086] In Equation (4), is the column vector formed by arranging by rows. The elements of the infinite homography are obtained based on the SVD decomposition method under scale uncertainty. Then, the RQ decomposition is used to decompose into and , and the initial translation value .
[0087] Furthermore, for non-first-frame images, the optimized camera internal parameters of the previous frame are used as the initial internal parameters of the current frame, and the initial pose of the current frame is obtained by solving the PnP (Perspective-n-Point, PnP) problem.
[0088] Next, in steps S130 and S140, the initial camera internal parameters and the initial pose of the current frame image are optimized.
[0089] In this embodiment, a feature point detection algorithm is used to match the feature points in the adjacent image plane regions, and the perspective transformation reprojection relationship of the 3D key points and the homography transformation reprojection relationship between the planar feature points are established through the camera internal parameters and the pose. Using the multi-view geometric constraint information in the sequence images, the focal length, principal point, and relative pose that constitute the camera internal parameters are used as the parameters to be optimized, and the objective function is established by minimizing the 3D key point reprojection residual and the inter-frame homography transformation reprojection residual. By solving this optimization problem, the joint estimation of the camera internal parameters and the relative 6D pose is realized.
[0090] As Figure 3 shows, since the pose relationship between the monocular zoom camera and the tracking platform can be pre-calibrated and kept fixed, therefore, in this application, the pose relationship between the two moving platforms is substantially equivalent to the relative pose relationship between the camera and the target platform. Figure 3 In is the camera coordinate system, whose origin is located at the camera optical center, is the object coordinate system, whose coordinate origin is located on the target platform.
[0091] As Figure 3 shows, in this embodiment, is used to represent the rigid body transformation from the object coordinate system to the camera coordinate system, that is, the relative pose between the two coordinate systems, namely:
[0092] (5)
[0093] In formula (5), and respectively represent the special Euclidean group and the special orthogonal group, and are respectively the rotation and translation components of.
[0094] Furthermore, the intrinsic matrix of the zoom camera is expressed as:
[0095] (6)
[0096] In formula (6), and are respectively and the equivalent focal lengths in the directions, and
[0097] is the coordinate of the camera principal point. Since the 3D model of the tracking target is known, a vertex of the tracking target is expressed in homogeneous form
[0098] (7)
[0099] In formula (7), is the 2D image point, .
[0100] Furthermore, the homography matrix describes the mapping relationship between two planes. If the feature points in the scene all lie on the same plane, then motion estimation can be performed through homography. Assume that the target plane in the object coordinate system is , and its plane equation is:
[0101] (8)
[0102] In formula (8), is the plane normal vector. Let a pair of adjacent frame images captured by the zoom camera be and , and there is a pair of matching feature points and on the plane. Based on the camera intrinsics and the 6D pose representation , the homography reprojection relationship can be further established:
[0103] (9)
[0104] In this embodiment, when performing joint optimization, it is traced back from the current frame to construct a sliding window for frames. By comparing the re-projection of three-dimensional key points with the two-dimensional key point detection values using the intrinsic camera parameters and the initial 6D pose values of the images within the sliding window, the established perspective re-projection residual function is expressed as:
[0105] (10)
[0106] In formula (10), represents the two-dimensional projection of the detected th predefined three-dimensional key point on the th image, represents the intrinsic camera parameters to be optimized, represents the pose to be optimized of the target image in the th frame, represents the homogeneous coordinates of the th predefined three-dimensional key point, and respectively represent the number of selected images and key points.
[0107] Since the residual function represented by formula (10) is constructed only based on the relative pose between the object system and the camera system and does not consider the relative pose constraint between images. Therefore, in this embodiment, the target plane area in the image is located and ORB (Oriented FAST and Rotated BRIEF) feature points are extracted on the plane area, and further the ORB feature points between adjacent frame images are matched.
[0108] In this embodiment, the homography re-projection residual function is constructed by re-projecting the plane feature points of the previous frame target image onto the current frame target image through a homography relationship.
[0109] Specifically, within the above-constructed sliding window, the plane feature points of the th frame image are re-projected onto the th frame image through the homography relationship shown in formula (9), and the established homography re-projection residual function is expressed as:
[0110] (11)
[0111] In formula (11), represents the number of feature point matches between the target images of the th frame and the th frame, and respectively represent the th and th feature points on the target images of the th frame, , respectively represent the camera internal parameters of the target images of the -th frame and the -th frame. , respectively represent the rotation values of the poses of the target images of the -th frame and the -th frame. , respectively represent the translation components of the target images of the -th frame and the -th frame. and respectively represent the plane normal vector and the plane equation intercept in the object coordinate system.
[0112] In this embodiment, combining formula (10) and formula (11), the camera internal parameters and the initial 6D pose values within the above sliding window are used as the parameters to be optimized, and an optimization objective function is established by minimizing the reprojection residual, expressed as:
[0113] (12)
[0114] In formula (12), only the relative pose between frames is constrained, which is an underdetermined equation. There is redundancy in the degrees of freedom in the optimization problem regarding the camera internal parameters and the absolute pose, resulting in an infinite number of solution spaces. To limit the randomness of the solution, when jointly optimizing the camera internal parameters and the initial pose values of the current frame using the perspective reprojection residual function and the homography reprojection residual function: If the camera internal parameters of the target images of the current frame and the previous frame are the same in the homography reprojection residual function, then the camera internal parameters in the homography reprojection residual function are fixed, and only the pose is optimized, that is, if , then in is fixed, and only the pose is optimized. If the camera internal parameters of the target images of the current frame and the previous frame are different, then the camera internal parameters of the target image of the previous frame in the homography reprojection residual function are fixed, and the camera internal parameters and the pose of the target image of the current frame are optimized, that is, if , then in is fixed, and only and the 6D pose are optimized.
[0115] As shown in Figure 4 and Figure 5 , it is the implementation method framework diagram and the algorithm flow chart of this method.
[0116] In the above monocular vision 6D pose measurement method for the segmented variable focal length scenario, a target detection model is used to locate the imaging position of the target platform. Then, a key point detection algorithm commonly used in human pose estimation is used to detect the key points of the target platform. Secondly, the infinite homography matrix is solved through the estimated optical center value and decomposed to obtain the initial camera internal parameters and the initial pose values. Using the multi-view geometric constraint information in the sequence images, the focal length, principal point, and relative pose that constitute the camera internal parameters are used as parameters to be optimized, and an optimization objective function is established by minimizing the perspective reprojection error of the 3D key points and the homography transformation reprojection residuals of the feature points between frames. By solving this optimization problem, the joint estimation of the camera internal parameters and the relative 6D pose is realized. This method realizes the robust, efficient, and high-precision relative pose measurement between platforms in monocular vision guidance. At the same time, the camera internal parameters and 6D pose can be optimized in real time. In this method, the absolute constraint based on the perspective reprojection residuals of the 3D key points and the relative constraint based on the homography reprojection residuals of the 2D feature points are combined to establish an objective function for minimizing the reprojection residuals, and the camera internal parameters and 6D pose are jointly optimized.
[0117] It should be understood that although Figure 1 the steps in the flowchart of Figure 1 are shown sequentially in the order indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover,
[0118] In one embodiment, as Figure 6 shown, a monocular vision 6D pose measurement device for the segmented variable focal length scenario is provided, including: an image sequence set acquisition module 200, a target key point extraction module 210, a camera internal parameter and initial pose value obtaining module 220, a joint optimization module 230, and a pose measurement value obtaining module 240, where:
[0119] The image sequence set acquisition module 200 is configured to acquire an image sequence set of the tracking target in the segmented variable focal length scenario, and the image sequence set includes multiple frames of target images sorted by time;
[0120] The target key point extraction module 210 is configured to use a target detection network to identify the region where the target is located in each frame of target image, obtain a target bounding box, and perform key point identification within the target bounding box using a preset 3D sparse key point set of the target, so as to obtain the target key points in each frame of image;
[0121] The camera internal parameter and pose initial value obtaining module 220 is configured to, for each frame of target image, if it is the first frame of target image, obtain the camera internal parameter and the first frame pose initial value based on the initial value of the optical center, and if it is a non-first frame of target image, obtain the current frame pose initial value according to the optimized camera internal parameter of the previous frame of image;
[0122] The joint optimization module 230 is configured to use the 3D sparse key point set of the target and the planar feature points to respectively perform reprojection to obtain the 3D key point reprojection and the planar feature point reprojection. For each frame of target image, a perspective reprojection residual function is constructed using the corresponding target key points and the 3D key point reprojection, and at the same time, a homography reprojection residual function is constructed using the corresponding planar feature points in the current frame of target image and the previous frame of target image;
[0123] The pose measurement value obtaining module 240 is configured to jointly optimize the camera internal parameter and the pose initial value of the current frame by minimizing the perspective reprojection residual function and the homography reprojection residual function, so as to obtain the measurement value of the 6D pose in each frame of target image.
[0124] For the specific limitations of the monocular vision 6D pose measurement device for a segmented variable focal length scene, reference can be made to the limitations of the monocular vision 6D pose measurement method for a segmented variable focal length scene in the above text, which will not be elaborated here. Each module in the above monocular vision 6D pose measurement device for a segmented variable focal length scene can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.
[0125] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 7As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it realizes a monocular vision 6D pose measurement method for a segmented variable focal length scenario. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the outer shell of the computer device, or an external keyboard, a touchpad, or a mouse, etc.
[0126] Those skilled in the art can understand that Figure 7 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.
[0127] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented:
[0128] Obtain an image sequence set of the tracking target in a segmented variable focal length scenario, where the image sequence set includes multiple frames of target images sorted by time;
[0129] Use a target detection network to identify the region where the target is located in each frame of the target image to obtain a target bounding box, and use a preset set of 3D sparse key points of the target to perform key point recognition within the target bounding box to obtain the target key points in each frame of the image;
[0130] For each frame of the target image, if it is the first frame of the target image, obtain the camera internal parameters and the initial pose value of the first frame based on the initial value of the optical center. If it is not the first frame of the target image, obtain the initial pose value of the current frame according to the optimized camera internal parameters of the previous frame of the image;
[0131] Use the set of 3D sparse key points of the target and planar feature points to perform reprojection respectively to obtain 3D key point reprojection and planar feature point reprojection. For each frame of the target image, use the corresponding target key points and the 3D key point reprojection to construct a perspective reprojection residual function. At the same time, use the corresponding planar feature points in the current frame of the target image and the previous frame of the target image to construct a homography reprojection residual function;
[0132] Jointly optimize the camera internal parameters and the initial pose of the current frame by minimizing the perspective reprojection residual function and the homography reprojection residual function, and obtain the measurement values of the 6D pose in each frame of the target image.
[0133] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:
[0134] Obtain an image sequence set of the tracking target in a segmented variable focal length scene, where the image sequence set includes multiple frames of target images sorted by time;
[0135] Use a target detection network to identify the region where the target is located in each frame of the target image, obtain the target bounding box, and use a preset set of 3D sparse key points of the target to perform key point identification within the target bounding box to obtain the target key points in each frame of the image;
[0136] For each frame of the target image, if it is the first frame of the target image, obtain the camera internal parameters and the initial pose of the first frame based on the initial value of the optical center. If it is a non-first frame of the target image, obtain the initial pose of the current frame according to the optimized camera internal parameters of the previous frame image;
[0137] Use the set of 3D sparse key points of the target and the planar feature points to perform reprojection respectively to obtain the 3D key point reprojection and the planar feature point reprojection. For each frame of the target image, use the corresponding target key points and the 3D key point reprojection to construct a perspective reprojection residual function. At the same time, use the corresponding planar feature points in the current frame of the target image and the previous frame of the target image to construct a homography reprojection residual function;
[0138] Jointly optimize the camera internal parameters and the initial pose of the current frame by minimizing the perspective reprojection residual function and the homography reprojection residual function, and obtain the measurement values of the 6D pose in each frame of the target image.
[0139] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above various methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.
[0140] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.
[0141] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.
Claims
1. A monocular vision 6D pose measurement method for segmented variable focal length scenarios, characterized in that, The method is applied to a non - cooperative system, which includes a tracking target and a tracking party in relative motion. The method is implemented in the tracking party: Obtain an image sequence set of the tracking target in a segmented variable - focal - length scenario. The image sequence set includes multiple target images sorted by time. Meanwhile, monitor the imaging size of the tracking target in each frame of the target image in real - time. When it is detected that the imaging size exceeds a preset threshold, adjust the focal length of the imaging system; Use a target detection network to identify the target area in each frame of the target image to obtain a target bounding box, and use a preset set of 3D sparse key points of the target to perform key - point recognition within the target bounding box to obtain the target key points in each frame of the image; For each frame of the target image, if it is the first - frame target image, obtain the initial camera internal parameters and the initial pose of the first frame based on the initial value of the optical center. If it is a non - first - frame target image, obtain the initial pose of the current frame according to the optimized camera internal parameters of the previous frame image; Use the set of 3D sparse key points of the target and planar feature points to perform reprojection respectively to obtain 3D key - point reprojection and planar feature - point reprojection. For each frame of the target image, use the corresponding target key points and the 3D key - point reprojection to construct a perspective reprojection residual function. Meanwhile, reproject the planar feature points of the previous frame of the target image to the current frame of the target image through a homography relationship to construct a homography reprojection residual function. The homography reprojection residual function is expressed as: ； In the above formula, represents the number of feature point matches between the -th and -th frame target images, , respectively represent the -th and -th feature points on the -th frame target image, , respectively represent the internal camera parameters of the -th and -th frame target images, , respectively represent the rotation values of the poses of the -th and -th frame target images, , respectively represent the translation components of the -th and -th frame target images, and respectively represent the plane normal vector and the plane equation intercept in the object coordinate system; Jointly optimize the camera internal parameters and the initial pose of the current frame by minimizing the perspective reprojection residual function and the homography reprojection residual function to obtain the measurement values of the 6D pose in each frame of the target image.
2. The monocular vision 6D pose measurement method for segmented variable focal length scenarios according to claim 1, characterized in that, When constructing the set of 3D sparse key points of the target: Generate a target three - dimensional dense model based on the target entity; Use a human - pose estimation method to extract multiple target key points with significant structural features on the target three - dimensional dense model; Generate the set of 3D sparse key points of the target according to the multiple target key points.
3. The monocular vision 6D pose measurement method for segmented variable focal length scenarios according to claim 2, wherein When using a target detection network to identify the target area in each frame of the target image to obtain a target bounding box: Use the target detection network to generate a first - frame target bounding box according to the first - frame target image; Based on the first - frame target bounding box, use the STAPLE algorithm to detect the target bounding box in the subsequent frames of the target image.
4. The monocular vision 6D pose measurement method for segmented variable focal length scenarios according to claim 3, characterized in that, The target detection network uses the YOLOX neural network.
5. The monocular vision 6D pose measurement method for segmented variable focal length scenarios according to claim 4, characterized in that The perspective reprojection residual function is expressed as: ； In the above formula, represents the two-dimensional projection of the th detected predefined three-dimensional key point on the th image, represents the camera internal parameters to be optimized, represents the pose to be optimized of the th frame of the target image, represents the homogeneous coordinates of the th predefined three-dimensional key point, and respectively represent the number of selected images and key points.
6. The monocular vision 6D pose measurement method for segmented variable focal length scenes according to claim 5, characterized in that, When jointly optimizing the camera internal parameters and the initial pose of the current frame by using the perspective reprojection residual function and the homography reprojection residual function: In the homography reprojection residual function, if the camera internal parameters of the current frame and the previous frame of the target image are the same, fix the camera internal parameters in the homography reprojection residual function and only optimize the pose; If the camera internal parameters of the current frame and the previous frame of the target image are different, fix the camera internal parameters of the previous frame of the target image in the homography reprojection residual function and only optimize the camera internal parameters and the pose of the current frame of the target image.
7. A monocular vision 6D pose measurement device for segmented variable focal length scenarios, characterized in that, The device includes: An image sequence set acquisition module, which is used to acquire an image sequence set of a tracking target in a segmented variable focal length scene. The image sequence set includes multiple frames of target images sorted by time. At the same time, the imaging size of the tracking target in each frame of the target image is monitored in real time. When it is detected that the imaging size exceeds a preset threshold, the focal length of the imaging system is adjusted; A target key point extraction module, which is used to identify the area where the target is located in each frame of the target image by using a target detection network to obtain a target bounding box, and perform key point identification within the target bounding box by using a preset target 3D sparse key point set to obtain the target key points in each frame of the image; A camera internal parameter and pose initial value obtaining module, which is used for each frame of the target image. If it is the first frame of the target image, the camera internal parameter and the first frame pose initial value are obtained based on the initial value of the optical center. If it is a non-first frame of the target image, the current frame pose initial value is obtained according to the optimized camera internal parameter of the previous frame image; A joint optimization module, which is used to respectively perform reprojection by using the target 3D sparse key point set and the planar feature points to obtain the three-dimensional key point reprojection and the planar feature point reprojection. For each frame of the target image, a perspective reprojection residual function is constructed by using the corresponding target key points and the three-dimensional key point reprojection. At the same time, the planar feature points of the previous frame of the target image are reprojected onto the current frame of the target image through a homography relationship to construct a homography reprojection residual function. The homography reprojection residual function is expressed as: ； In the above formula, represents the number of feature point matches between the -th and the -th frame target images, , respectively represent the -th and the -th feature points on the -th frame target image, , respectively represent the internal camera parameters of the -th and the -th frame target images, , respectively represent the rotation values of the poses of the -th and the -th frame target images, , respectively represent the translation components of the -th and the -th frame target images, and respectively represent the plane normal vector and the plane equation intercept in the object coordinate system; A pose measurement value obtaining module, which is used to jointly optimize the camera internal parameter and the pose initial value of the current frame by minimizing the perspective reprojection residual function and the homography reprojection residual function to obtain the measurement value of the 6D pose in each frame of the target image.