A light field camera based target object 6D pose detection method

Abstract

The application discloses a target object 6D pose detection method based on a light field camera, and comprises the following steps: 1, collecting light field original images of a target object under N different poses and establishing an offline template library; 2, performing feature detection and matching on a scene all-focus image and a template all-focus image, mapping and intercepting window coordinates to obtain a target region light field image and a target region all-focus image; 3, performing parallax estimation on the target region light field image, guiding target background separation by using a parallax map obtained, and then fusing edge detection results of the target region all-focus image to obtain a target edge parallax map; and 4, obtaining a target edge point cloud and performing registration processing on the target edge point cloud and a template point cloud to obtain the 6D pose of the target object. The application can effectively solve the interference of a non-target region in target pose estimation, overcome the target blur problem and monocular scale uncertainty problem caused by defocus of a camera, and improve the 6D pose detection precision of the target object.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a 6D pose detection method for target objects based on a light field camera. Background Technology

[0002] 6D pose estimation of target objects, as one of the key research directions in computer vision technology, aims to solve the rigid transformation that occurs between the body coordinate system and the camera coordinate system of a 3D target object in a scene, namely the rotation matrix R and the translation t. It has wide applications in fields such as industrial robotic arm control, on-orbit space services, autonomous driving, and augmented reality. Therefore, vision-based 6D pose detection technology for target objects has significant research value and importance.

[0003] Currently, vision-based 6D pose detection of target objects mainly falls into three categories: monocular, binocular, and depth camera methods. Monocular vision-based pose estimation methods primarily extract salient feature points from the image, construct a highly representative descriptor, and obtain the 3D coordinates of the feature points in the object coordinate system and the corresponding 2D pixel coordinates in the image pixel coordinate system through feature matching. Then, perspective projection is used to calculate the 6D pose of the target object. Binocular vision-based pose estimation methods primarily use cameras at different left and right positions to simultaneously acquire two images of the object. Feature points are extracted from the left and right images, and matching point pairs are established through stereo matching. Depth information is calculated using the disparity between the matching point pairs in different images, thereby determining the 3D coordinates of the matching point pairs on the target object in the camera's coordinate system. Finally, the pose is determined based on point cloud registration. Depth camera-based pose estimation methods can directly acquire the depth information of the target object using a depth sensor, thereby calculating the 3D coordinates of the target object, and finally determining the pose using point cloud registration.

[0004] The above technical methods have the following problems in practical implementation:

[0005] Monocular vision-based pose estimation methods, lacking depth information due to their limited 2D image capabilities, cannot effectively handle objects with varying scales and are susceptible to background interference. Binocular vision-based methods rely on the spatial positions of feature points, but these positions are affected by numerous factors, including model errors, image noise, and quantization errors. This results in high computational complexity for depth information recovery, susceptibility to noise, and complex camera configuration and calibration. Depth camera-based pose estimation methods are limited by the camera's inherent characteristics: short measurement distances, susceptibility to strong light, and high energy consumption, making them unsuitable for moving scenes. Furthermore, current vision-based 6D object pose detection methods often suffer from target blurring because the camera cannot simultaneously focus on objects at different depth distances. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a 6D pose detection method for target objects based on a light field camera. This method can effectively solve the interference from non-target regions in target pose estimation, overcome the target blurring problem caused by camera defocusing and the monocular scale uncertainty problem, thereby improving the 6D pose detection accuracy of target objects.

[0007] The present invention adopts the following technical solution to solve the technical problem:

[0008] The present invention provides a 6D pose detection method for a target object based on a light field camera, characterized by the following steps:

[0009] Step 1: Use a light field camera to acquire original light field images of the target object in N different poses;

[0010] The original light field image is refocused to obtain N fully focused images in different poses;

[0011] The true pose of the target object in N different poses is obtained using the ArUco marker plate in the full-focus image;

[0012] The original light field image is subjected to disparity estimation to obtain N disparity images under different poses. The edge detection results of the fully focused image and the intrinsic parameters of the light field camera are combined to obtain N edge point clouds under different poses.

[0013] An offline template library is established based on the full-focus image, ground truth pose, and edge point cloud.

[0014] Step 2: Use a light field camera to acquire the original light field image of the scene where the target object is located, and obtain the scene's fully focused image I through refocusing calculation.scene ;

[0015] The SURF algorithm was used to process the scene's fully focused image I. scene By performing feature detection on any full-focus image from the offline template library, a scene full-focus image I is obtained. scene The feature point set and the feature point set of the full-focus image in the offline template library are used. Fast approximate nearest neighbor matching is applied to the two feature point sets, and the full-focus image in the offline template library that satisfies the matching relationship is found according to the set threshold for the number of matching points. This full-focus image is denoted as the template full-focus image I. model And obtain n matching point pairs λ1,λ2,...,λ n Where λ n This represents the nth matching pair;

[0016] Given n matching points λ1, λ2, ..., λ n Solving for the template total focus image I model With scene full-focus image I scene The homography matrix between them is used to fully focus the template image I. model The four vertex pixel coordinates are mapped to the scene's fully focused image I. scene In the middle, and obtain the scene fully focused image I scene The cropping window coordinates (x) min ,y min ),(x min ,y max ),(x max ,y min ),(x max ,y max Using the cropping window coordinates, the original light field image and the fully focused image of the scene containing the target object are captured separately. scene By cropping, the light field image I of the target area is obtained. LF and target region full-focus image I obj ;

[0017] Step 3: Analyze the light field image I of the target region. LF Disparity estimation is performed to obtain disparity map I. disp and the disparity map I disp The disparity values ​​d1, d2, ..., d of all pixels in the array. i Histogram statistics are performed to obtain the disparity distribution histogram, where d i Represents the disparity value of the i-th pixel; i represents the disparity map I. disp The total number of pixels in the middle, and i = (x max -x min +1)×(y max -y min+1);

[0018] Given n matching points λ1, λ2, ..., λ n In scene full-focus image I scene The pixel coordinates in the image are (x1, y1), (x2, y2), ..., (x n ,y n Transformed to the disparity map I disp The corresponding pixel coordinates (x1-x) min ,y1-y min ),(x2-x min ,y2-y min ),...,(x n -x min ,y n -y min ), where (x n ,y n ) represents the nth matching point pair λ n In scene full-focus image I scene Pixel coordinates in;

[0019] According to the disparity map I disp The pixel coordinates (x1-x) of n matching point pairs min ,y1-y min ),(x2-x min ,y2-y min ),...,(x n -x min ,y n -y min ), and the disparity value distribution of n matching point pairs in the disparity distribution histogram is obtained as [d p1 ,d q1 ],[d p2 ,d q2 ],...,[d pk ,d qk ], where [d pk ,d qk [] represents the distribution of the k-th disparity value, k≤n;

[0020] Find the disparity distribution [d] p1 ,d q1 ],[d p2 ,d q2 ],...,[d pk ,d qk The distribution of disparity values ​​containing the largest number of matching point pairs [d] pmax ,d qmax ], max∈[1,k]; thus, the disparity range function of the target object is constructed using equation (1):

[0021]

[0022] In equation (1), d bin The range width of the distribution for each disparity value, [d] p ,d q [] represents the disparity distribution of the target object; Δ represents a constant;

[0023] The disparity map I disp The center is located in the disparity value distribution [d] p ,d q All disparity values ​​outside the region are set to 0, thus setting the disparity map I... disp The target object in the image is separated from the background, and the resulting disparity map I' is obtained. disp ;

[0024] Fully focused image I of the target region obj Perform edge detection to obtain the target edge image I edge ;

[0025] The disparity map I' disp and target edge image I edge Pixel-by-pixel fusion is performed to obtain the target edge disparity map I. disp-edge ;

[0026] Step 4: Combine camera intrinsic parameters to obtain the target edge disparity map I. disp-edge Target edge point cloud P scene and the template full-focus image I model The corresponding template edge point cloud P model Registration is performed to obtain the rotation matrix R and translation t, which are then used as the 6D pose of the target object.

[0027] The characteristic of the 6D pose detection method for target objects based on a light field camera described in this invention is that, in step 3, the target edge disparity map I is obtained using equation (2). disp-edge The disparity value I corresponding to any pixel coordinate (x, y) in the image. disp-edge (x, y):

[0028]

[0029] In equation (2), x∈[0,x] max -x min ],y∈[0,y max -y min ], I' disp (x, y) represents the disparity value corresponding to the pixel coordinates (x, y) in the disparity map after the target object and background are separated. edge(x, y) represents the pixel value corresponding to the pixel coordinates (x, y) in the target edge image.

[0030] Compared with existing technologies, the beneficial effects of this invention are reflected in:

[0031] 1. This invention effectively reduces interference from non-target areas and improves the pose detection accuracy of target objects by capturing the target area of ​​the scene and using a disparity map to guide the separation of the target object and the background.

[0032] 2. This invention can effectively acquire clear images of target objects at any depth and distance in a scene through light field full-focusing imaging, solving the problem of target blurring caused by defocusing in ordinary cameras.

[0033] 3. The light field parallax estimation method used in this invention can effectively overcome the shortcomings of indistinct feature points and large noise errors when recovering depth using binocular stereo vision, and obtain accurate depth information.

[0034] 4. The light field camera used in this invention is small in size and can record the position and angle information of the light field simultaneously in a single shot. The scene depth information can be recovered using a single frame of light field image. Compared with depth cameras, it consumes less power, is less affected by strong light, and is suitable for moving scenes. Attached Figure Description

[0035] Figure 1 The diagram shown is a schematic representation of a practical application scenario of the present invention;

[0036] Figure 2 This is a flowchart of a 6D pose detection method for a target object based on a light field camera according to the present invention. Detailed Implementation

[0037] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art.

[0038] In this embodiment, a target 6D pose detection method based on a light field camera is applied to, for example... Figure 1 The scene shown includes a light field camera 1, a camera bracket 2, a scene 3, a target object 4, a PC 5, and an offline template library 6. The target object 4 only needs to be within the shooting range of the light field camera 1.

[0039] This 6D pose detection method has the advantages of small equipment footprint, simple working mode, and low energy consumption. (Refer to...) Figure 2 Specifically, it is done in the following steps:

[0040] Step 1: Use light field camera 1 to acquire raw light field images of the target object 4 containing the ArUco marker from N different angles and distances;

[0041] The original light field image is refocused to generate a focus stack image, and the gradient value of each layer of the focus stack image is calculated. Then, the multi-focus image is fused based on the maximum gradient value index of each pixel coordinate, thereby obtaining N fully focused images under different poses.

[0042] The ground truth pose of the target object 4 in N different poses is obtained by using the ArUco marker in the full-focus image. Then, the region where the target object is located in the full-focus image and the original light field image is cropped. The parallax is estimated by the sub-pixel multi-view stereo matching algorithm based on the phase shift theorem in the original light field image of the cropped target object. The parallax images in N different poses are obtained. The edge detection results of the full-focus image of the cropped target object and the intrinsic parameters of the light field camera 1 are combined to obtain the edge point cloud. Thus, an offline template library 6 containing full-focus image, template edge point cloud and ground truth pose is established.

[0043] Step 2: Use light field camera 1 to acquire the original light field image of the scene where the target object 4 is located. Refocus the original light field image of the scene to generate a focus stack image, and calculate the gradient value of each layer of the focus stack image. Based on the maximum gradient value index of each pixel coordinate, perform multi-focus image fusion to obtain the scene fully focused image I. scene ;

[0044] The SURF algorithm was used to process the scene's fully focused image I. scene By performing feature detection on any full-focus image from the offline template library 6, a full-focus image of the scene, I, is obtained. scene The feature point set and the feature point set of the full-focus image in the offline template library are used. Fast approximate nearest neighbor matching is applied to the two feature point sets, and the full-focus image in the offline template library that satisfies the matching relationship is found according to the set threshold for the number of matching points. This full-focus image is denoted as the template full-focus image I. model And obtain n matching point pairs λ1,λ2,...,λ n , where λ n This represents the nth matching pair; in this embodiment, the threshold is set to 10.

[0045] Given n matching points λ1, λ2, ..., λ n Solving for the template total focus image I model With scene full-focus image I scene The homography matrix between them is used to fully focus the template image I. model The four vertex pixel coordinates are mapped to the scene's fully focused image I. scene In the middle, and obtain the scene fully focused image I scene The cropping window coordinates (x) min ,y min ),(x min,y max ),(x max ,y min ),(x max ,y max Using the cropping window coordinates, the original light field image and the fully focused image of the scene containing the target object are captured separately. scene By cropping, the light field image I of the target area is obtained. LF and target region full-focus image I obj ;

[0046] Step 3: Image I of the light field of the target region LF A sub-pixel multi-view stereo matching algorithm based on the phase shift theorem is used for disparity estimation, and disparity map I is used. disp Guided target-background separation, then fusion of the target region's fully focused image I obj The edge detection results are used to obtain the target edge disparity map I. disp-edge ;

[0047] Step 3.1, for disparity map I disp The disparity values ​​d1, d2, ..., d of all pixels in the array. i Histogram statistics are performed to obtain the disparity distribution histogram, where d i Represents the disparity value of the i-th pixel; i represents the disparity map I. disp The total number of pixels in the middle, and i = (x max -x min +1)×(y max -y min +1);

[0048] Step 3.2: Match the n matching points λ1, λ2, ..., λ n In scene full-focus image I scene The pixel coordinates in the image are (x1, y1), (x2, y2), ..., (x n ,y n Transform to disparity map I disp The corresponding pixel coordinates (x1-x) min ,y1-y min ),(x2-x min ,y2-y min ),...,(x n -x min ,y n -y min ), where (x n ,y n ) represents the nth matching point pair λ n In scene full-focus image I scene Pixel coordinates in;

[0049] According to disparity map I disp The pixel coordinates (x1-x) of n matching point pairs min ,y1-y min ),(x2-x min ,y2-y min ),...,(x n -x min ,y n -y min ), and the disparity value distribution of n matching point pairs in the disparity distribution histogram is obtained as [d p1 ,d q1 ],[d p2 ,d q2 ],...,[d pk ,d qk ], where [d pk ,d qk [] represents the distribution of the k-th disparity value, k≤n;

[0050] Find the disparity distribution [d] p1 ,d q1 ],[d p2 ,d q2 ],...,[d pk ,d qk The distribution of disparity values ​​containing the largest number of matching point pairs [d] pmax ,d qmax ], max∈[1,k]; thus, the disparity range function of the target object is constructed using equation (1):

[0051]

[0052] In equation (1), d bin The range width of the distribution for each disparity value, [d] p ,d q [] represents the disparity distribution of the target object; Δ represents a constant, set to 3;

[0053] Step 3.3: Assign disparity map I disp The middle is located in the disparity value distribution [d p ,d q All disparity values ​​outside the region are set to 0, thus setting the disparity map I... disp The target object in the image is separated from the background, and the resulting disparity map I' is obtained. disp ;

[0054] Step 3.4: Fully focus on the target region image I obj Perform edge detection to obtain the target edge image I edge ;

[0055] Step 3.5: Assign disparity map I' disp and target edge image I edge Pixel-by-pixel fusion is performed to obtain the target edge disparity map I. disp-edge Specifically, the target edge disparity map I is obtained using equation (2). disp-edge The disparity value I corresponding to any pixel coordinate (x, y) in the image. disp-edge (x, y):

[0056]

[0057] In equation (2), x∈[0,x] max -x min ],y∈[0,y max -y min ], I' disp (x, y) represents the disparity value corresponding to the pixel coordinates (x, y) in the disparity map after the target object and background are separated. edge (x, y) represents the pixel value corresponding to the pixel coordinates (x, y) in the target edge image.

[0058] Step 4: Combine camera intrinsic parameters to obtain the target edge disparity map I. disp-edge Target edge point cloud P scene and the template full-focus image I model The corresponding template edge point cloud P model Registration is performed, and the relative pose is obtained using the iterative nearest-point algorithm, then combined with the template fully focused image I. model The corresponding true pose is used to obtain the rotation matrix R and translation t of the current camera coordinate system relative to the body coordinate system, and is used as the 6D pose of the target object.

Claims

1. A method for 6D pose detection of a target object based on a light field camera, characterized in that: Follow these steps: Step 1: Use a light field camera to acquire original light field images of the target object in N different poses; The original light field image is refocused to obtain N fully focused images in different poses; The true pose of the target object in N different poses is obtained using the ArUco marker plate in the full-focus image; The original light field image is subjected to disparity estimation to obtain N disparity images under different poses. The edge detection results of the fully focused image and the intrinsic parameters of the light field camera are combined to obtain N edge point clouds under different poses. An offline template library is established based on the full-focus image, ground truth pose, and edge point cloud. Step 2: Use a light field camera to acquire the original light field image of the scene where the target object is located, and obtain the scene's fully focused image I through refocusing calculation. scene ; The SURF algorithm was used to process the scene's fully focused image I. scene By performing feature detection on any full-focus image from the offline template library, a scene full-focus image I is obtained. scene The feature point set and the feature point set of the full-focus image in the offline template library are used. Fast approximate nearest neighbor matching is applied to the two feature point sets, and the full-focus image in the offline template library that satisfies the matching relationship is found according to the set threshold for the number of matching points. This full-focus image is denoted as the template full-focus image I. model And obtain n matching point pairs λ1,λ2,...,λ n , where λ n This represents the nth matching pair; Given n matching points λ1, λ2, ..., λ n Solving for the template total focus image I model With scene full-focus image I scene The homography matrix between them is used to fully focus the template image I. model The four vertex pixel coordinates are mapped to the scene's fully focused image I. scene In the middle, and obtain the scene fully focused image I scene The cropping window coordinates (x) min ,y min ),(x min ,y max ),(x max ,y min ),(x max ,y max Using the cropping window coordinates, the original light field image and the fully focused image of the scene containing the target object are captured separately. scene By cropping, the light field image I of the target area is obtained. LF and target region full-focus image I obj ; Step 3: Analyze the light field image I of the target region. LF Disparity estimation is performed to obtain disparity map I. disp and the disparity map I disp The disparity values ​​d1, d2, ..., d of all pixels in the array. i Histogram statistics are performed to obtain the disparity distribution histogram, where d i Represents the disparity value of the i-th pixel; i represents the disparity map I. disp The total number of pixels in the middle, and i = (x max -x min +1)×(y max -y min +1); Given n matching points λ1, λ2, ..., λ n In scene full-focus image I scene The pixel coordinates in the image are (x1, y1), (x2, y2), ..., (x n ,y n Transformed to the disparity map I disp The corresponding pixel coordinates (x1-x) min ,y1-y min ),(x2-x min ,y2-y min ),...,(x n -x min ,y n -y min ), where (x n ,y n ) represents the nth matching point pair λ n In scene full-focus image I scene Pixel coordinates in; According to the disparity map I disp The pixel coordinates (x1-x) of n matching point pairs min ,y1-y min ),(x2-x min ,y2-y min ),...,(x n -x min ,y n -y min ), and the disparity value distribution of n matching point pairs in the disparity distribution histogram is obtained as [d p1 ,d q1 ],[d p2 ,d q2 ],...,[d pk ,d qk ], where [d pk ,d qk [] represents the distribution of the k-th disparity value, k≤n; Find the disparity distribution [d] p1 ,d q1 ],[d p2 ,d q2 ],...,[d pk ,d qk The distribution of disparity values ​​containing the largest number of matching point pairs [d] pmax ,d qmax ], max∈[1,k]; thus, the disparity range function of the target object is constructed using equation (1): In equation (1), d bin The range width of the distribution for each disparity value, [d] p ,d q [] represents the disparity distribution of the target object; Δ represents a constant; The disparity map I disp The center is located in the disparity value distribution [d] p ,d q All disparity values ​​outside the region are set to 0, thus setting the disparity map I... disp The target object in the image is separated from the background, and the resulting disparity map I' is obtained. disp ; Fully focused image I of the target region obj Perform edge detection to obtain the target edge image I edge ; The disparity map I' disp and target edge image I edge Pixel-by-pixel fusion is performed to obtain the target edge disparity map I. disp-edge ; Step 4: Combine camera intrinsic parameters to obtain the target edge disparity map I. disp-edge Target edge point cloud P scene and the template full-focus image I model The corresponding template edge point cloud P model Registration is performed to obtain the rotation matrix R and translation t, which are then used as the 6D pose of the target object.

2. The 6D pose detection method for a target object based on a light field camera according to claim 1, characterized in that, In step 3, the target edge disparity map I is obtained using equation (2). disp-edge The disparity value I corresponding to any pixel coordinate (x, y) in the image. disp-edge (x, y): In equation (2), x∈[0,x] max -x min ],y∈[0,y max -y min ], I' disp (x, y) represents the disparity value corresponding to the pixel coordinates (x, y) in the disparity map after the target object and background are separated. edge (x, y) represents the pixel value corresponding to the pixel coordinates (x, y) in the target edge image.

Images