Object pose detection method and device based on PVNet and double-branch feature fusion

Abstract

The application discloses a kind of object pose detection method and device based on PVNet and double-branch feature fusion, belong to computer vision and object pose detection technical field, method includes steps: two-dimensional image of the object to be detected is obtained by camera, two-dimensional image is input into the CNN network of double-branch feature fusion module to generate the semantic segmentation image of the object to be detected, the CNN network of double-branch feature fusion module includes basic architecture and double-branch feature fusion module;The semantic segmentation image is pixelated voting, and the two-dimensional feature point of the object to be detected is predicted to generate;The two-dimensional feature point of prediction and known three-dimensional key points of the object to be detected are substituted into uncertainty-driven PnP algorithm, and the 6D pose parameter of object is calculated.The application enhances the fusion ability of image semantic features and spatial details, significantly improves the precision and robustness of object pose detection in complex scene.

Description

Technical Field

[0001] This invention relates to an object pose detection method and apparatus based on PVNet and dual-branch feature fusion, belonging to the field of computer vision and object pose detection technology. Background Technology

[0002] Object pose estimation (6D Pose Estimation), a core technology in computer vision, aims to determine the position (translation) and orientation (rotation) of a target object in three-dimensional space using monocular or RGB-D images. Its output consists of a rotation matrix R∈SO(3) and a translation vector t∈R³, representing six degrees of freedom. The development of this technology has gone through several stages: early methods employed traditional methods based on handcrafted features (such as SIFT feature matching combined with the ICP algorithm); with the rise of deep learning, methods for directly regressing keypoint coordinates emerged; currently, the mainstream method is based on semantic segmentation and geometric reasoning techniques (such as PVNet and DenseFusion). Among these, PVNet achieves keypoint prediction through semantic segmentation and pixelation voting mechanisms, demonstrating good performance in pose estimation tasks.

[0003] Object pose estimation has significant value in fields such as industrial automation (e.g., improving the grasping accuracy of robotic arms to 92%), augmented reality (achieving sub-centimeter-level virtual registration), and autonomous driving (Tesla Autopilot core module). However, it still faces challenges such as occlusion handling (40% of samples in the LINEMOD dataset are occluded), multiple solutions for symmetric objects, few-shot learning, and real-time requirements (<50ms latency).

[0004] The existing PVNet network has insufficient ability to balance global semantic information and local detailed features in the semantic segmentation stage: global feature extraction is prone to losing local spatial information, and local feature extraction is difficult to model long-range dependencies, resulting in limited accuracy of subsequent keypoint prediction and pose calculation. Therefore, this invention provides an object pose detection method and apparatus based on PVNet and dual-branch feature fusion. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes an object pose detection method and apparatus based on PVNet and dual-branch feature fusion, which can improve the accuracy of object pose estimation without increasing computational overhead.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] In a first aspect, the present invention provides an object pose detection method using PVNet and dual-branch feature fusion, comprising the following steps:

[0008] Step S1: Acquire a 2D image of the object to be detected using a camera. Input the 2D image into a CNN network with a dual-branch feature fusion module to generate a semantic segmentation image of the object. The CNN network with the dual-branch feature fusion module is an improved semantic segmentation network architecture, including a basic architecture and a dual-branch feature fusion module. The basic architecture uses a fully convolutional neural network as the basic semantic segmentation framework and ResNet as the feature extraction backbone network of the encoder-decoder structure. The dual-branch feature fusion module includes a global context extraction path, a spatial information extraction path, and a dual-branch information fusion and feature recalibration mechanism.

[0009] Step S2: Perform pixel-wise voting on the semantic segmentation image to predict and generate two-dimensional feature points of the object to be detected;

[0010] Step S3: Substitute the predicted 2D feature points and the known 3D key points of the object to be detected into the uncertainty-driven PnP (Perspective-n-Point) algorithm to calculate the 6D pose parameters of the object.

[0011] As one possible implementation of this embodiment, the global context extraction path is used for:

[0012] Feature map for each channel Perform global average pooling to obtain channel context information. :

[0013] ,

[0014] Where H and W are the height and width of the input two-dimensional image, respectively. This indicates a global average pooling operation. For channel c in The pixel value of the location;

[0015] The channel context information P is processed using a lightweight MLP to model long-range dependencies between channels.

[0016] As one possible implementation of this embodiment, the lightweight MLP includes two fully connected layers, and the number of channels in the fully connected layers is 1 / 16 of the number of input feature channels (the value is selected through a large number of comparative experiments to find the optimal balance between accuracy and parameter quantity). The lightweight MLP is implemented by 1×1 convolutional layers.

[0017] As one possible implementation of this embodiment, the spatial information extraction path is used for:

[0018] A downsampling operation is performed on the input feature map, and a non-linear transformation is performed on the downsampled feature map through a lightweight convolutional layer. The lightweight convolutional layer is a depthwise separable convolution, and the convolution kernel is selected according to the size of the input image: if the input image is a small dataset, a 3x3 convolution kernel is used; if it is a normal-sized image, a 3x3 dilated convolution with a dilation factor of 2 is used.

[0019] Global average pooling is performed on the convolutional feature map to generate spatial information statistics. , The formula for calculating the c-th element is:

[0020] ,

[0021] in, and These are the height and width of the downsampled feature map, respectively. This is the feature map after downsampling. This indicates a global average pooling operation.

[0022] The lightweight convolutional layer has the same number of output channels as the number of input feature channels, and the convolution stride is set to 1 to achieve spatial downsampling.

[0023] As one possible implementation of this embodiment, the dual-branch information fusion and feature recalibration mechanism is used for:

[0024] After scaling the output P of the global context extraction path and the output S of the spatial information extraction path, sum them element by element to obtain the fused feature F = S + P.

[0025] The fused feature F is excited by two fully connected layers. The operation of the first fully connected layer is as follows: ,in It is the ReLU function. For batch normalization operations, The weights of the first fully connected layer are; the operations of the second fully connected layer are... ,in It is the sigmoid activation function. These are the weights for the second fully connected layer;

[0026] The global path weights are calculated by applying the softmax function to the second fully connected layer operation F''. Spatial path weights :

[0027] ,

[0028] Where A, B∈ , That is The C-th row element of the eigenvector, and ;

[0029] Weighting the original feature map Perform recalibration to generate the final feature map. :

[0030] ,

[0031] in , .

[0032] As one possible implementation of this embodiment, step S2 includes:

[0033] Two pixels are randomly selected from the semantically segmented target region to be detected, and their vector intersection is used as the hypothetical keypoint. Repeat N times to generate a set of key points The key assumptions;

[0034] All pixels within the target area vote on the hypothetical key point. The voting score is:

[0035] ,

[0036] in For index functions, The threshold is set to 0.99. Represents pixels Belongs to object ;

[0037] Based on the voting scores, key hypotheses are selected, and the mean of their spatial probability distribution is calculated. Covariance This is used for subsequent PnP algorithm calculations.

[0038] As one possible implementation of this embodiment, the value of N ranges from 100 to 500, and multiple samplings are used to ensure the diversity and representativeness of the assumed key points.

[0039] As one possible implementation of this embodiment, the mean Covariance The calculation formulas are as follows:

[0040] ,

[0041] ,

[0042] in, Assuming key points The mean distribution of the probability weights.

[0043] As one possible implementation of this embodiment, step S3 includes:

[0044] Input prediction of the spatial probability distribution of two-dimensional key points ( , ) and known 3D key points ;

[0045] The pose parameters are solved by minimizing the Mahalanobis distance. R , t ):

[0046] ,

[0047] ,

[0048] in, The three-dimensional coordinates of the key points for Two-dimensional projection, It is a linear transformation matrix. For sample categories, This is the perspective projection function.

[0049] Secondly, an object pose detection device based on PVNet and dual-branch feature fusion provided by embodiments of the present invention includes:

[0050] The image acquisition and processing module is used to acquire a two-dimensional image of the object to be detected through a camera, and input the two-dimensional image into a CNN network inserted into the dual-branch feature fusion module to generate a semantic segmentation image of the object to be detected. The CNN network inserted into the dual-branch feature fusion module is an improved semantic segmentation network architecture, including a basic architecture and a dual-branch feature fusion module. The basic architecture uses a fully convolutional neural network as the basic semantic segmentation framework and ResNet as the feature extraction backbone network of the encoder-decoder structure. The dual-branch feature fusion module includes a global context extraction path, a spatial information extraction path, and a dual-branch information fusion and feature recalibration mechanism.

[0051] The pixelation voting module is used to perform pixelation voting on the semantic segmentation image to predict and generate two-dimensional feature points of the object to be detected.

[0052] The 6D pose parameter calculation module is used to substitute the predicted 2D feature points and the known 3D key points of the object to be detected into the uncertainty-driven PnP algorithm to calculate the 6D pose parameters of the object.

[0053] Thirdly, an electronic device provided by an embodiment of the present invention includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the object pose detection method such as the above-described arbitrary PVNet and dual-branch feature fusion method.

[0054] Fourthly, embodiments of the present invention provide a storage medium storing a computer program, which, when run by a processor, executes the steps of the object pose detection method described above using arbitrary PVNet and dual-branch feature fusion.

[0055] The beneficial effects of the technical solutions in the embodiments of the present invention are as follows:

[0056] This invention generates high-precision semantic segmentation images through an improved semantic segmentation network, providing accurate region constraints for subsequent keypoint prediction. The dual-branch feature fusion module improves feature extraction accuracy by balancing global semantics and local details. Based on the semantic segmentation results, a pixelated voting mechanism predicts the positions and uncertainty distribution of 2D feature points, providing reliable input data for pose estimation. The voting mechanism also enhances robustness against occlusion and noise. By combining 2D feature points with 3D keypoints, an uncertainty-driven PnP algorithm is used to achieve high-precision pose estimation. Introducing covariance information optimizes the solution process, improving the accuracy of pose estimation.

[0057] This invention enhances the fusion capability of image semantic features and spatial details through a dual-branch feature fusion module. Combined with the uncertainty-driven PnP algorithm, it improves the accuracy of object pose estimation without increasing computational overhead, significantly enhancing the accuracy and robustness of object pose detection in complex scenes. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating an object pose detection method using PVNet and dual-branch feature fusion according to an exemplary embodiment.

[0059] Figure 2 This is a schematic diagram of an object pose detection device based on an exemplary embodiment of PVNet and dual-branch feature fusion.

[0060] Figure 3 This is a structural diagram of a dual-branch enhanced channel attention module according to an exemplary embodiment;

[0061] Figure 4 This is a flowchart illustrating the specific implementation of the present invention for object pose detection in a PVNet network based on the insertion of a dual-branch feature fusion module. Detailed Implementation

[0062] To more clearly illustrate the technical features of the present invention, the present invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0063] like Figure 1 As shown in the figure, the object pose detection method based on PVNet and dual-branch feature fusion provided by this embodiment of the invention includes the following steps:

[0064] Step S1: Acquire a 2D image of the object to be detected using a camera. Input the 2D image into a CNN network with a dual-branch feature fusion module to generate a semantic segmentation image of the object. The CNN network with the dual-branch feature fusion module is an improved semantic segmentation network architecture, including a basic architecture and a dual-branch feature fusion module. The basic architecture uses a fully convolutional neural network as the basic semantic segmentation framework and ResNet as the feature extraction backbone network of the encoder-decoder structure. The dual-branch feature fusion module includes a global context extraction path, a spatial information extraction path, and a dual-branch information fusion and feature recalibration mechanism.

[0065] Step S2: Perform pixel-wise voting on the semantic segmentation image to predict and generate two-dimensional feature points of the object to be detected;

[0066] Step S3: Substitute the predicted 2D feature points and the known 3D key points of the object to be detected into the uncertainty-driven PnP (Perspective-n-Point) algorithm to calculate the 6D pose parameters of the object.

[0067] This invention enhances the fusion capability of image semantic features and spatial details through a dual-branch feature fusion module. Combined with the uncertainty-driven PnP algorithm, it significantly improves the accuracy and robustness of object pose detection in complex scenes. The dual-branch structure of this invention effectively fuses global semantic and local detail features while maintaining lightweight computational overhead, improving the segmentation accuracy of small and occluded targets and reducing the cumulative error of pose estimation.

[0068] As one possible implementation of this embodiment, the global context extraction path is used for: feature maps of each channel Perform global average pooling to obtain channel context information. :

[0069] ,

[0070] Where H and W are the height and width of the input two-dimensional image, respectively. This indicates a global average pooling operation. For channel c in The pixel value of the location;

[0071] A lightweight MLP is used to process the channel context information P, modeling long-range dependencies between channels. Modeling long-range dependencies between channels preserves global semantic information, helping the network understand the overall category and outline of objects.

[0072] As one possible implementation of this embodiment, the lightweight MLP includes two fully connected layers, and the number of channels in the fully connected layers is 1 / 16 of the number of input feature channels. The lightweight MLP is implemented by a 1×1 convolutional layer.

[0073] As one possible implementation of this embodiment, the spatial information extraction path is used for:

[0074] A downsampling operation is performed on the input feature map, and a non-linear transformation is performed on the downsampled feature map through a lightweight convolutional layer. The lightweight convolutional layer is a depthwise separable convolution, and the convolution kernel is selected according to the size of the input image: if the input image is a small dataset, a 3x3 convolution kernel is used; if it is a normal-sized image, a 3x3 dilated convolution with a dilation factor of 2 is used.

[0075] Global average pooling is performed on the convolutional feature map to generate spatial information statistics. , The formula for calculating the c-th element is:

[0076] ,

[0077] in, and These are the height and width of the downsampled feature map, respectively. This is the feature map after downsampling. This represents the global average pooling operation. The spatial information extraction path mitigates information loss in the global path by extracting multi-scale local statistical features, enhancing the network's ability to perceive the spatial location of objects.

[0078] The lightweight convolutional layer has the same number of output channels as the number of input feature channels, and the convolution stride is set to 1 to achieve spatial downsampling.

[0079] As one possible implementation of this embodiment, the dual-branch information fusion and feature recalibration mechanism is used to: scale-align P, the output of the global context extraction path, and S, the output of the spatial information extraction path, and then sum them element by element to obtain the fused feature F=S+P;

[0080] The fused feature F is excited by two fully connected layers. The operation of the first fully connected layer is as follows: ,in It is the ReLU function. For batch normalization operations, The weights of the first fully connected layer are; the operations of the second fully connected layer are... ,in It is the sigmoid activation function. These are the weights for the second fully connected layer;

[0081] The global path weights are calculated by applying the softmax function to the second fully connected layer operation F''. Spatial path weights :

[0082] ,

[0083] Where A, B∈ , That is The C-th row element of the eigenvector, and ;

[0084] Weighting the original feature map Perform recalibration to generate the final feature map. :

[0085] ,

[0086] in , The dual-branch fusion and recalibration mechanism achieves an adaptive balance between global and local features, enabling the network to dynamically adjust feature weights based on the input content, thereby improving the flexibility and accuracy of semantic segmentation.

[0087] As one possible implementation of this embodiment, step S2 includes:

[0088] Two pixels are randomly selected from the semantically segmented target region to be detected, and their vector intersection is used as the hypothetical keypoint. Repeat N times to generate a set of key points The key assumptions;

[0089] All pixels within the target area vote on the hypothetical key point. The voting score is:

[0090] ,

[0091] in For index functions, The threshold is set to 0.99. Represents pixels Belongs to object ;

[0092] Based on the voting scores, key hypotheses are selected, and the mean of their spatial probability distribution is calculated. Covariance This information is used for subsequent PnP algorithm calculations. Step S2 uses multiple rounds of assumptions and voting to select the optimal key points, reducing the random error of a single prediction and improving the accuracy of key point localization.

[0093] As one possible implementation of this embodiment, the value of N is all the pixels of the target to be detected after image segmentation, and the diversity and representativeness of the hypothetical key points are ensured by multiple samplings.

[0094] As one possible implementation of this embodiment, the mean Covariance The calculation formulas are as follows:

[0095] ,

[0096] ,

[0097] in, Assuming key points The mean distribution of the probability weights.

[0098] As one possible implementation of this embodiment, step S3 includes:

[0099] Input prediction of the spatial probability distribution of two-dimensional key points ( , ) and known 3D key points ;

[0100] The pose parameters (R,t) are solved by minimizing the Mahalanobis distance:

[0101] ,

[0102] ,

[0103] in, The three-dimensional coordinates of the key points for Two-dimensional projection, It is a linear transformation matrix. For sample categories, This is the perspective projection function. Step S3 fully utilizes the probability distribution information of key points, reduces the impact of outliers on the solution results, and improves the robustness of pose estimation in complex scenes.

[0104] This invention incorporates the uncertainty of keypoint estimation into the PnP solution process and reduces the impact of low-confidence keypoints on the pose solution through Mahalanobis distance weighted optimization. Compared with the traditional PnP algorithm, the pose accuracy is improved by 20%-30% in occluded and noisy scenes.

[0105] The uncertainty-driven PnP algorithm specifically includes the following steps:

[0106] Step S31, Initialization:

[0107] Input the predicted two-dimensional keypoint mean and its covariance matrix Known 3D key points Camera intrinsic parameter matrix K;

[0108] Initialize pose parameters The initial solution can be obtained using traditional algorithms such as EPNP and RANSAC; Step S32, iterative optimization:

[0109] For the i-th iteration, perform the following sub-steps:

[0110] Step S321, Projection Calculation: Project the 3D key points Based on the current pose Projected onto the image plane:

[0111] ;

[0112] Step S322, Error Calculation: Calculate the projection point With prediction of two-dimensional key points The residual vector between: ;

[0113] Step S323, Weight Calculation: Calculate the Mahalanobis distance based on the residual vector and covariance matrix. ;

[0114] Step S324, Weighted Least Squares Solution: Construct the weighted error function:

[0115] ,

[0116] Among them, weight , To prevent small constants with a denominator of zero;

[0117] Step S325, Pose Update: Use the Levenberg-Marquardt algorithm to find the minimum value of the error function and update the pose parameters.

[0118] ;

[0119] Step S336, Termination condition determination:

[0120] When the number of iterations reaches the preset maximum value (Recommended value is 20), or the pose update amount is less than the threshold. (Recommended value) When the iteration terminates, output the final pose parameters. .

[0121] The process of constructing the covariance matrix is ​​as follows:

[0122] Obtain the covariance matrix of keypoint estimation from the pixelation voting step. ;

[0123] To account for the effects of camera noise, a diagonal noise matrix is ​​added. ,in The standard deviation of pixel noise (recommended value is 1.0);

[0124] The final covariance matrix is: .

[0125] This invention incorporates the uncertainty of key point estimation into the PnP solution process and reduces the impact of low-confidence key points on the pose solution through Mahalanobis distance weighted optimization. Compared with the traditional PnP algorithm, the pose accuracy is improved by 20%-30% in occluded and noisy scenes.

[0126] In the iterative optimization of step S32, the rotation matrix R is represented by a Lie group, and the rotation is parameterized into a three-dimensional vector. Through exponential mapping Calculate the rotation matrix, where for The corresponding antisymmetric matrix.

[0127] The pose update step specifically includes:

[0128] Use the current rotation matrix Convert to rotation vector ;

[0129] Calculate pose increment :

[0130] ,

[0131] in For Jacobian matrices, It is a diagonal weight matrix. It is the residual vector;

[0132] Update the rotation vector and translation vector:

[0133] ;

[0134] The updated rotation vector is converted into a rotation matrix using an exponential mapping:

[0135] .

[0136] This invention uses Lie group representation to avoid the orthogonality constraint problem of rotation matrix, improves optimization efficiency and stability, and enables the algorithm to maintain good convergence in large-angle rotation scenarios.

[0137] Step S3 further includes:

[0138] Step S34, uncertainty propagation:

[0139] Calculate the covariance matrix of pose parameters Used to assess the uncertainty of pose estimation:

[0140] ,

[0141] Where J is the Jacobian matrix of the final iteration step, and W is the corresponding weight matrix;

[0142] Step S35, outlier detection and removal:

[0143] Calculate the standardized residual for each keypoint: ;

[0144] Set a threshold T (recommended value is 3.0), and... Key points are marked as outliers;

[0145] Re-execute step S32, eliminate outliers, and then perform iterative optimization.

[0146] This invention provides confidence assessment of pose estimation through uncertainty propagation, and combined with an outlier removal mechanism, further improves the robustness of the algorithm in complex scenarios, especially suitable for cases of partial occlusion or missing features.

[0147] like Figure 2 As shown in the figure, an object pose detection device based on PVNet and dual-branch feature fusion provided by an embodiment of the present invention includes:

[0148] The image acquisition and processing module is used to acquire a two-dimensional image of the object to be detected through a camera, and input the two-dimensional image into a CNN network inserted into the dual-branch feature fusion module to generate a semantic segmentation image of the object to be detected. The CNN network inserted into the dual-branch feature fusion module is an improved semantic segmentation network architecture, including a basic architecture and a dual-branch feature fusion module. The basic architecture uses a fully convolutional neural network as the basic semantic segmentation framework and ResNet as the feature extraction backbone network of the encoder-decoder structure. The dual-branch feature fusion module includes a global context extraction path, a spatial information extraction path, and a dual-branch information fusion and feature recalibration mechanism.

[0149] The pixelation voting module is used to perform pixelation voting on the semantic segmentation image to predict and generate two-dimensional feature points of the object to be detected.

[0150] The 6D pose parameter calculation module is used to substitute the predicted 2D feature points and the known 3D key points of the object to be detected into the uncertainty-driven PnP algorithm to calculate the 6D pose parameters of the object.

[0151] The object pose detection method based on PVNet and the dual-branch fusion module mainly includes the following parts.

[0152] 1. Improved semantic segmentation network architecture:

[0153] Regarding the choice of infrastructure: a fully convolutional neural network is adopted as the basic semantic segmentation framework, and its encoder-decoder structure is the feature extraction backbone network ResNet.

[0154] like Figure 3 As shown, the dual-branch fusion module inserted into the backbone network includes:

[0155] a) Global context extraction path: retain GAP and lightweight MLP to maintain the ability to model long-range dependencies between channels;

[0156] b) Spatial information extraction path: Introduce lightweight convolutional layers (such as depthwise separable convolution) to perform spatial downsampling and nonlinear transformation on feature maps to extract multi-scale local statistical features;

[0157] c) Dual-branch information fusion and feature recalibration: Scale alignment and weighted fusion are performed on the global / local features of the dual-path outputs to generate a spatially sensitive channel attention map. This allows the network to adaptively balance semantic abstraction and detail preservation based on the input content, achieving recalibration of the dual-branch feature weights.

[0158] 2. Pixelated voting mechanism:

[0159] Keypoint prediction is performed using a semantic segmentation map generated by a semantic segmentation network. The prediction method employs a pixel-based voting mechanism to predict the location of the required keypoints in the 2D image. This generates the keypoint distribution locations of the 2D object to be detected.

[0160] 3. PnP pose calculation includes:

[0161] The predicted 2D keypoints and the known 3D keypoints of the object to be detected are fed into the uncertainty-driven PnP algorithm to calculate the final 6D pose parameters of the object.

[0162] like Figure 4 As shown, the specific process of object pose detection in the PVNet network based on the insertion of a dual-branch feature fusion module in this invention is as follows.

[0163] Step S1: Acquire a two-dimensional image of the object to be detected using a camera. The two-dimensional image is then used to generate a semantic segmentation image of the object to be detected through a CNN network with a dual-branch feature fusion module inserted.

[0164] The CNN network with the inserted dual-branch feature fusion module is an improved semantic segmentation network architecture, including a basic architecture and a dual-branch feature fusion module. The basic architecture uses a fully convolutional neural network as the basic semantic segmentation framework and ResNet as the feature extraction backbone network of the encoder-decoder structure. The dual-branch feature fusion module includes a global context extraction path, a spatial information extraction path, and a dual-branch information fusion and feature recalibration mechanism.

[0165] Global context extraction path for input 2D image The process of extracting contextual information is as follows: First, global average pooling is performed on the input image for each channel. This ensures that each feature map generated by the convolution is consistent with the contextual information. It includes only the global context information for that region and also compresses the feature information of each channel into the channel descriptor. Let the global average pooled context information of a certain channel be... Then the channel information after global average pooling of the input feature map The calculation formula is:

[0166] ,

[0167] Here, H and W represent the height and width of the input image, respectively. This step allows the backbone network to better capture long-range dependencies between input image channels, thereby improving the semantic segmentation accuracy of the backbone network for the objects to be detected.

[0168] The specific process of the local detail extraction path is as follows: The input image is downsampled only once, and the downsampled convolutional feature map is then subjected to global average pooling, similar to the context extraction method. The convolutional kernel selection is based on the size of the input image, using two kernel types: a 3x3 kernel for smaller datasets, and a dilated convolution with a 3x3 kernel but a dilation of 2 for normal-sized images. Finally, the convolutional feature map is subjected to global average pooling to generate spatial information statistics. , The formula for calculating the c-th element is:

[0169] ,

[0170] This step enables the original backbone network to extract multi-scale local statistical features, mitigating the information loss caused by the context extraction path. This allows the backbone network to better perceive the location of the object to be detected within the image.

[0171] The specific process of dual-branch information fusion and feature recalibration is as follows:

[0172] The aforementioned context extraction path and local detail extraction path are scale-aligned and weighted to generate spatially sensitive channel attention vectors, enabling the network to adaptively balance semantic abstraction and detail preservation based on the input content.

[0173] The dual-branch information fusion part involves summing the image feature extraction values ​​from the two branches one by one:

[0174] ,

[0175] The information from the two fully connected layers triggers the operation of the first fully connected layer as follows:

[0176] ,

[0177] in It is the ReLU function. For batch normalization operations, the operation for the second fully connected layer is defined after the first fully connected layer as follows:

[0178] ,

[0179] in It is the sigmoid activation function. It is still a batch normalization operation.

[0180] Feature recalibration involves applying a softmax operation to the feature vectors generated by the feature fusion process. This is applied here to the feature vectors of corresponding channels from the two pathways, specifically:

[0181] ,

[0182] in For compact feature description channels, A, B∈ , That is The Cth row element of the eigenvector. This represents the weight value of the Cth element of the respective path, which can be determined from the characteristics of the softmax function. Final feature map It is obtained through attention weights on various kernels:

[0183] ,

[0184] in , Finally, a recalibrated feature map was obtained. The feature map shows the weight ratios between different channels based on the information weights of the two branches. Thus, the semantic segmentation backbone network can perform semantic segmentation on the output 2D image based on the recalibrated feature map, generating a semantic segmentation image of the object to be detected and outputting this image to the subsequent keypoint prediction part.

[0185] In step S2, the network then performs pixel-wise voting on the semantically segmented object feature images to predict and generate two-dimensional feature points of the object to be detected.

[0186] First, two pixels within the target object to be detected after semantic segmentation need to be randomly selected, and the intersection of their vectors is used as the hypothetical key point. Repeating this process N times will generate a set of key points. The hypothetical keypoints are then determined, and all pixels belonging to the object vote on these hypothetical keypoints. The voting score is defined as:

[0187] ,

[0188] in For index functions, The threshold is set to 0.99. Represents pixels Belongs to object Intuitively, a higher vote score implies a more accurate hypothesis because it aligns with more predicted directions. The resulting hypothesis describes the spatial probability distribution of a keypoint in the image, with a mean of... The covariance ∑k is as follows:

[0189] ,

[0190] ,

[0191] in, Assuming key points The mean distribution, i.e., the probability weights; the mean and covariance are used to calculate the PnP pose parameters in the next stage.

[0192] Step S3: Input the predicted two-dimensional feature points and the known three-dimensional key points of the object to be detected into the PnP algorithm to calculate the 6D pose parameters of the object.

[0193] Given the 3D keypoint locations of each object and the keypoint locations of the object in the 2D image predicted by the above process, its 6D pose parameters can be calculated using an off-the-shelf PnP solver. However, since different keypoints may have different confidence levels and uncertainty patterns, this should be taken into account when solving the PnP.

[0194] Therefore, the spatial probability distribution (mean) of each keypoint obtained in the pixelated voting part is used to calculate the pixelated voting part. Covariance 6D pose is calculated by combining minimizing Mahalanobis distance and the PnP algorithm. The final object pose is obtained:

[0195] ,

[0196] ,

[0197] in, The three-dimensional coordinates of the key points for Two-dimensional projection, This is the perspective projection function.

[0198] This invention utilizes the attention mechanism in deep learning, specifically designing a dual-branch enhanced channel attention module. This module achieves channel dimension recalibration and spatial multi-scale feature extraction through a parallel branch structure: the channel branch introduces a squeeze-excitation mechanism to dynamically model inter-channel dependencies, while the spatial branch uses differentiated convolutional kernels to construct multi-scale feature representations. Adaptive fusion of dual-path features is achieved through a gated attention mechanism, enabling the network to simultaneously enhance its global semantic information capture and local detail feature extraction capabilities. By inserting this module into the semantic segmentation part of the original PVNet network, a new semantic segmentation network architecture is formed, thereby improving the accuracy of the semantic segmentation module. This, in turn, leads to higher accuracy in subsequent keypoint prediction and PnP solving. Ultimately, this improves the overall accuracy of the PVNet network in estimating the pose of the object to be detected.

[0199] An electronic device provided by an embodiment of the present invention includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the object pose detection method such as the arbitrary PVNet and dual-branch feature fusion described above.

[0200] Specifically, the aforementioned memory and processor can be general-purpose memory and processor, without any specific limitations. When the processor runs the computer program stored in the memory, it can execute the aforementioned PVNet and dual-branch feature fusion object pose detection method.

[0201] Corresponding to the above application startup method, this embodiment of the invention also provides a storage medium storing a computer program, which is executed by a processor to perform the steps of the object pose detection method such as the above arbitrary PVNet and dual-branch feature fusion method.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for object pose detection using PVNet and dual-branch feature fusion, characterized in that, Includes the following steps: Step S1: Acquire a two-dimensional image of the object to be detected through a camera, and input the two-dimensional image into a CNN network with a dual-branch feature fusion module to generate a semantic segmentation image of the object to be detected. The CNN network with a dual-branch feature fusion module includes a basic architecture and a dual-branch feature fusion module. The basic architecture uses a fully convolutional neural network as the basic semantic segmentation framework and ResNet as the feature extraction backbone network of the encoder-decoder structure. The dual-branch feature fusion module includes a global context extraction path, a spatial information extraction path, and a dual-branch information fusion and feature recalibration mechanism. Step S2: Perform pixel-based voting on the semantic segmentation image to predict and generate two-dimensional feature points of the object to be detected; Step S3: Substitute the predicted 2D feature points and the known 3D key points of the object to be detected into the uncertainty-driven PnP algorithm to calculate the 6D pose parameters of the object. The uncertainty-driven PnP algorithm specifically includes the following steps: Step S31, Initialization: Input the predicted two-dimensional keypoint mean and its covariance matrix Known 3D key points Camera intrinsic parameter matrix K; Initialize pose parameters The initial solution is obtained using traditional algorithms such as EPNP or RANSAC. Step S32, iterative optimization: For the i-th iteration, perform the following sub-steps: Step S321, Projection Calculation: Project the 3D key points Based on the current pose Projected onto the image plane: ； Step S322, Error Calculation: Calculate the projection point With prediction of two-dimensional key points The residual vector between: ; Step S323, Weight Calculation: Calculate the Mahalanobis distance based on the residual vector and covariance matrix. ; Step S324, Weighted Least Squares Solution: Construct the weighted error function: ， Among them, weight , To prevent small constants with a denominator of zero; Step S325, Pose Update: Use the Levenberg-Marquardt algorithm to find the minimum value of the error function and update the pose parameters. ； The pose update step specifically includes: Use the current rotation matrix Convert to rotation vector ; Calculate pose increment : ， in For Jacobian matrices, It is a diagonal weight matrix. It is the residual vector; Update the rotation vector and translation vector: ； The updated rotation vector is converted into a rotation matrix using an exponential mapping: ； Step S326, Termination condition determination: When the number of iterations reaches the preset maximum value Or the pose update amount is less than the threshold When the iteration terminates, output the final pose parameters. .

2. The object pose detection method based on PVNet and dual-branch feature fusion according to claim 1, characterized in that, The global context extraction path is used for: Feature map for each channel Perform global average pooling to obtain channel context information. : ， Where H and W are the height and width of the input two-dimensional image, respectively. This indicates a global average pooling operation. For channel c in The pixel value of the location; The channel context information P is processed using a lightweight MLP to model long-range dependencies between channels.

3. The object pose detection method based on PVNet and dual-branch feature fusion according to claim 2, characterized in that, The spatial information extraction path is used for: A downsampling operation is performed on the input feature map, and a non-linear transformation is performed on the downsampled feature map through a lightweight convolutional layer. The lightweight convolutional layer is a depthwise separable convolution, and the convolution kernel is selected according to the size of the input image: if the input image is a small dataset, a 3x3 convolution kernel is used; if it is a normal-sized image, a 3x3 dilated convolution with a dilation factor of 2 is used. Global average pooling is performed on the convolutional feature map to generate spatial information statistics. , The formula for calculating the c-th element is: ， in, and These are the height and width of the downsampled feature map, respectively. This is the feature map after downsampling. This indicates a global average pooling operation.

4. The object pose detection method based on PVNet and dual-branch feature fusion according to claim 3, characterized in that, The dual-branch information fusion and feature recalibration mechanism is used for: After scaling the output P of the global context extraction path and the output S of the spatial information extraction path, sum them element by element to obtain the fused feature F = S + P. The fused feature F is excited by two fully connected layers. The operation of the first fully connected layer is as follows: ,in It is the ReLU function. For batch normalization operations, The weights of the first fully connected layer are; the operations of the second fully connected layer are... ,in It is the sigmoid activation function. These are the weights for the second fully connected layer; The global path weights are calculated by applying the softmax function to the second fully connected layer operation F''. Spatial path weights : ， Where A, B∈ , That is The C-th row element of the eigenvector, and ; Weighting the original feature map Perform recalibration to generate the final feature map. .

5. The object pose detection method based on PVNet and dual-branch feature fusion according to any one of claims 1-4, characterized in that, Step S2 includes: Two pixels are randomly selected from the semantically segmented target region to be detected, and their vector intersection is used as the hypothetical keypoint. Repeat N times to generate a set of key points The key assumptions; All pixels within the target area vote on the hypothetical key point. The voting score is: ， in For index functions, For the threshold, Represents pixels Belongs to object ; Based on the voting scores, key hypotheses are selected, and the mean of their spatial probability distribution is calculated. Covariance This is used for subsequent PnP algorithm calculations.

6. The object pose detection method based on PVNet and dual-branch feature fusion according to claim 5, characterized in that, The mean Covariance The calculation formulas are as follows: ， ， in, Assuming key points The mean distribution of the probability weights.

7. The object pose detection method based on PVNet and dual-branch feature fusion according to claim 5, characterized in that, Step S3 includes: Input prediction of the spatial probability distribution of two-dimensional key points ( , ) and known 3D key points ; The pose parameters are solved by minimizing the Mahalanobis distance. : ， ， in, The three-dimensional coordinates of the key points for Two-dimensional projection, It is a linear transformation matrix. For sample categories, This is the perspective projection function.

8. An object pose detection device based on PVNet and dual-branch feature fusion, used to implement the object pose detection method based on PVNet and dual-branch feature fusion as described in any one of claims 1-7, characterized in that, include: The image acquisition and processing module is used to acquire a two-dimensional image of the object to be detected through a camera, and input the two-dimensional image into the CNN network of the dual-branch feature fusion module to generate a semantic segmentation image of the object to be detected. The CNN network of the dual-branch feature fusion module includes a basic architecture and a dual-branch feature fusion module. The basic architecture uses a fully convolutional neural network as the basic semantic segmentation framework and ResNet as the feature extraction backbone network of the encoder-decoder structure. The dual-branch feature fusion module includes a global context extraction path, a spatial information extraction path, and a dual-branch information fusion and feature recalibration mechanism. The pixelation voting module is used to perform pixelation voting on the semantic segmentation image to predict and generate two-dimensional feature points of the object to be detected. The 6D pose parameter calculation module is used to substitute the predicted 2D feature points and the known 3D key points of the object to be detected into the uncertainty-driven PnP algorithm to calculate the 6D pose parameters of the object.

9. An electronic device, characterized in that, The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions that the processor can execute. When the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the object pose detection method of PVNet and dual-branch feature fusion as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, performs the steps of the object pose detection method of PVNet and dual-branch feature fusion as described in any one of claims 1-7.

Images