Six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion, electronic equipment and computer program product

Abstract

The embodiment of the invention relates to a six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion, electronic equipment and a computer program product, and belongs to the technical field of image processing, and the method comprises the following steps: generating a three-dimensional point cloud of a target object based on a depth image, extracting three-dimensional point cloud key points from the three-dimensional point cloud by using a farthest point sampling algorithm, and calculating the three-dimensional point cloud key points; extracting corresponding geometric feature vectors through a reparameterized residual PointNet network, projecting the geometric feature vectors to an RGB image plane to obtain color features, inputting the color features into an adaptive weighted fusion network, calculating a weight coefficient representing the reliability of color information based on the geometric and color features, weighting the color features according to the weight coefficient, and splicing the weighted color features with the geometric features to obtain the color information reliability. And finally predicting the six-degree-of-freedom pose of the object based on the fused features. According to the embodiment of the invention, the precision of estimating the six-degree-of-freedom pose in complex scenes such as shielding and illumination variation is improved, and the robustness of the system is improved.

Description

Technical Field

[0001] This application relates to a six-degree-of-freedom pose estimation method, electronic device, and computer program product based on adaptive weighted feature fusion, belonging to the technical field of image processing. Background Technology

[0002] In intelligent manufacturing environments, industrial robots are tasked with performing various complex tasks, such as welding, movement, assembly, and painting. These operations are mostly implemented through offline programming or manual teaching, resulting in limited intelligence levels exhibited by the robots during task execution. Faced with environmental changes and to meet the ever-growing demand for intelligent technologies, especially in the critical task of precise grasping, robots must replan their operational paths. In industrial production processes, to complete tasks such as loading, sorting, and assembly, robots must be able to accurately grasp workpieces. This often requires workpieces to be arranged in a specific order or sparsely, relying on a 2D vision system for positioning until the grasping task is completed. If workpieces are scattered or piled up, the positioning difficulty increases significantly, making it difficult for robots to accurately grasp workpieces through programming or offline training. Therefore, developing the ability for industrial robots to identify and grasp scattered workpieces is of great significance for improving production efficiency, simplifying processing, and enhancing the flexibility and autonomy of the system.

[0003] In vision-controlled assembly operations, object detection is typically limited to two-dimensional space, determining the target's position and orientation through translation and rotation on a plane. This is because occlusion is uncommon in traditional assembly processes, allowing traditional point-to-line feature matching methods to effectively perform pose recognition tasks. However, as industrial scenarios become increasingly complex, object pose recognition from a single viewpoint can no longer meet the demands of more complex operations. Traditional methods often perform poorly in environments with occlusion or stacked objects. In recent years, deep learning technology has developed rapidly, demonstrating superior performance compared to traditional methods in terms of computational efficiency, resistance to sensor noise, and adaptability to complex environments, thus becoming the main development direction for 6DoF object pose estimation.

[0004] With the rapid improvement of industrial automation in my country, the demand for grasping technology capable of performing grasping operations in complex environments with chaos and partial obstruction is gradually increasing. Intelligent grasping operations show broad application prospects in intelligent logistics, manufacturing assembly, and high-risk operation scenarios, but at the same time, they also pose significant challenges to visual recognition and positioning technologies. Researching and developing high-precision, robust, and fast image recognition and positioning algorithms to accurately estimate object poses has become a focus in the field of automated manufacturing. By establishing robust visual recognition and positioning systems, the efficiency of robots performing repetitive and complex tasks such as welding, assembly, and painting can be improved. 6DoF pose estimation, involving the position and rotation of an object in three-dimensional space, is crucial for accurately determining the spatial position and orientation of an object and its pose in the camera coordinate system. It also has great significance for promoting the popularization of industrial automation and alleviating labor shortages.

[0005] As can be seen from the above description of existing technologies, the challenges of pose estimation in the field of intelligent industry are extensive and complex, directly affecting the performance of algorithms and the feasibility of practical applications. 6DoF pose estimation aims to determine the position and orientation of an object in three-dimensional space, which is crucial for applications such as automated production, robot operation, and quality inspection.

[0006] In real-world industrial environments, target objects may be occluded or overlapped by other objects, making accurate estimation of their complete pose more difficult. This is especially true when multiple similar objects are closely arranged, making it particularly challenging to identify the specific location and orientation of a single object. Industrial applications involve a wide variety of objects, which can differ significantly in size, shape, material, and color. This diversity demands that pose estimation algorithms possess high generalization capabilities to adapt to the identification and localization of different objects. Environmental factors such as lighting variations, reflections, shadows, and background noise all affect the accuracy of pose estimation. For example, strong lighting or reflections may prevent some features of the target object from being correctly captured. Pose estimation typically relies on data collected by various sensors, such as cameras and LiDAR. The resolution, accuracy, and operating range of these sensors can limit the performance of pose estimation. Furthermore, the data obtained from sensors may contain noise, requiring complex preprocessing steps for cleanup. Many industrial applications require pose estimation under real-time or near-real-time conditions to ensure the continuity and efficiency of production processes. High real-time requirements mean that the algorithm must complete complex calculations within a short time, placing high demands on computing resources. Effective pose estimation typically requires a large amount of training data to train deep learning models. However, for specific industrial objects and environments, it may be difficult to obtain sufficient high-quality training samples. Furthermore, manually labeling pose information from large amounts of data is both time-consuming and costly. Therefore, developing algorithms with good adaptability and generalization ability is a significant challenge.

[0007] Solving these problems requires comprehensive consideration of multiple factors, including algorithm design, data processing, sensor technology, and computing resource optimization, as well as continuous exploration and innovation to improve the accuracy, robustness, and real-time performance of pose estimation. With technological advancements, particularly the development of deep learning and 3D vision processing technologies, the performance of pose estimation in intelligent industrial applications is constantly improving.

[0008] Traditional networks extract feature information from RGB and depth images separately, followed by post-processing. DenseFusion unifies the processing of RGB and depth images through feature fusion, significantly improving network speed. Furthermore, DenseFusion introduces a differentiable iterative pose refinement process, which iteratively refines the initially predicted pose to obtain a more accurate one. This embodiment selects DenseFusion as the baseline network for improvement. While DenseFusion achieves good results in pose estimation in complex environments, its network structure still has shortcomings. First, DenseFusion uses a random selection method for keypoints in point cloud data. However, in complex scenes, directly using randomly selected points as keypoints for pose estimation is often inefficient and may affect the accuracy of the estimation results due to the unrepresentative nature of the selected points. Additionally, the geometric feature extraction part of the DenseFusion network is a shallow variant of PointNet, which limits the network's ability to extract geometric features. Furthermore, the DenseFusion network, when fusing features from two different modalities, does not consider the relationship between them; it simply concatenates color features, geometric features, and the generated global features together. When the external environment is complex, this may introduce more erroneous information and fail to fully utilize the complementarity between the two modalities to improve the robustness of pose estimation.

[0009] In conclusion, existing technologies can no longer fully meet people's needs and urgently need to be improved. Summary of the Invention

[0010] The main objective of this application is to provide a six-degree-of-freedom pose estimation method, electronic device, and computer program product based on adaptive weighted feature fusion, thereby addressing the shortcomings of existing technologies.

[0011] The embodiments of this application are implemented using the following technical solutions: According to one aspect of the embodiments of this application, a six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion is provided, comprising: generating a three-dimensional point cloud of a target object based on a depth image; selecting a point from the three-dimensional point cloud as an initial point using a farthest point sampling algorithm; calculating the distance from each point in the three-dimensional point cloud to the initial point to generate a multi-dimensional distance array; selecting the maximum value point in the multi-dimensional distance array; iteratively calculating the distance from each point to the maximum value point until the number of sampling points reaches a preset requirement to generate three-dimensional point cloud key points; processing the three-dimensional point cloud key points using a reparameterizable residual PointNet network to extract geometric features larger than the three-dimensional point cloud key points to obtain a geometric feature vector corresponding to each three-dimensional point cloud key point; projecting the three-dimensional point cloud key points onto an RGB image plane using camera parameters to obtain corresponding color features; fusing the geometric features and color features of the three-dimensional point cloud key points using an adaptive weighted fusion network, wherein the adaptive weighted fusion network calculates weight coefficients representing the reliability of color information based on the geometric features and color features; weighting the color features according to the weight coefficients and then concatenating them with the geometric features; and predicting the six-degree-of-freedom pose of the object based on the fused features.

[0012] According to at least one specific embodiment of the present application, the step of selecting the maximum value point in the multidimensional distance array, iteratively calculating the distance from each point to the maximum value point until the number of sampling points reaches a preset requirement, and generating 3D point cloud key points, further includes: adding the selected initial point to the sampling point set; calculating the minimum distance from each point in the 3D point cloud to all points in the sampling point set in each iteration, and updating the multidimensional distance array; selecting the point corresponding to the current maximum value in the multidimensional distance array as a new sampling point and adding it to the sampling point set; repeating this iterative process until the number of points in the sampling point set reaches the preset requirement, and the sampling point set is the generated 3D point cloud key points.

[0013] According to at least one specific embodiment of the present application, the farthest point sampling algorithm takes any point of the three-dimensional point cloud as the initial point and uses Euclidean distance in three-dimensional space to calculate the distance between the initial point and other points.

[0014] According to at least one specific embodiment of the present application, the step of processing the 3D point cloud keypoints through a reparameterizable residual PointNet network to extract geometric features larger than the 3D point cloud keypoints and obtain a geometric feature vector corresponding to each 3D point cloud keypoint further includes: processing the coordinate data of the 3D point cloud keypoints through a reparameterizable residual submodule; during the training phase of the reparameterizable residual submodule, performing forward propagation and gradient backpropagation through a parallel branch structure containing residual connections to update network parameters; during the inference phase, fusing the weights of the convolutional layers in the parallel branches with the parameters of the batch normalization layers to generate an equivalent single-path convolutional kernel; and transforming the input features through the equivalent single-path convolutional kernel to obtain a geometric feature vector corresponding to each 3D point cloud keypoint.

[0015] According to at least one specific embodiment of the present application, in the process of processing the key points of the 3D point cloud through the reparameterizable residual PointNet network, the coordinates of the key points of the 3D point cloud are input into the reparameterizable residual submodule, and the output features of the reparameterizable residual submodule are upgraded by two channel-dilation convolutional layers; the upgraded features are aggregated based on global average pooling, and the features before aggregation and the features after aggregation are added through a residual connection; based on the result of the addition, the deep geometric information representing each key point of the 3D point cloud is calculated, and the corresponding geometric feature vector is output.

[0016] According to at least one specific embodiment of the present application, the step of projecting 3D point cloud key points onto an RGB image plane using camera parameters to obtain corresponding color features, and fusing the geometric and color features of the 3D point cloud key points through an adaptive weighted fusion network, wherein the adaptive weighted fusion network calculates weight coefficients characterizing the reliability of color information based on the geometric and color features, weights the color features according to the weight coefficients and then concatenates them with the geometric features, and predicts the six-degree-of-freedom pose of the object based on the fused features, further includes: projecting the 3D coordinates of the 3D point cloud key points onto a 2D RGB image plane using a camera intrinsic parameter matrix to obtain the 2D pixel coordinates corresponding to each 3D point cloud key point, and fusing the geometric and color features of the 3D point cloud key points based on the 2D pixel coordinates from the RGB image plane. The global feature map extracted from the B image is sampled using bilinear interpolation to obtain a color feature vector corresponding to each 3D point cloud key point. A fusion intermediate feature is generated based on the color feature vector and the geometric feature vector. This fusion intermediate feature is then passed through another fully connected layer and processed by a normalization function to calculate scalar weight coefficients characterizing the reliability of the color information at that point. Based on the calculated scalar weight coefficients, the color feature vector is weighted element-wise, and the weighted color feature vector is concatenated with the geometric feature vector along the channel dimension to generate the final point-by-point fusion feature vector. Based on the point-by-point fusion feature vector of all key points, a pose regression network is used to calculate the six-degree-of-freedom pose parameters characterizing the object's position and rotation in 3D space.

[0017] According to at least one specific embodiment of the present application, the step of generating a fused intermediate feature based on the color feature vector and the geometric feature vector specifically involves: performing point-by-point fusion processing on the color feature vector and the geometric feature vector through an adaptive weighted fusion network, and adding the color feature vector and the geometric feature vector after passing through a fully connected layer and being transformed by an activation function.

[0018] According to another aspect of the embodiments of this application, a six-DOF pose estimation system based on adaptive weighted feature fusion is provided to implement the aforementioned six-DOF pose estimation method based on adaptive weighted feature fusion. The system includes: a multi-dimensional distance data generation module, which generates a three-dimensional point cloud of a target object based on a depth image; selects a point from the three-dimensional point cloud as an initial point using a farthest point sampling algorithm; calculates the distance from each point in the three-dimensional point cloud to the initial point; and generates a multi-dimensional distance array. A three-dimensional point cloud keypoint generation module selects the maximum value point in the multi-dimensional distance array; iteratively calculates the distance from each point to the maximum value point until the number of sampling points reaches a preset requirement; and a geometric feature vector calculation module processes the three-dimensional point cloud keypoints using a reparameterizable residual PointNet network, extracts geometric features greater than those of the three-dimensional point cloud keypoints, and obtains the geometric feature vector corresponding to each three-dimensional point cloud keypoint. The object six-DOF pose prediction module projects key points of a 3D point cloud onto an RGB image plane using camera parameters to obtain corresponding color features. It then fuses the geometric and color features of the key points of the 3D point cloud using an adaptive weighted fusion network. The adaptive weighted fusion network calculates weight coefficients representing the reliability of color information based on the geometric and color features. After weighting the color features according to the weight coefficients, the network concatenates them with the geometric features and predicts the object's six-DOF pose based on the fused features.

[0019] According to another aspect of the embodiments of this application, an electronic device is provided, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method.

[0020] According to another aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described herein.

[0021] The beneficial technical effects of the embodiments of this application are: The technical solution provided in this application adopts a progressive technical route of data selection, feature extraction, and information fusion. By utilizing the synergy and complementarity of the three core steps, it achieves the goal of real-time pose estimation in complex scenes. It realizes the technical effect of being able to stably and accurately output the spatial position and orientation of objects even when there are interferences such as occlusion, lighting changes, and texture loss. In this way, it solves the technical problems of low pose estimation accuracy and poor environmental adaptability caused by insufficient utilization of point cloud information, insufficient geometric feature extraction capabilities, and a single multimodal data fusion strategy.

[0022] This application embodiment selects multiple spatially distributed key points from the original point cloud using the farthest point sampling algorithm, obtaining the most geometrically representative and informative point set. This provides high-quality raw data for feature extraction, enabling the subsequent reparameterizable residual PointNet network to avoid computation on a large number of redundant or invalid points. This allows for the extraction of deeper and more discriminative geometric feature vectors. The aforementioned geometric key points are then precisely projected onto an RGB image using camera parameters to obtain the corresponding color features. Finally, an adaptive weighted fusion network is used to fuse the two features. Based on the fusion features of geometric and color features, a weight coefficient is dynamically calculated to evaluate the reliability of the current color information. By evaluating and weighting, the complementarity of the two modalities is fully utilized, improving the system's adaptability to complex working environments. Attached Figure Description

[0023] To more clearly illustrate the specific implementation methods of the embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the specific implementation methods or the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion.

[0025] Figure 1a It is a key point map (point cloud data) of the lampshade in the LineMOD dataset.

[0026] Figure 1b This is a key point image (RGB image) of a lampshade from the LineMOD dataset. Figure 1c This is a schematic diagram of the farthest point sampling algorithm.

[0027] Figure 2 It is a reparameterizable residual PointNet network (Rep-Res-PointNet network).

[0028] Figure 3 This is a schematic diagram of the reparameterization process of the Rep-Res module.

[0029] Figure 4 This is a schematic diagram of the structure of a pixel-wise feature adaptive weighted fusion network.

[0030] Figure 5 This is a schematic diagram of the feature fusion submodule.

[0031] Figure 6This is an architecture diagram of a six-degree-of-freedom pose estimation system based on adaptive weighted feature fusion.

[0032] Figure 7 This is a schematic diagram of the electronic device. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the specific implementation methods in the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the embodiments of this application.

[0034] The following section provides a further description of the existing technology solutions in conjunction with the background technology content.

[0035] Existing technologies extract feature information from RGB and depth images separately, and then perform certain post-processing. DenseFusion is a method for accurate 6DoF pose estimation based on RGB-D images. It was proposed by Wang et al. in 2019 and is a typical representative of 6DoF pose estimation algorithms.

[0036] The main contribution of DenseFusion lies in its ability to densely fuse color information from RGB images and geometric information from depth maps, thereby generating a feature representation for each pixel. This dense fusion method differs from traditional pose estimation methods, which typically process color and geometric information independently and then merge them at some stage. DenseFusion is an end-to-end network, mainly divided into two parts: a feature extraction stage and a pose estimation stage. In the feature extraction stage, the network first uses two parallel convolutional networks to process the RGB image and the depth image respectively to extract color and geometric features. For the RGB image, a pre-trained convolutional network is used to extract color features; for the depth map, another convolutional network is used to extract geometric features. In the feature fusion stage, for each pixel in the RGB image, its color features and the corresponding geometric features in the depth image are fused at the pixel level. This step is achieved through a point cloud alignment network, which tightly combines the two types of features to generate a fused feature for each pixel. In the pose estimation stage, the fused features are fed into the pose estimation network, which uses a fully connected layer to predict the object's pose parameters, including rotation (represented by quaternions) and translation vectors. Furthermore, DenseFusion proposes a differentiable pose iteration refinement network, which is more efficient in pose iteration refinement compared to traditional ICP methods, significantly reducing time costs.

[0037] In the initial stage of DenseFusion, semantic segmentation masks and bounding boxes are created from RGB images. The segmented color and depth image information is then encoded. Different feature extraction networks are used to fuse the color and geometric features of each pixel, and pose prediction is performed using these deeply fused features. In this way, the features provided by each pixel participate in pose prediction, and a voting mechanism is used to select the final 6DoF pose prediction result.

[0038] Furthermore, while the ICP algorithm is widely used in various pose estimation tasks for fine-tuning pose, its optimization results often fail to meet real-time requirements in real-time scenarios. Therefore, DenseFusion employs a deep learning-based iterative optimization module, which can improve the final pose prediction results with faster speed and better robustness.

[0039] like Figure 1 As shown in the embodiments of this application, the six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion includes the following steps: Step S1: Generate a 3D point cloud of the target object based on the depth image. Select a point from the 3D point cloud as an initial point using the farthest point sampling algorithm. Calculate the distance from each point in the 3D point cloud to the initial point to generate a multi-dimensional distance array.

[0040] Step S2: Select the maximum value point in the multidimensional distance array, and iteratively calculate the distance from each point to the maximum value point until the number of sampling points reaches the preset requirement, thereby generating the key points of the three-dimensional point cloud.

[0041] Step S3: Process the key points of the 3D point cloud using a reparameterizable residual PointNet network, extract geometric features larger than the key points of the 3D point cloud, and obtain the geometric feature vector corresponding to each key point of the 3D point cloud.

[0042] Step S4: Project the key points of the 3D point cloud onto the RGB image plane using camera parameters to obtain the corresponding color features. Then, fuse the geometric and color features of the key points of the 3D point cloud using an adaptive weighted fusion network. The adaptive weighted fusion network calculates weight coefficients that characterize the reliability of color information based on the geometric and color features. After weighting the color features according to the weight coefficients, it is concatenated with the geometric features. The six-degree-of-freedom pose of the object is predicted based on the fused features.

[0043] The technical solutions provided in steps S1 to S4, through data selection, feature enhancement, and fusion processing, focus on the problem of 6DoF pose estimation of objects based on RGB-D images. The goal is to accurately predict the 6DoF pose of objects with low texture or partial occlusion in complex environments, while ensuring good robustness and inference speed. This achieves the effect of predicting the six degrees of freedom pose of objects. Its technical effects can be summarized as follows: Steps S1 and S2 employ the farthest point sampling algorithm to select the most widely distributed and geometrically representative 3D point cloud key points from the original point cloud. The selected 3D point cloud key points provide high-quality raw data for the reparameterizable residual PointNet network in step S3. The reparameterizable residual PointNet network can process the selected 3D point cloud key points with its powerful feature extraction capabilities. During data processing, depth is ensured through residual connections, and reparameterization balances training performance and inference efficiency, ultimately extracting deeper geometric feature vectors.

[0044] Step S3 outputs a high-quality geometric feature vector based on the high-quality raw data provided in steps S1 and S2. This vector, along with the color features obtained from the RGB image through camera projection, uses the geometric features extracted in the previous steps as one of the benchmarks and combines them with the color features to dynamically evaluate the reliability of the current color information and calculate adaptive weights. This enables the system to intelligently cope with complex interferences such as changes in illumination and occlusion, providing high-quality input data for the adaptive weighted fusion network in step S4.

[0045] In step S4, the key points of the 3D point cloud are projected onto the RGB image plane using camera parameters. Based on the precise 3D coordinates provided in steps S1 and S2, the acquisition of color features can achieve pixel-level spatial alignment with geometric features. Step S4 also obtains a reliable geometric judgment benchmark based on the reparameterizable residual PointNet network provided in step S3, which can extract high-dimensional and deep geometric feature vectors. Through the deep structure and reparameterization design provided by the reparameterizable residual PointNet network, the geometric features represented by the geometric feature vectors can stably characterize the shape and spatial structure of the object, with less influence and interference from texture and lighting changes.

[0046] The weighting coefficients representing the reliability of color information in step S4 are based on the geometric features provided in step S3, and can be used to perform intelligent judgment through an adaptive weighted fusion network. For example: When determining whether the RGB appearance (color features) of an object is consistent with its inherent geometric structure (geometric features), if the object's surface suffers from color information distortion or loss due to strong reflections, shadows, or occlusion, the corresponding geometric features remain stable and reliable because the system possesses sufficient robustness. The fusion network can simultaneously analyze both RGB appearance and geometric structure features, identifying inconsistencies and generating a corresponding lower color weight. This automatically suppresses unreliable color information during fusion, relying more on reliable geometric information. Conversely, in areas with clear texture and normal lighting, where geometric and color features are highly consistent, the network assigns higher color weights, fully utilizing rich texture information to correctly predict the object's six-DOF pose.

[0047] In summary, the technical solutions provided in steps S1 to S4, through data selection, feature extraction, and information fusion, achieve real-time pose estimation in complex scenes. They enable stable and accurate output of the spatial position and orientation of objects even when there are interferences such as occlusion, lighting changes, and texture loss. This solves the technical problems of low pose estimation accuracy and poor environmental adaptability caused by insufficient utilization of point cloud information and insufficient geometric feature extraction capabilities. It fully utilizes the advantages of multimodal data fusion and combines the complementary advantages of different modalities, thereby improving the ability to predict the six-degree-of-freedom pose of objects in complex working conditions.

[0048] Preferably, in step S2, selecting the maximum value point in the multidimensional distance array and iteratively calculating the distance from each point to the maximum value point until the number of sampling points reaches a preset requirement to generate 3D point cloud key points further includes: Step S21: Add the selected initial point to the sampling point set; in each iteration, calculate the minimum distance from each point in the 3D point cloud to all points in the sampling point set, and update the multidimensional distance array. The farthest point sampling algorithm uses any point in the 3D point cloud as the initial point and calculates the distance between the initial point and other points using Euclidean distance in 3D space.

[0049] Step S22: Select the point corresponding to the current maximum value in the multidimensional distance array as a new sampling point and add it to the sampling point set.

[0050] Step S23: Repeat the iterative process until the number of points in the sampling point set reaches the preset requirement. The sampling point set is the generated 3D point cloud key points.

[0051] The optimization solutions provided in steps S21 to S23 employ an iterative active selection mechanism, using spatial distance as the criterion, to filter out a subset of points with optimal spatial representativeness from the original 3D point cloud. In steps S21 to S23, a farthest-point sampling algorithm is used, rather than random or uniform downsampling; it consistently selects the point farthest from the currently selected point set to add, ensuring the widest possible distribution coverage of the sampled point set in 3D space. Specifically: Step S21 initializes and updates distances. By setting the algorithm's starting point (an arbitrary initial point) and metric (Euclidean distance), it calculates the minimum distance from each point to all points in the sampled point set and updates the distance array. Step S21 provides the decision basis for step S22. Each element in the updated distance array L reflects the distance between the corresponding point and the selected set. Step S21 continuously maintains this global distance information. In step S22, by reading the distance array L provided by step S21, the point corresponding to the maximum value in the array is selected. Reading the point corresponding to the maximum value again maximizes the filling of gaps in the current selected set's space. The newly added points are then fed back to step S21 in the next iteration, changing the distance pattern of the entire point cloud and driving the selection process forward. Step S23 controls the stopping condition of the entire algorithm; iteration stops when the number of samples reaches a preset requirement, ensuring that the algorithm outputs a controllable set of key points that meets the needs of subsequent processing.

[0052] like Figure 1a As shown in the example, when performing 6DoF pose estimation for an object, effectively utilizing the object's geometric information is crucial for improving the accuracy and efficiency of the estimation, especially in complex scenes. Directly selecting N points randomly from an image or point cloud as keypoints for pose estimation is often inefficient and may affect the accuracy of the estimation results due to the unrepresentative nature of the selected points. Conversely, a method based on feature extraction from sampling points at the object's geometric boundaries can more effectively capture the object's geometric features, thereby improving the performance of pose estimation. The core of this method lies in using keypoints at the object's geometric boundaries as the basis for pose estimation. These keypoints are mainly distributed on the object's outer edges and are important sources of information describing the object's shape and spatial position. By accurately selecting these points on the geometric boundaries, not only can the efficiency of geometric information extraction be improved, but the number of required sampling points can also be reduced, thus improving the algorithm's real-time performance.

[0053] When estimating the pose of a target object using only RGB images, 6DoF pose estimation is typically achieved by identifying keypoints of the target object in the 2D image. However, projecting a rigid 3D object onto a 2D image not only results in the loss of some geometric information but also causes overlapping projections of different keypoints of the target object, thus increasing the difficulty of pose estimation. When estimating the pose of a target object using 3D point cloud data, corner points on the 3D bounding box of the target object are often used as keypoints for pose estimation. However, since these corner points are virtual points of the target object, it becomes difficult for the network to extract information about the target object in the vicinity of these points, leading to significant estimation errors. Figure 1a The image shown is a schematic diagram illustrating the keypoint selection for the "lampshade" in the LineMOD dataset, where orange dots represent selected keypoints. Figure 1a The key points of the point cloud are mainly distributed at the geometric boundaries of the target object. Figure 1b When the orange keypoints are projected onto a 2D image, some overlap occurs between the keypoints.

[0054] like Figure 1c As shown, to effectively select these key points, this embodiment employs the FPS algorithm. This algorithm selects the center of the object's point cloud as the initial key point, and then iteratively adds the point furthest from the selected key point to the key point set. Each iteration uses the newly added point as the new initial point until N key points are collected. This method not only ensures a wide and uniform distribution of the key point set but also effectively covers the geometric features of the object. For 3D point cloud data, Euclidean distance is generally used to measure the distance between points, i.e., the straight-line distance between two points in space. Figure 1c This is a schematic diagram of the algorithm for sampling the farthest feature point. The specific process of the algorithm is shown below: 1) For those containing Input a point cloud of points, and select a point in the point cloud. As the initial point, a set of sampling points is formed. ; 2) For each point to Calculate the distance and generate Distance array of dimensions Next, select the point in the array that corresponds to the maximum distance. And update the sampling point set accordingly. ; 3) For each point to The distance is calculated, and for each point... If it arrives The distance is less than So, update Therefore, array Always keep the set of sampling points for each point Distance to the nearest point in the middle; 4) Select array The point corresponding to the maximum value is taken as And update the sampling point set accordingly. ; 5) Repeat steps 2) to 4) until the desired result is reached. Until the required number of sampling points are reached.

[0055] As can be seen from the above, the features extracted from these carefully selected key points can provide a strong basis for 6DoF pose estimation. This key point selection and feature extraction strategy based on geometric boundary points can not only improve the accuracy of pose estimation compared with the traditional random sampling method, but also reduce the consumption of computing resources while ensuring real-time performance, providing a more effective and practical solution for pose estimation in complex scenarios.

[0056] Preferably, in step S3, the step of processing the 3D point cloud key points using a reparameterizable residual PointNet network to extract geometric features larger than the 3D point cloud key points and obtain the geometric feature vector corresponding to each 3D point cloud key point further includes: Step S31: Based on the coordinate data of the key points of the 3D point cloud, the data is processed through the reparameterized residual submodule. During the training phase of the reparameterized residual submodule, forward propagation and gradient backpropagation are performed through a parallel branch structure containing residual connections to update the network parameters. Step S32: During the inference phase, the weights of the convolutional layers in the parallel branch are fused with the parameters of the batch normalization layer to generate an equivalent single-path convolutional kernel; the input features are transformed by the equivalent single-path convolutional kernel to obtain the geometric feature vector corresponding to each 3D point cloud key point.

[0057] For example, it further includes: during the process of processing the key points of the 3D point cloud through the reparameterizable residual PointNet network, the coordinates of the key points of the 3D point cloud are input into the reparameterizable residual submodule, and the output features of the reparameterizable residual submodule are up-dimensionalized through two channel-dilation convolutional layers.

[0058] The upscaled features are aggregated using global average pooling, and the features before aggregation are added to the features after aggregation through a residual connection.

[0059] Based on the summed result, the deep geometric information representing each 3D point cloud key point is calculated, and the corresponding geometric feature vector is output.

[0060] In the optimization scheme provided in steps S31 to S32, the reparameterizable residual PointNet network integrates two techniques: structural reparameterization and residual connections with hierarchical processing. During training, complex parallel branches (including residual connections) are used to enhance model capacity and optimization capabilities. During inference, multiple branches are fused into a single efficient convolution through mathematical equivalence transformations, balancing performance and speed. Residual connections are introduced at multiple levels (within submodules and after global pooling) to promote gradient flow and feature reuse. A multi-stage processing flow (submodule processing, channel expansion, pooling aggregation) from coordinate input to feature output is designed to gradually extract geometric information. Step S31, during the training phase, provides multiple forward and gradient backpropagation paths through a parallel branching structure including residual connections, increasing the network's representational power and optimization space, which helps learn more complex and accurate geometric patterns. In the inference phase of step S3, an equivalent single-path convolutional kernel is generated by fusing the convolutional layer weights with the batch normalization (BN) layer parameters. This mathematical transformation eliminates the conditional judgments and additional computations caused by branching while fully preserving the network's functionality. It can be seen that step S31 is responsible for maximizing the network's performance potential during training, while step S32 is responsible for deploying the trained capabilities in the simplest and fastest way. Through the synergy of steps S31 and S32, the contradiction between the high accuracy requirements and the low latency and high efficiency deployment requirements of deep learning models is resolved.

[0061] like Figure 2 As shown, as an example, the reparameterizable residual PointNet network (Rep-Res-PointNet network) in step S3 is further explained below: Traditional methods treat depth images as additional image channels to RGB images using convolutional neural networks; however, this approach ignores the inherent three-dimensional structure of the depth channels. In contrast, this embodiment first uses known camera characteristics to convert segmented depth pixels into 3D point clouds, and then uses a PointNet-like architecture to extract geometric features, overcoming the shortcomings of traditional methods.

[0062] Rep-Res-PointNet Network Structure: This application improves upon the PointNet geometric feature extraction part of the baseline network DenseFusion by designing the Rep-Res-PointNet geometric feature extraction network. Specifically, the Rep-Res-PointNet network structure improves upon the PointNet part of the DenseFusion network. Rep-Res-PointNet enhances the network's ability to process point cloud data by embedding two Rep-Res sub-modules. Then, it performs two channel expansions on the features output by the Rep-Res sub-modules through convolution, and finally adds residual connections after average pooling to output the result.

[0063] Through the designed network structure, the improved model has achieved results in improving feature extraction efficiency, alleviating the gradient vanishing problem, and increasing network depth without causing performance degradation. At the same time, it has also optimized the computational efficiency of the model, making it more suitable for practical application needs.

[0064] The Rep-Res module: In the DenseFusion framework, the PointNet model used includes four convolutional layers and one pooling layer. This relatively shallow network structure limits its ability to extract deep feature information, thus affecting the network's performance. Unlike current mainstream methods that improve accuracy by designing complex and detailed feature extraction modules, this application proposes a Rep-Res module. This module is designed based on a residual structure and can be reparameterized. The Rep-Res module aims to enhance the geometric feature extraction capability by increasing the depth of the PointNet network within DenseFusion. Its concise design also helps control the number of parameters in the entire improved DenseFusion network, achieving a balance between performance and efficiency.

[0065] Detailed Explanation of the Rep-Res Module Structure: like Figure 3 As shown, the network architecture proposed in this application improves the performance of deep networks through meticulous design, avoiding the problems of gradient vanishing and network degradation that occur as the network deepens. In the Rep-Res module structure, the core construction consists of a 1×1 convolutional kernel, followed by a batch normalization layer and an activation layer. This configuration sequence is concatenated twice to build the basic module. This design aims to reduce the number of parameters by minimizing the convolutional kernel size while maintaining the network's expressive power.

[0066] To address the vanishing gradient problem common in deep networks, the Rep-Res module employs a strategy of cross-layer residual connections. This design allows gradient flow to proceed directly from the back end to the front end of the network, effectively preserving gradient flow and thus avoiding the vanishing gradient problem during training. Furthermore, residual connections also mitigate network degradation, i.e., the phenomenon where performance no longer improves or even declines as network depth increases.

[0067] Short-circuit connections are introduced before the two activation layers to connect features from different convolutional layers. This allows the network to effectively fuse high-level abstract features with low-level detailed features, which not only increases feature richness but also enhances the network's ability to represent input data. Fusing features from different levels helps the network better understand and process complex data, thereby improving overall model performance. Considering the limitations of computational resources in practical applications, especially in scenarios requiring fast inference, the short-circuit connections are merged into the convolutional branches using reparameterization. The application of reparameterization not only reduces the model's inference time but also lowers its computational cost while maintaining performance. This allows the network to maintain high accuracy and be suitable for resource-constrained deployment environments, meeting the dual requirements of real-time performance and efficiency.

[0068] The Rep-Res computation process for the input feature map is as follows: MERGEFORMAT (4-1) MERGEFORMAT (4-2) In the formula, Represents input features, represent Convolution kernel, Indicates BN layer, This represents the processing result of the first activation layer. This represents the final output of Rep-Res.

[0069] 2) The structural reparameterization process of the Rep-Res submodule The specific process of reparameterizing the Rep-Res submodule mainly consists of three steps:

[0070] First, the short-circuit connection branch can be represented as a convolutional kernel with a value of 1 on the left diagonal and a kernel of 0 at all other positions. Convolution kernel.

[0071] Secondly, respectively and Incorporating convolution kernels and

[53] . The equivalent convolution after fusing convolution and BN layers is: MERGEFORMAT(4-3) In the formula, The weights represent the original convolution. , , , These represent the mean, variance, learning rate factor, and bias of the BN layer. The weights represent the equivalent convolution. This represents the deviation from the equivalent convolution. (The sentence is incomplete and requires more context to translate accurately.) Equivalent to the weights of the fused convolution. This is equivalent to the deviation of the fusion convolution, thus completing the fusion of the BN layer and the convolution.

[0072] Finally, since the convolutions on both branches have the same size, they are added together and merged into a single convolution: MERGEFORMAT (4-4) The final result This refers to convolution during the inference phase.

[0073] Preferably, in step S4, the step of projecting the key points of the 3D point cloud onto the RGB image plane using camera parameters to obtain the corresponding color features, and fusing the geometric and color features of the key points of the 3D point cloud using an adaptive weighted fusion network, wherein the adaptive weighted fusion network calculates weight coefficients representing the reliability of color information based on the geometric and color features, and then weights the color features according to the weight coefficients and concatenates them with the geometric features, and predicts the six-degree-of-freedom pose of the object based on the fused features, further includes: Step S41: Project the three-dimensional coordinates of the key points of the three-dimensional point cloud onto the two-dimensional RGB image plane using the camera intrinsic parameter matrix to obtain the two-dimensional pixel coordinates corresponding to each key point of the three-dimensional point cloud. Based on the two-dimensional pixel coordinates, obtain the color feature vector corresponding to each key point of the three-dimensional point cloud by bilinear interpolation sampling from the global feature map extracted from the RGB image.

[0074] Step S42: Generate a fusion intermediate feature based on the color feature vector and the geometric feature vector. Pass the fusion intermediate feature through another fully connected layer and process it through a normalization function to calculate the scalar weight coefficient that characterizes the reliability of the color information at that point.

[0075] Step S43: Based on the calculated scalar weight coefficients, the color feature vector is weighted element by element, and the weighted color feature vector is concatenated with the geometric feature vector by channel dimension to generate the final point-by-point fused feature vector; based on the point-by-point fused feature vector of all key points, the six-degree-of-freedom pose parameters representing the position and rotation of the object in three-dimensional space are calculated by the pose regression network.

[0076] As an example, in step S41, generating a fusion intermediate feature based on the color feature vector and the geometric feature vector specifically involves: performing point-by-point fusion processing on the color feature vector and the geometric feature vector through an adaptive weighted fusion network, and adding the color feature vector and the geometric feature vector after passing through a fully connected layer and being transformed by an activation function.

[0077] The optimization techniques provided in steps S41 to S43 achieve adaptive, data-driven intelligent fusion of multimodal information (color features and geometric features), and calculate the six-degree-of-freedom pose parameters used to characterize the object's position and rotation in three-dimensional space, wherein: Step S41 aligns and samples the 3D geometric keypoints onto the 2D image plane using camera intrinsics and obtains the corresponding color feature vectors through bilinear interpolation. This ensures a strict semantic and spatial correspondence between the color feature vectors and the geometric feature vectors, preventing the adaptive mechanism in subsequent steps from failing. Step S42 receives the feature information provided in Step S41, generates a fusion intermediate feature through a neural network, and calculates scalar weight coefficients. Then, Step S43 performs element-wise weighting of the color feature vectors based on the scalar weight coefficients and concatenates them with the geometric features. If the weights are high, more color information is retained; conversely, if the weights are low, color information is suppressed, and reliance on geometric information is maintained.

[0078] In summary, steps S41 to S42, through collaborative processing, constitute a closed-loop intelligent decision-making unit from decision-making to execution. Step S41 can make fine adjustments based on the specific situation of each point in the 3D point cloud keypoints, determining whether the object is in a shadow or reflective area. Then, through subsequent steps, it achieves a process from progressively refined processing to global aggregated decision-making, ensuring that the regression network can make more accurate and robust global spatial judgments based on these optimized local evidences. The optimization techniques provided by steps S41 to S43 can automatically resist unreliable information, such as the tendency of certain colors to change, and prevent errors in occluded environments, emphasizing the contribution of geometric information. Ultimately, this significantly improves the accuracy, robustness, and generalization ability of six-DOF pose estimation in challenging real-world scenarios.

[0079] The following provides a detailed explanation of step S4 in conjunction with specific implementation methods. Step S4 first employs an adaptive weighted fusion network for RGB and point cloud features. To overcome the negative impact of instance segmentation errors on the feature extraction stage, this embodiment adopts an innovative method in the feature fusion stage of step S4. Instead of simply fusing the global information of color features and geometric features directly, a pixel-level adaptive weighted fusion network is designed, aiming to perform feature fusion in a more refined and accurate manner. The core idea of ​​the adaptive weighted fusion network is to first perform pixel-by-pixel fusion processing on local features, and then concatenate these local features with global information. This method not only increases the amount of information carried by each key point, but also enriches points with semantic image features by utilizing point cloud data point by point, achieving adaptive evaluation of the importance of color information during the fusion process.

[0080] Specifically, the adaptive weighted fusion network first analyzes each pixel independently, combining color features with corresponding geometric features in the point cloud to form a rich local feature representation. Then, the local features are concatenated with the global features. In this way, the adaptive weighted fusion network not only preserves local details but also understands and utilizes the global structure of the entire scene. This is crucial for accurately understanding the spatial relationships and contextual information of objects, helping to improve the accuracy and reliability of the final feature representation. Furthermore, the adaptive weighted fusion network incorporates an adaptive mechanism to evaluate the relative importance of color and geometric information during the fusion process. This means the network can dynamically adjust the contribution ratio of color and geometric information in the feature representation based on the specific task and data characteristics. By enhancing its adaptive capabilities, the network can prioritize geometric information when dealing with scenes where color information may be unreliable due to factors such as lighting changes or occlusion, thereby improving the robustness of feature fusion.

[0081] The pixel-level feature adaptive weighted fusion network provided in step S4 not only optimizes the feature fusion effect at both the local and global levels, but also endows the adaptive weighted fusion network with the ability to adaptively evaluate and utilize different types of information, thereby greatly improving the richness and reliability of features and providing a solid foundation for subsequent pose estimation tasks, especially in complex and variable environments.

[0082] like Figure 4As shown, further explanation of the pixel-level feature adaptive weighted fusion network structure follows: The main task in current research is to effectively fuse dense features extracted from image and 3D point cloud inputs to accurately estimate the pose of the target object. Considering that simply merging color and depth features into a single global feature vector may not adequately address occlusion and segmentation errors in complex scenes, this application proposes a pixel-level feature adaptive weighted fusion network. This network aims to handle fusion tasks more finely, especially in cases of severe occlusion and inaccurate segmentation, effectively integrating extracted features and improving estimation accuracy.

[0083] The pixel-level feature adaptive weighted fusion network operates by fusing pixels one by one, allowing the network to make independent predictions at each pixel level based on the fused features. The advantage of this method is that it can fully utilize the visible parts of objects for prediction, minimizing the negative impact of occlusion and segmentation errors on estimation performance. Specifically, firstly, the geometric features of each point in the point cloud are matched with the color features of its corresponding pixel on the image plane. This process utilizes the camera's intrinsic parameters to project points in 3D space onto the 2D image plane, resulting in local feature pairs. Then, the color and geometric features are fed into a feature fusion submodule for adaptive weighted fusion processing, further generating more reliable local features. These features are then fed into another deep network for further processing. At this stage, a symmetric reduction function is used to extract and generate a fixed-size global feature vector from these feature pairs. This global feature vector not only contains information extracted from each dense pixel feature but also incorporates global background information, thus providing the network with a richer and more comprehensive information view.

[0084] like Figure 5 As shown, an important concept in this application embodiment is the feature fusion submodule. The core of the pixel-level feature adaptive weighted fusion network is the point cloud-guided feature fusion submodule. The feature fusion submodule consists of a mesh generator, an image sampler, and a fusion layer. The feature fusion submodule includes two parts: point-to-point correspondence generation and point cloud-guided fusion layer.

[0085] Specifically, the point cloud points are projected onto the camera image, and the mapping matrix is ​​denoted as... Point cloud data and mapping matrix By inputting the mesh generator, point-to-point mappings between point clouds and camera images at various resolutions can be generated. Specifically, for any specific point within the point cloud... This method can determine its corresponding position in the camera image. The relationship is as follows: MERGEFORMAT (4-5) After establishing the point-to-point mapping, an image sampler is used to extract the semantic features of each point. Specifically, the image sampler samples the locations of the points. and image features As input, generate corresponding image features for each sampling point. .

[0086] Since the sampling positions may not correspond perfectly to each pixel, bilinear interpolation is used to obtain the image features that fall within the coordinates between pixels. This can be formulated as follows: MERGEFORMAT (4-6) In the formula, For point Corresponding image features It is a bilinear interpolation function. Sampling location Image features of adjacent pixels.

[0087] While camera-captured image data is rich in information, it is often affected by variations in lighting, occlusion, and other visual noise. This can cause features extracted directly from image data to contain misleading information, negatively impacting task performance. In contrast, point cloud data provides stable geometric information but lacks rich surface texture and color information. Therefore, designing a mechanism to evaluate and utilize features from both types of data to improve the overall performance of the system has become a challenging problem.

[0088] To address this issue, embodiments of this application design a fusion layer, which adaptively evaluates the importance of image features at each point and adjusts the fusion strategy between point cloud features and image features accordingly. Specifically, the fusion layer begins by combining point cloud features... Image features corresponding to each point The features are fed in parallel into a fully connected layer and processed using the tanh activation function. The purpose of this step is to map the two sets of features to the same dimensional space for further processing. Subsequently, the two mapped features are combined element-wise to form a compact feature representation that integrates point cloud geometric information and image visual information. This compact feature representation is then fed into another fully connected layer, whose task is to compress the features into a single-channel weight map. This weight map represents the relative importance of image features at each point, and it is calculated by considering the combined information of point cloud features and image features. Finally, the weight map is normalized using the sigmoid activation function, limiting the weight values ​​to the range [0,1]. The calculation process is shown in the following formula.

[0089] MERGEFORMAT (4-7) in , , This represents the learnable weight matrix in the Fusion layer. This represents the sigmoid activation function.

[0090] In this way, the weight value at each point can intuitively represent the contribution of the image features at that point to the final task. A high weight value means that the image features at the corresponding point are more important to the task, while a low weight value indicates that the image features may be more disturbed and their contribution to the task is relatively small.

[0091] In obtaining the weighted graph Then, the point cloud features With semantic image features By stitching together the data, more reliable local features can be generated, which can be formalized as follows: MERGEFORMAT (4-8) In this way, the fusion layer can adaptively evaluate and utilize image features and point cloud features, and can dynamically adjust the feature fusion strategy when faced with complex situations such as changes in lighting and occlusion, thereby improving the robustness and performance of the overall system.

[0092] For the method steps disclosed in the above embodiments, the method steps are described as a series of actions for the purpose of simplicity. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of this application.

[0093] Any flowchart or other description of a process or method can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed and implemented not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, or in accordance with program structures such as loops, branches, etc., as will be readily understood by those skilled in the art when implementing the embodiments of this application.

[0094] like Figure 6 As shown, this application provides a six-DOF pose estimation system based on adaptive weighted feature fusion, used to implement the six-DOF pose estimation method based on adaptive weighted feature fusion described in any specific embodiment of this application, including: The multidimensional distance data generation module generates a 3D point cloud of the target object based on the depth image. It selects a point from the 3D point cloud as an initial point using the farthest point sampling algorithm, calculates the distance from each point in the 3D point cloud to the initial point, and generates a multidimensional distance array.

[0095] The 3D point cloud key point generation module selects the maximum value point in the multi-dimensional distance array, iteratively calculates the distance from each point to the maximum value point, until the number of sampling points reaches the preset requirement, and generates 3D point cloud key points.

[0096] The geometric feature vector calculation module processes the key points of the 3D point cloud through a reparameterizable residual PointNet network, extracts geometric features larger than the key points of the 3D point cloud, and obtains the geometric feature vector corresponding to each key point of the 3D point cloud.

[0097] The object six-DOF pose prediction module projects key points of a 3D point cloud onto an RGB image plane using camera parameters to obtain corresponding color features. It then fuses the geometric and color features of the key points of the 3D point cloud using an adaptive weighted fusion network. The adaptive weighted fusion network calculates weight coefficients representing the reliability of color information based on the geometric and color features. After weighting the color features according to the weight coefficients, the network concatenates them with the geometric features and predicts the object's six-DOF pose based on the fused features.

[0098] The implementation methods of the system described above are merely illustrative. For example, the various functional modules, units, or subsystems within the system may or may not be physically separate, or they may or may not be physical units; that is, they may be located in the same place or distributed across multiple different systems and their subsystems or modules. Those skilled in the art can select some or all of the functional modules, units, or subsystems to achieve the objectives of the embodiments of the present invention according to actual needs. Those skilled in the art can understand and implement the above-described situations without any creative effort.

[0099] like Figure 7 As shown, this application provides a six-degree-of-freedom pose estimation method and system based on adaptive weighted feature fusion, and also provides corresponding electronic devices and computer program products: An electronic device includes: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method.

[0100] A computer program product includes a computer program that, when executed by a processor, implements the method.

[0101] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 7 As shown, device 800 includes a processor 801, a memory 802, a communication interface 803, and a bus 804. The processor 801, memory 802, and communication interface 803 communicate via bus 804, or via other means such as wireless transmission. The memory 802 stores instructions, and the processor 801 executes the instructions stored in the memory 802. The memory 802 stores program code 8021, and the processor 801 can call the program code 8021 stored in the memory 802 to execute the steps of the six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion.

[0102] It should be understood that in the embodiments of this application, processor 801 may be a CPU, or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors may be microprocessors or any conventional processors, etc.

[0103] The memory 802 may include read-only memory (ROM) and random access memory (RAM), and provides instructions and data to the processor 801. The memory 802 may also include non-volatile random access memory. The memory 802 may be volatile memory or non-volatile memory, or may include both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0104] In addition to the data bus, bus 804 may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 804 in the diagram.

[0105] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive (SSD).

[0106] In the description of the embodiments of this application, the reference to terms such as "an embodiment," "example," "specific example," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0107] Furthermore, the technical solutions of the various implementation methods in this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the embodiments of this application.

[0108] All features disclosed in the embodiments of this application, or all steps in the disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps. Any feature disclosed in the specification of the embodiments of this application, unless specifically stated otherwise, may be replaced by other equivalent or similar alternative features. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features. Throughout the specification, the same reference numerals indicate the same elements.

[0109] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification of embodiments (including the corresponding claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification of embodiments (including the corresponding claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them. Although the embodiments of this application have been described in detail with reference to the foregoing specific embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein, and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the specific embodiments of this application.

Claims

1. A six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion, characterized in that, include: A 3D point cloud of the target object is generated based on the depth image. A point is selected from the 3D point cloud as an initial point using the farthest point sampling algorithm. The distance from each point in the 3D point cloud to the initial point is calculated to generate a multi-dimensional distance array. Select the maximum value point in the multidimensional distance array, and iteratively calculate the distance from each point to the maximum value point until the number of sampling points reaches the preset requirement, thereby generating key points of a three-dimensional point cloud. The key points of the 3D point cloud are processed by the reparameterizable residual PointNet network to extract geometric features larger than the key points of the 3D point cloud, and to obtain the geometric feature vector corresponding to each key point of the 3D point cloud. The key points of the 3D point cloud are projected onto the RGB image plane using camera parameters to obtain the corresponding color features. The geometric and color features of the key points of the 3D point cloud are then fused using an adaptive weighted fusion network. The adaptive weighted fusion network calculates weight coefficients that characterize the reliability of color information based on the geometric and color features. The color features are then weighted according to the weight coefficients and concatenated with the geometric features. The six-degree-of-freedom pose of the object is predicted based on the fused features.

2. The six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion according to claim 1, characterized in that, The step of selecting the maximum value point in the multidimensional distance array, iteratively calculating the distance from each point to the maximum value point until the number of sampling points reaches a preset requirement, and generating 3D point cloud key points, further includes: The selected initial point is added to the sampling point set; in each iteration, the minimum distance from each point in the 3D point cloud to all points in the sampling point set is calculated, and the multidimensional distance array is updated; Select the point corresponding to the current maximum value in the multidimensional distance array as a new sampling point and add it to the sampling point set; Repeat this iterative process until the number of points in the sampling point set reaches the preset requirement, and the sampling point set is the generated 3D point cloud key points.

3. The six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion according to claim 2, characterized in that, The farthest point sampling algorithm takes any point in the three-dimensional point cloud as the initial point and uses Euclidean distance in three-dimensional space to calculate the distance between the initial point and other points.

4. The six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion according to claim 1, characterized in that, The step of processing the 3D point cloud key points using a reparameterizable residual PointNet network to extract geometric features larger than the 3D point cloud key points and obtain the geometric feature vector corresponding to each 3D point cloud key point further includes: Based on the coordinate data of the key points of the 3D point cloud, the data is processed through the reparameterized residual submodule. During the training phase of the reparameterized residual submodule, forward propagation and gradient backpropagation are performed through a parallel branch structure containing residual connections to update the network parameters. During the inference phase, the weights of the convolutional layers in the parallel branches are fused with the parameters of the batch normalization layer to generate an equivalent single-path convolutional kernel. The input features are then transformed using the equivalent single-path convolutional kernel to obtain the geometric feature vector corresponding to each 3D point cloud key point.

5. The six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion according to claim 4, characterized in that, In the process of processing the key points of the 3D point cloud through the reparameterizable residual PointNet network, the coordinates of the key points of the 3D point cloud are input into the reparameterizable residual submodule, and the output features of the reparameterizable residual submodule are upgraded by two channel-dilated convolutional layers. The upscaled features are aggregated using global average pooling, and the features before aggregation are added to the features after aggregation through a residual connection. Based on the summed result, the deep geometric information representing each 3D point cloud key point is calculated, and the corresponding geometric feature vector is output.

6. The six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion according to claim 1, characterized in that, The process involves projecting key points of a 3D point cloud onto an RGB image plane using camera parameters to obtain corresponding color features. An adaptive weighted fusion network then fuses the geometric and color features of the key points. This adaptive weighted fusion network calculates weight coefficients representing the reliability of the color information based on the geometric and color features. The color features are then weighted according to these weight coefficients and concatenated with the geometric features. The fused features are then used to predict the six-degree-of-freedom pose of the object. The process further includes: The three-dimensional coordinates of the key points of the three-dimensional point cloud are projected onto the two-dimensional RGB image plane using the camera intrinsic parameter matrix to obtain the two-dimensional pixel coordinates corresponding to each key point of the three-dimensional point cloud. Based on the two-dimensional pixel coordinates, the color feature vector corresponding to each key point of the three-dimensional point cloud is obtained by bilinear interpolation sampling from the global feature map extracted from the RGB image. A fusion intermediate feature is generated based on the color feature vector and the geometric feature vector. The fusion intermediate feature is then passed through another fully connected layer and processed by a normalization function to calculate the scalar weight coefficients that characterize the reliability of the color information at that point. Based on the calculated scalar weight coefficients, the color feature vector is weighted element by element, and the weighted color feature vector is concatenated with the geometric feature vector by channel dimension to generate the final point-by-point fused feature vector; based on the point-by-point fused feature vector of all key points, the pose regression network calculates the six-degree-of-freedom pose parameters representing the position and rotation of the object in three-dimensional space.

7. The six-degree-of-freedom pose estimation method based on adaptive weighted feature fusion according to claim 6, characterized in that, The step of generating a fusion intermediate feature based on the color feature vector and the geometric feature vector specifically involves: performing point-by-point fusion processing on the color feature vector and the geometric feature vector through an adaptive weighted fusion network, and adding the color feature vector and the geometric feature vector together after passing through a fully connected layer and being transformed by an activation function.

8. A six-DOF pose estimation system based on adaptive weighted feature fusion, used to implement the six-DOF pose estimation method based on adaptive weighted feature fusion as described in any one of claims 1 to 7, characterized in that, include: The multidimensional distance data generation module generates a 3D point cloud of the target object based on the depth image. It selects a point from the 3D point cloud as an initial point using the farthest point sampling algorithm, calculates the distance from each point in the 3D point cloud to the initial point, and generates a multidimensional distance array. The 3D point cloud key point generation module selects the maximum value point in the multi-dimensional distance array, iteratively calculates the distance from each point to the maximum value point, until the number of sampling points reaches the preset requirement, and generates 3D point cloud key points. The geometric feature vector calculation module processes the key points of the 3D point cloud through a reparameterizable residual PointNet network, extracts geometric features larger than the key points of the 3D point cloud, and obtains the geometric feature vector corresponding to each key point of the 3D point cloud. The object six-DOF pose prediction module projects key points of a 3D point cloud onto an RGB image plane using camera parameters to obtain corresponding color features. It then fuses the geometric and color features of the key points of the 3D point cloud using an adaptive weighted fusion network. The adaptive weighted fusion network calculates weight coefficients representing the reliability of color information based on the geometric and color features. After weighting the color features according to the weight coefficients, the network concatenates them with the geometric features and predicts the object's six-DOF pose based on the fused features.

9. An electronic device, characterized in that, include: The system includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 6.

Images