9D pose estimation method and device based on multi-frame image feature fusion

Abstract

The embodiment of the invention discloses a 9D pose estimation method and device based on multi-frame image feature fusion, and relates to the technical field of computer vision and target pose estimation, and the method comprises the steps: continuously collecting multi-frame RGB images of a target scene, carrying out the preprocessing of the multi-frame RGB images, combining a Query query mechanism with an MLP through a pre-training Transform encoder, and carrying out the recognition of the multi-frame RGB images, and extracting a global feature vector of each frame of image and generating a target candidate frame feature. Performing coding matching on the candidate frame features of the adjacent frames through a self-attention and cross-attention mechanism, screening high-correlation target features, and performing adaptive weighted fusion to obtain a fusion feature vector; and inputting the vector into a Transform decoder, outputting pose parameters in parallel through a prediction head, carrying out loss calculation and model optimization by taking a key frame label as supervision, and outputting a category-level 9D multi-target pose result containing 3D translation, rotation and scale. The problems that in the prior art, the 6D pose error is large, robustness is insufficient, and the requirements for real-time performance and high precision are difficult to meet are effectively solved.

Description

A 9D pose estimation method and apparatus based on multi-frame image feature fusion Technical Field

[0001] This invention relates to the field of computer vision and target pose estimation technology, and in particular to a 9D pose estimation method and apparatus based on multi-frame image feature fusion. Background Technology

[0002] With the deep integration of computer vision and robotics, high-precision target pose estimation has become a key technical requirement in scenarios such as autonomous driving, industrial robot grasping, and augmented reality. Among them, category-level 9D multi-target pose estimation needs to simultaneously predict the target's 6D pose (3D translation + 3D rotation) and 3D scale (length, width, height), and must have the ability to generalize to unseen instances of the same category.

[0003] However, traditional monocular RGB pose estimation methods (such as the "You Only Pose Once" architecture based on Transformer) have significant limitations: they rely on global feature extraction from a single frame image, are highly sensitive to environmental noise, local occlusion, and changes in illumination, resulting in large 6D pose prediction errors; at the same time, they do not make full use of the temporal correlation of multiple frames, which significantly reduces the robustness of pose estimation in dynamic scenes or when the camera pose changes rapidly.

[0004] Furthermore, noise in single-frame features directly affects the stability of 3D scale prediction, and traditional methods lack auxiliary optimization mechanisms for scale parameters, making them prone to scale fluctuations. More importantly, existing technologies have failed to form a positive cycle of "feature fusion - scale stabilization - pose accuracy improvement," making it difficult to meet the dual requirements of real-time performance (e.g., ≥15fps for robot grasping) and high accuracy.

[0005] Therefore, there is an urgent need for a 9D pose estimation method based on multi-frame image feature fusion that can fully utilize the temporal correlation of multiple frames, improve the robustness of 9D pose estimation based on multi-frame image feature fusion, and meet the requirements of real-time performance and high accuracy. Summary of the Invention

[0006] The embodiments of the present invention provide a 9D pose estimation method based on multi-frame image feature fusion to solve the problems of existing technologies that rely on single-frame features, have large 6D pose errors and insufficient robustness, and are prone to fluctuations in 3D scale prediction, making it difficult to meet the requirements of real-time performance and high accuracy. The technical solution is as follows: According to one aspect of the present invention, a 9D pose estimation method based on multi-frame image feature fusion, the method comprising: acquiring multiple consecutive images of a target scene to obtain an image sequence; performing time synchronization and preprocessing on the image sequence; the preprocessing includes image cropping, normalization, and pixel value standardization; the image sequence contains information from different perspectives of the target scene; using a pre-trained Transformer encoder to encode features of each preprocessed image frame, and generating target candidate box features of each image frame through a query mechanism and a multilayer perceptron (MLP) to obtain a global feature vector of each image frame; encoding and matching candidate box features of adjacent images based on the global feature vector through self-attention and cross-attention mechanisms, filtering out target features that meet the set relevance requirements, and performing adaptive weighted fusion to obtain a fused feature vector; inputting the feature vector into a Transformer decoder, performing pose prediction through a preset pose prediction head to obtain the 6D pose parameters and 3D scale parameters of the target, and performing post-processing to obtain a 9D multi-target pose result; the post-processing includes rotation matrix orthogonalization and scale parameter threshold constraint.

[0007] In one embodiment, an image sequence is obtained by acquiring multiple consecutive frames of images of a target scene. Time synchronization and preprocessing of the image sequence are achieved through the following steps: multiple RGB images of the target scene are continuously acquired using a monocular camera to obtain an image sequence; the image sequence is time-synchronized; a dynamic loader is designed to divide the image sequence into key frames and historical frames, and pose annotation is performed on the key frames; the time synchronization is used to eliminate inter-frame time deviation; the dynamic loader is used to crop each image to a uniform resolution, normalize pixel values ​​to the [0,1] range, adjust pixel distribution through standardization, and perform data augmentation; the dynamic loader is used to adjust the number of frame groups and batch size of the image sequence.

[0008] In one embodiment, a pre-trained Transformer encoder is used to encode features of each pre-processed frame of the image, and a query mechanism and a multilayer perceptron (MLP) are used to generate target candidate bounding box features for each frame of the image, resulting in a global feature vector for each frame. This is achieved through the following steps: The pre-trained Transformer encoder locates target candidate regions from each frame of the image using the query mechanism, and combines this with the MLP to generate multiple candidate bounding box features, resulting in a global feature vector for each frame. In another embodiment, self-attention and cross-attention mechanisms are used to encode and match candidate bounding box features of adjacent frames of the image based on the global feature vector. This is achieved through the following steps: The self-attention mechanism strengthens the spatiotemporal correlation of intra-frame candidate bounding box features based on the global feature vectors of adjacent frames; the cross-attention mechanism calculates the correlation matrix of inter-frame candidate bounding boxes of adjacent frames; and correlation values ​​are obtained by comparing the element values ​​of the matrix. Finally, frames that satisfy the correlation criteria are selected.

[0009] In one embodiment, the target features that meet the set relevance requirements are selected and adaptively weighted and fused to obtain the fused feature vector. This is achieved through the following steps: selecting multiple candidate box pairs whose relevance values ​​meet the set requirements, matching candidate box features belonging to the same target, and adaptively weighting and fusing the candidate box features belonging to the same target to obtain the feature vector belonging to the same target.

[0010] In one embodiment, the feature vector is input into the Transformer decoder, and pose prediction is performed using a preset pose prediction head to obtain the target's 6D pose parameters and 3D scale parameters. This is achieved through the following steps: the feature vector is used in parallel to predict the pose of the target using a preset pose prediction head in the Transformer decoder to obtain the target's 6D pose parameters and 3D scale parameters; the 6D pose parameters include translation vectors, rotation matrices, and 3D scale parameters; the pose prediction head includes a 6D pose branch and a 3D scale branch; the Transformer decoder is trained and optimized by calculating the losses for the translation vectors, rotation matrices, and 3D scale parameters using the annotations of the keyframes as supervision; the 6D pose branch uses fully connected layers and an attention mechanism; the 3D scale branch includes fully connected layers and L1 loss.

[0011] In one embodiment, post-processing to obtain the 9D multi-object pose result is achieved through the following steps: the 6D pose parameters and 3D scale parameters are orthogonally forced through SVD decomposition, a threshold constraint is applied to the 3D scale parameters, and a category-level 9D multi-object pose result is output; the 9D multi-object pose result includes 3D translation, 3D rotation, and 3D scaling.

[0012] According to one aspect of the present invention, a 9D pose estimation device based on multi-frame image feature fusion is provided. The device includes: an image acquisition and preprocessing module, configured to acquire multiple consecutive images of a target scene to obtain an image sequence, and perform time synchronization and preprocessing on the image sequence; the preprocessing includes image cropping, normalization, and pixel value standardization; the image sequence contains information from different viewpoints of the target scene; and a feature encoding and candidate box generation module, configured to perform feature encoding on each preprocessed frame of the image using a pre-trained Transformer encoder, and generate target candidate box features for each frame of the image through a query mechanism (Query) and a multilayer perceptron (MLP), thereby obtaining the feature fusion of each frame. The system comprises: a global feature vector of the image; a temporal feature matching and fusion module, used to encode and match candidate box features of adjacent frames of the image based on the global feature vector using self-attention and cross-attention mechanisms, and to filter out target features that meet the set relevance requirements for adaptive weighted fusion to obtain a fused feature vector; and a 9D pose estimation module based on multi-frame image feature fusion, used to input the feature vector into a Transformer decoder, perform pose prediction through a preset pose prediction head, obtain the 6D pose parameters and 3D scale parameters of the target, and perform post-processing to obtain the 9D multi-target pose result; the post-processing includes rotation matrix orthogonalization and scale parameter threshold constraints.

[0013] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein computer-readable instructions are stored on the memory; the computer-readable instructions are executed by one or more of the processors to cause the electronic device to implement the 9D pose estimation method based on multi-frame image feature fusion as described above.

[0014] According to one aspect of the present invention, a storage medium has computer-readable instructions stored thereon, which are executed by one or more processors to implement the 9D pose estimation method based on multi-frame image feature fusion as described above.

[0015] The beneficial effects of the technical solution provided by this invention are as follows: Firstly, this invention continuously acquires multiple frames of RGB images of the target scene using a monocular camera. After time synchronization, cropping, normalization, and standardization preprocessing, a pre-trained Transformer encoder combined with a query mechanism and MLP is used to extract global feature vectors from each frame and generate target candidate box features. Subsequently, self-attention and cross-attention mechanisms are used to encode and match the candidate box features of adjacent frames, and highly correlated target features are selected for adaptive weighted fusion to obtain a fused feature vector. This vector is input into the Transformer decoder, and pose parameters are output in parallel through a preset 6D pose and 3D scale prediction head. Loss calculation and model optimization are performed using keyframe annotations as supervision. Finally, the 6D pose parameters are orthogonalized by rotation matrices, and threshold constraints are applied to the 3D scale parameters to output a category-level 9D multi-target pose result including 3D translation, rotation, and scale. This method effectively suppresses single-frame noise and viewpoint bias through multi-frame feature fusion, significantly improving the accuracy and robustness of pose estimation, while maintaining 3D scale stability, thus meeting the dual requirements of real-time performance and high accuracy in scenarios such as robot grasping and autonomous driving. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0017] Figure 1 is a flowchart illustrating a 9D pose estimation method based on multi-frame image feature fusion according to an exemplary embodiment; Figure 2 is a schematic flowchart illustrating the 9D pose estimation method based on multi-frame image feature fusion in an application scenario; Figure 3 is a block diagram illustrating a 9D pose estimation device based on multi-frame image feature fusion according to an exemplary embodiment; Figure 4 is a hardware structure diagram of an electronic device according to an exemplary embodiment; Figure 5 is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0019] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this disclosure means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0020] This invention provides a 9D pose estimation method based on multi-frame image feature fusion. Through a multi-frame target-level feature matching and fusion mechanism, it achieves high-precision 9D pose estimation based on multi-frame image feature fusion, solving the problems of traditional single-frame methods such as sensitivity to environmental changes, insufficient robustness, and unstable 3D scale prediction. This significantly improves the accuracy and real-time performance of pose estimation, meeting the needs of scenarios such as robot grasping and autonomous driving. This 9D pose estimation method based on multi-frame image feature fusion is applicable to 9D pose estimation devices based on multi-frame image feature fusion, which can be electronic devices. The 9D pose estimation method based on multi-frame image feature fusion in this invention can be applied to various scenarios, such as 9D pose estimation based on multi-frame image feature fusion.

[0021] Please refer to Figure 1. This embodiment of the invention provides a 9D pose estimation method based on multi-frame image feature fusion, which is applicable to electronic devices.

[0022] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0023] As shown in Figure 1, the method may include the following steps: Step 110, obtaining multiple consecutive images of the target scene to obtain an image sequence, and performing time synchronization and preprocessing on the image sequence.

[0024] In one possible implementation, multiple frames of RGB images of the target scene are continuously acquired by a monocular camera to obtain an image sequence. The image sequence is time-synchronized, and a dynamic loader is designed to divide the image sequence into key frames and historical frames. The key frames are then labeled with poses. The dynamic loader is used to crop each image to a uniform resolution, normalize the pixel values ​​to the range of [0,1], adjust the pixel distribution through standardization, and perform data augmentation.

[0025] Among them, time synchronization is used to eliminate inter-frame time deviation; dynamic loader is used to adjust the number of frame groups and batch size of the image sequence; the image sequence contains information from different perspectives of the target scene; preprocessing includes image cropping, normalization, pixel value standardization, etc., which are not limited here.

[0026] Specifically, multiple frames of RGB images of the target scene are continuously acquired using a monocular camera to form an image sequence. This step ensures the continuity and temporal order of the image data, providing a foundation for subsequent processing. The acquired image sequence undergoes time synchronization processing to eliminate inter-frame temporal deviations and ensure temporal consistency among the images. Image preprocessing includes cropping images to a uniform resolution, normalizing pixel values ​​to the [0,1] range, adjusting pixel distribution through standardization, and performing data augmentation (such as random flipping and brightness / contrast adjustment). These operations aim to improve image quality, reduce noise interference, and provide stable input for subsequent feature extraction.

[0027] Furthermore, during the image acquisition phase, a high-resolution, high-frame-rate monocular camera is selected to ensure the capture of detailed changes in the target scene. Time synchronization is achieved through software algorithms, accurate to the millisecond level, ensuring accurate alignment of each frame on the timeline. In image preprocessing, cropping removes invalid areas at image edges, reducing computational load; normalization and standardization operations make pixel value distribution more uniform, improving the stability of feature extraction; data augmentation increases the model's adaptability to environmental changes by simulating images under different lighting and angle conditions.

[0028] In the above process, the embodiments of the present invention significantly improve the quality of input image data through precise image acquisition and comprehensive preprocessing, providing a stable and reliable foundation for subsequent feature extraction and achieving effective adaptation to complex environments.

[0029] Step 120: Use the pre-trained Transformer encoder to encode the features of each preprocessed frame image, and generate the target candidate box features of each frame image through the Query mechanism and the Multilayer Perceptron (MLP) to obtain the global feature vector of each frame image.

[0030] In one possible implementation, a pre-trained Transformer encoder is used to locate target candidate regions from each frame of the image through a query mechanism, and multiple candidate box features are generated by combining a multilayer perceptron (MLP) to obtain the global feature vector for each frame.

[0031] Specifically, a pre-trained Transformer encoder is used to encode features of each preprocessed frame image. The encoder captures global and local information in the image through a self-attention mechanism, forming rich feature representations. By combining a query mechanism with a multilayer perceptron (MLP), candidate regions that may contain the target are located from the encoded feature map, and multiple candidate bounding box features are generated.

[0032] In the feature encoding stage, pre-trained Transformer models, such as the Swin-VIT or ViT architecture, are used. These models are pre-trained on large-scale datasets and have powerful feature extraction capabilities. The query mechanism interacts with the feature map through the learned query vector to accurately locate the target candidate region. The MLP further refines the features of the candidate region, generating discriminative candidate box features.

[0033] In the above process, the embodiments of the present invention achieve efficient extraction and accurate localization of target features through an advanced Transformer encoder and Query mechanism, providing rich candidate box information for subsequent multi-frame feature matching and improving the accuracy of pose estimation.

[0034] Step 130: Using self-attention and cross-attention mechanisms, candidate box features of adjacent frames are encoded and matched based on the global feature vector. Target features that meet the set relevance requirements are selected and adaptively weighted and fused to obtain the fused feature vector.

[0035] In one possible implementation, the spatiotemporal correlation of intra-frame candidate box features is enhanced by a self-attention mechanism based on the global feature vectors of adjacent frames. The correlation matrix of inter-frame candidate boxes is calculated using a cross-attention mechanism, and the correlation value is obtained by comparing the element values ​​of the matrix.

[0036] In one possible implementation, multiple candidate box pairs that meet the set relevance requirements are selected, and the candidate box features belonging to the same target are matched. The candidate box features belonging to the same target are then adaptively weighted and fused to obtain the feature vector belonging to the same target.

[0037] The weights are dynamically allocated based on feature consistency.

[0038] Specifically, a self-attention mechanism is used to enhance the spatiotemporal correlation of intra-frame candidate box features, and a cross-attention mechanism is used to calculate the correlation matrix of inter-frame candidate boxes in adjacent frames. Candidate box pairs whose correlation values ​​meet the set requirements are selected and subjected to adaptive weighted fusion to obtain a fused feature vector belonging to the same target.

[0039] Furthermore, in the feature matching stage, the self-attention mechanism strengthens the spatiotemporal correlation of intra-frame features by calculating the similarity between candidate box features, making the features of the same target more consistent across different frames. The cross-attention mechanism calculates the correlation of candidate boxes across frames to find candidate box pairs belonging to the same target. In the feature fusion stage, weights are dynamically allocated based on the correlation values, and the candidate box features of the same target are weighted and fused to form a more representative fused feature vector.

[0040] In the above process, the embodiments of the present invention achieve effective matching and fusion of target features in multiple frames of images through self-attention and cross-attention mechanisms, suppress noise and viewpoint bias of single-frame features, improve the robustness and accuracy of feature representation, and provide a reliable basis for subsequent pose prediction.

[0041] Step 140: Input the feature vector into the Transformer decoder, perform pose prediction through the preset pose prediction head, obtain the 6D pose parameters and 3D scale parameters of the target, and perform post-processing to obtain the 9D multi-target pose result; post-processing includes rotation matrix orthogonalization and scale parameter threshold constraint.

[0042] In one possible implementation, the pose prediction head in the Transformer decoder is used to predict the pose of the feature vectors in parallel to obtain the target's 6D pose parameters and 3D scale parameters. The annotation of keyframes is used as supervision to calculate the loss of the translation vector, rotation matrix and 3D scale parameters respectively to train and optimize the Transformer decoder.

[0043] The 6D pose parameters include translation vectors, rotation matrices, and 3D scale parameters; the pose prediction head includes a 6D pose branch and a 3D scale branch; the 6D pose branch uses fully connected layers and an attention mechanism; the 3D scale branch includes fully connected layers and L1 loss.

[0044] In one possible implementation, the 6D pose parameters and 3D scale parameters of the SVD decomposition are orthogonally forced, and a threshold constraint is applied to the 3D scale parameters to output the category-level 9D multi-object pose results.

[0045] The 9D multi-target pose results include 3D translation, 3D rotation, and 3D scaling.

[0046] Specifically, the fused feature vectors are input into the Transformer decoder, which outputs the target's 6D pose parameters (translation vectors, rotation matrices) and 3D scale parameters in parallel through a pre-defined pose prediction head (including 6D pose branches and 3D scale branches). The 6D pose parameters output by the decoder are orthogonalized by the rotation matrix, and threshold constraints are applied to the 3D scale parameters to output the final category-level 9D multi-target pose result.

[0047] Furthermore, in the pose prediction stage, the Transformer decoder utilizes the temporal and geometric information from the fused feature vectors to optimize translation and rotation predictions through fully connected layers and attention mechanisms, and improves the stability of scale prediction through L1 loss constraints. In the post-processing stage, orthogonalization of the rotation matrix ensures the rationality of the rotation parameters, while threshold constraints on the scale parameters prevent unreasonable fluctuations in scale prediction.

[0048] In the above process, the embodiments of the present invention achieve high-precision prediction of target pose and scale through Transformer decoder and post-processing mechanism, ensuring the rationality and stability of output results, and meeting the accurate pose estimation requirements of robot grasping, autonomous driving and other scenarios.

[0049] Through the above process, this embodiment of the invention achieves high-precision estimation of the 3D position, pose, and scale information of a target by image acquisition and preprocessing, feature encoding and candidate box generation, multi-frame feature matching and fusion, pose prediction and post-processing. This method significantly reduces 6D pose estimation error, improves the stability of 3D scale prediction, and exhibits strong robustness, providing effective technical support for fields such as robot grasping, autonomous driving, and augmented reality.

[0050] In one application scenario, the 9D pose estimation method based on multi-frame image feature fusion of the present invention is used in a robot grasping scenario to perform 9D pose estimation based on multi-frame image feature fusion.

[0051] As shown in Figure 2, the following steps may be included: Step S1: Image acquisition and preprocessing.

[0052] Specifically, in the robotic grasping workstation, a monocular camera continuously acquires multiple frames of RGB images of the target object (such as industrial parts) to form an image sequence. The image sequence is then time-synchronized to eliminate inter-frame time deviations and ensure temporal consistency among the frames.

[0053] Furthermore, the image is cropped to a uniform resolution (e.g., 640×640) using a dynamic loader, the pixel values ​​are normalized to the range of [0,1], and the pixel distribution is adjusted by standardization (mean = 0.485, variance = 0.229). At the same time, data augmentation (e.g., random flipping, brightness and contrast adjustment) is performed to improve image quality and increase the model's adaptability to environmental changes.

[0054] In the above process, the embodiments of the present invention ensure the quality and consistency of the input data through precise image acquisition and comprehensive preprocessing, providing a stable and reliable foundation for subsequent feature extraction.

[0055] Step S2: Feature encoding and candidate box generation.

[0056] Specifically, a pre-trained Swin-VIT model is used as the backbone network to encode features of each pre-processed frame image. Swin-VIT captures global and local information in the image through a self-attention mechanism, forming rich feature representations.

[0057] Furthermore, by combining the query mechanism with the multilayer perceptron (MLP), candidate regions that may contain the target object are located from the encoded feature map, and multiple candidate box features are generated. Each candidate box feature is represented by a feature block of a different color (such as orange, yellow, green, blue, etc.).

[0058] In the above process, the embodiments of the present invention achieve efficient extraction and accurate localization of target features through the advanced Swin-VIT model and Query mechanism, providing rich candidate box information for subsequent multi-frame feature matching.

[0059] Step S3: Multi-frame feature matching and fusion.

[0060] Specifically, a self-attention mechanism is used to enhance the spatiotemporal correlation of intra-frame candidate box features, and a cross-attention mechanism is used to calculate the correlation matrix of inter-frame candidate boxes in adjacent frames. Further, candidate box pairs whose correlation values ​​meet set requirements are selected and adaptively weighted fused to obtain a fused feature vector belonging to the same target. The weights are dynamically allocated based on feature consistency to ensure that the fused features are more representative.

[0061] In the above process, the embodiments of the present invention achieve effective matching and fusion of target features in multiple frames of images through self-attention and cross-attention mechanisms, suppress noise and viewpoint bias of single-frame features, and improve the robustness and accuracy of feature representation.

[0062] Step S4: Pose Prediction and Post-processing. Specifically, the fused feature vector is input into the Transformer decoder, and the target's 6D pose parameters (translation vector, rotation matrix) and 3D scale parameters are output in parallel through a preset pose prediction head (including a 6D pose branch and a 3D scale branch). Further, the 6D pose parameters output by the decoder are orthogonalized using rotation matrices to ensure the reasonableness of the rotation parameters; threshold constraints are applied to the 3D scale parameters to prevent unreasonable fluctuations in scale prediction, ultimately outputting a category-level 9D multi-target pose result.

[0063] In the above process, the embodiments of the present invention achieve high-precision prediction of target pose and scale through Transformer decoder and post-processing mechanism, ensuring the rationality and stability of output results, and providing accurate pose information for robot grasping.

[0064] Through the above process, this embodiment of the invention achieves 9D pose estimation of industrial parts based on multi-frame image feature fusion in four steps: multi-frame image acquisition and preprocessing, feature encoding and candidate box generation, multi-frame feature matching and fusion, and pose prediction and post-processing. This method utilizes the Swin-VIT model and Transformer decoder, combined with self-attention and cross-attention mechanisms, to effectively improve the accuracy and robustness of pose estimation. Dynamic loaders and data augmentation techniques enhance the model's adaptability to environmental changes. The final output 9D pose result provides accurate pose and scale information for robot grasping, significantly improving the grasping success rate and efficiency.

[0065] The following are embodiments of the apparatus of the present invention, which can be used to execute the 9D pose estimation method based on multi-frame image feature fusion involved in the present invention. For details not disclosed in the apparatus embodiments of the present invention, please refer to the method embodiments of the 9D pose estimation method based on multi-frame image feature fusion involved in the present invention.

[0066] Please refer to Figure 3. In this embodiment of the invention, a 9D pose estimation device 800 based on multi-frame image feature fusion is provided.

[0067] The 9D pose estimation device 800 based on multi-frame image feature fusion includes, but is not limited to: an image acquisition and preprocessing module 810, a feature encoding and candidate box generation module 830, a temporal feature matching and fusion module 850, and a 9D pose estimation module 870 based on multi-frame image feature fusion.

[0068] The image acquisition and preprocessing module 810 is used to acquire multiple consecutive images of the target scene to obtain an image sequence, and to perform time synchronization and preprocessing on the image sequence; the preprocessing includes image cropping, normalization, and pixel value standardization; the image sequence contains information from different perspectives of the target scene.

[0069] The feature encoding and candidate box generation module 830 is used to encode the features of each frame of the preprocessed image using a pre-trained Transformer encoder, and generate the target candidate box features of each frame of the image through a query mechanism and a multilayer perceptron (MLP) to obtain the global feature vector of each frame of the image.

[0070] The temporal feature matching and fusion module 850 is used to encode and match the candidate box features of adjacent frames based on the global feature vector through self-attention and cross-attention mechanisms, and to filter out target features that meet the set requirements for adaptive weighted fusion to obtain the fused feature vector.

[0071] The 9D pose estimation module 870 based on multi-frame image feature fusion is used to input feature vectors into the Transformer decoder, perform pose prediction through a preset pose prediction head, obtain the 6D pose parameters and 3D scale parameters of the target, and perform post-processing to obtain the 9D multi-target pose result; the post-processing includes rotation matrix orthogonalization and scale parameter threshold constraint.

[0072] It should be noted that the 9D pose estimation based on multi-frame image feature fusion provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the 9D pose estimation device based on multi-frame image feature fusion will be divided into different functional modules to complete all or part of the functions described above.

[0073] Furthermore, the 9D pose estimation device based on multi-frame image feature fusion and the 9D pose estimation method based on multi-frame image feature fusion provided in the above embodiments belong to the same concept. The specific way each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0074] Figure 4 shows a schematic diagram of the structure of an electronic device according to an exemplary embodiment.

[0075] It should be noted that this electronic device is merely an example adapted to the present invention and should not be construed as providing any limitation on the scope of the invention. Furthermore, this electronic device should not be interpreted as requiring or needing to have one or more of the components of the exemplary electronic device 2000 shown in FIG4.

[0076] The hardware structure of the electronic device 2000 can vary greatly depending on the configuration or performance. As shown in Figure 4, the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0077] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 2000.

[0078] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted to this invention, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, as shown in FIG4, but this is not intended to be a specific limitation.

[0079] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0080] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0081] Application 253 is a computer-readable instruction based on operating system 251 that performs at least one specific task. It may include at least one module (not shown in Figure 4), and each module may contain computer-readable instructions for electronic device 2000. For example, a 9D pose estimation device based on multi-frame image feature fusion can be regarded as application 253 deployed on electronic device 2000.

[0082] Data 255 may be signal information, etc., and is stored in memory 250.

[0083] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby enabling the computation and processing of massive amounts of data 255 in the memory 250. For example, a 9D pose estimation method based on multi-frame image feature fusion can be implemented by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0084] Furthermore, the present invention can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of the present invention is not limited to any specific hardware circuit, software, or combination thereof.

[0085] Please refer to Figure 5. In this embodiment of the invention, an electronic device 4000 is provided. The electronic device 4000 may include: a desktop computer, a laptop computer, a server, etc., with sensor recognition capabilities.

[0086] In Figure 5, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0087] Data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in Figure 5, but this does not indicate that there is only one bus or one type of bus.

[0088] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0089] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0090] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 4000, but not limited thereto.

[0091] The memory 4003 stores computer-readable instructions, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.

[0092] The computer-readable instructions are executed by one or more processors 4001 to implement the 9D pose estimation method based on multi-frame image feature fusion in the above embodiments.

[0093] Furthermore, this embodiment of the invention provides a storage medium storing computer-readable instructions, which are executed by one or more processors to implement the 9D pose estimation method based on multi-frame image feature fusion as described above.

[0094] This invention provides a computer program product, which includes computer-readable instructions stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the 9D pose estimation method based on multi-frame image feature fusion as described above.

[0095] Compared with related technologies, the beneficial effects of the present invention are: 1. The present invention can improve the accuracy and robustness of target feature extraction; by using a pre-trained Swin-VIT model as the backbone network, combined with the query mechanism and multilayer perceptron (MLP), the target candidate region is accurately located and rich candidate box features are generated, effectively capturing global and local information in the image and adapting to complex environmental changes.

[0096] 2. This invention has the ability to effectively utilize multi-frame temporal information; it strengthens the spatiotemporal correlation of intra-frame candidate box features through a self-attention mechanism, and uses a cross-attention mechanism to calculate the inter-frame candidate box correlation of adjacent frame images, thereby achieving effective matching and fusion of multi-frame features, suppressing single-frame noise and viewpoint bias, and improving the stability of feature representation.

[0097] 3. This invention can achieve high-precision 9D pose estimation based on multi-frame image feature fusion; through the Transformer decoder and the preset pose prediction head, the 6D pose parameters (translation vector, rotation matrix) and 3D scale parameters of the target are output in parallel. Combined with the orthogonalization of the rotation matrix and the threshold constraint of the scale parameter, the rationality and accuracy of the output results are ensured.

[0098] 4. This invention has strong adaptability to dynamic scenes; by adjusting the number of frame groups and batch size of image sequences through a dynamic loader, and through data augmentation technology, the model can adapt to image input under different frame rates, resolutions and lighting conditions, meeting the needs of real-time application scenarios such as robot grasping and autonomous driving.

[0099] 5. This invention can improve the execution efficiency and success rate of tasks such as robot grasping; by providing accurate 9D pose results, it provides reliable pose and scale information for robot grasping planning, reduces positioning errors and collision risks during the grasping process, and improves the efficiency and stability of task execution.

[0100] 6. This invention has high efficiency in model training and optimization; by using keyframe annotations as supervision, the loss is calculated for translation vectors, rotation matrices and 3D scale parameters respectively, and the Transformer decoder is trained and optimized, thereby accelerating model convergence, improving model performance and reducing training costs.

[0101] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0102] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A 9D pose estimation method based on multi-frame image feature fusion, characterized in that, The method includes: acquiring multiple consecutive frames of images of a target scene to obtain an image sequence; performing time synchronization and preprocessing on the image sequence; the preprocessing includes image cropping, normalization, and pixel value standardization; the image sequence contains information from different perspectives of the target scene; using a pre-trained Transformer encoder to encode features of each frame of the preprocessed image, and generating target candidate box features of each frame of the image through a query mechanism and a multilayer perceptron (MLP) to obtain a global feature vector of each frame of the image; using self-attention and cross-attention mechanisms to encode and match the candidate box features of adjacent frames of the image based on the global feature vector, filtering out target features that meet the set relevance requirements, and performing adaptive weighted fusion to obtain a fused feature vector; inputting the feature vector into a Transformer decoder, performing pose prediction through a preset pose prediction head to obtain the 6D pose parameters and 3D scale parameters of the target, and performing post-processing to obtain a 9D multi-target pose result; the post-processing includes rotation matrix orthogonalization and scale parameter threshold constraints.

2. The 9D pose estimation method based on multi-frame image feature fusion as described in claim 1, characterized in that, The process of acquiring multiple consecutive images of the target scene to obtain an image sequence, and performing time synchronization and preprocessing on the image sequence includes: continuously acquiring multiple RGB images of the target scene using a monocular camera to obtain an image sequence; performing time synchronization on the image sequence; designing a dynamic loader to divide the image sequence into key frames and historical frames, and performing pose annotation on the key frames; the time synchronization is used to eliminate inter-frame time deviations; the dynamic loader is used to crop each image to a uniform resolution, normalize pixel values ​​to the [0,1] range, adjust pixel distribution through standardization, and perform data augmentation; the dynamic loader is used to adjust the number of frame groups and batch size of the image sequence.

3. The 9D pose estimation method based on multi-frame image feature fusion as described in claim 1, characterized in that, The process of using a pre-trained Transformer encoder to encode features of each frame of the preprocessed image, and generating target candidate box features of each frame of the image through a query mechanism and a multilayer perceptron (MLP) to obtain the global feature vector of each frame of the image includes: using a pre-trained Transformer encoder to locate target candidate regions from each frame of the image through a query mechanism, and combining a multilayer perceptron (MLP) to generate multiple candidate box features to obtain the global feature vector of each frame.

4. The 9D pose estimation method based on multi-frame image feature fusion as described in claim 1, characterized in that, The step of encoding and matching candidate bounding box features of adjacent frames based on the global feature vector using self-attention and cross-attention mechanisms includes: strengthening the spatiotemporal correlation of intra-frame candidate bounding box features based on the global feature vector of adjacent frames using a self-attention mechanism; calculating the correlation matrix of inter-frame candidate bounding boxes of adjacent frames using a cross-attention mechanism; obtaining correlation values ​​by comparing the element values ​​of the matrix; and filtering out boxes that meet the correlation requirements.

5. The 9D pose estimation method based on multi-frame image feature fusion as described in claim 4, characterized in that, The step of adaptively weighting and fusing the target features that meet the set relevance requirements to obtain the fused feature vector includes: selecting multiple candidate box pairs whose relevance values ​​meet the set requirements, matching the candidate box features belonging to the same target, and adaptively weighting and fusing the candidate box features belonging to the same target to obtain the feature vector belonging to the same target.

6. The 9D pose estimation method based on multi-frame image feature fusion as described in claim 2, characterized in that, The step of inputting the feature vector into the Transformer decoder and performing pose prediction through a preset pose prediction head to obtain the target's 6D pose parameters and 3D scale parameters includes: performing pose prediction on the feature vector in parallel through a preset pose prediction head in the Transformer decoder to obtain the target's 6D pose parameters and 3D scale parameters; the 6D pose parameters include translation vectors, rotation matrices, and 3D scale parameters; the pose prediction head includes a 6D pose branch and a 3D scale branch; using the annotations of the keyframes as supervision, calculating the loss for the translation vectors, rotation matrices, and 3D scale parameters respectively to train and optimize the Transformer decoder; the 6D pose branch uses fully connected layers and an attention mechanism; the 3D scale branch includes fully connected layers and L1 loss.

7. The 9D pose estimation method based on multi-frame image feature fusion as described in claim 1, characterized in that, The post-processing to obtain 9D multi-object pose results includes: forcibly orthogonalizing the 6D pose parameters and 3D scale parameters through SVD decomposition, applying threshold constraints to the 3D scale parameters, and outputting category-level 9D multi-object pose results; the 9D multi-object pose results include 3D translation, 3D rotation, and 3D scaling.

8. A 9D pose estimation device based on multi-frame image feature fusion, characterized in that, The device includes: an image acquisition and preprocessing module, used to acquire multiple consecutive images of a target scene to obtain an image sequence, and to perform time synchronization and preprocessing on the image sequence; the preprocessing includes image cropping, normalization, and pixel value standardization; the image sequence contains information from different perspectives of the target scene; a feature encoding and candidate box generation module, used to perform feature encoding on each preprocessed image frame using a pre-trained Transformer encoder, and to generate target candidate box features for each image frame through a query mechanism and a multilayer perceptron (MLP), obtaining a global feature vector for each image frame; and a temporal feature matching module. The matching and fusion module is used to encode and match the candidate box features of the images in adjacent frames based on the global feature vector using self-attention and cross-attention mechanisms, and to filter out target features that meet the set relevance requirements for adaptive weighted fusion to obtain the fused feature vector; the 9D pose estimation module based on multi-frame image feature fusion is used to input the feature vector into the Transformer decoder, perform pose prediction through a preset pose prediction head, obtain the 6D pose parameters and 3D scale parameters of the target, and perform post-processing to obtain the 9D multi-target pose result; the post-processing includes rotation matrix orthogonalization and scale parameter threshold constraint.

9. An electronic device, characterized in that, include: At least one processor and at least one memory, wherein computer-readable instructions are stored on the memory; the computer-readable instructions are executed by one or more of the processors to cause the electronic device to implement the 9D pose estimation method based on multi-frame image feature fusion as described in any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that, The computer-readable instructions are executed by one or more processors to implement the 9D pose estimation method based on multi-frame image feature fusion as described in any one of claims 1 to 7.

Images