An object pose estimation method based on online multi-view fusion

Abstract

The application discloses an object pose estimation method based on online multi-view fusion, which acquires multi-view continuous image frames and synchronizes camera internal parameters and pose information, obtains two-dimensional boundary boxes, semantic features and confidence through open vocabulary target detection, and then obtains the spatial position, pose orientation and observation confidence of the target by two-dimensional to three-dimensional pose promotion. In a unified coordinate system, multi-constraint target correlation matching based on spatial distance, semantic features and pose difference is performed, and target motion information is estimated according to a historical state to generate a predicted state. An incremental online fusion strategy is adopted to perform weighted fusion on the current observation state, the historical fusion state and the predicted state, and the global target state is updated in real time, and the estimation result containing the target identity, stable three-dimensional pose and motion state is output after repeated target suppression. The method improves the stability and cross-frame consistency of target positioning in a shielding environment and the real-time perception ability in a dynamic scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to an online multi-view fusion method for object pose estimation. Background Technology

[0002] In tasks such as autonomous robot navigation, intelligent obstacle avoidance, and perception of complex dynamic environments, accurately and stably acquiring the 3D spatial position and orientation of surrounding objects is a prerequisite for achieving safe and efficient motion planning and behavioral decision-making. However, real-world application scenarios often involve various uncertainties, making reliable 3D pose estimation a prominent technical challenge. The sources of these technical problems are mainly reflected in the following aspects: First, single-view acquisition devices are limited by a fixed field of view and cannot fully cover target objects in the scene. When a target is partially occluded by other objects or exhibits an atypical appearance due to rotation, the target information contained in a single frame is severely insufficient, easily leading to detection omissions, positioning deviations, or pose misjudgments. Second, in multi-view configurations, differences in the viewing angles of different acquisition devices, lighting variations, and sensor noise cause significant inconsistencies in the appearance and geometric features of the same target across different image frames. Traditional single-frame independent estimation methods struggle to establish stable and consistent target correspondences in both the temporal and spatial dimensions. Furthermore, the introduction of dynamic targets complicates the problem. The position and orientation of moving targets change continuously over time. If estimation is performed independently based solely on the current image frame, the results often contain significant noise and lack motion continuity, failing to provide reliable dynamic state information for subsequent navigation and obstacle avoidance decisions. Finally, environmental perception systems typically require online operation, placing stringent demands on computational efficiency and real-time response capabilities. Offline batch processing-based multi-view fusion methods cannot meet real-time constraints. Therefore, the technical problem addressed in this application is clearly defined as: in online perception scenarios with factors such as partial occlusion, multi-view observation, and target motion, how to achieve stable association of target objects across time frames and viewpoints, and fuse multi-source observation information to obtain real-time 3D pose estimation with spatiotemporal consistency.

[0003] To address the aforementioned technical challenges, existing technologies typically employ several main approaches. One group focuses on improving the accuracy and robustness of single-frame 3D pose estimation. For example, deep convolutional neural networks can be used to directly regress the 3D bounding box parameters of the target from monocular or binocular images, or geometric priors provided by depth images can be combined for 3D position inference. These methods often enhance adaptability to occlusion and viewpoint changes by increasing model capacity, introducing attention mechanisms, or fusing multi-scale features. Another group attempts to improve the reliability of pose estimation through multi-view geometric constraints. A typical approach is to perform target detection and pose estimation separately at multiple fixed viewpoints, and then use camera intrinsic and extrinsic parameters to project the estimation results from each viewpoint onto a unified 3D coordinate system for offline fusion. Furthermore, there are batch processing multi-target tracking and 3D reconstruction methods based on complete video sequences. By establishing a graphical model or energy function over the entire sequence, the optimal 3D trajectory and pose of the target in all frames can be solved simultaneously to achieve high spatiotemporal consistency. For the processing of dynamic targets, some existing solutions introduce classical state estimation methods such as Kalman filtering or particle filtering. After obtaining the observation results, the target state is filtered and smoothed to reduce noise and restore the continuity of motion.

[0004] However, the aforementioned existing technologies still have significant drawbacks in addressing the technical problems addressed in this application. While single-frame methods can achieve high accuracy under ideal conditions, they essentially only perform inference at the pixel or local image region level, lacking the ability to integrate information across time frames and viewpoints. When faced with occlusion and drastic changes in viewpoint, the estimation results are prone to significant jumps, severely impacting the stability of downstream navigation tasks. Offline batch processing fusion methods, although able to obtain relatively consistent pose sequences through global optimization, require overall processing of the entire video segment, resulting in high computational costs and large output delays, fundamentally contradicting the real-time perception requirements of robots and augmented reality systems. Traditional filtering tracking methods, while able to smooth state sequences, typically rely on relatively simple linear motion assumptions or require predefined motion models, limiting their adaptability to complex and irregular motions. Furthermore, the filtering process is independent of the pose estimation stage, making it difficult to correct original observation errors, leading to problems such as tracking drift or target association failure. Furthermore, the aforementioned methods lack effective online mechanisms for identifying and suppressing recurring targets and maintaining consistency in cross-view target association. This can easily lead to redundant targets in overlapping view areas or when long-term occlusion recurs, reducing the usability of global perception results. Therefore, an online multi-view fusion object pose estimation method is needed to address these issues. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an online multi-view fusion object pose estimation method, which aims to overcome the problems of poor cross-frame consistency in occluded scenes, difficulty in obtaining dynamic target state in real time and stability, and offline fusion being unable to meet online requirements in existing pose estimation methods. It has the feature of combining multi-view observation, historical state and motion prediction in real time through incremental online fusion to output spatiotemporally consistent 3D pose.

[0006] To achieve the above technical solution, the technical solution adopted by the present invention is as follows: An online multi-view fusion method for object pose estimation includes the following steps: S1, Image and pose information acquisition: Acquire continuous image frames formed by the acquisition device during the movement, or acquire multi-view image frames acquired by multiple view acquisition devices in the same scene. Obtain the camera intrinsic parameters and camera pose information corresponding to the image frame, synchronize the image frame and pose information in time, and form a data packet to be processed. S2, Open vocabulary object detection: Input the image frames in the data packet to be processed into the open vocabulary object detection model, detect the target objects in the image, and obtain the two-dimensional bounding box, target category, semantic features and detection confidence of the target objects; S3, 2D to 3D pose enhancement: Based on 2D bounding box, image features, camera intrinsics and camera pose information, generate the pose observation results of the target object in 3D space. The pose observation results include 3D spatial position, pose orientation information and observation confidence. S4, Target observation state construction: Transform the attitude observation results to a unified coordinate system and construct the target observation state of the current frame; S5, Target Association Matching: The current frame target observation state is associated with the historical target state. If the match is successful, the corresponding historical target state is updated; if the match fails, a new target object is created. S6, Motion State Update: Based on the historical target states of the same target object in consecutive frames, estimate the motion state information of the target object and generate the target prediction state; S7, Online Multi-View Fusion: Incrementally fuse the current frame target observation state, historical target fusion state and target prediction state online to obtain the global fusion state of the target object; S8, Repeated Target Suppression and State Feedback: Merge or delete repeatedly generated target objects, and write the updated global fusion state into the historical state cache; S9, Attitude estimation results output: Output the target object's target ID, target category, stable 3D attitude, motion state information, observation confidence, and spatial location results.

[0007] Preferably, in step S1, the image and pose data acquisition method specifically includes: Add a timestamp to each frame of the image and store the image frame, camera intrinsics and camera pose information in a buffer. Image frames with similar timestamps and camera pose information are selected from the buffer to form the same data packet to be processed. When there is a time difference between the image frame and the camera pose information, the camera pose corresponding to the current image frame is interpolated or corrected based on the adjacent pose information. When using multiple perspective acquisition devices, select image frames whose timestamp difference is less than a preset threshold from each perspective to form a multi-perspective data packet to be processed.

[0008] Preferably, in step S2, the open vocabulary target detection method specifically includes: Input the target category text, natural language description, or preset semantic prompts into the open vocabulary target detection model; The open vocabulary object detection model is used to obtain the two-dimensional bounding box, object category, semantic features and detection confidence of the target object; Detection results with a confidence level lower than a preset threshold are discarded, and overlapping detection results are filtered to obtain the target detection results for the current frame.

[0009] Preferably, in step S3, the two-dimensional to three-dimensional attitude enhancement method specifically includes: Extract target region image features from image frames based on two-dimensional bounding boxes; The target region image features, camera intrinsic parameters, and camera pose information are input into a 2D-to-3D pose enhancement model; the 2D-to-3D pose enhancement model is used to estimate the center position, orientation information, and observation confidence of the target object in 3D space; When it is necessary to obtain the spatial occupancy of a target object, further estimate the three-dimensional size information of the target object.

[0010] Preferably, in step S4, the target observation state construction transforms the target object from the camera coordinate system to a unified coordinate system through coordinate transformation. The coordinate transformation relationship is as follows: ; In the formula, This represents the position of the i-th target as observed from the k-th viewpoint in the camera coordinate system. Let represent the transformation matrix from the camera coordinate system to the unified coordinate system for the k-th viewpoint at time t. This indicates the target's three-dimensional position after transformation to a unified coordinate system; the unified coordinate system is one of the world coordinate system, navigation coordinate system, or local map coordinate system.

[0011] Preferably, the target association matching in step S5 determines whether the target in the current frame and the historical target belong to the same object through a matching cost function, wherein the matching cost function is: ; In the formula, This represents the matching cost between the current i-th objective at time t and the historical j-th objective. Represents the distance term in three-dimensional space. Represents semantic feature differences. Indicates the difference in orientation. Indicates category or confidence level difference items. , , , These represent the weight coefficients of the corresponding items; When the matching cost is less than the preset matching threshold, it is determined that the target in the current frame and the historical target belong to the same object; when multiple historical targets meet the matching conditions, the historical target with the lowest matching cost is selected as the associated object; when there is no historical target that meets the matching conditions, a new target object is created and a target ID is assigned.

[0012] Preferably, the motion state update in S6 specifically includes: For each target object, a continuous frame history state sequence is saved; based on the three-dimensional spatial position change and time interval of the same target object in adjacent frames, the velocity information and position change trend of the target object are estimated; when the position change of the target object is less than a preset threshold, it is determined as a static target; when the position change of the target object is greater than or equal to the preset threshold, it is determined as a dynamic target, and its motion state information is used to generate the target prediction state.

[0013] Preferably, in step S7, the online multi-view fusion updates the current frame target observation state, historical target fusion state, and target prediction state through incremental fusion. The incremental fusion formula is: ; ; in, This represents the target global state after fusion at the current moment. Indicates the target observation status in the current frame. This represents the target predicted state obtained based on historical states. This indicates the target fusion state at the previous moment. , , These represent the fusion weights for the current observation state, the target prediction state, and the historical fusion state, respectively.

[0014] Preferably, in step S8, the repetitive target suppression method specifically includes: The existing target objects are grouped according to target category or semantic features. Within the same group, the three-dimensional spatial distance, attitude orientation difference, spatial overlap and historical trajectory overlap between different target objects are calculated. When two target objects meet the duplicate target determination conditions, the target object with more continuous historical trajectory, more stable fusion state or higher observation confidence is retained, and the effective observation information of the other target object is merged into the retained target object, or the other target object is deleted.

[0015] Preferably, the output results in step S9 specifically include: For each global target object, output the target ID, target category, three-dimensional spatial position, orientation information, motion state information, observation confidence and update timestamp. When the target object includes three-dimensional size information, output the spatial occupancy range of the target object. When the target object is determined to be a dynamic target, output its future spatial position change trend.

[0016] The beneficial effects of this invention are as follows: 1. This invention integrates open-vocabulary target detection with 2D-to-3D pose enhancement, along with time synchronization and coordinate system-one transformation, to achieve real-time 3D pose perception of target objects under multi-view or multi-frame observation. Open-vocabulary detection eliminates the dependence on fixed category lists, enhancing the ability to recognize unfamiliar objects and text-described targets. The pose enhancement model utilizes image features extracted from 2D bounding boxes and camera geometric information to directly generate 3D observation results including spatial location, pose orientation, and observation confidence. This scheme organically integrates single-frame observation results from different sources and at different times. Through multi-constraint association matching based on 3D spatial distance, semantic feature differences, pose orientation differences, and category confidence differences, it constructs target identity consistency across time frames. This effectively overcomes the target drift and discontinuity problems caused by occlusion or viewpoint changes in traditional single-frame methods. It can maintain continuous and stable localization and pose extraction of the same target object even under conditions of local occlusion and scene scale changes, solving the defects of occlusion sensitivity and poor cross-frame consistency in existing technologies.

[0017] 2. This invention introduces a motion state update mechanism based on continuous frame historical states to generate motion state information for target objects, including velocity and position change trends, and provides a predicted target state accordingly. This method no longer passively receives observations from the current frame but actively uses historical trajectories to predict the target's possible spatial location in the next moment, simultaneously using the predicted state for target association matching guidance and online fusion processes. When a target moves or is briefly occluded, the predicted state provides reasonable prior information to help maintain correct target association relationships and avoid target identity interruption or incorrect association due to brief observation gaps. Furthermore, by distinguishing between static and dynamic targets based on position changes, the system can adopt differentiated processing strategies for targets with different motion characteristics, providing richer and more reliable dynamic behavior information for intelligent navigation systems and automatic obstacle avoidance modules. This solves the shortcomings of existing technologies in handling dynamic targets and providing continuous motion state information.

[0018] 3. This invention employs an incremental online multi-view fusion mechanism. At the moment each frame of image processing is completed, the current frame's target observation state, historical target fusion state, and target prediction state are weighted and fused, updating the target's global fusion state in real time. The updated result is immediately written to the historical cache for processing in the next frame. This incremental fusion method eliminates the need to wait for the complete sequence acquisition before global batch processing optimization, ensuring that the system's response latency and output frequency meet the stringent real-time perception requirements of mobile robots and augmented reality devices. Simultaneously, by adaptively adjusting the fusion weights of the current observation, prediction, and historical states, this scheme dynamically balances the contribution ratio of new observation information to historical information based on the observation confidence and target motion state from different perspectives. When the target is observed with low quality or is occluded, it moderately relies on the historical fusion state and prediction state to maintain stable output. When high-quality observations are obtained, it rapidly converges to the accurate pose, significantly improving the real-time performance and robustness of the overall perception system in complex interactive scenarios, and overcoming the shortcomings of insufficient real-time performance in existing offline fusion technologies. Attached Figure Description

[0019] Figure 1 This is the overall system flowchart of the present invention; Figure 2 This is a flowchart of the online multi-view fusion update process of the present invention; Figure 3 This is a schematic diagram of motion state update and state feedback in the present invention; Figure 4 This is a schematic diagram of multi-view target association according to the present invention; Figure 5 This is a comparison chart of target pose estimation errors in occlusion scenarios according to embodiments of the present invention; Figure 6This is a comparison chart of dynamic target tracking errors in an embodiment of the present invention. Detailed Implementation

[0020] Example 1: This invention first uses a robot platform to collect continuous environmental images and camera pose information, and obtains the two-dimensional target detection results through an open vocabulary target detection model; then, it uses the Boxer two-dimensional to three-dimensional pose enhancement model to predict the three-dimensional pose of the target; then, it uses an online multi-view fusion module to associate, predict and incrementally fuse the target state under different time frames and different viewpoints; finally, it outputs the fused stable target state to the navigation system to realize the robot's dynamic environmental perception and real-time navigation.

[0021] like Figure 1 As shown, it includes the following steps: S1, Image and pose information acquisition: Acquire continuous image frames formed by the acquisition device during the movement, or acquire multi-view image frames acquired by multiple view acquisition devices in the same scene, and acquire the camera intrinsic parameters and camera pose information corresponding to the image frames, synchronize the image frames and pose information in time, and form a data packet to be processed. S2, Open vocabulary object detection: Input the image frames in the data packet to be processed into the open vocabulary object detection model to detect the target objects in the image and obtain the two-dimensional bounding box, target category, semantic features and detection confidence of the target objects; S3, 2D to 3D pose enhancement: Based on the 2D bounding box, image features, camera intrinsics and camera pose information, generate the pose observation results of the target object in 3D space. The pose observation results include 3D spatial position, pose orientation information and observation confidence. S4, Target observation state construction: Transform the attitude observation results to a unified coordinate system to construct the target observation state of the current frame; S5, Target Association Matching: The current frame target observation state is associated with the historical target state. If the match is successful, the corresponding historical target state is updated; if the match fails, a new target object is created. S6, Motion State Update: Based on the historical target states of the same target object in consecutive frames, estimate the motion state information of the target object and generate the target prediction state; S7, Online Multi-View Fusion: Incrementally fuse the current frame target observation state, historical target fusion state and target prediction state online to obtain the global fusion state of the target object; S8, Repeated Target Suppression and State Feedback: Merge or delete repeatedly generated target objects, and write the updated global fusion state into the historical state cache; S9, Attitude estimation results output: Output the target object's target ID, target category, stable 3D attitude, motion state information, observation confidence, and spatial location results.

[0022] This embodiment runs on Ubuntu 22.04 and is implemented using the PyTorch deep learning framework. The hardware platform includes an Intel i9 processor, an NVIDIA RTX 4090 GPU, and 64GB of RAM. The edge computing platform uses the JetsonOrin computing platform for online inference deployment. The robot platform uses a differential motion robot chassis and utilizes the ROS2 robot operating system for data communication between the perception and navigation modules. The above hardware and software environment is only one feasible implementation; in practical applications, other computing platforms can be used depending on computing power, cost, and real-time requirements.

[0023] Preferably, in step S1, this embodiment of the invention uses an Intel RealSense D455 RGB-D camera as the visual input device. For image frames requiring open-vocabulary target detection, the system scales the input image to a resolution of 960×960. For real-time deployment at the edge, the input of the auxiliary processing branch can also be uniformly scaled to 640×640 according to computing power constraints to reduce the amount of inference computation. The system synchronizes the image frame and camera pose information according to the timestamp and inputs the synchronized data into the pose estimation module.

[0024] The data to be processed at time t for the k-th viewpoint is represented as follows: ; In the formula, This represents the input image at time t from the k-th viewpoint. Indicates camera intrinsic parameters. Represents the camera pose transformation matrix. It represents inertial measurement information.

[0025] In one specific implementation of this embodiment, an indoor multi-view object pose dataset was used for experimental verification. Experimental scenarios included offices, laboratories, warehouse areas, and indoor corridor environments. Target categories included chair, table, box, bottle, backpack, person, and dynamic obstacles. The robot autonomously moved and collected data in different scenarios, acquiring approximately 12,000 frames of RGB images and corresponding camera pose information.

[0026] In this embodiment, the training set, validation set, and test set are divided in a 70:10:20 ratio. The training set is used for model training, the validation set is used for parameter tuning, and the test set is used for online fusion performance evaluation.

[0027] Preferably, in step S2, the open-vocabulary object detection module uses OWLv2 as the detection model. The system inputs the target category text, natural language description, or preset semantic prompts into OWLv2 to perform open-vocabulary detection on the target objects in the image, obtaining the two-dimensional bounding box of the target object, the target category, semantic features, and detection confidence. In other embodiments, DETIC or other open-vocabulary object detection models can also be used as alternative detection modules.

[0028] The i-th two-dimensional detection result of the current frame is represented as: ; In the formula, Represents a two-dimensional bounding box. Indicates the target category. Represents semantic features, This indicates the confidence level of the detection.

[0029] Preferably, in step S3, BoxerNet receives the target's two-dimensional bounding box, target region image features, camera intrinsic parameters, camera pose information, and optional depth or semi-dense point cloud information, and outputs the target object's three-dimensional directed bounding box in the camera coordinate system; the three-dimensional directed bounding box includes the three-dimensional center position, three-dimensional size, pose orientation information, and three-dimensional observation confidence.

[0030] BoxerNet uses a Transformer structure to fuse target region features, 2D bounding box positions, camera geometry, and optional depth information, and then performs regression prediction on the 3D oriented bounding box of the target object.

[0031] The 3D pose observation result of the i-th target in the k-th view is represented as follows: ; In the formula, This indicates the target's three-dimensional spatial position in the camera coordinate system. This indicates the target's orientation information in the camera coordinate system. Indicates the three-dimensional dimensions of the target. This represents the confidence level of the three-dimensional observation.

[0032] In this embodiment, the BoxerNet model uses ViT-Base as the backbone network for visual feature extraction, with a feature dimension of 768. The AdamW optimizer is used during model training or adaptation, with an initial learning rate of 1e-4, a batch size of 8, and a total training epoch count of 50. Further data augmentation techniques such as random rotation, random occlusion, random brightness perturbation, and Gaussian noise perturbation are employed during training to improve the model's robustness in complex environments. The above parameters represent a feasible engineering implementation configuration; in practical applications, they can be adjusted based on data scale, memory capacity, and real-time requirements.

[0033] Preferably, in step S4, in order to achieve target association between different time frames and different viewpoints, the present invention further transforms the target observation results to the world coordinate system: ; In the formula, This represents the position of the i-th target as observed from the k-th viewpoint in the camera coordinate system. This represents the transformation matrix from the camera coordinate system to the unified coordinate system. This indicates the target's three-dimensional position after transformation to a unified coordinate system; like Figure 4 As shown, the system performs two-dimensional detection and three-dimensional pose enhancement on target objects from two different perspectives, and then transforms the two three-dimensional observation results to a unified coordinate system through coordinate transformation. After the transformation, the three-dimensional positions of the two observation results in the unified coordinate system are close, and the target categories and semantic features are consistent, so target association matching can be performed.

[0034] Preferably, in step S5, the system associates and matches the current frame target observation state with the historical target states. In the target association stage, this invention simultaneously utilizes three-dimensional spatial distance, semantic feature similarity, pose orientation difference, and category consistency to construct a joint matching cost function: .

[0035] In the formula, This represents the matching cost between the current i-th objective at time t and the historical j-th objective. Represents the distance term in three-dimensional space. Represents semantic feature differences. Indicates the difference in orientation. Indicates category or confidence level difference items. , , , These represent the weight coefficients of the corresponding items.

[0036] In one feasible implementation, spatial distance weighting Set to 0.45, semantic feature weight Set to 0.30, attitude direction weight Set to 0.15, class consistency weight Set it to 0.10.

[0037] When the spatial distance between the targets is less than 0.5m and the joint matching cost is lower than the preset threshold, the system determines that the current target and the historical target belong to the same target object.

[0038] Furthermore, in step S6, the system further predicts the target's state at the next moment based on the historical trajectory; such as... Figure 3 As shown, the target is at t 2. t 1. At time t, the same target ID is maintained. The system forms a historical trajectory based on the historical position, and obtains the position change trend from the historical trajectory, thereby predicting the possible position at time t+1. The target speed is expressed as: ; In the formula, This represents the velocity information of the j-th target at time t. This represents the three-dimensional spatial position of the j-th target at time t. This represents the three-dimensional spatial position of the j-th target at the previous time step. This represents the time interval between the current moment and the previous moment; The predicted position of the target at the next moment can be represented as: ; In the formula, This represents the predicted position of the j-th target at the next time step. Indicates the prediction time interval; When the target speed is less than the preset speed threshold, the target is identified as a static target; when the target speed is greater than or equal to the preset speed threshold, the target is identified as a dynamic target, and the predicted position is used for target association matching and online multi-view fusion in the next frame.

[0039] In this embodiment, the online multi-view fusion module adopts a sliding window caching mechanism, and the system maintains the historical target state of the most recent 20 frames as the fusion cache window.

[0040] like Figure 2 As shown, when the same target is observed simultaneously from multiple perspectives, the system first performs weighted fusion based on the confidence levels of the 3D observations from different perspectives: ; In the formula, This represents the overall observation position of the j-th target at time t. This represents the set of viewpoints from which the j-th target is observed at time t. This represents the confidence level of the k-th viewpoint in the three-dimensional observation of the target. This represents the target position after the k-th viewpoint is transformed to a unified coordinate system; Subsequently, the system further performs online incremental updates by combining the current observation status, motion prediction status, and historical fusion status: ; in, This represents the target global state after fusion at the current moment. Indicates the target observation status in the current frame. This represents the target predicted state obtained based on historical states. This indicates the target fusion state at the previous moment. , , Let represent the fusion weights of the current observation state, the target prediction state, and the historical fusion state, respectively, and satisfy the following: .

[0041] In one feasible implementation, the current observation state weight Set to 0.5, motion prediction state weight Set to 0.2, historical fusion state weight Set it to 0.3.

[0042] The system dynamically adjusts the fusion weights based on the confidence level of 3D observations from different perspectives, thereby reducing the impact of occluded perspectives and low-quality observation results on the fusion state.

[0043] Preferably, in step S8, the system suppresses duplicate target objects. The system groups the current global target list according to target category or semantic features, and calculates the 3D spatial distance, pose orientation difference, spatial overlap, and historical trajectory overlap between different target objects within the same group. When two target objects meet the duplicate target determination criteria, the target object with more continuous historical trajectory, more stable fusion state, or higher observation confidence is retained, and the effective observation information of the other target object is merged into the retained target object, or the other target object is deleted. After the duplicate target suppression is completed, the system writes the updated global fusion state into the historical state cache, which is used for target association matching, motion state update, and online multi-view fusion in the next frame.

[0044] Preferably, in step S9, the system generates attitude estimation results based on the global fusion state. The attitude estimation results include target ID, target category, three-dimensional spatial position, attitude orientation information, motion state information, observation confidence, and update timestamp. When the target object includes three-dimensional size information, the system further outputs the spatial occupancy range of the target object. When the target object is determined to be a dynamic target, the system further outputs its future spatial position change trend. The attitude estimation results can be sent to the navigation system, obstacle avoidance module, path planning module, augmented reality display module, or three-dimensional environment perception module via ROS2 messages, shared memory, network packets, or other communication interfaces.

[0045] In the experiment, the system's average online inference speed reached 18 FPS, which can meet the real-time navigation requirements of indoor mobile robots.

[0046] Example 2: This embodiment deploys the method of the present invention on an indoor mobile robot platform to verify its online multi-view fusion object pose estimation effect in a dynamic and complex environment. The application scenario is an indoor experimental field with an area of ​​12m × 8m, which is equipped with static desks and chairs, movable storage boxes, and dynamic pedestrians played by experimental personnel. The mobile robot adopts a differential chassis and is equipped with two Intel RealSense D455 RGB-D cameras, one facing forward and the other facing right. The two cameras achieve hard synchronization through a hardware trigger line, acquiring color images and corresponding depth maps at a resolution of 1280×720 at a frame rate of 15 frames per second, and scaling the images to a resolution of 960×960 as input to the detection model. The computing platform is NVIDIA Jetson Orin, running the Ubuntu 22.04 operating system and ROS2 Humble middleware. The camera intrinsic parameters were pre-acquired using the Zhang Zhengyou calibration method, while the camera extrinsic parameters were calibrated using a unified robot calibration board to obtain the transformation matrices of the forward camera-to-head coordinate system and the right-side camera-to-head coordinate system. The camera pose for each frame was obtained by fusing the robot's odometry and IMU. A total of 20,000 consecutive image frames were acquired in the experiment, covering various motion modes such as stationary observation, straight-line movement, in-situ rotation, and circling.

[0047] The parameter settings are as follows: The open-vocabulary target detection model used was OWLv2, with target category text prompts set to "chair, table, box, person". The detection confidence threshold was set to 0.3, and overlapping boxes were eliminated using non-maximum suppression with a threshold of 0.45. The 2D-to-3D pose enhancement model used BoxerNet with a ViT-Base backbone and 768 feature dimensions. The regression output included the target's 3D center position, orientation quaternion, 3D dimensions, and observation confidence. The time synchronization threshold was set to 0.02 seconds; synchronization was considered effective when the difference between the image frame and the pose information timestamp was less than this threshold. The coordinate system used was the world coordinate system with the site center as the origin. In target association matching, the weight coefficients were set to... The matching cost threshold is set to 1.2. During motion state updates, the preset speed threshold is set to 0.1 m / s to distinguish between moving and static states. Incremental fusion weights. The values ​​were set to 0.5, 0.2, and 0.3 respectively. When occlusion caused the observation confidence level to fall below 0.3, the values ​​were automatically adjusted. Reduced to 0.2 The threshold is increased to 0.5 to enhance the ability to maintain historical fusion states; the spatial distance threshold for repeated target suppression is 0.5 meters, and the attitude orientation difference threshold is 15 degrees.

[0048] Two existing methods were used for comparison. The first is a single-frame pose estimation method, which also uses OWLv2 and BoxerNet to independently detect and perform 3D pose enhancement for each frame, but does not perform any inter-frame correlation, motion prediction, or multi-view fusion; it simply outputs the result of the current frame directly. The second is an offline batch processing fusion method. After acquiring the entire sequence, this method uses the observation results of all frames as input to construct a pose graph optimization model. It uses the Levenberg-Marquardt algorithm to minimize the reprojection error and smoothness constraints between the observations of each frame and the target state, and outputs the optimized global target trajectory. The batch processing window is 200 frames, and 50 frames are retained between adjacent windows to ensure continuity.

[0049] To comprehensively evaluate the performance of this method, four typical modes were selected for statistical analysis: static high-visibility targets, static transiently occluded targets, dynamic uniform-speed targets, and dynamic variable-speed turning targets. Table 1 lists the 3D position error, attitude orientation error, mean observation confidence, and average processing time per frame for the method of this invention under different modes; Table 2 shows the comparison of key indicators between this method and two comparative methods in occluded and dynamic tracking scenarios.

[0050] Table 1: Performance data of the method of the present invention under different typical modes;

[0051] Table 2: Performance comparison of this method with existing methods in key scenarios;

[0052] As can be seen from the data in the table above, the average position error of the method in the occluded scenario is significantly reduced compared to the single-frame estimation method. Furthermore, it only requires 3 frames to converge to normal accuracy after occlusion, while the single-frame method requires 18 frames. In dynamic target tracking, especially during turning, the position error of this method remains consistently below one decimeter, significantly outperforming both the single-frame method and the offline fusion method. Simultaneously, this method experiences only one target loss throughout the entire process and does not generate duplicate targets, demonstrating the effectiveness of cross-frame association and repetition suppression. In terms of real-time performance, the single-frame processing time of this method is 55 milliseconds, achieving a stable online running speed of 18 frames / second, meeting the real-time perception requirements of mobile robots. While the offline fusion method has lower accuracy, it cannot be applied to online scenarios because it requires waiting for a complete sequence. In summary, this invention combines multi-view observation, historical state, and motion prediction in real-time through incremental online fusion, achieving high-precision and high-stability 3D pose estimation under complex conditions such as occlusion, dynamics, and multiple targets. It combines real-time performance with robustness, significantly outperforming existing single-frame methods and offline batch processing methods. like Figure 5 As shown, the red curve represents the pose estimation result based solely on the current single-frame image, while the blue curve represents the result of the online multi-view fusion method proposed in this invention. In the 0 to 40 seconds before occlusion occurs, the errors of both methods are small, approximately 0.08 meters. From the 40th to the 55th second after occlusion begins, the single-frame method suffers from observation loss due to the target partially disappearing from the field of view or a significant drop in detection confidence. This is manifested in the red curve as a sudden interruption or jump to a maximum value, often leading to target loss in practical applications. In contrast, the method of this invention uses incremental fusion based on the historical fusion state and motion prediction state of previous frames, maintaining a reasonable estimate of the target position during occlusion. The blue curve shows only a slight increase in error, approximately 0.15 meters, avoiding the identity switching problem caused by target loss and re-association. After occlusion ends, this method quickly fuses newly acquired high-quality observations, and the error returns to normal levels. This advantage directly solves the occlusion sensitivity defect of existing single-frame methods, demonstrating that this invention can significantly improve the continuity and stability of pose estimation in complex scenes through cross-frame state preservation and motion prediction mechanisms.

[0053] like Figure 6As shown, the 3D position tracking errors of dynamic pedestrian targets are compared between the single-frame estimation method and the method of this invention. From 0 to 80 seconds, the pedestrian first moves in a uniform straight line, then accelerates and completes a 90-degree turn within 4 seconds from 80 to 84 seconds, and then continues to move in a straight line at a new speed. The red curve shows that the single-frame estimation error is about 0.12 meters during the uniform speed phase, but the error increases significantly to more than 0.25 meters during the turning phase. This is because the single-frame method cannot utilize motion trend information, and each estimation is performed independently, resulting in a large deviation when the motion state changes abruptly. The blue curve represents the result of the method of this invention, which uses a continuous frame motion state update mechanism and combines incremental fusion to weight the observation and prediction, reducing the error to about 0.05 meters during the uniform speed phase and only increasing to 0.10 meters during the turning phase, which is much smaller than that of the single-frame method. This shows that the present invention effectively suppresses noise amplification through motion prediction and online fusion of dynamic targets, solving the deficiency of existing technologies in obtaining continuous and stable motion state information, and providing more reliable dynamic target tracking capabilities for robot navigation and automatic obstacle avoidance.

Claims

1. An online multi-view fusion method for object pose estimation, characterized in that, Includes the following steps: S1, Image and pose information acquisition: Acquire continuous image frames formed by the acquisition device during the movement, or acquire multi-view image frames acquired by multiple view acquisition devices in the same scene. Obtain the camera intrinsic parameters and camera pose information corresponding to the image frame, synchronize the image frame and pose information in time, and form a data packet to be processed. S2, Open vocabulary object detection: Input the image frames in the data packet to be processed into the open vocabulary object detection model, detect the target objects in the image, and obtain the two-dimensional bounding box, target category, semantic features and detection confidence of the target objects; S3, 2D to 3D pose enhancement: Based on 2D bounding box, image features, camera intrinsics and camera pose information, generate the pose observation results of the target object in 3D space. The pose observation results include 3D spatial position, pose orientation information and observation confidence. S4, Target observation state construction: Transform the attitude observation results to a unified coordinate system and construct the target observation state of the current frame; S5, Target Association Matching: The current frame target observation state is associated with the historical target state. If the match is successful, the corresponding historical target state is updated; if the match fails, a new target object is created. S6, Motion State Update: Based on the historical target states of the same target object in consecutive frames, estimate the motion state information of the target object and generate the target prediction state; S7, Online Multi-View Fusion: Incrementally fuse the current frame target observation state, historical target fusion state and target prediction state online to obtain the global fusion state of the target object; S8, Repeated Target Suppression and State Feedback: Merge or delete repeatedly generated target objects, and write the updated global fusion state into the historical state cache; S9, Attitude estimation results output: Output the target object's target ID, target category, stable 3D attitude, motion state information, observation confidence, and spatial location results.

2. The online multi-view fusion object pose estimation method according to claim 1, characterized in that, In step S1, the method for acquiring image and pose data specifically includes: Add a timestamp to each frame of the image and store the image frame, camera intrinsics and camera pose information in a buffer. Image frames with similar timestamps and camera pose information are selected from the buffer to form the same data packet to be processed. When there is a time difference between the image frame and the camera pose information, the camera pose corresponding to the current image frame is interpolated or corrected based on the adjacent pose information. When using multiple perspective acquisition devices, select image frames whose timestamp difference is less than a preset threshold from each perspective to form a multi-perspective data packet to be processed.

3. The online multi-view fusion object pose estimation method according to claim 1, characterized in that, In step S2, the open vocabulary target detection method specifically includes: Input the target category text, natural language description, or preset semantic prompts into the open vocabulary target detection model; The open vocabulary object detection model is used to obtain the two-dimensional bounding box, object category, semantic features and detection confidence of the target object; Detection results with a confidence level lower than a preset threshold are discarded, and overlapping detection results are filtered to obtain the target detection results for the current frame.

4. The online multi-view fusion object pose estimation method according to claim 1, characterized in that, The two-dimensional to three-dimensional pose enhancement method in S3 specifically includes: Extract target region image features from image frames based on two-dimensional bounding boxes; The target region image features, camera intrinsic parameters, and camera pose information are input into a 2D-to-3D pose enhancement model; the 2D-to-3D pose enhancement model is used to estimate the center position, orientation information, and observation confidence of the target object in 3D space; When it is necessary to obtain the spatial occupancy of a target object, further estimate the three-dimensional size information of the target object.

5. The online multi-view fusion object pose estimation method according to claim 1, characterized in that, In step S4, the target observation state construction transforms the target object from the camera coordinate system to a unified coordinate system through coordinate transformation. The coordinate transformation relationship is as follows: ； In the formula, This represents the position of the i-th target as observed from the k-th viewpoint in the camera coordinate system. Let represent the transformation matrix from the camera coordinate system to the unified coordinate system for the k-th viewpoint at time t. This indicates the target's three-dimensional position after transformation to a unified coordinate system; the unified coordinate system is one of the world coordinate system, navigation coordinate system, or local map coordinate system.

6. The online multi-view fusion object pose estimation method according to claim 1, characterized in that, In step S5, target association matching determines whether the target in the current frame and historical targets belong to the same object through a matching cost function. The matching cost function is: ； In the formula, This represents the matching cost between the current i-th objective at time t and the historical j-th objective. Represents the distance term in three-dimensional space. Represents semantic feature differences. Indicates the difference in orientation. Indicates category or confidence level difference items. , , , These represent the weight coefficients of the corresponding items; When the matching cost is less than the preset matching threshold, it is determined that the target in the current frame and the historical target belong to the same object; when multiple historical targets meet the matching conditions, the historical target with the lowest matching cost is selected as the associated object; when there is no historical target that meets the matching conditions, a new target object is created and a target ID is assigned.

7. The online multi-view fusion object pose estimation method according to claim 1, characterized in that, The motion state update in S6 specifically includes: For each target object, a continuous frame history state sequence is saved; based on the three-dimensional spatial position change and time interval of the same target object in adjacent frames, the velocity information and position change trend of the target object are estimated; when the position change of the target object is less than a preset threshold, it is determined as a static target; when the position change of the target object is greater than or equal to the preset threshold, it is determined as a dynamic target, and its motion state information is used to generate the target prediction state.

8. The online multi-view fusion object pose estimation method according to claim 1, characterized in that, In step S7, online multi-view fusion updates the target observation state, historical target fusion state, and target prediction state of the current frame through incremental fusion. The incremental fusion formula is: ； ； in, This represents the target global state after fusion at the current moment. Indicates the target observation status in the current frame. This represents the target predicted state obtained based on historical states. This indicates the target fusion state at the previous moment. , , These represent the fusion weights for the current observation state, the target prediction state, and the historical fusion state, respectively.

9. The online multi-view fusion object pose estimation method according to claim 1, characterized in that, In step S8, the repetitive target suppression method specifically includes: The existing target objects are grouped according to target category or semantic features. Within the same group, the three-dimensional spatial distance, attitude orientation difference, spatial overlap and historical trajectory overlap between different target objects are calculated. When two target objects meet the duplicate target determination conditions, the target object with more continuous historical trajectory, more stable fusion state or higher observation confidence is retained, and the effective observation information of the other target object is merged into the retained target object, or the other target object is deleted.

10. The online multi-view fusion object pose estimation method according to claim 9, characterized in that, The output results in step S9 specifically include: For each global target object, output the target ID, target category, three-dimensional spatial position, orientation information, motion state information, observation confidence and update timestamp. When the target object includes three-dimensional size information, output the spatial occupancy range of the target object. When the target object is determined to be a dynamic target, output its future spatial position change trend.

Images