An interactive system based on monocular vision and feature point cloud fusion

Abstract

This invention relates to the field of human-computer interaction technology, and in particular to an interactive system based on the fusion of monocular vision and feature point cloud. The system includes: an inertial measurement unit (IMU) for synchronously acquiring motion acceleration and angular velocity data from a monocular image acquisition module; a feature processing module for extracting ORB feature points from consecutive image frames and performing inter-frame matching, outputting feature matching pairs; a scale and pose calculation module for fusing feature matching pairs with data acquired by the IMU, recovering the absolute scale of the human-computer interaction scene through a tightly coupled optimization algorithm, and calculating the real-time pose of the interactive system; and a point cloud generation and optimization module for triangulating ORB feature points from consecutive image frames based on the absolute scale and real-time pose to generate an initial sparse point cloud, removing dynamic interference points through a dynamic point culling algorithm, and filling point cloud holes through a point cloud completion algorithm, outputting a dense optimized point cloud. This invention provides an interactive system capable of meeting high-precision interaction requirements.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to an interactive system based on the fusion of monocular vision and feature point cloud. Background Technology

[0002] With the rapid development of human-computer interaction technology, visual perception-based interaction systems have become a core application area in fields such as smart terminals, industrial control, and AR / VR due to their characteristics of not requiring additional wearable devices and offering natural and intuitive operation. Among them, the interaction system that combines monocular vision with feature point cloud fusion has the advantage of achieving 3D environmental perception and interaction with only a single camera, significantly reducing hardware deployment costs, simplifying system architecture, and avoiding the equipment complexity and high price issues of solutions such as binocular vision or LiDAR. It has been widely promoted in consumer electronics and lightweight industrial scenarios.

[0003] However, in practical applications, monocular vision cannot directly measure physical depth and must indirectly recover 3D information through motion triangulation, leading to inherent scale uncertainty in the system. Triangulation noise is significant for distant feature points, while close-range features are easily lost due to occlusion. Depth measurement range is limited and reliability is low, failing to meet high-precision interaction requirements. Existing systems rely on image texture to extract feature points; in low-texture environments, the success rate of feature point extraction and matching drops sharply, even causing localization and point cloud mapping to collapse. Repetitive texture scenes easily lead to mismatched features, causing pose jumps and point cloud misalignment. Furthermore, the system is extremely sensitive to changes in lighting; strong light, backlight, shadows, or low light environments can disrupt the stability of feature descriptors, resulting in a sharp decrease in the number of features and reduced matching accuracy. Interference from dynamic objects can also cause point cloud ghosting and map contamination, further reducing interaction reliability. Existing systems only construct point clouds based on sparse feature points, lacking dense surface information coverage of the scene. The point clouds have many holes and poor continuity, limiting interaction dimensions and operational precision, making it difficult to meet the interaction needs of complex scenes. System initialization relies on sufficient translational motion and parallax, and initialization cannot be completed in pure rotational scenes. During tracking, feature points being occluded, large-angle changes in viewpoint, or rapid movement can all lead to tracking loss, and the success rate of relocalization after loss is low. Especially in the case of scene changes or no common-view features, it is difficult to achieve closed-loop optimization, and errors continue to accumulate. Summary of the Invention

[0004] The purpose of this invention is to provide an interactive system based on the fusion of monocular vision and feature point cloud, so as to solve the technical problems of scale drift, inaccurate depth, poor robustness and unstable interaction in the above-mentioned background technology.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: An interactive system based on the fusion of monocular vision and feature point cloud includes: A monocular image acquisition module is used to acquire continuous image frames in real time in human-computer interaction scenarios; An inertial measurement unit is used to synchronously acquire motion acceleration and angular velocity data from the monocular image acquisition module. The feature processing module extracts ORB feature points from the consecutive image frames and performs inter-frame matching, outputting feature matching pairs. The scale and pose calculation module integrates the feature matching pairs with the motion acceleration and angular velocity data from the monocular image acquisition module, and recovers the absolute scale of the human-computer interaction scene through a tightly coupled optimization algorithm to calculate the real-time pose of the interaction system. The point cloud generation and optimization module, based on the absolute scale and the real-time pose, triangulates the ORB feature points of continuous image frames to generate an initial sparse point cloud, removes dynamic interference points through a dynamic point culling algorithm, and then fills the point cloud holes through a point cloud completion algorithm to output a dense optimized point cloud. The interactive decision-making module, based on the dense optimized point cloud and real-time pose, realizes collision detection, accurate trajectory tracking and interactive command generation, thus completing human-computer interaction.

[0006] Furthermore, the monocular image acquisition module is a high-definition monocular camera used to acquire continuous RGB image frames of human-computer interaction scenarios in real time, with a frame rate of no less than 30fps for the continuous RGB image frames.

[0007] Furthermore, the inertial measurement unit is time-synchronized with the monocular image acquisition module.

[0008] Furthermore, the feature processing module also includes an illumination adaptive adjustment unit, which enhances the feature recognition of continuous image frames in low light environment by equalizing the image brightness histogram, and uses an adaptive threshold to filter the ORB feature points of continuous image frames, and completes inter-frame matching in combination with BRISK descriptors.

[0009] Furthermore, the scale and pose calculation module adopts a sliding window iterative optimization strategy. In the initialization stage, a motion recovery structure is constructed through monocular vision. The preliminary three-dimensional structure of the human-computer interaction scene is reconstructed through the motion recovery structure. The scale factor and gravity direction are calibrated online by combining the motion acceleration data and motion angular velocity data collected by the inertial measurement unit.

[0010] Furthermore, the dynamic point removal algorithm is based on the label consistency detection principle. It separates and marks ground points in the human-computer interaction scenario, calculates the proportion of ground points among the nearest neighbors of non-ground points, and determines that a point is dynamic and removes it when the proportion is higher than a preset threshold. At the same time, it sets an uncertain point container to temporarily store points to be determined, and performs a second verification in combination with the reconstruction results of the human-computer interaction scenario.

[0011] Furthermore, the point cloud completion algorithm includes a depth hinting module and a geometry preservation fusion module. The depth hinting module projects the initial sparse point cloud into a depth map, fills the holes using a 2D repair model, and generates an RGB image. The geometry preservation fusion module combines the RGB image with the 3D generation model to generate a complete shape point cloud. After ICP alignment and dynamic scale adaptation, the complete shape point cloud is fused with the initial sparse point cloud to preserve the original geometric structure.

[0012] Furthermore, it also includes a loop closure detection and optimization module, which detects the common viewing area of ​​the human-computer interaction scene based on the bag-of-words model. When a loop closure is detected, the pose error and point cloud coordinates are corrected through a global graph optimization algorithm.

[0013] Furthermore, the loopback is the historical access area.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention achieves precise scale and reliable depth in a monocular interactive system by setting up an inertial measurement unit, a scale and pose calculation module, and a point cloud generation and optimization module. By using a tightly coupled optimization algorithm to fuse visual and inertial data, it solves the inherent scale uncertainty problem of monocular vision, eliminates scale drift, expands the depth measurement range, and improves the depth accuracy at near and far distances. Through dynamic point culling and point cloud completion, it reduces point cloud ghosting and map pollution caused by dynamic objects, fills sparse point cloud holes, enhances point cloud continuity, effectively improves the feature extraction and matching stability of the system in low-texture and repetitive texture environments, avoids localization crashes and pose jumps, improves the interaction accuracy and environmental adaptability in complex scenes, and meets the requirements of high-precision interaction.

[0015] 2. This invention achieves long-term stable operation and efficient human-computer interaction by setting up an adaptive illumination adjustment unit, a loop closure detection and optimization module, and an interactive decision-making module. The adaptive illumination processing can resist interference from strong light, weak light, and backlight, ensuring the stability of feature descriptors and improving the robustness of matching under different illumination conditions. Loop closure detection and global graph optimization can correct accumulated pose errors, solve the problem of difficult relocation after tracking loss, and ensure the long-term stable operation of the system. At the same time, the dense optimized point cloud provides complete geometric information for collision detection and trajectory tracking, expands the interaction dimension and operation accuracy, and is more suitable for consumer electronics and lightweight industrial interaction scenarios. Attached Figure Description

[0016] Figure 1 This is a system block diagram of the present invention; Detailed Implementation The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0017] See Figure 1As shown, an interactive system based on monocular vision and feature point cloud fusion includes: a monocular image acquisition module, an inertial measurement unit, a feature processing module, a scale and pose calculation module, a point cloud generation and optimization module, an interactive decision-making module, and a loop closure detection and optimization module.

[0018] The monocular image acquisition module is a high-definition monocular camera, fixed to the intelligent interactive terminal, which acquires continuous RGB image frames of human-computer interaction scenes in real time with a frame rate of no less than 30fps to ensure image continuity.

[0019] The inertial measurement unit is time-synchronized with the high-definition monocular camera, and collects the motion acceleration and angular velocity data of the high-definition monocular camera in real time, providing motion priors for scale recovery.

[0020] The feature processing module preprocesses consecutive image frames, performs brightness histogram equalization through an illumination adaptive adjustment unit to suppress the impact of sudden illumination changes, then extracts ORB feature points from consecutive image frames, combines BRISK descriptors to enhance feature robustness, completes inter-frame feature matching, eliminates mismatched pairs, and outputs stable feature matching results.

[0021] The scale and pose calculation module adopts a sliding window iterative optimization strategy, which tightly couples the feature matching pairs with the motion acceleration and angular velocity data collected by the inertial measurement unit. In the initialization stage, a motion recovery structure is constructed through monocular vision. The preliminary 3D structure of the human-computer interaction scene is reconstructed through the motion recovery structure. The scale factor and gravity direction are calibrated online by combining the motion acceleration and angular velocity data collected by the inertial measurement unit to achieve absolute scale calculation, avoid the problem of pure rotation not being able to initialize, and output stable pose in real time.

[0022] The point cloud generation and optimization module triangulates feature points based on the real-time pose of the interactive system and the scale information of the human-computer interaction scene to generate an initial sparse point cloud. A dynamic point removal algorithm based on label consistency is adopted to first mark ground points and then judge and remove dynamic targets through neighborhood statistics. Subsequently, the point cloud holes are filled by the depth prompting module and the geometry preservation fusion module, and the continuous, interference-free dense optimized point cloud is generated.

[0023] The loop closure detection and optimization module detects the common viewing area in human-computer interaction scenarios based on the bag-of-words model. When a historical access area is identified, global graph optimization is triggered to correct pose drift and point cloud offset, and suppress the accumulation of errors over a long period of time.

[0024] The interactive decision-making module performs spatial collision detection and target trajectory tracking based on dense optimized point clouds, analyzes gestures or interactive target movement intentions, generates stable interactive commands, and achieves high-precision, low-latency human-computer interaction.

[0025] Actual testing shows that this system can still track stably in scenes with low texture, complex lighting, and dynamic interference, significantly improving point cloud integrity and pose accuracy. It also has low hardware cost and simple structure, making it suitable for lightweight scenarios such as AR interaction and smart terminal control.

[0026] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. An interactive system based on monocular vision and feature point cloud fusion, characterized in that, include: A monocular image acquisition module is used to acquire continuous image frames in real time for human-computer interaction scenarios; An inertial measurement unit is used to synchronously acquire motion acceleration and angular velocity data from the monocular image acquisition module. The feature processing module extracts ORB feature points from the consecutive image frames and performs inter-frame matching, outputting feature matching pairs. The scale and pose calculation module integrates the feature matching pairs with the motion acceleration and angular velocity data from the monocular image acquisition module, and recovers the absolute scale of the human-computer interaction scene through a tightly coupled optimization algorithm to calculate the real-time pose of the interaction system. The point cloud generation and optimization module, based on the absolute scale and the real-time pose, triangulates the ORB feature points of continuous image frames to generate an initial sparse point cloud, removes dynamic interference points through a dynamic point culling algorithm, and then fills the point cloud holes through a point cloud completion algorithm to output a dense optimized point cloud. The interactive decision-making module, based on the dense optimized point cloud and real-time pose, realizes collision detection, accurate trajectory tracking and interactive command generation, thus completing human-computer interaction. 2.The interactive system based on monocular vision and feature point cloud fusion of claim 1, wherein: The monocular image acquisition module is a high-definition monocular camera used to acquire continuous RGB image frames of human-computer interaction scenarios in real time, with a frame rate of no less than 30fps for the continuous RGB image frames. 3.The interactive system based on monocular vision and feature point cloud fusion of claim 1, wherein: The inertial measurement unit is time-synchronized with the monocular image acquisition module. 4.The interactive system based on monocular vision and feature point cloud fusion of claim 1, wherein: The feature processing module further includes an illumination adaptive adjustment unit, which enhances the feature recognition of continuous image frames in low light environment by equalizing the image brightness histogram, and uses an adaptive threshold to filter ORB feature points of continuous image frames, and completes inter-frame matching in combination with BRISK descriptors.

5. The interactive system based on monocular vision and feature point cloud fusion according to claim 1, characterized in that: The scale and pose calculation module adopts a sliding window iterative optimization strategy. In the initialization stage, a motion recovery structure is constructed through monocular vision. The initial three-dimensional structure of the human-computer interaction scene is reconstructed through the motion recovery structure. The scale factor and gravity direction are calibrated online by combining the motion acceleration data and motion angular velocity data collected by the inertial measurement unit. 6.The interactive system based on monocular vision and feature point cloud fusion of claim 1, wherein: The dynamic point removal algorithm is based on the label consistency detection principle. It separates and marks ground points in the human-computer interaction scene, calculates the proportion of ground points among the nearest neighbors of non-ground points, and determines that a point is dynamic and removes it when the proportion is higher than a preset threshold. At the same time, it sets an uncertain point container to temporarily store points to be determined, and performs secondary verification in combination with the reconstruction results of the human-computer interaction scene. 7.The interactive system based on monocular vision and feature point cloud fusion of claim 1, wherein: The point cloud completion algorithm includes a depth hinting module and a geometry preservation fusion module. The depth hinting module projects the initial sparse point cloud into a depth map, fills the holes with a 2D repair model, and generates an RGB image. The geometry preservation fusion module combines the RGB image with the 3D generation model to generate a complete shape point cloud. After ICP alignment and dynamic scale adaptation, the complete shape point cloud is fused with the initial sparse point cloud to preserve the original geometric structure. 8.The interactive system based on monocular vision and feature point cloud fusion of claim 1, characterized in that: Also includes: The loop closure detection and optimization module detects the common viewing area of ​​the human-computer interaction scene based on the bag-of-words model. When a loop closure is detected, the global graph optimization algorithm is used to correct the pose error and point cloud coordinates. 9.The interactive system based on monocular vision and feature point cloud fusion of claim 8, wherein: The loopback refers to the historical access area.

Images