Arm three-dimensional modeling method and system based on image processing and inertial sensing

Abstract

The invention relates to an arm three-dimensional modeling method and system based on image processing and inertial sensing, and belongs to the technical field of computer vision and three-dimensional modeling. The method comprises the following steps: acquiring multiple frames of RGB images through a binocular camera and synchronously acquiring motion data provided by an IMU (Inertial Measurement Unit); segmenting the RGB image by using a deep learning model, and extracting an arm region; generating depth mapping through a stereo matching algorithm of a binocular camera, and converting pixel coordinates and depth values into three-dimensional point cloud data; performing preliminary calibration on the point cloud in combination with motion parameters provided by an IMU (Inertial Measurement Unit); performing dynamic registration and fusion on the multi-frame point cloud data to generate a complete three-dimensional arm model; the modeling process is checked in real time through a user interaction interface, and a parameter optimization result is adjusted. The system comprises a data acquisition module, an image processing and region extraction module, a point cloud generation and preliminary calibration module, a point cloud fusion and optimization module and a user interaction module, which cooperatively complete a high-precision modeling task of an arm. The method and the system can efficiently and automatically generate an accurate arm three-dimensional model, remarkably improve modeling efficiency, reduce manual intervention and equipment cost, and are suitable for various application scenes such as medical rehabilitation and ergonomic design.

Description

Technical field:

[0001] The present invention relates to the field of three-dimensional modeling technology, and specifically to a three-dimensional modeling method and system for an arm based on image processing and inertial sensing, which aims to generate an accurate arm model efficiently and in real time. It can be applied to the three-dimensional modeling stage in scenarios such as medical rehabilitation and ergonomic analysis. The system automatically filters out the arm area for the RGB image taken by a binocular camera, eliminates the interference of duplicate background, and converts it into a point cloud that only retains the arm area. At the same time, it combines with IMU to realize point cloud registration, thereby quickly generating an accurate arm model. This method significantly shortens the modeling time, improves the modeling accuracy, and reduces the dependence on manual operation. Background technology:

[0002] In medical rehabilitation and ergonomic design, three-dimensional modeling of the arm is a core link in evaluation and design. However, traditional methods usually rely on manual operations, and technicians gradually build models through CAD tools, which is time-consuming, inefficient, and requires high skills. It is also prone to frequent modifications due to human errors, which increases costs and prolongs the development cycle. In medical rehabilitation, the dynamic changes of patients' arms are frequent, making modeling and verification more cumbersome. Although multi-view image modeling can improve accuracy, it requires a large amount of high-quality dense view image data, relies on expensive equipment and complex scene settings, is sensitive to background environments, and has poor deployment flexibility, making it difficult to meet the actual application needs in the fields of medicine and ergonomics.

[0003] Therefore, how to achieve simple operation and automatically generate high-precision arm 3D models is an urgent problem to be solved. Summary of the invention:

[0004] In view of this, the purpose of the present invention is to provide a three-dimensional arm modeling method and system based on image processing and inertial sensing, which can automatically identify the arm area and generate its three-dimensional model using a small amount of image data at different angles, so as to quickly obtain a high-precision model for subsequent medical scenario applications.

[0005] The first aspect of the present invention discloses a method for three-dimensional modeling of an arm based on image processing and inertial sensing, the method comprising:

[0006] Use a binocular camera to capture multiple frames of RGB images, and simultaneously obtain the camera's internal and external parameters and the motion information of the IMU sensor (such as the perspective change matrix and rotation angle). To improve the capture quality of the arm area, it is recommended to optimize the shooting environment, such as keeping the background simple and the light consistent. By collecting different angles of the arm from multiple perspectives, ensure that complete arm data is obtained, providing high-quality input for subsequent modeling.

[0007] The collected RGB images are processed by deep learning to automatically identify and segment the arm area, which specifically includes the following steps: Arm region recognition: The DeepLabv3+ model is used to extract multi-scale contextual features through dilated convolution and spatial pyramid pooling (ASPP) technology to accurately segment the arm region from the RGB image. The decoder module is used to further optimize the segmentation boundary, ensure the integrity of the arm region, and eliminate background and other irrelevant interference. Optional post-processing (depending on the specific scenario): If there are small area noise or boundary discontinuity problems in the segmentation results, morphological operations (such as closing operations) can be applied for optimization. Use area threshold filtering to remove small pseudo-target areas and retain the main structure that conforms to the arm shape characteristics.

[0008] The segmented arm area image is converted into 3D point cloud data, which includes the following steps: Depth map generation: The stereo matching algorithm of the binocular camera is used to calculate the depth information (z value) of each pixel and obtain the position (u, v, z) of each pixel in the image coordinate system. Depth mapping is the basis of point cloud generation and provides the distance information of each pixel to the camera. Coordinate system conversion: Use the camera intrinsic parameters (focal length fx, fy and principal point coordinates cx, cy) to convert the two-dimensional pixel coordinates (u, v) and depth value z into a three-dimensional point (x, y, z) in the camera coordinate system. Preliminary calibration: Use the motion parameters provided by the IMU (such as rotation matrix and translation vector) to align each frame of point cloud from the camera coordinate system to the global coordinate system. The IMU data provides the preliminary position and posture information of each frame of point cloud during the camera shooting process, compensates for the camera's motion changes, ensures that each frame of point cloud is represented in a unified reference frame, and lays the foundation for subsequent point cloud fusion.

[0009] Based on the preliminary calibration, the multi-view point clouds are fused to generate a complete 3D model of the arm. The specific steps are as follows: Dynamic registration: Based on the geometric relationship between multi-frame point clouds, combined with the rotation matrix and translation vector provided by IMU, multi-frame point clouds are dynamically registered with high precision. The point cloud registration method (such as ICP algorithm or feature point-based matching method) is used to optimize the alignment results of multi-frame point clouds and compensate for the accumulated error of IMU data and the perspective deviation in the visual method. Noise processing and optimization: Statistical filtering and outlier removal are performed on the initially fused point cloud to remove discrete noise points in the point cloud. The point cloud surface is further smoothed and optimized to improve data quality and provide reliable input for the final model generation. Point cloud fusion: Based on the internal and external parameters of the camera, the point cloud is unified and stitched together, and all point cloud data are integrated into a global model. The final 3D point cloud file (such as PLY format) is output to fully describe the 3D structure of the arm and ensure the integrity and consistency of the model.

[0010] Through the user interaction interface developed based on PyQt, the modeling process can be monitored in real time and adjusted dynamically: Dynamic display: Users can view the point cloud generation and registration process, as well as the complete 3D model that is gradually constructed, in real time through the interface. Parameter adjustment: Provides adjustable options such as filter strength, segmentation threshold, etc., so that users can optimize the results according to modeling needs. Result export: After modeling is completed, the user can export a high-precision 3D arm model with one click to facilitate subsequent analysis and application.

[0012] The first aspect of the present invention discloses an arm three-dimensional modeling system based on image processing and inertial sensing, which includes a data acquisition module, an image processing and region extraction module, a point cloud generation and preliminary calibration module, a point cloud fusion and optimization module, and a user interaction module.

[0013] The data acquisition module uses a binocular camera to collect multiple frames of RGB images and synchronously obtains motion data provided by the IMU (such as perspective change matrix and rotation angle). It supports multi-view acquisition to ensure coverage of different angles and complete structural data of the arm. The data acquisition process is controlled by the module, allowing manual or automatic mode switching.

[0014] The image processing and region extraction module uses the DeepLabv3+ model to segment the arm region, optimize the segmentation boundary, and remove background interference. If further optimization is required, morphological operations and area filtering can be applied to remove noise or discontinuous boundary regions.

[0015] The point cloud generation and preliminary calibration module generates a depth map based on stereo matching of the binocular camera, converting pixel coordinates and depth values ​​into three-dimensional point cloud data in the camera coordinate system. Using the motion parameters provided by the IMU, each frame of the point cloud is calibrated from the camera coordinate system to the global coordinate system, providing a basis for point cloud fusion.

[0016] The point cloud fusion and optimization module uses IMU data and point cloud registration algorithms (such as ICP) to dynamically align multi-frame point clouds and optimize the consistency of multi-view point clouds. Noise points are removed by filtering and outlier removal, and the fused point clouds are integrated into a complete 3D model and output as a standard format file (such as PLY).

[0017] The user interaction module is based on the interface developed by PyQt, which supports real-time viewing of point cloud generation, calibration and fusion processes. It provides parameter adjustment functions (such as filter strength and segmentation threshold) and supports one-click export of high-precision 3D models. Beneficial effects:

[0018] The present invention significantly improves modeling efficiency, reduces manual participation, and reduces the possibility of human error through a three-dimensional arm modeling method based on image processing and inertial sensing. Compared with the traditional manual modeling method, this method uses a binocular camera and a deep learning algorithm to achieve automatic segmentation and point cloud generation, greatly shortens the modeling time, and reduces dependence on manual skills. Compared with the high-cost modeling method that relies on high-precision equipment and complex operations, the present invention uses conventional binocular cameras and IMU data, combined with algorithm optimization, to achieve efficient point cloud registration and fusion, effectively reducing equipment and computing costs. At the same time, the model quality is improved by dynamic filtering and noise removal to ensure that the generated three-dimensional arm model is accurate and consistent. The present invention also provides a graphical user interface that supports real-time adjustment of modeling parameters and result export, further improving practicality and ease of use. Description of the drawings:

[0019] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings: Figure 1 A schematic flow chart of a method for three-dimensional arm modeling based on image processing and inertial sensing disclosed in Embodiment 1 of the present invention; Figure 2 This is a structural schematic diagram of an arm three-dimensional modeling system based on image processing and inertial sensing disclosed in Embodiment 2 of the present invention. Specific implementation method:

[0020] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0021] Embodiment 1

[0022] The embodiment of the present invention discloses a method for three-dimensional modeling of an arm based on image processing and inertial sensing. Figure 1 , Figure 1 A schematic flow chart of a method for three-dimensional modeling of an arm based on image processing and inertial sensing disclosed in an embodiment of the present invention. The method comprises the following steps:

[0023] Step 1: Data Collection Use a binocular camera to capture multiple frames of RGB images, and synchronously obtain the motion information provided by the IMU (such as the view change matrix and rotation angle). Acquisition settings: The camera's internal and external parameters have been preset, and the resolution is set to 640×480 pixels. Try to ensure consistent lighting and a simple background in the shooting environment to reduce interference and improve data quality. Multi-view acquisition: Adjust the camera position and angle to collect images of the arm in different directions to ensure complete coverage of the arm area and provide high-quality input data for subsequent modeling.

[0024] Step 2: Image processing and region segmentation The collected RGB images are processed using deep learning to accurately segment the arm area: Arm region recognition: Using the DeepLabv3+ model, we extract multi-scale contextual features through dilated convolution and spatial pyramid pooling (ASPP) techniques to accurately segment the arm region from the RGB image. We use the decoder module to optimize the segmentation boundaries to ensure that the segmentation results are complete and accurate. Post-processing optimization (optional): For problems with small area noise or discontinuous boundaries, morphological operations (such as closing operations) can be used to further optimize the segmentation results. Use area thresholds to filter out smaller pseudo-target areas and retain only the main areas that meet the arm shape characteristics.

[0025] Step 3: Generate 3D point cloud Convert the segmented arm area image into 3D point cloud data: Depth map generation: The depth value z of each pixel is calculated through the stereo matching algorithm of the binocular camera, and a depth map is generated. The calculation formula of the depth value is: z=f*b / d Where f is the focal length of the camera (unit: mm); b is the baseline length of the binocular camera (the distance between the left and right eyes, unit: mm); d is the pixel disparity value (unit: pixel), that is, the horizontal position difference between the corresponding pixels of the left and right eyes. Coordinate system conversion: Convert the image coordinates (u, v) and depth value z to three-dimensional points (x, y, z) in the camera coordinate system. The conversion formula is: x=(u-cx)*z / fx y=(v-cy)*z / fy z=z Where u, v are pixel coordinates (positions on the image); cx, cy are the coordinates of the principal point of the image (the intersection of the optical axis and the image plane, unit: pixel); fx, fy are the pixel units of the camera focal length in the image coordinate system (i.e., the focal length value expressed in pixels); z is the depth value calculated from stereo matching.

[0026] Step 4: Point cloud calibration and fusion Preliminary calibration: Use the motion parameters (rotation matrix R and translation vector T) provided by the IMU to transform each frame of the point cloud from the camera coordinate system to the global coordinate system: P_global=R*P_camera+T Where P_camera is the point cloud coordinate in the camera coordinate system; R is the rotation matrix, which describes the change in the camera's posture; T is the translation vector, which describes the change in the camera's spatial position; P_global is the point cloud coordinate in the global coordinate system. Point cloud fusion: Based on dynamic registration methods (such as ICP algorithm or feature point matching), the alignment accuracy of multi-frame point clouds is further optimized, the accumulated errors are eliminated, and the point cloud fusion is completed.

[0027] Step 5: Point cloud denoising and optimization Noise removal: Use radius filtering to remove isolated noise points: Delete points whose distance to their nearest neighbor is greater than a certain threshold. Use statistical filtering to remove outliers: Remove abnormal points based on the statistical information of the local distribution of the point cloud (such as the average distance from the point to the nearest neighbor). Model optimization: Smooth the point cloud surface to reduce the impact of noise on the quality of the 3D model.

[0028] Step 6: Model visualization and export Dynamic visualization: Use a PyQt-based user interface to display the point cloud generation and fusion process in real time. Users can dynamically view the gradual construction of the model and operate the model (rotate, zoom, etc.) through the interactive interface. Result export: Users can export the final 3D model with one click, supporting PLY and OBJ file formats for subsequent applications.

[0029] Embodiment 2

[0030] The second aspect of the present invention discloses a three-dimensional modeling system of an arm based on image processing and inertial sensing, see Figure 2 , Figure 2 The present invention discloses a structural diagram of an arm three-dimensional modeling system based on image processing and inertial sensing, which comprises a data acquisition module, an image processing and region extraction module, a point cloud generation and preliminary calibration module, a point cloud fusion and optimization module, and a user interaction module.

[0031] The data acquisition module uses a binocular camera to collect multiple frames of RGB images, and simultaneously obtains motion data provided by the IMU (such as perspective change matrix and rotation angle). It supports multi-perspective acquisition to ensure that different angles of the arm are covered and complete structural data is obtained. It provides manual and automatic mode switching, and the acquisition process is automatically controlled by the module.

[0032] The image processing and region extraction module uses the DeepLabv3+ model to segment the RGB image, accurately extract the arm area, and remove background interference. According to actual needs, morphological operations and area filtering can be used to remove noise points or boundary discontinuities in the segmentation to ensure the integrity of the segmentation results.

[0033] The point cloud generation and preliminary calibration module generates a depth map based on the stereo matching algorithm of the binocular camera and converts the segmented image into 3D point cloud data. Using the motion parameters provided by the IMU, each frame of the point cloud is aligned from the camera coordinate system to the global coordinate system to form a unified reference frame, providing a basis for point cloud fusion.

[0034] The point cloud fusion and optimization module uses IMU data combined with point cloud registration algorithms (such as ICP) to perform high-precision dynamic alignment of multi-frame point clouds to ensure the spatial consistency of multi-view point cloud data. It removes noise in the point cloud through filtering and outlier removal, and optimizes the surface smoothness of the model. The fused point cloud is used to generate a complete 3D arm model and output as a standard format file (such as PLY).

[0035] The user interaction module develops the user interface based on PyQt, which supports real-time viewing of the dynamic process of point cloud generation, calibration and fusion. It provides parameter adjustment functions such as filter intensity and segmentation threshold, and users can optimize the results according to their needs. It supports one-click export of high-precision 3D models in PLY or OBJ file formats for subsequent application and analysis.

[0036] It should be noted that the specific implementation process of the second embodiment is similar to that of the first embodiment and will not be repeated in the second embodiment.

[0037] Finally, it should be noted that the three-dimensional arm modeling method and system based on image processing and inertial sensing disclosed in the embodiment of the present invention only discloses the preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A three-dimensional modeling method of an arm based on image processing and inertial sensing, characterized in that: The following steps are involved: Step 1: Use a binocular camera to collect multiple frames of RGB images and synchronously obtain motion data provided by the IMU sensor; Step 2: deep learning-driven processing is performed on the collected RGB images to automatically identify and segment the arm area; Step 3: Generate a depth map based on the stereo matching algorithm of the binocular camera, and convert the pixel coordinates and depth values ​​into three-dimensional point cloud data in the camera coordinate system; Step 4: Use the motion parameters provided by the IMU to calibrate each frame of point cloud from the camera coordinate system to the global coordinate system; Step 5: Dynamically register and fuse the multi-frame point clouds to generate a complete three-dimensional arm model; Step 6: View the modeling process in real time through the user interface, adjust parameters and export the final model.

2. The method according to claim 1, characterized in that The arm area is identified using the DeepLabv3+ model, multi-scale context feature extraction is achieved through atrous convolution and spatial pyramid pooling (ASPP) technology, and the segmentation boundary is optimized using a decoder module.

3. The method according to claim 1, characterized in that The stereo matching algorithm calculates the depth value of each pixel and generates a depth map based on the camera intrinsic parameters (focal length fx, fy and principal point coordinates cx, cy).

4. The method according to claim 1, characterized in that: The dynamic registration is based on IMU data and a point cloud registration algorithm (such as ICP) to achieve high-precision alignment of multi-view point cloud data.

5. The method according to claim 1, characterized in that The point cloud optimization includes removing noise points by statistical filtering and outlier removal, and smoothing the point cloud surface.

6. The method according to claim 1, characterized in that The user interaction interface is developed based on PyQt, supports real-time monitoring of point cloud generation, calibration and fusion, and provides parameter adjustment and model export functions.

7. An arm three-dimensional modeling system based on image processing and inertial sensing, characterized in that: Includes the following modules: Data acquisition module, used to collect multiple frames of RGB images and IMU data through a binocular camera; The image processing and region extraction module is used to perform deep learning segmentation on the collected RGB images and extract the arm region; Point cloud generation and preliminary calibration module, used to generate 3D point cloud data and calibrate it through IMU data; Point cloud fusion and optimization module, used to register, fuse and optimize multi-frame point clouds; User interaction module, used to monitor the modeling process in real time and export the model.

8. The system according to claim 7, characterized in that The image processing and region extraction module uses the DeepLabv3+ model to segment the arm region and further optimizes the segmentation result through morphological operations.

9. The system according to claim 7, characterized in that The point cloud fusion and optimization module combines IMU data and point cloud registration algorithm to achieve dynamic alignment and fusion of multi-frame point clouds.

10. The system according to claim 7, characterized in that The user interaction module supports real-time display, rotation, scaling and translation of the model, and can optimize the modeling results through parameter adjustment.

Images