Arm three-dimensional modeling method and system using binocular depth camera to shoot

Abstract

The invention discloses a method and a system for performing three-dimensional modeling on an arm by using a binocular depth camera, which are suitable for the fields of medical rehabilitation, ergonomic analysis and the like. The method comprises the steps that a binocular depth camera is used for collecting multiple frames of arm depth images, Gaussian filtering and morphological processing are carried out on the images, noise is eliminated, and background interference is removed; hue standardization is carried out through histogram equalization, Gamma correction and reference frame matching; and converting the depth map into three-dimensional point cloud data by using internal parameters and external parameters of the camera, and fusing multiple frames of point clouds to obtain a complete model. The system is composed of a data acquisition module, a data preprocessing module, a point cloud generation and modeling module and a user interaction module. The system improves the modeling efficiency, reduces the manual participation degree, and has the characteristics of low cost, simplicity and convenience in operation and high precision.

Description

Technical Field

[0001] The present invention relates to the field of 3D modeling technology, and more particularly to a method and system for 3D modeling of an arm using a binocular depth camera, which aims to improve the real-time and simplicity of 3D modeling of an arm, and can be applied to the 3D conceptual modeling stage in scenarios such as medical rehabilitation and ergonomic analysis. By automatically processing depth map data, the present invention significantly shortens the modeling time and reduces the reliance on manual operations. Background Art

[0002] In medical rehabilitation and ergonomic design, 3D modeling of arms is an important part of evaluation and design. Traditional 3D modeling methods usually rely on manual operations, and technicians use CAD tools to gradually create models on computers. This process is time-consuming and requires high skills of modelers, especially in medical rehabilitation scenarios. Modeling and verification need to be repeated according to the dynamic changes of the patient's arm, which is inefficient and prone to human errors, resulting in repeated revisions of subsequent designs, extending the development cycle and increasing costs. If you want to generate a 3D model through multi-view images, you usually need a large amount of high-quality image data from dense perspectives. This method requires high hardware equipment costs, complex scene settings, and is difficult to flexibly deploy in practical applications.

[0003] Therefore, how to achieve simple operation and automatically generate a three-dimensional model of the arm is an urgent problem to be solved. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method and system for three-dimensional modeling of an arm using a binocular depth camera, which can automatically generate a three-dimensional model of the arm through image data from a relatively sparse perspective, so as to quickly obtain a high-precision model for subsequent medical scenarios.

[0005] The first aspect of the present invention discloses a method for three-dimensional modeling of an arm photographed by a binocular depth camera, the method comprising:

[0006] Use a binocular depth camera to collect multi-frame depth map data and monocular RGB image data of the arm. During the acquisition process, set a background plate to simplify the shooting environment, reduce the interference of complex backgrounds, and ensure that the main structure of the arm can be accurately captured. Gaussian filtering is performed on the acquired depth map to reduce noise, and morphological operations are used to remove background noise and retain the main outline of the arm.

[0007] During the depth map acquisition process, the tones of different depth maps may differ greatly due to changes in lighting, unstable camera parameters, or hardware limitations. In order to reduce the impact of these differences on subsequent processing, the following steps are used to standardize the tones: Histogram equalization: Apply histogram equalization to each depth map to enhance the overall contrast and details, thereby reducing the brightness differences between different depth maps. Gamma Correction: Use gamma correction to adjust each image to ensure visual consistency across frames and reduce tonal differences caused by different exposures. Normalization: Normalize the pixel values ​​of all depth maps to the same range (e.g. [0, 1]) so that the subsequent image segmentation and point cloud conversion processes are more unified. Tone adjustment based on reference frame: Select a relatively ideal depth map as the reference frame, perform tone matching on other depth maps, and make them consistent with the reference frame by adjusting the average brightness and contrast to ensure that the tones between different frames are consistent, thereby improving the stability of the modeling results.

[0008] Through depth value analysis, the image is automatically segmented according to the depth characteristics of the background plate and the arm area. By setting the depth threshold to distinguish the foreground from the background, the arm area is effectively extracted to reduce the workload of subsequent calculations. For possible interference after segmentation, connected component analysis is used to retain the largest arm area that meets the preset shape characteristics.

[0009] The segmented depth map is converted into 3D point cloud data. In this process, the depth information of each pixel is converted into coordinate values ​​in 3D space based on the camera's intrinsic parameters (such as focal length, principal point coordinates) and extrinsic parameters (such as camera position and posture). The details are as follows: Use of intrinsic parameters: The camera intrinsic parameters are used to convert the 2D image coordinates (u, v) together with the depth z into 3D point coordinates (x, y, z) in the camera coordinate system. The intrinsic parameters include the focal length (fx, fy) and the principal point coordinates (cx, cy), which are used to calculate the position of each pixel in 3D space. Use of external parameters: Camera external parameters define the position and posture of the camera in the world coordinate system. Through the rotation matrix and translation vector, the point cloud can be transformed from the camera coordinate system to the global coordinate system. When modeling depth maps from multiple perspectives, external parameters are used to calibrate point cloud data from different perspectives so that they can be fused in a unified global coordinate system. The use of camera external parameters is crucial to ensure the spatial accuracy of the point cloud and helps eliminate model errors caused by changes in camera position.

[0010] The point cloud data collected from different perspectives are fused, discrete noise points are removed, and statistical filtering is used to improve the point cloud quality. After the coordinate system is converted based on internal and external parameters, all point clouds are in the same global coordinate system and can be accurately spliced. The fused point cloud data is saved as a 3D point cloud file (such as PLY format) for subsequent 3D modeling and visualization.

[0011] Furthermore, a user interaction interface is developed based on PyQt, through which users can control operations such as depth map acquisition, preprocessing, and 3D modeling. The interactive interface also provides a visualization function, through which users can view the generated 3D model, adjust parameters, and optimize the modeling effect in real time.

[0012] The second aspect of the present invention discloses a three-dimensional modeling system of an arm using a binocular depth camera. The system includes a data acquisition module, a data preprocessing module, a point cloud generation and modeling module, and a user interaction module.

[0013] The data module uses a binocular depth camera to collect depth map data of the arm. The camera setting can automatically adjust the shooting angle to ensure that the depth map collected from multiple perspectives can cover different parts of the arm. The module can also be operated through a user interface, allowing the user to manually or automatically control the shooting process as needed.

[0014] The data preprocessing module performs noise processing, background removal, and arm area extraction on the collected depth map. Through the application of Gaussian filtering, morphological operations, and depth thresholds, the arm contour extracted during the preprocessing process is ensured to be as complete and clear as possible. In particular, through histogram equalization, gamma correction, pixel value normalization, and reference frame-based color adjustment, the color tones between depth map frames are ensured to be consistent, reducing the impact of differences on modeling.

[0015] The point cloud generation and modeling module converts the pre-processed depth map into 3D point cloud data and fuses point clouds from different perspectives. The coordinates of each pixel are accurately converted using the camera's internal and external parameters to ensure the spatial accuracy of the point cloud model. The fused point cloud data is filtered to remove noise points and finally saved in a standard 3D file format for subsequent analysis and processing.

[0016] The user interaction module is implemented based on the PyQt framework and provides a friendly graphical interface. Users can set parameters, view the point cloud generation process in real time, and operate the generated 3D model on the interface. The interface also integrates the model visualization function, and the model can be refined and adjusted through interactive tools. Beneficial effects:

[0017] The present invention realizes automatic three-dimensional modeling of the arm by using depth map data, which significantly improves the modeling efficiency, reduces manual participation, and reduces the possibility of manual errors. Compared with the traditional manual modeling method, this method greatly shortens the modeling time and reduces the reliance on manual skills through the use of depth cameras and algorithm optimization. In addition, by using a background plate, the system can more effectively reduce the interference of complex environments on the modeling process, thereby improving the accuracy of the model. By introducing standardized processing of tonal differences, the consistency between different depth map frames is further guaranteed, thereby improving the final quality of the model. By introducing external parameters, the precise fusion of point cloud data from different perspectives is achieved, ensuring the consistency and accuracy of the global model. By using monocular RGB images, the three-dimensional model is dyed, further reflecting the color state of the arm skin. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings: Figure 1 A schematic flow chart of a method for three-dimensional modeling of an arm photographed by a binocular depth camera disclosed in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the structure of a system for three-dimensional modeling of an arm using a binocular depth camera, as disclosed in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0020] Embodiment 1

[0021] The embodiment of the present invention discloses a method for three-dimensional modeling of an arm using a binocular depth camera. Figure 1 , Figure 1 A flowchart of a method for three-dimensional modeling of an arm using a binocular depth camera is disclosed in an embodiment of the present invention. The method comprises the following steps:

[0022] Step 1: Depth map data collection The depth camera is used to collect multi-frame depth map data of the arm area. The resolution of the depth map is 640×480 pixels and is stored in uint16 format. The collected depth map data is transmitted to the data processing module for subsequent depth map preprocessing and 3D modeling.

[0023] Step 2: Depth map preprocessing The collected depth map data is processed for noise. First, each depth map is smoothed using Gaussian filtering to reduce the interference of noise on the image. Then, the background plate area and the arm area are segmented using the average depth and standard deviation of the left and right areas. Specifically, the image is divided into the left area and the right area, and the average depth value and standard deviation of the two areas are calculated respectively. According to the depth difference and standard deviation of the left and right areas, the background plate area and the arm area are judged, and a mask for segmentation is generated.

[0024] Step 3: Tone Difference Processing When there are tonal differences in the continuously acquired depth images, histogram equalization is used to balance the tones. This can reduce the illumination differences introduced by changes in the acquisition environment. Furthermore, based on the regional color balancing technology, the depth values ​​between the left and right regions are adjusted to make the depth of the arm region as consistent as possible in each frame.

[0025] Step 4: Generate 3D point cloud The preprocessed depth map data is traversed pixel by pixel, and the depth value of each pixel is converted to the (x, y, z) coordinates in three-dimensional space. The conversion process is completed based on the camera's intrinsic parameters (including focal length and principal point position). The specific calculation formula is as follows: x=(u-cx)*z / fx y=(v-cy)*z / fy z=D / 1000 Among them, fx and fy are the focal lengths expressed in pixels, cx and cy are the pixel coordinates of the center of the optical axis, and D is the depth value in millimeters.

[0026] Step 5: Introduce camera external parameters for coordinate system transformation In order to align the point cloud data of multiple frames of depth maps, the camera external parameters are used to transform the coordinate system of each point cloud. The external parameters include the rotation matrix R and the translation vector T. The conversion formula is as follows: P_global=R*P_camera+T Through coordinate transformation, the point cloud of each frame can be transformed from the camera coordinate system to the global coordinate system, thereby realizing data fusion between different frames and obtaining a complete arm model.

[0027] Step 6: Point cloud denoising and optimization Radius filtering is used to remove isolated noise points, that is, to delete points that are too far away from their neighbors. Statistical filtering is further used to remove outliers based on nearest neighbor statistics, thus obtaining a smooth 3D arm model.

[0028] Step 7: 3D model display Use a 3D display module (such as Open3D) to visualize the generated arm model so that it can be rotated and scaled manually. During the display process, users can also dye the model. When using the dyeing function, the monocular RGB image generated synchronously when the binocular depth camera is used to calibrate with the point cloud coordinates, retain the color information corresponding to the point cloud used to generate the initial 3D model, realize model dyeing, and display the real arm skin color status. Furthermore, for areas of interest to the user, such as marking damaged areas or important measurement points, the user can manually select points in the point cloud of the area and enter text information in the UI interface to achieve labeling of the area.

[0029] Embodiment 2

[0030] The second aspect of the present invention discloses a system for three-dimensional modeling of an arm using a binocular depth camera. Figure 2 , Figure 2 The present invention discloses a system for three-dimensional modeling of an arm using a binocular depth camera. The system includes a data acquisition module, a data preprocessing module, a point cloud generation and modeling module, and a user interaction module.

[0031] The data acquisition module includes a binocular depth camera and its connection module. It is used to collect multi-frame depth map data and monocular RGB map data of the arm area. The resolution of the depth map is 640×480 pixels, and the resolution of the monocular RGB map is also 640×480 pixels. The data collected by the binocular depth camera is transmitted to the data preprocessing module.

[0032] Data preprocessing module The data preprocessing module is used to process the collected depth map data, including but not limited to Gaussian filtering, background plate segmentation and tone alignment processing. The details are as follows: Gaussian filtering is used to remove random noise in the depth map. Background plate segmentation is used to extract the arm region to ensure that only the valid part of the arm is considered in subsequent modeling. Hue alignment is used to unify the tones of multiple frames of images to avoid color deviation caused by differences in lighting conditions.

[0033] The point cloud generation and modeling module is used to generate a 3D point cloud model of the arm based on the preprocessed depth map data. When generating the point cloud, the camera's intrinsic parameters are used to convert the depth information into 3D space coordinates. Furthermore, combined with the camera's extrinsic parameters, the point clouds of all frames are aligned with global coordinates. After generating the point cloud, the point cloud generation and modeling module also performs denoising on the point cloud to remove isolated points and outliers to make the model smoother.

[0034] The user interaction module is used to visualize and interactively operate the 3D arm model generated by the modeling unit. The user interaction module supports a variety of interactive operations, such as rotation, scaling, translation, etc. Users can observe the overall shape of the arm through the interactive interface, or zoom in to view the details of the arm. At the same time, users can also perform further operations on the model through the interface, such as calling up monocular RGB image data for coloring, manually selecting point clouds and entering text information for annotation, etc.

[0035] It should be noted that the specific implementation process of the second embodiment is similar to that of the first embodiment and will not be repeated in the second embodiment.

[0036] Finally, it should be noted that the method and system for three-dimensional modeling of an arm using a binocular depth camera disclosed in the embodiment of the present invention only discloses a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for three-dimensional modeling of an arm using a binocular depth camera, characterized in that: The method comprises: Step 1, collecting multi-frame depth map data of the arm area, the depth map is collected by a binocular depth camera, the resolution is 640×480 pixels, and it is stored in uint16 format; the collected depth map data is transmitted to the data preprocessing module; Step 2: preprocess the depth map data, including: Gaussian filtering: Use Gaussian filtering to smooth each depth map to reduce noise; Region segmentation: Based on the depth information, the background plate and the arm area are distinguished and a segmentation mask is generated; Tone difference processing: Reduce the tone difference between depth map frames through histogram equalization, gamma correction, pixel value normalization and tone adjustment based on reference frames; Step 3: Generate 3D point cloud data based on the preprocessed depth map, specifically: The two-dimensional image coordinates (u, v) and the depth value z are converted into three-dimensional coordinates (x, y, z) in the camera coordinate system using the camera intrinsic parameters, wherein the intrinsic parameters include the focal length fx, fy and the principal point coordinates cx, cy; Introduce camera extrinsics, use rotation matrix R and translation vector T to transform point cloud from camera coordinate system to global coordinate system, so as to realize point cloud fusion under multiple perspectives; Step 4: De-noising the fused point cloud data, including: Radius filtering method removes isolated points; Statistical filtering removes outliers; Step 5: Generate a complete 3D model of the arm and visualize it through a 3D display module. Users can rotate, scale, view, color, and annotate the model through the interactive interface.

2. The arm three-dimensional modeling method according to claim 1, characterized in that: The tone difference processing also includes tone adjustment based on a reference frame, wherein the reference frame is selected as an ideal depth map, and other depth maps are kept consistent with the reference frame by adjusting average brightness and contrast.

3. The arm three-dimensional modeling method according to claim 1, characterized in that: The denoising process includes radius filtering and statistical filtering to remove isolated points and outliers and improve the smoothness and quality of the three-dimensional arm model.

4. The arm three-dimensional modeling method according to claim 1, characterized in that: The display process of the three-dimensional model uses a three-dimensional display module and allows the user to color and mark various parts of the model.

5. A three-dimensional modeling system of an arm using a binocular depth camera, characterized in that: The system includes the following modules: Data acquisition module: used to collect multi-frame depth map data of the arm area through a binocular depth camera, with a resolution of 640×480 pixels, and the depth map data is stored in uint16 format; Data preprocessing module: used to process the collected depth map data, including Gaussian filtering, background plate segmentation and tone alignment processing; Point cloud generation and modeling module: used to convert the pre-processed depth map data into a 3D point cloud, and use the camera's intrinsic and extrinsic parameters to globally align the point clouds of multiple perspectives, generate a complete 3D arm model, and perform denoising. User interaction module: used to display and interact with the arm 3D model through a graphical interface, including rotation, scaling, translation, coloring and annotation of the model.

6. The system according to claim 5, characterized in that The data preprocessing module includes a tone adjustment unit, which is used to reduce the tone difference between depth map frames by means of histogram equalization, gamma correction, pixel normalization and reference frame tone matching.

7. The system according to claim 5, characterized in that The point cloud generation and modeling module further uses external parameters to transform the point cloud data from the camera coordinate system to the global coordinate system to ensure that point clouds from different perspectives can be correctly fused.

8. The system according to claim 5, characterized in that The user interaction module is implemented based on the PyQt framework. Users can set parameters through the interface, view the point cloud generation process in real time, and operate the generated three-dimensional model.

9. The system according to claim 5, characterized in that The user interaction module also allows the user to color the model and mark specific areas, such as damaged parts of the arm or other parts of interest, by means of sliders or buttons.

Images