Image data fusion and coloring enhancement processing method

Abstract

The present invention discloses an image data fusion and coloring enhancement processing method. The present invention scans through CT mode to obtain detailed anatomical data of the face, and uses a high-definition camera to obtain RGBD image data; reconstructs the CT scan data to obtain detailed tomographic images, and uses the RGBD data collected by the high-definition camera to perform surface reconstruction, and finally generates three-dimensional point cloud data of the face; the reconstructed CT data and facial point cloud data are subjected to surface coloring processing to generate the final three-dimensional image. The initial parameters of the TSDF algorithm are marked by the CT reconstruction results, and combined with CT and face scans, more abundant patient diagnostic information can be obtained. CT and face scans are mixed and shot at the same time, and the face scan results are used as a coloring supplement for CT data, which can further improve the efficiency of doctor-patient communication. CT and face scans are mixed and shot at the same time, and the initial parameters of the TSDF algorithm are marked by the CT reconstruction results, and the results are more accurate.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to an image data fusion and coloring enhancement processing method. Background Art

[0002] Currently, many oral CBCT (Cone Beam Computed Tomography) devices are equipped with cameras for auxiliary positioning, the most typical application example is Planmeca ProFace photos. The cameras in these devices are mainly used to capture the frontal and lateral facial images of patients to help doctors make preliminary positioning and diagnosis. Although this method provides certain auxiliary functions, its actual application potential has not yet been fully developed and utilized.

[0003] The specific problems and shortcomings are as follows:

[0004] 1. Functional limitations

[0005] At present, the camera in the oral CBCT device is mainly used to assist positioning. Although this function helps to make a preliminary match between facial images and CT data, its application scope is relatively narrow. The main function of the camera is to provide a two-dimensional facial image, which is usually used to help doctors determine the specific position and angle of the patient before the CBCT scan. Although this auxiliary function can improve the accuracy of positioning in some cases, it does not bring out the full potential of the combination of camera and CT technology.

[0006] 2. Not fully utilizing the combined advantages of camera and CT technology

[0007] In current applications, cameras and CT imaging technologies are considered two independent systems, and their combination has not been explored in depth. Facial images acquired by cameras and bone data acquired by CT imaging are usually processed separately, lacking effective fusion and comprehensive utilization. This means that although cameras can provide detailed information about the face, their integration and application with CT images are still limited, and comprehensive analysis and use of the data cannot be achieved.

[0008] 3. The potential of equipment functions has not been tapped

[0009] Existing configurations mainly utilize the auxiliary positioning function of the camera, while ignoring the potential application advantages of combining the camera with CT technology. The deep integration of camera and CT technology can achieve more complex data processing and analysis, such as simultaneous facial scanning and CT imaging, and accurate pairing and fusion of the two data. This can not only improve the matching accuracy of facial and bone data, but also obtain more dimensional information on the same device, thereby improving the comprehensive effect of clinical diagnosis and treatment.

[0010] 4. Limitations of practical applications

[0011] Due to the relatively limited functions and applications of existing equipment, many potential clinical application scenarios have not been fully developed. For example, facial scan data can be combined with CT images to achieve more accurate facial surface reconstruction, preoperative planning, and postoperative evaluation. These advanced applications require that the equipment can not only perform CT imaging and facial scanning independently, but also be able to efficiently integrate and process these two types of data. Existing camera configurations fail to provide this comprehensive capability, limiting the application value and clinical effectiveness of the equipment. Summary of the invention

[0012] The purpose of the present invention is to provide an image data fusion and coloring enhancement processing method to solve the above problems.

[0013] To achieve the above object, the present invention provides the following technical solutions:

[0014] Image data fusion and coloring enhancement processing method, to obtain facial data, scan through CT mode to obtain detailed anatomical data of the face, and use a high-definition camera to obtain RGBD image data; data reconstruction, reconstruct the CT scan data to obtain detailed tomographic images, and use the RGBD data collected by the high-definition camera to perform surface reconstruction, and finally generate three-dimensional point cloud data of the face; surface coloring processing, perform surface coloring processing on the reconstructed CT data and facial point cloud data to generate the final three-dimensional image, wherein the specific method steps of the surface coloring processing are: three-dimensional cube encirclement, use the TSDF algorithm to encircle the target surface with a three-dimensional cube, and divide the cube into several volume pixel arrays according to the required reconstruction accuracy, and the voxels are used for further surface construction and processing; depth map scoring, score each voxel in each depth map, determine the most accurate surface position by integrating depth data from different perspectives, and finally construct the fine surface structure of the face; surface coloring, color the final voxel array based on the reconstructed CT data.

[0015] As a further solution of the present invention: acquiring facial data includes the following steps: acquiring facial data scanned in CT mode, wherein the step includes scanning the patient's face through a CT scanning module of an oral CBCT device to acquire facial CT data with a three-dimensional structure, and the facial CT data includes detailed information of bone structure and soft tissue; acquiring RGBD image data taken by a high-definition camera, wherein the step includes photographing the patient's face through a high-definition camera integrated in the oral CBCT device to acquire RGBD image data with depth information, and the RGBD image data includes texture, color and depth information of the facial surface.

[0016] As a further solution of the present invention: the acquisition of facial data scanned in CT mode includes the following steps: patient preparation and positioning, ensuring that the patient is accurately positioned on the scanning bed, scanning parameter selection: selecting appropriate scanning parameters according to the patient's specific conditions, such as facial features, medical history and scanning requirements; scanning data acquisition, before starting the scan, ensuring that all settings are correctly configured. During the scanning process, the X-ray tube and detector rotate around the patient to collect facial tomographic data at multiple angles, signal acquisition and conversion: the detector receives the X-ray signal after penetrating the human body, and converts the signal into an electrical signal and transmits it to the computer to generate preliminary projection data; data storage, all collected raw data will be securely stored in the system for subsequent image reconstruction and analysis.

[0017] As a further solution of the present invention: the acquisition of RGBD image data taken by a high-definition camera includes the following steps: motion trajectory setting, planning the motion trajectory of the high-definition camera according to the specific application scenario or task requirements, the trajectory includes a straight line, a curve or a complex path, and is accurately set through a controller or dedicated software to ensure the coverage and consistency of data acquisition; high-definition camera calibration, before starting shooting, the high-definition camera is fully calibrated, including adjusting the camera's focal length, exposure time and depth sensor sensitivity parameters to obtain high-quality image data; data acquisition, as the high-definition camera moves along a preset trajectory, the RGBD image is captured in real time, the image includes not only traditional RGB color information, but also a depth channel, thereby describing the distance from each object in the scene to the camera.

[0018] As a further solution of the present invention: data reconstruction includes the following steps: reconstructing CT data, wherein the step includes processing the acquired facial CT scan data using a three-dimensional reconstruction algorithm to generate a three-dimensional stereo model, which describes in detail the internal structure of the face, including the distribution of bones and soft tissues; surface reconstructing RGBD image data, wherein the step includes processing the acquired RGBD image data by using a deep image reconstruction technology to generate a three-dimensional model of the external surface of the face, which includes the texture of the facial skin, color information and corresponding depth data.

[0019] As a further solution of the present invention: reconstructing CT data includes the following steps:

[0020] CT data preprocessing: Perform necessary corrections on the collected raw CT data, such as radiation hardening correction, detector response correction, etc., to eliminate possible artifacts and distortions, ensure data accuracy, and convert projection data in the time domain into image data in the space domain.

[0021] Image reconstruction uses advanced reconstruction algorithms to reconstruct the corrected projection data into high-resolution tomographic images, and further processes the reconstructed images, such as noise removal, contrast enhancement, and edge sharpening, to improve image quality and visualization effects.

[0022] As a further solution of the present invention: surface reconstruction of RGBD image data comprises the following steps:

[0023] Coordinate system consistency reconstruction, when reconstructing the surface data of the HD camera, the same coordinate system as the CT data is used;

[0024] Application of TSDF algorithm,Use TSDF algorithm to reconstruct the surface of RGBD data, and generate three-dimensional point cloud data of the face by calculating the implicit surface in three-dimensional space to describe the external morphology of the face.

[0025] Data integration display integrates the reconstructed surface scan data and CT data in the same software for display, and adjusts the transparency of the surface scan data.

[0026] As a further solution of the present invention: the depth map scoring comprises the following steps:

[0027] Initialize all the marked voxels. The initial marking value of all voxels is 1, which means that all voxels are to be processed in the initial state. The marking voxel value of 1 means that the voxel will participate in the subsequent depth map data processing and scoring.

[0028] Initialize weight voxels, Initialize weight voxels ;The weight voxel W is used to calculate the weighted average of each voxel to fuse data from different perspectives;

[0029] Initialize voxel color, initialize voxel color RGB to all 0 (black); prepare the initial state for voxel coloring;

[0030] Process each depth image;

[0031] Determine the best surface processing result. The final voxel data will present a distribution similar to the target surface. The area where the TSDF value is close to 0 represents the position of the surface. Determine the best surface position, that is, the position where the TSDF value is 0.

[0032] Surface extraction, uses extraction algorithms to extract surface meshes from voxel meshes, generates visual 3D surface models from voxel meshes, and displays the final reconstruction results.

[0033] As a further solution of the present invention: the steps of processing each depth image are as follows:

[0034] Traverse each voxel and perform the following calculations and updates: Calculate the projection position of the voxel:

[0035] Calculate the projection position of voxel P on the depth image; obtain the position of the corresponding point of the voxel in the depth image;

[0036] Get the color of the projection position: Get the color rgb (P) of the projection position from the depth image. If the voxel has no corresponding point in the depth image, skip the voxel and extract the color information of the voxel in the depth image.

[0037] Get the depth value: Get the depth value ds of the projection position of point P in the depth image; extract the depth information in the depth map;

[0038] Calculate the depth from voxel to camera: Calculate the actual depth value dv from voxel P to camera; obtain the depth information of voxel in three-dimensional space;

[0039] Calculate the SDF value from voxel P to the target surface, which is used to measure the distance from the voxel to the actual surface;

[0040]

[0041] Calculate the truncated signed distance function TSDF and truncate the SDF value to the range [-1, 1] to meet the needs of TSDF. t controls the range of truncation:

[0042]

[0043] Among them, the parameters: t is the preset threshold used to truncate the SDF value;

[0044] Calculate weights: Calculate the weights of voxels, usually taking into account perspective and depth, with closer voxels and more frontal perspectives having higher weights;

[0045]

[0046] Among them, the parameters: θ is the angle between the voxel view and the camera, is the actual depth value of the voxel to the camera;

[0047] Fusion of TSDF and color data: Use the weighted average method to fuse the TSDF and color data of the current voxel, and update the weight value of the voxel at the same time; the fusion formula will take into account historical data and new data to improve the accuracy of the final reconstruction:

[0048]

[0049]

[0050] .

[0051] As a further solution of the present invention, coloring the final voxel array based on the reconstructed CT data includes the following steps:

[0052] Threshold segmentation is performed on CT reconstruction voxels. Threshold segmentation: voxels are divided into air parts and non-air parts according to the voxel value. The voxels in the air part are marked as 1, and the voxels in the non-air part are marked as -1;

[0053] Three-dimensional Gaussian filtering and voxel labeling: three-dimensional Gaussian filtering is performed on the labeled CT voxel data to smooth the voxel data and remove noise; the processed voxel data is used as the initial TSDF label;

[0054] Initialize weight voxels, initialize weight voxels W: define the weight voxel set ,in is an adjustable weight parameter;

[0055] Initialize voxel color, initialize color RGB: the initial color of all voxels is set to all 0 (black);

[0056] Process the depth image and traverse each depth image;

[0057] Determine the best surface position and color, determine the best surface position: after processing, the center of the transition zone from 1 to -1 (0 separation zone) is the best surface position. The color of this position is determined by the corresponding RGB voxel value.

[0058] As a further solution of the present invention: processing the depth image includes the following steps:

[0059] Calculate the projection position: Calculate the projection position of the voxel P on the depth image, obtain the color rgb(P) of the projection position, and skip the voxel if there is no color information at the position;

[0060] Get depth value: Get the depth value of the projection position of point P on the depth image ;

[0061] Calculate voxel depth: calculate the depth dv from voxel P to the camera;

[0062] Calculate the SDF value: Calculate the SDF value corresponding to voxel P: ;

[0063] Calculate the TSDF value: Calculate the TSDF value of voxel P:

[0064]

[0065] Among them, t is the preset threshold;

[0066] Calculate the weight of the current voxel:

[0067]

[0068] in, is the viewing angle of the voxel in the depth image;

[0069] Fusion of TSDF and color: Fusion of the current TSDF value and color with the historical TSDF value. The fusion formula is as follows:

[0070]

[0071]

[0072] .

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] Accurate marking of the initial parameters of the TSDF algorithm: Marking the initial parameters of the TSDF algorithm through CT reconstruction results effectively improves the accuracy of the algorithm calculation, making the generated 3D model more accurate and better reflecting the patient's actual anatomical structure.

[0075] Rich patient diagnostic information: This invention combines CT imaging and facial scanning technology to provide more comprehensive diagnostic data by simultaneously acquiring the patient's internal structure and external morphology. This multi-dimensional information fusion can help doctors better understand the patient's condition during diagnosis and treatment planning, and improve the accuracy of diagnosis.

[0076] Efficient doctor-patient communication: During the mixed shooting of CT and facial scan, the facial scan results can be used as a coloring supplement for CT data. This intuitive 3D image can not only show the patient's lesion more clearly, but also greatly improve the efficiency of doctor-patient communication, helping patients better understand their condition and treatment plan.

[0077] Enhanced accuracy and fusion effect: The initial parameters of the TSDF algorithm are marked by the CT reconstruction results. When CT and face scans are taken simultaneously, it not only ensures the high accuracy of the algorithm, but also optimizes the fusion effect of face scan data and CT data, making the final image data more detailed and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a flowchart of the image data fusion and coloring enhancement processing method.

[0079] Figure 2 A flowchart of a method for obtaining facial data.

[0080] Figure 3A flowchart of a method for obtaining RGBD image data taken by a high-definition camera.

[0081] Figure 4 Schematic diagram of the process of reconstructing CT data.

[0082] Figure 5 A flowchart of a method for surface reconstruction of RGBD image data.

[0083] Figure 6 Schematic diagram of voxel grid.

[0084] Figure 7 Flowchart of the steps of the depth map scoring algorithm.

[0085] Figure 8 Schematic diagram of the process of coloring the final voxel array based on the reconstructed CT data. DETAILED DESCRIPTION

[0086] The present invention will be further described in detail below in conjunction with the accompanying drawings and implementations. It is to be understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present invention. It is also necessary to explain that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings.

[0087] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0088] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as exemplary features that provide various details of some ways in which the technical concept of the present invention can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present invention.

[0089] The terms used herein are for the purpose of describing specific embodiments, rather than being restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or provided values ​​that will be recognized by those of ordinary skill in the art.

[0090] The following takes the application scenario of CBCT (Cone Beam CT) as an example and describes the image data fusion and coloring enhancement processing method of the present invention with reference to the accompanying drawings.

[0091] Figure 1 FIG. 1 is a flow chart of an image data fusion and color enhancement processing method according to an embodiment of the present invention. Figure 1 The present invention provides an image data fusion and coloring enhancement processing method M10. The image data fusion and coloring enhancement processing method M10 of this embodiment may include the following steps S100, S200 and S300.

[0092] S100: Acquiring facial data

[0093] In this step, a CT scan will be performed to obtain detailed anatomical data of the face, while a high-definition camera will be used to obtain RGBD image data. These data will be used for subsequent reconstruction and analysis to ensure accurate representation of the facial structure.

[0094] S110: Acquiring facial data of CT scan

[0095] Figure 2 FIG. 1 is a flow chart of a method for obtaining facial data in an image data fusion and color enhancement processing method according to an embodiment of the present invention. Figure 2 The method S110 for acquiring facial data of this embodiment may include the following steps S1101, S1102 and S1103.

[0096] S1101 Patient Preparation and Positioning:

[0097] Patient positioning: Ensuring that the patient is accurately positioned on the scanning table is critical to reducing motion artifacts. It is often necessary for the patient to remain absolutely still to avoid image blurring due to motion.

[0098] Scan parameter selection: Select appropriate scan parameters based on the patient's specific conditions, such as facial features, medical history, and scanning requirements. These parameters may include the energy level, layer thickness, and rotation speed of the scan.

[0099] Patient Positioning:

[0100] Operation steps during positioning:

[0101] Head fixation: Use specialized head fixation devices, such as head support pads or restraints, to ensure that the patient's head remains stationary during the scan. These devices can effectively prevent the patient from involuntary movement during the scan.

[0102] Posture adjustment: According to the scanning area (such as a specific part of the face), adjust the patient's posture to obtain the best scanning angle. For example, ask the patient to tilt his head slightly back or forward to obtain a clearer image at certain specific angles.

[0103] Breathing control: In some cases, the patient may be asked to hold their breath for a few seconds to avoid subtle movements of the chest and face caused by breathing. This is especially important when the scan involves the face and upper neck area.

[0104] Communication and confirmation:

[0105] Patient education: Before the scan begins, have a thorough discussion with the patient, explaining the entire scanning process and why they must remain absolutely still. A clear understanding of the process can reduce the patient's tension and reduce unnecessary movement.

[0106] Position confirmation: Before scanning, the technician should reconfirm that the patient is positioned correctly and ensure that the fixation device is secure to avoid accidental movement during scanning.

[0107] Scan parameter selection:

[0108] Customized parameter settings based on patient conditions:

[0109] Facial features: The facial anatomy of different patients varies, such as facial bone density, soft tissue distribution, etc. Based on these differences, the scanning parameters should be adjusted accordingly. For example, for patients with thinner faces, the energy level of the scan may need to be reduced to avoid overexposure of bones and soft tissues.

[0110] Specific settings of the main scanning parameters:

[0111] Energy Level: Adjusts the voltage level of the X-ray tube to control the penetration ability of the X-rays. For thin facial tissues, a lower kVp may be sufficient; for thicker or bone-dense areas, a higher kVp may be needed to ensure adequate penetration and imaging.

[0112] Slice thickness: Thinner slice thicknesses provide higher resolution images, but may increase scanning time. Thicker slice thicknesses are suitable for general diagnosis and can cover large areas more quickly.

[0113] Rotation speed: Adjust the rotation speed of the scanner to suit different imaging needs. A faster rotation speed can shorten the scanning time and reduce the discomfort caused by patients staying still for a long time; but for facial scans that require high precision, a slower rotation speed is required to obtain a more detailed image.

[0114] Optimization and balance of scanning parameters:

[0115] Image quality and radiation dose: While optimizing image quality, it is also necessary to consider the control of radiation dose. Higher scanning energy and smaller layer thickness will increase the radiation dose, but can provide clearer images. Therefore, it is very important to reduce the patient's radiation exposure as much as possible while ensuring image quality.

[0116] S1102 Scanning Data Collection:

[0117] Start the CT scanner: Before starting the scan, make sure all settings are configured correctly. During the scan, the X-ray tube and detector will rotate around the patient to collect tomographic data of the face from multiple angles.

[0118] Signal acquisition and conversion: The detector receives the X-ray signals after penetrating the human body, converts these signals into electrical signals and transmits them to the computer to generate preliminary projection data.

[0119] Start scanning: After confirming that all preparations are correct, the operator can start the CT scanner. During the startup process, the X-ray tube begins to emit X-rays and the detector rotates around the patient to scan. The rotation speed and trajectory should be consistent with the previously set parameters to ensure multi-angle tomographic data acquisition.

[0120] Dynamic Adjustment: If any abnormality is detected during monitoring, such as slight patient movement or poor image quality, the scan can be paused immediately, patient positioning or device parameters can be adjusted, and then the scan can be continued.

[0121] X-ray signal acquisition:

[0122] Multi-angle data collection: While the X-ray tube is emitting rays, the detector rotates around the patient and receives X-ray signals after penetrating the human body at different angles. The penetration depth and attenuation of the rays at each angle are different, and the detector records this information as raw data. These multi-angle projection data are the basis for subsequent image reconstruction.

[0123] Generate preliminary projection data:

[0124] Projection data formation: As the detector rotates around the patient, the computer continuously receives and processes X-ray signals from different angles, gradually generating a series of projection data. These projection data are two-dimensional image sets formed after multiple sampling and processing, and each image represents the attenuation information of X-rays after penetrating the face from a specific angle.

[0125] Data integrity check: After data acquisition is completed, the technician should check the generated projection data to ensure data integrity and consistency. The check includes whether the data covers the entire scan area and whether there are obvious artifacts or defects. If problems are found, it may be necessary to readjust the parameters and scan again.

[0126] S1103 Data Storage:

[0127] System storage: All collected raw data will be securely stored in the system for subsequent image reconstruction and analysis. The data should be stored in a complete and correct manner to avoid data loss or damage.

[0128] S120: Get RGBD image data captured by a high-definition camera

[0129] Figure 3 1 is a flow chart of a method for obtaining RGBD image data captured by a high-definition camera in an image data fusion and color enhancement processing method according to an embodiment of the present invention. Figure 3 The present invention provides an image data fusion and coloring enhancement processing method M10. The method S120 for obtaining RGBD image data taken by a high-definition camera in this embodiment may include the following steps S1201, S1202 and S1203.

[0130] S1201 motion trajectory setting:

[0131] Trajectory planning: Plan the motion trajectory of the HD camera according to the specific application scenario or task requirements. The trajectory can be a straight line, curve or complex path, and is precisely set by the controller or dedicated software to ensure the coverage and consistency of data collection.

[0132] Trajectory type and planning:

[0133] Straight track: Suitable for scenes that require uniform coverage of a specific area, such as smooth movement from top to bottom or from left to right in facial scanning. Straight track can ensure the linear consistency of data acquisition and is suitable for capturing continuous and uniform image sequences.

[0134] Curved trajectories: Curved trajectories can be used to bypass obstacles or capture nonlinear features, such as capturing facial contours from multiple angles. Curved trajectories are particularly important in some applications that require precise coverage of complex surfaces because they can better adapt to the geometry of the target object.

[0135] Complex paths: When you need to capture data in a complex 3D space, you may need to plan a complex path that includes multiple motion modes such as straight lines, curves, and even spirals. This path ensures that the camera captures all information about the target object from every possible angle, especially when performing comprehensive 3D modeling.

[0136] Data collection coverage: Trajectory planning needs to ensure that the camera's movement can fully cover the target area, avoiding any blind spots or data loss. For example, in facial scanning, the trajectory must ensure that all important features of the entire face are covered, including key parts such as the bridge of the nose, eye sockets, and chin.

[0137] S1202 HD Camera Calibration:

[0138] Calibration preparation: Before you start shooting, the HD camera must be fully calibrated to ensure the accuracy of the RGB (red, green, blue) and D (depth) sensors. This includes adjusting the camera's focal length, exposure time, and the sensitivity of the depth sensor to obtain high-quality image data.

[0139] RGB sensor calibration:

[0140] Focus adjustment: First, adjust the focus of the camera to ensure that the subject is clearly imaged in the image. Focus adjustment is usually done manually or through an autofocus mechanism, and the appropriate focus is selected based on the shooting distance and scene characteristics. Especially in facial scanning, the precise setting of the focus can capture subtle facial features such as skin texture, hair, etc.

[0141] Exposure time setting: Exposure time directly affects the brightness and detail capture of the image. During the calibration process, the exposure time needs to be adjusted according to the ambient lighting conditions and the reflective characteristics of the subject. Too long an exposure time may cause the image to be overexposed, while too short an exposure time may cause the image to be dim or lose details. The ideal exposure time setting should be able to evenly display the details in the scene in each color channel (red, green, and blue).

[0142] White balance and color correction: To ensure the color accuracy of the RGB channels, the camera also needs to perform white balance adjustment and color correction. White balance adjustment can eliminate color deviations under different light sources, making the white in the image look real and natural. Color correction is used to calibrate the response of each color channel so that the camera can accurately reproduce the true colors in the scene.

[0143] Depth sensor calibration:

[0144] Sensitivity adjustment: The sensitivity of the depth sensor determines its ability to capture distance information. During the calibration process, the sensitivity of the sensor needs to be adjusted according to the shooting distance and environmental characteristics. Too high a sensitivity may cause increased noise, while too low a sensitivity may cause inaccurate or lost depth information. Through precise sensitivity adjustment, ensure that the sensor can capture clear depth information.

[0145] Depth range calibration: Depth sensors usually have a limited working range, and their effective range needs to be calibrated before starting to shoot to ensure that all important features of the target object can be captured. This process may involve testing the sensor's ranging capabilities and adjusting its working range to match the needs of specific application scenarios.

[0146] Eliminate depth errors: During the calibration process, special calibration tools or software are also needed to eliminate possible errors in the depth sensor. These errors may be caused by hardware defects of the sensor, interference from ambient light, or other factors. Through a series of tests and adjustments, the noise and artifacts in the depth data are reduced to ensure that the captured 3D structure information is accurate and reliable.

[0147] S1203 Data Collection:

[0148] Real-time acquisition: As the high-definition camera moves along a preset trajectory, RGBD images are captured in real time. These images not only include traditional RGB color information, but also add a depth channel (D) to describe the distance of each object in the scene from the camera. This enables the resulting image to present the structure of the object in three dimensions.

[0149] Real-time capture process:

[0150] Track motion and data synchronization: When the HD camera moves along a preset track, the real-time acquisition system automatically synchronizes the camera's motion and the capture of image data. Whether the camera moves along a straight line, curve, or complex path, each frame of the image captured strictly corresponds to the camera's current position. This synchronization mechanism ensures the coherence and integrity of the image data, so that the image at each position can accurately reflect the perspective of that position.

[0151] Frame rate and data stream processing: During real-time acquisition, the camera continuously captures image data at a certain frame rate. Depending on the application requirements, the frame rate can be adjusted to adapt to different scene dynamic or static environments. At the same time, the collected data will be transmitted to the computing system through a high-speed data stream for preliminary processing and storage, ensuring that the entire acquisition process is recorded without delay.

[0152] RGB image information capture:

[0153] Color and brightness information: In each frame, the RGB channel records the color and brightness information of the objects in the scene. These data are collected by the camera's red, green, and blue sensors respectively, and can accurately reproduce the color characteristics of the scene. High-quality RGB images can not only show the surface color of the object, but also show the shape and texture of the object through brightness and shadow.

[0154] Resolution and detail capture: The RGB images acquired in real time have high resolution and can capture the subtle details in the scene. This is essential for accurately describing the appearance characteristics of objects, such as texture, reflection, and transparency. With high-resolution images, subsequent image processing and analysis can more accurately identify and reconstruct the 3D structure of objects.

[0155] Data acquisition of depth channel (D):

[0156] Distance measurement and depth information: In each frame, in addition to RGB color information, the D channel also records the distance of each object in the scene to the camera. The depth sensor obtains the distance data of the object surface by measuring the return time of light or based on other depth perception technologies. This distance information constitutes a depth map, which shows the distance relationship between each pixel in the scene and the camera.

[0157] Three-dimensional structure description: By combining RGB color information and data from the depth channel, the acquisition system can present the structure of the object in three dimensions.

[0158] S200: Data Reconstruction

[0159] In this step, the CT scan data will be reconstructed to obtain detailed tomographic images, and the RGBD data collected by the high-definition camera will be used for surface reconstruction to finally generate three-dimensional point cloud data of the face.

[0160] RGBD data acquisition and surface reconstruction, high-definition cameras and depth sensors (usually using RGBD technology) are used to capture the external features of the face. RGBD cameras can simultaneously obtain color images (RGB) and depth information (Depth), the former records the color and texture of the face, and the latter records the geometric depth of the facial surface. By fusing the depth map and the RGB image, a three-dimensional surface model of the face can be generated. This model contains rich texture information and can be used to accurately express the micro details of the facial surface.

[0161] Data fusion and 3D point cloud generation, CT reconstruction data and RGBD surface data have been obtained, and the next step is to fuse these two types of data. Since CT data mainly reflects the internal structure of the face, and RGBD data focuses on external surface features, the combination of the two can generate a complete facial 3D model.

[0162] When data is fused, the CT data and RGBD data are precisely aligned. First, the key feature points in the two sets of data are found through the feature extraction algorithm, and then these feature points are aligned using the registration algorithm (such as the ICP algorithm or deep learning registration technology) to achieve accurate data fusion. After the registration is completed, the tomographic information of the CT data can be combined with the surface information of the RGBD data to generate complete three-dimensional point cloud data.

[0163] S210: Reconstruct CT data

[0164] Figure 4 FIG. 1 is a flow chart of a method for reconstructing CT data according to an embodiment of the present invention. Figure 4 ,The CT data reconstruction method S210 of this embodiment may include the following steps S2101 and S2102.

[0165] S2101 CT data preprocessing:

[0166] Data correction: Perform necessary corrections on the collected raw CT data, such as radiation hardening correction, detector response correction, etc., to eliminate possible artifacts and distortions and ensure data accuracy.

[0167] Data conversion: converting projection data in the time domain into image data in the spatial domain. This is done by reconstructing the original data from one-dimensional projection information into a two-dimensional or three-dimensional image form through mathematical algorithms.

[0168] During the CT scanning process, the raw data obtained is usually affected by various physical factors, which may introduce artifacts and distortions. These problems will reduce the accuracy of the image and affect subsequent analysis and reconstruction. In order to ensure the accuracy and reliability of CT data, the raw data must be corrected as necessary.

[0169] Image reconstruction: By superimposing the back-projection results from multiple angles, a complete two-dimensional image can be reconstructed. Each layer of back-projection data provides information for a certain part of the final image, and the superposition of all back-projection data produces the final CT image.

[0170] Initial image estimate: First, generate an initial image estimate based on the projection data. This initial estimate can be obtained by simple back-projection.

[0171] Projection difference calculation: The projection of the initial image is then compared with the actual acquired projection data and the differences between the two are calculated. These differences reflect the deviation between the initial image and the true image.

[0172] Image update: Based on the projection difference, the initial image is updated to make it closer to the actual projection data. This process is performed iteratively, and each iteration fine-tunes the image to gradually reduce the gap with the real image.

[0173] Convergence judgment: When the difference between the updated projection and the actual projection data drops to a small enough range, or reaches a predetermined number of iterations, the algorithm stops, and the final image generated is the reconstructed image.

[0174] 3D Image Reconstruction:

[0175] When a 3D image needs to be reconstructed, a CT scan acquires a series of continuous 2D slice images. These slice images represent cross-sections of the object at different depths. By superimposing these 2D slices, a 3D image can be generated. The 3D reconstruction process usually uses convolution or interpolation algorithms to ensure the continuity and accuracy of the image in the vertical direction.

[0176] S2102 Image Reconstruction:

[0177] Application of reconstruction algorithms: The corrected projection data are reconstructed into high-resolution tomographic images using advanced reconstruction algorithms such as filtered back projection, iterative reconstruction, etc. These images provide a detailed view of the internal structures of the face.

[0178] Image post-processing: Further processing is performed on the reconstructed image, such as noise removal, contrast enhancement, and edge sharpening, to improve image quality and visualization effect.

[0179] S220: Surface reconstruction of RGBD image data

[0180] Figure 5 FIG. 1 is a flow chart of a method for surface reconstruction of RGBD image data according to an embodiment of the present invention. Figure 5, the surface reconstruction RGBD image data method S220 of this embodiment may include the following steps S2201, S2203 and S2203.

[0181] S2201 Coordinate System Consistency Reconstruction:

[0182] Reconstruction coordinate system: When reconstructing the surface data of the HD camera, the same coordinate system as the CT data is used. This consistency can make the reconstruction of the facial surface structure more accurate, and based on the reconstruction results of CT, the speed and accuracy of surface reconstruction are further optimized.

[0183] Reconstruct the consistency of the coordinate system:

[0184] Background: In the three-dimensional reconstruction of facial structures, the surface data captured by high-definition cameras and the internal structure data obtained by CT scans often come from different devices. These devices usually have their own independent coordinate systems, and directly fusing these data may lead to inconsistencies and errors in the reconstruction results. Therefore, it is particularly important to unify these data into the same coordinate system for processing.

[0185] Coordinate system transformation: First, the coordinate system of the HD camera needs to be calibrated and converted to the CT coordinate system. After the coordinate transformation, it is necessary to ensure that the HD camera and CT data are reconstructed under the same spatial reference. Any slight error will affect the final reconstruction accuracy. Therefore, a high-precision registration algorithm is required, such as feature point-based registration or iterative closest point (ICP) algorithm to ensure strict alignment of different data sources.

[0186] Capture and reconstruction of surface data: HD cameras can capture high-resolution surface textures and details of the face, but lack internal structure information. CT scans, on the other hand, provide complete internal structure information. By integrating these two types of data in the same coordinate system, accurate reconstruction of the facial surface structure can be achieved. Based on the unified coordinate system, CT data can be used as a reference frame for surface reconstruction to help quickly locate and calibrate the surface data of the HD camera. Using CT data to guide the HD camera surface reconstruction can automatically correct possible errors during the reconstruction process and improve the accuracy of the final surface model. This method is particularly helpful in solving the errors that may occur when HD cameras capture edges and complex structures, ensuring that the details and shapes of the face are accurately reconstructed.

[0187] S2202 TSDF algorithm application:

[0188] Surface reconstruction: The TSDF algorithm (Truncated Signed Distance Function) is used to reconstruct the surface of the RGBD data. This algorithm generates 3D point cloud data of the face by calculating the implicit surface in 3D space, which accurately describes the external morphology of the face.

[0189] By accumulating depth data from different perspectives, a voxel grid is constructed, and each voxel stores the TSDF value at that location. Then, the implicit surface in the three-dimensional space is determined by zero-crossing point interpolation, that is, the surface of the target object is reconstructed.

[0190] RGBD data processing: RGBD cameras simultaneously capture color images (RGB) and depth images (D) of the scene, providing color information for each pixel and distance information to the camera. In this way, rich details of the scene can be obtained.

[0191] Data fusion: Using RGBD data from multiple perspectives, by fusing these data in a unified 3D space, the implicit surface representation is gradually accumulated and optimized. This multi-perspective fusion can eliminate the occlusion and noise problems caused by a single perspective and generate a more complete and accurate 3D point cloud.

[0192] 3D point cloud generation,After completing the construction of the TSDF voxel grid, Marching Cubes or similar algorithms are used to extract the surface from the voxel grid to generate a 3D point cloud. The point cloud data describes the external morphology of the face, including surfaces, edges, and details.

[0193] High-precision reconstruction, surface reconstruction of RGBD data through TSDF algorithm.

[0194] S2203 data integration display:

[0195] 3D data visualization: The reconstructed facial scan data and CT data are integrated in the same software for display. By adjusting the transparency of the facial scan data, the patient's facial appearance and internal anatomical structure can be viewed simultaneously to achieve a comprehensive visualization effect.

[0196] Data integration: Fusion of face scan data and CT data: To fuse face scan data and CT data in the same 3D space, the two types of data need to be aligned in the coordinate system. Face scan data (usually point cloud data generated by RGBD cameras) and CT data (voxel data) may come from different devices, so they need to be mapped to the same spatial reference frame through coordinate system transformation.

[0197] Data conversion: Use high-precision coordinate registration technology (such as feature-based registration, iterative closest point algorithm, etc.) to convert the coordinate system of the surface scan data to the coordinate system of the CT data to ensure accurate alignment of the data.

[0198] Data import: Import the integrated data into the software, and the software will automatically load and display it according to the data format and coordinate system information.

[0199] Face scan data transparency adjustment: When displaying face scan data, its transparency can be adjusted so that both the facial appearance and the internal anatomy can be viewed simultaneously. Transparency settings can usually be adjusted through the visualization tools in the software.

[0200] Effect display: By reducing the transparency of the face scan data, the internal structure of the CT data can be revealed; conversely, increasing the transparency can highlight the appearance details of the face scan data. In this way, doctors and researchers can intuitively understand the relationship between the facial surface and internal structure.

[0201] Layered display: The software usually supports multi-layered display modes, which can display details of different data layers at the same time. For example, CT data can be set as the bottom layer, and semi-transparent surface scan data can be displayed on the upper layer to form a clear multi-layer view.

[0202] Interactive Adjustment: The software allows users to dynamically adjust 3D data through interactive operations (such as rotation, scaling, cutting, etc.), observe data from different angles and sections, and help conduct detailed analysis.

[0203] S300: Surface coloring

[0204] The final step is to perform surface coloring on the reconstructed CT data and facial point cloud data to generate the final three-dimensional image.

[0205] Data Fusion: Colorization of Facial Point Cloud Data,Facial point cloud data usually contains color information (RGB).

[0206] CT data colorization: CT data are usually grayscale images. By mapping grayscale values ​​to color space or combining with facial point cloud data, the color distribution of different tissues is given.

[0207] Data mapping: The grayscale value or segmentation result of CT data needs to be mapped to the color space so that different tissues and structures appear in different colors on the 3D model.

[0208] Texture mapping: The color information of the facial point cloud data is mapped as a texture onto the 3D surface model of the CT data.

[0209] Color mapping: Map the grayscale value or segmentation information of CT data to a color table, and use colors to represent different tissue types or structures.

[0210] Hybrid rendering: To simultaneously display the facial appearance and internal structure, hybrid rendering technology can be used to synthesize the facial point cloud data and the color information of the CT data into a final 3D model. By adjusting the transparency and blending parameters, the surface and internal structure can be effectively displayed.

[0211] Generation of the final image: Model rendering, using 3D rendering software to render the colored 3D model into the final image. These software can generate high-quality image and video output, supporting observation and dynamic display from different perspectives.

[0212] Image quality optimization: Adjust rendering settings (such as lighting, shadows, reflections, etc.) to improve image quality, making the final 3D image more realistic and detailed.

[0213] S310 3D cube enclosure:

[0214] Cube segmentation: The target surface is surrounded by a 3D cube using the TSDF algorithm, and the cube is divided into several voxel (volume pixel) arrays according to the required reconstruction accuracy. These voxels will be used for further surface construction and processing.

[0215] Define the target region: Define a cubic grid in 3D space to surround the target surface or object. This cubic grid is the basis of the entire voxelization process.

[0216] Grid size and resolution: Determine the size and resolution of the cube grid. The size of the cube determines the size of the voxels, which also affects the accuracy of the final reconstructed model. The higher the resolution, the smaller the voxels, the richer the model details, but the computational and storage requirements will also increase.

[0217] Voxel grid construction:

[0218] Voxel grid initialization: The cube grid is divided into several small cube units, which are called voxels. Each voxel has a fixed volume in space and occupies a position in the voxel grid.

[0219] Voxel arrangement: Voxels are usually arranged in a regular 3D array to form a voxel grid. These voxels will be used to store the distance information (TSDF value) of the target surface at that location.

[0220] Voxel separation and accuracy

[0221] Voxelization process:

[0222] Voxel size setting: Set the voxel size according to the desired reconstruction accuracy. The smaller the voxel size, the higher the resolution of the mesh, which can capture finer surface details.

[0223] Precision and computational resources: The increase in precision will increase the number of voxels, which will increase the computational burden and storage requirements. It is necessary to balance between precision and computational resources and choose the appropriate voxel size to meet the application requirements.

[0224] Voxel data storage:

[0225] Distance value storage: Each voxel stores the signed distance from the location to the target surface, which is calculated based on the TSDF algorithm. The TSDF value represents the distance from the voxel to the nearest surface and has a sign (positive or negative) to indicate whether the voxel is inside or outside the surface.

[0226] Voxel Update: Voxel values ​​are continuously updated based on input data to reflect the latest surface information. Using data from multiple viewpoints can further improve the accuracy and completeness of the voxel grid.

[0227] Voxel processing and distance function calculation in TSDF algorithm:

[0228] Calculation process: Use a depth sensor or other data source to obtain the distance information of the target object. This distance information is used to calculate the TSDF value of each voxel, which is a truncated version of the closest distance from the voxel to the target surface.

[0229] Signed distance: TSDF values ​​have a sign, indicating whether the voxel is inside or outside the surface. This sign information is crucial for surface reconstruction because it helps distinguish between internal and external surfaces.

[0230] Voxel grid optimization:

[0231] Data fusion: If there is data from multiple perspectives, it is necessary to fuse this data into a voxel grid. By combining data from different perspectives, the accuracy of the voxel grid can be improved and noise and errors can be reduced.

[0232] Voxel Reconstruction: Updating and optimizing the voxel grid includes handling inconsistencies in the data, filling in missing data areas, and smoothing surfaces to improve model quality.

[0233] S320 Depth Map Scoring:

[0234] Surface construction: Each voxel in each depth map is scored, and the most accurate surface position is determined by combining depth data from different perspectives to ultimately construct the fine surface structure of the face.

[0235] Depth map scoring is a key step in the 3D reconstruction process, which involves scoring each voxel in each depth map to determine the most accurate surface location to construct the fine surface structure of the face.

[0236] Mapping of voxels to depth maps:

[0237] Voxel definition: The three-dimensional space is divided into several voxels, each voxel represents a small cube in the space. Each pixel in the depth map corresponds to a voxel, indicating the depth information of the voxel from the camera's perspective.

[0238] Data mapping: Map the depth information in each depth map to the voxel grid. Each voxel will update its corresponding TSDF value based on the data in the depth map.

[0239] Voxel Scoring:

[0240] Depth data fusion: For each voxel, extract the corresponding depth information from the depth maps of all views and fuse them. Use weighted averaging or other fusion methods to combine the data from multiple views to obtain the final depth estimate of the voxel.

[0241] Scoring algorithm: Each voxel is scored, usually based on the following criteria:

[0242] Depth consistency: The consistency of the depth values ​​of voxels in different depth maps. The smaller the change of the depth value of a voxel under different viewing angles, the higher its score.

[0243] Data coverage: The data coverage of voxels in different viewing angles. Voxels with good data coverage will be scored higher.

[0244] Matching degree: The matching degree between the voxel depth value and the target surface. If the depth value of the voxel is close to the target surface, the score is higher.

[0245] Accurate surface position determination:

[0246] Optimal depth selection: Determine the optimal depth of a voxel by comparing the scoring results in different depth maps. The depth with the highest score is selected as the final depth value of the voxel, thereby accurately determining the surface position.

[0247] Outlier processing: Process outliers and noise data. Usually, filtering or smoothing methods are used to remove noise and erroneous data in the depth map to improve the accuracy of scoring.

[0248] Implicit surface extraction: Use the final voxel depth value calculated by the TSDF algorithm to generate a 3D mesh model of the target surface through an implicit surface extraction method (such as the Marching Cubes algorithm).

[0249] Mesh optimization: Smooth, refine, and repair the generated 3D mesh to improve surface quality, remove noise and holes, and ensure the fineness and accuracy of the final model.

[0250] Model Fusion:

[0251] Multi-view fusion: The final 3D surface model is constructed by integrating data from different viewpoints. The fused model can more comprehensively display the surface details of the target object and solve the problems of occlusion and blind spots under a single viewpoint.

[0252] Texture mapping: Apply texture information to the generated 3D model to enhance the visual effect. Texture mapping can provide more detailed information and make the model more realistic.

[0253] Figure 7 is a flowchart of the steps of a depth map scoring algorithm according to one embodiment of the present invention. Figure 7 The depth map scoring algorithm of this embodiment includes the following steps S3201, S3202, S3203, S3204, S3205 and step S3206.

[0254] Depth map scoring algorithm steps:

[0255] In 3D reconstruction, enclosing the target surface with a 3D cube and scoring each voxel according to the depth map is a complex process; the following are the detailed algorithm steps:

[0256] S3201 Initialize all labeled voxels:

[0257] Steps: Initialize the marker value of all voxels to 1, indicating that all voxels are to be processed in the initial state;

[0258] Purpose: Marking a voxel as 1 means that the voxel will participate in the subsequent depth map data processing and scoring;

[0259] S3202 Initialize weight voxels:

[0260] Step: Initialize weight voxels ;

[0261] Purpose: The weight voxel W is used to calculate the weighted average of each voxel to fuse data from different perspectives;

[0262] S3203 Initialize voxel color:

[0263] Steps: Initialize the voxel color RGB to all 0 (black);

[0264] Purpose: Prepare the initial state for voxel coloring;

[0265] S3204 processes each depth image and performs the following steps:

[0266] Traverse each voxel and perform the following calculations and updates:

[0267] Compute the projected position of a voxel:

[0268] Steps: Calculate the projection position of voxel P on the depth image;

[0269] Objective: Get the position of the corresponding point of the voxel in the depth image;

[0270] Get the color of the projected position:

[0271] Steps: Get the color rgb (P) of the projection position from the depth image. If the voxel has no corresponding point in the depth image, skip the voxel.

[0272] Purpose: Extract the color information of voxels in depth images;

[0273] Get the depth value:

[0274] Steps: Get the depth value ds of the projection position of point P in the depth image;

[0275] Purpose: Extract depth information from the depth map;

[0276] Calculate the depth from voxel to camera: Calculate the actual depth value dv from voxel P to the camera;

[0277] Purpose: To obtain the depth information of voxels in three-dimensional space;

[0278] Calculate the Signed Distance Function (SDF):

[0279]

[0280] Purpose: Calculate the Signed Distance Function value from voxel P to the target surface, which is used to measure the distance from the voxel to the actual surface;

[0281] Calculate Truncated Signed Distance Function (TSDF):

[0282]

[0283] Parameters: t is the preset threshold used to cut off the SDF value;

[0284] Purpose: Truncate the SDF value to the range of [-1, 1] to meet the needs of TSDF. t controls the range of truncation.

[0285] Calculate weights:

[0286]

[0287] Among them, the parameters: θ is the angle between the voxel view and the camera, is the actual depth value of the voxel to the camera;

[0288] Purpose: Calculate the weight of voxels, usually taking into account perspective and depth, with closer voxels and more frontal perspectives having higher weights;

[0289] Fusion of TSDF and color data:

[0290]

[0291]

[0292]

[0293] Purpose: Use the weighted average method to fuse the TSDF and color data of the current voxel, and update the weight value of the voxel at the same time; the fusion formula will take into account historical data and new data to improve the accuracy of the final reconstruction;

[0294] S3205 Determine the best surface treatment results:

[0295] After being processed by the above algorithm, the final voxel data will present a distribution similar to the target surface; Figure 6 In the voxel grid shown, the area where the TSDF value is close to 0 represents the location of the surface, which is usually the transition zone from positive to negative TSDF values;

[0296] Purpose: To determine the best surface locations, i.e., locations where the TSDF value is 0; these locations represent the most likely surface locations, since the difference between the depth map measurements and the 3D reconstructed model is the smallest here;

[0297] S3206 Surface Extraction, extracting a surface mesh from a voxel mesh using an extraction algorithm (such as the Marching Cubes algorithm);

[0298] Purpose: To generate a visual 3D surface model from a voxel grid and display the final reconstruction results.

[0299] Through the above steps, the voxel scoring algorithm can accurately determine the optimal position of the target surface and generate a high-quality 3D model. This method has broad application prospects in applications that deal with complex surfaces and high precision requirements.

[0300] S330 Surface Coloring:

[0301] Voxel coloring: The final voxel array is colored based on the reconstructed CT data. The coloring process combines the density information in the CT data, so that the final generated 3D image not only shows the appearance of the face, but also reflects the density distribution of the internal structure, providing a more intuitive basis for diagnosis.

[0302] Voxel data preparation:

[0303] Voxel grid generation: First, a voxel grid is generated using the TSDF algorithm or other reconstruction methods. These voxel grids represent the 3D structure of the target object (such as a face).

[0304] CT data alignment: Align the CT data with the voxel grid to ensure that the voxel grid matches the structures in the CT data. This usually requires transforming the CT data and the voxel grid to the same coordinate system.

[0305] Density Mapping:

[0306] Density data extraction: Extract the density value corresponding to each voxel position from the CT data. The position of each voxel will correspond to the pixel position in the CT slice, thereby obtaining density information.

[0307] Density Mapping to Voxels: Map the density values ​​in the CT data to a voxel grid. The color or transparency of each voxel will be adjusted according to its corresponding density value.

[0308] Coloring:

[0309] Color mapping: Color voxels according to the density value in the CT data. Common methods include:

[0310] Colormap mapping: Convert density values ​​to color values ​​using a predefined color map (e.g., pseudo color mapping). For example, low density areas can be mapped to blue, while high density areas can be mapped to red.

[0311] Gradient Map: Use a gradient color map to map the density value range to a continuous color gradient to reflect the change from low density to high density.

[0312] Transparency Adjustment: Adjusts the transparency of voxels based on their density values. Areas with higher density may have lower transparency, while areas with lower density may have higher transparency. This adjustment helps to highlight internal structures in 3D views.

[0313] Optimizing voxel shading

[0314] Detail Enhancement:

[0315] Texture refinement: Refine the texture of the voxel grid to enhance the expression of image details. Refinement can be achieved by increasing texture resolution, adjusting color mapping parameters, etc.

[0316] Edge Enhancement: Use edge detection algorithms to enhance edge details in images and improve the visualization of internal structures.

[0317] Visualization Adjustment:

[0318] Lighting and shading: Apply lighting and shading effects during voxel shading to enhance the visual quality of 3D models. This helps to better reveal structures and details in volumetric data.

[0319] Interactive Adjustment: Provides interactive adjustment capabilities that enable users to modify color mapping, transparency, and lighting settings in real time to meet different analysis needs.

[0320] Figure 8 FIG. 4 is a flow chart of coloring the final voxel array based on the reconstructed CT data according to an embodiment of the present invention. Figure 8 The method S330 of coloring the final voxel array based on the reconstructed CT data of this embodiment may include the following steps S3301, S3302, S3303, S3304, S3305 and step S3306.

[0321] Combined with the surface coloring algorithm of the CT reconstruction voxel result V, this algorithm aims to color the surface of the CT reconstruction voxel data to improve the visual effect and accuracy of the reconstructed model. The specific steps are as follows:

[0322] S3301 performs threshold segmentation on the CT reconstruction voxels. Threshold segmentation: Voxels are divided into air parts and non-air parts according to voxel values. Voxels in the air part are marked as 1, and voxels in the non-air part are marked as -1.

[0323] S3302 3D Gaussian filtering, voxel labeling: Perform 3D Gaussian filtering on the labeled CT voxel data to smooth the voxel data and remove noise. The processed voxel data is similar to the TSDF (Truncated Signed Distance Function) calculation result and serves as the initial TSDF labeling.

[0324] S3303 Initialize weight voxels, initialize weight voxels W: define weight voxel set ,in is an adjustable weight parameter.

[0325] S3304 Initialize voxel color, initialize color RGB: the initial color of all voxels is set to all 0 (black).

[0326] S3305 processes the depth image and traverses each depth image: performing the following operations on each voxel:

[0327] a. Calculate the projection position: calculate the projection position of the voxel P on the depth image, obtain the color rgb(P) of the projection position, and skip the voxel if there is no color information at the position;

[0328] b. Get the depth value: Get the depth value of the projection position of point P on the depth image ;

[0329] c. Calculate voxel depth: Calculate the depth of voxel P to the camera ;

[0330] d. Calculate the SDF value: Calculate the SDF value corresponding to voxel P: ;

[0331] e. Calculate TSDF value: Calculate the TSDF value of voxel P:

[0332]

[0333] Among them, t is the preset threshold.

[0334] f. Calculate current weight: Calculate the weight of the current voxel:

[0335]

[0336] in, is the viewing angle of the voxel in the depth image.

[0337] g. Fusion of TSDF and color: Fusion of the current TSDF value and color with the historical TSDF value. The fusion formula is as follows:

[0338]

[0339]

[0340]

[0341] S3306 determines the best surface position and color, determines the best surface position: after processing, the center of the transition zone from 1 to -1 (0 separation zone) is the best surface position. The color of this position is determined by the corresponding RGB voxel value.

[0342] The algorithm can improve the visualization of the model and ensure the accuracy of surface reconstruction by accurately coloring the surface of CT reconstructed voxel data and combining it with the color information in the depth image.

[0343] The above describes in detail the preferred implementation of this patent, but this patent is not limited to the above implementation. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of this patent.

Claims

1. A method for image data fusion and color enhancement processing, characterized in that: Acquire facial data. Scan the face in CT mode to obtain detailed anatomical data of the face, and use a high-definition camera to obtain RGBD image data. Data reconstruction: CT scan data will be reconstructed to obtain detailed tomographic images, and the surface will be reconstructed using RGBD data collected by high-definition cameras to ultimately generate three-dimensional point cloud data of the face; as well as Surface coloring processing: performing surface coloring processing on the reconstructed CT data and facial point cloud data to generate the final three-dimensional image. The specific method and steps of surface coloring processing are as follows: Three-dimensional cube encirclement: The target surface is enclosed by a three-dimensional cube using the TSDF algorithm, and the cube is divided into several volume pixel arrays according to the required reconstruction accuracy. The voxels are used for further surface construction and processing; Depth map scoring: Scoring each voxel in each depth map, determining the most accurate surface position by combining depth data from different perspectives, and ultimately constructing the fine surface structure of the face; Surface shading, which shades the final voxel array based on the reconstructed CT data, The coloring process of the final voxel array based on the reconstructed CT data includes the following steps: Threshold separation is performed on CT reconstruction voxels: voxels are divided into air parts and non-air parts according to the voxel value, the voxels of the air part are marked as 1, and the voxels of the non-air part are marked as -1; Three-dimensional Gaussian filtering and voxel labeling: The labeled CT voxel data is subjected to three-dimensional Gaussian filtering to smooth the voxel data and remove noise; the processed voxel data is used as the initial TSDF label; Initialize weight voxel W: define the weight voxel set ,in is an adjustable weight parameter; Initialize voxel color RGB: the initial color of all voxels is set to all 0; Process the depth image and traverse each depth image; Determine the optimal surface position and color: After processing, the center of the transition zone from 1 to -1 is the optimal surface position, and the color of this position is determined by the corresponding RGB voxel value.

2. The image data fusion and coloring enhancement processing method according to claim 1, characterized in that: Obtaining facial data includes the following steps: Acquire facial data scanned in CT mode, wherein the step includes scanning the patient's face through a CT scanning module of an oral CBCT device to acquire facial CT data with a three-dimensional structure, wherein the facial CT data contains detailed information of bone structure and soft tissue; Acquire RGBD image data captured by a high-definition camera, wherein this step includes photographing the patient's face with a high-definition camera integrated in the oral CBCT device to obtain RGBD image data with depth information, wherein the RGBD image data includes texture, color, and depth information of the facial surface.

3. The image data fusion and coloring enhancement processing method according to claim 2, characterized in that: The method of obtaining facial data scanned in CT mode comprises the following steps: Patient preparation and positioning: ensuring the patient is accurately positioned on the scanning bed; Scan parameter selection: selecting appropriate scan parameters based on the patient's specific circumstances, including facial features, medical history, and scanning requirements; Scanning data acquisition, before starting the scan, make sure all settings are configured correctly. During the scan, the X-ray tube and detector rotate around the patient to collect facial tomographic data at multiple angles. Signal acquisition and conversion: The detector receives the X-ray signal after penetrating the human body, and converts the signal into an electrical signal and transmits it to the computer to generate preliminary projection data; Data storage: All collected raw data will be securely stored in the system for subsequent image reconstruction and analysis.

4. The image data fusion and coloring enhancement processing method according to claim 2, characterized in that: The method of obtaining the RGBD image data captured by the high-definition camera comprises the following steps: Motion trajectory setting: according to the specific application scenario or task requirements, the motion trajectory of the high-definition camera is planned. The trajectory includes straight lines, curves or complex paths, and is accurately set through the controller or dedicated software to ensure the coverage and consistency of data collection; HD camera calibration: Before starting to shoot, the HD camera is fully calibrated, including adjusting the camera's focal length, exposure time, and depth sensor sensitivity parameters to obtain high-quality image data; Data Acquisition,As the high-definition camera moves along a preset trajectory, it captures RGBD images in real time.,The images include not only the traditional RGB color information,,but also a depth channel to describe the distance of each object in the scene,from the camera.

5. The image data fusion and coloring enhancement processing method according to claim 1, characterized in that: Data reconstruction includes the following steps: Reconstructing the CT data, wherein the step includes processing the acquired facial CT scan data using a three-dimensional reconstruction algorithm to generate a three-dimensional stereoscopic model, which describes the internal structure of the face in detail, including the distribution of bones and soft tissues; Surface reconstruction of RGBD image data, wherein this step includes processing the acquired RGBD image data by using a deep image reconstruction technique to generate a three-dimensional model of the external surface of the face, the model including the texture and color information of the facial skin and corresponding depth data.

6. The image data fusion and coloring enhancement processing method according to claim 5, characterized in that: Reconstructing CT data includes the following steps: CT data preprocessing: Correct the collected raw CT data, including ray hardening correction or detector response correction, to eliminate possible artifacts and distortions, ensure data accuracy, and convert projection data in the time domain into image data in the space domain. Image reconstruction: using advanced reconstruction algorithms to reconstruct the corrected projection data into high-resolution tomographic images, and further processing the reconstructed images, including noise removal, contrast enhancement, and edge sharpening, to improve image quality and visualization effects.

7. The image data fusion and color enhancement processing method according to claim 5, characterized in that: Surface reconstruction of RGBD image data includes the following steps: Coordinate system consistency reconstruction, when reconstructing the surface data of the HD camera, the same coordinate system as the CT data is used; Application of TSDF algorithm, using TSDF algorithm to reconstruct the surface of RGBD data, by calculating the implicit surface in three-dimensional space, generating three-dimensional point cloud data of the face to describe the external morphology of the face, Data integration display integrates the reconstructed surface scan data and CT data in the same software for display, and adjusts the transparency of the surface scan data.

8. The image data fusion and coloring enhancement processing method according to claim 1, characterized in that: The depth map scoring includes the following steps: Initialize all the marked voxels. The initial marking value of all voxels is 1, which means that all voxels are to be processed in the initial state. The marking voxel value of 1 means that the voxel will participate in the subsequent depth map data processing and scoring. Initialize weight voxels, Initialize weight voxels ; The weight voxel W is used to calculate the weighted average of each voxel to fuse data from different perspectives; Initialize voxel color, initialize voxel color RGB to all 0; prepare the initial state for voxel coloring; Process each depth image; Determine the best surface processing result, and the final voxel data will present a distribution similar to the target surface; The area where the TSDF value is close to 0 represents the position of the surface, and the best surface position is determined, that is, the position where the TSDF value is 0; Surface extraction, uses extraction algorithms to extract surface meshes from voxel meshes, generates visual 3D surface models from voxel meshes, and displays the final reconstruction results.

Images