A spinal region segmentation method and system based on multi-modal three-dimensional gaussian representation

Abstract

The application discloses a kind of based on multi-modal three-dimensional reconstruction's spinal region segmentation method and system.The method includes: the multi-view visible light image of target object back and infrared thermal imaging image are synchronously collected, and cross-modal pixel level registration is realized by global optimal depth estimation;Initial sparse point cloud is generated based on foreground mask constraint, and multi-modal three-dimensional Gaussian point cloud model of temperature attribute is constructed;Through introducing foreground mask weighted joint loss function, differentiable rendering optimization is carried out to Gaussian point cloud, and noise cleaning is carried out to point cloud based on covariance perception mahalanobis distance;Finally, the three-dimensional segmentation result of spinal region is obtained by point cloud segmentation network of fusion space geometry feature and temperature feature.The application realizes the unified expression of visible light geometry information and infrared physiological information in three-dimensional space, effectively suppresses background interference and abnormal noise, improves the precision and robustness of spinal region segmentation, and has the advantages of no radiation, non-contact, low cost and the like.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and 3D vision computing technology, specifically relating to a method for 3D reconstruction and segmentation of the spinal region based on multimodal image fusion. This method combines visible light and infrared thermal imaging data to generate a Gaussian point cloud with fused temperature information through joint reconstruction. This point cloud segmentation network is then used to achieve accurate identification and segmentation of the spinal region on the human back. This method can be applied to non-contact, non-invasive spinal screening, health assessment, and assisted diagnosis scenarios. Background Technology

[0002] The morphology and functional state of the spinal structure are of great significance to human health. Scoliosis and postural abnormalities are common in adolescents and adults. If not detected and intervened in a timely manner, they may further lead to muscle imbalances, chronic pain, and functional impairments. Therefore, accurate detection and analysis of spinal regions are of great value for early screening, rehabilitation assessment, and clinical diagnostic support.

[0003] Currently, commonly used methods for spinal examination in clinical practice include X-ray imaging, computed tomography (CT), and magnetic resonance imaging (MRI). While X-rays and CT can clearly reflect bone structure, they pose a risk of ionizing radiation and are not suitable for high-frequency examinations. MRI, although radiation-free, has high equipment costs and long examination cycles, making it difficult to meet the needs of large-scale screening and routine monitoring. Therefore, developing radiation-free, low-cost spinal examination methods is of significant practical importance.

[0004] To address this, researchers have proposed non-invasive detection methods based on optical 3D measurement, such as using structured light, binocular vision, or multi-view reconstruction techniques to obtain the 3D geometry of the human back. These methods can reconstruct the surface structure of the back relatively accurately; however, because the spine is located within the subcutaneous tissue, relying solely on surface geometry information is insufficient to accurately reflect its actual position. Uneven distribution of back muscles, localized fat accumulation, and natural protrusions of bony structures can all interfere with the 3D morphology, leading to deviations in segmentation results based on geometric features.

[0005] On the other hand, infrared thermal imaging technology can reflect the temperature distribution information of the human body surface, and to a certain extent, it can reflect the characteristics of tissue metabolism and blood circulation. The spinal region has a certain temperature distribution pattern under specific physiological conditions, and introducing infrared information into spinal detection can help provide supplementary discrimination criteria. However, infrared images usually have problems such as low resolution and insufficient spatial structure representation ability, making it difficult to complete fine region localization and segmentation tasks on their own.

[0006] With the development of deep learning technology, point cloud segmentation methods have been applied to the field of 3D human body analysis. However, existing point cloud segmentation methods mostly use spatial coordinates and color information as the main input features, lacking the ability to fuse multimodal information. In scenes such as the human back with gentle geometric changes and indistinct texture features, traditional methods struggle to extract discriminative features, easily leading to blurred segmentation boundaries and insufficient recognition accuracy. Furthermore, existing 3D representation methods have limited ability to express additional physiological information (such as temperature features), which also restricts the improvement of segmentation performance.

[0007] In summary, existing technologies still have the following shortcomings in spinal region detection and segmentation: (1) Existing medical imaging methods have problems such as radiation risks or high costs, making it difficult to meet the needs of high-frequency, non-invasive detection. (2) Detection methods based on a single mode (visible light or infrared) have limited information dimensions, are easily affected by external factors, and lack robustness. (3) Existing 3D reconstruction and point cloud segmentation methods mainly rely on geometric or color features and lack effective integration of physiological information such as temperature, resulting in low segmentation accuracy under complex body surface conditions. (4) Existing three-dimensional representation methods are mostly discrete points or mesh structures, which have limited ability to express multimodal continuous attributes (such as temperature information) and make it difficult to achieve unified modeling and efficient fusion of multi-source information.

[0008] To address the aforementioned issues, current technologies lack a solution that can effectively integrate geometric and temperature information in three-dimensional space while simultaneously ensuring both reconstruction accuracy and segmentation precision. Therefore, there is an urgent need to propose a three-dimensional reconstruction method that integrates visible light and infrared information, combined with novel point cloud representation and segmentation techniques, to achieve high-precision, non-contact three-dimensional segmentation and localization of the spinal region to meet practical application requirements. Summary of the Invention: To address the problems in existing technologies, such as radiation risks in medical imaging, lack of physiological information and susceptibility to surface morphology interference in single visible light 3D reconstruction, lack of spatial structural information in single infrared thermal imaging, and insufficient utilization of multimodal features in existing point cloud segmentation methods, this invention aims to provide a spinal region segmentation method and system based on multimodal 3D reconstruction. By fusing visible light images and infrared thermal imaging data, temperature information is introduced into 3D space, and combined with improved point cloud representation and segmentation methods, high-precision, radiation-free, and non-contact 3D segmentation of the spinal region is achieved.

[0009] To achieve the above objectives, the present invention provides a spinal region segmentation method based on multimodal three-dimensional reconstruction, the method comprising the following steps: Step S1 Multimodal data acquisition: Simultaneously acquire the three-dimensional geometric information and temperature distribution information of the back surface of the target object, and constrain the temporal synchronization and spatial consistency of the data acquisition process; Step S2 Multimodal Image Registration and Mapping: Establish pixel-level mapping relationship between the two modal images through joint calibration of visible light camera and infrared camera; Step S3: Foreground mask extraction and 3D reconstruction initialization: Foreground segmentation is performed on the visible light image to obtain the foreground mask of the target region. The foreground mask is then used to filter the feature points extracted during the 3D reconstruction process, retaining only the feature points located in the foreground region to generate an initial sparse point cloud. Step S4 Multimodal 3D Gaussian Modeling and Joint Optimization: Based on the sparse point cloud, a multimodal 3D Gaussian point cloud model containing temperature attributes is constructed, and the Gaussian point cloud model is iteratively optimized through differentiable rendering based on the joint loss function with foreground mask constraints. Step S5 Point Cloud Optimization Processing: For the optimized Gaussian point cloud model, calculate the statistical distance based on the covariance matrix of each Gaussian unit, and remove abnormal Gaussian units according to the statistical distance to obtain point cloud data with consistent structure. Step S6: Multimodal Feature Fusion and Point Cloud Segmentation: The optimized multimodal Gaussian point cloud is input into a pre-trained point cloud segmentation network. By fusing spatial geometric features and temperature features, the three-dimensional segmentation result of the spine region is obtained.

[0010] In step S2, a cross-modal mapping model dependent on depth parameters is constructed based on the joint calibration parameters of the visible light camera and the infrared camera. The optimal depth parameter is determined through candidate search and multi-view statistical optimization within a preset depth range, so as to achieve pixel-level alignment of infrared temperature information to the visible light image coordinate system. The determination of the optimal global depth parameter includes: calculating the chamfer distance between the infrared projection edge and the visible light edge at each candidate depth and the gradient direction consistency error, and performing statistical optimization on the error of the multi-view image pair, and selecting the depth value that minimizes the comprehensive error as the optimal global depth.

[0011] In step S2, for each candidate depth, a comprehensive error evaluation index is constructed by calculating the spatial matching degree between the infrared edge and the visible light edge after projection transformation, as well as the consistency of their gradient directions. , in, For the set of visible light edges, For depth-based The set of infrared edges after projection mapping; and This corresponds to the gradient direction field; Used to evaluate the geometric distance error of the edge point set; Used to evaluate differences in gradient direction distribution, such as calculations based on cosine similarity or angular deviation; To balance the hyperparameters.

[0012] In step S4, each Gaussian unit in the multimodal 3D Gaussian point cloud model includes spatial position parameters, covariance parameters, color attributes, opacity parameters, and temperature attributes.

[0013] In step S4, the joint loss function is composed of a weighted sum of the photometric consistency loss based on the rendered color map and the real visible light image, and the temperature consistency loss based on the rendered temperature map and the real infrared thermal imaging image. The foreground mask generated in step S3 is introduced into the loss function as a weighting factor to suppress the interference of the background region.

[0014] In step S4, the joint loss function The following relationship must be satisfied: in, and These are real visible light and infrared images, respectively; and These are rendered visible light and infrared images, respectively; This represents the photometric error between real visible light and rendered visible light. , M is a preset weighting coefficient used to balance the contributions of color loss and temperature loss; 2D Foreground mask.

[0015] In step S5, the statistical distance is the Mahalanobis distance based on the covariance matrix of two Gaussian units, specifically defined as: in, , Let be the spatial mean vector of the Gaussian cell. , Let be the corresponding covariance matrix.

[0016] In step S6, the point cloud segmentation network uses a multi-branch structure to extract spatial geometric features and temperature features respectively, and outputs a segmentation mask for the spine region after weighted fusion of the features through an attention mechanism. The point cloud segmentation network is built based on PointNet++, DGCNN, or PointTransFormer architecture.

[0017] A spinal region segmentation system based on multimodal 3D Gaussian representation, characterized in that the system, used to execute the above method, comprises: Multimodal data synchronous acquisition module: It is equipped with a single-node multimodal imaging module, an electronically controlled turntable and a synchronous trigger control unit, which are used to synchronously acquire visible light image sequences and infrared thermal imaging image sequences of the back of the target object from multiple perspectives; The registration and 3D reconstruction initialization module stores the intrinsic and extrinsic calibration parameters of the visible light camera and the infrared camera. It is configured to perform multimodal image registration and mapping based on the globally optimal depth and to generate an initial sparse point cloud using the structure-of-motion algorithm. Multimodal 3D Gaussian Modeling and Joint Optimization Module: Configured to instantiate multimodal Gaussian units containing spatial location, covariance, color, opacity, and temperature attributes based on the sparse point cloud; calculate and render color maps and render temperature maps based on a differentiable rendering mechanism; and iteratively optimize the parameters of each Gaussian unit through backpropagation according to the joint loss function based on foreground mask constraints to obtain a multimodal 3D Gaussian model. Point cloud cleaning and noise reduction module: configured to perform neighborhood consistency judgment on Gaussian cells in the multimodal 3D Gaussian model using a statistical distance metric based on covariance perception, remove isolated abnormal Gaussian cells, and output optimized Gaussian point cloud; Multimodal feature fusion and point cloud segmentation module: It is preloaded with a pre-trained point cloud segmentation network, configured to extract spatial geometric features and physiological features from the optimized Gaussian point cloud, and use the attention mechanism to perform feature fusion, outputting a 3D segmentation mask of the spinal region. Visualization output module: configured to combine the spinal region segmentation results with temperature attributes to generate a three-dimensional visualization model, and present it through a display device.

[0018] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention introduces temperature information in three-dimensional space by fusing visible light and infrared thermal imaging data, which overcomes the limitation of single-modal information dimension and improves the accuracy of spinal region identification.

[0019] (2) By constructing a multimodal three-dimensional Gaussian point cloud model, this invention realizes the unified expression of geometric and physiological information, enhances the model's ability to represent complex body surface structures, and solves the problem that existing representation methods have limited ability to express multimodal continuous attributes.

[0020] (3) This invention effectively suppresses background interference and improves the accuracy of 3D reconstruction by introducing a joint optimization mechanism based on foreground mask.

[0021] (4) The present invention uses a point cloud optimization method based on covariance perception to adaptively perceive anisotropic structures using Mahalanobis distance, thereby removing noise points while maintaining the continuity of the point cloud structure and overcoming the defect of traditional Euclidean distance filtering that easily deletes effective structural points.

[0022] (5) The present invention uses a point cloud segmentation method based on multimodal feature fusion, which enables the network to adaptively adjust the contribution weights of geometric information and temperature information, thereby improving the robustness and accuracy of spinal region segmentation. (6) The present invention adopts a non-contact detection method, which does not require the use of medical imaging equipment with radiation, and has the advantages of high safety, low cost and suitability for large-scale screening.

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0024] Figure 1 Method flowchart.

[0025] Figure 2 : Schematic diagram of multimodal acquisition.

[0026] Figure 3 Schematic diagram of multimodal 3D reconstruction and optimization.

[0027] The attached figures are labeled as follows: 1. Target object; 2. Electrically controlled turntable; 3. Multimodal imaging module; 31. Visible light camera; 32. Infrared thermal imaging camera; 4. Host computer; 5. Cold light source. Detailed Implementation

[0028] This invention provides a method for spinal region segmentation based on joint reconstruction of infrared and visible light images. This method innovatively introduces human physiological temperature features into the underlying 3D Gaussian point cloud architecture, and also incorporates foreground mask constraints and a noise removal mechanism based on Mahalanobis statistical distance, achieving deep coupling of multimodal features and high-precision segmentation in complex environments.

[0029] To facilitate understanding of the overall technical concept of the present invention, the general flow of the method is first summarized. Figure 1 This diagram illustrates a flowchart of a spinal region segmentation method based on joint reconstruction of infrared and visible light images, provided by an embodiment of the present invention. (Refer to...) Figure 1 This method mainly includes the following six core steps: synchronous acquisition of multimodal data (S1), multimodal image registration and mapping based on global optimal depth (S2), foreground mask extraction and 3D reconstruction initialization (S3), multimodal 3D Gaussian modeling and joint optimization (S4), point cloud cleaning and noise reduction based on covariance-aware Mahalanobis statistical distance (S5), and point cloud segmentation and result output based on multimodal feature fusion (S6). The steps are closely interconnected, jointly achieving end-to-end processing from raw multi-view multimodal data to high-precision 3D segmentation results of the spine region.

[0030] The following detailed explanation of each step is provided in conjunction with specific embodiments.

[0031] Step S1: Multimodal data acquisition In this embodiment, in order to simultaneously acquire the three-dimensional geometric information and temperature distribution information of the target object's back surface, constraints are imposed on the temporal synchronization and spatial consistency of the data acquisition process.

[0032] like Figure 2 As shown, the data acquisition system includes a single-node multimodal imaging module 3 and an electrically controlled turntable 2. The multimodal imaging module 3 integrates a visible light camera 31 and an infrared thermal imaging camera 32, which are rigidly connected and fixedly installed to maintain their relative positional relationship. The electrically controlled turntable 2 is used to carry the target object and realize multi-view acquisition.

[0033] To reduce the impact of the external environment on the collected data, the preferred acquisition environment is a closed space with constant temperature and no significant airflow interference, and is equipped with a uniformly distributed cold light source 5 to reduce the interference of shadow effects and environmental thermal radiation changes on infrared imaging, thereby improving the stability of data acquisition.

[0034] The target object 1 has its back skin area exposed and is positioned at the center of the electrically controlled turntable 2 in a natural standing posture. An automated acquisition process is executed by the host computer 4, employing a "step-hover-synchronous acquisition" control method. Specifically, the host computer 4 sends a rotation command to the electrically controlled turntable 2, causing it to rotate stepwise at preset angular intervals. When the turntable reaches the target angle and comes to a standstill, the host computer 4 sends a synchronous trigger signal to the multimodal imaging module 3 via a software interface or hardware trigger.

[0035] Under the control of the trigger signal, the visible light camera 31 and the infrared thermal imaging camera 32 synchronously complete the data acquisition of the current viewing angle in time, and transmit the corresponding visible light image and infrared thermal imaging image to the host computer 4 for storage. By repeating the above-mentioned stepping rotation and synchronous acquisition process, multi-view image data covering the back area of ​​the target object is obtained, forming a visible light image sequence and a corresponding infrared image sequence.

[0036] Preferably, the rotation range of the turntable covers a predetermined angular range of the target object's back to obtain complete or nearly complete back information.

[0037] Step S2: Multimodal image registration and mapping based on globally optimal depth Since the visible light camera and the infrared thermal imaging camera in the single-node multimodal imaging module differ in parameters such as field of view and focal length, a cross-modal pixel-level mapping relationship needs to be established to achieve accurate alignment between the two modal images.

[0038] In this embodiment, a calibration board is first used to jointly calibrate the visible light camera and the infrared camera. Preferably, the calibration board is a checkerboard structure with different thermal response characteristics, enabling it to form clear corner features in both visible light and infrared images. Based on the calibration images, the intrinsic parameter matrix of the visible light camera is solved separately. and the intrinsic parameter matrix of the infrared camera Furthermore, the relative pose relationship between the two cameras is calculated, including the rotation matrix. With translation vector This allows for the establishment of a unified camera coordinate system.

[0039] Considering that the baseline distance between the two cameras is relatively small compared to the target observation distance, the scene depth distribution corresponding to the visible light image and the infrared image at the same viewpoint has a high degree of consistency. Based on this assumption, this embodiment constructs an adaptive mapping model dependent on depth parameters to realize the projection transformation of the infrared image to the visible light image coordinate system.

[0040] In the specific implementation process, edge detection and gradient calculation are first performed on the visible light image and the infrared image respectively to extract the edge features and gradient direction information under the two modalities. Since there are differences in gray-scale distribution between different modalities, but high consistency in the contour and structural boundaries of the human back, edge features and gradient direction are used as the constraint basis for cross-modal registration.

[0041] Subsequently, multiple candidate depth parameters are selected within a preset depth range, and corresponding mapping relationships are constructed based on camera intrinsic and extrinsic parameters. The infrared image is then projected and transformed into the visible light image coordinate system under different depth assumptions. For each candidate depth, a comprehensive error evaluation index is constructed by calculating the spatial matching degree between the projected infrared edge and the visible light edge, as well as the consistency of their gradient directions. Specifically, for a given candidate depth parameter The indicator is defined as follows: in, For the set of visible light edges, For depth-based The set of infrared edges after projection mapping; and This corresponds to the gradient direction field; Used to evaluate the geometric distance error of the edge point set; Used to evaluate differences in gradient direction distribution, such as calculations based on cosine similarity or angular deviation; To balance the hyperparameters.

[0042] To improve the robustness of the registration results, this embodiment employs a statistical optimization strategy for multi-view image pairs. Specifically, the aforementioned error indices are calculated for multiple sets of image pairs from different viewpoints, and the error of the same candidate depth across all image pairs is statistically analyzed. The median is preferably used as the global evaluation index to reduce the impact of outliers on the results. By traversing and searching all candidate depth parameters, the depth value that minimizes the global evaluation index is selected as the optimal global depth.

[0043] Finally, the optimal depth parameters are substituted into the adaptive mapping model to achieve accurate mapping from the infrared image to the visible light image coordinate system, so that the infrared temperature information can be aligned with the visible light image at the pixel level, thereby providing a high-precision data foundation for subsequent multimodal 3D reconstruction and Gaussian point cloud modeling.

[0044] Step S3: Foreground mask extraction and 3D reconstruction initialization In real-world applications, acquired images often contain background areas (such as walls, equipment, or other non-target objects). This redundant information can interfere with feature matching and camera pose estimation during 3D reconstruction, thus affecting reconstruction accuracy and computational efficiency. Therefore, this step introduces a two-dimensional foreground mask to constrain the 3D reconstruction process.

[0045] like Figure 3 As shown, firstly, the registered visible light image sequence obtained in step S2 is input into a pre-trained two-dimensional visual segmentation model, such as a human segmentation network based on U-Net, DeepLabV3+, or Segment Anything Model (SAM) architecture, to perform pixel-level classification of the image, distinguishing the back region of the target object from the background region. The model output is then thresholded to generate a corresponding two-dimensional foreground mask. For example, pixels with values ​​greater than a preset threshold (e.g., 0.5) in the probability map output by the model are identified as target regions and assigned a value of 1, while other pixels are assigned a value of 0. The foreground mask is used to limit the effective region in subsequent processing.

[0046] After obtaining the foreground mask, a 3D reconstruction method based on multi-view geometry is used to model the target region. Specifically, features are extracted and described from the visible light image sequence (e.g., using SIFT, SURF, or SuperPoint algorithms) to obtain a set of feature points with scale and rotation invariance. Further, the feature points and their corresponding foreground masks are spatially filtered, retaining only valid feature points located within the foreground region and discarding feature points located within the background region.

[0047] Subsequently, the selected feature points are used for feature matching and camera pose estimation. A structure-based motion approach is used to establish the correspondence between multiple viewpoints, and bundle adjustment is employed to jointly optimize the camera parameters. Based on this, spatial point positions are recovered through triangulation calculations, generating an initial sparse point cloud model of the target object's back region.

[0048] By using the above processing method, the two-dimensional foreground mask is used to impose prior constraints on the three-dimensional reconstruction process, which can effectively reduce background interference, improve the accuracy of feature matching and the stability of camera pose estimation, and reduce computational complexity, thereby obtaining a high-quality initial three-dimensional structure that is more focused on the target region.

[0049] It should be understood that the extraction of the foreground mask is not limited to deep learning segmentation models, but can also be obtained by traditional background subtraction algorithms or depth sensor threshold segmentation methods, as long as they can distinguish between the target region and the background region.

[0050] Step S4: Multimodal 3D Gaussian Modeling and Joint Optimization After obtaining the sparse point cloud, in order to achieve high-precision modeling of the back region of the target object, this embodiment constructs a multimodal three-dimensional Gaussian point cloud model that incorporates temperature information and performs joint optimization based on a differentiable rendering mechanism.

[0051] like Figure 3 As shown, firstly, using the sparse point cloud generated in step S3 as the spatial initialization basis, a corresponding three-dimensional Gaussian distribution unit is constructed at the location of each point, thereby forming a continuous three-dimensional Gaussian point cloud representation. Each Gaussian unit includes at least the following attributes: spatial location parameter, covariance parameter, color attribute, opacity parameter, and temperature attribute. The temperature attribute is assigned by the cross-modal mapping relationship established in step S2 and is used to characterize the infrared thermal information of the corresponding spatial location, thereby achieving a unified expression of geometric and physiological information.

[0052] Based on this, the multimodal Gaussian point cloud is projected onto a two-dimensional image plane at a given viewpoint, and pixel values ​​are cumulatively calculated based on volumetric rendering principles. Specifically, for each pixel, multiple Gaussian units are weighted and fused along its viewing direction. The contribution weight of different Gaussian units is controlled by an opacity parameter, thereby obtaining the corresponding rendered color value and rendered temperature value. In this way, a unified modeling of the appearance and temperature distribution of visible light can be achieved in continuous space.

[0053] Furthermore, to improve the model's fitting ability to the target region and suppress background interference, this embodiment constructs a joint optimization model based on foreground mask constraints. The joint loss function... The following relationship must be satisfied: in, and These are real visible light and infrared images, respectively; and These are rendered visible light and infrared images, respectively; The photometric error between real visible light and rendered visible light can be addressed, for example, by using a weighted combination of L1 loss and structural similarity (SSIM) loss; The temperature error between the real infrared image and the rendered infrared image can be represented by, for example, a weighted combination of L1 loss, structural similarity loss (SSIM) loss, and mean squared error (MSE) loss. , M is a preset weighting coefficient used to balance the contributions of color loss and temperature loss. 2D Foreground mask.

[0054] The gradient of the total loss function with respect to the parameters of each Gaussian unit is calculated using the backpropagation algorithm, and the parameters of each Gaussian unit are iteratively updated according to the gradient descent direction. During the optimization process, the Gaussian unit set can be dynamically added to and removed based on the cumulative gradient magnitude and opacity value of the Gaussian units. For example, Gaussian units located in under-reconstructed regions with large gradients are cloned or split, and Gaussian units with opacity below a preset threshold are pruned. Finally, a multimodal 3D Gaussian model that faithfully reproduces the geometric appearance and temperature distribution of the target object's back is obtained.

[0055] By introducing mask constraints and a multimodal joint optimization mechanism, this embodiment can effectively reduce the generation of invalid Gaussian units in the background region, suppress the spread of noise points, and improve the modeling accuracy of the human back structure and temperature distribution, ultimately obtaining a multimodal three-dimensional Gaussian model that integrates high spatial resolution and high-precision temperature information.

[0056] Step S5: Point cloud optimization processing based on covariance perception After completing the multimodal 3D Gaussian point cloud modeling and joint optimization, there may still be anomalous Gaussian units in the model due to camera calibration errors, environmental interference, or insufficient optimization. These anomalous units usually appear as isolated noise points in space, requiring further optimization of the Gaussian point cloud.

[0057] Considering the continuous and smooth geometric features of the human back surface, a large number of Gaussian units in the Gaussian point cloud exhibit anisotropic distribution along the body surface direction, meaning that there are scale differences in different directions within local regions. If a traditional filtering method based on Euclidean distance is used, it is easy to misclassify effective Gaussian units distributed along the curved surface direction as outliers, thereby disrupting the continuity of the point cloud structure.

[0058] To address this, this embodiment introduces a distance metric based on covariance perception to evaluate the spatial relationships between Gaussian units. Specifically, for any Gaussian unit, its spatial location parameters are used as the center, and a local spatial distribution model is constructed based on its covariance matrix. During the neighborhood search process, the influence of the covariance matrix on the spatial scale is considered, and a weighted distance calculation is performed on the neighboring Gaussian units, thereby achieving adaptive perception of anisotropic structures.

[0059] In a preferred embodiment, the distance metric is calculated using Mahalanobis distance. For any two Gaussian units... and Let their spatial mean vectors be respectively and The covariance matrix is and Its Mahalanobis distance Defined as: This distance normalizes the spatial distance in different directions by introducing the inverse of the covariance matrix, which compresses the distance in the principal direction of the Gaussian unit and amplifies the distance in the vertical direction, thereby more accurately reflecting the true anisotropic structural relationship of the point cloud in the local region.

[0060] Based on the aforementioned distance metric, a local neighborhood is constructed for each Gaussian cell, and a consistency judgment is made based on the distribution of Gaussian cells within the neighborhood. When the spatial distribution difference between a Gaussian cell and its neighborhood exceeds a preset threshold, it is identified as an outlier and is removed or marked; otherwise, it is retained as a valid structural point.

[0061] The point cloud optimization method based on covariance perception described above can effectively remove isolated noise points in space while preserving the continuous Gaussian structure distributed along the curved surface of the human back. This avoids the erroneous deletion of effective structures by traditional methods, thereby improving the purity and structural integrity of point cloud data and providing high-quality input for subsequent point cloud segmentation.

[0062] Step S6: Point cloud segmentation and result output based on multimodal feature fusion After obtaining the optimized multimodal 3D Gaussian point cloud, in order to achieve accurate identification of the spine region of the target object, this embodiment processes the point cloud based on a point cloud segmentation method of multimodal feature fusion.

[0063] Specifically, the multimodal Gaussian point cloud is input into a pre-trained point cloud segmentation network, which may employ a deep learning model based on PointNet++, DGCNN, or PointTransformer architecture. Feature encoding is performed on each Gaussian unit or its corresponding point. The features include at least spatial geometric features and physiological features, where the spatial geometric features consist of the position parameters and covariance information of the Gaussian unit, and the physiological features consist of color and temperature information. For example, three-dimensional coordinates, covariance matrix eigenvalues, RGB color values, and normalized temperature values ​​can be concatenated into a multidimensional feature vector as network input.

[0064] In the feature extraction process, a multi-branch feature encoding structure is preferred, which independently models different modal features to fully preserve the expressive power of various features. In the feature fusion stage, an attention-based fusion strategy is introduced to model the correlation between different modal features. Specifically, by constructing the correlation between features, different modal features are weighted and fused, enabling the network to adaptively adjust the contribution of geometric and temperature information in the segmentation process according to the differences in local region features.

[0065] Through the aforementioned multimodal feature fusion mechanism, the network can enhance its sensitivity to temperature distribution differences in regions with relatively gentle geometric changes, and strengthen its response to geometric changes in regions where temperature features are not significant, thereby improving the recognition accuracy and robustness of the spinal region.

[0066] After feature extraction and fusion, the point cloud segmentation network outputs a corresponding category label for each Gaussian unit, generating a 3D segmentation result for the spine region. Based on the segmentation result, point cloud data belonging to the spine region is extracted, and a corresponding 3D segmentation mask is constructed.

[0067] Furthermore, the point cloud of the spinal region is visualized to generate a corresponding 3D model on a display device, intuitively showcasing the spatial structure and morphological characteristics of the spine. Simultaneously, by incorporating temperature information from the point cloud, the temperature distribution of the target region is visualized, providing data support for subsequent posture analysis, health assessment, or assisted diagnosis.

[0068] It should be understood that the point cloud segmentation network is not limited to the example architecture described above, and any deep learning model capable of processing point cloud data and outputting point-by-point category labels can be applied to this step.

[0069] Example 2: A spinal region segmentation system based on multimodal 3D reconstruction Based on the same inventive concept, embodiments of the present invention also provide a spinal region segmentation system. This system includes the following modules: Multimodal data synchronous acquisition module: including single-node multimodal imaging module, electronically controlled turntable and synchronous trigger control unit, configured to synchronously acquire visible light image sequence and infrared thermal imaging image sequence of the back of the target object from different perspectives.

[0070] The registration and 3D reconstruction initialization module stores the intrinsic and extrinsic calibration parameters of the visible light camera and the infrared camera. It is configured to perform multimodal image registration and mapping based on the global optimal depth as described in step S2, and to generate the initial sparse point cloud of the back region of the target object using the structure-of-motion algorithm.

[0071] Multimodal 3D Gaussian Modeling and Joint Optimization Module: Configured to instantiate multimodal Gaussian units containing spatial location, covariance, color, opacity, and temperature attributes based on the sparse point cloud; calculate and render color maps and render temperature maps based on a differentiable rendering mechanism; and iteratively optimize the parameters of each Gaussian unit through backpropagation according to the photometric and temperature joint loss function based on foreground mask constraints described in step S4 to obtain a multimodal 3D Gaussian model.

[0072] The point cloud cleaning and noise reduction module based on covariance perception is configured to perform neighborhood consistency judgment on the Gaussian units in the multimodal 3D Gaussian model using the Mahalanobis statistical distance based on covariance perception described in step S5, remove isolated abnormal Gaussian units, and output the optimized Gaussian point cloud.

[0073] Multimodal feature fusion and point cloud segmentation module: It is preloaded with a pre-trained point cloud segmentation network, configured to extract spatial geometric features and physiological features from the optimized Gaussian point cloud, and use the attention mechanism to perform feature fusion, outputting a 3D segmentation mask of the spinal region.

[0074] Visualization output module: configured to combine the spinal region segmentation results with temperature attributes to generate a 3D visualization model with pseudo-color temperature mapping, and present it through a display device.

[0075] For details on the specific implementation of each module, please refer to the description of the corresponding steps in Example 1, which will not be repeated here.

[0076] Example 3: A computer-readable storage medium Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing computer program instructions thereon. When the computer program instructions are executed by a processor, they implement all or part of the steps of the spinal region segmentation method based on multimodal three-dimensional reconstruction as described in Embodiment 1.

[0077] Specifically, the storage medium includes, but is not limited to, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0078] The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), or a dedicated neural network acceleration chip, etc. When the processor loads and executes the computer program instructions, it can drive the corresponding hardware devices to perform the following operations: control the multimodal imaging module and the electronically controlled turntable to complete the synchronous acquisition of multi-view multimodal data (S1); perform cross-modal image registration and mapping based on the globally optimal depth (S2); generate a foreground mask and complete the initialization of three-dimensional reconstruction (S3); construct a multimodal three-dimensional Gaussian point cloud model with fused temperature attributes and perform joint optimization (S4); perform point cloud cleaning and noise reduction based on covariance perception using Mahalanobis statistical distance (S5); and perform point cloud segmentation and result output based on multimodal feature fusion (S6).

[0079] The computer-readable storage medium provided in this embodiment enables the method of the present invention to be delivered and deployed in the form of a software product, facilitating integration into existing medical imaging workstations or portable computing devices, thereby expanding the application scenarios of the present invention.

Claims

1. A method for spinal region segmentation based on multimodal 3D reconstruction, characterized in that, Includes the following steps: Step S1 Multimodal data acquisition: Simultaneously acquire the three-dimensional geometric information and temperature distribution information of the back surface of the target object, and constrain the temporal synchronization and spatial consistency of the data acquisition process; Step S2 Multimodal Image Registration and Mapping: Establish pixel-level mapping relationship between the two modal images through joint calibration of visible light camera and infrared camera; Step S3: Foreground mask extraction and 3D reconstruction initialization: Foreground segmentation is performed on the visible light image to obtain the foreground mask of the target region. The foreground mask is then used to filter the feature points extracted during the 3D reconstruction process, retaining only the feature points located in the foreground region to generate an initial sparse point cloud. Step S4 Multimodal 3D Gaussian Modeling and Joint Optimization: Based on the sparse point cloud, a multimodal 3D Gaussian point cloud model containing temperature attributes is constructed, and the Gaussian point cloud model is iteratively optimized through differentiable rendering based on the joint loss function with foreground mask constraints. Step S5 Point Cloud Optimization Processing: For the optimized Gaussian point cloud model, calculate the statistical distance based on the covariance matrix of each Gaussian unit, and remove abnormal Gaussian units according to the statistical distance to obtain point cloud data with consistent structure. Step S6: Multimodal Feature Fusion and Point Cloud Segmentation: The optimized multimodal Gaussian point cloud is input into a pre-trained point cloud segmentation network. By fusing spatial geometric features and temperature features, the three-dimensional segmentation result of the spine region is obtained.

2. The spinal region segmentation method based on multimodal three-dimensional reconstruction according to claim 1, characterized in that, In step S2, a cross-modal mapping model dependent on depth parameters is constructed based on the joint calibration parameters of the visible light camera and the infrared camera. The optimal depth parameter is determined through candidate search and multi-view statistical optimization within a preset depth range, so as to achieve pixel-level alignment of infrared temperature information to the visible light image coordinate system. The determination of the optimal global depth parameter includes: calculating the chamfer distance between the infrared projection edge and the visible light edge at each candidate depth and the gradient direction consistency error, and performing statistical optimization on the error of the multi-view image pair, and selecting the depth value that minimizes the comprehensive error as the optimal global depth.

3. The spinal region segmentation method based on multimodal three-dimensional reconstruction according to claim 2, characterized in that, In step S2, for each candidate depth, a comprehensive error evaluation index is constructed by calculating the spatial matching degree between the infrared edge and the visible light edge after projection transformation, as well as the consistency of their gradient directions. , in, For the set of visible light edges, For depth-based The set of infrared edges after projection mapping; and This corresponds to the gradient direction field; Used to evaluate the geometric distance error of the edge point set; Used to evaluate differences in gradient direction distribution, such as calculations based on cosine similarity or angular deviation; To balance the hyperparameters.

4. The spinal region segmentation method based on multimodal three-dimensional reconstruction according to claim 1, characterized in that, In step S4, each Gaussian unit in the multimodal 3D Gaussian point cloud model includes spatial position parameters, covariance parameters, color attributes, opacity parameters, and temperature attributes.

5. The spinal region segmentation method based on multimodal three-dimensional reconstruction according to claim 1, characterized in that, In step S4, the joint loss function is composed of a weighted sum of the photometric consistency loss based on the rendered color map and the real visible light image, and the temperature consistency loss based on the rendered temperature map and the real infrared thermal imaging image. The foreground mask generated in step S3 is introduced into the loss function as a weighting factor to suppress the interference of the background region.

6. The spinal region segmentation method based on multimodal three-dimensional reconstruction according to claim 5, characterized in that, In step S4, the joint loss function The following relationship must be satisfied: in, and These are real visible light and infrared images, respectively; and These are rendered visible light and infrared images, respectively; This represents the photometric error between real visible light and rendered visible light. , M is a preset weighting coefficient used to balance the contributions of color loss and temperature loss; 2D Foreground mask.

7. The spinal region segmentation method based on multimodal three-dimensional reconstruction according to claim 1, characterized in that, In step S5, the statistical distance is the Mahalanobis distance based on the covariance matrix of two Gaussian units, specifically defined as: in, , Let be the spatial mean vector of the Gaussian cell. , Let be the corresponding covariance matrix.

8. The spinal region segmentation method based on multimodal three-dimensional reconstruction according to claim 1, characterized in that, In step S6, the point cloud segmentation network uses a multi-branch structure to extract spatial geometric features and temperature features respectively, and outputs the segmentation mask of the spine region after weighted fusion of the features through an attention mechanism.

9. The spinal region segmentation method based on multimodal three-dimensional reconstruction according to claim 6, characterized in that, The point cloud segmentation network is built based on PointNet++, DGCNN, or PointTransFormer architecture.

10. A spinal region segmentation system based on multimodal 3D Gaussian representation, characterized in that, The system for performing the method according to any one of claims 1 to 9, the system comprising: Multimodal data synchronous acquisition module: It is equipped with a single-node multimodal imaging module, an electronically controlled turntable and a synchronous trigger control unit, which are used to synchronously acquire visible light image sequences and infrared thermal imaging image sequences of the back of the target object from multiple perspectives; The registration and 3D reconstruction initialization module stores the intrinsic and extrinsic calibration parameters of the visible light camera and the infrared camera. It is configured to perform multimodal image registration and mapping based on the globally optimal depth and to generate an initial sparse point cloud using the structure-of-motion algorithm. Multimodal 3D Gaussian Modeling and Joint Optimization Module: Configured to instantiate multimodal Gaussian units containing spatial location, covariance, color, opacity, and temperature attributes based on the sparse point cloud; calculate and render color maps and render temperature maps based on a differentiable rendering mechanism; and iteratively optimize the parameters of each Gaussian unit through backpropagation according to the joint loss function based on foreground mask constraints to obtain a multimodal 3D Gaussian model. Point cloud cleaning and noise reduction module: configured to perform neighborhood consistency judgment on Gaussian cells in the multimodal 3D Gaussian model using a statistical distance metric based on covariance perception, remove isolated abnormal Gaussian cells, and output optimized Gaussian point cloud; Multimodal feature fusion and point cloud segmentation module: It is preloaded with a pre-trained point cloud segmentation network, configured to extract spatial geometric features and physiological features from the optimized Gaussian point cloud, and use the attention mechanism to perform feature fusion, outputting a 3D segmentation mask of the spinal region. Visualization output module: configured to combine the spinal region segmentation results with temperature attributes to generate a three-dimensional visualization model, and present it through a display device.

Images