Medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning

Abstract

The application discloses a medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning, and comprises the following steps: step S1, acquiring and preprocessing in-situ medical hyperspectral image data; step S2, constructing a fractional Fourier transform representation; step S3, constructing a fractional domain spectrum consistent rotating sector mask strategy SCRSM; step S4, constructing fractional domain degradation input and cooperative reconstruction pre-training; step S5, constructing a shared fractional domain space-spectrum encoder and a multi-domain feature extraction module; step S6, constructing a fractional domain representation enhancement module and a learnable order multi-head fractional self-attention; step S7, constructing a segmentation head and a classification head and performing supervised fine-tuning; step S8, reasoning a model and outputting a result; the application solves the problem of how to effectively combine fractional domain representation, self-supervised mask learning and downstream segmentation / classification tasks.

Description

Technical Field

[0001] This invention relates to the field of medical hyperspectral image segmentation and classification, and particularly to a medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning. Background Technology

[0002] This invention belongs to the fields of medical image processing, in-situ medical hyperspectral imaging, and artificial intelligence. Specifically, it relates to a fractional domain mask representation learning method for in-situ medical hyperspectral data, which can be used for lesion region segmentation, tissue category discrimination, and related auxiliary diagnostic tasks in medical hyperspectral images acquired while the tissue is in its anatomical position, natural surface state, or clinical operation state.

[0003] In situ medical hyperspectral imaging is a non-contact imaging technique that combines spectral measurement with medical image acquisition without damaging tissue structure or relying on staining or contrast agents. Compared to traditional RGB images or single-modal medical images, in situ medical hyperspectral images not only preserve the spatial morphology, texture, and boundary contours of tissue surfaces, but also provide continuous spectral response information reflecting tissue biochemical composition, microstructural differences, and pathological changes. Therefore, this technology has significant value in in situ applications such as intraoperative navigation, bedside examination, organ or tissue surface analysis, lesion boundary identification, tissue viability assessment, and computer-aided diagnosis of medical images.

[0004] However, in-situ medical hyperspectral data is characterized by high-dimensional spectra, strong band correlations, complex spatial structures, subtle differences in lesions, and difficulties in obtaining annotations. Compared with standardized offline samples, in-situ acquisition is also susceptible to uneven illumination, specular reflection, tissue surface curvature, blood or body fluid obstruction, respiratory movements or slight displacement during handheld acquisition, changes in the effective field of view, and differences in cross-device band responses. In practical applications, the spectral responses of different tissues or lesion regions may be highly similar, while pathological changes often manifest as subtle perturbations in local texture, boundary morphology, and spectral curves. Relying solely on the original spatial domain features or conventional convolution features is often insufficient to fully characterize such non-stationary spatial-spectral coupling relationships.

[0005] Existing methods for medical hyperspectral image analysis generally include traditional feature extraction methods, supervised deep learning methods, and self-supervised representation learning methods. Traditional methods typically utilize principal component analysis, spectral angle mapping, handcrafted texture features, or machine learning classifiers for image analysis. Their advantage lies in their relative simplicity, but they often rely on manually designed features and struggle to adapt to the complex lighting, non-flat tissue surfaces, and high-dimensional spectral responses that combine to create diverse data distributions in in-situ medical settings. Supervised deep learning methods can learn discriminative features end-to-end and achieve good segmentation or classification results when sufficient labeled samples are available. However, fine labeling of in-situ medical hyperspectral data usually requires support from professional physicians, pathology results, or multimodal image registration, which is costly and involves limited sample sizes, thus restricting the model's generalization ability and robustness.

[0006] In recent years, self-supervised learning methods such as mask modeling have been used to reduce reliance on manual annotation. However, most methods primarily perform random pixel or block masking in the original image domain, making it difficult to simulate the structural loss patterns in in-situ medical hyperspectral images caused by illumination direction, tissue surface, local occlusion, specular reflection suppression, and cross-band consistent degradation. Furthermore, while direct modeling in the Fourier frequency domain can enhance frequency representation, it may weaken local spatial details, hindering the accurate recovery of lesion boundaries, tissue texture, and subtle transition regions in in-situ medical images.

[0007] The fractional Fourier transform (FFT) is a generalization of the classical Fourier transform in the fractional order sense, enabling the establishment of a continuous transition representation between the original domain and the frequency domain. Through adjustable fractional-order parameters, fractional-domain representation can simultaneously characterize local structure and frequency variations, providing a new technical foundation for modeling non-stationary spectral responses, tissue texture variations, boundary contours, and cross-band correlations in in-situ medical hyperspectral data caused by tissue surface inhomogeneities, acquisition angle variations, and spectral response perturbations. Therefore, effectively combining fractional-domain representation, self-supervised mask learning, and downstream segmentation / classification tasks is a pressing technical problem to be solved in the intelligent analysis of in-situ medical hyperspectral images.

[0008] This invention addresses the shortcomings of existing in-situ medical hyperspectral image analysis methods by proposing a medical hyperspectral image segmentation and classification method based on fractional-domain mask representation learning. This method constructs a mask degradation model with directional structure and spectral consistency in the fractional domain, performs self-supervised pre-training by co-reconstructing from the original and fractional domains, and then transfers the pre-trained spatial-spectral encoder to the in-situ medical hyperspectral image segmentation and classification task for supervised fine-tuning. This improves the representational ability, annotation efficiency, and generalization performance of medical hyperspectral image analysis under in-situ data conditions.

[0009] This invention mainly solves the following technical problems:

[0010] 1. To address the problem that existing in-situ medical hyperspectral image analysis methods rely excessively on manual annotation, pathological correspondence, or expert delineation, and reduce the need for large-scale, finely annotated data for model training;

[0011] 2. To address the problem that the original domain random masking method is difficult to characterize the specular reflection, local occlusion, illumination direction changes, and spatial-spectral coupling degradation features in in-situ acquisition;

[0012] 3. To address the problem that single spatial domain or single frequency domain modeling cannot simultaneously take into account local details of in-situ tissues, global frequency response, surface texture changes, and cross-band correlations;

[0013] 4. To address the problem that conventional self-attention mechanisms mainly calculate correlations in the original domain and are insufficient to fully characterize the non-stationary spatial-spectral dependence in in-situ medical hyperspectral data;

[0014] 5. Address the issues of insufficient model transferability, unstable weak boundary recognition, difficulty in locating low-contrast lesions, and weak ability to discriminate fine-grained tissue differences in downstream segmentation and classification tasks of in-situ medical hyperspectral images.

[0015] The purpose of this invention is to provide a unified medical hyperspectral image learning framework that enables models to learn robust spatial-spectral representations under unlabeled or minimally labeled conditions, and to simultaneously adapt to pixel-level segmentation and image-level classification tasks during supervised fine-tuning. By introducing fractional domain masks, self-supervised collaborative reconstruction, multi-domain feature extraction, and a learnable fractional attention mechanism, this invention can improve the accuracy of lesion region localization, tissue boundary restoration, and category discrimination under in-situ conditions. Summary of the Invention

[0016] This invention addresses the shortcomings of existing technologies by providing a medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning, thus solving the problem of how to effectively combine fractional domain representation, self-supervised mask learning, and downstream segmentation / classification tasks.

[0017] The specific technical solution of this invention is as follows:

[0018] A medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning includes the following steps:

[0019] Step S1: Acquire and preprocess in-situ medical hyperspectral image data;

[0020] Step S2: Construct the fractional Fourier transform representation;

[0021] Step S3: Construct a fractional-domain spectrally consistent rotating sector mask strategy (SCRSM);

[0022] Step S4: Construct fractional domain degenerate input and co-reconstruction pre-training;

[0023] Step S5: Construct a shared fractional-domain spatial-spectral encoder and a multi-domain feature extraction module;

[0024] Step S6: Construct a fractional domain representation enhancement module and a learnable multi-head fractional self-attention module;

[0025] Step S7: Construct the segmentation head and classification head and perform supervised fine-tuning;

[0026] Step S8: Inference Model and Output Results.

[0027] Preferably, step S1 includes the following steps:

[0028] Step S11: Obtain the in-situ medical hyperspectral image data to be analyzed, and establish the corresponding dataset according to the specific task;

[0029] Step S12: The collection objects may include tissues exposed during the operation, tissues on the body surface or organ surface, tissue areas that maintain their original structural relationships under endoscopic or microscopic vision, and tissue samples that retain their original surface state after being excised.

[0030] Step S13: During acquisition, record the device's wavelength range, center wavelength, exposure time, light source conditions, acquisition distance, white board / dark field reference, and case or site information;

[0031] Step S14: For segmentation tasks, the dataset includes in-situ hyperspectral images and their corresponding pixel-level tissue or lesion annotations; for classification tasks, the dataset includes in-situ hyperspectral images, regions of interest and their corresponding image-level or region-level category annotations.

[0032] Step S15: The hyperspectral image can be represented as , where S represents the number of spectral bands, and H and W represent the image height and width, respectively;

[0033] Step S16: Preprocess the acquired in-situ medical hyperspectral images;

[0034] Step S17: For in-situ data obtained from different cases, different collection batches, or different devices, further processing such as case-based classification, batch normalization, and ROI consistency can be performed to reduce data leakage and collection bias.

[0035] Step S18: After preprocessing, an unlabeled in-situ hyperspectral image set for self-supervised pre-training and a segmentation or classification labeled dataset for supervised fine-tuning are formed.

[0036] Preferably, step S2 includes the following steps:

[0037] Step S21: Perform fractional Fourier transform modeling on the in-situ medical hyperspectral image;

[0038] Step S22: Let the fractional-order parameter be... Its corresponding rotation angle is ,when When, the fractional Fourier transform degenerates into the identity transform; when At that time, it degenerates into a standard Fourier transform, which can be adjusted by... The model can obtain a continuous representation between the original domain and the frequency domain;

[0039] Step S23: For spatial structures in in situ medical hyperspectral images, a two-dimensional fractional Fourier transform (FRFT2D) can be performed along the two-dimensional spatial direction to enhance the expression of natural tissue boundaries, surface textures, organ or mucosal surface morphological changes, and transition areas of local lesions.

[0040] Step S24: For spectral sequence or channel features, a one-dimensional fractional Fourier transform (FRFT1D) can be performed along the spectral dimension to enhance the modeling ability for cross-band correlation, reflectance curve shape changes, and non-stationary spectral responses.

[0041] Preferably, the fractional domain masking process in step S3 can be represented as follows:

[0042]

[0043] in, For fractional-order parameters; This indicates the position of the c-th spectral band in the fractional domain space before the mask is applied. The value at that location, This indicates the corresponding fractional range value after applying SCRSM; This indicates that the two-dimensional sector mask M is located at position The values ​​at the specified locations are: c is the spectral band index, C is the total number of bands, and u and v are spatial location indices.

[0044] Preferably, step S4 includes the following steps:

[0045] Step S41: After completing the fractional field mask, represent the masked fractional field. The inverse two-dimensional fractional Fourier transform maps the image back to the original image domain, forming a degraded input. This process can be represented as

[0046]

[0047] in, This represents the degraded input obtained after fractional domain masking and inverse transformation back to the original image domain; This represents the original input hyperspectral image; Represents fractional parameters The two-dimensional fractional Fourier transform of the following This represents the corresponding inverse two-dimensional fractional Fourier transform; This indicates that a spectrally consistent rotating sector mask operation is performed on the fractional domain representation;

[0048] Step S42: Degenerate input A shared fractional-domain spatial-spectral encoder is input to extract deep spatial-spectral features; subsequently, a reconstruction decoder is connected to gradually restore the spatial resolution and spectral response, and the reconstruction result is output. The calculation formula can be expressed as:

[0049]

[0050] in, Indicates degenerate input; This represents a shared fractional-domain spatial-spectral encoder used to extract deep spatial-spectral features; Indicates the reconstruction decoder; This represents the reconstructed hyperspectral image obtained by the model.

[0051] Step S43: The pre-training phase employs joint constraints between the original domain and the score domain;

[0052] Step S44: During training, fractional-order parameters A phased sampling strategy can be adopted.

[0053] Preferably, step S5 includes the following steps:

[0054] Step S51: The shared fractional domain spatial-spectral encoder is used for both self-supervised pre-training and supervised fine-tuning stages, and its core component is the multi-domain feature extraction module.

[0055] Step S52: In view of the characteristics of in-situ medical hyperspectral data, which simultaneously contain tissue surface morphology, local texture, reflectance curve and cross-band response differences, the multi-domain feature extraction module performs collaborative modeling through original domain multi-scale convolution branch, two-dimensional fractional domain spatial branch and one-dimensional fractional domain spectral branch, thereby obtaining complementary spatial-spectral features.

[0056] Step S53: In the original domain multi-scale convolution branch, the input features are divided into multiple subgroups along the channel dimension, and multi-scale local spatial responses are extracted by convolution operations with different kernel sizes. Then, channel fusion is performed by pointwise convolution.

[0057] Step S54: In the two-dimensional fractional domain spatial branch, perform a two-dimensional fractional Fourier transform on the input features along the spatial dimension to obtain the complex-valued fractional domain representation. Concatenate the real and imaginary parts and integrate the features through pointwise convolution. Then, return to the original feature space through an inverse two-dimensional fractional Fourier transform.

[0058] Step S55: In the one-dimensional fractional domain spectral branch, perform a one-dimensional fractional Fourier transform on the input features along the spectral or channel correlation dimension, and fuse the real and imaginary parts of the complex-valued features, and then obtain the spectral enhancement features through inverse transform;

[0059] Step S56: To avoid inconsistencies in response scales between different branches, the multi-domain feature extraction module normalizes the output of the fractional domain branches and introduces a learnable scaling factor to adaptively adjust the contribution of each branch.

[0060] Step S57: Finally, the three types of branch features are spliced ​​and fused, and the module output is obtained through residual connection and nonlinear activation.

[0061] Preferably, the calculation formula for step S6 is:

[0062]

[0063] In the formula, h represents the attention head index; This represents the output of the h-th attention head; and These represent the query features respectively. Bond features In fractional order The result of the one-dimensional fractional Fourier transform; The magnitude of the complex-valued feature. Represents a logarithmic mapping. A tiny positive number introduced to prevent numerical instability; T represents matrix transpose; d represents the feature dimension of a single attention head, used to scale attention relevance; This represents the value feature of the h-th attention head; Used to normalize the relevance weights.

[0064] Preferably, step S7 includes the following steps:

[0065] Step S71: In the segmentation task, a decoder-based pixel-level prediction head is adopted. This prediction head receives multi-layer features processed by a fractional-domain spatial-spectral encoder and a fractional-domain representation enhancement module, gradually fuses multi-scale skip connection features from the encoder, and restores spatial resolution through upsampling, finally outputting pixel-level category predictions in in-situ medical hyperspectral images.

[0066] Step S72: To improve the effectiveness of skip connection features, a lightweight skip feature refinement module is introduced before fusing skip features with decoding features. This module consists of pointwise convolution, batch normalization, depthwise convolution, and efficient channel attention, and can be represented as:

[0067]

[0068] in, The encoder skip connection features represent the input; This represents pointwise convolution, used for channel mapping and fusion; Indicates batch normalization; This represents depthwise convolution, used to extract lightweight local spatial information; This indicates efficient channel attention, used to enhance critical channel response; This represents the output features after processing by the jump feature refinement module;

[0069] Step S73: The supervision loss for the segmentation task includes cross-entropy loss and Dice loss. The final segmentation loss calculation formula is as follows:

[0070]

[0071] Among them, cross-entropy loss Dice loss is used for pixel-level category constraints. Used to constrain the overlap between the predicted region and the true region.

[0072] Preferably, step S8 includes the following steps:

[0073] Step S81: In the application phase, the in-situ medical hyperspectral image to be analyzed is processed according to the same preprocessing procedure as in the training phase, and then input into the trained model.

[0074] Step S82: For segmentation tasks, the model outputs the tissue category or lesion category corresponding to each pixel, and generates a segmentation mask, color visualization results, and quantitative analysis indicators. For classification tasks, the model outputs the category probability of the image or region of interest and the final category discrimination result.

[0075] The beneficial effects of the medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning in this invention are as follows:

[0076] 1. This invention introduces fractional Fourier transform into in-situ medical hyperspectral image representation learning, establishing a continuous fractional domain representation between the original domain and the frequency domain, which can take into account both local tissue details and frequency response information.

[0077] 2. The Spectral Consistent Rotating Sector Mask Strategy (SCRSM) designed in this invention directly acts on the fractional domain, which can form missing modes with directional structure, random rotational perturbation and cross-band consistency. This is more in line with the modeling requirements of directional illumination changes, curved surface structures, local occlusion or reflection and spatial-spectral coupling features in in-situ acquisition.

[0078] 3. This invention employs self-supervised pre-training by combining original domain reconstruction, fractional domain missing region consistency, and edge consistency constraints. This enables full utilization of the intrinsic structural information of in-situ medical hyperspectral images under unlabeled or minimally labeled conditions, reducing reliance on manual annotation.

[0079] 4. The multi-domain feature extraction module in this invention performs multi-domain collaborative feature extraction through original domain multi-scale convolution, two-dimensional fractional domain spatial modeling and one-dimensional fractional domain spectral modeling, which can simultaneously enhance the expression of natural tissue texture, lesion boundaries, reflectance curve differences, spectral correlation and fine-grained pathological differences.

[0080] 5. The learnable order multi-head fractional self-attention in this invention introduces learnable fractional order parameters for different attention heads, enabling the attention mechanism to adaptively calculate correlations in multiple fractional domains, thereby improving the ability to model non-stationary spatial-spectral dependence under in-situ acquisition conditions.

[0081] 6. This invention can be adapted to in-situ medical hyperspectral image segmentation and classification tasks based on the same pre-trained encoder, and has good task versatility, transferability and engineering deployment value.

[0082] 7. This invention can improve the lesion boundary recovery, tissue region segmentation, and image-level or region-level category discrimination in in situ data, providing a more stable and reliable technical solution for medical hyperspectral image-assisted diagnosis. Attached Figure Description

[0083] The present invention may provide the following figures, which can be further drawn into formal flowcharts or structural block diagrams according to the actual patent agency requirements:

[0084] Figure 1 This is a flowchart illustrating the overall process of the medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning, as described in this invention.

[0085] Figure 2 This is a flowchart illustrating the overall process of fractional domain masking and collaborative reconstruction of in-situ medical hyperspectral images during the self-supervised pre-training phase of this invention.

[0086] Figure 3 This is the overall flowchart of the supervised fine-tuning stage of the present invention.

[0087] Figure 4This is a schematic diagram of the shared fractional domain spatial-spectral encoder and multi-domain feature extraction module of the present invention.

[0088] Figure 5 This is a schematic diagram of the learnable multi-head fractional self-attention mechanism structure of the present invention. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0090] This invention provides a medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning, comprising steps such as in-situ data preparation, fractional domain self-supervised pre-training, supervised fine-tuning, segmentation / classification inference, and result output. This method, named FracMask-Former, has the following core idea: First, it uses fractional Fourier transform to map the in-situ medical hyperspectral image to a continuous fractional domain between the original domain and the frequency domain, constructing a mask degradation model with directional structure and spectral consistency within this domain. Then, it performs self-supervised learning through a shared fractional domain spatial-spectral encoder and reconstruction decoder, subjecting the model to constraints of original domain reconstruction, fractional domain missing region consistency, and edge structure consistency. Finally, it loads the pre-trained encoder into a downstream task, combining it with a fractional domain representation enhancement module and a task head to complete medical hyperspectral image segmentation or classification.

[0091] Step 1: Acquisition and Preprocessing of In-situ Medical Hyperspectral Image Data

[0092] Acquire in-situ medical hyperspectral image data to be analyzed and establish corresponding datasets according to specific tasks. Acquisition objects may include intraoperatively exposed tissues, tissues on the body surface or organ surface, tissue regions that maintain their original structural relationships under endoscopic or microscopic views, and tissue samples that retain their in-situ surface state after ex vivo. During acquisition, record the device's wavelength range, center wavelength, exposure time, light source conditions, acquisition distance, whiteboard / dark field reference, and case or site information. For segmentation tasks, the dataset includes in-situ hyperspectral images and their corresponding pixel-level tissue or lesion annotations; for classification tasks, the dataset includes in-situ hyperspectral images, regions of interest, and their corresponding image-level or region-level category annotations. Hyperspectral images can be represented as... Where S represents the number of spectral bands, and H and W represent the image height and width, respectively.

[0093] The acquired in-situ medical hyperspectral images undergo preprocessing, including but not limited to dark white plate correction, reflectance conversion, removal of bad or low signal-to-noise ratio bands, spatial cropping, size uniformity, spectral normalization, specular reflection and saturation pixel suppression, background / instrument / blood or body fluid interference region masking, spatial registration and motion correction, training / validation / test set partitioning, and data augmentation. For in-situ data obtained from different cases, different acquisition batches, or different devices, further processing can be performed by case partitioning, batch normalization, and ROI consistency to reduce data leakage and acquisition bias. The preprocessed data forms an unlabeled in-situ hyperspectral image set for self-supervised pre-training, and a segmentation or classification labeled dataset for supervised fine-tuning.

[0094] Step 2: Construct the fractional Fourier transform representation

[0095] Fractional Fourier transform modeling is performed on in-situ medical hyperspectral images. Let the fractional-order parameter be... Its corresponding rotation angle is .when When, the fractional Fourier transform degenerates into the identity transform; when At that time, it degenerates into a standard Fourier transform. By adjusting... The model can obtain a continuous representation between the original domain and the frequency domain.

[0096] For spatial structures in in situ medical hyperspectral images, a two-dimensional fractional Fourier transform (FRFT2D) can be performed along the two-dimensional spatial direction to enhance the expression of natural tissue boundaries, surface textures, morphological changes of organ or mucosal surfaces, and transitional zones of local lesions. For spectral sequences or channel features, a one-dimensional fractional Fourier transform (FRFT1D) can be performed along the spectral dimension to enhance the modeling ability of cross-band correlations, changes in reflectance curve shape, and non-stationary spectral responses.

[0097] Step 3: Construct a fractional-domain spectrally consistent rotating sector mask strategy (SCRSM)

[0098] The SCRSM masking process during the self-supervised pre-training phase is summarized as follows: First, the input in-situ medical hyperspectral image X is mapped to a fractional domain representation through a two-dimensional fractional Fourier transform. Subsequently, a spectrally consistent rotating sector mask is constructed in this fractional domain. The specific mask generation process is as follows.

[0099] In the fractional-domain plane, with the center point as the origin of polar coordinates, the polar angle of each position relative to the center point is calculated. To avoid introducing prior biases through a fixed-direction mask and to simulate the differences in directional response caused by changes in illumination direction, probe angle, and tissue surface curvature during in-situ acquisition, a global rotation angle is randomly sampled during each forward propagation to perturb the overall angle distribution. Subsequently, an angle folding operation is performed to map the complete angle space to the basic angle interval, forming a fan-shaped distribution with a multi-symmetric structure. When a folded corner at a certain position falls within the preset fan-shaped width, it is set as the mask position; otherwise, it is set as a reserved position.

[0100] In the fractional-domain plane, with the center point as the origin of polar coordinates, the polar angle of each position relative to the center point is calculated. To avoid introducing prior biases through a fixed-direction mask and to simulate the differences in directional response caused by changes in illumination direction, probe angle, and tissue surface curvature during in-situ acquisition, a global rotation angle is randomly sampled during each forward propagation to perturb the overall angle distribution. Subsequently, an angle folding operation is performed to map the complete angle space to the basic angle interval, forming a fan-shaped distribution with a multi-symmetric structure. When a folded corner at a certain position falls within the preset fan-shaped width, it is set as the mask position; otherwise, it is set as a reserved position.

[0101] The generated two-dimensional sector mask middle, This indicates that the position of the fraction field is preserved. This indicates that the location is masked. To maintain spectral consistency of in-situ medical hyperspectral data at the same tissue location, this two-dimensional sector mask is broadcast along the spectral dimension to all bands, allowing the same spatial location to share a consistent mask pattern across different bands. The fractional-domain masking process can be represented as:

[0102] in, For fractional-order parameters; This indicates the position of the c-th spectral band in the fractional domain space before the mask is applied. The value at that location, This indicates the corresponding fractional range value after applying SCRSM; This indicates that the two-dimensional sector mask M is located at position The values ​​at the specified locations are: c is the spectral band index, C is the total number of bands, and u and v are spatial location indices.

[0103] The aforementioned SCRSM enables the model to learn more robust spatial-spectral representations by addressing directional, structural, and spectral consistency loss patterns that are more consistent with those encountered in in-situ acquisitions, such as changes in illumination direction, specular reflection suppression, tissue surface structure, and local occlusion, during the pre-training process.

[0104] Step 4: Fractional Domain Degenerate Input Construction and Co-reconstruction Pre-training

[0105] After completing the fractional field mask, the fractional field representation to be masked The inverse two-dimensional fractional Fourier transform maps the image back to the original image domain, forming a degraded input. This process can be represented as

[0106]

[0107] in, This represents the degraded input obtained after fractional domain masking and inverse transformation back to the original image domain; This represents the original input hyperspectral image; Represents fractional parameters The two-dimensional fractional Fourier transform of the following This represents the corresponding inverse two-dimensional fractional Fourier transform; This indicates that a spectrally consistent rotating sector mask operation is performed on the fractional domain representation;

[0108] Degenerate input A shared fractional-domain spatial-spectral encoder is input to extract deep spatial-spectral features; subsequently, a reconstruction decoder is connected to gradually restore the spatial resolution and spectral response, and the reconstruction result is output. The calculation formula can be expressed as:

[0109]

[0110] in, Indicates degenerate input; This represents a shared fractional-domain spatial-spectral encoder used to extract deep spatial-spectral features; Indicates the reconstruction decoder; This represents the reconstructed hyperspectral image obtained by the model.

[0111] The pre-training phase employs joint constraints from the original domain and the score domain; firstly, the reconstruction results are... Mapping back to the fractional domain, we get and in the mask missing area First, the fractional domain consistency loss is calculated to enable the model to focus on recovering the masked discriminative fractional domain components. Second, an L1 reconstruction loss constraint is applied in the original domain. Consistency with the original input X is assessed to recover the basic spatial structure, reflectance distribution, and spectral response under in-situ acquisition. Furthermore, an edge consistency loss is introduced to constrain the consistency of the first-order differences in the horizontal and vertical directions, enhancing the recovery of natural tissue boundaries, lesion transition zones, and texture variations. The total pre-training loss can be expressed as:

[0112]

[0113] in, This represents the consistency loss due to missing regions in the fractional domain. Indicates the loss during original domain reconstruction. This represents the edge consistency loss. and These are the corresponding weighting coefficients.

[0114] Fractional parameters during training A phased sampling strategy can be adopted. In the initial stage of training, sampling should begin from a narrower range. To ensure optimization stability, the sampling range is gradually expanded as training progresses, thereby improving the model's adaptability to different score domain representations and different missing patterns.

[0115] Step 5: Construct a shared fractional-domain spatial-spectral encoder and a multi-domain feature extraction module

[0116] The shared fractional-domain spatial-spectral encoder is used for both self-supervised pre-training and supervised fine-tuning stages, with its core component being the multi-domain feature extraction module. Addressing the characteristics of in-situ medical hyperspectral data, which simultaneously includes tissue surface morphology, local texture, reflectance curves, and cross-band response differences, the multi-domain feature extraction module collaboratively models complementary spatial-spectral features through multi-scale convolutional branches in the original domain, two-dimensional fractional-domain spatial branches, and one-dimensional fractional-domain spectral branches.

[0117] In the original domain multi-scale convolution branch, the input features are divided into multiple subgroups along the channel dimension, and convolution operations with different kernel sizes are used to extract multi-scale local spatial responses. Then, channel fusion is performed through pointwise convolution. This branch is mainly used to preserve local texture, edge response, surface morphology, and details of small lesion structures.

[0118] In the two-dimensional fractional domain branch, a two-dimensional fractional Fourier transform is performed on the input features along the spatial dimension to obtain a complex-valued fractional domain representation. The real and imaginary parts are concatenated and the features are integrated through pointwise convolution, and then the inverse two-dimensional fractional Fourier transform is used to return to the original feature space. This branch is used to enhance the ability to express natural boundaries of organizations, surface texture changes, local low-contrast structures, and non-stationary spatial structures.

[0119] In the one-dimensional fractional-domain spectral branch, a one-dimensional fractional Fourier transform is performed on the input features along the spectral or channel correlation dimension. The real and imaginary parts of the complex-valued features are then fused, and an inverse transform is performed to obtain the spectral enhancement features. This branch is used to model inter-band correlations, differences in reflectance curve shape, and potential spectral differences related to changes in blood oxygenation, water content, or tissue composition.

[0120] To avoid inconsistencies in response scales between different branches, the multi-domain feature extraction module normalizes the output of the fractional domain branch and introduces a learnable scaling factor to adaptively adjust the contribution of each branch. Finally, the features of the three types of branches are concatenated and fused, and the module output is obtained through residual connections and nonlinear activation.

[0121] Step 6: Construct a fractional domain representation enhancement module and a learnable multi-head fractional self-attention module.

[0122] During the supervised fine-tuning phase, to enhance the high-level discriminative relation modeling capability, a fractional domain representation enhancement module is introduced after the shared encoder. The core unit of the fractional domain representation enhancement module is a learnable-order multi-head fractional self-attention mechanism.

[0123] For input token features First, the query Q, key K, and value V are obtained through linear mapping, and then divided into multiple attention heads. For the h-th attention head, a learnable fractional-order parameter is assigned. and to and Performing a one-dimensional fractional Fourier transform yields and Since the fractional domain features are represented by complex values, their magnitudes are used and logarithmic mapping is performed to improve numerical stability, and then the attention correlation is calculated. The output of the h-th attention head can be expressed as:

[0124]

[0125] In the formula, h represents the attention head index; This represents the output of the h-th attention head; and These represent the query features respectively. Bond features In fractional order The result of the one-dimensional fractional Fourier transform; The magnitude of the complex-valued feature. Represents a logarithmic mapping. A tiny positive number introduced to prevent numerical instability; T represents matrix transpose; d represents the feature dimension of a single attention head, used to scale attention relevance; This represents the value feature of the h-th attention head; Used to normalize the relevance weights.

[0126] Learnable-order multi-head fractional self-attention enables the model to adaptively model inter-regional structural associations, cross-band dependencies, and fine-grained tissue differences across multiple fractional domains by assigning learnable fractional orders to different attention heads. Compared to conventional self-attention, which only computes similarity in the original domain, this mechanism is better able to capture non-stationary spatial-spectral dependencies in in-situ medical hyperspectral data.

[0127] Step 7: Construct the segmentation head and classification head and perform supervised fine-tuning.

[0128] In the segmentation task, a decoder-based pixel-level prediction head is employed. This prediction head receives multi-layer features processed by a fractional-domain spatial-spectral encoder and a fractional-domain representation enhancement module, progressively fuses multi-scale skip connection features from the encoder, and restores spatial resolution through upsampling, ultimately outputting pixel-level category predictions in in-situ medical hyperspectral images.

[0129] To improve the effectiveness of skip connection features, a lightweight skip feature refinement module is introduced before fusing skip features with decoding features. This module consists of pointwise convolution, batch normalization, depthwise convolution, and efficient channel attention, and can be represented as:

[0130] in, The encoder skip connection features represent the input; This represents pointwise convolution, used for channel mapping and fusion; Indicates batch normalization; This represents depthwise convolution, used to extract lightweight local spatial information; This indicates efficient channel attention, used to enhance critical channel response; This represents the output features after processing by the jump feature refinement module;

[0131] The supervised loss for the segmentation task includes cross-entropy loss and Dice loss. The final segmentation loss is calculated using the following formula:

[0132]

[0133] Among them, cross-entropy loss Dice loss is used for pixel-level category constraints. Used to constrain the overlap between the predicted region and the true region.

[0134] In classification tasks, image-level or region-of-interest (ROI) level classification heads are employed. High-level features are first enhanced by convolution and global average pooling to obtain global semantic representations, which are then input into a multilayer perceptron to obtain image-level or region-level class probabilities. Cross-entropy loss is used for supervised optimization in classification tasks.

[0135] Step 8: Model Inference and Result Output

[0136] In the application phase, the in-situ medical hyperspectral images to be analyzed are processed according to the same preprocessing procedure as in the training phase and then input into the trained model. For segmentation tasks, the model outputs the tissue category or lesion category corresponding to each pixel and generates segmentation masks, color visualization results, and quantitative analysis indicators; for classification tasks, the model outputs the category probability of the image or region of interest and the final category discrimination result.

[0137] This method can be deployed in in-situ medical hyperspectral acquisition systems, medical hyperspectral image analysis workstations, intraoperative auxiliary diagnostic platforms, bedside detection equipment, or offline medical image processing systems to assist doctors in lesion localization, tissue identification, and category discrimination.

[0138] Example 1: In-situ medical hyperspectral imaging lesion region segmentation

[0139] This embodiment addresses the task of lesion or tissue region segmentation in in-situ medical hyperspectral images. First, hyperspectral images of the tissue to be analyzed are acquired using medical hyperspectral acquisition equipment under intraoperative, bedside, endoscopic, microscopic, or organ surface observation conditions. Pixel-level annotations are then provided by professionals, taking into account the surgical field of view, pathological results, postoperative diagnosis, or multimodal image registration results. The acquired data undergoes dark whiteboard correction, reflectivity conversion, band filtering, specular reflection and saturation region processing, background and instrument region masking, spatial registration, normalization, cropping, and size standardization to form training, validation, and test sets divided by case or acquisition batch.

[0140] In the self-supervised pre-training phase, only the in-situ hyperspectral images from the training set are used as input, without pixel-level labels. A two-dimensional fractional Fourier transform is performed on the input images, and a rotating sector-shaped missing pattern is constructed in the fractional domain using SCRSM to simulate structural degradation caused by directional illumination changes, local occlusion, or reflection suppression during in-situ acquisition. The degraded input is obtained through inverse fractional Fourier transform and fed into the fractional-domain spatial-spectral encoder and reconstruction decoder for recovery training. During training, the model is jointly optimized using fractional-domain missing region consistency loss, original domain reconstruction loss, and edge consistency loss, enabling the encoder to learn a general spatial-spectral representation related to tissue structure, boundary changes, and spectral response.

[0141] During the supervised fine-tuning phase, pre-trained fractional-domain spatial-spectral encoder weights are loaded and connected to the fractional-domain representation enhancement module and the segmentation decoding head. The segmentation head fuses multi-scale encoded features and introduces a skip feature refinement module consisting of PWConv, BN, DWConv, and ECA at skip connections to enhance key channel responses and boundary details. The model employs joint optimization using cross-entropy loss and Dice loss. After training, the model outputs pixel-level segmentation results when a test in-situ hyperspectral image is input, generating lesion region masks, tissue region visualizations, and quantitative evaluation metrics.

[0142] Through this embodiment, the method can improve the ability to restore lesion boundaries and the stability of region segmentation in in situ medical hyperspectral images with fewer labeled samples. It is applicable to scenarios such as in situ medical tissue segmentation, lesion localization, intraoperative margin judgment and bedside auxiliary analysis.

[0143] Example 2: In-situ Medical Hyperspectral Imaging Tissue Classification

[0144] This embodiment addresses the task of classifying in-situ medical hyperspectral images or regions of interest (ROIs) labeled by doctors. First, in-situ hyperspectral images of different tissue types, pathological states, or anatomical locations are acquired, and image-level or region-level category labels are established based on pathological diagnoses, expert annotations, or clinical records. The hyperspectral data undergoes dark whiteboard correction, reflectivity conversion, bad band removal, spectral normalization, ROI extraction, reflection and background region processing, and training / validation / testing partitioning to obtain a standard dataset for model training.

[0145] In the pre-training phase, similar to Example 1, fractional-domain mask self-supervised learning is performed using unlabeled in-situ hyperspectral images. Spectral consistent directional missing patterns are constructed using SCRSM, and the fractional-domain spatial-spectral encoder is trained by co-reconstructing the target using the original domain and fractional domain. This enables the model to acquire general expressive capabilities for different tissue reflectance responses, cross-band correlations, surface spatial structure differences, and in-situ acquisition perturbations.

[0146] In the supervised classification phase, pre-trained fractional-domain spatial-spectral encoder weights are loaded and connected to a fractional-domain representation enhancement module and an image-level or region-level classification head. The fractional-domain representation enhancement module performs learnable-order multi-head fractional self-attention modeling on high-level features. Different attention heads can calculate correlations at different fractional orders, thereby enhancing the ability to discriminate non-stationary spectral differences and inter-regional structural relationships in in-situ data. The classification head outputs class probabilities through convolutional enhancement, global average pooling, and a multilayer perceptron, and is optimized using cross-entropy loss during training.

[0147] After training, the model is input into the in-situ medical hyperspectral image or region of interest to be classified, and it outputs the corresponding category probability and final category label. This embodiment can be used for tasks such as tissue type identification, lesion / non-lesion discrimination, tissue state assessment, and pathological category auxiliary analysis in in-situ medical hyperspectral images.

[0148] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. For those skilled in the art, equivalent substitutions, combinations, or improvements made to the fractional sampling range, mask ratio, number of network layers, loss weights, task header structure, data preprocessing methods, or deployment platforms without departing from the core idea of ​​the present invention should all fall within the scope of protection of the present invention.

Claims

1. A medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning, characterized in that, Includes the following steps: Step S1: Acquire and preprocess in-situ medical hyperspectral image data; Step S2: Construct the fractional Fourier transform representation; Step S3: Construct a fractional-domain spectrally consistent rotating sector mask strategy (SCRSM); Step S4: Construct fractional domain degenerate input and co-reconstruction pre-training; Step S5: Construct a shared fractional-domain spatial-spectral encoder and a multi-domain feature extraction module; Step S6: Construct a fractional domain representation enhancement module and a learnable order multi-head fractional self-attention; Step S7: Construct the segmentation head and classification head and perform supervised fine-tuning; Step S8: Inference Model and Output Results.

2. The medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning according to claim 1, characterized in that, Step S1 includes the following steps: Step S11: Obtain the in-situ medical hyperspectral image data to be analyzed, and establish the corresponding dataset according to the specific task; Step S12: The collection objects may include tissues exposed during the operation, tissues on the body surface or organ surface, tissue areas that maintain their original structural relationships under endoscopic or microscopic vision, and tissue samples that retain their original surface state after being excised. Step S13: During acquisition, record the device's wavelength range, center wavelength, exposure time, light source conditions, acquisition distance, white board / dark field reference, and case or site information; Step S14: For segmentation tasks, the dataset includes in-situ hyperspectral images and their corresponding pixel-level tissue or lesion annotations; for classification tasks, the dataset includes in-situ hyperspectral images, regions of interest and their corresponding image-level or region-level category annotations. Step S15: The hyperspectral image can be represented as , where S represents the number of spectral bands, and H and W represent the image height and width, respectively; Step S16: Preprocess the acquired in-situ medical hyperspectral images; Step S17: For in-situ data obtained from different cases, different collection batches, or different devices, further processing such as case-based classification, batch normalization, and ROI consistency can be performed to reduce data leakage and collection bias. Step S18: After preprocessing, an unlabeled in-situ hyperspectral image set for self-supervised pre-training and a segmentation or classification labeled dataset for supervised fine-tuning are formed.

3. The medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning according to claim 1, characterized in that, Step S2 includes the following steps: Step S21: Perform fractional Fourier transform modeling on the in-situ medical hyperspectral image; Step S22: Let the fractional-order parameter be... Its corresponding rotation angle is ,when When, the fractional Fourier transform degenerates into the identity transform; when At that time, it degenerates into a standard Fourier transform, which can be adjusted by... The model can obtain a continuous representation between the original domain and the frequency domain; Step S23: For spatial structures in in situ medical hyperspectral images, a two-dimensional fractional Fourier transform (FRFT2D) can be performed along the two-dimensional spatial direction to enhance the expression of natural tissue boundaries, surface textures, organ or mucosal surface morphological changes, and transition areas of local lesions. Step S24: For spectral sequence or channel features, a one-dimensional fractional Fourier transform (FRFT1D) can be performed along the spectral dimension to enhance the modeling ability for cross-band correlation, reflectance curve shape changes, and non-stationary spectral responses.

4. The medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning according to claim 1, characterized in that, The fractional field masking process in step S3 can be represented as follows: ， in, For fractional-order parameters; This indicates the position of the c-th spectral band in the fractional domain space before the mask is applied. The value at that location, This indicates the corresponding fractional range value after applying SCRSM; This indicates that the two-dimensional sector mask M is located at position The values ​​at the specified locations are: c is the spectral band index, C is the total number of bands, and u and v are spatial location indices.

5. The medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: After completing the fractional field mask, represent the masked fractional field. The inverse two-dimensional fractional Fourier transform maps the image back to the original image domain, forming a degraded input. This process can be represented as ， in, This represents the degraded input obtained after fractional domain masking and inverse transformation back to the original image domain; This represents the original input hyperspectral image; Represents fractional parameters The two-dimensional fractional Fourier transform of the following This represents the corresponding inverse two-dimensional fractional Fourier transform; This indicates that a spectrally consistent rotating sector mask operation is performed on the fractional domain representation; Step S42: Degenerate input A shared fractional-domain spatial-spectral encoder is input to extract deep spatial-spectral features; subsequently, a reconstruction decoder is connected to gradually restore the spatial resolution and spectral response, and the reconstruction result is output. The calculation formula can be expressed as: ， in, Indicates degenerate input; This represents a shared fractional-domain spatial-spectral encoder used to extract deep spatial-spectral features; Indicates the reconstruction decoder; This represents the reconstructed hyperspectral image obtained from the model recovery; Step S43: The pre-training phase employs joint constraints between the original domain and the score domain; Step S44: During training, fractional-order parameters A phased sampling strategy can be adopted.

6. The medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning according to claim 1, characterized in that, Step S5 includes the following steps: Step S51: The shared fractional domain spatial-spectral encoder is used for both self-supervised pre-training and supervised fine-tuning stages, and its core component is the multi-domain feature extraction module. Step S52: In view of the characteristics of in-situ medical hyperspectral data, which simultaneously contain tissue surface morphology, local texture, reflectance curve and cross-band response differences, the multi-domain feature extraction module performs collaborative modeling through original domain multi-scale convolution branch, two-dimensional fractional domain spatial branch and one-dimensional fractional domain spectral branch, thereby obtaining complementary spatial-spectral features. Step S53: In the original domain multi-scale convolution branch, the input features are divided into multiple subgroups along the channel dimension, and multi-scale local spatial responses are extracted by convolution operations with different kernel sizes. Then, channel fusion is performed by pointwise convolution. Step S54: In the two-dimensional fractional domain spatial branch, perform a two-dimensional fractional Fourier transform on the input features along the spatial dimension to obtain the complex-valued fractional domain representation. Concatenate the real and imaginary parts and integrate the features through pointwise convolution. Then, return to the original feature space through an inverse two-dimensional fractional Fourier transform. Step S55: In the one-dimensional fractional domain spectral branch, perform a one-dimensional fractional Fourier transform on the input features along the spectral or channel correlation dimension, and fuse the real and imaginary parts of the complex-valued features, and then obtain the spectral enhancement features through inverse transform; Step S56: To avoid inconsistencies in response scales between different branches, the multi-domain feature extraction module normalizes the output of the fractional domain branches and introduces a learnable scaling factor to adaptively adjust the contribution of each branch. Step S57: Finally, the three types of branch features are spliced ​​and fused, and the module output is obtained through residual connection and nonlinear activation.

7. The medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning according to claim 1, characterized in that, The calculation formula for step S6 is as follows: ， In the formula, h represents the attention head index; This represents the output of the h-th attention head; and These represent the query features respectively. Bond features In fractional order The result of the one-dimensional fractional Fourier transform; The magnitude of the complex-valued feature. Represents a logarithmic mapping. A tiny positive number introduced to prevent numerical instability; T represents matrix transpose; d represents the feature dimension of a single attention head, used to scale attention relevance; This represents the value feature of the h-th attention head; Used to normalize the relevance weights.

8. The medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning according to claim 1, characterized in that, Step S7 includes the following steps: Step S71: In the segmentation task, a decoder-based pixel-level prediction head is adopted. This prediction head receives multi-layer features processed by a fractional-domain spatial-spectral encoder and a fractional-domain representation enhancement module, gradually fuses multi-scale skip connection features from the encoder, and restores spatial resolution through upsampling, finally outputting pixel-level category predictions in in-situ medical hyperspectral images. Step S72: To improve the effectiveness of skip connection features, a lightweight skip feature refinement module is introduced before fusing skip features with decoding features. This module consists of pointwise convolution, batch normalization, depthwise convolution, and efficient channel attention, and can be represented as: ， in, The encoder skip connection features represent the input; This represents pointwise convolution, used for channel mapping and fusion; Indicates batch normalization; This represents depthwise convolution, used to extract lightweight local spatial information; This indicates efficient channel attention, used to enhance critical channel response; This represents the output features after processing by the jump feature refinement module; Step S73: The supervision loss for the segmentation task includes cross-entropy loss and Dice loss. The final segmentation loss calculation formula is as follows: ， Among them, cross-entropy loss Dice loss is used for pixel-level category constraints. Used to constrain the overlap between the predicted region and the true region.

9. The medical hyperspectral image segmentation and classification method based on fractional domain mask representation learning according to claim 1, characterized in that, Step S8 includes the following steps: Step S81: In the application phase, the in-situ medical hyperspectral image to be analyzed is processed according to the same preprocessing procedure as in the training phase, and then input into the trained model. Step S82: For segmentation tasks, the model outputs the tissue category or lesion category corresponding to each pixel, and generates a segmentation mask, color visualization results, and quantitative analysis indicators. For classification tasks, the model outputs the category probability of the image or region of interest and the final category discrimination result.

Images