Hyperspectral multi-view medical image classification method and system based on random band projection

Abstract

The application discloses a hyperspectral multi-view medical image classification method and system based on random band projection, and the method comprises the following steps: performing random band projection on hyperspectral data to form a hyperspectral multi-view image representation, extracting semantic features from auxiliary examination report texts, and extracting visual features from auxiliary medical images; a multi-modal fusion classification model is constructed, and a band perception feature modulation backbone, view reliability weighted aggregation, multi-modal reliability bias attention fusion and sample condition missing modal compensation mechanism are introduced; a multi-view of a target medical object to be measured is generated, and available modal features are extracted, then the trained model is input, and a classification result and a confidence level are output. The application generates complementary multi-views while compressing dimensions through random band projection, so that the model can perceive the band source, view quality and modal integrity corresponding to each projection view, and the accuracy, robustness and missing modal adaptability of the medical image three-classification task are improved.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and artificial intelligence, and in particular to a hyperspectral multi-view medical image classification method and system based on random band projection. Background Technology

[0002] Medical image classification is one of the important applications of medical artificial intelligence. Traditional medical image analysis usually relies on single-modality images, such as ultrasound images, pathological images, endoscopic images, skin images, fundus images, or other imaging data, which are judged by human experience or automatically classified by computer-aided diagnostic models. With the increasing demand for precision clinical diagnosis and treatment, relying solely on a single image modality is often insufficient to fully characterize the complex differences in the target medical object at the levels of microstructure, macromorphology, and clinical semantics.

[0003] Hyperspectral data can reflect differences in tissue structure, composition, scattering characteristics, absorption characteristics, and spectral response of target medical objects, and has significant potential in medical image classification, tissue identification, lesion classification, and auxiliary diagnosis. Meanwhile, conventional medical images provide visual information such as the macroscopic morphology, boundaries, texture, grayscale, blood flow, and structural relationships of the target object, while auxiliary examination report text can condense the doctor's comprehensive description of image features, clinical manifestations, or examination results in natural language. Joint analysis of hyperspectral images, conventional auxiliary medical images, and report text is expected to overcome the information bottleneck of a single modality and improve the accuracy and robustness of medical image classification.

[0004] However, achieving the aforementioned multimodal fusion classification faces multiple technical challenges. First, hyperspectral data combines images and spectra, resulting in extremely high dimensionality. A single sample typically contains a large number of spectral-spatial measurements, and directly using this data as input to a deep learning model can easily lead to the curse of dimensionality, causing severe overfitting on a limited number of medical samples. Among existing dimensionality reduction methods, methods based on mathematical transformations can compress dimensionality, but the transformed feature components lack intuitive physical meaning, potentially weakening or erasing subtle spectral differences that play a crucial role in classification. Methods based on fixed band selection, while retaining some spectral interpretability, typically utilize only a small number of fixed bands, easily discarding a large amount of potential discriminative information, and have limited generalization ability across different devices, tissue types, and medical tasks. Second, most existing medical image-assisted classification systems rely only on a single image modality or simple clinical features, lacking deep fusion of hyperspectral data, auxiliary images, and report text. Furthermore, they typically assume complete data modalities, making it difficult to adapt to situations in real clinical or research scenarios where auxiliary report text, auxiliary medical images, or other modalities are missing to varying degrees. Furthermore, many systems model only around specific diseases or pathological names, and the category naming and task definition are too specific, resulting in limited scope of protection and application.

[0005] Therefore, automatically constructing multiple differentiated low-dimensional view representations from the same hyperspectral data, and adaptively fusing auxiliary medical image features and text semantic features through a multimodal attention mechanism, while designing a learning strategy robust to missing modalities, holds promise for achieving medical image classification by comprehensively utilizing multi-source heterogeneous information while avoiding high-dimensional input. How to automatically generate complementary multi-view representations while achieving adaptive multimodal fusion under modality loss remains a pressing technical challenge. Summary of the Invention

[0006] The problem this invention aims to solve is to provide a hyperspectral multi-view medical image classification method and system based on random band projection. Each band of the hyperspectral data is treated as a coordinate axis in a high-dimensional spectral space. Through multiple independent random band projections, the high-dimensional spectral data is mapped into multiple complementary low-dimensional color views. Simultaneously, combining the semantic features of auxiliary examination report text and the visual features of auxiliary medical images, the multimodal fusion classification model introduces band-aware feature modulation, view reliability weighting, multimodal reliability bias attention, and a sample condition missing modality compensation mechanism. This enables the classification of the target medical object into normal, suspicious, and abnormal categories, and provides tolerance for varying degrees of missing auxiliary modalities.

[0007] This invention adopts the following technical solution: a hyperspectral multi-view medical image classification method based on random band projection, comprising the following steps:

[0008] S1. Obtain multimodal data of the target medical object and assign classification labels. The multimodal data includes at least hyperspectral raw data and at least one pseudo-color synthetic image generated from the hyperspectral data, as well as auxiliary examination report text and / or auxiliary medical images associated with the target medical object.

[0009] S2. Perform multiple independent random band projections on the hyperspectral raw data. Each time, three bands are randomly selected from all bands. After percentage truncation, linear stretching, normalization, and enhancement, they are synthesized into a three-channel color image, forming multiple hyperspectral multi-view images of the same target medical object. Combined with the existing pseudo-color image of the target medical object, a multi-view image set is formed.

[0010] S3. Extract auxiliary modal features, including: extracting textual features from the auxiliary examination report text to obtain a text semantic vector; and extracting visual features from the auxiliary medical image to obtain an auxiliary medical image feature vector.

[0011] S4. Construct a multimodal fusion classification model, which receives the multi-view image set, text semantic vector, and auxiliary medical image feature vector as inputs. Internally, it uses a band-sensing feature modulation backbone network, view reliability weighted multi-view aggregation, multimodal reliability bias attention fusion mechanism, and sample condition missing modality compensation module to output classification probability. The multimodal fusion classification model is trained using multimodal data and corresponding three-class labels, and the best model is saved.

[0012] S5. Generate a multi-view image set from the hyperspectral data of the target medical object according to step S2, extract auxiliary modal features according to step S3, input the trained multimodal fusion classification model, and output the classification result and confidence level.

[0013] As a preferred embodiment, the classification labels in step S1 include normal, suspicious, or abnormal categories, representing three different hyperspectral-image joint characterization patterns. These three categories serve as general category names for model training, classification inference, and system output, and are not directly limited to any specific disease, specific tissue state, specific pathological name, or specific risk level.

[0014] For specific medical image application scenarios, the actual medical meaning of the three types of labels can be determined by expert annotation, pathological results, follow-up results, classification standards, or clinical consensus.

[0015] As a preferred embodiment, the random band projection in step S2 includes:

[0016] S2.1 Perform K independent random band projections on the hyperspectral raw data. Each time, three different band numbers are randomly selected without replacement from all B band numbers as a group, and the corresponding two-dimensional single-band images are extracted respectively. Repeat K times. The band combinations selected in each projection are different, so that the generated K views have differences in spectral response.

[0017] S2.2. Normalization enhancement is performed by percentage truncation linear stretching, and the three enhanced grayscale images are used as R, G, and B channels respectively to synthesize a three-channel color image.

[0018] The percentage-truncation linear stretching uses the 2% and 98% quantiles of the band data distribution as the minimum and maximum stretching values, respectively, to linearly map the original reflection intensity to the range of 0-255, and truncates extreme values ​​to unify the contrast of each view.

[0019] S2.3. Generate K complementary color views through K projections, which together with the existing pseudo-color images form a hyperspectral multi-view image set of the same target medical object.

[0020] As a preferred embodiment, step S3, extracting auxiliary modal features, includes:

[0021] Step 3.1: Extract the text features of the auxiliary examination report and use a pre-trained natural language processing model, including but not limited to ClinicalBERT, PubMedBERT, BioBERT, and medical domain large language model encoder, to convert the report text into a fixed-dimensional semantic vector.

[0022] Step 3.2: Extract visual features from auxiliary medical images. Use a pre-trained convolutional neural network or visual Transformer to extract high-level visual features from auxiliary medical images.

[0023] Step 3.3: If the target medical object contains auxiliary examination report text, extract the semantic vector of the entire report using a pre-trained language model; otherwise, set the corresponding text features to learnable modality missing embedding vectors. If the target medical object contains auxiliary medical images, extract the visual feature vectors of the auxiliary medical images using a pre-trained visual model; otherwise, set them to missing embedding vectors.

[0024] As a preferred embodiment, the multimodal fusion classification model described in step S4 includes: a band-aware image view encoder, a view reliability weighted multi-view pooler, a multimodal reliability bias attention fusion module, a sample condition missing modality compensation module, and a classification head.

[0025] The band-aware image view encoder employs a shared-weight deep visual backbone network to independently extract visual features from all K input hyperspectral multi-view images and existing pseudo-color images. For each random band projection view, its corresponding band number, center wavelength, band interval, and band order are encoded into a band combination vector. Channel scaling coefficients and bias coefficients are generated through a feature modulation layer to modulate the shallow, middle, or deep features of the visual backbone network, resulting in a set of image feature vectors that simultaneously contain spatial texture information and band source information.

[0026] The view reliability weighted multi-view pooler calculates the view reliability weight based on the image feature vector, band combination vector and view quality statistics of each view, and performs weighted aggregation on a set of image feature vectors according to the view reliability weight. Then, it combines the results of max pooling, average pooling or attention pooling to obtain a fixed-size multi-view aggregated feature vector.

[0027] The multimodal reliability bias attention fusion module receives the multi-view aggregated feature vector, text semantic vector, and auxiliary medical image feature vector, maps them to a modality token of uniform dimension, and adds a modality presence flag, modality reliability score, and class prototype consistency score to each modality token; a bias term generated by the modality presence flag, modality reliability score, and class prototype consistency score is added to the attention logits of multi-head self-attention or cross-attention, so that modalities with higher reliability and more consistent with the target class prototype receive higher fusion weights;

[0028] The sample condition missing modality compensation module generates a sample condition pseudo-modality vector corresponding to the missing modality based on available modality features and modality presence indicators, and simultaneously outputs the uncertainty score. When the text modality or auxiliary medical image modality is missing, the multimodal reliability bias attention fusion module uses the sample condition pseudo-modality vector as a weak compensation input and reduces its fusion weight according to the uncertainty score. The classification head consists of a Dropout layer, a fully connected layer, a nonlinear activation layer, and a three-dimensional fully connected output layer, outputting the corresponding classification logits. After Softmax normalization, the classification prediction probability is obtained.

[0029] As a preferred embodiment, the model training in step S4 employs a weighted cross-entropy loss function, calculating class weights based on the proportions of normal, suspicious, and abnormal samples in the training set to alleviate class imbalance. During training, text modalities or auxiliary medical image modalities in the complete samples are randomly obscured with a preset probability. The distance between the true modal vector before obscuration and the conditional pseudo-modal vector of the sample after obscuration is constrained by consistency constraints or distillation loss, and multi-view... Figure 1 Consistency constraints or entropy regularization constraints are used to stabilize the reliability weights of the view; the optimizer is AdamW, with an initial learning rate of 0.001 and a weight decay coefficient of 1×10^-4. The ReduceLROnPlateau strategy is used to reduce the learning rate when the validation loss does not improve for several consecutive epochs; random horizontal flipping, random vertical flipping, random rotation, and color jitter are applied to the input multi-view images for data augmentation; the weighted F1 score or macro-F1 score on the validation set is monitored, and the model weights with the best performance are saved.

[0030] As a preferred embodiment, the classification result in step S5 is obtained through one of the following methods:

[0031] Calculate the arithmetic mean of the predicted probabilities of all K views of the same target medical object, and take the category with the highest probability;

[0032] Alternatively, a majority voting method can be used, where the predicted category of each view is treated as one vote, and the category with the most votes is taken as the final classification result.

[0033] If auxiliary modal features are available, the multimodal fusion prediction result will be directly used as the final classification result output.

[0034] The present invention also provides: a hyperspectral multi-view medical image classification system based on random band projection, comprising:

[0035] The multimodal data acquisition module is used to acquire the hyperspectral raw data of the target medical object, existing pseudo-color images, and optional auxiliary examination report text and auxiliary medical images, and to uniformly manage the classification labels;

[0036] The random band projection multi-view generation module is used to randomly select multiple band combinations from the hyperspectral full band and perform multiple random band projections, extract band data and perform normalization enhancement, synthesize multiple three-channel color images, and combine them with existing pseudo-color images to form a multi-view image set.

[0037] The modality feature extraction module performs textual semantic feature extraction on the auxiliary examination report text, visual feature extraction on the auxiliary medical images, and handles modality loss.

[0038] The multimodal fusion classification model training module is used to construct a neural network that includes a band-sensing feature modulation backbone, a view reliability weighted multi-view aggregator, a multimodal reliability bias attention fusion module, and a sample condition missing modality compensation module. It is trained using multi-view image sets, text semantic features, auxiliary medical image features, and three-class labels, and the best model is saved.

[0039] The classification reasoning module is used to generate multi-view images of the target medical object and extract available modal features, load the best model to perform forward reasoning, and output the classification results and confidence scores of normal, suspicious or abnormal classes.

[0040] As a preferred embodiment, the system further includes a data acquisition device and a data access interface;

[0041] The data acquisition equipment includes hyperspectral imaging equipment, pseudo-color image acquisition or generation equipment, auxiliary medical image acquisition equipment, and auxiliary examination report entry or access terminal.

[0042] The data access interface is used to uniformly access hyperspectral raw data, multiple pseudo-color composite images, auxiliary examination report text, and auxiliary medical images according to the target medical object number.

[0043] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0044] 1. This invention describes the classification task as a general medical image three-classification task, using normal, suspicious and abnormal classes as neutral category names, avoiding limiting the method to specific diseases or pathological types, and improving the universality and scope of protection of the patent text.

[0045] 2. The random band projection strategy automatically generates multiple views. Without requiring experts to preset bands, it retains the physical meaning of the spectrum, achieves efficient data dimension compression and view complementarity, and greatly reduces the risk of overfitting.

[0046] 3. This invention integrates multimodal information such as hyperspectral images, auxiliary medical images, and auxiliary examination report text, which is beneficial for characterizing target medical objects from multiple dimensions such as microscopic spectroscopy, macroscopic vision, and clinical semantics, and improves the three-class classification recognition capability.

[0047] 4. An adaptive missing modality handling mechanism is introduced, which enables the model to work normally even when auxiliary examination report text or auxiliary medical images are missing. This conforms to the normal situation of incomplete data in real medical scenarios and enhances the universality and robustness of the system.

[0048] 5. Low computational cost: Only three bands need to be processed each time to generate a view. After multi-view pooling and combined with a lightweight fusion module, the overall inference speed can meet the real-time requirements of medical image-assisted analysis. Attached Figure Description

[0049] Figure 1 This is a flowchart of the hyperspectral multi-view medical image classification method of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0051] In one embodiment of the present invention, the overall process of the hyperspectral multi-view medical image classification method based on random band projection is as follows: Figure 1As shown, the process includes: S1, acquiring multimodal data; S2, performing random band projection and generating hyperspectral multiviews; S3, extracting textual features from auxiliary examination reports and visual features from auxiliary medical images; S4, constructing and training a multimodal fusion classification model, outputting probabilities for three classes: normal, suspicious, and abnormal; S5, performing inference on the target medical object and outputting the classification results and confidence levels. The specific implementation is as follows:

[0052] (a) Acquiring multimodal data of the target medical subjects

[0053] In this embodiment, the raw hyperspectral data can be obtained through a pushbroom hyperspectral imaging system, a snapshot hyperspectral imaging system, or other hyperspectral acquisition devices. Each sample can correspond to a raw data file, which contains information on both spatial and spectral dimensions. The hyperspectral data can cover the visible light, near-infrared, or other spectral ranges suitable for medical imaging.

[0054] All samples are labeled with three categories: Normal, Suspicious, and Abnormal. These labels are encoded as 0, 1, and 2 during data storage and model training, respectively. In the output, the results are displayed as Normal, Suspicious, and Abnormal, along with their corresponding confidence levels.

[0055] At least one pseudo-color composite image is generated in advance using a fixed band combination or other methods as a standard reference view. Simultaneously, based on the data conditions of the specific medical task, auxiliary examination report text and corresponding auxiliary medical images are acquired.

[0056] In this embodiment, the auxiliary medical images can be one or more of the following: ultrasound images, pathological images, endoscopic images, skin images, fundus images, breast images, CT images, MRI images, optical coherence tomography images, or other medical images. The auxiliary examination report text can be an examination report, image description, pathology summary, structured examination fields, or other clinical text.

[0057] (ii) Category naming and mapping rules

[0058] The three categories in the medical image classification task are named Normal, Suspicious, and Abnormal, with the following meanings:

[0059] Normal category: This refers to the image category in which the target medical object presents a state that is consistent with or substantially consistent with the preset normal state in hyperspectral data, multi-view images, and available auxiliary modalities;

[0060] Suspicious category: This category represents images in which the target medical object presents a distinguishable difference from the normal category in hyperspectral data, multi-view images, and available auxiliary modalities, but is not yet directly classified into the abnormal category.

[0061] Anomaly category: This refers to the image category in which the target medical object deviates significantly from the preset normal state in hyperspectral data, multi-view images, and available auxiliary modalities.

[0062] It is important to note that the three categories of names mentioned above do not directly express specific diseases or pathological types, nor are they directly equivalent to fixed clinical diagnostic conclusions. In practical applications, these three categories of labels can be determined by expert annotations, pathological results, follow-up results, clinical classification standards, or multi-center data consensus. The system can save a mapping table for specific tasks in the background, but when outputting externally, it prioritizes using the normal, suspicious, and abnormal categories to improve the cross-disease applicability of the method.

[0063] (III) Generation of multiple views for random band projection

[0064] Enter the random band projection multi-view generation stage, and set the number of projections to K. Perform K random band projections independently for each target medical object: without replacement, select 3 different bands from band numbers 1 to B as a group, which is recorded as one projection.

[0065] After each projection, a two-dimensional image of the corresponding band is extracted from the hyperspectral three-dimensional data volume and normalized using percentage-truncation linear stretching.

[0066] Specifically, the 2% quantile and 98% quantile of the image for this band are calculated respectively. If the upper limit is less than or equal to the lower limit, the global minimum and maximum values ​​are used instead.

[0067] The original reflection intensity is linearly mapped to [0,1] using the following formula, then multiplied by 255 and cropped to convert it into an 8-bit grayscale image.

[0068] ;

[0069] in, The input is the grayscale value of the original image pixels. and This represents the maximum and minimum values ​​of all original reflection intensities in the entire image.

[0070] The stretching results of the three bands are assigned to the R, G, and B channels respectively, and then merged into a single three-channel color image. K projections yield K views at different spectral angles, which, together with the existing fixed-band pseudo-color image, constitute a hyperspectral multi-view image set for the target medical object. All views can be uniformly adjusted to 256×256 pixels, or adjusted to other sizes as needed for the specific model.

[0071] In this embodiment, preferably, the original hyperspectral data contains 294 bands, and K is 9. Three different bands are randomly selected from the 294 band numbers each time to form nine complementary three-channel color views. In this way, the model does not need to directly process the complete high-dimensional spectral cube, but instead processes multiple low-dimensional pseudo-color views, thereby reducing the input dimensionality and preserving the spectral differences under different band combinations.

[0072] (iv) Modal feature extraction

[0073] For auxiliary examination report text, all Chinese and English characters, numbers, examination descriptions, structured fields, or unstructured descriptions are uniformly encoded, concatenated into sentences, paragraphs, or structured text sequences, and input into a pre-trained medical language model. The sentence-level vectors or hidden states at specific locations are then used as the semantic representation of the text. Optional models include ClinicalBERT, PubMedBERT, BioBERT, pre-trained Chinese models in the medical field, cross-language medical text encoding models, or other text encoding models.

[0074] Specifically, if a target medical object has no auxiliary examination report text, a learnable text missing embedding vector is defined, initialized as a zero vector or a random vector, and optimized during training; alternatively, the text features can be directly set to an all-zero vector, and the model can be notified of the missing modality through a modality presence flag.

[0075] For assisting medical images, a pre-trained visual model on a natural or medical image dataset can be loaded, the final classification layer removed, and the output high-level visual feature vectors compressed to a uniform dimension via a linear layer. Optional visual models include ResNet, EfficientNet, DenseNet, ConvNeXt, Swing Transformer, Vision Transformer, or other medical image-specific encoders.

[0076] Specifically, if auxiliary medical images are missing, a learnable image missing embedding vector or an all-zero vector is similarly used, and the corresponding modality presence marker is set as missing.

[0077] (v) Multimodal fusion classification model

[0078] The multimodal fusion classification model includes a band-aware image view encoder, a view reliability weighted multi-view aggregator, a multimodal reliability bias attention fusion module, a sample condition missing modality compensation module, and a classification head.

[0079] A band-aware image view encoder is used to extract features for each hyperspectral view. In this embodiment, the encoder employs EfficientNet-B4, ResNet, ConvNeXt, Swin Transformer, Vision Transformer, or other deep vision backbone networks. Each view is independently forward-propagated through the visual backbone, and all views share the same set of network weights. Unlike directly applying a conventional RGB visual backbone, this embodiment synchronously records the three band numbers, corresponding center wavelengths, band intervals, and channel arrangement order for each random projection view, and encodes them into a band combination vector.

[0080] After the band combination vector is input into the modulation layer, it generates channel scaling coefficients and bias coefficients for at least one intermediate feature layer in the corresponding visual backbone network. Feature-level affine modulation is then performed on the feature map of this layer, enabling the model to distinguish pseudo-color views formed by different random band combinations and reducing the model's erroneous reliance on the semantics of ordinary RGB channels. Subsequently, linear projection maps the features of each view to a unified dimension.

[0081] A view reliability-weighted multi-view aggregator is used to aggregate all view features of the same target medical object. In this embodiment, the view reliability weight is first calculated based on the image feature vector, band combination vector, and view quality statistics for each view. The view quality statistics include one or more of the following: view brightness distribution, saturated pixel ratio, contrast, edge intensity, or inter-band correlation. Then, all view features are weighted and summed using the view reliability weight, and the sum is concatenated or added to the results of max pooling, average pooling, attention pooling, or gated pooling to obtain a fixed-dimensional multi-view aggregated vector. This vector represents the comprehensive hyperspectral image representation of the target medical object under multiple random band combination perspectives.

[0082] The multimodal reliability bias attention fusion module normalizes and linearly projects the multi-view aggregated vector, text semantic vector, and auxiliary medical image feature vector, mapping them to a unified dimension and stacking them into multiple modality tokens. For each modality token, the model simultaneously calculates a modality presence flag, a modality reliability score, and a category prototype consistency score. The category prototype consistency score is obtained by the similarity between the current modality token and the learnable category prototypes of the normal, suspicious, and abnormal classes, and is used to measure the degree of support of the modality for three-class classification.

[0083] In multi-head self-attention layers or cross-attention layers, the modality reliability score, modality presence flag, and class prototype consistency score are converted into attention bias terms and added to attention logits. This gives higher interaction weights to modality tokens that are more reliable, not missing, and more consistent with the class prototype, while low-quality or missing modalities are automatically suppressed. Through this mechanism, the model can learn the complementary relationships between hyperspectral multi-view information, auxiliary image visual information, and textual semantic information, while reducing the impact of modality noise and modality conflict on classification results.

[0084] The sample condition missing modality compensation module is used to handle potentially missing text or auxiliary medical image modalities. For missing modalities, instead of using only fixed zero vectors or fixed missing embeddings, it inputs available multi-view aggregate vectors, available auxiliary modality vectors, and modality presence flags into the pseudo-modality generator to generate sample condition pseudo-modality vectors related to the current target medical object.

[0085] Simultaneously, the sample condition missing modality compensation module outputs an uncertainty score, which is used to generate uncertainty gating coefficients. If the uncertainty of the pseudo-modal vector is high, the corresponding gating coefficient is reduced, allowing it to participate in fusion only as weak compensation information; if the uncertainty is low, it is allowed to make a higher contribution in attention fusion. Thus, when auxiliary examination report text or auxiliary medical images are missing, the model can still use hyperspectral multiviews and available modalities to infer the corresponding compensation representation, while avoiding over-reliance on unreliable compensation information.

[0086] The fused features from multi-head self-attention or cross-attention outputs are fed into the classification head after residual connection and normalization. In this embodiment, the classification head may include a Dropout layer, a fully connected layer, a ReLU or GELU activation layer, a second Dropout layer, and a final 3D fully connected layer.

[0087] The final output consists of three values, which are then processed by Softmax to obtain the probabilities of the corresponding normal, suspicious, and abnormal classes.

[0088] (vi) Model Training

[0089] During the training phase, the training and validation sets can be divided according to a preset ratio, such as 4:1, 7:3, or 8:2. The loss function uses weighted cross-entropy, and the class weights can be calculated based on the proportion of normal, suspicious, and abnormal samples in the training set to alleviate the class imbalance problem.

[0090] The optimizer can be AdamW, with an initial learning rate of 0.001 and weight decay of 1e-4. The learning rate scheduling strategy can be ReduceLROnPlateau, cosine annealing, linear warmup, or other strategies. Each batch contains a collection of views from different target medical objects, and the data loader must support fixed-number or variable-length view inputs.

[0091] The input hyperspectral multiview and existing pseudocolor images can be enhanced with random horizontal flipping, random vertical flipping, random rotation, cropping, scaling, brightness perturbation, contrast perturbation, saturation perturbation, or other enhancement methods suitable for medical images.

[0092] During training, monitor metrics such as precision, recall, F1-score, macro-F1, weighted-F1, or AUC on the validation set, and save the weights of the best-performing model.

[0093] In particular, to enhance the robustness of missing modalities, during training, auxiliary examination report text and auxiliary medical images can be randomly masked with a certain probability, and the true features of the complete modality before masking can be used as the teacher signal.

[0094] For example, text features are masked with a 20% probability, and auxiliary medical image features are masked with a 20% probability, both independently. After masking, a pseudo-modal vector is generated by the sample conditional missing modality compensation module, and the pseudo-modal vector is constrained to approximate the true modality vector before masking by mean squared error, cosine distance, contrastive learning loss, or distillation loss. Furthermore, multi-view weights can be applied to the view reliability weights. Figure 1 Consistency constraints, or entropy regularization constraints, prevent the model from relying on a single random projected view for a long time. In this way, the model can maintain stable output when faced with partial modality loss during the actual inference phase.

[0095] (vii) Reasoning methods

[0096] During inference, the same hyperspectral data preprocessing and random band projection process as in the training phase is performed on the target medical object to generate a multi-view image set; simultaneously, available auxiliary examination report text features and auxiliary medical image features are extracted. For missing modalities, the sample conditional missing modality compensation module generates corresponding sample conditional pseudo-modal vectors based on available modalities and sets their fusion weights according to the uncertainty score; when the compensation uncertainty is higher than a preset threshold, the missing modality feature can also be set as a missing embedding vector and its attention weight reduced.

[0097] Load the best model saved during the training phase, perform forward propagation, and output the three-class classification probabilities: normal, suspicious, and abnormal. The class with the highest probability is taken as the final classification result, and the confidence score for that class is also output. To reduce the variance introduced by random band projection, the random band projection can be repeated multiple times on the same target medical object, and the average probability can be taken. Alternatively, a majority voting method can be used to determine the final classification result.

[0098] In one embodiment of the present invention, a hyperspectral multi-view medical image three-classification system based on random band projection is also provided, comprising: a data acquisition device, data access and management software, random band projection multi-view generation software, multimodal fusion classification software, missing modality adaptive processing software, and a classification result output terminal. The data acquisition device may include a hyperspectral imaging device, a pseudo-color image acquisition or generation device, an auxiliary medical image acquisition device, and an auxiliary examination report entry terminal.

[0099] Furthermore, in a specific application, the target medical object is a thyroid-related tissue specimen or a thyroid-related imaging region, the auxiliary medical image is an ultrasound image, and the auxiliary examination report text is an ultrasound diagnostic report text. This embodiment is only used to illustrate one application of the present invention in a thyroid-related image classification task and does not limit the application of the present invention to other medical image three-classification tasks.

[0100] First, the raw hyperspectral data is acquired from target tissue specimens using a pushbroom hyperspectral imaging system, outputting as RAW format binary files. Each sample corresponds to one file, containing spatial pixels and reflectance intensity across continuous spectral bands. The data type can be a 16-bit unsigned integer, covering the visible to near-infrared spectral range. A pseudo-color composite image is pre-generated using a fixed band combination or other methods as a standard reference view.

[0101] All samples are accompanied by a three-category label, uniformly coded as follows: 0 for normal, 1 for suspicious, and 2 for abnormal. The three labels can be derived from existing expert annotations, postoperative results, clinical records, or multidisciplinary consensus in this thyroid-related image task. In model training, system inference, and result output, specific pathological names are not used directly; instead, the categories of normal, suspicious, and abnormal are used uniformly.

[0102] Based on hospital information system records, retrieve text descriptions from ultrasound diagnostic reports regarding the target area's size, boundaries, echogenicity, calcification, and blood flow signals, along with corresponding static ultrasound images or video keyframes. This data is optional and may be missing.

[0103] Next, the random band projection multi-view generation stage begins, with K=9. Nine random band projections are performed independently for each sample: three different bands are randomly selected without replacement from band numbers 1 to 294 as a group. After each projection, the corresponding two-dimensional image of the band is extracted from the hyperspectral 3D data volume and normalized using percentage-truncated linear stretching. The stretching results of the three bands are assigned to the R, G, and B channels respectively and merged into a single three-channel color PNG image. Nine projections yield nine views from different spectral angles, which, together with the existing fixed-band pseudo-color image, constitute the hyperspectral multi-view image set for that sample. All views can be resized to 256×256 pixels.

[0104] Next, on the modal feature extraction side, for ultrasound diagnostic report text, the Chinese, English, and numerical characters are uniformly encoded, concatenated into a sentence or paragraph, and input into a pre-trained medical language model. The sentence-level vector is then used as the semantic representation of the text. If a sample has no report, a learnable text missing embedding vector is defined. For ultrasound images, a visual model pre-trained on a medical image or natural image dataset and fine-tuned with relevant data is loaded. The final classification layer is removed, and the output visual feature vector is then compressed to a uniform dimension through a linear layer. If an ultrasound image is missing, a learnable image missing embedding vector is used similarly.

[0105] Subsequently, a multimodal fusion classification model is constructed, with the backbone employing EfficientNet-B4 with band-aware feature modulation. Each view is independently propagated forward through EfficientNet-B4, sharing weights. Simultaneously, the three random band numbers, center wavelength, and band intervals corresponding to that view are encoded into a band combination vector. The modulation layer generates channel scaling and bias coefficients for the intermediate feature layer, modulating the EfficientNet-B4 features to extract the band combination-aware image feature vector, which is then linearly projected to 512 dimensions. All view features of the same sample are weighted and aggregated using view reliability weights, and combined with the max-pooling result to obtain a 512-dimensional multi-view aggregated vector. The multimodal reliability bias attention fusion module normalizes the multi-view aggregated vector, text feature vector, and ultrasound image feature vector using LayerNorm, then stacks them into three tokens input to a multi-head self-attention layer. Attention weights between tokens are calculated by combining modality presence flags, modality reliability scores, and class prototype consistency scores.

[0106] For potentially missing text or ultrasound image modalities, the sample conditional missing modality compensation module generates a corresponding sample conditional pseudo-modal vector based on available multi-view aggregated vectors, available auxiliary modal vectors, and modality presence flags, and outputs an uncertainty score. If the modality is actually missing, the pseudo-modal vector enters the attention fusion module in a weakly compensated form after uncertainty gating. During training, text or ultrasound images are randomly marked as missing with a certain probability to simulate real missing patterns, and the pseudo-modal vector after occlusion is supervised by the real modal vector before occlusion, enabling the model to learn to perform sample-level compensation in missing cases.

[0107] The fused features from the multi-head self-attention output are fed into the classification head after residual connections and LayerNorm. The classification head may include Dropout, fully connected layers, ReLU activation layers, further Dropout, and a 3D fully connected layer. Three values ​​are output, which are then processed by Softmax to obtain the probabilities of the corresponding normal, suspicious, and abnormal classes.

[0108] During training, the training and validation set ratio can be 4:1. The loss function uses weighted cross-entropy, with weights calculated based on the number of samples from each of the three classes. The optimizer is AdamW, with an initial learning rate of 0.001, weight decay of 1e-4, and a batch size of 32. The learning rate is monitored using ReduceLROnPlateau to track the validation set loss. At each epoch, precision, recall, and F1-score are calculated on the validation set, and macro-F1 is used as the model selection metric, saving the weights of models with the highest macro-F1 score.

[0109] During inference, the same random band projection is performed on the test sample to generate multiple views and extract available ultrasound modal features. For missing ultrasound report text or ultrasound image modalities, the sample conditional pseudo-modal vector is first generated by the sample conditional missing modal compensation module, and then its fusion weight is attenuated according to the uncertainty score. The best model is loaded for forward propagation, and the category with the highest probability is taken as the classification result.

[0110] For a more reliable level of confidence, the random band projection process can be repeated multiple times and the probability averaged to reduce the variance caused by the randomness of the projection.

[0111] Furthermore, in one or more embodiments, the classification system of the present invention can be deployed in a medical imaging workstation. After the hyperspectral imaging device acquires the raw data of the target medical object, the software system parses it into a three-dimensional hyperspectral data volume; the random band projection unit generates multiple color views according to a preset K value; the modal feature extraction unit reads the auxiliary examination report text and auxiliary medical images and processes their missing states; the multimodal fusion inference unit loads the trained model and outputs the three-class classification probability; the result output unit writes the classification category, confidence level, selected random band combination, and missing modal state into the sample database or case database.

[0112] Furthermore, in one or more embodiments, an electronic device and a computer-readable storage medium are also provided. The electronic device may be a medical imaging workstation with a GPU, a server, an edge computing device, or a cloud computing node; the computer-readable storage medium may be a hard disk, a solid-state drive, flash memory, an optical disk, a mobile storage medium, or a network storage medium, wherein the program instructions stored thereon, when executed, enable the electronic device to complete hyperspectral data reading, random band projection, multi-view generation, auxiliary modal feature extraction, sample condition missing modality compensation, reliability bias fusion, and classification result output.

[0113] It is particularly important to note that in some embodiments, the normal, suspicious, and abnormal categories may also be displayed as Category 0, Category 1, and Category 2, or as Category A, Category B, and Category C, depending on the specific system interface or project management needs. All of the above display methods belong to the same three-category labeling system, the purpose of which is to avoid limiting the classification task to a specific disease or pathological type.

[0114] Furthermore, in some embodiments, each projection of the random band projection may select more than three bands, and form a three-channel image through channel compression, main channel selection, linear mapping or learnable mapping; multiple projection views may also be input into a three-dimensional convolutional network, graph neural network or temporal modeling network for further fusion.

[0115] Furthermore, in some embodiments, multimodal fusion methods can employ feature splicing, gated fusion, bilinear pooling, cross-modal Transformer, hybrid expert models, reliability biased attention, category prototype guided attention, or other learnable fusion methods; missing modality processing methods can employ modality dropout, missing embedding, sample conditional pseudomodal generation, generative modality completion, distillation learning, or uncertainty weighting, etc.; the band-aware image backbone can also be implemented using feature-level affine modulation, conditional convolution, lightweight adapter, channel attention modulation, or band position coding injection, etc.

[0116] Furthermore, the method described in this invention can be used for three-class classification of skin lesions, three-class classification of digestive tract lesions, three-class classification of breast-related images, three-class classification of pathological section regions, three-class classification of fundus images, three-class classification of oral mucosa images, three-class classification of intraoperative tissue identification, or other medical image three-class classification tasks. For different tasks, normal, suspicious, and abnormal classes are respectively mapped to the three target categories in the task, while the technical framework of random band projection, multi-view generation, multi-modal fusion, and missing modality processing remains consistent.

[0117] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A hyperspectral multi-view medical image classification method based on random band projection, characterized in that, Includes the following steps: S1. Obtain multimodal data of the target medical object and assign classification labels. The multimodal data includes at least hyperspectral raw data and at least one pseudo-color synthetic image generated from the hyperspectral data, as well as auxiliary examination report text and / or auxiliary medical images associated with the target medical object. S2. Perform multiple independent random band projections on the hyperspectral raw data. Each time, three bands are randomly selected from all bands. After percentage truncation, linear stretching, normalization and enhancement, they are synthesized into a three-channel color image, forming multiple hyperspectral multi-view images of the same target medical object. Combine the existing pseudo-color images of the target medical object to form a multi-view image set; S3. Extract auxiliary modal features, including: extracting textual features from the auxiliary examination report text to obtain a text semantic vector; and extracting visual features from the auxiliary medical image to obtain an auxiliary medical image feature vector. S4. Construct a multimodal fusion classification model, which receives the multi-view image set, text semantic vector, and auxiliary medical image feature vector as inputs. Internally, it uses a band-aware feature modulation backbone network, view reliability weighted multi-view aggregation, multimodal reliability bias attention fusion mechanism, and sample condition missing modality compensation module to output classification probability. The multimodal fusion classification model is trained using multimodal data and corresponding classification labels, and the best model is saved. S5. Generate a multi-view image set from the hyperspectral data of the target medical object according to step S2, and extract auxiliary modal features according to step S3. Input the trained multimodal fusion classification model and output the classification result and confidence level.

2. The hyperspectral multi-view medical image classification method according to claim 1, characterized in that: The classification labels mentioned in step S1 include normal, suspicious, or abnormal categories. The actual medical meaning of the three categories of labels is determined by expert annotation, pathological results, follow-up results, classification standards, or common clinical practice. The classification results mentioned in step S5 include three classification results: normal, suspicious, and abnormal.

3. The hyperspectral multi-view medical image classification method according to claim 1, characterized in that: The random band projection in step S2 includes: S2.1 Perform K independent random band projections on the hyperspectral raw data. Each time, three different band numbers are randomly selected without replacement from all B band numbers as a group, and the corresponding two-dimensional single-band images are extracted respectively. Repeat this K times. The band combinations selected in each projection are different, so that the generated K views have differences in spectral response. S2.

2. Normalization enhancement is performed by percentage truncation linear stretching, and the three enhanced grayscale images are used as R, G, and B channels respectively to synthesize a three-channel color image. The percentage-truncation linear stretching uses the 2% and 98% quantiles of the band data distribution as the minimum and maximum stretching values, respectively, to linearly map the original reflection intensity to the range of 0-255, and truncates extreme values ​​to unify the contrast of each view. S2.

3. Generate K complementary color views through K projections, which together with the existing pseudo-color images form a hyperspectral multi-view image set of the same target medical object.

4. The hyperspectral multi-view medical image classification method according to claim 1, characterized in that: Step S3, extracting auxiliary modal features, includes: Step 3.1: Extract the text features of the auxiliary examination report and use a pre-trained natural language processing model, including but not limited to ClinicalBERT, PubMedBERT, BioBERT, and medical domain large language model encoder, to convert the report text into a fixed-dimensional semantic vector. Step 3.2: Extract visual features from auxiliary medical images. Use a pre-trained convolutional neural network or visual Transformer to extract high-level visual features from auxiliary medical images. Step 3.3: If a target medical object lacks auxiliary examination report text, then its corresponding text semantic vector is set to an all-zero vector or a learnable modality missing embedding vector; if it lacks auxiliary medical images, then its corresponding auxiliary medical image feature vector is processed in the same way.

5. The hyperspectral multi-view medical image classification method according to claim 1, characterized in that: The multimodal fusion classification model described in step S4 includes: Image view encoder: A deep vision backbone network with shared weights is used, and the random band number, wavelength position and band interval corresponding to each hyperspectral multiview image are encoded into a band combination vector; the band combination vector is modulated by a modulation layer to generate channel scaling coefficients and bias coefficients, and after feature modulation on at least one intermediate feature layer of the deep vision backbone network, a set of image feature vectors with band combination perception capability is obtained. Multi-view pooler: Calculates view reliability weights based on the image feature vector, band combination vector, and view quality statistics of each view, performs reliability-weighted pooling on the set of image feature vectors, and combines max pooling, average pooling, or attention pooling to obtain a fixed-size multi-view aggregated feature vector. Multimodal attention fusion module: Receives the multi-view aggregated feature vector, text semantic vector, and auxiliary medical image feature vector. Applies a modality presence flag, modality reliability score, and category prototype consistency score to each modality token. Adds a bias term generated by the modality reliability score and modality presence flag to the attention logits of the multi-head self-attention or cross-attention mechanism to reduce the impact of low reliability or missing modalities on the fusion result, and generates fused multimodal features. Adaptive missing modality processing module: For missing text or auxiliary medical image modalities, it generates sample conditional pseudo-modal vectors and uncertainty scores based on available modal features and modality presence indicators, and controls the contribution of the pseudo-modal vectors into the attention fusion module through uncertainty gating coefficient; The classification head consists of a Dropout layer, a fully connected layer, a non-linear activation layer, and a three-dimensional fully connected output layer, which outputs the logits of the corresponding classification. After Softmax normalization, the classification prediction probability is obtained.

6. The hyperspectral multi-view medical image classification method according to claim 5, characterized in that: The adaptive missing modality processing module includes a missing flag encoder, a pseudo-modality generator, and an uncertainty gate; The missing flag encoder encodes the presence state of the text modality and the auxiliary medical image modality into a missing state vector; The pseudo-modal generator takes the multi-view aggregated feature vector and available auxiliary modal features as input to generate a sample conditional pseudo-modal vector corresponding to the current target medical object for the missing modality; The uncertainty gate outputs the uncertainty score corresponding to the missing mode, and reduces the weight of the pseudo-mode vector in attention fusion based on the uncertainty score.

7. The hyperspectral multi-view medical image classification method according to claim 6, characterized in that: The model training described in step S4 uses a weighted cross-entropy loss function to calculate class weights based on the proportion of normal, suspicious, and abnormal samples in the training set, in order to alleviate the class imbalance problem. During training, text modalities or auxiliary medical image modalities in the complete samples are randomly masked, and consistency constraints or distillation loss are applied to the real modal vectors before masking and the sample conditional pseudo-modal vectors generated after masking. Apply multi-view consistency constraints or entropy regularization constraints to the view reliability weights; The optimizer used was AdamW, with an initial learning rate of 0.001 and a weight decay coefficient of 1×10^-4. The ReduceLROnPlateau strategy was adopted to reduce the learning rate when the validation loss did not improve for several consecutive epochs. Apply random horizontal flipping, random vertical flipping, random rotation, and color jitter to the input multi-view images to perform data augmentation; monitor the weighted F1 score or macro-F1 score on the validation set and save the model weights with the best performance.

8. The hyperspectral multi-view medical image classification method according to claim 1, characterized in that: The classification result in step S5 is obtained through one of the following methods: Calculate the arithmetic mean of the predicted probabilities of all K views of the same target medical object, and take the category with the highest probability; Alternatively, a majority voting method can be used, where the predicted category of each view is treated as one vote, and the category with the most votes is taken as the final classification result. If auxiliary modal features are available, the multimodal fusion prediction result will be directly used as the final classification result output.

9. A hyperspectral multi-view medical image classification system based on random band projection, characterized in that, include: The multimodal data acquisition module is used to acquire the hyperspectral raw data of the target medical object, existing pseudo-color images, and optional auxiliary examination report text and auxiliary medical images, and to uniformly manage the classification labels; The random band projection multi-view generation module is used to randomly select multiple band combinations from the hyperspectral full band and perform multiple random band projections, extract band data and perform normalization enhancement, synthesize multiple three-channel color images, and combine them with existing pseudo-color images to form a multi-view image set. The modality feature extraction module performs textual semantic feature extraction on the auxiliary examination report text, visual feature extraction on the auxiliary medical images, and handles modality loss. The multimodal fusion classification model training module is used to construct a neural network that includes a band-sensing feature modulation backbone, a view reliability weighted multi-view aggregator, a multimodal reliability bias attention fusion module, and a sample condition missing modality compensation module. It is trained using multi-view image sets, text semantic features, auxiliary medical image features, and three-class labels, and the best model is saved. The classification reasoning module is used to generate multi-view images of the target medical object and extract available modal features, load the best model to perform forward reasoning, and output the classification results and confidence scores of normal, suspicious or abnormal classes.

10. The hyperspectral multi-view medical image classification system according to claim 9, characterized in that: The system also includes data acquisition equipment and data access interfaces; The data acquisition equipment includes hyperspectral imaging equipment, pseudo-color image acquisition or generation equipment, auxiliary medical image acquisition equipment, and auxiliary examination report entry or access terminal. The data access interface is used to uniformly access hyperspectral raw data, multiple pseudo-color composite images, auxiliary examination report text, and auxiliary medical images according to the target medical object number.

Images