Multi-modal data classification method based on large model

Abstract

The invention discloses a multi-modal data classification method based on a large model, and relates to the technical field of data classification. The method comprises the following steps: collecting and preprocessing a professional modal image, an auxiliary modal image and text data to obtain professional modal image standard data, auxiliary modal image standard data and text coding data; extracting feature vectors of the two types of image standard data through a visual feature extractor, and enhancing the text coding data through a T5 model and a bidirectional converter model to obtain enhanced text feature vectors; then constructing a double-branch learnable prompt word, performing cross-modal semantic alignment on the feature vectors of the standard data of the two types of images and the feature vectors of the enhanced text, and generating a fused feature vector through a dynamic attention fusion method of category decoupling and non-negative matrix factorization; and inputting the fusion feature vector into an integrated learning model, and outputting a final classification result.

Description

Technical Field

[0001] This invention relates to the field of data classification technology, specifically to a multimodal data classification method based on a large model. Background Technology

[0002] With the deep application of artificial intelligence technology in fields such as medical diagnosis and industrial quality inspection, the demand for multimodal data classification is becoming increasingly prominent. Relying solely on single-modal data is insufficient to fully characterize the overall features, and the collaborative utilization of multimodal data has become a key prerequisite for improving classification accuracy.

[0003] Traditional multimodal data classification methods involve independently extracting and simply fusing features from each modality: for image modalities, a common convolutional neural network is used to extract global visual features; for text modalities, the TF-IDF method is used to extract word frequency statistical features; then the feature vectors obtained from each modality are directly concatenated into a single feature matrix, which is then input into a classifier to complete the final classification.

[0004] However, traditional methods achieve multimodal fusion by directly splicing data without designing collaborative mechanisms for the semantic characteristics of different modalities. This makes it impossible to establish an effective association between the visual semantics of the image modality and the linguistic semantics of the text modality. Consequently, the fused features cannot leverage the semantic complementary value of multimodal data, resulting in the classification model being unable to capture key cross-modal association information and ultimately failing to meet the classification accuracy requirements of practical applications. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal data classification method based on a large model to solve the problems existing in the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal data classification method based on a large model, comprising the following steps: Step S1: Collect professional modal image data, auxiliary modal image data, and text data, and preprocess them to obtain professional modal image standard data, auxiliary modal image standard data, and text encoded data; Step S2: Use a visual feature extractor to extract features from the professional modality image standard data and the auxiliary modality image standard data respectively, and obtain the professional modality image feature vector and the auxiliary modality image feature vector respectively. Based on the T5 model and the bidirectional transformer model, perform text enhancement and encoding on the text encoding data to obtain the enhanced text feature vector. Step S3: Construct bi-branch learnable cue words. Based on the bi-branch learnable cue words, perform cross-modal semantic alignment on the professional modality image feature vector, the auxiliary modality image feature vector, and the enhanced text feature vector to obtain the professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector. Step S4: The professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector are fused by a dynamic attention fusion method based on category decoupling and nonnegative matrix factorization to obtain a fused feature vector; Step S5: Input the fused feature vector into the pre-trained ensemble learning model and output the final classification result to achieve classification of multimodal data.

[0007] Preferably, the process of acquiring professional modal image data, auxiliary modal image data, and text data, and preprocessing them to obtain professional modal image standard data, auxiliary modal image standard data, and text encoded data, includes the following specific steps: Acquire professional modal image data Modality1, auxiliary modal image data Modality2, and text data. ; The resolutions of Modality1 and Modality2 are unified to 224×224×3. A bilinear interpolation algorithm is used to achieve the size transformation, as shown in the following formula:

[0008] in, For the original image data, The image data after resolution standardization. and These are the coordinates of the neighboring pixels corresponding to the target pixel in the original image. and These are the weighting coefficients for bilinear interpolation, used to smoothly transition neighboring pixel values. and To unify the pixel row and column coordinates of the image, For the RGB channels of the image; Three enhancement operations—random horizontal flipping, adaptive brightness adjustment, and small-angle rotation—were performed on Modality1 and Modality2 after resolution unification to obtain professional modal image standard data. Standard data for auxiliary modal images ; For the collected text data Data cleaning, word segmentation, and encoding are performed to obtain the final encoded text data. .

[0009] Preferably, the step of using a visual feature extractor to extract features from the professional modality image standard data and the auxiliary modality image standard data respectively, to obtain the professional modality image feature vector and the auxiliary modality image feature vector, includes the following steps: Professional modal image standard data The sequence is transformed into a processable patch sequence. A convolutional operation is used to map pixel-level data to patch-level features. The formula for calculating the feature map after patch embedding is as follows:

[0010]

[0011] in, For professional modal image standard data, This represents the patch size, with a value of 16. This represents the number of feature channels after patch embedding. The feature map after patch embedding. This is the flattened patch sequence. The dimension of the feature map after patch embedding. Total number of patches; Flatten the patch sequence The input is a three-stage EfficientViT Block module, which, through the synergistic effect of deep convolution, cascaded group attention, and feedforward neural networks, progressively extracts global semantic features of the image to obtain a professional modality image feature vector. This professional modality image feature vector includes both global and local feature vectors. The third-stage output of the EfficientViT Block module Perform global average pooling to fuse the features into a single global feature vector. The formula for calculating the global feature vector of a professional modality image is as follows:

[0012] in, For professional modal image global feature vectors, This represents the total number of patches output in the third phase. This is the nth patch feature output from the third stage; Similarly, output auxiliary modal image standard data. Global feature vector of auxiliary modal image ; Output characteristics of the auxiliary modal phase of the EfficientViT Block module Patches with significant texture changes are retained through variance screening. The formula for calculating the feature set of the selected local candidate patches is as follows:

[0013] in, For the nth patch feature, Let be the channel variance of the nth patch feature. A larger variance indicates that the patch contains more detail. As the filtering threshold, 10% of the maximum variance of all patches is used. This is the set of local candidate patch features after filtering. Local max pooling is performed on the candidate patch features to fuse the detailed information of the candidate regions and generate local feature vectors. The formula for calculating the local feature vector of a professional modality image is as follows:

[0014] in, For professional modal image local feature vectors, Take the maximum value of candidate patches by channel. This is the nth local patch feature of the output features in the auxiliary modality phase; Similarly, output auxiliary modal image standard data. Auxiliary modal image local feature vector .

[0015] Preferably, the text enhancement and encoding of text-encoded data based on the T5 model and the bidirectional transformer model to obtain enhanced text feature vectors includes the following steps: The T5 model is used, taking the category set of the target classification task as input, to generate supplementary descriptive text related to the categories, thus solving the problem of incomplete semantics in the original case text. Taking the skin disease diagnosis task as an example, the specific formula and parameters are as follows: To guide T5 in generating high-quality category descriptions, the Prompt template is designed as follows: Prompt="Generate descriptions of skin disease categories". The typical characteristics are described, including the lesion's morphology, color, and location. For the kth category of the target task, the Prompt length should be controlled within 30 characters to avoid the T5 generation being off-topic due to excessive length. Input the Prompt into the T5 model to generate category description text. The formula for calculating the category description text is:

[0016] in, The maximum length of the generated text is set to 50. This is the beam search width, used to balance the diversity and accuracy of the generated text; its value is 4. Describe the category text; Category description text Perform data cleaning, word segmentation, and encoding processes to generate standardized codes. and Ensure consistency with the original text encoding format; Encode the original text With generated text encoding The sequences are concatenated to form an enhanced text code, which is then input into the BERT bidirectional transformer model for encoding. Since the maximum input length of BERT is 512, to avoid the fused sequence being too long, a strategy of truncating the original text w+ and concatenating the resulting text is adopted. The formula for calculating the fused attention mask sequence is as follows: ; ; in, The truncation length of the original text encoding. The input ID sequence after fusion. The fused attention mask sequence; Fusion coding ( , The input is a BERT model with a 12-layer Transformer encoder, 768-dimensional hidden layers, and 12 attention heads. The output is a feature sequence H with dimensions 1×512×768. The BERT output sequence features are transformed into fixed-dimensional text feature vectors. A token feature + global average pooling fusion pooling strategy is used to enhance the robustness of the features. The formula for calculating the enhanced text feature vector is as follows:

[0017] in, This is the feature of the 0th token output by BERT. To determine the number of valid tokens in the merged text. This is a global average pooling feature. To integrate the text features after pooling, and For linear transformation parameters, For layer normalization operation, To enhance text feature vectors.

[0018] Preferably, the construction of the dual-branch learnable prompt words includes the following steps: A global Prompt branch and a local Prompt branch are constructed. Both types of Prompts are learnable vector sequences, capturing global semantic associations and local detail associations across modalities, respectively. The structure, dimensions, and initialization strategies of the two types of Prompts are consistent, differing only in the adaptation objects in the subsequent alignment stage. The specific design is as follows: Global Prompt ( ) and local Prompt ( ) is represented as: ; ; in, Let m be the m-th learnable vector in the global Prompt. Let m be the m-th learnable vector of the local prompt. This is a global prompt sequence used to align global image features with global text semantics. This is a local prompt sequence used to align local image features with semantic details in the text. The length of the Prompt sequence. The dimension of the Prompt vector; To avoid gradient explosion caused by a large difference in distribution between the Prompt vector and the feature vector during the initial training phase, a zero-mean Gaussian distribution is used for initialization, calculated as follows: ,in, The standard deviation is 0.02. This standard deviation can quickly enter the effective training phase and avoid semantic shift caused by excessively large initial values. To incorporate the semantic information of the text enhancement features into the Prompt sequence, the enhanced text feature vector from step S2 is... As a semantic guidance vector, it is element-wise added to each vector in the Prompt sequence. The formula for calculating the global and local Prompt sequences that fuse textual semantics is as follows: ; ; in, A global Prompt sequence that integrates textual semantics. A local Prompt sequence for incorporating textual semantics.

[0019] Preferably, the step of performing cross-modal semantic alignment of the professional modality image feature vector, the auxiliary modality image feature vector, and the enhanced text feature vector based on the dual-branch learnable cue words to obtain the professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector includes the following steps: Since the dimensions of the image feature vector and the Prompt sequence are 1×768 and 16×768 respectively, the image features need to be mapped to a feature matrix matching the dimensions of the Prompt sequence through linear projection. The calculation formula is as follows: ; ; in, This is the projection result of the global feature vector of the professional modality image. This is the projection result of the local feature vectors of a professional modal image. To aid in the projection results of the global feature vector of the modal image, To aid in the projection of local feature vectors of modal images, and The weight matrix and bias terms for the global projection are given. and Here are the weight matrix and bias terms for the local projection; Cosine similarity is used to measure the semantic consistency between the projected image feature vector and the Prompt sequence. The cosine similarity value ranges from -1 to 1. The closer the value is to 1, the more semantically consistent it is; the closer it is to -1, the more semantically divergent it is. The calculation formula is as follows: ; ; in, This is the similarity matrix between the projection result of the global feature vector of the professional modality image and the global Prompt. To assist in the similarity matrix between the projection results of the global feature vectors of the modal image and the global Prompt, This is the similarity matrix between the projection results of local feature vectors of a professional modality image and the global Prompt. This is the similarity matrix between the projection results of local feature vectors of a professional modality image and the global Prompt. The L2 norm is used to normalize the characteristic matrix. A global Prompt sequence that integrates textual semantics. A local Prompt sequence that integrates textual semantics; To enhance cross-modal semantic consistency, a contrastive loss function is designed. By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, the semantic alignment between the Prompt vector and the projected image feature vector is optimized. The formula for calculating the total cross-modal alignment loss is as follows: ; ; ; in, The total cross-modal alignment loss is represented by a smaller loss value, indicating better cross-modal semantic alignment performance. For global alignment loss, For local alignment loss, and The number of positive sample pairs is 2. , Let be the similarity between the global projection feature vector of the i-th image and the t-th token. Let be the similarity between the local projection feature vector of the i-th image and the t-th token. This is the temperature coefficient, with a value of 0.07. The sum of the similarity indices of negative sample pairs is used to suppress the similarity of negative sample pairs and avoid cross-modal semantic confusion. During training, the minimum value is minimized through backpropagation. The learnable Prompt vector and projection parameters are updated synchronously until the loss converges. Even after alignment, the image feature vectors may still exhibit distribution shifts or noise interference, requiring optimization of the feature representation through residual connections and layer normalization to ultimately obtain the local alignment feature vectors for the professional modality image. Auxiliary modality image local alignment feature vector and text alignment feature vectors .

[0020] Preferably, the step of optimizing feature representation through residual connections and layer normalization includes the following steps: The projected global feature vector is residually joined with the aligned Prompt feature vector, and then layer normalization is used to eliminate distribution offset, resulting in the professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector. The professional modality image alignment feature vector includes the professional modality image global alignment feature vector and the professional modality image local alignment feature vector. The auxiliary modality image alignment feature vector includes the auxiliary modality image global alignment feature vector and the auxiliary modality image local alignment feature vector. The formulas for calculating the professional modality image global alignment feature vector and the auxiliary modality image global alignment feature vector are as follows: ; ; in, For professional modal image global alignment feature vectors, To assist in the global alignment of feature vectors in modal images, This is the weight coefficient for the Prompt feature, with a value of 0.05. The global Prompt feature is weighted by similarity. For layer normalization operation; The optimization process for local alignment feature vectors is the same as that for global alignment. After alignment optimization, local alignment feature vectors for professional modal images are obtained. Auxiliary modality image local alignment feature vector ; Enhance text feature vectors The aligned image feature dimensions need to be adapted. Dimensional uniformity is achieved through repeated expansion and layer normalization. The formula for calculating the text alignment feature vector is:

[0021] in, For text alignment feature vectors, This indicates that the text feature vector is repeated along the token dimension. Next, generate the feature matrix. To enhance text feature vectors.

[0022] Preferably, the step of fusing the professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector using a dynamic attention fusion method based on category decoupling and nonnegative matrix factorization to obtain a fused feature vector includes the following specific steps: A category decoupling fusion module is constructed. Through a process of category-related feature extraction, category-independent feature expansion, and linear integration, the fused features retain both category specificity and category adaptability. The specific design is as follows: Category-related features focus on the known category semantics of the target task, and category semantic projection is performed on the aligned feature vector from step S3:

[0023] in, The class projection results of the globally aligned feature vectors of professional modal images. To assist in the global alignment of feature vectors in modal images with category projection results, The class projection results of the local alignment feature vectors of professional modal images. The class projection results of the local alignment feature vectors of professional modal images. The class projection results of the text-aligned feature vectors. and The weight matrix and bias terms for class projection. , This represents the number of category-related feature channels, with a value of 512. Category-independent features are used to adapt to scenarios with unknown categories or changing categories. They are generated by introducing a learnable category-independent prompt sequence and projecting it. The formula for calculating the category-independent feature matrix is ​​as follows:

[0024] in, The class-independent Prompt sequence is initialized using a zero-mean Gaussian distribution. and The weight matrix and bias terms for class-independent projections. , This represents the number of category-independent feature channels, with a value of 256. The feature matrix is ​​class-independent. This represents a Gaussian distribution with zero mean. Take 0.02; By integrating class-related and class-independent features into a unified feature matrix through a linear layer, the formula for calculating the class decoupling result of the i-th feature is as follows:

[0025] in, The category decoupling result for the i-th feature. For splicing operations, and To integrate the weight matrix and bias terms of the linear layer, Xavier initialization is used. , For activation function, The category projection result for the i-th feature; Based on features after category decoupling A dynamic fusion strategy with cross-attention and semantic weight adaptation is adopted to focus the fusion process on features with high semantic relevance, resulting in a dynamically fused feature matrix. Common features are then extracted through non-negative matrix factorization, and a combination of residual connections and layer normalization is performed to enhance feature representation, as detailed below: Nonnegative matrix factorization (NMF) is used to extract the dynamically fused feature matrix. To extract common features across multiple modalities, the core idea of ​​Non-Modal Matrix Factorization (NMF) is to decompose a high-dimensional non-negative input matrix into the product of two low-dimensional non-negative matrices, as shown in the formula: ,in, The matrix is ​​a basis matrix, and the column vectors are modal common basis, where each column vector represents a core feature pattern shared across modalities. The coefficient matrix consists of row vectors representing the coefficients of the common basis, which represent the projection intensity of each token feature onto the common basis. The optimization objective of NMF is to minimize the reconstruction error, ensuring that the decomposed matrix can approximately reconstruct the input features while extracting commonalities. The formula is as follows: ,in, For the Frobenius norm, iterative optimization using the Lee-Seung alternating multiplication method is employed, with the number of iterations... =100, the change in reconstruction error over three consecutive iterations is less than the threshold. Finally, the modal common feature matrix extracted by NMF is obtained. ; Dynamically fuse feature matrices With common feature matrix Perform residual connections to supplement mode-specific information and avoid feature loss caused by NMF. The formula for calculating the enhanced feature matrix after residual integration is as follows:

[0026] in, This is the enhanced feature matrix after residual integration. This is the common feature weighting coefficient, with a value of 0.3. This is a matrix of common characteristics; right The execution layer normalizes the feature distribution to eliminate bias, and then global average pooling transforms the 16×768 matrix features into a 1×768 fused feature vector. The formula for calculating the fused feature vector is as follows: ; ; in, for The channel mean, for The channel variance, To avoid tiny values ​​where the denominator is 0, the value is selected. , The enhanced features are those obtained after normalization. To fuse feature vectors.

[0027] Preferably, the dynamic fusion strategy employing cross-attention and semantic weight adaptation is used to obtain the dynamically fused feature matrix. The specific steps are as follows: Decoupling features by text category To guide this process, cross-attention fusion is performed with features decoupled from the four image categories respectively: ; ; ; in, For querying the matrix, The key matrix, For value matrices, For the dimension of attention head, This is used to scale the attention score, preventing excessively large values ​​from causing Softmax saturation. For professional modal image global-text fusion features after cross-attention fusion, For global-text fusion features of auxiliary modal images after cross-attention fusion, For professional modal image local-text fusion features after cross-attention fusion, The auxiliary modality image local-text fusion features are obtained after cross-attention fusion; Based on the cross-modal similarity matrix from step S3, dynamic weights are calculated to weight the cross-attention fusion features:

[0028] in, The dynamic weight of the i-th fused feature is... Let be the similarity between the i-th image feature and the t-th token of the Prompt. For average similarity, The feature matrix after dynamic fusion. This represents the number of tokens in the Prompt.

[0029] Preferably, the specific steps for inputting the fused feature vector into a pre-trained ensemble learning model and outputting the final classification result are as follows: By integrating the inference results of three base classifiers—a fully connected layer classifier, a support vector machine (SVM), and a classification and regression tree (CART)—each classifier is trained and infers independently, outputting its own category probability distribution. The complementarity between classifiers enhances the overall generalization ability. The fused feature vector is then input into a pre-trained multi-classifier to obtain the final classification result of the target object. The specific process is as follows: The fully connected layer classifier is built on a deep learning framework and is specifically adapted to high-dimensional target features. It maps to the class space through a non-linear transformation. The formula for calculating the output probability distribution of the fully connected layer classifier is:

[0030] in, To fuse feature vectors, and The weight matrix and bias terms of the professional modal hidden layer. For activation function, This represents the probability of Dropout failure, with a value of 0.3. and The weight matrix and bias terms of the output layer are as follows: This represents the output probability distribution of the fully connected layer classifier. An SVM classifier employing radial basis function kernels is specifically designed to address the linear inseparability of high-dimensional features and is adaptable to various applications. Given its 768-dimensional high-dimensionality, the formula for calculating the output probability distribution of SVM is:

[0031] in, For the fused feature vector of the i-th sample in the training set, This is the bandwidth parameter for the RBF core, with a value of 0.001. This is the raw output of SVM. The probability calibration function transforms the raw output of the SVM into a probability distribution. The output probability distribution of SVM, Radial basis functions; Using the Classification and Regression Tree (CART) as the third classifier, the formula for calculating the output probability distribution is as follows:

[0032] in, For the sample set corresponding to the decision tree node, For the subset of the sample set that belongs to the k-th class, The Gini index measures the purity of node samples. for The set of leaf node samples that are included during inference. Let be the number of samples belonging to the k-th class in this leaf node. The output probability distribution of the decision tree; After independent reasoning by the three classifiers, three probability distributions are output: , , ; To integrate the advantages of the three classifiers, a weighted voting strategy based on validation set accuracy is adopted. This strategy uses dynamic weight allocation to highlight the contribution of the high-confidence classifier, as shown in the following formula:

[0033] in, The weights of the classifier, These correspond to fully connected layer, SVM, and decision tree, respectively. This represents the accuracy of the classifier on the validation set. The probability distributions of the three classifiers are summed by weights to obtain the preliminary probability distribution and classification labels after ensemble. The formula for calculating the probability distribution after ensemble is as follows:

[0034] in, The probability distribution after integration. For preliminary classification labels, Let be the integration probability that the target object belongs to the k-th class; Temperature scaling is used for probability calibration to improve the reliability and practicality of the results. The formula for calculating the probability of belonging to the k-th class after calibration is as follows:

[0035] in, This represents the probability of belonging to the k-th class after calibration. This is the temperature calibration coefficient, with a value range of [0.5, 2.0]. To integrate the logarithm of probabilities, avoid numerical overflow, and ensure stable exponential operations.

[0036] This invention provides a multimodal data classification method based on a large model, involving machine learning and deep learning techniques, which has the following beneficial effects: (1) Based on the characteristics of image and text data, while preserving visual details such as the microstructure and macromorphology of lesions, single-modal refined extraction supplements semantic information through text enhancement, avoiding the one-sidedness of feature extraction, and providing high-quality and highly discriminative basic features for subsequent cross-modal processing, which is suitable for the precise capture of subtle features in medical diagnosis.

[0037] (2) Based on the refined extraction of single modality, the complementary fusion of text and image features makes up for the limitations of single modality information, and the description of medical history in medical text and the visualization of lesions in images are mutually corroborated, reducing misjudgment caused by single modality noise or missing information, and improving the comprehensiveness and robustness of features.

[0038] (3) The dual-branch learnable Prompt cross-modal alignment accurately adapts to global and local dual-granularity features. By guiding Prompt learning through text semantics, it effectively eliminates the cross-modal semantic gap between images and text, and maps different modal features to a unified semantic space, ensuring the consistency of cross-modal information such as "lesion morphology description" and "image texture features" in medical diagnosis.

[0039] (4) Cross-attention dynamic fusion is based on cross-modal alignment. It focuses on highly correlated features by adjusting semantic weights to avoid diluting effective signals with invalid information. At the same time, it incorporates modal common feature enhancement. Even when there is redundancy or difference in multimodal data, it can still output reliable classification results stably, improving the accuracy and adaptability of classification in complex scenarios. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the steps of a multimodal data classification method based on a large model proposed in this invention. Figure 2 This is a hierarchical diagram of the steps involved in obtaining the aligned feature vectors of specialized modal images in a multimodal data classification method based on a large model proposed in this invention. Figure 3 This is a hierarchical diagram of the steps in obtaining the final classification result in a multimodal data classification method based on a large model proposed in this invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Please see Figures 1-3 This invention provides a technical solution: a multimodal data classification method based on a large model.

[0044] S1: Collect professional modal image data, auxiliary modal image data, and text data, and preprocess them to obtain professional modal image standard data, auxiliary modal image standard data, and text encoded data.

[0045] First, for the target classification task (such as skin disease diagnosis, industrial quality inspection, etc.), three core modal data of the target object are collected, as follows: Professional modal image data (denoted as Modality1): refers to professional image data that can reflect the core visual features of the target object, such as dermoscopic images in skin disease diagnosis tasks. High-resolution images of the lesion area are acquired through dermoscopic equipment (resolution range 2048×2048~4096×4096, pixel depth 24-bit RGB). This image can clearly present the microstructure of the lesion (such as pigment network, punctate blood vessels), providing key visual basis for subsequent feature extraction. Modality-2 auxiliary image data refers to auxiliary visual data that complements professional modality-2 image data. For example, clinical images taken in the task of dermatological diagnosis are captured by high-definition cameras to obtain macroscopic images of the patient's lesion area (resolution 1920×1080, pixel depth 24-bit RGB). These images contain global information such as the location, shape, and surrounding skin condition of the lesion, which can make up for the limited field of view of dermoscopy images. Text data (denoted as ModalityT): refers to structured / unstructured text that describes the attributes, state, or background of a target object, such as electronic medical record text in a dermatology diagnosis task. It includes basic patient information (gender, age), lesion description (location: left forearm; shape: round; color: light red), medical history (no history of allergies), etc. The text length is controlled between 200 and 500 characters.

[0046] It should be noted that the data collection process must comply with data security regulations. All image data must be de-identified (removing patient faces and privacy markers), and text data must be anonymized (replacing names and ID numbers with placeholders) to ensure compliance.

[0047] Because the collected professional modal image data differs from the auxiliary modal image data in resolution, brightness, and angle, directly inputting it into the visual feature extractor (EfficientViT) will lead to feature extraction bias. Therefore, standardization is necessary, as follows: The resolution of Modality1 and Modality2 is unified to 224×224×3 (this size is the default input size of the EfficientViT pre-trained model, which maximizes the reuse of pre-trained weights while balancing feature detail preservation and computational efficiency). The size transformation is achieved using a bilinear interpolation algorithm, and the specific formula is as follows:

[0048] in, For the original image data, The image data after resolution standardization. and These are the coordinates of the neighboring pixels corresponding to the target pixel in the original image. and These are the weighting coefficients for bilinear interpolation, used to smoothly transition neighboring pixel values. and To unify the pixel row and column coordinates of the image, These are the RGB channels corresponding to the image.

[0049] It should be noted that the formula for calculating the weighting coefficients of bilinear interpolation is: , , ,in, , The height and width of the original image. , The target image has a fixed height and width, and the resolution is set according to the pre-training input of EfficientViT on the ImageNet-1K dataset, which is 224×224. Reusing this size can reduce the domain offset of the feature extractor. Secondly, at the 224×224 size, the computational cost of a single image is about 1 / 320 of that of a 4096×4096 image, which can significantly reduce the computational cost of subsequent steps, while preserving the key visual details of the lesion.

[0050] To improve the model's robustness to image noise and avoid overfitting, three enhancement operations are performed on the image after resolution unification: random horizontal flipping, adaptive brightness adjustment, and small-angle rotation, as detailed below: The image is horizontally flipped with a probability p_1=0.5 (this probability is a common value in computer vision, balancing the ratio of enhanced data to original data and avoiding feature orientation distortion caused by excessive flipping). The brightness of the flipped image is adjusted, with the brightness coefficient ranging from [0.8, 1.2]. The brightness-adjusted image is then randomly rotated within the range of [-10°, 10°] (rotation angles >15° will cause lesion morphology distortion). Nearest neighbor interpolation is used to repair pixel gaps after rotation. The final output of the image standardization is: professional modal image standard data. Standard data for auxiliary modal images Both have dimensions of 224×224×3.

[0051] Collected text data The text contains special characters and semantic redundancy, requiring standardization through two steps: data cleaning, word segmentation, and encoding. This ensures that the subsequent BERT text feature extractor can accurately capture semantic information, as detailed below: Remove The text is cleaned by removing irrelevant special characters and redundant content. Specific operations include: deleting "#", "@", "&", "newline character (\n)", and "tab character (\t)"; deleting repeated phrases while retaining the first occurrence; identifying repeated segments using a string matching algorithm (such as KMP algorithm); filtering when the repeated length is ≥5 characters to prevent BERT from truncating key information due to excessive text length; standardizing numbers and units in the text; and recording the cleaned text as follows. If the length is controlled between 100 and 500 characters (if the original text is > 500 characters), the core paragraphs of "lesion description" and "diagnosis related" are retained, and irrelevant duplicate content of "patient's chief complaint" is deleted; if the original text is < 100 characters, the default description of the modality (such as "no additional text information") is added to avoid empty sequences in subsequent encoding.

[0052] Tokenizer using the BERT model Perform word segmentation and encoding to generate the standard input format for the BERT model. The specific process is as follows: Will The data is split into a sequence of tokens from the BERT vocabulary. The WordPiece segmentation strategy is used (out-of-vocabulary words are split into sub-words, e.g., "dermatoscope" is split into "skin", "skin", and "scope") to ensure that medical terminology can be effectively segmented and recognized. Since the maximum input sequence length of the BERT model is 512 tokens, padding or truncation is required to ensure a uniform input sequence length. The padded token sequence is then converted into an ID sequence, with each token corresponding to a unique ID in the BERT vocabulary. An attention mask is created based on the input IDs, with valid tokens having a value of 1 and padded tokens having a value of 0, ultimately yielding the encoded text data. = .

[0053] S2: A visual feature extractor is used to extract features from the professional modality image standard data and the auxiliary modality image standard data respectively, and the corresponding professional modality image feature vector and auxiliary modality image feature vector are obtained. Based on the T5 model and the bidirectional transformer model, the text encoding data is enhanced and encoded to obtain the enhanced text feature vector.

[0054] A visual feature extractor is used to extract features from the standard data of the professional modality image and the standard data of the auxiliary modality image, resulting in feature vectors for the professional modality image and the auxiliary modality image. Since the feature extraction process for the two image modalities is consistent, the following uses the professional modality image as an example. For example, auxiliary modal images Only the input data is replaced; the parameters and process are completely reused.

[0055] First, the 224×224×3 normalized image is transformed into a patch sequence that can be processed by EfficientViT. A convolutional operation is then used to map pixel-level data to patch-level features. The formula for calculating the feature map after patch embedding is as follows: ; ; in, For professional modal image standard data, This represents the patch size, with a value of 16. This represents the number of feature channels after patch embedding. The feature map after patch embedding. This is the flattened patch sequence. The dimension of the feature map after patch embedding. This represents the total number of patches.

[0056] It should be noted that the core function of patch embedding is to decompose a high-resolution image into local patches of fixed size, which not only preserves the pixel correlation within each patch, but also extracts preliminary local features through convolution operations, laying the foundation for subsequent multi-stage feature processing.

[0057] Flatten the patch sequence The input is a three-stage EfficientViT Block module (including two downsampling operations). Through the synergistic effect of depthwise convolution, cascaded group attention, and feedforward neural networks, global semantic features of the image are extracted step by step to obtain feature vectors for the professional modality image and auxiliary modality image. The feature vector for the professional modality image includes global feature vectors and local feature vectors for the professional modality image. The feature vector for the auxiliary modality image includes global feature vectors and local feature vectors for the auxiliary modality image. The specific process and formulas are as follows: The EfficientViT Block1 structure consists of the following modules connected in series along the forward propagation direction: a professional modality deep convolutional submodule (DWConv1) → a third feedforward neural network submodule (FFN3) → a cascaded grouped attention submodule (CGA1) → an auxiliary modality deep convolutional submodule (DWConv2) → a fourth feedforward neural network submodule (FFN4). The parameters and functions of each submodule are as follows: Depthwise convolution (DWConv1 / DWConv2): uses a depthwise convolution with a kernel size of k=3×3 and a stride of s=1.

[0058] Feedforward Neural Networks (FFN3 / FFN4): Employ a linear transformation → GELU activation → linear transformation structure, with a hidden layer dimension of 4 × C_0 = 3072 (4 times the number of channels to balance nonlinear expression and computational cost).

[0059] Cascaded Grouped Attention (CGA1): Divides the input features into G=12 groups according to channels ( =768÷12=64), each group has 64 channels. Self-attention is calculated independently for each group and then concatenated. The number of self-attention heads is 8. The computational complexity is reduced by grouping attention, while capturing the local correlation of patches within the group.

[0060] Downsampling 1: Using a 1×1 convolution with stride = 2, the number of patches for the output features in the professional modality stage is reduced from 196 (14×14) to 49 (7×7), while the number of channels remains unchanged. =768, the output is .

[0061] Auxiliary Modal Stage (EfficientViT Block2+ Downsampling 2): Reuses the professional modal stage structure, with the input being... (1×49×768), after downsampling, the number of patches is reduced to 16 (4×4), the number of channels remains 768, the receptive field is expanded to a 64×64 area, and the output... ; Phase 3 (EfficientViT Block 3): No downsampling, input is... (1×16×768), through 3 iterations of EfficientViT Block, the global semantic association is further strengthened, and deep features are output. (1×16×768).

[0062] For the output of the third stage Global average pooling (GAP) is performed to fuse the 16 patch features into a single global feature vector. The formula for calculating the global feature vector of a professional modality image is as follows:

[0063] in, For professional modal image global feature vectors, This represents the total number of patches output in the third phase. This is the nth patch feature output from the third stage.

[0064] Similarly, auxiliary modal image standard data After the above process, the global feature vector of the auxiliary modality image is output. (1×768).

[0065] To capture local details in the image, after the auxiliary modality stage output of EfficientViT (before the third-stage downsampling), high-resolution patch features are preserved and local feature vectors are extracted to avoid detail loss caused by the third-stage global pooling. The specific process is as follows: Output characteristics of the auxiliary mode stage Patches with significant texture / grayscale changes (likely lesion areas) are retained through variance screening. The formula for calculating the feature set of the filtered local candidate patches is as follows:

[0066] in, For the nth patch feature, Let be the channel variance of the nth patch feature. A larger variance indicates that the patch contains more detail. As the filtering threshold, 10% of the maximum variance of all patches is used. This is the feature set of the filtered local candidate patches.

[0067] It should be noted that the screening threshold is based on experimental verification. On the dermatology dataset, this value can effectively screen out more than 95% of significant texture patches. The screening threshold can be adaptively adjusted through training data and is not a fixed value.

[0068] Perform Local Max Pooling (LMP) on the candidate patch features to fuse the detailed information of the candidate regions and generate local feature vectors. The formula for calculating the local feature vector of a professional modality image is as follows:

[0069] in, For professional modal image local feature vectors, Take the maximum value of candidate patches by channel. This is the nth local patch feature of the output features in the auxiliary modality phase.

[0070] Similarly, auxiliary modal image standard data After the above process, the local feature vector of the auxiliary modality image is output. .

[0071] The text encoding data is enhanced and encoded based on the T5 model and the bidirectional transformer model to obtain the enhanced text feature vector. The specific process is as follows: The T5 model (pre-trained corpus includes medical text, adapted for case semantic generation) is used as input to generate supplementary descriptive text related to the categories of the target classification task, thus solving the problem of incomplete semantics in the original case text. Taking the skin disease diagnosis task as an example, the specific formula and parameters are as follows: To guide T5 in generating high-quality category descriptions, the prompt template is designed as follows: Prompt="Generate descriptions of skin disease categories". The typical characteristics are described, including the lesion's morphology, color, and location. For the k-th category of the target task, the prompt length should be controlled within 30 characters to avoid the T5 generation being too long and deviating from the topic.

[0072] Input the Prompt into the T5 model to generate category description text. The formula for calculating the category description text is:

[0073] in, The maximum length of the generated text is set to 50. This is the beam search width, used to balance the diversity and accuracy of the generated text; its value is 4. Describe the category text.

[0074] Category description text Perform the same data cleaning-segmentation-encoding process as step S1 to generate standardized codes. (1×512) and (1×512), to ensure consistency with the original text encoding format.

[0075] Encode the original text With generated text encoding The sequences are concatenated to form an enhanced text code, which is then input into the bidirectional transformer model BERT (pre-trained on the medical text corpus PubMedBERT to improve case semantic understanding) for encoding. The specific process is as follows: Since the maximum input length of BERT is 512, to avoid the fused sequence being too long, a strategy of truncating the original text w+ and concatenating the resulting text is adopted. The formula for calculating the fused attention mask sequence is as follows: ; ; in, The truncation length of the original text encoding. The input ID sequence after fusion. This is the fused attention mask sequence.

[0076] Fusion coding ( , The input is a BERT model with a 12-layer Transformer encoder, 768-dimensional hidden layers, and 12 attention heads. The output is a feature sequence H with dimensions 1×512×768. The BERT output sequence features are transformed into fixed-dimensional text feature vectors. A token feature + global average pooling fusion pooling strategy is used to enhance the robustness of the features. The formula for calculating the enhanced text feature vector is as follows:

[0077] in, This is the feature of the 0th token output by BERT. To determine the number of valid tokens in the merged text. This is a global average pooling feature. To integrate the text features after pooling, and For linear transformation parameters, For layer normalization operation, To enhance text feature vectors.

[0078] S3: Construct dual-branch learnable cue words, and perform cross-modal semantic alignment on the professional modality image feature vector, the auxiliary modality image feature vector, and the enhanced text feature vector based on the dual-branch learnable cue words to obtain professional modality image alignment feature vector, auxiliary modality image alignment feature vector, and text alignment feature vector.

[0079] To meet the alignment requirements of the global-local dual-granularity image features and text enhancement features output from step S2, a global Prompt branch and a local Prompt branch are constructed. Both types of Prompts are learnable vector sequences (unlike fixed text Prompts, learnable Prompts can adapt to the semantic distribution of the target task through training). They respectively capture cross-modal global semantic associations and local detail associations. The structure, dimensions, and initialization strategies of the two types of Prompts are consistent, and the only difference lies in the adaptation objects in the subsequent alignment stage. The specific design is as follows: Global Prompt ( ) and local Prompt ( ) is represented as: ; ; in, Let m be the m-th learnable vector in the global Prompt. Let m be the m-th learnable vector of the local prompt. This is a global prompt sequence used to align global image features with global text semantics. This is a local prompt sequence used to align local image features with semantic details in the text. The length of the Prompt sequence. The dimension of the Prompt vector.

[0080] To avoid gradient explosion caused by a large difference in distribution between the Prompt vector and the feature vector during the initial training phase, a zero-mean Gaussian distribution is used for initialization, calculated as follows: ,in, The standard deviation is 0.02. This standard deviation allows for a quick entry into the effective training phase and avoids semantic shifts caused by excessively large initial values.

[0081] To incorporate the semantic information of the text enhancement features into the Prompt sequence, the enhanced text feature vector from step S2 is... As a semantic guidance vector, it is element-wise added to each vector in the Prompt sequence. The formula for calculating the global and local Prompt sequences that fuse textual semantics is as follows: ; ; in, A global Prompt sequence that integrates textual semantics. A local Prompt sequence for incorporating textual semantics.

[0082] It should be noted that element-level addition injects the global semantics of the text into each Prompt vector, so that the Prompt has text semantic guidance from the early stage of training, avoiding the bias of "emphasizing images and neglecting text" when aligning across modalities.

[0083] Based on the bi-branch learnable cue words, cross-modal semantic alignment is performed on the professional modality image feature vector, the auxiliary modality image feature vector, and the enhanced text feature vector to obtain the professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector, as detailed below: Since the dimensions of the image feature vector and the Prompt sequence are 1×768 and 16×768 respectively, the image features need to be mapped to a feature matrix matching the dimensions of the Prompt sequence through linear projection. The calculation formula is as follows: ; ; in, This is the projection result of the global feature vector of the professional modality image. This is the projection result of the local feature vectors of a professional modal image. To aid in the projection results of the global feature vector of the modal image, To aid in the projection of local feature vectors of modal images, and The weight matrix and bias terms for the global projection are given. and Here are the weight matrix and bias terms for the local projection.

[0084] It should be noted that the purpose of projection is to transform the image feature vector in single-vector form into a matrix with the same structure as the Prompt sequence, so that the semantic similarity of each token can be directly calculated, avoiding the loss of semantic information caused by the calculation of single-vector similarity.

[0085] Cosine similarity is used to measure the semantic consistency between the projected image feature vector and the Prompt sequence. The cosine similarity value ranges from -1 to 1. The closer the value is to 1, the more semantically consistent it is; the closer it is to -1, the more semantically divergent it is. The calculation formula is as follows: ; ; in, This is the similarity matrix between the projection result of the global feature vector of the professional modality image and the global Prompt. To assist in the similarity matrix between the projection results of the global feature vectors of the modal image and the global Prompt, This is the similarity matrix between the projection results of local feature vectors of a professional modality image and the global Prompt. This is the similarity matrix between the projection results of local feature vectors of a professional modality image and the global Prompt. It is the L2 norm, used to normalize the characteristic matrix.

[0086] To enhance cross-modal semantic consistency, a contrastive loss function is designed. By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, the semantic alignment between the Prompt vector and the projected image feature vector is optimized. The formula for calculating the total cross-modal alignment loss is as follows: ; ; ; in, The total cross-modal alignment loss is represented by a smaller loss value, indicating better cross-modal semantic alignment performance. For global alignment loss, For local alignment loss, and The number of positive sample pairs is 2. , Let be the similarity between the global projection feature vector of the i-th image and the t-th token. Let be the similarity between the local projection feature vector of the i-th image and the t-th token. This is the temperature coefficient, with a value of 0.07. The sum of the similarity indices for negative sample pairs is used to suppress the similarity of negative sample pairs and avoid cross-modal semantic confusion.

[0087] It should be noted that positive sample pairs are defined as feature vector-Prompt pairs of the same modality type, while negative sample pairs are defined as feature vector-Prompt pairs of different modality types.

[0088] During training, the minimum value is minimized through backpropagation. The learnable Prompt vector and projection parameters are updated synchronously until the loss converges (the convergence threshold is set to...). That is, the change in loss over 5 consecutive epochs is less than (Stop alignment optimization when needed).

[0089] Even after alignment, the image feature vectors may still exhibit distribution shifts or noise interference. Therefore, a combination of residual connections and layer normalization is needed to optimize the feature representation, ensuring the stability and discriminative power of the output features. The specific process is as follows: The projected global feature vector is residually joined with the aligned Prompt feature vector, and then layer normalization is used to eliminate distribution offset, resulting in the professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector. The professional modality image alignment feature vector includes the professional modality image global alignment feature vector and the professional modality image local alignment feature vector. The auxiliary modality image alignment feature vector includes the auxiliary modality image global alignment feature vector and the auxiliary modality image local alignment feature vector. The formulas for calculating the professional modality image global alignment feature vector and the auxiliary modality image global alignment feature vector are as follows: ; ; in, For professional modal image global alignment feature vectors, To assist in the global alignment of feature vectors in modal images, This is the weight coefficient for the Prompt feature, with a value of 0.05. The global Prompt feature is weighted by similarity. This is a layer normalization operation.

[0090] It should be noted that, This is a simplified representation of multiplication; in actual calculations, multiplication is first performed... Perform row normalization to ensure its dimensions are compatible with the data. The form of multiplication is essentially based on similarity pairs. Each vector is weighted and adjusted without changing its core dimensional structure.

[0091] The optimization process for local alignment feature vectors is the same as that for global alignment. After alignment optimization, local alignment feature vectors for professional modal images are obtained. Auxiliary modality image local alignment feature vector .

[0092] Enhance text feature vectors The aligned image feature dimensions need to be adapted. Dimensional uniformity is achieved through repeated expansion and layer normalization. The formula for calculating the text alignment feature vector is:

[0093] in, For text alignment feature vectors, This indicates that the text feature vector is repeated along the token dimension. Next, generate the feature matrix.

[0094] S4: The professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector are fused by a dynamic attention fusion method based on category decoupling and nonnegative matrix factorization to obtain a fused feature vector.

[0095] The professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector are fused using a dynamic attention fusion method based on category decoupling and nonnegative matrix factorization to obtain a fused feature vector. The specific process is as follows: A category decoupling fusion module is constructed. Through the process of "category-related feature extraction + category-independent feature expansion + linear integration", the fused features retain both category specificity and category adaptability. The specific design is as follows: Category-related features focus on the known category semantics of the target task, and category semantic projection is performed on the aligned feature vector from step S3:

[0096] in, The class projection results of the globally aligned feature vectors of professional modal images. To assist in the global alignment of feature vectors in modal images with category projection results, The class projection results of the local alignment feature vectors of professional modal images. The class projection results of the local alignment feature vectors of professional modal images. The class projection results of the text-aligned feature vectors. and The weight matrix and bias terms for class projection. , This represents the number of category-related feature channels, with a value of 512.

[0097] Category-independent features are used to adapt to scenarios with unknown categories or changing categories. They are generated by introducing a learnable category-independent prompt sequence and projecting it. The formula for calculating the category-independent feature matrix is ​​as follows:

[0098] in, The class-independent Prompt sequence is initialized using a zero-mean Gaussian distribution. and The weight matrix and bias terms for class-independent projections. , This represents the number of category-independent feature channels, with a value of 256. The feature matrix is ​​class-independent. It follows a zero-mean Gaussian distribution.

[0099] By integrating class-related and class-independent features into a unified feature matrix through a linear layer, the formula for calculating the class decoupling result of the i-th feature is as follows:

[0100] in, The category decoupling result for the i-th feature. For splicing operations, and To integrate the weight matrix and bias terms of the linear layer, Xavier initialization is used. , For activation function, This is the category projection result for the i-th feature.

[0101] Based on features after category decoupling A dynamic fusion strategy combining cross-attention and semantic weight adaptation is adopted to focus the fusion process on features with high semantic relevance. The specific process is as follows: Decoupling features by text category To guide this process, cross-attention fusion is performed with features decoupled from the four image categories respectively: ; ; ; in, For querying the matrix, The key matrix, For value matrices, For the dimension of attention head, This is used to scale the attention score, preventing excessively large values ​​from causing Softmax saturation. For professional modal image global-text fusion features after cross-attention fusion, For global-text fusion features of auxiliary modal images after cross-attention fusion, For professional modal image local-text fusion features after cross-attention fusion, This refers to the local-text fusion features of the auxiliary modality image after cross-attention fusion.

[0102] It should be noted that setting the text category decoupling feature as K allows the image feature (Q) to accurately locate the corresponding text semantic anchor point by calculating its similarity with K (such as matching the visual feature of "irregular edges" in the image with the text description of "irregular edges"); setting it as V allows the image feature to be directly integrated into the text content that is semantically consistent with K during weighted fusion, realizing "semanticization of visual features" and avoiding noise or redundant information in the image from dominating the fusion result.

[0103] Based on the cross-modal similarity matrix from step S3, dynamic weights are calculated to weight the cross-attention fusion features:

[0104] in, The dynamic weight of the i-th fused feature is... Let be the similarity between the i-th image feature and the t-th token of the Prompt. For average similarity, The feature matrix after dynamic fusion. This represents the number of tokens in the Prompt.

[0105] To ensure cross-modal consistency of features after dynamic fusion, an intramodal loss function is introduced:

[0106] in, The loss is dynamic fusion loss; the smaller the loss, the better the consistency after fusion of the various modalities. is the L2 norm squared, used to measure the difference between the fused features of a single modality and the weighted integrated features.

[0107] This formula avoids fusion imbalance caused by excessive weighting of a certain modality feature (such as only professional modality image features dominating the fusion result), and ensures a balanced contribution of the four image features and text features.

[0108] During training, the minimum value is minimized through backpropagation. Simultaneously update the cross-attention parameters and parameters related to dynamic weight calculation until the loss converges (the convergence threshold is set to...). ).

[0109] Dynamically fused feature matrix There may still be issues with modal redundancy or insufficient discriminative power. Feature representation is enhanced through a combination of nonnegative matrix factorization (NMF) commonality extraction, residual connections, and layer normalization, as detailed below: Dynamic fusion feature matrix It may contain negative values; global Min-Max normalization will be used to normalize them. All elements are linearly mapped to the [0,1] interval. This method preserves the relative magnitude relationship between features (without destroying discriminability) and ensures that the NMF input is non-negative. This step avoids decomposition failure caused by negative values. To avoid individual elements remaining negative due to extreme noise, a non-negativity verification step is added before the NMF input, replacing any potentially tiny negative elements with small positive values. This satisfies the non-negativity requirement of NMF while avoiding the gradient vanishing problem caused by zero values.

[0110] Using nonnegative matrix factorization (NMF) from To extract common features across multiple modalities, the core idea of ​​Non-Modal Matrix Factorization (NMF) is to decompose a high-dimensional non-negative input matrix into the product of two low-dimensional non-negative matrices, as shown in the formula: ,in, The matrix is ​​a base matrix, and the column vectors are modal common bases. Each column vector represents a core feature pattern shared across modalities (such as the common semantics of "lesion edge texture" and "lesion color description in text"). Let be the coefficient matrix, and let the row vectors be the coefficients of the common basis, representing the projection intensity of each token feature onto the common basis.

[0111] The optimization objective of NMF is to minimize the reconstruction error, ensuring that the decomposed matrix can approximately reconstruct the input features while extracting commonalities. The formula is as follows: ,in, For the Frobenius norm, iterative optimization using the Lee-Seung alternating multiplication method is employed, with the number of iterations... =100, the change in reconstruction error over three consecutive iterations is less than the threshold. Finally, the modal common feature matrix extracted by NMF is obtained. .

[0112] Dynamically fuse feature matrices With common feature matrix Perform residual connections to supplement mode-specific information and avoid feature loss caused by NMF. The formula for calculating the enhanced feature matrix after residual integration is as follows:

[0113] in, This is the enhanced feature matrix after residual integration. This is the common feature weighting coefficient, with a value of 0.3.

[0114] right The execution layer normalizes the feature distribution to eliminate bias, and then global average pooling transforms the 16×768 matrix features into a 1×768 fused feature vector. The formula for calculating the fused feature vector is as follows: ; ; in, for The channel mean, for The channel variance, To avoid tiny values ​​where the denominator is 0, the value is selected. , The enhanced features are those obtained after normalization. To fuse feature vectors.

[0115] S5: Input the fused feature vector into the pre-trained ensemble learning model and output the final classification result to achieve classification of multimodal data.

[0116] This step employs an ensemble learning framework, integrating the inference results of three base classifiers: a fully connected layer classifier, a support vector machine (SVM), and a classification and regression tree (CART). These three classifiers are trained and infer independently, each outputting a class probability distribution. The complementarity between the classifiers enhances the overall generalization ability. The fused feature vector is then input into a pre-trained multi-classifier, yielding the final classification result for the target object. The specific process is as follows: The fully connected layer classifier is built on a deep learning framework and is specifically adapted to high-dimensional target features. It maps to the class space through a non-linear transformation. The formula for calculating the output probability distribution of the fully connected layer classifier is:

[0117] in, To fuse feature vectors, and The weight matrix and bias terms of the professional modal hidden layer. For activation function, This represents the probability of Dropout failure, with a value of 0.3. and The weight matrix and bias terms of the output layer are as follows: This represents the output probability distribution of the fully connected layer classifier.

[0118] An SVM classifier employing a radial basis function (RBF) kernel is specifically designed to address the linear inseparability of high-dimensional features and is adaptable to various applications. Given its 768-dimensional high-dimensionality, the formula for calculating the output probability distribution of SVM is:

[0119] in, For the fused feature vector of the i-th sample in the training set, This is the bandwidth parameter for the RBF core, with a value of 0.001. This is the raw output of SVM. The probability calibration function transforms the raw output of the SVM into a probability distribution. This represents the output probability distribution of the SVM.

[0120] The Classification and Regression Tree (CART) is used as the third classifier, with node splitting based on the Gini index to emphasize the interpretability of the classification results (e.g., in medical diagnosis where the classification criteria need to be clearly defined). The formula for calculating the output probability distribution of the decision tree is as follows:

[0121] in, For the sample set corresponding to the decision tree node, For the subset of the sample set that belongs to the k-th class, The Gini index measures the purity of node samples. for The set of leaf node samples that are included during inference. Let be the number of samples belonging to the k-th class in this leaf node. This represents the output probability distribution of the decision tree.

[0122] After independent reasoning by the three classifiers, three probability distributions are output: (Fully connected layer) (SVM) (Decision tree).

[0123] To integrate the advantages of the three classifiers, a weighted voting strategy based on validation set accuracy is adopted. This avoids the problem of weak classifiers diluting the results of strong classifiers in simple voting, and highlights the contribution of high-confidence classifiers through dynamic weight allocation. The formula is as follows: The weights are calculated based on the classifier's accuracy on the validation set, ensuring a positive correlation between the weights and the classifier's performance. The formula is as follows:

[0124] in, The weights of the classifier, These correspond to fully connected layer, SVM, and decision tree, respectively. This represents the accuracy of the classifier on the validation set.

[0125] Every 10 epochs of training, the validation set accuracy is recalculated and the weights are updated to avoid insufficient adaptability caused by fixed weights (such as automatic weight adjustment after SVM performance improves in the later stages of training).

[0126] The probability distributions of the three classifiers are summed by weights to obtain the preliminary probability distribution and classification labels after ensemble. The formula for calculating the probability distribution after ensemble is as follows:

[0127] in, The probability distribution after integration. For preliminary classification labels, Let be the integration probability that the target object belongs to the k-th class.

[0128] The integrated probability distribution avoids the bias of a single classifier. For example, a fully connected layer may misjudge early lesions, SVM can capture subtle feature differences, and decision trees can be corrected through rules. The weighted combination of the three improves the reliability of classification.

[0129] During training, to enhance the consistency of ensemble decisions, an ensemble loss function is introduced. The formula for calculating the mean squared error loss is as follows:

[0130] in, For the true labels of the target object, The mean squared error loss is used; the smaller the loss, the closer the ensemble probability distribution is to the true label.

[0131] By simultaneously optimizing the parameters of the three classifiers (fully connected layer weights, SVM kernel parameters, and decision tree splitting threshold) through backpropagation, we can ensure that the output directions of the three classifiers are consistent and avoid the results canceling out after integration.

[0132] Probability distribution after integration There may be issues with the probability distribution being too steep or too flat (e.g., the output probability of a fully connected layer is concentrated in 0 or 1, lacking intermediate confidence levels). Temperature scaling is used for probability calibration to improve the reliability and usability of the results. The formula for calculating the probability of belonging to the k-th class after calibration is as follows:

[0133] in, This represents the probability of belonging to the k-th class after calibration. This is the temperature calibration coefficient, with a value range of [0.5, 2.0]. To integrate the logarithm of probabilities, avoid numerical overflow, and ensure stable exponential operations.

[0134] The formula for calculating the calibration loss function is:

[0135] in, To determine the number of samples in the validation set, Let i be the true label of the i-th validation set sample. Let be the calibrated probability of the i-th validation set sample. The cross-entropy calibration loss is used; the smaller the loss, the better the calibrated probability matches the true label.

[0136] During training, the three classifiers and ensemble weights are fixed, and only optimization is performed. until Convergence (convergence threshold) This ensures that the calibration process does not affect the performance of the optimized classifier.

[0137] After calibration, the final classification result is output, including the classification label and the post-calibration confidence score, as shown in the following formula: ; ; in, For the final category label, This represents the final confidence level.

[0138] The classification labels use standardized medical terminology, and the confidence level is retained to two decimal places to facilitate medical record recording and statistical analysis.

[0139] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, the phrase "comprising an element defined as..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0140] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal data classification method based on a large model, characterized in that: Includes the following steps: Step S1: Collect professional modal image data, auxiliary modal image data, and text data, and preprocess them to obtain professional modal image standard data, auxiliary modal image standard data, and text encoded data; Step S2: Use a visual feature extractor to extract features from the professional modality image standard data and the auxiliary modality image standard data respectively, and obtain the professional modality image feature vector and the auxiliary modality image feature vector respectively. Based on the T5 model and the bidirectional transformer model, perform text enhancement and encoding on the text encoding data to obtain the enhanced text feature vector. Step S3: Construct bi-branch learnable cue words. Based on the bi-branch learnable cue words, perform cross-modal semantic alignment on the professional modality image feature vector, the auxiliary modality image feature vector, and the enhanced text feature vector to obtain the professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector. Step S4: The professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector are fused by a dynamic attention fusion method based on category decoupling and nonnegative matrix factorization to obtain a fused feature vector; Step S5: Input the fused feature vector into the pre-trained ensemble learning model and output the final classification result to achieve classification of multimodal data.

2. The multimodal data classification method based on a large model according to claim 1, characterized in that: The process of acquiring professional modal image data, auxiliary modal image data, and text data, and preprocessing them to obtain professional modal image standard data, auxiliary modal image standard data, and text encoded data, includes the following specific steps: Acquire professional modal image data Modality1, auxiliary modal image data Modality2, and text data. ; The resolutions of Modality1 and Modality2 are unified to 224×224×3. A bilinear interpolation algorithm is used to achieve the size transformation, as shown in the following formula: ； in, For the original image data, The image data after resolution standardization. and These are the coordinates of the neighboring pixels corresponding to the target pixel in the original image. and These are the weighting coefficients for bilinear interpolation, used to smoothly transition neighboring pixel values. and To unify the pixel row and column coordinates of the image, For the RGB channels of the image; Three enhancement operations—random horizontal flipping, adaptive brightness adjustment, and small-angle rotation—were performed on Modality1 and Modality2 after resolution unification to obtain professional modal image standard data. Standard data for auxiliary modal images ; For the collected text data Data cleaning, word segmentation, and encoding are performed to obtain the final encoded text data. .

3. The multimodal data classification method based on a large model according to claim 2, characterized in that: The step of using a visual feature extractor to extract features from the professional modality image standard data and the auxiliary modality image standard data, respectively, to obtain the professional modality image feature vector and the auxiliary modality image feature vector, includes the following steps: Professional modal image standard data The sequence is transformed into a processable patch sequence. A convolutional operation is used to map pixel-level data to patch-level features. The formula for calculating the feature map after patch embedding is as follows: ； ； in, For professional modal image standard data, This represents the patch size, with a value of 16. This represents the number of feature channels after patch embedding. The feature map after patch embedding. This is the flattened patch sequence. The dimension of the feature map after patch embedding. Total number of patches; Flatten the patch sequence The input is a three-stage EfficientViT Block module, which, through the synergistic effect of deep convolution, cascaded group attention, and feedforward neural networks, progressively extracts global semantic features of the image to obtain a professional modality image feature vector. This professional modality image feature vector includes both global and local feature vectors. The third-stage output of the EfficientViT Block module Perform global average pooling to fuse the features into a single global feature vector. The formula for calculating the global feature vector of a professional modality image is as follows: ； in, For professional modal image global feature vectors, This represents the total number of patches output in the third phase. This is the nth patch feature output from the third stage; Similarly, output auxiliary modal image standard data. Global feature vector of auxiliary modal image ; Output characteristics of the auxiliary modal phase of the EfficientViT Block module Patches with significant texture changes are retained through variance screening. The formula for calculating the feature set of the selected local candidate patches is as follows: ； in, For the nth patch feature, Let be the channel variance of the nth patch feature. A larger variance indicates that the patch contains more detail. As the filtering threshold, 10% of the maximum variance of all patches is used. This is the set of local candidate patch features after filtering. Local max pooling is performed on the candidate patch features to fuse the detailed information of the candidate regions and generate local feature vectors. The formula for calculating the local feature vector of a professional modality image is as follows: ； in, For professional modal image local feature vectors, Take the maximum value of candidate patches by channel. This is the nth local patch feature of the output features in the auxiliary modality phase; Similarly, output auxiliary modal image standard data. Auxiliary modal image local feature vector .

4. The multimodal data classification method based on a large model according to claim 3, characterized in that: The text enhancement and encoding of text-encoded data based on the T5 model and bidirectional transformer model to obtain enhanced text feature vectors includes the following steps: The T5 model is used, taking the category set of the target classification task as input, to generate supplementary descriptive text related to the categories, thus solving the problem of incomplete semantics in the original case text. Taking the skin disease diagnosis task as an example, the specific formula and parameters are as follows: To guide T5 in generating high-quality category descriptions, the Prompt template is designed as follows: Prompt="Generate descriptions of skin disease categories" The typical characteristics are described, including the lesion's morphology, color, and location. For the kth category of the target task, the Prompt length should be controlled within 30 characters to avoid the T5 generation being off-topic due to excessive length. Input the Prompt into the T5 model to generate category description text. The formula for calculating the category description text is: ； in, The maximum length of the generated text is set to 50. This is the beam search width, used to balance the diversity and accuracy of the generated text; its value is 4. Describe the category text; Category description text Perform data cleaning, word segmentation, and encoding processes to generate standardized codes. and Ensure consistency with the original text encoding format; Encode the original text With generated text encoding The sequences are concatenated to form an enhanced text code, which is then input into the BERT bidirectional transformer model for encoding. Since the maximum input length of BERT is 512, to avoid the fused sequence being too long, a strategy of truncating the original text w+ and concatenating the resulting text is adopted. The formula for calculating the fused attention mask sequence is as follows: ； ； in, The truncation length of the original text encoding. The input ID sequence after fusion. The fused attention mask sequence; Fusion coding ( , The input is a BERT model with a 12-layer Transformer encoder, 768-dimensional hidden layers, and 12 attention heads. The output is a feature sequence H with dimensions 1×512×768. The BERT output sequence features are transformed into fixed-dimensional text feature vectors. A token feature + global average pooling fusion pooling strategy is used to enhance the robustness of the features. The formula for calculating the enhanced text feature vector is as follows: ； in, This is the feature of the 0th token output by BERT. To determine the number of valid tokens in the merged text. This is a global average pooling feature. To integrate the text features after pooling, and For linear transformation parameters, For layer normalization operation, To enhance text feature vectors.

5. The multimodal data classification method based on a large model according to claim 4, characterized in that: The construction of the dual-branch learnable prompt words includes the following steps: A global Prompt branch and a local Prompt branch are constructed. Both types of Prompts are learnable vector sequences, capturing global semantic associations and local detail associations across modalities, respectively. The structure, dimensions, and initialization strategies of the two types of Prompts are consistent, differing only in the adaptation objects in the subsequent alignment stage. The specific design is as follows: Global Prompt ( ) and local Prompt ( ) is represented as: ； ； in, Let m be the m-th learnable vector in the global Prompt. Let m be the m-th learnable vector of the local prompt. This is a global prompt sequence used to align global image features with global text semantics. This is a local prompt sequence used to align local image features with semantic details in the text. The length of the Prompt sequence. The dimension of the Prompt vector; To avoid gradient explosion caused by a large difference in distribution between the Prompt vector and the feature vector during the initial training phase, a zero-mean Gaussian distribution is used for initialization, calculated as follows: ,in, The standard deviation is 0.

02. This standard deviation can quickly enter the effective training phase and avoid semantic shift caused by excessively large initial values. To incorporate the semantic information of the text enhancement features into the Prompt sequence, the enhanced text feature vector from step S2 is... As a semantic guidance vector, it is element-wise added to each vector in the Prompt sequence. The formula for calculating the global and local Prompt sequences that fuse textual semantics is as follows: ； ； in, A global Prompt sequence that integrates textual semantics. A local Prompt sequence for incorporating textual semantics.

6. The multimodal data classification method based on a large model according to claim 5, characterized in that: The method of performing cross-modal semantic alignment of the professional modality image feature vector, the auxiliary modality image feature vector, and the enhanced text feature vector based on the dual-branch learnable cue words to obtain the professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector includes the following steps: Since the dimensions of the image feature vector and the Prompt sequence are 1×768 and 16×768 respectively, the image features need to be mapped to a feature matrix matching the dimensions of the Prompt sequence through linear projection. The calculation formula is as follows: ； ； in, This is the projection result of the global feature vector of the professional modality image. This is the projection result of the local feature vectors of a professional modal image. To aid in the projection results of the global feature vector of the modal image, To aid in the projection of local feature vectors of modal images, and The weight matrix and bias terms for the global projection are given. and Here are the weight matrix and bias terms for the local projection; Cosine similarity is used to measure the semantic consistency between the projected image feature vector and the Prompt sequence. The cosine similarity value ranges from -1 to 1. The closer the value is to 1, the more semantically consistent it is; the closer it is to -1, the more semantically divergent it is. The calculation formula is as follows: ； ； in, This is the similarity matrix between the projection result of the global feature vector of the professional modality image and the global Prompt. To assist in the similarity matrix between the projection results of the global feature vectors of the modal image and the global Prompt, This is the similarity matrix between the projection results of local feature vectors of a professional modality image and the global Prompt. This is the similarity matrix between the projection results of local feature vectors of a professional modality image and the global Prompt. The L2 norm is used to normalize the characteristic matrix. A global Prompt sequence that integrates textual semantics. A local Prompt sequence that integrates textual semantics; To enhance cross-modal semantic consistency, a contrastive loss function is designed. By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, the semantic alignment between the Prompt vector and the projected image feature vector is optimized. The formula for calculating the total cross-modal alignment loss is as follows: ； ； ； in, The total cross-modal alignment loss is represented by a smaller loss value, indicating better cross-modal semantic alignment performance. For global alignment loss, For local alignment loss, and The number of positive sample pairs is 2. , Let be the similarity between the global projection feature vector of the i-th image and the t-th token. Let be the similarity between the local projection feature vector of the i-th image and the t-th token. This is the temperature coefficient, with a value of 0.

07. The sum of the similarity indices of negative sample pairs is used to suppress the similarity of negative sample pairs and avoid cross-modal semantic confusion. During training, the minimum value is minimized through backpropagation. The learnable Prompt vector and projection parameters are updated synchronously until the loss converges. Even after alignment, the image feature vectors may still exhibit distribution shifts or noise interference, requiring optimization of the feature representation through residual connections and layer normalization to ultimately obtain the local alignment feature vectors for the professional modality image. Auxiliary modality image local alignment feature vector and text alignment feature vectors .

7. The multimodal data classification method based on a large model according to claim 6, characterized in that: The specific steps for optimizing feature representation through residual connections and layer normalization are as follows: The projected global feature vector is residually joined with the aligned Prompt feature vector, and then layer normalization is used to eliminate distribution offset, resulting in the professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector. The professional modality image alignment feature vector includes the professional modality image global alignment feature vector and the professional modality image local alignment feature vector. The auxiliary modality image alignment feature vector includes the auxiliary modality image global alignment feature vector and the auxiliary modality image local alignment feature vector. The formulas for calculating the professional modality image global alignment feature vector and the auxiliary modality image global alignment feature vector are as follows: ； ； in, For professional modal image global alignment feature vectors, To assist in the global alignment of feature vectors in modal images, This is the weight coefficient for the Prompt feature, with a value of 0.

05. The global Prompt feature is weighted by similarity. For layer normalization operation; The optimization process for local alignment feature vectors is the same as that for global alignment. After alignment optimization, local alignment feature vectors for professional modal images are obtained. Auxiliary modality image local alignment feature vector ; Enhance text feature vectors The aligned image feature dimensions need to be adapted. Dimensional uniformity is achieved through repeated expansion and layer normalization. The formula for calculating the text alignment feature vector is: ； in, For text alignment feature vectors, This indicates that the text feature vector is repeated along the token dimension. Next, generate the feature matrix. To enhance text feature vectors.

8. The multimodal data classification method based on a large model according to claim 7, characterized in that: The method of fusing the professional modality image alignment feature vector, the auxiliary modality image alignment feature vector, and the text alignment feature vector using a dynamic attention fusion method based on category decoupling and nonnegative matrix factorization to obtain a fused feature vector includes the following specific steps: A category decoupling fusion module is constructed. Through a process of category-related feature extraction, category-independent feature expansion, and linear integration, the fused features retain both category specificity and category adaptability. The specific design is as follows: Category-related features focus on the known category semantics of the target task, and category semantic projection is performed on the aligned feature vector from step S3: ； in, The class projection results of the globally aligned feature vectors of professional modal images. To assist in the global alignment of feature vectors in modal images with category projection results, The class projection results of the local alignment feature vectors of professional modal images. The class projection results of the local alignment feature vectors of professional modal images. The class projection results of the text-aligned feature vectors. and The weight matrix and bias terms for class projection. , This represents the number of category-related feature channels, with a value of 512. Category-independent features are used to adapt to scenarios with unknown categories or changing categories. They are generated by introducing a learnable category-independent prompt sequence and projecting it. The formula for calculating the category-independent feature matrix is ​​as follows: ； in, The class-independent Prompt sequence is initialized using a zero-mean Gaussian distribution. and The weight matrix and bias terms for class-independent projections. , This represents the number of category-independent feature channels, with a value of 256. The feature matrix is ​​class-independent. This represents a Gaussian distribution with zero mean. Take 0.02; By integrating class-related and class-independent features into a unified feature matrix through a linear layer, the formula for calculating the class decoupling result of the i-th feature is as follows: ； in, The category decoupling result for the i-th feature. For splicing operations, and To integrate the weight matrix and bias terms of the linear layer, Xavier initialization is used. , For activation function, The category projection result for the i-th feature; Based on features after category decoupling A dynamic fusion strategy with cross-attention and semantic weight adaptation is adopted to focus the fusion process on features with high semantic relevance, resulting in a dynamically fused feature matrix. Common features are then extracted through non-negative matrix factorization, and a combination of residual connections and layer normalization is performed to enhance feature representation, as detailed below: Nonnegative matrix factorization (NMF) is used to extract the dynamically fused feature matrix. To extract common features across multiple modalities, the core idea of ​​Non-Modal Matrix Factorization (NMF) is to decompose a high-dimensional non-negative input matrix into the product of two low-dimensional non-negative matrices, as shown in the formula: ,in, The matrix is ​​a basis matrix, and the column vectors are modal common basis, where each column vector represents a core feature pattern shared across modalities. The coefficient matrix consists of row vectors representing the coefficients of the common basis, which represent the projection intensity of each token feature onto the common basis. The optimization objective of NMF is to minimize the reconstruction error, ensuring that the decomposed matrix can approximately reconstruct the input features while extracting commonalities. The formula is as follows: ,in, For the Frobenius norm, iterative optimization using the Lee-Seung alternating multiplication method is employed, with the number of iterations... =100, the change in reconstruction error over three consecutive iterations is less than the threshold. Finally, the modal common feature matrix extracted by NMF is obtained. ; Dynamically fuse feature matrices With common feature matrix Perform residual connections to supplement mode-specific information and avoid feature loss caused by NMF. The formula for calculating the enhanced feature matrix after residual integration is as follows: ； in, This is the enhanced feature matrix after residual integration. This is the common feature weighting coefficient, with a value of 0.

3. This is a matrix of common characteristics; right The execution layer normalizes the feature distribution to eliminate bias, and then global average pooling transforms the 16×768 matrix features into a 1×768 fused feature vector. The formula for calculating the fused feature vector is as follows: ； ； in, for The channel mean, for The channel variance, To avoid tiny values ​​where the denominator is 0, the value is selected. , The enhanced features are those obtained after normalization. To fuse feature vectors.

9. The multimodal data classification method based on a large model according to claim 8, characterized in that: The dynamic fusion strategy employing cross-attention and semantic weight adaptation yields a dynamically fused feature matrix. The specific steps are as follows: Decoupling features by text category To guide this process, cross-attention fusion is performed with features decoupled from the four image categories respectively: ； ； ； in, For querying the matrix, The key matrix, For value matrices, For the dimension of attention head, This is used to scale the attention score, preventing excessively large values ​​from causing Softmax saturation. For professional modal image global-text fusion features after cross-attention fusion, For global-text fusion features of auxiliary modal images after cross-attention fusion, For professional modal image local-text fusion features after cross-attention fusion, The auxiliary modality image local-text fusion features are obtained after cross-attention fusion; Based on the cross-modal similarity matrix from step S3, dynamic weights are calculated to weight the cross-attention fusion features: ； in, The dynamic weight of the i-th fused feature is... Let be the similarity between the i-th image feature and the t-th token of the Prompt. For average similarity, The feature matrix after dynamic fusion. This represents the number of tokens in the Prompt.

10. A multimodal data classification method based on a large model according to claim 9, characterized in that: The specific steps for inputting the fused feature vector into the pre-trained ensemble learning model and outputting the final classification result are as follows: By integrating the inference results of three base classifiers—a fully connected layer classifier, a support vector machine (SVM), and a classification and regression tree (CART)—each classifier is trained and infers independently, outputting its own category probability distribution. The complementarity between classifiers enhances the overall generalization ability. The fused feature vector is then input into a pre-trained multi-classifier to obtain the final classification result of the target object. The specific process is as follows: The fully connected layer classifier is built on a deep learning framework and is specifically adapted to high-dimensional target features. It maps to the class space through a non-linear transformation. The formula for calculating the output probability distribution of the fully connected layer classifier is: ； in, To fuse feature vectors, and The weight matrix and bias terms of the professional modal hidden layer. For activation function, This represents the probability of Dropout failure, with a value of 0.

3. and The weight matrix and bias terms of the output layer are as follows: This represents the output probability distribution of the fully connected layer classifier. An SVM classifier employing radial basis function kernels is specifically designed to address the linear inseparability of high-dimensional features and is adaptable to various applications. Given its 768-dimensional high-dimensionality, the formula for calculating the output probability distribution of SVM is: ； in, For the fused feature vector of the i-th sample in the training set, This is the bandwidth parameter for the RBF core, with a value of 0.

001. This is the raw output of SVM. The probability calibration function transforms the raw output of the SVM into a probability distribution. The output probability distribution of SVM, Radial basis functions; Using the Classification and Regression Tree (CART) as the third classifier, the formula for calculating the output probability distribution is as follows: ； in, For the sample set corresponding to the decision tree node, For the subset of the sample set that belongs to the k-th class, The Gini index measures the purity of node samples. for The set of leaf node samples that are included during inference. Let be the number of samples belonging to the k-th class in this leaf node. The output probability distribution of the decision tree; After independent reasoning by the three classifiers, three probability distributions are output: , , ; To integrate the advantages of the three classifiers, a weighted voting strategy based on validation set accuracy is adopted. This strategy uses dynamic weight allocation to highlight the contribution of the high-confidence classifier, as shown in the following formula: ； in, The weights of the classifier, These correspond to fully connected layer, SVM, and decision tree, respectively. This represents the accuracy of the classifier on the validation set. The probability distributions of the three classifiers are summed by weights to obtain the preliminary probability distribution and classification labels after ensemble. The formula for calculating the probability distribution after ensemble is as follows: ； in, The probability distribution after integration. For preliminary classification labels, Let be the integration probability that the target object belongs to the k-th class; Temperature scaling is used for probability calibration to improve the reliability and practicality of the results. The formula for calculating the probability of belonging to the k-th class after calibration is as follows: ； in, This represents the probability of belonging to the k-th class after calibration. This is the temperature calibration coefficient, with a value range of [0.5, 2.0]. To integrate the logarithm of probabilities, avoid numerical overflow, and ensure stable exponential operations.

Images