Training method and classification method of multi-modal classification model and multi-modal classification model

Abstract

The application discloses a training method and a classification method of a multi-modal classification model, and the multi-modal classification model. The training method comprises the following steps: performing feature extraction on a first image sample, a second image sample, a first health-related sample and a second health-related sample respectively; performing corresponding multi-granularity contrast learning and individual commonness representation learning on the extracted first image features and first health-related features, obtaining a contrast learning loss and an individual commonness representation learning loss, and iteratively updating the multi-modal classification model; performing individual commonness representation learning on the extracted second image features and second health-related features to obtain individual representation and commonness representation; performing weighted splicing on the individual representation and the commonness representation to obtain spliced features, performing classification processing on the spliced features, obtaining a classification loss, and iteratively updating the multi-modal classification model again. The two-stage training strategy improves the adaptability and classification accuracy of the model to complex multi-modal data.

Description

Technical Field

[0001] This application relates to the field of multimodal fusion technology, and in particular to a training method, classification method and multimodal classification model for a multimodal classification model. Background Technology

[0002] Currently, doctors typically combine multimodal information such as 3D medical images, electronic health records (EHRs), and medical reports for diagnosis, enabling a more comprehensive understanding of a patient's health status and improving diagnostic accuracy and efficiency. 3D imaging data, such as CT and MRI, can generate high-resolution images, accurately identifying internal details; EHRs provide comprehensive background information including clinical history, demographic data, and medication use. However, traditional clinical diagnostic methods heavily rely on physician experience and manual analysis, exhibiting significant limitations when handling complex multimodal data: when faced with a large volume of CT images, repeated review by doctors is not only time-consuming and labor-intensive but also prone to errors due to workload and time pressure; simultaneously, the quality of CT images is constrained by equipment and technology, also affecting diagnostic accuracy. Furthermore, traditional methods demand extremely high levels of physician expertise, making it difficult for non-specialists to participate, and the acquisition cycle for multi-source data (such as images, text, and laboratory indicators) is long, further increasing the workload of doctors.

[0003] To overcome these limitations, multimodal AI technology has emerged. However, existing AI models still have significant shortcomings when processing cross-modal data. Although they can integrate multiple data modalities, they struggle to uncover deep semantic relationships between different modalities due to a lack of effective modal interaction design. Furthermore, existing methods often neglect the multi-granularity of features and the quality differences within modalities, resulting in considerable room for improvement in interaction details and stability. Summary of the Invention

[0004] Therefore, it is necessary to provide a training method, a classification method, and a multimodal classification model to address the aforementioned technical problems and solve at least one of the problems existing in the prior art.

[0005] Firstly, a training method for a multimodal classification model is provided, applicable to a multimodal classification model, the method comprising:

[0006] Feature extraction is performed on the first image sample, the second image sample, the first health-related sample, and the second health-related sample respectively to obtain the first image feature, the second image feature, the first health-related feature, and the second health-related feature;

[0007] Multi-granularity contrastive learning and individual common representation learning are performed on the first image features and the first health-related features to obtain contrastive learning loss and individual common representation learning loss, so as to iteratively update the multimodal classification model based on the contrastive learning loss and individual common representation learning loss;

[0008] Individual and common representation learning is performed on the second image features and the second health-related features to obtain individual and common representations under different loss constraints;

[0009] The individual and common representations are weighted and concatenated to obtain concatenated features, and classification processing is performed based on the concatenated features to obtain classification loss;

[0010] Based on the classification loss, the multimodal classification model is iteratively updated again.

[0011] In one possible implementation, the step of performing multi-granularity contrastive learning and individual commonality representation learning on the first image features and the first health-related features to obtain contrastive learning loss and individual commonality representation learning loss includes:

[0012] Multi-granularity contrastive learning is performed on the first image features and the first health-related features to obtain the contrastive learning loss and cross-modal features, wherein the contrastive learning loss includes intra-modal contrastive learning loss, inter-modal contrastive learning loss and sample-level contrastive learning loss;

[0013] Individual common representation learning is performed on the cross-modal features to obtain the individual common representation learning loss.

[0014] In one possible implementation, the step of learning individual common representations for the cross-modal features to obtain the individual common representation learning loss includes: determining the individual representation learning loss based on the individual representations;

[0015] Based on the common representation, determine the common representation learning loss;

[0016] Based on the individual representation loss and the commonality representation loss, the individual commonality representation learning loss is obtained.

[0017] In one possible implementation, the weighted concatenation of the individual representation and the common representation to obtain the concatenated feature includes:

[0018] Determine the sample quality of the image samples and health-related samples corresponding to the personality representation;

[0019] Based on the sample quality, determine the individual weight and commonality weight of the corresponding individual representation;

[0020] Based on the weights of the individual representations, the common weights corresponding to the common representations are determined;

[0021] The individual weights and common weights are respectively assigned to the individual representation and the common representation;

[0022] The weighted individual representation and the common representation are concatenated to obtain the concatenated feature.

[0023] In one possible implementation, determining the sample quality of the image samples corresponding to the personality representation and the health-related samples includes:

[0024] Based on the mean of the individual features corresponding to all training samples, create positive sample prototypes and negative sample prototypes.

[0025] The similarity scores between the positive sample prototype and the negative sample prototype and the personality representation are determined respectively.

[0026] Based on the similarity score and the real label, the sample quality of the image samples and health-related samples corresponding to the personality representation is determined.

[0027] In one possible implementation, the classification loss includes a prototype update loss and a prediction loss, and the classification processing based on the concatenated features to obtain the classification loss includes:

[0028] The positive and negative sample prototypes are updated based on high-quality samples, and the prototype update loss is determined.

[0029] The splicing features are classified and predicted to obtain the classification prediction results;

[0030] The prediction loss is obtained based on the classification prediction results and the true labels;

[0031] Based on the prototype update loss and the prediction loss, the classification loss is obtained.

[0032] In one possible implementation, the individual representation includes individual image representation and individual health-related representation, and the commonality representation includes common image representation and common health-related representation. The step of weighting and concatenating the individual and commonality representations to obtain concatenated features, and then performing classification processing based on the concatenated features, includes:

[0033] The individual image representation and the common image representation are weighted and then stitched together to obtain image stitching features; the individual health-related representation and the common image representation are weighted and then stitched together to obtain health-related stitching features.

[0034] Image classification results are predicted based on the image stitching features, and health-related classification results are predicted based on the health-related stitching features.

[0035] The image classification result and the health-related classification result are weighted separately, and the classification result is obtained based on the weighted result.

[0036] Secondly, a classification method for a multimodal classification model is provided, applied to a multimodal classification model trained using the aforementioned training method. The method includes:

[0037] Acquire the image data to be detected and the health-related data to be detected, and extract features from the image data to be detected and the health-related data to be detected respectively to obtain the image features to be detected and the health-related features to be detected.

[0038] Individual and common representation learning is performed on the features of the image to be detected and the health-related features to be detected, respectively, to obtain individual image representation, common image representation, individual health-related representation and common health-related representation under different loss constraints;

[0039] The individual image representation and the common image representation are spliced ​​together to obtain image splicing features, and the individual health-related representation and the common image representation are spliced ​​together to obtain health-related splicing features;

[0040] Image classification results are predicted based on the image stitching features, and health-related classification results are predicted based on the health-related stitching features.

[0041] The image classification results and the health-related classification results are weighted separately, and a comprehensive classification result is obtained based on the weighted results.

[0042] Thirdly, a multimodal classification model is provided, which includes a multimodal classification model corresponding to a first training stage and a multimodal classification model corresponding to a second training stage, wherein the multimodal classification model corresponding to the second training stage is trained based on the multimodal classification model corresponding to the first training stage.

[0043] The first-stage multimodal classification model includes an image feature extraction module, a health-related feature extraction module, and a feature learning module. The feature learning module is used to perform multi-granularity comparative learning and individual commonality representation learning on the extracted image features and health-related features.

[0044] The second stage of the multimodal classification model includes an image feature extraction module, a health-related feature extraction module, a feature learning module, and a classification module. The feature learning module is used to learn individual and common representations of the extracted image features and health-related features. The classification module is used to classify and predict the individual and common representations learned by the feature learning module.

[0045] In one possible implementation, the multimodal classification model corresponding to the second stage includes an image modality prediction branch and a health-related modality prediction branch. The image modality prediction branch includes an image feature extraction module, an image feature learning module, and an image classification module. The health-related modality prediction branch includes a health-related feature extraction module, a health-related feature learning module, and a health-related classification module. The image classification module is used to output image prediction results, and the health-related classification module is used to output health-related prediction results. The image prediction results and the health-related prediction results are weighted and fused to obtain the final prediction result.

[0046] The training method, classification method, and multimodal classification model described above include the following steps: feature extraction is performed on a first image sample, a second image sample, a first health-related sample, and a second health-related sample to obtain first image features, second image features, first health-related features, and second health-related features; multi-granularity contrastive learning and individual common representation learning are performed on the first image features and the first health-related features to obtain contrastive learning loss and individual common representation learning loss, and the multimodal classification model is iteratively updated based on the contrastive learning loss and the individual common representation learning loss; individual common representation learning is performed on the second image features and the second health-related features to obtain individual representations and common representations corresponding to different loss constraints; the individual representations and common representations are weighted and concatenated to obtain concatenated features, and classification processing is performed based on the concatenated features to obtain a classification loss; the multimodal classification model is iteratively updated again based on the classification loss. In this embodiment, the model training phase is divided into two stages. In the first stage, multi-granularity contrastive learning and individual common representation learning are performed on the extracted image features and health-related features. The model is then iteratively updated based on the contrastive learning loss and the individual common representation learning loss. In the second stage, based on the model trained in the first stage, individual common representation learning is performed on the extracted image features and health-related features. The learned individual and common representations are then weighted and concatenated, and classification is performed based on the concatenated features. The model is then iteratively updated again based on the classification loss. By adopting the above two-stage training strategy, the first stage, through multi-granularity contrastive learning and individual common representation learning, deeply explores the differences between images and health-related features within modalities and the semantic relationships between modalities, enhancing feature discriminativeness. The second stage, based on the results of the first stage, further learns individual and common representations and weights them for classification. This effectively handles the multi-granularity characteristics and quality differences of features, strengthens modal interaction, and thus improves the model's adaptability, diagnostic accuracy, and generalization ability to complex multimodal data. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of a model architecture for a multimodal classification model in one embodiment of this application;

[0049] Figure 2This is a flowchart illustrating a training method for a multimodal classification model in one embodiment of this application;

[0050] Figure 3 This is a flowchart illustrating a classification method of a multimodal classification model in one embodiment of this application;

[0051] Figure 4 This is a schematic diagram of a training device for a multimodal classification model in one embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the structure of a classification device for a multimodal classification model in one embodiment of this application;

[0053] Figure 6 This is a schematic diagram of a computer device according to one embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] The multimodal classification model training method and classification method provided in this embodiment can be applied to, for example... Figure 1 The multimodal classification model architecture shown includes a multimodal classification model corresponding to a first training phase and a multimodal classification model corresponding to a second training phase. The multimodal classification model corresponding to the second training phase is trained based on the multimodal classification model corresponding to the first training phase. It can be understood that after training in the first training phase, the multimodal classification model obtains updated model parameters, which are then fed into the second training phase for further iteration, resulting in the final trained multimodal classification model.

[0056] The first-stage multimodal classification model includes an image feature extraction module, a health-related feature extraction module, and a feature learning module. First, the image feature extraction module extracts features from the input image samples to obtain image features. Then, the health-related feature extraction module extracts features from the input health-related samples to obtain health-related features. Next, based on the extracted image features and health-related features, the feature learning module performs multi-granularity contrastive learning and individual common representation learning, calculating the contrastive learning loss and individual common representation learning loss obtained during the learning process, which are used to iteratively update the multimodal classification model.

[0057] The second stage of the multimodal classification model includes an image feature extraction module, a health-related feature extraction module, a feature learning module, and a classification module. First, the image feature extraction module extracts features from the input image samples to obtain image features. Then, the health-related feature extraction module extracts features from the input health-related samples to obtain health-related features. Next, the feature learning module performs individual and common representation learning on the extracted image features and health-related features. Finally, the classification module performs classification prediction on the individual and common representations learned by the feature learning module.

[0058] It should be noted that the multimodal classification model in the second stage includes an image modality prediction branch and a health-related modality prediction branch. The image modality prediction branch includes an image feature extraction module, an image feature learning module, and an image classification module. The health-related modality prediction branch includes a health-related feature extraction module, a health-related feature learning module, and a health-related classification module. First, the image feature extraction module extracts features from the input image samples to obtain image features. Then, the feature learning module learns individual and common representations of the image features to obtain individual image representations and common image representations. Finally, in the classification module, the individual and common image representations are weighted and concatenated, and prediction is performed based on the concatenated features to obtain the image prediction result. Similarly, health-related samples also go through the above process to obtain health-related prediction results. A prediction result weight learning method is introduced to weight and fuse the image prediction results and health-related prediction results to obtain the final prediction result.

[0059] It should be noted that the image feature extraction module can adopt a PENet architecture based on 3D convolution, including a 3D convolutional backbone network and a perceptron MLP for global feature aggregation and encoding. The 3D convolutional neural network contains the backbone of PENet, and the multilayer perceptron consists of three hidden layers. The health-related feature extraction module can use a multilayer perceptron for feature encoding, consisting of two hidden layers. The classification module can include a Dynamic Prototype Fusion Module and a classifier. The classification module includes the Dynamic Prototype Fusion Module and a classifier (composed of several fully connected layers or convolutional layers).

[0060] Example 1

[0061] In one embodiment, such as Figure 2 As shown, a training method for a multimodal classification model is provided, which can be applied to applications such as... Figure 1 The multimodal classification model corresponding to the first training phase shown includes the following steps:

[0062] In step S110, feature extraction is performed on the first image sample, the second image sample, the first health-related sample, and the second health-related sample to obtain the first image feature, the second image feature, the first health-related feature, and the second health-related feature.

[0063] Optionally, a training sample dataset can be obtained, which stores a large number of training samples for training the multimodal classification models corresponding to the first and second training phases. The multimodal classification models corresponding to the first and second training phases can be trained using the same training samples.

[0064] It should be noted that the training sample dataset can be an existing publicly available dataset or a dataset generated from training data samples collected through integration with hospital platforms. Existing publicly available datasets may include RadFusion, designed for the diagnosis of pulmonary embolism, and Lung-PET-CT-Dx, used for classifying lung adenocarcinoma and squamous cell carcinoma. The RadFusion dataset contains 1,837 studies from 1,794 patients, including 1,111 negative cases and 726 positive cases. Each study includes complete CTPA and EHR data. The CTPA data consists of multiple single-channel 2D images with a slice thickness of 1.25 mm, and the number of slices per study ranges from 28 to 2,600. The EHR data consists of multiple tables, including demographic data, outpatient and inpatient medications, vital signs, ICD codes, and laboratory test results, totaling 2,912 attributes. The Lung-PET-CT-Dx dataset, used for lung cancer detection, contains 355 samples. Based on histopathological diagnosis, patients were classified into four categories: adenocarcinoma (251 cases), small cell carcinoma (38 cases), large cell carcinoma (5 cases), and squamous cell carcinoma (61 cases). Imaging information for some patients included CT volumetric data at a resolution of 512x512. The patients' EHR data included demographic data, clinical stage, pathological data, and lifestyle records.

[0065] The image samples can be CT (Computed Tomography) images or PET (Positron Emission Tomography) images. The health-related samples can be tabular or textual modal data, specifically including electronic medical record data corresponding to the image samples, which may include health-related data such as identity information, medical records, medical progress records, and physical examination results.

[0066] Image samples are typically stored in DICOM format. DICOM (Digital Imaging and Communications in Medicine) is an international standard for medical images and related information. Therefore, a data structure in the form of a data table can be extracted from the DICOM file corresponding to the medical image to be examined. This data table is then read to obtain the raw image data. The key component of the raw image data is the pixel sequence, which records information such as the grayscale value of each pixel in the image.

[0067] The read pixel sequence is a raw grayscale value matrix stored in a DICOM file, with each pixel corresponding to a specific value. By filtering and mapping these grayscale values, the target window level and width are determined. Window level and window width are crucial concepts in the display of medical images. The window width determines the range of displayed grayscale values, while the window level is the center value of that range. Different window level and width settings can highlight details of different tissues in the image. Therefore, based on the determined target window level and width, the original image data can be processed, mapping the grayscale values ​​within the corresponding range to a new grayscale image, thus obtaining a grayscale image that highlights only the lung region. To increase data diversity, grayscale image enhancement operations can be performed, such as flipping the grayscale image at random angles, including horizontal flipping, vertical flipping, and rotation at arbitrary angles. This can simulate lung images from different shooting angles, allowing the model to learn richer image features and avoid overfitting. Furthermore, image sharpening algorithms can enhance the edge and detail information of the image, making the boundaries of lung tissue clearer and the texture more pronounced. After these data augmentation operations, an enhanced grayscale image is obtained, which contains more diverse image information, helping to improve the effectiveness of subsequent analysis and model training. Then, the image feature extraction module extracts features from the enhanced grayscale image to obtain image features, which may include lesion texture, shape features, etc.

[0068] For health-related samples, redundant attributes lacking information can be removed, and the data can be structured to generate health table data. This health table data is then digitized; for example, categorical variables are converted to numerical values ​​using label encoding, and binary variables such as medication records, disease diagnostic markers, and medical examination indicators are encoded using binary encoding. Finally, all tables are standardized, merged, and attribute filtering is performed using support vector machines to complete preprocessing. Then, health-related features are extracted from the preprocessed table data, which may include symptom keywords, physiological indicator values, etc.

[0069] In step S120, multi-granularity contrastive learning and individual common representation learning are performed on the first image features and the first health-related features to obtain contrastive learning loss and individual common representation learning loss, so as to iteratively update the multimodal classification model based on the contrastive learning loss and individual common representation learning loss;

[0070] Optionally, in the first training phase, a multi-granularity contrastive learning algorithm and a unique-common representation learning algorithm are used simultaneously. The multi-granularity contrastive learning algorithm achieves cross-modal alignment, while the unique-common representation learning algorithm decouples modal unique features. Multi-granularity contrastive learning can shorten the feature distance between similar samples and widen the feature distance between dissimilar samples, thereby improving the class discrimination of features. Unique features within a modality (individuality) and shared features across modalities (commonality) can be separated through unique-common representation learning, enhancing the semantic relevance of features and improving the model's generalization ability. Specifically, a contrastive learning loss function can be used to perform feature alignment at different levels to determine inter-modal loss, intra-modal loss, and sample-level loss. Then, the contrastive learning loss is calculated based on the inter-modal loss, intra-modal loss, and sample-level loss. For the cross-modal features after contrastive learning, individual and common representation learning can be performed again. First, the cross-modal features are decomposed into common features and individual features by the encoder. Then, based on the common loss function and the common features, the common loss is calculated, and based on the individual loss function, the individual loss is calculated. Then, based on the individual loss and the common loss, the individual and common representation learning loss is obtained. Finally, based on the total contrastive learning loss and the individual and common representation learning loss, the total learning loss is obtained. If the total learning loss is greater than a preset loss threshold, or the number of iterations is less than a preset number, the model is iteratively updated; otherwise, the model trained in the first training stage is obtained.

[0071] In step S130, the second image features and the second health-related features are subjected to individual common representation learning to obtain individual representations and common representations under different loss constraints;

[0072] Optionally, in the second training phase, based on the model parameters trained in the first phase, individual common representation learning is performed on the newly input second-phase image features and health-related features (which have no overlap with the first-phase samples). Under the constraints of different loss functions, the second image features and second health-related features are decomposed into common representations with cross-modal shared characteristics and individual representations that retain unique information of a single modality. Specifically, by designing common representation learning loss (such as minimizing the difference in common representations between images and health features) and individual representation retention loss (such as ensuring that single-modal features can be reconstructed through individual representations), the model is forced to separate two types of representations from the mixed features: common representations capture the common patterns of images and health data at the semantic level (such as the common mapping of the pathological mechanisms of diseases in images and indicators), while individual representations retain the detailed information unique to each modality (such as the texture details of images and the temporal records of health data). This loss-constraint-based decoupling process can achieve cross-modal semantic unification through common representations and maintain modality specificity through individual representations, providing a feature foundation that is both relevant and complete for subsequent feature fusion and classification tasks, thereby improving the model's ability to comprehensively analyze multi-source heterogeneous data.

[0073] In step S140, the individual representation and the common representation are weighted and then concatenated to obtain concatenated features, and classification processing is performed based on the concatenated features to obtain classification loss;

[0074] Optionally, since individual representations capture the most discriminative information, while common representations are relatively stable, individual representations are more sensitive to quality changes. First, the dynamic prototype fusion module can integrate common and individual representations based on an adaptive mechanism. It first calculates the importance weights of the two types of representations based on the input individual and common representations using a dynamic weight generation mechanism (such as attention networks or multilayer perceptrons). Then, the module performs linear combination or nonlinear transformation on the individual and common representations according to the weights to complete feature concatenation, forming concatenated features. Finally, a classification head performs predictive processing on the concatenated features to obtain the classification result.

[0075] In step S150, the multimodal classification model is iteratively updated again based on the classification loss.

[0076] It should be noted that the classification loss may include prediction loss and prototype update loss. The prediction loss can be calculated based on the classification result, while the prototype update loss is calculated based on the prototype update performed by the dynamic prototype fusion module. If the classification loss is greater than a preset loss threshold, or the number of iterations is less than a preset number, the model is iteratively updated; otherwise, the model trained in the second training phase is obtained.

[0077] In this embodiment, the model training phase is divided into two stages. In the first stage, multi-granularity contrastive learning and individual common representation learning are performed on the extracted image features and health-related features. The model is then iteratively updated based on the contrastive learning loss and the individual common representation learning loss. In the second stage, based on the model trained in the first stage, individual common representation learning is performed on the extracted image features and health-related features. The learned individual and common representations are then weighted and concatenated, and classification is performed based on the concatenated features. The model is then iteratively updated again based on the classification loss. By adopting the above two-stage training strategy, the first stage, through multi-granularity contrastive learning and individual common representation learning, deeply explores the differences between images and health-related features within modalities and the semantic relationships between modalities, enhancing feature discriminativeness. The second stage, based on the results of the first stage, further learns individual and common representations and weights them for classification. This effectively handles the multi-granularity characteristics and quality differences of features, strengthens modal interaction, and thus improves the model's adaptability, diagnostic accuracy, and generalization ability to complex multimodal data.

[0078] In one embodiment of this application, the step of performing multi-granularity contrastive learning and individual commonality representation learning on the first image features and the first health-related features to obtain contrastive learning loss and individual commonality representation learning loss includes:

[0079] Multi-granularity contrastive learning is performed on the first image features and the first health-related features to obtain the contrastive learning loss and cross-modal features, wherein the contrastive learning loss includes intra-modal contrastive learning loss, inter-modal contrastive learning loss and sample-level contrastive learning loss;

[0080] Individual common representation learning is performed on the cross-modal features to obtain the individual common representation learning loss.

[0081] Optionally, for multi-granularity contrastive learning, a unified contrastive learning loss function can be used for optimization at different levels, including those containing... For batch data of samples, these samples can be represented as ,in Indicates the first One image modality, Represents a table modality. Used for comparative learning of tags. This indicates that feature extraction is performed through an image feature extraction module and a health-related feature extraction module, respectively, to generate corresponding image features. Health-related characteristics .

[0082] It should be noted that, based on the label softening strategy, the label distribution of samples within each batch can be adjusted. For example, the category distribution of batch data can be represented as follows: ,in This represents the category of each sample. For the first... The label vector of each sample , It can be calculated using the following formula:

[0083] ;

[0084] in, Indicates the batch and the first The total number of samples belonging to the same category.

[0085] The label for the entire batch is: .

[0086] Then, feature alignment is performed at the intra-modal, inter-modal, and sample levels, respectively.

[0087] In-modal: Let Representing modal data The characteristics of. The first The self-similarity vector of each sample is represented as: , The following formula can be used to calculate it:

[0088] ;

[0089] in, It is a learnable temperature parameter. Used to calculate similarity scores via dot product. This represents the feature vector of the j-th sample in the modal data S. The feature vector of the k-th sample in the modal data S. The feature vector of the m-th sample in the modal data S, where N represents the number of samples.

[0090] Since both the softened label and the self-similarity score matrix are probability distributions, the Kullback-Leibler (KL) divergence can be used, and the loss is calculated as follows:

[0091] ;

[0092] Where q(s) represents the probability distribution of sample self-similarity scores, and Y represents the probability distribution corresponding to the softened label.

[0093] Therefore, the intra-modal contrast loss can be calculated using the above method for both image features and health-related features, and the intra-modal contrast loss of the image is... The intra-modal contrast loss for health-related features is Then the intramodal global contrastive learning loss can be expressed as: .

[0094] For intermodal relationships, cross-modal relationships can be modeled from a macroscopic perspective by comparing and analyzing the features of different modalities. Specifically, similarity scores are calculated from image features to health-related features and from health-related features to image features. These are represented as follows: and Specifically, it can be calculated using the following formula:

[0095]

[0096] in, This represents the similarity score between image features and health-related features. The similarity score between health-related features and image features is represented by τ, where τ represents the temperature parameter. Used to calculate similarity scores via dot product. This represents the feature vector of the j-th sample in the image modality. This represents the feature vector of the k-th sample in the image modality. This represents the feature vector of the m-th sample in the image modality. This represents the feature vector of the j-th sample in the health-related modality. This represents the feature vector of the k-th sample in the health-related modality. Let represent the feature vector of the m-th sample in the health-related modality.

[0097] Similarly, the inter-modal contrastive learning loss can be obtained based on the calculation method of the intra-modal contrastive learning loss, which can be specifically expressed as:

[0098] .

[0099] in, This represents the intermodal contrastive learning loss from image features to health-related features. This represents the intermodal contrastive learning loss from health-related features to image features.

[0100] At the sample level, all modal information from a single sample can be integrated to obtain global information. Starting from a global granularity, the physiological and pathological characteristics of patients can be captured more comprehensively. By... and Adding them together yields the global representation: Subsequently, Treating it as a modality, we introduce the loss calculation method of intra-modal contrastive learning to obtain the sample-level contrastive learning loss. Finally, the total loss for multi-granularity contrastive learning is defined as follows:

[0101]

[0102] Then, under the constraints of different loss functions, the second image features and the second health-related features are decomposed into a common representation with cross-modal sharing characteristics and a personalized representation that retains unique information of a single modality. The common representation may include a common image representation. and related expressions of common health Personality representation may include personality image representation. Personal health related expressions .

[0103] In one embodiment of this application, the step of performing individual common representation learning on the cross-modal features to obtain the individual common representation learning loss includes: determining the individual representation learning loss based on the individual representation;

[0104] Based on the common representation, determine the common representation learning loss;

[0105] Based on the individual representation loss and the commonality representation loss, the individual commonality representation learning loss is obtained.

[0106] Alternatively, orthogonal projection loss can be used. This method minimizes the similarity between individual and common features within each modality. The loss value is calculated as follows:

[0107]

[0108] in, For common image representation, For common health-related indications, express , This indicates information related to individual health.

[0109] The overall loss of personality expression is: .

[0110] For generating common representations, the Jensen-Shannon divergence (JSD) can be used as the loss function, and the loss value is calculated as follows:

[0111]

[0112] in, For common image representation, This refers to information related to common health conditions.

[0113] The overall loss of the commonality representation is: .

[0114] Minimize the loss value to maximize the commonality among these features. In summary, the overall loss for individual-commonality representation learning is defined as follows:

[0115]

[0116] In one embodiment of this application, the step of weighting and concatenating the individual representation and the common representation to obtain the concatenated feature includes:

[0117] Determine the sample quality of the image samples and health-related samples corresponding to the personality representation;

[0118] Based on the sample quality, determine the individual weight and commonality weight of the corresponding individual representation;

[0119] The individual weights and common weights are respectively assigned to the individual representation and the common representation;

[0120] The weighted individual representation and the common representation are concatenated to obtain the concatenated feature.

[0121] Optionally, since individual representations can capture the most discriminative information, while common features are relatively stable, individual features are more sensitive to quality changes. First, the dynamic prototype fusion module can integrate common and individual representations based on an adaptive mechanism. First, positive and negative sample prototypes are constructed, and the similarity score between these prototypes and individual features is calculated. Finally, based on the score and the true label, the sample quality is determined. Then, based on the input individual and common representations, the importance weights of the two types of representations can be calculated using a dynamic weight generation mechanism (such as attention networks or multilayer perceptrons) according to the classification and inconsistency scores of the sample quality. Subsequently, the module performs linear combination or nonlinear transformation on the individual and common representations according to the weights to complete feature concatenation and form concatenated features.

[0122] In one embodiment of this application, determining the sample quality of the image samples corresponding to the personality representation and the health-related samples includes:

[0123] Based on the mean of the individual features corresponding to all training samples, create positive sample prototypes and negative sample prototypes.

[0124] The similarity scores between the positive sample prototype and the negative sample prototype and the personality representation are determined respectively.

[0125] Based on the similarity score and the real label, the sample quality of the image samples and health-related samples corresponding to the personality representation is determined.

[0126] Optionally, positive and negative sample prototypes can be created by calculating the mean of the individual characteristics of all samples in the training set, which can be expressed by the following formula:

[0127]

[0128] in, This represents the total number of samples, while and These represent the number of positive and negative samples, respectively.

[0129] Then, the similarity scores between the positive and negative prototypes and the personality traits can be calculated using the dot product, and expressed as follows: and Then, the categories are initially determined based on the two obtained similarity scores, and finally compared with the labels. Inconsistent samples are defined as low-quality samples.

[0130] During the fusion phase, the weights of individual features can be dynamically adjusted based on the classification and similarity scores of the samples, assigning lower weights to low-quality samples. The weights are determined by the following piecewise function:

[0131]

[0132] Then, the individual and common features are multiplied by their respective weights, concatenated, and a classifier is used for final classification to obtain the output. This is done to address the class imbalance problem in the data distribution and optimize the classification process.

[0133] In one embodiment of this application, the classification loss includes a prototype update loss and a prediction loss, and the classification processing based on the concatenated features to obtain the classification loss includes:

[0134] The positive and negative sample prototypes are updated based on high-quality samples, and the prototype update loss is determined.

[0135] The splicing features are classified and predicted to obtain the classification prediction results;

[0136] The prediction loss is obtained based on the classification prediction results and the true labels;

[0137] Based on the prototype update loss and the prediction loss, the classification loss is obtained.

[0138] Optionally, high-quality samples can be used to update the prototype, and the loss is calculated as follows:

[0139]

[0140] Where ⊙ represents the XOR logical operation. This indicates sample quality, with a value of 1 representing high quality. (Loss value) and This is used to optimize the prototypes for both positive and negative samples separately. The overall prototype update loss is calculated as follows:

[0141]

[0142] For the classification process, the binary focusing loss function can be used to calculate the prediction loss, which can be obtained through the following formula:

[0143]

[0144] in, It is a focusing parameter used to adjust the weights of difficult and easy samples. It is the first The predicted probability value output for each sample.

[0145] For the combination of loss functions of the overall model, the total loss in the first training phase is defined as:

[0146]

[0147] in, and These are learnable weight parameters. These weights will be automatically adjusted during training to optimize the contribution of each item. The total loss for the second stage is defined as:

[0148]

[0149] in, and These are learnable weight parameters. These weights will also be automatically and dynamically adjusted during training.

[0150] In one embodiment of this application, the individual representation includes individual image representation and individual health-related representation, and the commonality representation includes common image representation and common health-related representation. The step of weighting and concatenating the individual representation and commonality representation to obtain concatenated features, and then performing classification processing based on the concatenated features, includes:

[0151] The individual image representation and the common image representation are weighted and then stitched together to obtain image stitching features; the individual health-related representation and the common image representation are weighted and then stitched together to obtain health-related stitching features.

[0152] Image classification results are predicted based on the image stitching features, and health-related classification results are predicted based on the health-related stitching features.

[0153] The image classification result and the health-related classification result are weighted separately, and the classification result is obtained based on the weighted result.

[0154] Optionally, firstly, a feature learning module learns individual and common representations of image features to obtain individual and common image representations. Finally, in the classification module, the individual and common image representations are weighted and concatenated, and prediction is performed based on the concatenated features to obtain the image prediction result. Similarly, health-related samples also go through the above process to obtain health-related prediction results. A prediction result weight learning method is introduced to weight and fuse the image prediction results and health-related prediction results to obtain the final prediction result.

[0155] It should be noted that the introduction of learnable parameters The weights representing the image prediction results, and corresponding The weights are assigned to the table prediction results. The comprehensive diagnostic results are used as the prediction results for classification algorithm optimization. After optimization, these weights can be used to achieve the later fusion of the prediction results for the image and table modalities to obtain the final classification result. In both the first and second stages of model training, SGD (Stochastic Gradient Descent) is used as the optimizer. The first stage simultaneously performs multi-granularity contrastive learning and individual-common representation learning, while the second stage calculates the prototype update loss and classification loss.

[0156] In this embodiment, the model training phase is divided into two stages. In the first stage, multi-granularity contrastive learning and individual common representation learning are performed on the extracted image features and health-related features. The model is then iteratively updated based on the contrastive learning loss and the individual common representation learning loss. In the second stage, based on the model trained in the first stage, individual common representation learning is performed on the extracted image features and health-related features. The learned individual and common representations are then weighted and concatenated, and classification is performed based on the concatenated features. The model is then iteratively updated again based on the classification loss. By adopting the above two-stage training strategy, the first stage, through multi-granularity contrastive learning and individual common representation learning, deeply explores the differences between images and health-related features within modalities and the semantic relationships between modalities, enhancing feature discriminativeness. The second stage, based on the results of the first stage, further learns individual and common representations and weights them for classification. This effectively handles the multi-granularity characteristics and quality differences of features, strengthens modal interaction, and thus improves the model's adaptability, diagnostic accuracy, and generalization ability to complex multimodal data.

[0157] Example 2

[0158] In one embodiment, such as Figure 3 As shown, a method for a multimodal classification model is provided, which is applied to a multimodal classification model trained in the second training phase, including the following steps:

[0159] In step S210, the image data to be detected and the health-related data to be detected are acquired, and feature extraction is performed on the image data to be detected and the health-related data to be detected respectively to obtain the image features to be detected and the health-related features to be detected.

[0160] In step S220, individual and common representation learning is performed on the features of the image to be detected and the health-related features to be detected, respectively, to obtain individual image representation, common image representation, individual health-related representation and common health-related representation under different loss constraints;

[0161] In step S230, the individual image representation and the common image representation are spliced ​​together to obtain image splicing features, and the individual health-related representation and the common image representation are spliced ​​together to obtain health-related splicing features;

[0162] In step S240, an image classification result is predicted based on the image stitching features, and a health-related classification result is predicted based on the health-related stitching features.

[0163] In step S250, the image classification result and the health-related classification result are weighted respectively, and a comprehensive classification result is obtained based on the weighted result.

[0164] Optionally, image data to be detected, such as lung CT images, and health-related data to be detected, such as electronic medical record data, can be acquired for the user to be detected. These data are then preprocessed (e.g., image denoising, medical record text cleaning and standardization). The image feature extraction module and the health-related feature extraction module then extract the image features and health-related features respectively. Next, individual and common representation learning is performed on the image features and health-related features to be detected, respectively, to obtain individual image representations, common image representations, individual health-related representations, and common health-related representations under different loss constraints. These are then input into the classification module. First, the individual and common image representations are weighted and concatenated in the classification module, and prediction is performed based on the concatenated features to obtain the image prediction result. Similarly, the above process is performed on the health-related data to be detected to obtain the health-related prediction result. Finally, prediction result weight learning is introduced to weight and fuse the image prediction result and the health-related prediction result to obtain the final prediction result.

[0165] In this embodiment, the model training phase is divided into two stages. In the first stage, multi-granularity contrastive learning and individual common representation learning are performed on the extracted image features and health-related features. The model is then iteratively updated based on the contrastive learning loss and the individual common representation learning loss. In the second stage, based on the model trained in the first stage, individual common representation learning is performed on the extracted image features and health-related features. The learned individual and common representations are then weighted and concatenated, and classification is performed based on the concatenated features. The model is then iteratively updated again based on the classification loss. By adopting the above two-stage training strategy, the first stage, through multi-granularity contrastive learning and individual common representation learning, deeply explores the differences between images and health-related features within modalities and the semantic relationships between modalities, enhancing feature discriminativeness. The second stage, based on the results of the first stage, further learns individual and common representations and weights them for classification. This effectively handles the multi-granularity characteristics and quality differences of features, strengthens modal interaction, and thus improves the model's adaptability, diagnostic accuracy, and generalization ability to complex multimodal data.

[0166] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0167] To further verify the performance and effectiveness of the multimodal classification model provided in this application, experiments were conducted on two multimodal disease diagnosis datasets. The first dataset, RadFusion, was designed for the diagnosis of pulmonary embolism. The second dataset, Lung-PET-CT-Dx, was used to classify lung adenocarcinoma and squamous cell carcinoma.

[0168] The RadFusion dataset contains 1,837 studies from 1,794 patients, including 1,111 negative cases and 726 positive cases. Each study includes complete CTPA and EHR data. CTPA data consists of multiple single-channel 2D images with a slice thickness of 1.25 mm, and the number of slices per study ranges from 28 to 2,600. EHR data consists of multiple tables including demographics, vital signs, outpatient and inpatient medications, ICD codes, and laboratory test results, totaling 2,912 attributes. The Lung-PET-CT-Dx dataset, used for lung cancer detection, contains 355 samples. Based on histopathological diagnosis, patients were categorized into four types: adenocarcinoma (251 cases), small cell carcinoma (38 cases), large cell carcinoma (5 cases), and squamous cell carcinoma (61 cases). Image information for some patients includes CT volumetric data at a resolution of 512x512. The EHR data for this patient includes clinical stage, demographics, pathological data, and lifestyle records.

[0169] Several baseline models were compared. Based on their modality fusion methods, they were categorized into early fusion methods and late fusion methods. Early fusion methods included: PE-MVCNet, DAFT, MMTM, MedFuse, DrFuse, TabMixer, and MMAD. Late fusion methods included: PEFusion and PECon.

[0170] The evaluation metrics include: Area under the receiver operating characteristic curve (AUROC), accuracy (ACC), balanced F score (F1Score), sensitivity (SEN), specificity (SPC), negative predictive value (NPV), and positive predictive value (PPV).

[0171] To adapt the datasets for experimental research, specific processing was performed as follows: For the RadFusion dataset, following previous research, positive samples associated with subsegmental types were excluded due to questionable clinical significance. To improve computational efficiency, the image data was scaled from 512x512 to 224x224. These images were then randomly cropped to 192x192 along either the height or width axis and randomly flipped and rotated by up to 15 degrees. For each EHR table, attributes with zero variance were removed, and all tables were standardized. Subsequently, multiple tables were merged, and a support vector machine algorithm was used to filter attributes for final processing. For the Lung-PET-CT-Dx dataset, to address the imbalance caused by significant differences in sample size between different categories, only the two categories with larger sample sizes—adenocarcinoma and squamous cell carcinoma—were retained. Samples with missing EHR or CT data were excluded. The same processing procedure as for the RadFusion dataset was used for the image data. For the EHR data, attributes with excessive missing values ​​were removed, and non-numerical information was digitized, with a small number of missing data points imputed using the mean of each category. To alleviate the class imbalance problem, a small number of samples in the training set were randomly oversampled by 3 times.

[0172] The experiments were conducted on an NVIDIA HGX A100 Tensor Core GPU. For optimization, a stochastic gradient descent optimizer with a momentum of 0.9 was used. In the first stage, training was performed for 100 epochs on both datasets. The learning rate was set to 0.01, and the batch size was 4. A linear warm-up strategy was used for learning rate scheduling, gradually increasing the learning rate over the first 10,000 iterations, followed by 600,000 cosine annealing iterations. For the RadFusion dataset, the feature projection dimension representing the learned part was set to 128, while for the Lung-PET-CT-Dx dataset, the dimension was set to 32. In the second stage, the image and table branches were trained separately, and training continued for an additional 50 epochs on both datasets, maintaining the same learning rate and batch size settings. The StepLR learning rate scheduler was used in this stage with a step size of 10 epochs. The final experimental results are shown in Tables 1 and 2 below.

[0173] Table 1: Experimental results on the RadFusion dataset

[0174]

[0175] Table 2: Experimental results on the Lung-PET-CT-Dx dataset

[0176]

[0177] As shown in Tables 1 and 2 above, the model provided in this application achieves superior performance on key metrics across the two datasets. On the RadFusion dataset, compared to the best late-stage fusion method PECon, the model provided in this application achieves a relative improvement of 1.04% in AUROC, 1.97% in ACC, and 1.85% in F1 Score. On the other four metrics, SEN reaches 98.8%, SPE reaches 87.5%, NPV reaches 98.6%, and PPV reaches 89.0%. On the Lung-PET-CT-Dx dataset, the advantage in key metrics is further amplified. Compared to PECon, AUROC is improved by 1.80%, ACC by 9.71%, and F1 Score by 18.33%. Furthermore, it also achieves best performance on the remaining four metrics.

[0178] Furthermore, the model presented in this application demonstrates superiority compared to other earlier fusion methods. On the RadFusion dataset, in the AUROC comparison, the model presented in this application improves performance by 1.78% compared to the closest model (95.3%), by 1.97% on ACC compared to the closest result (91.4%), and by 2.07% on F1 Score compared to the closest result (91.7%). On the Lung-PET-CT-Dx dataset, these advantages are 0.38%, 3.13%, and 3.20%, respectively. It is evident that the model presented in this application achieves best results on all key metrics across both datasets. However, due to optimization conflicts on certain metrics, such as increasing SEN leading to a decrease in SPE, the model presented in this application did not outperform all metrics on the RadFusion dataset, but its overall performance still shows a significant advantage.

[0179] In summary, compared with all other methods, the model presented in this application achieves the best results on the key metrics of both datasets. Furthermore, this application achieves excellent results of 93.7% AUROC, 90.7% ACC, and 91.1% F1 Score using only the tabular modal branch of the tabular modal data, surpassing the performance of most other multimodal models.

[0180] Example 3

[0181] In one embodiment, a training apparatus for a multimodal classification model is provided, which corresponds one-to-one with the training method for the multimodal classification model in the above embodiments. For example... Figure 4 As shown, the training device for this multimodal classification model includes a training feature extraction unit 101, a first iteration unit 102, a commonality learning training unit 103, a classification training unit 104, and a second iteration unit 105. Detailed descriptions of each functional module are as follows:

[0182] The training feature extraction unit 101 is used to extract features from the first image sample, the second image sample, the first health-related sample, and the second health-related sample respectively, so as to obtain the first image feature, the second image feature, the first health-related feature, and the second health-related feature.

[0183] The first iteration unit 102 is used to perform multi-granularity contrastive learning and individual common representation learning on the first image features and the first health-related features to obtain contrastive learning loss and individual common representation learning loss, so as to iteratively update the multimodal classification model based on the contrastive learning loss and individual common representation learning loss;

[0184] The common learning training unit 103 is used to learn individual common representations of the second image features and the second health-related features to obtain individual representations and common representations under different loss constraints;

[0185] The classification training unit 104 is used to weight and concatenate the individual representation and the common representation to obtain the concatenated features, and to perform classification processing based on the concatenated features to obtain the classification loss;

[0186] The second iteration unit 105 is used to iteratively update the multimodal classification model again based on the classification loss.

[0187] In one embodiment of this application, the first iteration unit 102 is used for:

[0188] Multi-granularity contrastive learning is performed on the first image features and the first health-related features to obtain the contrastive learning loss and cross-modal features, wherein the contrastive learning loss includes intra-modal contrastive learning loss, inter-modal contrastive learning loss and sample-level contrastive learning loss;

[0189] Individual common representation learning is performed on the cross-modal features to obtain the individual common representation learning loss.

[0190] In one embodiment of this application, the cross-modal features include individual features and common features, and the first iteration unit 102 is further configured to:

[0191] Based on the aforementioned personality representation, determine the personality representation learning loss;

[0192] Based on the common representation, determine the common representation learning loss;

[0193] Based on the individual representation loss and the commonality representation loss, the individual commonality representation learning loss is obtained.

[0194] In one embodiment of this application, the classification training unit 104 is further configured to:

[0195] Determine the sample quality of the image samples and health-related samples corresponding to the personality representation;

[0196] Based on the sample quality, determine the individual weight and commonality weight of the corresponding individual representation;

[0197] The individual weights and common weights are respectively assigned to the individual representation and the common representation;

[0198] The weighted individual representation and the common representation are concatenated to obtain the concatenated feature.

[0199] In one embodiment of this application, the classification training unit 104 is further configured to:

[0200] Based on the mean of the individual features corresponding to all training samples, create positive sample prototypes and negative sample prototypes.

[0201] The similarity scores between the positive sample prototype and the negative sample prototype and the personality representation are determined respectively.

[0202] Based on the similarity score and the real label, the sample quality of the image samples and health-related samples corresponding to the personality representation is determined.

[0203] In one embodiment of this application, the classification loss includes prototype update loss and prediction loss, and the classification training unit 104 is further configured to:

[0204] The positive and negative sample prototypes are updated based on high-quality samples, and the prototype update loss is determined.

[0205] The splicing features are classified and predicted to obtain the classification prediction results;

[0206] The prediction loss is obtained based on the classification prediction results and the true labels;

[0207] Based on the prototype update loss and the prediction loss, the classification loss is obtained.

[0208] In one embodiment of this application, the individual representation includes individual image representation and individual health-related representation, and the commonality representation includes common image representation and common health-related representation. The classification training unit 104 is further configured to:

[0209] The individual image representation and the common image representation are weighted and then stitched together to obtain image stitching features; the individual health-related representation and the common image representation are weighted and then stitched together to obtain health-related stitching features.

[0210] Image classification results are predicted based on the image stitching features, and health-related classification results are predicted based on the health-related stitching features.

[0211] The image classification result and the health-related classification result are weighted separately, and the classification result is obtained based on the weighted result.

[0212] In this embodiment, the model training phase is divided into two stages. In the first stage, multi-granularity contrastive learning and individual common representation learning are performed on the extracted image features and health-related features. The model is then iteratively updated based on the contrastive learning loss and the individual common representation learning loss. In the second stage, based on the model trained in the first stage, individual common representation learning is performed on the extracted image features and health-related features. The learned individual and common representations are then weighted and concatenated, and classification is performed based on the concatenated features. The model is then iteratively updated again based on the classification loss. By adopting the above two-stage training strategy, the first stage, through multi-granularity contrastive learning and individual common representation learning, deeply explores the differences between images and health-related features within modalities and the semantic relationships between modalities, enhancing feature discriminativeness. The second stage, based on the results of the first stage, further learns individual and common representations and weights them for classification. This effectively handles the multi-granularity characteristics and quality differences of features, strengthens modal interaction, and thus improves the model's adaptability, diagnostic accuracy, and generalization ability to complex multimodal data.

[0213] Specific limitations regarding the training device for the multimodal classification model can be found in the limitations on the training method for the multimodal classification model described above, and will not be repeated here. Each module in the training device for the aforementioned multimodal classification model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0214] Example 4

[0215] In one embodiment, a classification device for a multimodal classification model is provided. The training device for this multimodal classification model corresponds one-to-one with the classification method of the multimodal classification model in the above embodiment. As shown in Figure X, the classification device for the multimodal classification model includes a feature extraction unit 201, a commonality learning unit 202, a feature concatenation unit 203, a classification unit 204, and a comprehensive classification result acquisition unit 205. Detailed descriptions of each functional module are as follows:

[0216] The detection feature extraction unit 201 is used to acquire the image data to be detected and the health-related data to be detected, and to extract features from the image data to be detected and the health-related data to be detected respectively, so as to obtain the image features to be detected and the health-related features to be detected.

[0217] The commonality learning unit 202 is used to learn individual commonality representations for the features of the image to be detected and the health-related features to be detected, respectively, to obtain individual image representations, common image representations, individual health-related representations and common health-related representations under different loss constraints;

[0218] The feature splicing unit 203 is used to splice the individual image representation and the common image representation to obtain image splicing features, and to splice the individual health-related representation and the common image representation to obtain health-related splicing features;

[0219] The classification unit 204 is used to predict image classification results based on the image stitching features and to predict health-related classification results based on the health-related stitching features;

[0220] The comprehensive classification result acquisition unit 205 is used to weight the image classification result and the health-related classification result respectively, and obtain the comprehensive classification result based on the weighted result.

[0221] In this embodiment, the model training phase is divided into two stages. In the first stage, multi-granularity contrastive learning and individual common representation learning are performed on the extracted image features and health-related features. The model is then iteratively updated based on the contrastive learning loss and the individual common representation learning loss. In the second stage, based on the model trained in the first stage, individual common representation learning is performed on the extracted image features and health-related features. The learned individual and common representations are then weighted and concatenated, and classification is performed based on the concatenated features. The model is then iteratively updated again based on the classification loss. By adopting the above two-stage training strategy, the first stage, through multi-granularity contrastive learning and individual common representation learning, deeply explores the differences between images and health-related features within modalities and the semantic relationships between modalities, enhancing feature discriminativeness. The second stage, based on the results of the first stage, further learns individual and common representations and weights them for classification. This effectively handles the multi-granularity characteristics and quality differences of features, strengthens modal interaction, and thus improves the model's adaptability, diagnostic accuracy, and generalization ability to complex multimodal data.

[0222] Specific limitations regarding the classification device of the multimodal classification model can be found in the limitations on the classification method of the multimodal classification model above, and will not be repeated here. Each module in the classification device of the aforementioned multimodal classification model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0223] In one embodiment, a computer device is provided, which may be a terminal device, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium storing computer-readable instructions. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer-readable instructions implement a training method and a classification method for a multimodal classification model. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0224] In this application embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it implements the steps of the training method and classification method of the multimodal classification model as described above.

[0225] In one embodiment of the application, a readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, they implement the steps of the training method and classification method of the multimodal classification model described above.

[0226] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0227] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0228] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A training method for a multimodal classification model, characterized in that, The method, applied to a multimodal classification model, includes: Feature extraction is performed on the first image sample, the second image sample, the first health-related sample, and the second health-related sample respectively to obtain the first image feature, the second image feature, the first health-related feature, and the second health-related feature; Multi-granularity contrastive learning and individual common representation learning are performed on the first image features and the first health-related features, including: aligning features at different levels using a contrastive learning loss function to determine inter-modal loss, intra-modal loss, and sample-level loss; calculating the contrastive learning loss based on the inter-modal loss, intra-modal loss, and sample-level loss; decomposing the cross-modal features after contrastive learning into common features and individual features using an encoder; calculating the common loss based on the common loss function and the common features; calculating the individual loss based on the individual loss function; obtaining the individual common representation learning loss based on the individual loss and the common loss; obtaining the total learning loss based on the total contrastive learning loss and the individual common representation learning loss; if the total learning loss is greater than a preset loss threshold, or the number of iterations is less than a preset number, then the model is iteratively updated. Individual and common representation learning is performed on the second image features and the second health-related features to obtain individual and common representations under different loss constraints; The individual and common representations are weighted and concatenated to obtain concatenated features, and classification processing is performed based on the concatenated features to obtain classification loss; Based on the classification loss, the multimodal classification model is iteratively updated again.

2. The training method for the multimodal classification model as described in claim 1, characterized in that, The step of performing multi-granularity contrastive learning and individual commonality representation learning on the first image features and the first health-related features to obtain contrastive learning loss and individual commonality representation learning loss includes: Multi-granularity contrastive learning is performed on the first image features and the first health-related features to obtain the contrastive learning loss and cross-modal features, wherein the contrastive learning loss includes intra-modal contrastive learning loss, inter-modal contrastive learning loss and sample-level contrastive learning loss; Individual common representation learning is performed on the cross-modal features to obtain the individual common representation learning loss.

3. The training method for the multimodal classification model as described in claim 2, characterized in that, The step of learning individual common representations for the cross-modal features to obtain the individual common representation learning loss includes: Based on the aforementioned personality representation, determine the personality representation learning loss; Based on the common representation, determine the common representation learning loss; Based on the individual representation loss and the commonality representation loss, the individual commonality representation learning loss is obtained.

4. The training method for the multimodal classification model as described in claim 1, characterized in that, The weighted concatenation of the individual and common representations yields the concatenated features, including: Determine the sample quality of the image samples and health-related samples corresponding to the personality representation; Based on the sample quality, determine the individual weight and commonality weight of the corresponding individual representation; The individual weights and common weights are respectively assigned to the individual representation and the common representation; The weighted individual representation and the common representation are concatenated to obtain the concatenated feature.

5. The training method for the multimodal classification model as described in claim 4, characterized in that, Determining the sample quality of the image samples corresponding to the personality representation and the health-related samples includes: Based on the mean of the individual features corresponding to all training samples, create positive sample prototypes and negative sample prototypes. The similarity scores between the positive sample prototype and the negative sample prototype and the personality representation are determined respectively. Based on the similarity score and the real label, the sample quality of the image samples and health-related samples corresponding to the personality representation is determined.

6. The training method for the multimodal classification model as described in claim 5, characterized in that, The classification loss includes prototype update loss and prediction loss. The classification processing based on the concatenated features to obtain the classification loss includes: The positive and negative sample prototypes are updated based on high-quality samples, and the prototype update loss is determined. The splicing features are classified and predicted to obtain the classification prediction results; The prediction loss is obtained based on the classification prediction results and the true labels; Based on the prototype update loss and the prediction loss, the classification loss is obtained.

7. The training method for the multimodal classification model as described in any one of claims 1-6, characterized in that, The individual representation includes individual image representation and individual health-related representation, and the common representation includes common image representation and common health-related representation. The individual and common representations are weighted and concatenated to obtain concatenated features, and classification processing is performed based on these concatenated features, including: The individual image representation and the common image representation are weighted and then stitched together to obtain image stitching features; the individual health-related representation and the common image representation are weighted and then stitched together to obtain health-related stitching features. Image classification results are predicted based on the image stitching features, and health-related classification results are predicted based on the health-related stitching features. The image classification result and the health-related classification result are weighted separately, and the classification result is obtained based on the weighted result.

8. A classification method for a multimodal classification model, characterized in that, A multimodal classification model trained using the training method of any one of claims 1-7, wherein the method comprises: Acquire the image data to be detected and the health-related data to be detected, and extract features from the image data to be detected and the health-related data to be detected respectively to obtain the image features to be detected and the health-related features to be detected. Individual and common representation learning is performed on the features of the image to be detected and the health-related features to be detected, respectively, to obtain individual image representation, common image representation, individual health-related representation and common health-related representation under different loss constraints; The individual image representation and the common image representation are spliced ​​together to obtain image splicing features, and the individual health-related representation and the common image representation are spliced ​​together to obtain health-related splicing features; Image classification results are predicted based on the image stitching features, and health-related classification results are predicted based on the health-related stitching features. The image classification results and the health-related classification results are weighted separately, and a comprehensive classification result is obtained based on the weighted results.

9. A multimodal classification model, characterized in that, The multimodal classification model is used to implement the training method of the multimodal classification model as described in any one of claims 1-7. The multimodal classification model specifically includes a multimodal classification model corresponding to a first training stage and a multimodal classification model corresponding to a second training stage. The multimodal classification model corresponding to the second training stage is trained based on the multimodal classification model corresponding to the first training stage. The multimodal classification model corresponding to the first training phase includes an image feature extraction module, a health-related feature extraction module, and a feature learning module. The feature learning module is used to perform multi-granularity comparison learning and individual commonality representation learning on the extracted image features and health-related features. The multimodal classification model corresponding to the second training phase includes an image feature extraction module, a health-related feature extraction module, a feature learning module, and a classification module. The feature learning module is used to perform individual and common representation learning on the extracted image features and health-related features. The classification module is used to classify and predict the individual and common representations learned by the feature learning module.

10. The multimodal classification model as described in claim 9, characterized in that, The multimodal classification model corresponding to the second training phase includes an image modality prediction branch and a health-related modality prediction branch. The image modality prediction branch includes an image feature extraction module, an image feature learning module, and an image classification module. The health-related modality prediction branch includes a health-related feature extraction module, a health-related feature learning module, and a health-related classification module. The image classification module is used to output image prediction results, and the health-related classification module is used to output health-related prediction results. The image prediction results and the health-related prediction results are weighted and fused to obtain the final prediction result.

Images