A multimodal classification method and system based on maximizing coding rate reduction

Abstract

The present invention relates to the technical field of multimodal data classification of deep learning, and in particular to a multimodal classification method and system based on maximizing coding rate reduction. The method comprises constructing a specific feature representation based on maximizing coding rate reduction loss by maximizing the overall coding rate and compressing the coding rate of different subspaces, optimizing the specific feature representation of different perspectives; constructing the maximum mutual information loss with the goal of maximizing the mutual information between the specific feature representation and the global feature representation; constructing the cross entropy loss by mapping the specific feature representation to a specified number of category dimensions to obtain the score of each category; and training a multimodal classification network model based on the cross entropy loss, the maximizing coding rate reduction loss, and the maximum mutual information loss. These view-specific features are aligned and fused through spatial transformation and cross-sample fusion to achieve consistent features across multiple views.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data classification, and in particular to a multimodal classification method and system based on maximizing coding rate reduction. Background Art

[0002] In today's digital age, the sources of data are increasingly rich and diverse. With the continuous development of various sensor technologies, we can obtain data about the same object or phenomenon from multiple angles and multiple channels. Traditional classification and clustering methods usually only process data from a single view. However, data from a single view often cannot fully reflect all the characteristics of the object, which may lead to inaccurate classification results. For example, in image recognition, images taken from only one angle may affect the recognition effect due to occlusion, uneven lighting and other problems. In financial risk assessment, relying on only one source of data for analysis may ignore some important risk factors. In addition, there may be complementarity and correlation between data from different views. If these multimodal data cannot be effectively integrated, a lot of valuable information will be wasted.

[0003] Although some multimodal classification algorithms have been proposed, they still have many problems. On the one hand, the computational complexity of existing algorithms is usually high. When processing large-scale multimodal data, a lot of computing resources and time are required, which makes it difficult to meet the real-time requirements in practical applications. For example, some multimodal classification algorithms based on deep learning are very time-consuming in training and reasoning due to the complex model structure.

[0004] On the other hand, existing multimodal classification algorithms are sensitive to noise. In actual data, there are often various noises and outliers, which will have an adverse effect on the classification results. When processing noisy data, existing multimodal classification algorithms lack effective anti-interference mechanisms, resulting in a decrease in classification accuracy. In addition, the adaptability of existing algorithms needs to be improved. Multimodal data in different fields have different characteristics and distributions. Existing multimodal classification algorithms are often difficult to flexibly adapt to various types of multimodal data, limiting their application in a wider range of fields. Summary of the invention

[0005] In order to solve the above-mentioned problems, the present invention provides a multimodal classification method and system based on maximizing coding rate reduction.

[0006] In a first aspect, the present invention provides a multimodal classification method based on maximizing coding rate reduction, which adopts the following technical solution:

[0007] A multimodal classification method based on maximizing coding rate reduction, comprising:

[0008] Acquire multimodal image data;

[0009] Preprocess the acquired multimodal image data to obtain a training set and a test set;

[0010] Perform feature extraction on multimodal image data to extract feature representations from different perspectives;

[0011] By maximizing the overall coding rate and compressing the coding rates of different subspaces, the feature representation of different perspectives is optimized based on maximizing the coding rate and reducing the loss;

[0012] The maximum mutual information loss is constructed with the goal of maximizing the mutual information between feature representation and global feature representation;

[0013] The cross entropy loss is constructed by mapping the feature representation to a specified number of category dimensions to obtain a score for each category.

[0014] The multimodal classification network model is trained based on cross entropy loss, maximum coding rate reduction loss and maximum mutual information loss respectively;

[0015] The multimodal classification network model is used to perform predictive classification and obtain the predicted classification probability.

[0016] Furthermore, the preprocessing of the acquired multimodal image data to obtain a training set and a test set includes preprocessing the image data to obtain a processed multimodal image data set, including a training set and a test set, wherein the training set ,Include samples and views, the dataset is defined as ,

[0017] in Indicates View primitive features.

[0018] Further, the coding rate construction based on maximizing the overall coding rate and compressing different subspaces is based on maximizing the coding rate to reduce the loss, optimizing the specific feature representation of different perspectives, including optimizing the specific feature representation of each perspective in the training set by maximizing the coding rate to reduce the loss, and the loss is expressed as:

[0019] ,

[0020] in , , , and obtain the optimized feature representation after the coding rate is reduced ,in represents the optimized feature representation after the coding rate is reduced. With parameters Therefore, by optimizing To obtain .

[0021] Furthermore, the cross entropy loss is constructed by mapping the specific feature representation to a specified number of category dimensions to obtain the score of each category, including cascading the feature representation space transformation and alignment of the specific features of all views to obtain the global feature representation, which is expressed as:

[0022] ,

[0023] in, Indicates The view feature transformation matrix, represents the cascade feature, , through the block diagonal matrix , Mapping , ; ,in .

[0024] Furthermore, the cross entropy loss is constructed by mapping a specific feature representation to a specified number of category dimensions to obtain a score for each category, and further includes a fully connected layer Will Mapping to the dimension of the specified number of categories to obtain a classification matrix ,in are the parameters of the classification network, and the classification matrix is ​​trained through the cross-label loss function , where the cross entropy loss is expressed as:

[0025]

[0026]

[0027] in is the loss of a single sample, is a single code encoding of the true label, indicating whether the sample belongs to For all samples.

[0028] Furthermore, the constructing of the maximum mutual information loss with the goal of maximizing the mutual information between the specific feature representation and the global feature representation includes optimizing the consistency of the common features by maximizing the mutual information between the common features and each view feature, wherein the mutual information is expressed as:

[0029] ,

[0030] in is the similarity score between a pair of samples, is a common feature, using cosine distance to evaluate the similarity between these representations.

[0031] Furthermore, the multimodal classification network model is used to perform prediction classification to obtain the prediction classification probability, including Input into the training model to get the classification matrix , by indexing the largest classification matrix To generate the predicted class of each sample for evaluation.

[0032] In a second aspect, a multimodal classification system based on maximizing coding rate reduction comprises:

[0033] A data acquisition module is configured to acquire multimodal image data;

[0034] A preprocessing module is configured to preprocess the acquired multimodal image data to obtain a training set and a test set;

[0035] The feature extraction module is configured to perform feature extraction on the multimodal image data to extract specific feature representations of different viewing angles;

[0036] The maximum coding rate reduction loss module is configured to maximize the overall coding rate and compress the coding rates of different subspaces to construct a maximum coding rate reduction loss based on optimizing specific feature representations of different perspectives;

[0037] The maximum mutual information loss module is configured to construct the maximum mutual information loss with the goal of maximizing the mutual information between the specific feature representation and the global feature representation;

[0038] A cross entropy loss module is configured to construct a cross entropy loss by mapping a specific feature representation to a specified number of category dimensions to obtain a score for each category;

[0039] The training module is configured to train a multimodal classification network model based on a cross entropy loss, a maximum coding rate reduction loss, and a maximum mutual information loss, respectively;

[0040] The prediction module is configured to use the multimodal classification network model to perform prediction classification and obtain the prediction classification probability.

[0041] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the multimodal classification method based on maximizing coding rate reduction.

[0042] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, the processor being used to implement various instructions; the computer-readable storage medium being used to store multiple instructions, the instructions being suitable for being loaded by the processor and executing the multimodal classification method based on maximizing coding rate reduction.

[0043] In summary, the present invention has the following beneficial technical effects:

[0044] 1) Our method proposes a new multimodal classification framework from the perspective of information coding, aiming to explore compact feature representation within views and consistent representation between views.

[0045] 2) In the intra-view feature learning of the proposed method, the goal is to optimize the feature representation by maximizing the difference in encoding rate between the entire dataset and a single class. This process expands the feature representation space while compressing the representation in each class, resulting in a more compact feature representation in each viewpoint. In addition, these view-specific features are aligned and fused through spatial transformation and cross-sample fusion to achieve consistent features across multiple views.

[0046] 3) The proposed method maximizes information transfer and maintains the consistency and relevance of data representation between views. By maximizing the mutual information between the consensus representation and the specific view representation, our method ensures that the learned representation captures more concise intrinsic features and the relevance between different views. Experimental results on multimodal classification tasks have achieved good results. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a schematic diagram of a multimodal classification method based on maximizing coding rate reduction according to Embodiment 1 of the present invention;

[0048] Figure 2 Schematic diagram of the architecture of the multimodal classification network model of Embodiment 2 of the present invention;

[0049] Figure 3 Schematic diagram of the effect of evaluation parameters on the model in Example 1 of the present invention;

[0050] Figure 4 Schematic diagram of subjective confusion matrix of different data sets in Example 1 of the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0052] Example 1

[0053] Reference Figure 1 , a multimodal classification method based on information theory optimization in this embodiment, its overall implementation block diagram is as follows Figure 1As shown, it includes two processes: training phase and testing phase;

[0054] The specific steps of the training phase process are:

[0055] (1) Obtain multimodal image data and preprocess the image data set to obtain a processed multimodal image data set, including a training set and a test set. ,Include samples and views, the dataset is defined as ,in Indicates The training set obtains the potential representation through the initial layer , this latent representation best promotes subsequent layers in their classification task .in is specific to A representation of a view.

[0056] (2) In order to make the learned features discriminative, features of different categories should be as uncorrelated as possible, thereby maximizing the overall space occupied by the features. This is reflected in maximizing the overall encoding rate, indicating a higher overall sample density. On the other hand, within each category, it is essential that the features have a high degree of correlation and closeness. The representation in each category should span only the smallest subspace, and the encoding rate of different subspaces should be minimized as much as possible. Therefore, The ideal representation In the experience The partitioning is done by distinguishing between the complete set and its subsets by a significant contrast in coding rate.

[0057] Specifically, when dealing with subspace distributions of feature representations, with a specified precision An accurate estimate of the number of bits required to encode a finite number of samples with precision can be calculated as follows: ,in . That is, the coding rate of the complete set is obtained.

[0058] Feature Representation of Class Data Can cross A low-dimensional subspace. Divide into subsets, Represents the number of classifications, each subset is located in a different subspace. In contrast, features within the same class or cluster should show high correlation and consistency. Features from the same class or cluster can be expressed as: . Among them, the matrix are diagonal elements, indicating Classes / clusters The membership of samples. Using the matrix To indicate the sample Belongs to category ,otherwise Features from the same class or cluster should occupy a very small subspace to achieve the minimum encoding rate. Calculate the encoding rate of each subset, that is, get the compactness of different subsets:

[0059] ,in , .

[0060] Therefore, the coding rate reduction loss of view-specific feature representation is expressed as:

[0061] .

[0062] Get the optimized feature representation after the coding rate is reduced .in represents the optimized feature representation after the coding rate is reduced. With parameters Therefore, by optimizing To obtain .

[0063] (3) After maximizing the rate reduction of the feature representation of each view, these representations assign classes to different subspaces. In order to obtain consistent feature representations from multiple views, these features are first aligned and then feature fusion is performed. Considering that the representations of various samples within the same classification show similarities, the consensus feature representation of samples in different views must be strengthened by utilizing samples with learned relationship scores. Therefore, in order to achieve consistent multimodal data representation, the goal is to achieve cross-sample fusion by learning the relationship between samples and utilizing it to obtain consistent data representation. Specifically, in order to align each spatial representation between views, the feature space is first transformed and the transformed representation is connected to the consistent representation, that is:

[0064] ,

[0065] in Indicates The view feature transformation matrix, represents the cascade feature, .

[0066] By block diagonal matrix , Mapping , ; ,in . Through the transformed representation matrix and The relationship score of each pair of samples is calculated, and then the softmax function is used to convert the relationship score into a numerical value between 0 and 1. The relationship scores between samples are expressed as follows: ,in express The dimension of This makes the model training more accurate and smooth. Finally, the matrix Multiply them with their corresponding relation scores and then add them together to get the final output:

[0067] ,

[0068]

[0069] in Indicates The feature representation of a sample. Representation Matrix of OK.

[0070] In the equation, the i-th representation can be improved by other samples with high relationship scores. The enhanced feature representation matrix is ​​expressed as .because is derived from the concatenation of all views, so it usually contains redundant information. To alleviate this problem, fully connected non-linear and linear layers are used to map: ,in are the parameters of the network, It is the consistent feature representation after filtering. and The dimension of is kept consistent to facilitate the subsequent mutual information learning.

[0071] (4) Inter-view feature representation based on maximum mutual information: viewpoint-specific representation learned via maximum coding rate reduction It has inter-cluster distinction and intra-cluster similarity, and in order to maintain structural consistency, it further improves the public representation The learned consensus representation is maximized by maximizing the mutual information between the two representations. and each view Mathematically speaking, ,in represents mutual information, which is defined as: ,in is the similarity score between a pair of samples. Using cosine distance to evaluate the similarity between these representations. In order to maximize the mutual information, it can be transformed into maximizing the lower bound of the mutual information, that is, minimizing the loss The loss function encourages the similarity between the consensus feature representation and view-specific feature representation of the same sample, while discouraging the similarity between different samples.

[0072] (5) Through a fully connected layer Will Mapping to the dimension of the specified number of categories to obtain a classification matrix .in is the parameter of the classification network. Then the classification matrix is ​​trained by the cross-label loss function For a single sample, in a neural network based classifier, the cross-label loss is: ,in is the loss of a single sample, is a single code encoding of the true label, indicating whether the sample belongs to For all samples, the cross entropy loss will be: .

[0073] In this embodiment, the total loss of the multimodal classification network includes the maximum coding rate reduction loss function, the maximum mutual information loss function and the cross entropy loss function, namely:

[0074]

[0075] in, From the input To output The coding rate reduction loss is , yes and The approximate mutual information loss between is , is the cross entropy loss. The model reduces the feature representation of the maximum feature reduction rate within the viewpoint, which not only maximizes the feature representation space, but also compresses the feature representation within the class into a smaller representation space, making the feature representation within the viewpoint more compact. At the same time, consistent feature representations are learned through sample fusion to maintain consistency across different views. This causes the representations of pairs with high relationship scores to become more similar, because other samples with high correlation strengthen the consensus representation.

[0076] The specific steps of the testing phase process are:

[0077] Step 2_1: Test set Input into the training model to get the classification matrix , by indexing the largest classification matrix To generate the predicted class of each sample for evaluation.

[0078] In order to further verify the feasibility and effectiveness of the method of the present invention, experiments were conducted.

[0079] In order to further verify the feasibility and effectiveness of the method in this embodiment, relevant verification experiments are carried out. All data sets are converted into vectors as the input of the network, and the autoencoders of all views are implemented using a four-layer MLP and ReLU activation function to learn view-specific representations. We use the provided class labels to fill the diagonal matrix Assign values ​​as follows: Each sample ,set up and . Then keep Unchanged, optimize the total loss and learn the classification matrix Since the invention minimizes the spanning subspace of each class, the learned features can show a clear subspace structure. In order to evaluate the effectiveness of the learned common feature representation, we index the largest classification matrix To generate the predicted class for each sample. Here, two commonly used objective parameters for evaluating classification methods are used as evaluation indicators, namely accuracy and macro F1, to evaluate the classification performance. Experiments are conducted on six public multi-view datasets of different scales, namely Hdigit, Fashion, Synthetic3d, BBC, Caltech-5v, and BDGP. Specifically:

[0080] (1) Hdigit dataset: This dataset is a handwritten digit dataset with two views, 10,000 samples, and dimensions of 784 and 256 respectively.

[0081] (2) Fashion dataset: This dataset contains 28x28 pixel grayscale images of 10 categories (such as clothes, shoes, bags, etc.). We first expand each image into a 784-dimensional vector and then learn the features.

[0082] (3) Synthetic3d dataset: This dataset is a synthetic dataset containing three feature-related views, each of which contains three clusters.

[0083] (4) BBC dataset: This dataset contains 685 documents from BBC news in five categories (business, entertainment, politics, sports, and technology). The documents are described by four views with dimensions of 4659, 4633, 4665, and 4684.

[0084] (5) Caltech-5V dataset: This dataset is a synthetic dataset containing 3D vehicle models from 1,400 images of 7 categories. The dataset contains 5 views, each of which provides object observations from different angles, including WM, CENTRIST, LBP, GIST, and HOG.

[0085] (6) BDGP dataset: This dataset includes images of fruit fly embryos and consists of 2,500 samples representing 5 objects. Each sample has visual features (1,750) and text features (79).

[0086] Table 1 Evaluation results of the method of the present invention on the test set

[0087] ACC F1 Hdigit 95.85% 95.84% Fashion 91.20% 91.07% Synthetic3d 99.17% 99.13% BBC 96.35% 96.33% Caltech-5v 97.50% 97.45% BDGP 98.6% 98.32%

[0088] like Figure 3 As shown in FIG. 1 , it is a schematic diagram of the impact of evaluation parameters on the model, namely, the temperature parameter τ and the trade-off factor λ. That is, the λ parameter is kept unchanged, while the balance factor λ and the temperature parameter τ are changed. Figure 3 The variation of ACC results of datasets BDGP and Synthetic3d when λ and τ take different values ​​[0.1, 0.5, 1] ​​is described. λ varies from 10-3 to 103, and τ varies from 0.2 to 0.8. When ε = 0.5. In the range of λ = 0.001 ~ 1 and τ = 0.3 ~ 0.5, the classification results are stable, and λ and τ have no significant effect on the classification results. Therefore, ε = 0.5, λ = 0.01, τ = 0.5 are set.

[0089] Figure 3 (a) The impact of temperature parameter τ and trade-off factor λ on the model when ε = 0.1 is set for the BDGP dataset; (b) The impact of temperature parameter τ and trade-off factor λ on the model when ε = 0.1 is set for the Synthetic3d dataset; (c) The impact of temperature parameter τ and trade-off factor λ on the model when ε = 0.5 is set for the BDGP dataset; (d) The impact of temperature parameter τ and trade-off factor λ on the model when ε = 0.5 is set for the Synthetic3d dataset; (e) The impact of temperature parameter τ and trade-off factor λ on the model when ε = 1 is set for the BDGP dataset; (f) The impact of temperature parameter τ and trade-off factor λ on the model when ε = 1 is set for the Synthetic3d dataset.

[0090] like Figure 4 As shown, Figure 4 (a) is the subjective confusion matrix of the BDGP dataset. Figure 4 (b) is the subjective confusion matrix of the Fashion dataset. Figure 4(c) in the figure is the subjective confusion matrix of the Synthetic3d dataset. Figure 4 (d) in the figure is the subjective confusion matrix of the Calteth-5v dataset. Figure 4 (e) in the figure is the subjective confusion matrix of the Hdigit dataset. Figure 4 (f) in the figure is the subjective confusion matrix of the BBC dataset. The values ​​in the confusion matrix represent the number of accurate and inaccurate predictions for each class. In the graphical representation of the confusion matrix, each column represents the predicted class for each dataset, while each row represents the true class to which the data belongs. The diagonal elements represent the correct classification rate of each type of sample. Darker colors correspond to higher values, indicating better classification results. The denominator is obtained by summing the dimensions where the actual labels are located and then dividing the number of cases in each small cell within that dimension by this sum. This provides a comprehensive overview of the model's classification capabilities, including accurate and inaccurate predictions for each class. In the BDGP dataset, the classifier has a three-class classification accuracy of 93% for the original data, while the misclassification rate is 7%. Of these misclassifications, 6% were incorrectly labeled as class 4 and 1% were incorrectly labeled as class 2. For the Caltech5v data, the model tends to misclassify samples from other classes as class 1. This result may be due to the fact that the learned features of these samples are not sufficiently different from the features of samples belonging to class 1.

[0091] As a further embodiment,

[0092] Figure 2 The multimodal classification network framework to be trained in this embodiment is a multimodal classification network based on optimizing information theory metrics, which is composed of an intra-view feature representation module based on maximum coding rate reduction (Intra-ViewFeature Representation module based on Maximal Coding Rate Reduction, Intra-MCR 2 ), spatial transformation and Cross-samplefeature Aggregation (SC-FA) module and Inter-ViewFeature Representation module based on Maximal Mutual Information (Inter-MMI) module.

[0093] Example 2

[0094] This embodiment provides a multimodal classification system based on maximizing coding rate reduction, including:

[0095] A data acquisition module is configured to acquire multimodal image data;

[0096] A preprocessing module is configured to preprocess the acquired multimodal image data to obtain a training set and a test set;

[0097] The feature module is configured to extract features from multimodal image data using a multimodal classification network model and learn the intrinsic relationship of multimodal features using an attention mechanism;

[0098] The training module is configured to train a multimodal classification network model based on maximizing the coding rate reduction loss, improving the sample similarity and the cross entropy loss respectively;

[0099] The classification module is configured to use a multimodal classification network model to perform predictive classification and obtain a predicted classification probability.

[0100] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the multimodal classification method based on maximizing coding rate reduction.

[0101] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the multimodal classification method based on maximizing coding rate reduction.

[0102] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal classification method based on maximizing coding rate reduction, characterized in that include: Acquire multimodal image data; Preprocess the acquired multimodal image data to obtain a training set and a test set; Perform feature extraction on multimodal image data to extract feature representations from different perspectives; By maximizing the overall coding rate and compressing the coding rates of different subspaces, the feature representation of different perspectives is optimized based on maximizing the coding rate and reducing the loss; The maximum mutual information loss is constructed with the goal of maximizing the mutual information between feature representation and global feature representation; The cross entropy loss is constructed by mapping the feature representation to a specified number of category dimensions to obtain a score for each category. The multimodal classification network model is trained based on cross entropy loss, maximum coding rate reduction loss and maximum mutual information loss respectively; The multimodal classification network model is used to perform predictive classification and obtain the predicted classification probability.

2. A multimodal classification method based on maximizing coding rate reduction according to claim 1, characterized in that: The method of preprocessing the acquired multimodal image data to obtain a training set and a test set includes preprocessing the image data to obtain a processed multimodal image data set, including a training set and a test set, wherein the training set ,Include samples and views, the dataset is defined as , in Indicates View primitive features.

3. A multimodal classification method based on maximizing coding rate reduction according to claim 2, characterized in that: The method of maximizing the overall coding rate and compressing the coding rate of different subspaces is based on maximizing the coding rate to reduce the loss, optimizing the feature representation of different perspectives, including optimizing the feature representation of each perspective in the training set by maximizing the coding rate to reduce the loss, and the loss is expressed as: ， in , , , and obtain the optimized feature representation after the coding rate is reduced ,in represents the optimized feature representation after the coding rate is reduced. With parameters Therefore, by optimizing To obtain .

4. A multimodal classification method based on maximizing coding rate reduction according to claim 3, characterized in that: The cross entropy loss is constructed by mapping the feature representation to a specified number of category dimensions to obtain the score of each category, including cascading the features of all perspectives to obtain a global feature representation after aligning the feature representation space, which is expressed as: , in, Indicates The view feature transformation matrix, represents the cascade feature, , through the block diagonal matrix , Mapping , ; ,in .

5. A multimodal classification method based on maximizing coding rate reduction according to claim 4, characterized in that: The cross entropy loss is constructed by mapping the feature representation to a specified number of category dimensions to obtain a score for each category, and also includes a fully connected layer Will Mapping to the dimension of the specified number of categories to obtain a classification matrix ,in are the parameters of the classification network, and the classification matrix is ​​trained through the cross-label loss function , where the cross entropy loss is expressed as: in is the loss of a single sample, is a single code encoding of the true label, indicating whether the sample belongs to classes, for all samples.

6. A multimodal classification method based on maximizing coding rate reduction according to claim 5, characterized in that: The maximum mutual information loss is constructed with the goal of maximizing the mutual information between the feature representation and the global feature representation, including optimizing the consistency of the common features by maximizing the mutual information between the common features and each view feature, wherein the mutual information is expressed as: ， in is the similarity score between a pair of samples, is a common feature, using cosine distance to evaluate the similarity between these representations.

7. A multimodal classification method based on maximizing coding rate reduction according to claim 6, characterized in that: The multimodal classification network model is used to perform prediction classification to obtain the prediction classification probability, including the test set Input into the training model to get the classification matrix , by indexing the largest classification matrix To generate the predicted class of each sample for evaluation.

8. A multimodal classification system based on maximizing coding rate reduction, characterized in that include: A data acquisition module is configured to acquire multimodal image data; A preprocessing module is configured to preprocess the acquired multimodal image data to obtain a training set and a test set; The feature extraction module is configured to perform feature extraction on the multimodal image data to extract feature representations of different viewing angles; The maximum coding rate reduction loss module is configured to maximize the overall coding rate and compress the coding rates of different subspaces to construct feature representations based on maximizing the coding rate and reducing the loss, and optimize the feature representations of different perspectives; The maximum mutual information loss module is configured to construct the maximum mutual information loss with the goal of maximizing the mutual information between the feature representation and the global feature representation; A cross entropy loss module is configured to construct a cross entropy loss by mapping feature representations to a specified number of category dimensions to obtain a score for each category; The training module is configured to train a multimodal classification network model based on a cross entropy loss, a maximum coding rate reduction loss, and a maximum mutual information loss, respectively; The prediction module is configured to use the multimodal classification network model to perform prediction classification and obtain the prediction classification probability.

9. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the method according to claim 1 .

10. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method as claimed in claim 1 .

Images