Self-adaptive multi-modal classification device and method based on lost modal mining, equipment and medium

Abstract

The invention provides a self-adaptive multi-modal classification device and method based on lost modal mining, equipment and a medium, and the method comprises the steps: carrying out the preprocessing of originally input images and table data, and training a cross-modal cross mining model; when data input by a user is lost, extracting features by using an image or numerical encoder, restoring features of a missing mode from a database in a feature mining manner through a pre-trained cross-modal cross mining model, and then performing feature fusion; fusing the features of the two modes by using an improved bidirectional focusing algorithm to obtain a multi-mode fusion feature, enhancing the multi-mode fusion feature by using five-layer self-focusing module stacking, inputting the multi-mode fusion feature into a classification network, and training the classification network to obtain a multi-mode classification model; the method is used for performing category prediction on unknown data input by a user in actual use. The method solves the problem that the existing multi-mode classification process cannot adapt to the loss of multiple modes, and has higher robustness.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to an adaptive multimodal classification device, method, equipment and medium based on lost mode mining. Background Art

[0002] With the rapid development of deep learning technology, its application in various fields has become a hot topic of public concern. Traditional deep learning techniques focus solely on single-modal data, such as images, text, structured tables, and audio. These methods currently face bottlenecks in many areas and struggle to meet the practical requirements of production applications. However, multimodal deep learning methods, which combine multiple modalities, are beginning to demonstrate breakthrough potential. Compared to single-modal methods, multimodal learning offers numerous advantages. It can learn richer semantic information from data in various modalities, while also complementing features across modalities, thereby enhancing model robustness and stability. Furthermore, for most practical applications, the data required is inherently multimodal. For example, in clinical settings, doctors will utilize a comprehensive approach to diagnose using a combination of medical images, laboratory test results, and medical history data, among other modalities. In the metallurgical manufacturing industry, technicians use images, parameters, and other data to identify steel defects. The application of multimodal deep learning technology simulates real-world production scenarios.

[0003] However, while multimodal learning presents new opportunities, it also presents new challenges. In the real world, data is often imperfect, and modality loss often occurs. This modality loss can occur during model training or during actual deployment, potentially involving the loss of one or more modalities. The causes of modality loss can be summarized into two main categories. The first is the high cost or difficulty of acquiring data for a particular modality. For example, in medical scenarios, patients with an allergy to contrast agents cannot obtain complete medical image data. The second is low data quality, rendering the data unusable. For example, improper data storage can lead to data corruption. Once modality loss occurs, it can have multiple negative impacts on multimodal learning. First, for models that don't account for modality loss, modality loss in the input data can directly render the classification tool inoperable. Second, modality loss can alter the distribution of the data for that modality, leading to biased classification results. Recent research has also found that when data loss for a particular modality reaches a certain ratio, that modality becomes noise, causing the classification model's accuracy to fluctuate or even decline.

[0004] Due to previous technological limitations, data with missing modalities often had to be eliminated, leaving a large amount of valuable data unusable. Later studies attempted to use interpolation techniques to restore the lost data as much as possible, but this approach is impractical for image data loss and severe modality loss. In recent years, with the advancement of deep learning technology, many studies have used complex methods such as generative adversarial networks, knowledge distillation, and shared latent spaces to restore lost modalities. However, due to the highly specialized and heterogeneous nature of data in some fields, restoring authentic and reliable data features is difficult. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides an adaptive multimodal classification device, method, equipment and medium based on missing modal mining. By utilizing the fact that various modal data of the same category in the sample have similar features, the present invention solves the problem that existing multimodal classification methods have difficulty in effectively utilizing missing modal data through simple and efficient inter-modal feature mining.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] Adaptive multimodal classification method based on lost mode mining:

[0008] Preprocess the original input image data and table data for training cross-modal cross-mining models;

[0009] When the user inputs image data and table data at the same time, the numerical encoder and image encoder are directly used to extract features and then perform feature fusion;

[0010] When the user only inputs image data or table data, the image encoder or numerical encoder is used to extract features, and the pre-trained cross-modal cross-mining model is used to restore the features of the missing modality from the database through feature mining, and then perform feature fusion;

[0011] The feature fusion is specifically as follows: calculating the focusing weight of the image feature X1 to the table feature X2 to obtain the fusion feature of X2 based on X1; calculating the focusing weight of the image feature X2 to the table feature X1 to obtain the fusion feature of X1 based on X2; and concatenating the above two fusion features to obtain the final multimodal fusion feature;

[0012] Enhance multimodal fusion features: Use a five-layer stack of autofocus modules to capture the deep relationship between multimodal fusion features, perform information interaction layer by layer, and ultimately obtain fusion features that have been enhanced five times;

[0013] The fused features enhanced five times are input into the classification network, and the classification network is trained to obtain a multimodal classification model;

[0014] In actual use, after the user inputs unknown data, the multimodal classification model determines whether the input data is missing a modality. If the modality is missing, it is restored. After feature fusion, feature enhancement and multimodal classification, the category prediction result is finally obtained.

[0015] Furthermore, during training, the cross-modal cross-mining model uses a mutual information contrast loss function to constrain the model. The mutual information contrast loss function is:

[0016]

[0017] Where: f(z) and f(z′) are two features of a positive correlation pair, N is the total number of samples, and MI() represents the mutual information between the two features.

[0018] Furthermore, the focusing weight from the image feature X1 to the table feature X2 includes:

[0019] Assign learnable parameter matrices to the two modal features X1 and X2 respectively and We get two new features:

[0020]

[0021] Calculate the focus weight A from X1 to X2 by inner product 1→2 , reflecting the modal interaction relationship between X1 and X2:

[0022]

[0023] Where d is the dimension of X′1 and X′2, is the scaling factor;

[0024] Apply the focus weight to X2 to calculate the fused features of X2 based on X1:

[0025] O 1→2 =A 1→2 (X2W 1→2 )

[0026] Among them, W 1→2 are learnable parameters.

[0027] Furthermore, the five-layer autofocus module stack is used to capture the deep relationship of multimodal fusion features, specifically:

[0028] Multimodal fusion feature F fusion Make a copy and give two different learnable parameters W f (1) and W f(2) , construct two new fusion features F′1 and F′2:

[0029] F′1=F fusion W f (1)

[0030] F′2=F fusion W f (2)

[0031] Calculation based on F fusion Its own focus weight:

[0032]

[0033] Get the weight vector of the first autofocus module:

[0034] O1=A f (F fusion W f )

[0035] Among them, W f is a learnable parameter;

[0036] Except for the first autofocus module, the other autofocus modules take the output of the previous autofocus module as input. Residual connection and layer normalization are added after each autofocus module, and finally the fused features after 5 enhancements are obtained.

[0037] Furthermore, the image encoder consists of a convolutional layer, four residual blocks and a global maximum pooling layer. The global maximum pooling reduces the spatial dimension to 1*1, thereby outputting features of a fixed size of 2048 dimensions.

[0038] An adaptive multimodal classification device based on lost mode mining, comprising:

[0039] Data preprocessing module, which preprocesses the original input image data and table data;

[0040] A modal feature extraction module, comprising an image encoder and a numerical encoder, wherein the image encoder is used to extract image features, and the numerical encoder is used to extract table data features;

[0041] The modality restoration module, when the input modality is incomplete, uses a pre-trained cross-modal cross-mining model to restore the features of the missing modality from the database through feature mining;

[0042] Modal feature fusion module, used to integrate multimodal features;

[0043] The classification module uses the fused features to train the classification network and obtain a multimodal classification model.

[0044] An electronic device comprising a memory and a processor;

[0045] The memory is used to store computer programs;

[0046] The processor is configured to execute the computer program and implement the above-mentioned adaptive multimodal classification method when executing the computer program.

[0047] A storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the above-mentioned adaptive multimodal classification method.

[0048] The beneficial effects of the present invention are:

[0049] (1) The present invention uses multimodal data for classification tasks, simulating the real scenario of comprehensive consideration of multiple information in application practice; compared with traditional multimodal methods, the present invention can provide more accurate and more interpretable classification results, promoting the further development of intelligent applications.

[0050] (2) The modal restoration module designed in this invention can simply and efficiently enable the model to adapt to various modal loss situations, thereby improving the robustness of the multimodal classification program. Furthermore, due to its ability to address modal loss, data from missing modalities that would previously have been discarded can be fully utilized, significantly reducing data acquisition costs in practical applications. The method can also be applied to a variety of different real-world scenarios.

[0051] (3) The present invention uses multiple self-focusing modules for feature learning, which enables the model to have better representation learning capabilities for high-dimensional complex features, prompting the model to further improve the classification accuracy.

[0052] (4) The pre-trained cross-modal mining model in the modality restoration module of the present invention has added value. This model can not only be used to address the problem of modality loss, but can also be used independently as a multimodal data intelligent retrieval tool. In practical applications, as the scale of existing data expands, more data can be used to fine-tune the mining model, continuously improving the model's performance. As the mining model's performance improves, the model's restoration of lost modal data will become more realistic, thereby improving the overall efficiency of the model. The two complement each other, with good scalability and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of the adaptive multimodal classification based on lost mode mining according to the present invention;

[0054] Figure 2 This is a schematic diagram of the numerical encoder of the present invention;

[0055] Figure 3 Schematic diagram of the image encoder according to the present invention;

[0056] Figure 4 This is a schematic diagram of the modal restoration module of the present invention;

[0057] Figure 5 Schematic diagram of the multimodal feature fusion module of the present invention;

[0058] Figure 6 Schematic diagram of the multimodal classification module of the present invention. DETAILED DESCRIPTION

[0059] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0060] like Figure 1 As shown, the present invention provides an adaptive multimodal classification method based on lost mode mining, which specifically includes the following steps:

[0061] Step (1) preprocesses the original input image and table data, including image normalization, denoising and size standardization, and table normalization.

[0062] In order to reduce the differences between different image sources and speed up the convergence of the model, the pixel values ​​of the image need to be normalized. The specific operation is to divide each pixel value by the maximum pixel value so that the pixel value range is adjusted to between 0 and 1. The calculation process is as follows:

[0063]

[0064] Among them, I(x,y) represents the original pixel value at the coordinate (x,y) in the image, I min and I max are the minimum and maximum pixel values ​​in the image, I norm (x,y) is the normalized pixel value.

[0065] Images may contain noise, which can affect the model's learning process. Therefore, removing noise can help improve image quality and classification results. Image denoising is performed using Gaussian filtering, implemented using the GaussianBlur function in OpenCV.

[0066] Because different imaging devices can cause image sizes to vary, making it difficult for the machine to perform tensor calculations, all images need to be processed to the same dimension. OpenCV's resize function is used to uniformly resize the images. The largest image in the sequence is selected as the standard, and the smaller images are resized using interpolation.

[0067] In order to make the model learn more efficiently, the tabular data is also normalized. The z-score method is used to scale the data to a standard normal distribution. The specific implementation formula is as follows:

[0068]

[0069] Where: Z i is the value of the i-th data after normalization, X i is the i-th data value, It represents the mean value of a certain attribute feature of all samples.

[0070] Step (2), train the cross-modal cross-mining model.

[0071] After preprocessing in step (1), the complete training data of all modalities is used to train the cross-modal cross-mining model. In this step, two modality-specific encoders are first constructed to extract data features. Then, a multimodal contrast loss function is used to constrain the model convergence. The pre-trained model is saved for subsequent feature restoration of the lost modality.

[0072] 1) Build a numerical encoder

[0073] Numerical encoders such as Figure 2 As shown, the backbone network consists of two fully connected layers (FCs) for extracting features from tabular data. The input layer takes numeric data of dimension n, which is then mapped to 1024-dimensional intermediate features by the first FC layer and resized to 512 dimensions by the second FC layer. All FC layers use ReLU as the activation function.

[0074] 2) Build an image encoder

[0075] Image encoders such as Figure 3As shown, the backbone network is a residual network consisting of a convolutional layer, four residual blocks, and a pooling layer for extracting image features. The input image is a 224*224*3 RGB image. This is followed by a 64-channel convolutional layer with a 7*7 kernel size, followed by a 3*3 max pooling layer. Both the convolutional and pooling layers have a stride of 2. The four residual blocks contain 3, 4, 6, and 3 Bottleneck residual units, respectively. Each Bottleneck residual unit contains three convolution layers: the first 1*1 convolution is used for dimensionality reduction, the last 1*1 convolution is used for dimensionality increase, and the middle 3*3 convolution is a standard convolution operation. The Bottleneck residual units in each residual block use skip connections, directly adding the input of each Bottleneck residual unit to its output and feeding it into the next Bottleneck residual unit. This addresses the vanishing gradient problem associated with deeper network layers. The residual block design reduces the number of parameters while maintaining the original representation capability. After the last residual block, global max pooling (GMP) is used to reduce the spatial dimension to 1*1, outputting a fixed-size feature of 2048 dimensions. Finally, a linear transformation is performed to map the feature to 512 dimensions, making it the same as the feature dimension of the tabular data.

[0076] 3) Multimodal feature pairing pre-training

[0077] After the previous step, the image and table data are mapped to 512-dimensional features. The features of N samples form tokens, which are used for multimodal feature matching tasks. Figure 4 As shown in the figure, during the training process, the image and table data of the same sample are regarded as positively correlated pairs, and the data of all other different samples are regarded as negatively correlated pairs. The training task of the model is to pair all positive sample pairs as much as possible in the same feature space, while differentiating the negative sample pairs, so that the data of the two modalities are aligned in the same feature space.

[0078] The present invention uses mutual information to measure the similarity between modalities, which is specifically implemented as follows:

[0079]

[0080] Where x and y represent the data of the two modalities (image and table), sim(x,y) represents the similarity score of the two modalities, f(x) and f(y) represent the features of their respective modalities, MI(f(x), f(y)) represents the mutual information between the features of the two modalities, p(f(x)) and p(f(x′)) represent the marginal probability distributions of the two features, and p(f(x), f(x′)) represents the joint probability distribution of the two modalities.

[0081] Based on the measurement of the mutual information between the two modalities, this paper designs a mutual information contrast loss function to constrain the training of the model. The formula is as follows:

[0082]

[0083] Where: f(z) and f(z′) are two features of a positively correlated pair, and N is the total number of samples.

[0084] The ultimate training goal is to minimize the loss function L. The optimization function of the training process is Adam, and the initial learning rate is set to 10 -3 The training round is 1000, and the early stopping mechanism is used. When the loss does not decrease for 100 consecutive rounds, the training network is considered to have converged, and the model G is θ Save to device.

[0085] Step (3): adaptive modal feature restoration.

[0086] The present invention can adaptively select processing branches according to different multi-modal inputs, which are divided into three input situations. When the data modality input by the user is complete, model G θ No modal restoration is performed, and the numerical encoder and image encoder are directly used to extract features and execute step (4). When the user inputs image data or table data, the input modal data is encoded by the corresponding encoder to obtain features, and the pre-trained cross-modal cross-mining model G θ The features of the existing input data are used to restore the features of the missing modality from the database (formed by the original input image and table data) through feature mining. Specifically, based on the features of the existing input data, the pre-trained cross-modal cross-mining model G θ We extract features from the database for the other modality's data, calculate the similarity between the features of the input data and each feature of the other modality, sort the results from high to low, and determine the features of the top k (presumably set empirically) samples with the highest similarity. To improve efficiency, we use a fast selection sort algorithm implemented using NumPy's partition function. We perform a weighted sum of the similarities of the k most similar features, using the following formula:

[0087]

[0088] Among them: v is the reduced feature, v i is the i-th feature among the first k features, sim i and sim j Respectively represent the similarity between the features of the existing input data and the i-th and j-th features.

[0089] Since the information of data from different modalities of the same sample in the semantic space should be similar, and the same category often has similar image and numerical features, the present invention discovers features similar to other sample information from the existing data to restore the lost modal features.

[0090] Step (4), feature fusion.

[0091] The features X1 and X2 of the two modal data, image and table, are fused using the improved bidirectional multimodal focusing algorithm. Figure 5 As shown, in order to obtain a more effective multimodal fusion representation, the present invention is designed to allow the model to learn the interaction relationship between modalities and weight the features accordingly, so that the model can focus on more important features.

[0092] 1) Calculate the focus weight from X1 to X2

[0093] First, assign learnable parameter matrices to the two modal features X1 and X2 respectively and We get two new features:

[0094]

[0095] Then calculate the focus weight A from X1 to X2 by inner product 1→2 , reflecting the modal interaction relationship between X1 and X2, the formula is as follows:

[0096]

[0097] Where d is the dimension of X′1 and X′2, It is a scaling factor used to prevent the input from being too large, causing the output of the softmax function to approach 0 and causing the gradient to disappear;

[0098] Finally, the focus weight is applied to X2 to calculate the fusion feature of X2 based on X1:

[0099] O 1→2 =A 1→2 (X2W 1→2 )

[0100] Among them, W 1→2 are learnable parameters.

[0101] 2) Calculate the focus weight from X2 to X1

[0102] Similar to the above operation, calculate the fusion features of X1 based on X2:

[0103] O 2→1 =A 2→1 (X1W 2→1)

[0104] Among them, W 2→1 are learnable parameters.

[0105] 3) Multimodal bidirectional focus fusion

[0106] The final multimodal fusion feature F is obtained through the splicing operation fusion =[O 1→2 ;O 2→1 ], where [·;·] is a concatenation operation, and the resulting fusion feature contains the interactive information between the two modalities.

[0107] Step (5), feature enhancement.

[0108] After obtaining the fusion features of the two modalities in step (4), in order to further improve the representation effect, the features are further enhanced before inputting into the classification network. Specifically, a five-layer self-focusing module stack is used to capture the deep relationship of the features and to perform information interaction layer by layer, so that the network can focus on important features while suppressing redundant information. fusion Make a copy and give two different learnable parameters W f (1) and W f (2) , construct two new fusion features F′1 and F′2, the specific formula is as follows:

[0109] F′1=F fusion W f (1)

[0110] F′2=F fusion W f (2)

[0111] Then calculate based on F fusion Its own focus weight:

[0112]

[0113] Finally, the weight vector of the first autofocus module is obtained:

[0114] O1=A f (F fusion W f )

[0115] Among them, W f are learnable parameters.

[0116] The feature enhancement module consists of five stacked autofocus modules. Except for the first, all other modules use the output of the previous module as input. To stabilize training and prevent vanishing gradients, residual connections and layer normalization are added after each autofocus module. The resulting fused feature O5, after five enhancements, is used for classification prediction.

[0117] Step (6), train the classification network.

[0118] 1) Build a classification network

[0119] As indicated Figure 6 As shown in Figure 2, the classification network consists of three layers, of which the first two layers are fully connected layers and the last layer is the output classification layer. In step (5), the enhanced fusion feature O5 is first input into the two fully connected layers in sequence, and the activation function is ReLU. The specific formula is as follows:

[0120] h1=ReLU(O5W1+b1)

[0121] h2=ReLU(h1W2+b2)

[0122] Among them, h1 and h2 are hidden layer dimensions, W i and b i (i=1,2) are all learnable parameters.

[0123] In order to prevent overfitting, the dropout regularization method is used to randomly discard some neurons. Finally, the input is sent to the classification layer and the classification result is obtained through the softmax function. That is, the probability distribution of each category is as follows:

[0124]

[0125] Among them, W3 and b3 are both learnable parameters.

[0126] 2) Determine the loss function

[0127] Use cross entropy loss to constrain the classification network for training. Use one-hot encoding to define the true label y, and calculate the cross entropy between the true label and the predicted result using the following formula:

[0128]

[0129] Among them, y i and Denote the true label and prediction result of the i-th category respectively. The training goal of the classification network is to minimize this loss.

[0130] The optimizer used in the entire training process is Adam, and the initial learning rate is set to 10-5 The training round is 1000, and the early stopping mechanism is used. When the loss does not decrease after 100 consecutive rounds, the training network is considered to have converged, and the multimodal classification model M is used. θ Save to device.

[0131] Step (7), in actual use, after the user inputs unknown data, the multimodal classification model automatically determines whether the input data is missing a modality. If the modality is missing, it is restored, and then through feature fusion, feature enhancement and multimodal classification, the final category prediction result is obtained.

[0132] An adaptive multimodal classification device based on lost modality retrieval includes a data preprocessing module, a modality feature extraction module, a modality restoration module, a modality fusion module and a classification module.

[0133] The data preprocessing module normalizes, denoises and standardizes the original input image, and normalizes the original input table.

[0134] The modal feature extraction module consists of an image encoder and a numerical encoder. The image encoder is responsible for extracting image features. The network depth is 50 layers, including one convolutional layer, four residual blocks, and one pooling layer. The residual blocks are constructed by skip connections. The numerical encoder is responsible for extracting tabular data features and consists of two fully connected layers. All data are mapped into 512-dimensional feature vectors after passing through the corresponding encoder.

[0135] The modality restoration module is automatically called when the input modality is incomplete. It uses a pre-trained cross-modal cross-mining model to mine samples with the closest features from the existing modalities of similar samples in the database to restore the lost modality features.

[0136] The modal fusion module is used to integrate multimodal features, calculate the interaction information between the two modalities using a multimodal bidirectional focusing algorithm, and then perform weighted fusion to obtain a better representation, prompting the model to focus on important information while suppressing redundant information.

[0137] The classification module is responsible for using the fused features for category prediction. It consists of two fully connected layers and one classification layer, and the model convergence is constrained by cross entropy loss.

[0138] The present invention relates to an adaptive multimodal classification device based on missing modal retrieval, which can be implemented in the form of a computer program and can be run on a computing device. The computing device can be a terminal device or a server. The server can be independent or a server cluster; the terminal device includes a tablet computer, a desktop computer, a notebook computer, a mobile phone, a wearable device or a personal digital assistant, etc. The computing device is composed of a network interface, a memory and a processor connected by a system bus, wherein the memory includes a memory and a non-volatile storage medium, the latter of which can store an operating system and a computer program. The computer program contains program instructions, and when the processor executes these instructions, it can execute a multimodal classification method for missing modal data. The processor provides control and computing capabilities to ensure the normal operation of the computing device. The memory provides an operating environment for the computer program stored in the non-volatile storage medium, and when the processor executes these program instructions, classification can be performed.

[0139] In addition, the network interface supports network communication functions such as task distribution, etc. It should be noted that the processor can be a digital signal processor (DSP), a central processing unit (CPU), a general-purpose processor, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC) or other programmable logic device, a discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0140] The present invention also provides a computer-readable and writable storage medium storing a computer program containing program instructions. When a processor executes these instructions, a multimodal classification method utilizing missing modality data can be implemented. The computer-readable and writable storage medium can be an internal storage unit of a device, such as a memory hard disk, or an external storage device, such as a smart memory card, a plug-in hard disk, a flash memory card, a secure digital card (SD), etc.

[0141] The embodiments described are preferred implementations of the present invention, but the present invention is not limited to the above implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention are within the scope of protection of the present invention.

Claims

1. An adaptive multimodal classification method based on lost mode mining, characterized by: Preprocess the original input image data and table data for training cross-modal cross-mining models; When the user inputs image data and table data at the same time, the numerical encoder and image encoder are directly used to extract features, and then feature fusion is performed; When the user only inputs image data or table data, the image encoder or numerical encoder is used to extract features, and the pre-trained cross-modal cross-mining model is used to restore the features of the missing modality from the database through feature mining, and then feature fusion is performed; The feature fusion is specifically: calculating the focusing weight of the image feature X1 to the table feature X2 to obtain the fusion feature of X2 based on X1, calculating the focusing weight of the image feature X2 to the table feature X1 to obtain the fusion feature of X1 based on X2, and concatenating the above two fusion features to obtain the final multimodal fusion feature; Enhance the multimodal fusion features: Use 5 layers of self-focus modules to capture the deep relationship of multimodal fusion features, perform information interaction layer by layer, and finally obtain fusion features that have been enhanced 5 times; The fused features enhanced five times are input into the classification network, and the classification network is trained to obtain a multimodal classification model; In actual use, after the user inputs unknown data, the multimodal classification model determines whether the input data is missing a modality. If the modality is missing, it is restored. After that, feature fusion, feature enhancement and multimodal classification are performed to finally obtain the category prediction result.

2. The adaptive multimodal classification method according to claim 1, characterized in that: When training the cross-modal cross-mining model, a mutual information contrast loss function is used to constrain the model. The mutual information contrast loss function is: Among them: f(z) and f(z′) are two features of a positively correlated pair, N is the total number of samples, and MI() represents the mutual information between the two features.

3. The adaptive multimodal classification method according to claim 1, characterized in that: The focusing weight from the image feature X1 to the table feature X2 includes: Assign learnable parameter matrices to the two modal features X1 and X2 and We get two new features: Calculate the focus weight A from X1 to X2 through the inner product 1→2 , reflecting the modal interaction relationship between X1 and X2: Where d is the dimension of X′1 and X′2, is the scaling factor; Apply the focus weight to X2 to calculate the fused features of X2 based on X1: A 1→2 =A 1→2 (X2W 1→2 ) Among them, W 1→2 are learnable parameters.

4. The adaptive multimodal classification method according to claim 1, characterized in that: The five-layer self-focus module stack is used to capture the deep relationship of multimodal fusion features, specifically: For multimodal fusion feature F fusion Make a copy and give two different learnable parameters W f (1) and W f (2) , construct two new fusion features F′1 and F′2: F′1=F fusion W f (1) F′2=F fusion W f (2) Calculation based on F fusion Its own focus weight: Get the weight vector of the first autofocus module: O1=A f (F fusion W f ) Among them, W f is a learnable parameter; Except for the first autofocus module, other autofocus modules take the output of the previous autofocus module as input. Residual connection and layer normalization are added after each autofocus module, and finally the fused features are enhanced five times.

5. The adaptive multimodal classification method according to claim 1, characterized in that: The image encoder consists of a convolutional layer, four residual blocks and a global maximum pooling layer. The global maximum pooling reduces the spatial dimension to 1*1, thereby outputting features of a fixed size of 2048 dimensions.

6. A device for implementing the adaptive multimodal classification method according to any one of claims 1 to 5, characterized in that: include: Data preprocessing module, preprocessing the original input image data and table data; A modal feature extraction module, composed of an image encoder and a numerical encoder, wherein the image encoder is used to extract image features, and the numerical encoder is used to extract table data features; The modality restoration module, when the input modality is incomplete, restores the features of the missing modality from the database through feature mining using a pre-trained cross-modality cross-mining model; Modal feature fusion module, used to integrate multi-modal features; The classification module uses the fused features to train the classification network and obtain a multimodal classification model.

7. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the adaptive multimodal classification method according to any one of claims 1 to 5 when executing the computer program.

8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the adaptive multimodal classification method according to any one of claims 1 to 5.

Images