Multimodal data classification method and system based on deep learning

Abstract

The present invention proposes a multimodal data classification method and system based on deep learning. The method is as follows: obtain N groups of data and divide them into a training set, a validation set, and a test set; train the training set; record the validation result after the last training as a matrix S; use the test set to test the deep learning network model after training, and obtain a softmax matrix R for each group of test data; use the elements in the matrix S as weights to multiply the elements in the softmax matrix R corresponding to each group of test data according to the corresponding modal features of the corresponding classification to obtain a matrix F for each group of test data; add the probabilities corresponding to different modes of the same classification in the matrix F to obtain a matrix Fr; perform softmax processing on the matrix Fr to obtain the final classification result. This multimodal data classification method based on deep learning reduces the overfitting of the neural network and improves the classification ability of the multimodal neural network.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a multimodal data classification method and system based on deep learning. Background Art

[0002] Multimodality refers to the combination of two or more modalities in various forms. Each source or form of information can be referred to as a modality. Modal feature fusion is a recently emerging topic in deep learning, effectively improving the learning and classification capabilities of neural networks. Currently, processing multimodal data faces challenges such as complex network design and slow training speed, hindering the full potential of multimodal feature fusion. Summary of the Invention

[0003] In order to overcome the defects in the above-mentioned prior art, the purpose of the present invention is to provide a multimodal data classification method and system based on deep learning.

[0004] In order to achieve the above-mentioned object of the present invention, the present invention provides a multimodal data classification method based on deep learning, comprising the following steps:

[0005] Obtain N groups of data, each group of data including multiple modal features, and divide the N groups of data into a training set, a validation set, and a test set;

[0006] Constructing a deep learning network model, which includes multiple neural networks, and training the data of the training set in the deep learning network model. During training, the features of different modalities are trained using the neural networks corresponding to the modalities;

[0007] Use the validation set to validate the trained deep learning network model, and record the validation result after the last training as matrix S. Each element in the matrix S is the classification precision of each modal feature of each classification;

[0008] The deep learning network model after training is tested using the test set. During the test, the features of different modalities are calculated using the trained neural network corresponding to the modality. A softmax matrix R is obtained for each set of test data. Each element in the softmax matrix R is the softmax value of each modal feature of each classification.

[0009] The softmax matrix R corresponding to each set of test data is weighted and integrated, and the elements in the matrix S are used as weights to multiply the elements in the softmax matrix R corresponding to each set of test data according to the corresponding classification and modal features to obtain the matrix F of each set of test data;

[0010] Add the probabilities corresponding to different modes of the same classification in the matrix F corresponding to each set of test data to obtain the final softmax result matrix Fr of each set of test data;

[0011] Perform softmax processing on the final softmax result matrix Fr corresponding to each set of test data to obtain the final classification result of each set of test data.

[0012] This multimodal data classification method based on deep learning reduces the complexity of the network, makes the training of each modality independent, reduces the overfitting of the neural network, and improves the classification ability of the multimodal neural network.

[0013] The preferred solution of the multimodal data classification method based on deep learning: the format of the matrix S is Among them, P mn is the classification precision of the features of the mth classification and the nth mode;

[0014] The format of the softmax matrix R is Among them, R mn It is the softmax value of the mth classification and the nth mode. Each column vector represents the softmax value obtained after calculating a modal feature, and each row vector represents the softmax value of a classification after training with different modal features.

[0015] The format of the matrix F is The format of the final softmax result matrix Fr is

[0016] The preferred solution of the multimodal data classification method based on deep learning: the calculation formula of the classification precision rate in the matrix S is: Among them, TP is the number of predicted positive instances, and FP is the number of predicted negative instances.

[0017] The preferred solution of the multimodal data classification method based on deep learning: the softmax processing refers to obtaining the position of the maximum value in the matrix Fr, and the code of the position is the classification result.

[0018] The present invention also provides a multimodal data classification system, comprising a data acquisition module, a storage module, and a processing module. The data acquisition module and the storage module are respectively connected to the processing module. The data acquisition module receives or collects N sets of multimodal data information and sends the multimodal data information to the processing module. The storage module is used to store at least one executable instruction, which causes the processing module to perform an operation corresponding to the above-mentioned deep learning-based multimodal data classification method on the multimodal data information. This multimodal data classification system has all the advantages of the above-mentioned deep learning-based multimodal data classification method.

[0019] The beneficial effects of the present invention are: the present invention addresses the problem of insufficient accuracy in multimodal deep learning, utilizes the precision rates of validation sets of networks with different modalities, assigns weights to different classification objects, and finally adds the softmax layers obtained from different modal features with weights to improve the classification accuracy of the neural network. The present invention can significantly improve the classification capabilities of the network and can be used in different fields.

[0020] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 It is a schematic diagram of a flow chart of the present invention;

[0023] Figure 2 Flowchart for multimodal network training;

[0024] Figure 3 Flowchart for obtaining the validation set result matrix S;

[0025] Figure 4 Flowchart for obtaining the softmax matrix R of the test set. DETAILED DESCRIPTION

[0026] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0027] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0028] like Figure 1 As shown, the present invention provides an embodiment of a multimodal data classification method based on deep learning, which specifically includes the following steps:

[0029] Obtain N groups of data, where each group of data includes data of multiple modalities. Taking human motion information as an example, these N groups of data are human motion information data, and each group of data includes multiple modal features. Different modalities also refer to different motion information. For example, the acceleration and angular velocity collected by the IMU module; the human surface electromyography collected by the sEMG module; and the plantar pressure collected by the plantar pressure sensor are all different modal information.

[0030] The N sets of data are divided into training sets, validation sets, and test sets. The classification methods for the training and validation sets can be diverse, including fixed assignment and various cross-validation methods. Different modes of determining data features can be divided based on data type, initial data processing method, data structure, and other aspects.

[0031] Construct a deep learning network model that contains multiple neural networks, and train the data of the training set in the deep learning network model. The neural network can be of different types and specifications, such as convolutional neural networks, recurrent neural networks, etc., which are determined by the characteristics of the data. During training, the features of different modalities are trained using the neural networks corresponding to the modalities, and the number of neural networks is the same as the number of modalities. Figure 2 As shown, the process of using the training set for training in this embodiment is to first separate the training set according to the modality, and then independently use the corresponding neural network for training, and finally different neural networks obtain corresponding weights.

[0032] After each training is completed, that is, after the features of each modality are trained independently in turn, the trained deep learning network model is verified using the validation set.

[0033] After all the training is completed, the validation set result matrix S after the last training is recorded. Each element in the matrix S is the classification precision rate of each modal feature of each classification.

[0034] The format of the matrix S is Among them, P mnis the classification precision of the feature of the mth classification and the nth mode. The calculation formula of the classification precision is: Among them, TP is the number of predicted positive instances, and FP is the number of predicted negative instances.

[0035] like Figure 3 As shown, the acquisition of the matrix S depends on the verification results of each modality. Here, the verification result is the value of the softmax layer in the format of 1×m, where m is the total number of categories.

[0036] The test set is used to test the deep learning network model after training. During the test, the features of different modes are calculated using the trained neural network corresponding to the mode. A softmax matrix R is obtained for each set of test data. Each element in the softmax matrix R is the softmax value of each modal feature of each classification.

[0037] The format of the softmax matrix R is Among them, R mn is the softmax value of the mth classification and nth mode, such as Figure 4 As shown in Figure 2, each column vector in the softmax matrix R represents the softmax value obtained after calculating a modal feature, and each row vector represents the softmax value of a classification after training with different modal features.

[0038] The softmax matrix R corresponding to each set of test data is weighted and integrated, and the elements in the matrix S are used as weights to multiply the elements in the softmax matrix R corresponding to each set of test data according to the corresponding classification and modal features to obtain the matrix F.

[0039] The format of the matrix F is

[0040] The probabilities corresponding to different modes of the same classification in the matrix F corresponding to each set of test data are added together. In this embodiment, the row vectors are added together to obtain the final softmax result matrix Fr corresponding to each set of test data. The format of the final softmax result matrix Fr is

[0041] The matrix Fr corresponding to each set of test data is softmax processed to obtain the final classification result of each set of test data. Specifically, the softmax processing mentioned here refers to obtaining the position of the maximum value in the matrix Fr, and the code of the position is the classification result.

[0042] The present application also proposes an embodiment of a multimodal data classification system. The multimodal data classification system in this embodiment includes a data acquisition module, a storage module and a processing module. The data acquisition module and the storage module are respectively connected to the processing module. The data acquisition module receives or collects N groups of multimodal data information and sends the multimodal data information to the processing module. The storage module is used to store at least one executable instruction, and the executable instruction enables the processing module to perform operations corresponding to the above-mentioned deep learning-based multimodal data classification method on the multimodal data information.

[0043] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0044] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A multimodal data classification method based on deep learning, characterized in that: The following steps are involved: Acquire N sets of human motion information data, each set of data including at least two modal features of human motion acceleration and / or angular velocity, human surface electromyography, and plantar pressure, and divide the N sets of data into a training set, a validation set, and a test set according to the set; Constructing a deep learning network model, which includes multiple neural networks, and training the data of the training set in the deep learning network model. During training, the features of different modalities are trained using the neural networks corresponding to the modalities; Use the validation set to validate the trained deep learning network model, and record the validation result after the last training as matrix S. Each element in the matrix S is the classification precision of each modal feature of each classification; The deep learning network model after training is tested using the test set. During the test, the features of different modalities are calculated using the trained neural network corresponding to the modality. A softmax matrix R is obtained for each set of test data. Each element in the softmax matrix R is the softmax value of each modal feature of each classification. The softmax matrix R corresponding to each set of test data is weighted and integrated, and the elements in the matrix S are used as weights to multiply the elements in the softmax matrix R corresponding to each set of test data according to the corresponding classification and modal features to obtain the matrix F of each set of test data; Add the probabilities corresponding to different modes of the same classification in the matrix F corresponding to each set of test data to obtain the final softmax result matrix Fr of each set of test data; Perform softmax processing on the final softmax result matrix Fr corresponding to each set of test data to obtain the final classification result of each set of test data.

2. The multimodal data classification method based on deep learning according to claim 1, characterized in that: The format of the matrix S is Among them, P mn is the classification precision of the features of the mth classification and the nth mode; The format of the softmax matrix R is Among them, R mn It is the softmax value of the mth classification and the nth mode. Each column vector represents the softmax value obtained after calculating a modal feature, and each row vector represents the softmax value of a classification after training with different modal features. The format of the matrix F is The format of the final softmax result matrix Fr is 3. The multimodal data classification method based on deep learning according to claim 1 or 2, characterized in that: The calculation formula of the classification precision in the matrix S is: Among them, TP is the number of predicted positive instances, and FP is the number of predicted negative instances.

4. The multimodal data classification method based on deep learning according to claim 1, characterized in that: The softmax processing is to obtain the position of the maximum value in the matrix Fr, and the code of the position is the classification result.

5. A multimodal data classification system, characterized in that: The system comprises a data acquisition module, a storage module and a processing module, wherein the data acquisition module and the storage module are respectively connected to the processing module, the data acquisition module receives or collects N groups of multimodal data information and sends the multimodal data information to the processing module, and the storage module is used to store at least one executable instruction, and the executable instruction enables the processing module to perform an operation corresponding to the multimodal data classification method based on deep learning as described in any one of claims 1 to 4 on the multimodal data information.

Images