Multi-modal Feature Fusion Method, Apparatus, Electronic Device and Readable Storage Medium

Abstract

The present application provides a multimodal feature fusion method, device, electronic device and readable storage medium, which obtain multiple heterogeneous data of a target object; for each heterogeneous data, extract the unimodal feature matrix of the heterogeneous data; for each unimodal feature matrix, based on the unimodal feature matrix and the multiple unimodal weight matrices corresponding to it, determine the fusion feature matrix of the unimodal feature matrix; for each fusion feature matrix, determine the fusion weight matrix between the fusion feature matrix and each fusion feature matrix; normalize each fusion weight matrix to obtain a normalized fusion weight matrix; based on each fusion feature matrix and the multiple normalized fusion weight matrices corresponding to each fusion feature matrix, determine the multimodal feature matrix describing the target object. In this way, the different modal features carried by the heterogeneous data can be fused to express the characteristics of the target object in a more detailed manner.

Description

Technical Field

[0001] The present application relates to the field of computer processing technology, and in particular to a multimodal feature fusion method, device, electronic device and readable storage medium. Background Art

[0002] Modality refers to the way a target object occurs or exists. Unimodality refers to a target object existing in only one mode, while multimodality refers to the ability of the same target object to occur or exist in two or more modes, or a combination of these modes. Data or information from the same data source is referred to as a modality. Common unimodal features include video, text, images, and audio.

[0003] The reason for fusing unimodal features from different data sources into multimodal features is that different modal features represent different target objects. In other words, different modal features view the same target object from different perspectives. Different modal features contain complementary information. If these features can be combined, the characteristics of the object itself can be more detailed. Therefore, how to fuse these modal features has become a pressing issue. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a multimodal feature fusion method, device, electronic device and readable storage medium, which can fuse the different modal features carried by heterogeneous data and express the characteristics of the target object in a more detailed manner.

[0005] The present invention provides a multimodal feature fusion method, which includes:

[0006] Acquire multiple heterogeneous data of the target object;

[0007] For each heterogeneous data, extract the single modal feature matrix of the heterogeneous data;

[0008] For each unimodal feature matrix, determining a unimodal weight matrix between the unimodal feature matrix and each unimodal feature matrix;

[0009] Determining a fusion feature matrix of the unimodal feature matrix based on the unimodal feature matrix and a plurality of unimodal weight matrices corresponding to the unimodal feature matrix;

[0010] For each fusion feature matrix, determine the fusion weight matrix between the fusion feature matrix and each fusion feature matrix;

[0011] For each fusion weight matrix corresponding to the fusion feature matrix, the fusion weight matrix is normalized to obtain a normalized fusion weight matrix;

[0012] Based on each fusion feature matrix and a plurality of normalized fusion weight matrices corresponding to each fusion feature matrix, a multimodal feature matrix for describing the target object is determined.

[0013] Furthermore, when the heterogeneous data includes audio data, extracting a unimodal feature matrix of the heterogeneous data includes:

[0014] Converting the audio data into mono audio data, and resampling the mono audio data to obtain resampled audio data;

[0015] Moving a Hanning time window of a preset window length on the resampled audio data, performing Fourier transform on the resampled audio data, and obtaining an audio spectrum of the audio data;

[0016] Mapping the audio spectrum into an initial Mel-cepstrum using a filter bank;

[0017] Performing logarithmic calculation on the initial Mel-cepstrum to obtain a stable Mel-cepstrum;

[0018] The stable Mel-frequency cepstrum is reorganized according to a preset time length to obtain a unimodal feature matrix of the audio data.

[0019] Furthermore, when the heterogeneous data includes text data, extracting a unimodal feature matrix of the heterogeneous data includes:

[0020] performing natural language preprocessing on the text data, extracting from the text data a plurality of text feature groups and an image feature group corresponding to each text feature group; wherein the text feature group includes at least two of Chinese character features, phrase features, and sentence features; and the text feature group is expressed in a one-hot vector format;

[0021] Determine a text feature matrix for each text feature group and a picture feature matrix for the picture feature group corresponding to each text feature group;

[0022] For each text feature group, fuse the text feature matrix of the text feature group with the image feature matrix of the image feature group corresponding to the text feature group to obtain a preliminary fusion matrix of the text feature group;

[0023] For each text feature group, based on the preliminary fusion matrix of the text feature group and the picture feature matrix of the picture feature group corresponding to the text feature group, determine a multidimensional correlation matrix between each sub-text feature in the text feature group and the sub-picture feature corresponding to the sub-text feature;

[0024] Determining a multidimensional attention weight of each sub-text feature based on a multidimensional correlation matrix of each sub-text feature in the text feature group;

[0025] Based on the multi-dimensional attention weight of each sub-text feature in each text feature group and the image feature matrix of the sub-image feature corresponding to each sub-text feature, a unimodal feature matrix of the text data is determined.

[0026] Furthermore, determining the text feature matrix of each text feature group and the picture feature matrix of the picture feature group corresponding to each text feature group includes:

[0027] For each text feature group, using a pre-trained feature extraction model corresponding to the text feature group, determine a text feature matrix that can characterize each sub-text feature in the text feature group;

[0028] The pre-trained convolutional neural network is used to determine the image feature matrix of the image feature group corresponding to the text feature group.

[0029] Furthermore, the fusing of the text feature matrix of the text feature group and the image feature matrix of the image feature group corresponding to the text feature group to obtain a preliminary fusion matrix of the text feature group includes:

[0030] For each sub-text feature in the text feature group, a sub-text matrix of the sub-text feature is fused with a sub-image matrix of a sub-image feature corresponding to the sub-text feature to obtain a sub-fusion matrix of each sub-text feature;

[0031] Based on the sub-fusion matrix of each sub-text feature, a preliminary fusion matrix of the text feature group is determined.

[0032] Furthermore, the determining of a multidimensional correlation matrix between each sub-text feature in the text feature group and the sub-picture feature corresponding to the sub-text feature based on the preliminary fusion matrix of the text feature group and the picture feature matrix of the picture feature group corresponding to the text feature group includes:

[0033] For each sub-text feature in the text feature group, a multidimensional correlation matrix of the sub-text feature is determined based on the sub-fusion matrix of the sub-text feature and the sub-image matrix of the sub-image feature corresponding to the sub-text feature.

[0034] Furthermore, the feature extraction model corresponding to the text feature group is obtained through the following steps:

[0035] Obtaining a pre-trained language pre-training model; the language pre-training model is used to extract a text feature matrix representing each sub-text feature of the text feature group from the one-hot vector of the text feature group;

[0036] Performing model distillation processing on the language pre-training model to compress parameters in the language pre-training model to obtain the feature extraction model.

[0037] Furthermore, when the heterogeneous data includes video data, the multimodal feature fusion method further includes:

[0038] dividing the video data into multiple frames of image data;

[0039] When the heterogeneous data includes image data, extracting a single-modal feature matrix of the heterogeneous data includes:

[0040] A deep neural network with a residual jump mechanism is used to extract the unimodal feature matrix of the image data.

[0041] The embodiment of the present application further provides a multimodal feature fusion device, the multimodal feature fusion device comprising:

[0042] A data acquisition module is used to acquire multiple heterogeneous data of the target object;

[0043] A matrix extraction module is used to extract the single-modal feature matrix of each heterogeneous data source;

[0044] A unimodal weight determination module is used to determine, for each unimodal feature matrix, a unimodal weight matrix between the unimodal feature matrix and each unimodal feature matrix;

[0045] a fusion matrix determination module, configured to determine a fusion feature matrix of the unimodal feature matrix based on the unimodal feature matrix and a plurality of unimodal weight matrices corresponding to the unimodal feature matrix;

[0046] A fusion weight determination module is used to determine, for each fusion feature matrix, a fusion weight matrix between the fusion feature matrix and each fusion feature matrix;

[0047] A matrix normalization module is used to normalize each fusion weight matrix corresponding to the fusion feature matrix to obtain a normalized fusion weight matrix;

[0048] The multimodal feature determination module is used to determine a multimodal feature matrix for describing the target object based on each fusion feature matrix and multiple normalized fusion weight matrices corresponding to each fusion feature matrix.

[0049] Furthermore, when the heterogeneous data includes audio data, the matrix extraction module, when used to extract a single-modal feature matrix of the heterogeneous data, is configured to:

[0050] Converting the audio data into mono audio data, and resampling the mono audio data to obtain resampled audio data;

[0051] Moving a Hanning time window of a preset window length on the resampled audio data, performing Fourier transform on the resampled audio data, and obtaining an audio spectrum of the audio data;

[0052] Mapping the audio spectrum into an initial Mel-cepstrum using a filter bank;

[0053] Performing logarithmic calculation on the initial Mel-cepstrum to obtain a stable Mel-cepstrum;

[0054] The stable Mel-frequency cepstrum is reorganized according to a preset time length to obtain a unimodal feature matrix of the audio data.

[0055] Furthermore, when the heterogeneous data includes text data, the matrix extraction module, when used to extract the unimodal feature matrix of the heterogeneous data, is configured to:

[0056] performing natural language preprocessing on the text data, extracting from the text data a plurality of text feature groups and an image feature group corresponding to each text feature group; wherein the text feature group includes at least two of Chinese character features, phrase features, and sentence features; and the text feature group is expressed in a one-hot vector format;

[0057] Determine a text feature matrix for each text feature group and a picture feature matrix for the picture feature group corresponding to each text feature group;

[0058] For each text feature group, fuse the text feature matrix of the text feature group with the image feature matrix of the image feature group corresponding to the text feature group to obtain a preliminary fusion matrix of the text feature group;

[0059] For each text feature group, based on the preliminary fusion matrix of the text feature group and the picture feature matrix of the picture feature group corresponding to the text feature group, determine a multidimensional correlation matrix between each sub-text feature in the text feature group and the sub-picture feature corresponding to the sub-text feature;

[0060] Determining a multidimensional attention weight of each sub-text feature based on a multidimensional correlation matrix of each sub-text feature in the text feature group;

[0061] Based on the multi-dimensional attention weight of each sub-text feature in each text feature group and the image feature matrix of the sub-image feature corresponding to each sub-text feature, a unimodal feature matrix of the text data is determined.

[0062] Furthermore, when the matrix extraction module is used to determine the text feature matrix of each text feature group and the image feature matrix of the image feature group corresponding to each text feature group, the matrix extraction module is used to:

[0063] For each text feature group, using a pre-trained feature extraction model corresponding to the text feature group, determine a text feature matrix that can characterize each sub-text feature in the text feature group;

[0064] The pre-trained convolutional neural network is used to determine the image feature matrix of the image feature group corresponding to the text feature group.

[0065] Furthermore, when the matrix extraction module is used to fuse the text feature matrix of the text feature group and the image feature matrix of the image feature group corresponding to the text feature group to obtain a preliminary fusion matrix of the text feature group, the matrix extraction module is used to:

[0066] For each sub-text feature in the text feature group, a sub-text matrix of the sub-text feature is fused with a sub-image matrix of a sub-image feature corresponding to the sub-text feature to obtain a sub-fusion matrix of each sub-text feature;

[0067] Based on the sub-fusion matrix of each sub-text feature, a preliminary fusion matrix of the text feature group is determined.

[0068] Furthermore, when the matrix extraction module is used to determine a multidimensional correlation matrix between each sub-text feature in the text feature group and the sub-picture feature corresponding to the sub-text feature based on the preliminary fusion matrix of the text feature group and the picture feature matrix of the picture feature group corresponding to the text feature group, the matrix extraction module is used to:

[0069] For each sub-text feature in the text feature group, a multidimensional correlation matrix of the sub-text feature is determined based on the sub-fusion matrix of the sub-text feature and the sub-image matrix of the sub-image feature corresponding to the sub-text feature.

[0070] Furthermore, the matrix extraction module is used to obtain a feature extraction model corresponding to the text feature group through the following steps:

[0071] Obtaining a pre-trained language pre-training model; the language pre-training model is used to extract a text feature matrix representing each sub-text feature of the text feature group from the one-hot vector of the text feature group;

[0072] Performing model distillation processing on the language pre-training model to compress parameters in the language pre-training model to obtain the feature extraction model.

[0073] Furthermore, when the heterogeneous data includes video data, the multimodal feature fusion device further includes a video segmentation module, which is configured to:

[0074] dividing the video data into multiple frames of image data;

[0075] When the heterogeneous data includes image data, the matrix extraction module is used to extract a single-modal feature matrix of the heterogeneous data, and the matrix extraction module is used to:

[0076] A deep neural network with a residual jump mechanism is used to extract the unimodal feature matrix of the image data.

[0077] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the multimodal feature fusion method as described above are performed.

[0078] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the multimodal feature fusion method described above are executed.

[0079] The multimodal feature fusion method, device, electronic device and readable storage medium provided in the embodiment of the present application obtain multiple heterogeneous data of the target object; for each heterogeneous data, extract the unimodal feature matrix of the heterogeneous data; for each unimodal feature matrix, respectively determine the unimodal weight matrix between the unimodal feature matrix and each unimodal feature matrix; based on the unimodal feature matrix and the multiple unimodal weight matrices corresponding to the unimodal feature matrix, determine the fusion feature matrix of the unimodal feature matrix; for each fusion feature matrix, respectively determine the fusion weight matrix between the fusion feature matrix and each fusion feature matrix; for each fusion weight matrix corresponding to the fusion feature matrix, normalize the fusion weight matrix to obtain a normalized fusion weight matrix; based on each fusion feature matrix and the multiple normalized fusion weight matrices corresponding to each fusion feature matrix, determine the multimodal feature matrix used to describe the target object. In this way, the different modal features carried by the heterogeneous data can be fused to express the characteristics of the target object in a more detailed manner.

[0080] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0082] Figure 1 A flowchart of a multimodal feature fusion method provided in an embodiment of the present application;

[0083] Figure 2 A schematic diagram of a single-modal feature matrix extraction process for audio data provided in an embodiment of the present application;

[0084] Figure 3 This is one of the structural diagrams of a multimodal feature fusion device provided in an embodiment of the present application;

[0085] Figure 4 This is a second structural diagram of a multimodal feature fusion device provided in an embodiment of the present application;

[0086] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0088] Research has found that different modal features represent different target objects. In other words, different modal features have different perspectives on the same target object. In essence, there is some cross-information between different modal features (so there will be information redundancy). However, in addition to the cross-information, what is more important is that there is complementary information between different modal features. If different modal features can be integrated together, the characteristics of the object itself can be represented more meticulously.

[0089] At present, there are three main methods of multimodal feature fusion: 1. Early-fusion, i.e., data-level fusion; 2. Late-fusion, i.e., decision-level fusion; 3. Intermediate-fusion.

[0090] Front-end fusion combines multiple independent datasets into a single feature vector, which is then fed into a machine learning classifier. This is because front-end fusion of multimodal data often fails to fully exploit the complementarity between multiple modal data, and the raw data from front-end fusion often contains a large amount of redundant information.

[0091] Back-end fusion combines the scores (decisions) output by classifiers trained on data from different modalities. The advantage of this approach is that errors in the fusion model originate from different classifiers, and errors from different classifiers are often independent of each other, preventing further accumulation of errors. Common back-end fusion methods include max-fusion, averaged-fusion, Bayesian rule-based fusion, and ensemble learning. Ensemble learning, as a typical back-end fusion method, is widely used in research fields such as communications, computer recognition, and speech recognition.

[0092] Intermediate fusion involves converting different modal data into high-dimensional feature representations before fusing them at the model's intermediate layers. Taking neural networks as an example, intermediate fusion first uses the neural network to convert the raw data into high-dimensional feature representations, then extracts the commonalities between the different modal data in that high-dimensional space.

[0093] Based on this, an embodiment of the present application provides a multimodal feature fusion method that can fuse single-modal features of heterogeneous data from different data sources, and can express the characteristics of the target object in a more detailed manner.

[0094] See also Figure 1 , Figure 1 This is a flow chart of a multimodal feature fusion method provided in an embodiment of the present application. Figure 1 As shown in , the multimodal feature fusion method provided in the embodiment of the present application includes:

[0095] S101: Acquire multiple heterogeneous data of a target object.

[0096] S102: For each heterogeneous data, extract a unimodal feature matrix of the heterogeneous data.

[0097] S103 : For each unimodal feature matrix, determine a unimodal weight matrix between the unimodal feature matrix and each unimodal feature matrix.

[0098] Among them, taking the extraction of N unimodal feature matrices from multiple heterogeneous data of the target object as an example, each unimodal feature matrix corresponds to N unimodal weight matrices; here, including the unimodal weight matrix between each unimodal feature matrix and itself.

[0099] S104: Determine a fusion feature matrix of the unimodal feature matrix based on the unimodal feature matrix and a plurality of unimodal weight matrices corresponding to the unimodal feature matrix.

[0100] Here, each single-modal feature matrix can be calculated to obtain a corresponding fusion feature matrix.

[0101] S105 : For each fused feature matrix, determine a fusion weight matrix between the fused feature matrix and each fused feature matrix.

[0102] Here, each single-modal feature matrix corresponds to a fusion feature matrix, and each fusion feature matrix can be calculated to obtain the corresponding N fusion weight matrices; here, including the fusion weight matrix between each fusion feature matrix and itself.

[0103] S106 : For each fusion weight matrix corresponding to the fusion feature matrix, normalize the fusion weight matrix to obtain a normalized fusion weight matrix.

[0104] Here, each single-modal feature matrix of heterogeneous data needs to execute steps S103 to S106 to realize the modal feature fusion between each heterogeneous data and other heterogeneous data in multiple heterogeneous data except itself, and obtain the fusion feature matrix corresponding to each heterogeneous data and multiple normalized fusion weight matrices corresponding to the fusion feature matrix.

[0105] S107 : Determine a multimodal feature matrix for describing the target object based on each fused feature matrix and a plurality of fusion weight matrices corresponding to each fused feature matrix.

[0106] Here, heterogeneous data refers to data with different data structures coming from different data sources. For the same target object, multiple heterogeneous data related to the target object can usually be obtained. For example, target object A attended a meeting and gave a speech at the meeting. For target object A, video data, picture data, audio data and text data about target object A can be obtained at the meeting. For video data and audio data, the acquisition devices of the two are different, so the two come from different data sources. Therefore, the two are heterogeneous data. Similarly, video data, picture data, audio data and text data are all heterogeneous data, that is, the heterogeneous data of the target object include one or more of video data, picture data, audio data and text data.

[0107] In step S102, for each heterogeneous data, if a subsequent feature fusion operation is to be performed on it, first, it is necessary to extract the unimodal feature matrix of the heterogeneous data from the heterogeneous data; and for the heterogeneous data obtained from different data sources, the extraction method of the unimodal feature matrix of the heterogeneous data is naturally different.

[0108] For further information, see Figure 2 , Figure 2 This is a flow chart of extracting a single-modal feature matrix of audio data provided by an embodiment of the present application. Figure 2 As shown, step S102 includes:

[0109] Step S201: convert the audio data into mono audio data, and resample the mono audio data to obtain resampled audio data.

[0110] In this step, the acquired audio data is converted into mono audio data, and the converted mono audio data is resampled to obtain audio data of a preset frequency, thereby obtaining resampled audio data.

[0111] Here, the preset frequency may be 16 kHz.

[0112] Step S202: Moving a Hanning window of a preset window length on the resampled audio data, performing Fourier transform on the resampled audio data, and obtaining an audio spectrum of the audio data.

[0113] In this step, a Hanning window (Hann window) for Fourier transforming the audio data is obtained, and the Hanning window of a preset window length is moved on the resampled audio data according to a preset window shift. The resampled audio data each time within the Hanning window is short-time Fourier transformed until all the resampled audio data have completed the short-time Fourier transform, thereby obtaining the audio spectrum of the audio data.

[0114] Here, the preset window length may be 25 ms, and the preset window shift may be 10 ms.

[0115] Step S203: Map the audio spectrum into an initial Mel-cepstrum using a filter bank.

[0116] In this step, the obtained audio spectrum is mapped to a 64-order mel filter bank, and the initial mel cepstrum of the audio data is calculated through the filter.

[0117] Step S204 : performing logarithmic calculation on the initial Mel-cepstrum to obtain a stable Mel-cepstrum.

[0118] In this step, in order to obtain a stable Mel-cepstrum, the calculated initial Mel-cepstrum needs to be logarithmically calculated. In order to avoid performing logarithmic calculation on audio data with a spectrum value of 0, a bias constant is used during the calculation process to ensure that the audio data does not have a spectrum value of 0, thereby avoiding taking the logarithm of 0. Specifically, the stable Mel-cepstrum is calculated using the following formula:

[0119] mels=log(mel-spectrum+0.01);

[0120] Among them, mel s To stabilize the Mel-spectrum, log is the logarithm calculation, mel-spectrum is the initial Mel-spectrum, and a is the bias constant. Here, the value of a is usually 0.01.

[0121] Step S205 : reorganizing the stable Mel-frequency cepstrum according to a preset time length to obtain a unimodal feature matrix of the audio data.

[0122] In this step, the calculated stable Mel-frequency cepstrum will be reframed with a duration of 0.96 seconds to extract feature data that can represent the audio data. There are no overlapping frames in the reorganized stable Mel-frequency cepstrum. Each frame of audio data (feature data) contains 64 Mel frequency bands with a duration of 10 milliseconds (i.e., a total of 96 frames of feature data are obtained after reorganization). A unimodal feature matrix of the audio data is constructed based on each frame of audio data (feature).

[0123] It should be noted that the output data format of the unimodal feature matrix corresponding to the audio data in this application is [nums_frames, 128], that is, the output data is a 128-dimensional high-level feature vector with semantics and meaning; where nums_frames is the frame length of each frame of feature data,

[0124] Here, 0.96 means that when extracting the unimodal feature matrix of the audio data, it will be reframed with a duration of 0.96s.

[0125] In another embodiment, when the heterogeneous data includes text data, step S102 includes:

[0126] Step 1: Perform natural language preprocessing on the text data, and extract multiple text feature groups and image feature groups corresponding to each text feature group from the text data; wherein the text feature groups include at least two of Chinese character features, phrase features, and sentence features; and the text feature groups are expressed in the form of one-hot vectors.

[0127] In this step, the acquired text data is subjected to natural language preprocessing, and a plurality of text feature groups are extracted from the text data respectively. The extraction can be based on the form of features, for example, Chinese characters, phrases, and sentences; in this way, the extracted text feature groups include at least two of Chinese character features, phrase features, and sentence features.

[0128] Here, the text data can be split into multiple sub - character features one by one for each Chinese character. For example, for the text "咬定青山不放松", it can be split into "咬", "定", "青", "山", "不", "放", and "松". In this way, "咬" can be used as a sub - character feature in a Chinese - character feature group, and so on. The sub - character features in the Chinese - character feature group include: "咬", "定", "青", "山", "不", "放", and "松"; and the multiple "Chinese characters" split out are respectively converted into the corresponding one - hot vector form; for example, "咬" [0, 1, 0, 0]; "定": [0, 1, 1, 0].

[0129] The text data can also be split into multiple sub - character features one by one for each phrase. For example, for the text "咬定青山不放松", it can be split into "咬定", "青山", "不", and "放松". In this way, "咬定" can be used as a sub - character feature in a phrase - feature group, and so on. The sub - character features in the phrase - feature group include: "咬定", "青山", "不", and "放松"; and the multiple "phrases" split out are respectively converted into the corresponding one - hot vector form; for example, "咬定" [1, 1, 0, 0]; "青山": [0, 1, 1, 1].

[0130] The text data can also be split into multiple sub - character features one by one for each sentence. For example, the whole sentence "咬定青山不放松" can be used as a sub - character feature in a sentence - feature group; and the "sentence" split out is converted into the corresponding one - hot vector form.

[0131] The picture feature group is the traditional Chinese characters and / or pictographic - character pictures corresponding to each sub - character feature in its corresponding text - feature group; corresponding to the above - mentioned embodiments, the picture feature group corresponding to the Chinese - character feature group includes the traditional Chinese characters and / or pictographic - character pictures corresponding to "咬", "定", etc.; the picture feature group corresponding to the phrase - feature group includes the traditional Chinese - character group and / or pictographic - character group pictures corresponding to "咬定"; the picture feature group corresponding to the sentence - feature group includes the traditional Chinese characters and / or pictographic - character pictures of the whole sentence corresponding to "咬定青山不放松".

[0132] When extracting the text - feature group, it is also necessary to determine the picture feature group corresponding to each text - feature group. Here, the text - feature group includes multiple sub - character features; correspondingly, the picture feature group includes the sub - picture features corresponding to each sub - character feature.

[0133] Specifically, when the text - feature group is a Chinese - character feature group, the picture feature group corresponding to the Chinese - character feature group includes the traditional Chinese characters and / or pictographic - character pictures corresponding to each sub - character feature (each Chinese character) in the Chinese - character feature group;

[0134] When the text feature group is a phrase feature group, the picture feature group corresponding to the phrase feature group includes pictures of traditional Chinese characters and / or pictographic Chinese characters corresponding to each sub-text feature (each phrase) in the phrase feature group;

[0135] When the text feature group is a sentence feature group, the picture feature group corresponding to the sentence feature group includes sentences composed of traditional Chinese characters and / or pictures of sentences composed of pictographic Chinese characters corresponding to each sub-text feature (each sentence) in the sentence feature group.

[0136] Step 2: Determine the text feature matrix of each text feature group and the image feature matrix of the image feature group corresponding to each text feature group.

[0137] In this step, in order to perform feature fusion in the subsequent process, it is necessary to unify the expression form of each text feature group and convert it into computer-recognizable data. The text feature matrix of each text feature group is determined separately, that is, the sub-text features in each text feature group are presented in the form of a matrix.

[0138] Specifically, when the text feature group is a Chinese character feature group, the sub-text feature is a Chinese character. At this time, the Chinese character must be converted into a matrix form, that is, the Chinese character must be represented in the form of a matrix; when the text feature group is a phrase feature group, the sub-text feature is a phrase. At this time, the phrase must be converted into a matrix form, that is, the phrase must be represented in the form of a matrix; when the text feature group is a sentence feature group, the sub-text feature is a sentence. At this time, the sentence must be converted into a matrix form, that is, the sentence must be represented in the form of a matrix.

[0139] In one embodiment, step 2 includes: for each text feature group, using a pre-trained feature extraction model corresponding to the text feature group, determining a text feature matrix that can characterize each sub-text feature in the text feature; using a pre-trained convolutional neural network, determining an image feature matrix of the image feature group corresponding to the text feature group.

[0140] In this step, the method of determining the text feature matrix of the text feature group is different from the method of determining the image feature matrix of the image feature group; for the text feature group, the pre-trained feature extraction model corresponding to the text feature group is used to determine the text feature matrix that can represent each sub-text feature (Chinese character, phrase, sentence) in the text feature group.

[0141] For the image feature group, a pre-trained convolutional neural network is used to determine the image feature matrix of the image feature group corresponding to the text feature group.

[0142] In one embodiment, the feature extraction model corresponding to the text feature group is obtained by the following steps:

[0143] First, a pre-trained language pre-training model is obtained; the language pre-training model is used to extract a text feature matrix representing each sub-text feature of the text feature group from the one-hot vector of the text feature group.

[0144] Here, the language pre-training model is trained through training data. However, since the language pre-training model (for example, the Bert model) has more parameters and a larger model size, it cannot be used on a computer with average performance, that is, the practicality of the language pre-training model is poor. At this time, it is necessary to compress the size of the language pre-training model while maintaining the accuracy of the original language pre-training model.

[0145] Then, the language pre-training model is subjected to model distillation processing to compress the parameters in the language pre-training model to obtain the feature extraction model.

[0146] Here, the feature extraction model is obtained by compressing the language pre-training model. While maintaining the good generalization performance of the language pre-training model (for example, the Bert model), the feature extraction model can also overcome the length limitation of the language pre-training model on sub-text features.

[0147] Step 3: For each text feature group, fuse the text feature matrix of the text feature group and the image feature matrix of the image feature group corresponding to the text feature group to obtain a preliminary fusion matrix of the text feature group.

[0148] In one embodiment, step 3 includes: for each sub-text feature in the text feature group, fusing the sub-text matrix of the sub-text feature with the sub-image matrix of the sub-image feature corresponding to the sub-text feature to obtain a sub-fusion matrix of each sub-text feature; and determining a preliminary fusion matrix of the text feature group based on the sub-fusion matrix of each sub-text feature.

[0149] In this step, a sub-text matrix for each sub-text feature in the text feature group is determined, and a sub-picture matrix for the sub-picture feature corresponding to each sub-text feature is determined from the picture feature matrix of the picture feature group corresponding to the text feature group;

[0150] For each sub-text feature, fuse the sub-text matrix of the sub-text feature with the sub-image matrix of the corresponding sub-image to obtain a sub-fusion matrix of the sub-text feature;

[0151] For each text feature group, a preliminary fusion matrix of the text feature group is determined based on the sub-fusion matrices of each sub-text feature in the text feature group.

[0152] Specifically, the sub-fusion matrix of the sub-text features is determined by the following formula:

[0153]

[0154] Among them, u i w is the sub-fusion matrix of the i-th sub-text feature in the w-th text feature group, h i w is the sub-text matrix of the i-th sub-text feature in the w-th text feature group, v i w is the sub-image matrix of the sub-image feature corresponding to the i-th sub-text feature in the w-th text feature group.

[0155] Step 4: For each text feature group, based on the fusion feature matrix of the text feature group and the picture feature matrix of the picture feature group corresponding to the text feature group, determine the multidimensional correlation matrix between each sub-text feature in the text feature group and the sub-picture feature corresponding to the sub-text feature.

[0156] In one embodiment, step 4 includes: for each sub-text feature in the text feature group, based on the sub-fusion matrix of the sub-text feature and the sub-image matrix of the sub-image feature corresponding to the sub-text feature, determining the multidimensional correlation matrix of the sub-text feature.

[0157] In this step, for each sub-text feature in each text feature group, a multidimensional correlation matrix between the sub-text feature and the corresponding sub-image feature can be calculated based on the sub-fusion matrix of the sub-text feature and the image feature matrix of the sub-image feature corresponding to the sub-text feature. Specifically, the multidimensional correlation matrix is calculated using the following formula:

[0158]

[0159] Among them, e i w is the multidimensional correlation matrix of the i-th sub-text feature in the w-th text feature group, u i w is the sub-fusion matrix of the i-th sub-text feature in the w-th text feature group, v i w is the image feature matrix of the sub-image feature corresponding to the i-th sub-text feature in the w-th text feature group.

[0160] Step 5: Based on the multidimensional correlation matrix of each sub-text feature in the text feature group, determine the multidimensional attention weight of the sub-text feature.

[0161] In this step, the multidimensional attention weight of each sub-text feature is calculated using the following formula:

[0162]

[0163] Among them, a i w is the multidimensional attention weight of the i-th sub-text feature in the w-th text feature group, e i w is the multidimensional correlation matrix of the i-th sub-text feature in the w-th text feature group, and z is the number of sub-text features in the w-th text feature group.

[0164] Step 6: Determine the unimodal feature matrix of the text data based on the multidimensional attention weight of each sub-text feature in each text feature group and the image feature matrix of the sub-image feature corresponding to each sub-text feature.

[0165] In this step, the unimodal feature matrix of the text data is determined by the following formula:

[0166]

[0167] Among them, S is the unimodal feature matrix of text data, a i w is the multidimensional attention weight of the i-th sub-text feature in the w-th text feature group, v i w is the image feature matrix of the sub-image feature corresponding to the i-th sub-text feature in the w-th text feature group, z is the number of sub-text features in the w-th text feature group, and l is the number of text feature groups.

[0168] In another embodiment, when the heterogeneous data includes video data, the multimodal feature fusion further includes: dividing the video data into multiple frames of image data.

[0169] In this step, if the obtained heterogeneous data is video data, the video data is divided into multiple frames of image data in the form of processable image data for subsequent extraction of a single-modal feature matrix.

[0170] In another embodiment, when the received video data is divided into multiple frames of image data, or the received heterogeneous data is image data, step S102 includes: using a deep neural network with a residual jump mechanism to extract a unimodal feature matrix of the image data.

[0171] In this step, a deep neural network with a residual jump mechanism is pre-trained using training set data, and a single-modal feature matrix that can characterize the image data is extracted from each image data using the deep neural network with a residual jump mechanism.

[0172] Here, for video data and image data, this application introduces a deep neural network ResNet with a residual jump structure; through the residual network structure of this network, the problems of gradient vanishing, explosion and network degradation caused by the increase in the number of convolutional network layers can be avoided, thereby improving the accuracy of the single-modal feature extraction results.

[0173] For each extracted unimodal feature matrix, each two unimodal feature matrices can be fused separately in terms of features, so that the unimodal feature matrix becomes a multimodal feature matrix, thereby representing the target object more meticulously.

[0174] In step S103, the unimodal weight matrix between every two unimodal feature matrices is calculated using the following formula:

[0175]

[0176] Among them, Q xy is the unimodal weight matrix between the unimodal feature matrix of the xth heterogeneous data and the unimodal feature matrix of the yth heterogeneous data, S x mn is the single modal feature matrix of the xth heterogeneous data, is the transpose of the unimodal feature matrix of the yth heterogeneous data, and m and n are the rows and columns of the matrix respectively.

[0177] Among them, N single-modal feature matrices S are extracted from multiple heterogeneous data of the target object. x mn For example, each single-mode feature matrix S x mn There are N single-modal weight matrices Q corresponding to xy ; Here, it includes the unimodal weight matrix between each unimodal feature matrix and itself.

[0178] In step S104, the fusion feature matrix of each single-modal feature matrix is calculated using the following formula:

[0179]

[0180] in, is the fusion feature matrix of the xth heterogeneous data, S x mn is the single modal feature matrix of the xth heterogeneous data, is the transpose of the unimodal weight matrix between the unimodal feature matrix of the x-th heterogeneous data and the unimodal feature matrix of the y-th heterogeneous data, and m and n are the rows and columns of the matrix respectively.

[0181] Here, the single-modal feature matrix S of each heterogeneous data is x mn The above formula can be used to calculate the corresponding fusion feature matrix

[0182] In step S105, the fusion weight matrix between each fusion feature matrix and each fusion feature matrix is calculated by the following formula:

[0183]

[0184] Among them, P xy is the fusion weight matrix between the single modal feature matrix of the xth heterogeneous data and the single modal feature matrix of the yth heterogeneous data, is the fusion feature matrix of the xth heterogeneous data, is the transpose of the fusion feature matrix of the yth heterogeneous data, m and n are the rows and columns of the matrix respectively.

[0185] Here, each unimodal feature matrix S x mn Corresponding to a fusion feature matrix And each fusion feature matrix The corresponding N fusion weight matrices P can be calculated by the above formula xy ; Here, it includes the fusion weight matrix between each fusion feature matrix and itself.

[0186] In step S106, the normalized fusion weight matrix is calculated using the following formula:

[0187] P xy' =tanh(P xy );

[0188] Among them, P xy' is the fusion weight matrix between the normalized unimodal feature matrix of the xth heterogeneous data and the unimodal feature matrix of the yth heterogeneous data, P xy is the fusion weight matrix between the unimodal feature matrix of the xth heterogeneous data and the unimodal feature matrix of the yth heterogeneous data.

[0189] Here, each fusion weight matrix P xy The corresponding normalized fusion weight matrix P can be calculated by the above formula xy' .

[0190] Here, for each single-modal feature matrix of heterogeneous data, steps S103 to S106 need to be executed to achieve modal feature fusion between each heterogeneous data and other heterogeneous data in multiple heterogeneous data except itself, and obtain a fusion feature matrix corresponding to each heterogeneous data and multiple normalized fusion weight matrices corresponding to the fusion feature matrix.

[0191] In step S107, multiple unimodal features of the target object are fused using the following formula to determine a multimodal feature matrix for describing the target object:

[0192]

[0193] Among them, G mn is the multimodal feature matrix of the target object, P xy' is the fusion weight matrix between the normalized unimodal feature matrix of the xth heterogeneous data and the unimodal feature matrix of the yth heterogeneous data, is the fusion feature matrix of the xth heterogeneous data, m and n are the rows and columns of the matrix respectively.

[0194] The multimodal feature fusion method provided in the embodiment of the present application obtains multiple heterogeneous data of the target object; for each heterogeneous data, extracts the unimodal feature matrix of the heterogeneous data; for each unimodal feature matrix, determines the unimodal weight matrix between the unimodal feature matrix and each unimodal feature matrix; based on the unimodal feature matrix and the multiple unimodal weight matrices corresponding to the unimodal feature matrix, determines the fusion feature matrix of the unimodal feature matrix; for each fusion feature matrix, determines the fusion weight matrix between the fusion feature matrix and each fusion feature matrix; for each fusion weight matrix corresponding to the fusion feature matrix, normalizes the fusion weight matrix to obtain a normalized fusion weight matrix; based on each fusion feature matrix and the multiple normalized fusion weight matrices corresponding to each fusion feature matrix, determines the multimodal feature matrix for describing the target object. In this way, the different modal features carried by the heterogeneous data can be fused to express the characteristics of the target object in a more detailed manner.

[0195] See also Figure 3 、 Figure 4 , Figure 3 This is one of the structural diagrams of a multimodal feature fusion device provided in an embodiment of the present application. Figure 4 This is a second structural diagram of a multimodal feature fusion device provided in an embodiment of the present application. Figure 3 As shown in , the multimodal feature fusion device 300 includes:

[0196] The data acquisition module 310 is used to acquire multiple heterogeneous data of the target object;

[0197] The matrix extraction module 320 is used to extract the single-modal feature matrix of each heterogeneous data source;

[0198] A unimodal weight determination module 330 is configured to determine, for each unimodal feature matrix, a unimodal weight matrix between the unimodal feature matrix and each unimodal feature matrix;

[0199] A fusion matrix determination module 340 is configured to determine a fusion feature matrix of the unimodal feature matrix based on the unimodal feature matrix and a plurality of unimodal weight matrices corresponding to the unimodal feature matrix;

[0200] A fusion weight determination module 350 is used to determine, for each fusion feature matrix, a fusion weight matrix between the fusion feature matrix and each fusion feature matrix;

[0201] A matrix normalization module 360 is used to normalize each fusion weight matrix corresponding to the fusion feature matrix to obtain a normalized fusion weight matrix;

[0202] The multimodal feature determination module 370 is configured to determine a multimodal feature matrix for describing the target object based on each fused feature matrix and a plurality of normalized fusion weight matrices corresponding to each fused feature matrix.

[0203] Further, such as Figure 4 As shown, when the heterogeneous data includes video data, the multimodal feature fusion device 300 further includes a video segmentation module 380, and the video segmentation module 380 is used to:

[0204] dividing the video data into multiple frames of image data;

[0205] When the heterogeneous data includes image data, the matrix extraction module 320 is used to extract a single-modal feature matrix of the heterogeneous data, and the matrix extraction module 320 is used to:

[0206] A deep neural network with a residual jump mechanism is used to extract the unimodal feature matrix of the image data.

[0207] Furthermore, when the heterogeneous data includes audio data, the matrix extraction module 320 is used to extract a single-modal feature matrix of the heterogeneous data, and the matrix extraction module 320 is used to:

[0208] Converting the audio data into mono audio data, and resampling the mono audio data to obtain resampled audio data;

[0209] Moving a Hanning time window of a preset window length on the resampled audio data, performing Fourier transform on the resampled audio data, and obtaining an audio spectrum of the audio data;

[0210] Mapping the audio spectrum into an initial Mel-cepstrum using a filter bank;

[0211] Performing logarithmic calculation on the initial Mel-cepstrum to obtain a stable Mel-cepstrum;

[0212] The stable Mel-frequency cepstrum is reorganized according to a preset time length to obtain a unimodal feature matrix of the audio data.

[0213] Furthermore, when the heterogeneous data includes text data, the matrix extraction module 320 is used to extract the unimodal feature matrix of the heterogeneous data, and the matrix extraction module 320 is used to:

[0214] performing natural language preprocessing on the text data, extracting from the text data a plurality of text feature groups and an image feature group corresponding to each text feature group; wherein the text feature group includes at least two of Chinese character features, phrase features, and sentence features; and the text feature group is expressed in a one-hot vector format;

[0215] Determine a text feature matrix for each text feature group and a picture feature matrix for the picture feature group corresponding to each text feature group;

[0216] For each text feature group, fuse the text feature matrix of the text feature group with the image feature matrix of the image feature group corresponding to the text feature group to obtain a preliminary fusion matrix of the text feature group;

[0217] For each text feature group, based on the preliminary fusion matrix of the text feature group and the picture feature matrix of the picture feature group corresponding to the text feature group, determine a multidimensional correlation matrix between each sub-text feature in the text feature group and the sub-picture feature corresponding to the sub-text feature;

[0218] Determining a multidimensional attention weight of each sub-text feature based on a multidimensional correlation matrix of each sub-text feature in the text feature group;

[0219] Based on the multi-dimensional attention weight of each sub-text feature in each text feature group and the image feature matrix of the sub-image feature corresponding to each sub-text feature, a unimodal feature matrix of the text data is determined.

[0220] Furthermore, when the matrix extraction module 320 is used to determine the text feature matrix of each text feature group and the image feature matrix of the image feature group corresponding to each text feature group, the matrix extraction module 320 is used to:

[0221] For each text feature group, using a pre-trained feature extraction model corresponding to the text feature group, determine a text feature matrix that can characterize each sub-text feature in the text feature group;

[0222] The pre-trained convolutional neural network is used to determine the image feature matrix of the image feature group corresponding to the text feature group.

[0223] Furthermore, when the matrix extraction module 320 is used to fuse the text feature matrix of the text feature group and the image feature matrix of the image feature group corresponding to the text feature group to obtain a preliminary fusion matrix of the text feature group, the matrix extraction module 320 is used to:

[0224] For each sub-text feature in the text feature group, a sub-text matrix of the sub-text feature is fused with a sub-image matrix of a sub-image feature corresponding to the sub-text feature to obtain a sub-fusion matrix of each sub-text feature;

[0225] Based on the sub-fusion matrix of each sub-text feature, a preliminary fusion matrix of the text feature group is determined.

[0226] Furthermore, when the matrix extraction module 320 is used to determine the multidimensional correlation matrix between each sub-text feature in the text feature group and the sub-picture feature corresponding to the sub-text feature based on the preliminary fusion matrix of the text feature group and the picture feature matrix of the picture feature group corresponding to the text feature group, the matrix extraction module 320 is used to:

[0227] For each sub-text feature in the text feature group, a multidimensional correlation matrix of the sub-text feature is determined based on the sub-fusion matrix of the sub-text feature and the sub-image matrix of the sub-image feature corresponding to the sub-text feature.

[0228] Furthermore, the matrix extraction module 320 is used to obtain a feature extraction model corresponding to the text feature group through the following steps:

[0229] Obtaining a pre-trained language pre-training model; the language pre-training model is used to extract a text feature matrix representing each sub-text feature of the text feature group from the one-hot vector of the text feature group;

[0230] Performing model distillation processing on the language pre-training model to compress parameters in the language pre-training model to obtain the feature extraction model.

[0231] The multimodal feature fusion device provided in the embodiment of the present application obtains multiple heterogeneous data of the target object; for each heterogeneous data, extracts the unimodal feature matrix of the heterogeneous data; for each unimodal feature matrix, determines the unimodal weight matrix between the unimodal feature matrix and each unimodal feature matrix; based on the unimodal feature matrix and the multiple unimodal weight matrices corresponding to the unimodal feature matrix, determines the fusion feature matrix of the unimodal feature matrix; for each fusion feature matrix, determines the fusion weight matrix between the fusion feature matrix and each fusion feature matrix; for each fusion weight matrix corresponding to the fusion feature matrix, normalizes the fusion weight matrix to obtain a normalized fusion weight matrix; based on each fusion feature matrix and the multiple normalized fusion weight matrices corresponding to each fusion feature matrix, determines the multimodal feature matrix for describing the target object. In this way, the different modal features carried by the heterogeneous data can be fused to express the characteristics of the target object in a more detailed manner.

[0232] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in FIG, the electronic device 500 includes a processor 510, a memory 520 and a bus 530.

[0233] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the above-mentioned Figure 1 The steps of the multimodal feature fusion method in the illustrated method embodiment and the specific implementation thereof can be found in the method embodiment and will not be described in detail here.

[0234] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the multimodal feature fusion method in the illustrated method embodiment and the specific implementation thereof can be found in the method embodiment and will not be described in detail here.

[0235] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0236] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0237] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0238] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0239] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0240] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multimodal feature fusion method, characterized in that: The multimodal feature fusion method includes: Acquire multiple heterogeneous data of the target object; For each heterogeneous data, extract the single modal feature matrix of the heterogeneous data; For each unimodal feature matrix, a unimodal weight matrix between the unimodal feature matrix and each unimodal feature matrix is determined by matrix division operation; the objects of the matrix division operation are the product of the unimodal feature matrix, the unimodal feature matrix and the transposed matrix of each unimodal feature matrix; Based on the unimodal feature matrix and a plurality of unimodal weight matrices corresponding to the unimodal feature matrix, determining a fusion feature matrix of the unimodal feature matrix by a first summation operation; objects of the first summation operation are quotients between the absolute values of the transposed matrices of the unimodal feature matrix and each unimodal weight matrix; For each fusion feature matrix, the fusion weight matrix between the fusion feature matrix and each fusion feature matrix is determined by taking the reciprocal operation; the object of the reciprocal operation is the absolute value of the product between the fusion feature matrix and the transposed matrix of each fusion feature matrix; For each fusion weight matrix corresponding to the fusion feature matrix, the fusion weight matrix is normalized to obtain a normalized fusion weight matrix; Based on each fusion feature matrix and multiple normalized fusion weight matrices corresponding to each fusion feature matrix, a multimodal feature matrix used to describe the target object is determined through a second summation operation; the objects of the second summation operation are the products of each fusion feature matrix and multiple normalized fusion weight matrices corresponding to each fusion feature matrix.

2. The multimodal feature fusion method according to claim 1, characterized in that: When the heterogeneous data includes audio data, extracting a unimodal feature matrix of the heterogeneous data includes: Converting the audio data into mono audio data, and resampling the mono audio data to obtain resampled audio data; Moving a Hanning time window of a preset window length on the resampled audio data, performing Fourier transform on the resampled audio data, and obtaining an audio spectrum of the audio data; Mapping the audio spectrum into an initial Mel-cepstrum using a filter bank; Performing logarithmic calculation on the initial Mel-cepstrum to obtain a stable Mel-cepstrum; The stable Mel-frequency cepstrum is reorganized according to a preset time length to obtain a unimodal feature matrix of the audio data.

3. The multimodal feature fusion method according to claim 1, characterized in that: When the heterogeneous data includes text data, extracting a unimodal feature matrix of the heterogeneous data includes: performing natural language preprocessing on the text data, extracting from the text data a plurality of text feature groups and an image feature group corresponding to each text feature group; wherein the text feature group includes at least two of Chinese character features, phrase features, and sentence features; and the text feature group is expressed in a one-hot vector format; Determine a text feature matrix for each text feature group and a picture feature matrix for the picture feature group corresponding to each text feature group; For each text feature group, fuse the text feature matrix of the text feature group with the image feature matrix of the image feature group corresponding to the text feature group to obtain a preliminary fusion matrix of the text feature group; For each text feature group, based on the preliminary fusion matrix of the text feature group and the picture feature matrix of the picture feature group corresponding to the text feature group, determine a multidimensional correlation matrix between each sub-text feature in the text feature group and the sub-picture feature corresponding to the sub-text feature; Determining a multidimensional attention weight of each sub-text feature based on a multidimensional correlation matrix of each sub-text feature in the text feature group; Based on the multi-dimensional attention weight of each sub-text feature in each text feature group and the image feature matrix of the sub-image feature corresponding to each sub-text feature, a unimodal feature matrix of the text data is determined.

4. The multimodal feature fusion method according to claim 3, characterized in that: The step of determining a text feature matrix for each text feature group and a picture feature matrix for the picture feature group corresponding to each text feature group includes: For each text feature group, using a pre-trained feature extraction model corresponding to the text feature group, determine a text feature matrix that can characterize each sub-text feature in the text feature group; The pre-trained convolutional neural network is used to determine the image feature matrix of the image feature group corresponding to the text feature group.

5. The multimodal feature fusion method according to claim 3, characterized in that: The fusing of the text feature matrix of the text feature group and the image feature matrix of the image feature group corresponding to the text feature group to obtain a preliminary fusion matrix of the text feature group includes: For each sub-text feature in the text feature group, a sub-text matrix of the sub-text feature is fused with a sub-image matrix of a sub-image feature corresponding to the sub-text feature to obtain a sub-fusion matrix of each sub-text feature; Based on the sub-fusion matrix of each sub-text feature, a preliminary fusion matrix of the text feature group is determined.

6. The multimodal feature fusion method according to claim 5, characterized in that: The step of determining a multidimensional correlation matrix between each sub-text feature in the text feature group and the sub-picture feature corresponding to the sub-text feature based on the preliminary fusion matrix of the text feature group and the picture feature matrix of the picture feature group corresponding to the text feature group includes: For each sub-text feature in the text feature group, a multidimensional correlation matrix of the sub-text feature is determined based on the sub-fusion matrix of the sub-text feature and the sub-image matrix of the sub-image feature corresponding to the sub-text feature.

7. The multimodal feature fusion method according to claim 1, characterized in that: When the heterogeneous data includes video data, the multimodal feature fusion method further includes: dividing the video data into multiple frames of image data; When the heterogeneous data includes image data, extracting a single-modal feature matrix of the heterogeneous data includes: A deep neural network with a residual jump mechanism is used to extract the unimodal feature matrix of the image data.

8. A multimodal feature fusion device, characterized in that: The multimodal feature fusion device comprises: A data acquisition module is used to acquire multiple heterogeneous data of the target object; A matrix extraction module is used to extract the single-modal feature matrix of each heterogeneous data source; A unimodal weight determination module is used to determine, for each unimodal feature matrix, a unimodal weight matrix between the unimodal feature matrix and each unimodal feature matrix by matrix division operation; the objects of the matrix division operation are the product of the unimodal feature matrix, the unimodal feature matrix and the transposed matrix of each unimodal feature matrix; a fusion matrix determination module, configured to determine a fusion feature matrix of the unimodal feature matrix by a first summation operation based on the unimodal feature matrix and a plurality of unimodal weight matrices corresponding to the unimodal feature matrix; wherein the objects of the first summation operation are the quotients between the absolute values of the transposed matrices of the unimodal feature matrix and each unimodal weight matrix; A fusion weight determination module is used to determine, for each fusion feature matrix, a fusion weight matrix between the fusion feature matrix and each fusion feature matrix by taking a reciprocal operation; the object of the reciprocal operation is the absolute value of the product between the fusion feature matrix and the transposed matrix of each fusion feature matrix; A matrix normalization module is used to normalize each fusion weight matrix corresponding to the fusion feature matrix to obtain a normalized fusion weight matrix; A multimodal feature determination module is used to determine a multimodal feature matrix used to describe the target object through a second summation operation based on each fusion feature matrix and multiple normalized fusion weight matrices corresponding to each fusion feature matrix; the objects of the second summation operation are the products of each fusion feature matrix and multiple normalized fusion weight matrices corresponding to each fusion feature matrix.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the multimodal feature fusion method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal feature fusion method according to any one of claims 1 to 7 are executed.

Images