Training method and device of multi-modal model

Abstract

This disclosure provides a method and apparatus for training a multimodal model. The method includes: constructing multimodal samples based on image and / or audio data; in a first training phase, inputting the multimodal samples into a first combination block for iterative training; constructing a multimodal model for the i-th training phase based on the i-th combination block and the first i-1 trained combination blocks; when i is greater than 2, obtaining the first i-1 trained combination blocks based on the multimodal model of the (i-1)-th training phase; in the i-th training phase, inputting the multimodal samples into the multimodal model of the i-th training phase for iterative training until the multimodal model of the N-th training phase is obtained, and determining the multimodal model of the N-th training phase as the target multimodal model. This embodiment can maintain stable accuracy of the multimodal model even in the event of random modality loss.

Description

Technical Field

[0001] This document relates to the field of artificial intelligence technology, and in particular to a training method and apparatus for a multimodal model. Background Technology

[0002] With the rapid development of internet technology, the application of multimodal data is becoming increasingly widespread. In related technologies, after multimodal data is input into a classification model, the model can extract data features corresponding to each modality from the data of various modalities, fuse the data features of multiple modalities, and use the fused features to classify specific objects.

[0003] However, in practical applications, modality loss frequently occurs. When any modality is missing, the classification model often needs to fill in the input with invalid values, leading to the introduction of a large amount of noise into the fused features and a significant decrease in classification accuracy. Therefore, how to maintain the accuracy of classification models in classifying multimodal data in the event of random modality loss has received increasing attention. Summary of the Invention

[0004] This disclosure provides a method and apparatus for training a multimodal model to maintain stable accuracy of the multimodal model in the event of random mode loss.

[0005] In a first aspect, embodiments of this disclosure provide a method for training a multimodal model, comprising: Construct multimodal samples based on image and / or audio data; In the first training phase, the multimodal samples are input into the first combination block for iterative training; the first combination block includes N single-modal branch networks for feature extraction, where N is the number of modal types of the multimodal samples; Based on the i-th combined block and the first i-1 combined blocks after training, a multimodal model for the i-th training stage is constructed; where i is greater than 1; the i-th combined block includes at least one multimodal branch network, each of which is used to extract features from the received sample features, and the sample features correspond to i modalities; when i is greater than 2, the first i-1 combined blocks after training are obtained based on the multimodal model for the (i-1)-th training stage after training. In the i-th training phase, the multimodal samples are input into the multimodal model of the i-th training phase for iterative training until the multimodal model of the N-th training phase is obtained, and the multimodal model of the N-th training phase is determined as the target multimodal model.

[0006] Secondly, embodiments of this disclosure provide a training apparatus for a multimodal model, comprising: The sample construction unit is used to construct multimodal samples based on image and / or audio data. The first training unit is used to input the multimodal samples into the first combination block for iterative training in the first training phase; the first combination block includes N single-modal branch networks for feature extraction, where N is the number of modal types of the multimodal samples; A model building unit is used to construct a multimodal model for the i-th training stage based on the i-th combination block and the first i-1 training combination blocks; where i is greater than 1; the i-th combination block includes at least one multimodal branch network, each of the multimodal branch networks is used to extract features from the received sample features, and the sample features correspond to i modalities; when i is greater than 2, the first i-1 training combination blocks are obtained based on the multimodal model for the (i-1)-th training stage. The second training unit is used to input the multimodal samples into the multimodal model of the i-th training stage for iterative training until the multimodal model of the N-th training stage is obtained, and the multimodal model of the N-th training stage is determined as the target multimodal model.

[0007] Thirdly, embodiments of this disclosure provide an electronic device, including: a memory, a processor, and computer-executable instructions stored in the memory and executable on the processor, wherein the computer-executable instructions, when executed by the processor, implement the method described in the first aspect above.

[0008] Fourthly, embodiments of this disclosure provide a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the method described in the first aspect above.

[0009] Fifthly, embodiments of this disclosure provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the method described in the first aspect above.

[0010] In one or more embodiments of this disclosure, firstly, multimodal samples are constructed based on image material data and / or audio material data; then, in the first training stage, the multimodal samples are input into the first combination block for iterative training; the first combination block includes N single-modal branch networks for feature extraction, where N is the number of modal types of the multimodal samples; next, based on the i-th combination block and the first i-1 training combination blocks, a multimodal model for the i-th training stage is constructed; i is greater than 1; the i-th combination block includes at least one multimodal branch network, each multimodal branch network being used to extract features from the received sample features, and the sample features correspond to i modalities; when i is greater than 2, the first i-1 training combination blocks are obtained based on the multimodal model for the (i-1)-th training stage; finally, in the i-th training stage, the multimodal samples are input into the multimodal model for the i-th training stage for iterative training until the multimodal model for the N-th training stage is obtained, and the multimodal model for the N-th training stage is determined as the target multimodal model. As can be seen, in this embodiment, the single-modal branch networks in the first combination block are trained in the first training stage. Then, the trained old combination blocks are reused, and the multimodal model built based on the old and new combination blocks is put into the model training of the next training stage. The new combination block includes a multimodal branch network, and the number of modal types corresponding to the sample features received by the multimodal branch network matches the next training stage. This makes the final target multimodal model integrate the combination blocks that have been fully trained in each training stage. Therefore, in the scenario of random modality missing, the target multimodal model does not need to rely on invalid value filling. It can make full use of the low-order modal branches in each combination block that match the input data after the modality is missing to complete the inference, so that the accuracy of the multimodal model remains stable. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in one or more embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a training method for a multimodal model provided in one embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of a three-modal model provided in one embodiment of the present disclosure; Figure 3A This is a schematic diagram of the first training stage in a training method for a multimodal model provided in an embodiment of this disclosure; Figure 3BThis is a schematic diagram of the k-th training stage in a training method for a multimodal model provided in an embodiment of this disclosure; Figure 4A This is a schematic diagram of the first training phase of a trimodal model provided in an embodiment of the present disclosure; Figure 4B This is a schematic diagram of the second training phase of a trimodal model provided in an embodiment of the present disclosure; Figure 4C This is a schematic diagram of the third training phase of a trimodal model provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of a dual-modal model provided in one embodiment of the present disclosure; Figure 6A This is a schematic diagram of the first training phase of a bimodal model provided in an embodiment of the present disclosure; Figure 6B This is a schematic diagram of the second training phase of a bimodal model provided in an embodiment of the present disclosure; Figure 7 A schematic diagram of the structure of a training device for a multimodal model provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this disclosure, the technical solutions in one or more embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of the embodiments. Based on one or more embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0014] This disclosure provides a method and apparatus for training a multimodal model, which can maintain the accuracy of the multimodal model in the event of random modality loss. The training method for the multimodal model can be applied to a server and implemented by a server-side device, which can be a single server, a server cluster consisting of several servers, or one or more cloud servers in a cloud computing platform.

[0015] Figure 1 This is a flowchart illustrating a training method for a multimodal model according to an embodiment of this disclosure. Figure 1 As shown, the process includes: Step S102: Construct multimodal samples based on image material data and / or audio material data.

[0016] Step S104: In the first training phase, the multimodal samples are input into the first combination block for iterative training; the first combination block includes N single-modal branch networks for feature extraction, where N is the number of modal types of the multimodal samples.

[0017] Step S106: Based on the i-th combination block and the first i-1 combined blocks after training, construct the multimodal model for the i-th training stage; i is greater than 1; the i-th combination block includes at least one multimodal branch network, each multimodal branch network is used to extract features from the received sample features, and the sample features correspond to i modalities; when i is greater than 2, the first i-1 combined blocks after training are obtained based on the multimodal model for the (i-1)-th training stage after training.

[0018] Step S108: In the i-th training stage, the multimodal samples are input into the multimodal model of the i-th training stage for iterative training until the multimodal model of the N-th training stage is obtained. The multimodal model of the N-th training stage is then determined as the target multimodal model.

[0019] In this embodiment, firstly, multimodal samples are constructed based on image and / or audio data. Then, in the first training phase, the multimodal samples are input into the first combination block for iterative training. The first combination block includes N single-modal branch networks for feature extraction, where N is the number of modal types in the multimodal samples. Next, based on the ith combination block and the first i-1 training combination blocks, a multimodal model for the ith training phase is constructed, where i is greater than 1. The ith combination block includes at least one multimodal branch network, each used to extract features from the received sample features, with each sample feature corresponding to an i-th modality. If i is greater than 2, the first i-1 training combination blocks are obtained based on the multimodal model from the (i-1)th training phase. Finally, in the ith training phase, the multimodal samples are input into the multimodal model from the ith training phase for iterative training until the multimodal model from the Nth training phase is obtained, and the multimodal model from the Nth training phase is determined as the target multimodal model. As can be seen, in this embodiment, the single-modal branch networks in the first combination block are trained in the first training stage. Then, the trained old combination blocks are reused, and the multimodal model built based on the old and new combination blocks is put into the model training of the next training stage. The new combination block includes a multimodal branch network, and the number of modal types corresponding to the sample features received by the multimodal branch network matches the next training stage. This makes the final target multimodal model integrate the combination blocks that have been fully trained in each training stage. Therefore, in the scenario of random modality missing, the target multimodal model does not need to rely on invalid value filling. It can make full use of the low-order modal branches in each combination block that match the input data after the modality is missing to complete the inference, so that the accuracy of the multimodal model remains stable.

[0020] In step S102 above, multimodal samples are constructed based on image material data and / or audio material data. Image material data includes, but is not limited to, photographs, thermal images, depth maps, etc. Audio material data includes, but is not limited to, speech, background noise, music audio, etc.

[0021] Multimodal samples can be composed of multiple unimodal sub-samples. In one example, multimodal samples include sub-samples of three modes: visible mode sub-sample 1, thermal imaging mode sub-sample 2, and audio mode sub-sample 3.

[0022] In one example, multimodal samples are represented by an audio spectrogram X, a video Y, and a hot video Z, where video Y has j frames and a size of (64 × 64 × 3); and hot video Z has j frames and a size of (64 × 64 × 1). For both visible and hot videos, the image in each visible or hot video sequence is segmented into 8 × 8 fixed-size patches and linearly embedded into a 128-dim (dimension) projection space. Subsequently, a learnable positional embedding is added to the linear embedding of each frame. The image embeddings are then concatenated to form a sequence vector, into which a frame embedding is added as a frame index, distinguishing the embedding vectors between the three frames in the input video sequence. The layer described above for patch extraction, positional embedding, and frame embedding is called the standard embedding layer. For audio data, the audio spectrogram serves as input to a three-layer Conv-1D (one-dimensional convolution) layer with 32, 64, and 128 filters of sizes 11, 11, and 3, respectively, a stride of 1, and a ReLU activation function. This architecture is called an audio convolutional layer.

[0023] In one example, multimodal samples are constructed based on image source data. For instance, a visible modality subsample 1 is constructed based on a photograph of person A, and a thermal imaging modality subsample 2 is constructed based on a thermal image of person A. Subsample 1 and subsample 2 are combined to obtain multimodal sample 1 related to person A.

[0024] In one example, multimodal samples are constructed based on audio source data. For instance, subsample 1 of modality Y1 is constructed based on speech X1, subsample 2 of modality Y2 is constructed based on background sound X2, and subsample 3 of modality Y3 is constructed based on music audio X3. Subsample 1, subsample 2, and subsample 3 are combined together to obtain multimodal sample 2.

[0025] In one example, multimodal samples are constructed based on image and audio data. For instance, a subsample 1 for the visible modality is constructed based on photographs, a subsample 2 for the thermal imaging modality is constructed based on thermal imaging, and a subsample 3 for the audio modality is constructed based on mixed audio data. Subsamples 1, 2, and 3 are then combined to obtain multimodal sample 3.

[0026] In one embodiment, the multimodal samples include complete training samples and missing training samples; constructing multimodal samples based on image material data and / or audio material data includes: constructing a first training sample and a second training sample based on image material data and / or audio material data; the first training sample includes sub-samples of N modalities, and the second training sample includes sub-samples of M modalities, where N is greater than M; the first training sample is determined as the complete training sample; the second training sample is filled with preset filling sub-samples to obtain the missing training sample.

[0027] N represents the number of modalities in the multimodal samples. In one example, N is 3 and M is 2, so the first training sample includes subsamples of 3 modalities, and the second training sample includes subsamples of 2 modalities.

[0028] Based on image and / or audio data, construct the first and second training samples, as illustrated in the examples above.

[0029] In one example, the first training sample and the second training sample can be constructed separately. For example, S1 first training samples and S2 second training samples can be constructed separately.

[0030] In another example, a first training sample can be constructed first. A target modality is then randomly selected from the various modalities of the first training sample. Subsamples of the target modality are then eliminated from the first training sample to obtain a second training sample. For example, S3 first training samples can be constructed first. Then, S4 first training samples can be randomly selected from these S3. For each of these S4 first training samples, one or more target modalities are randomly selected from the various modalities, and subsamples of the target modality are eliminated from the first training sample to obtain a second training sample.

[0031] The first training sample is determined as the complete training sample. The second training sample is then filled with pre-defined filler sub-samples to obtain the missing training sample. In one example, the first training sample 1 includes sub-samples of three modalities, and is thus determined as the complete training sample. The second training sample 1 includes sub-samples of modality 1. It is filled with pre-defined filler sub-samples of modality 2 and modality 3 to obtain the missing training sample 1. The second training sample 2 includes sub-samples of modality 1 and modality 3. It is filled with pre-defined filler sub-samples of modality 2 to obtain the missing training sample 2.

[0032] Preset fill subsamples for various modalities can be customized as needed. For example, in the visible modality, a pure black image of a specified size can be set as the preset fill subsample for that visible modality.

[0033] In one example, for multimodal data, if all modalities are present, the multimodal data can be referred to as "complete" data, i.e., complete training samples.

[0034] In one example, for trimodal data, if a single modality is missing, the data can be called "single-modal missing" data. If two modalities are missing, the data can be called "double-missing" data. The data corresponding to the missing modalities are represented by predefined fixed data. That is, missing training samples include "single-modal missing" data and "double-missing" data.

[0035] During the learning phase, by randomly absolving multimodal data into predefined fixed data, the random mode loss that occurs in real-world scenarios can be fully simulated.

[0036] As can be seen, through this embodiment, by constructing complete training data and missing training data during the model training phase, and using both complete training data and missing training data together for model training, the multimodal model can fully learn how to process data in scenarios with missing random modalities.

[0037] In step S104 above, in the first training stage, multimodal samples are input into the first combination block for iterative training. The first combination block includes N single-modal branch networks for feature extraction, where N is the number of modality types in the multimodal samples. A combination block is a parameterized unit that encapsulates a specific function. In this embodiment, the specific function encapsulated by the combination block can be a feature extraction function. This embodiment can employ a progressive learning strategy for the multimodal model to be trained, dividing the complete model training process into N training stages. In one example, the multimodal model can be a PLACH (Progressive Learning with K-combination blocks architecture Change) model. K and N are the same value.

[0038] Before step S104 is executed, single-modal models corresponding to various modalities can be constructed based on each single-modal branch network in the first combination block. Multimodal samples can include N sub-samples, each sub-sample corresponding to one modality. Iterative training of the multimodal samples into the first combination block can be achieved by inputting each sub-sample of the multimodal samples into the single-modal models of various modalities for iterative training. After step S104 is executed, the first combined block after training can be obtained.

[0039] In one embodiment, the multimodal samples include visible sub-samples, thermal imaging sub-samples, and audio sub-samples; the first combination block includes: a visible branch network, a thermal imaging branch network, and an audio branch network; inputting the multimodal samples into the first combination block for iterative training includes: inputting the visible sub-samples into the visible branch network for iterative training; inputting the thermal imaging sub-samples into the thermal imaging branch network for iterative training; and inputting the audio sub-samples into the audio branch network for iterative training.

[0040] In one example, the first combination block includes: a visible branch network for the visible modality, a thermal imaging branch network for the thermal imaging modality, and an audio branch network for the audio modality. Based on the visible branch network and the first classification head, a single-modal model for the visible modality is constructed; based on the thermal imaging branch network and the second classification head, a single-modal model for the thermal imaging modality is constructed; and based on the audio branch network and the third classification head, a single-modal model for the audio modality is constructed. The first, second, and third classification heads mentioned above can be three different preset classification heads.

[0041] During the iterative training of the visible branch network by inputting visible subsamples into the visible subsamples, the visible subsamples can be input into the visible branch network for feature extraction to obtain visible features. The visible features are then input into the first classification head for classification to obtain the first classification result. The first training loss is generated based on the first classification result, and the visible branch network is iteratively trained based on the first training loss.

[0042] During the iterative training of the thermal imaging sub-samples into the thermal imaging branch network, the thermal imaging sub-samples can be input into the thermal imaging branch network for feature extraction to obtain thermal imaging features. The thermal imaging features are then input into the second classification head for classification to obtain the second classification result. Based on the second classification result, a second training loss is generated, and the thermal imaging branch network is iteratively trained based on the second training loss.

[0043] During the iterative training of the audio sub-samples into the audio branch network, the audio sub-samples can be input into the audio branch network for feature extraction to obtain audio features. The audio features are then input into the third classification head for classification to obtain the third classification result. Based on the third classification result, a third training loss is generated, and the thermal imaging branch network is iteratively trained based on the third training loss.

[0044] As can be seen, by training the visible branch network, thermal imaging branch network and audio branch network in the first combined block respectively through this embodiment, the first combined block can learn the basic feature representation and independent classification ability of three types of single-modal data. At the same time, in the progressive learning process, it provides a combined block with fixed parameters, reusable functions and already trained for each subsequent training stage.

[0045] In step S106 above, a multimodal model for the i-th training stage is constructed based on the i-th combination block and the first i-1 combination blocks after training; i is greater than 1; the i-th combination block includes at least one multimodal branch network, each multimodal branch network is used to extract features from the received sample features, and the sample features correspond to i modes; when i is greater than 2, the first i-1 combination blocks after training are obtained based on the multimodal model for the (i-1)-th training stage after training.

[0046] In this embodiment, i is an integer greater than 1. When i=2, the multimodal model for the second training stage is constructed based on the second combined block and the first combined block after training. The first combined block after training is the first combined block after training. When i>2, the multimodal model for the i-th training stage is constructed based on the i-th combined block and the first i-1 combined blocks after training. For example, when i=3, the multimodal model for the third training stage is constructed based on the third combined block and the first two combined blocks after training. The first two combined blocks after training are obtained from the multimodal model for the second training stage after training.

[0047] The i-th combination block may include one or more multimodal branch networks, each of which is used to extract features from the received sample features, and the sample features correspond to i modalities. In one example, the second combination block includes three multimodal branch networks, each of which is used to extract features from the received sample features, and the sample features correspond to two modalities.

[0048] In one example, for a set S with K modalities, the power set P(S) is the set of all subsets of S; the distinct k-combinations of subsets in P(S) are represented by the set... Indicates; each Include One reason The given elements. In this example, the value of K is the same as the value of N mentioned above. For example, for a set S = {a, b, c}, that is, a set S has 3 elements, K=3. The power set mentioned above refers to the set of all subsets of a set S. For S = {a, b, c}, its power set P(S) = { ,{a}, {b}, {c}, {a,b}, {a,c}, {b,c}, {a,b,c}}. in, This represents the empty set. The k-combination subsets mentioned above refer to subsets containing exactly k elements. For a set S = {a, b, c}, the classification is based on the number of elements k: k=0: The k-combinatorial subset includes , ={ },quantity =1.

[0049] k=1: The k-combination subset includes {a}, {b}, {c}. = {{a}, {b}, {c}}, quantity =3.

[0050] k=2: The k-combinatorial subsets include {a,b}, {a,c}, {b,c}. ={{a,b}, {a,c}, {b,c}}, quantity =3.

[0051] k=3: The subset of k combinations includes {a,b,c}, so = {{a,b,c}}, quantity = 1.

[0052] Referring to the example above, the PLACH model comprises a total of K combinatorial blocks, each of which can be combined with the above... Correspondingly. For example, the first combinatorial block in the PLACCH model corresponds to That is, the first composite block includes three unimodal branch networks, where the first unimodal branch network corresponds to {a}, the second unimodal branch network corresponds to {b}, and the third unimodal branch network corresponds to {c}. The second composite block in the PLACCH model corresponds to... That is, the second combinatorial block includes three multimodal branch networks, where the first multimodal branch network corresponds to {a, b}, the second multimodal branch network corresponds to {a, c}, and the third multimodal branch network corresponds to {b, c}. The third combinatorial block in the PLACCH model corresponds to... That is, the third combinatorial block includes a multimodal branch network, which corresponds to {a,b,c}.

[0053] In one example, the branches in the PLACH model can be implemented using a multi-head attention transformer. These branches include the unimodal branch networks in the first combinatorial block and the multimodal branch networks in the i-th combinatorial block. The layers in the transformer are called standard transformer layers. Given a transformer input, the standard transformer layer receives the input and first passes it through a layer normalization layer, followed by a multi-head attention layer configured with four attention heads and a 0.1 dropout (random deactivation). The output of the attention layer is added to the input to obtain the attention output vector, which serves as the input to two MLP (Multilayer Perceptron) layers with 256 and 128 units respectively, and a dropout of 0.1. The MLP output is then added to the attention output vector to obtain the output of the transformer branch.

[0054] In one embodiment, if i equals 2, based on the i-th combination block and the first i-1 combined blocks after training, a multimodal model for the i-th training stage is constructed, including: determining the first connection relationship between the multimodal branch network and the single-modal branch network according to the two modes corresponding to the multimodal branch network in the second combination block and the one mode corresponding to the single-modal branch network in the first combined block after training; and constructing a multimodal model for the second training stage based on the second combination block, the first combined block after training, and the first connection relationship.

[0055] In one example, the first combo block can be used This means that the second combo block can be used Representation. Composite block Includes: visible branch networks Thermal imaging subnetwork Audio branch network Among them, the visible branch network Single-mode branching network for visible modes, thermal imaging branching network For thermal imaging modalities, a single-mode branch network and an audio branch network are provided. This is a single-modal branch network for audio modalities. (Combined block) Includes: Visible-thermal imaging subnetwork Audio-visible branch network Audio-thermal imaging subnetwork Among them, the visible-thermal imaging branch network Corresponding to two modalities: visible mode and thermal imaging mode; audio-visual branch network Corresponding to two modalities: audio mode and visible mode; audio-thermal imaging branch network It corresponds to two modes: audio mode and thermal imaging mode.

[0056] The above visible branch network The output sample features can be used This means that the same applies to other single-modal branch networks and multimodal branch networks.

[0057] The above visible-thermal imaging branch network The input can be determined by the output of a portion of the single-mode branch network in the first combination block. The specific input selection mechanism can be found in the following expression: (1) In the above expression (1), This refers to the visible-thermal imaging branch network. Output sample features, Refers to composite blocks The sample features output by the j-th unimodal branch network. Refers to composite blocks The j-th single-modal branch network, This represents the empty set. The entire expression (1) is used to represent: the set consisting of all sets that satisfy " and The condition of "existence of intersection" The set that constitutes.

[0058] For example, visible branch networks Output and If there is an intersection, then the visible-thermal imaging branch network exists. The input includes visible branch networks. The output of the thermal imaging branch network. Output and If there is an intersection, then the visible-thermal imaging branch network exists. The input includes a thermal imaging branch network. The output of the audio branch network. Output and If there is no intersection, then it is a visible-thermal imaging branch network. The input does not include the audio branch network. The output of the above audio-visible branch network. Audio-thermal imaging subnetwork The input can be referenced from the visible-thermal imaging branch network described above. The corresponding description section for the input.

[0059] In summary, the first connection relationships include: Visible branch network With audio-visible branch network Connected, and visible branch networks Previously, the audio-visual branch network Following; audio branch network With audio-visible branch network Connection, and audio branch network Previously, the audio-visual branch network After.

[0060] Thermal Imaging Branch Network With audio-thermal imaging subnetwork Connections, and thermal imaging branch networks Previously, the audio-thermal imaging subnetwork Following; audio branch network With audio-thermal imaging subnetwork Connection, and audio branch network Previously, the audio-thermal imaging subnetwork After.

[0061] In the process of constructing the multimodal model for the second training stage based on the second combined block, the first combined block after training, and the first connection relationship, the first combined block after training, the second combined block, and the preset classification head can be connected according to the first connection relationship to obtain the multimodal model for the second training stage.

[0062] In one example, the visible branch network will be shown. With visible-thermal imaging branch network Connect the thermal imaging branch network With visible-thermal imaging branch network Connect, and, the visible-thermal imaging branch network Connecting it to the fourth classification head yields the first multimodal model in the second training phase.

[0063] thermal imaging branch network With audio-thermal imaging subnetwork Connect, link the audio branch network With audio-thermal imaging subnetwork Connections, and the audio-thermal imaging branch network Connecting it to the fifth classification head yields the first multimodal model in the second training phase.

[0064] thermal imaging branch network With audio-thermal imaging subnetwork Connect, link the audio branch network With audio-thermal imaging subnetwork Connections, and the audio-thermal imaging branch network Connecting it to the sixth classification head yields the third multimodal model in the second training phase.

[0065] As can be seen, through this embodiment, the connection relationship of each branch network in adjacent combination blocks is determined by utilizing modal characteristics, and based on this connection relationship, the first combination block with fixed parameters and the second combination block to be trained are combined to construct the multimodal model of the second training stage, and then carry out staged iterative training.

[0066] In one embodiment, if i is greater than 2, a multimodal model for the i-th training stage is constructed based on the i-th combination block and the first i-1 combined blocks after training. This includes: determining a second connection relationship between the multimodal branch networks in the i-th combination block and the multimodal branch networks in the (i-1)-th combination block according to the i-th modes corresponding to the multimodal branch networks in the i-th combination block and the i-1 modes corresponding to the multimodal branch networks in the (i-1)-th combination block after training; and constructing a multimodal model for the i-th training stage based on the i-th combination block, the first i-1-th combined blocks after training, and the second connection relationship.

[0067] In one example, the i-th combinatorial block can be used Indicates a composite block The k-th multimodal branch network can be used This indicates that the multimodal branch network The output sample features can be used express.

[0068] When i is greater than 2, for any multimodal branch network in the i-th combination block, that multimodal branch network can selectively accept some or all of the outputs of the multimodal branch networks in the previous combination block. That is, the aforementioned multimodal branch networks... The input can be composed of combination blocks The output of some or all of the multimodal branch networks is determined, and the specific input selection mechanism is described in the following expression: (2) In the above expression (1), This refers to multimodal branching networks. Output sample features, Refers to composite blocks The sample features output by the j-th multimodal branch network. Refers to composite blocks The j-th multimodal branch network, This represents the empty set. The entire expression (2) is used to represent: the set consisting of all sets that satisfy " and The condition of "existence of intersection" The set that constitutes.

[0069] For example, visible-thermal imaging branch networks Output With audio-visual-thermal imaging subnetwork Output If there is an intersection, then the audio-visual-thermal imaging branch network is valid. The input includes a visible-thermal imaging branch network. The output of the audio-visible branch network. Output With audio-visual-thermal imaging subnetwork Output If there is an intersection, then the audio-visual-thermal imaging branch network is valid. The input includes audio-visible branch networks. The output of the audio-thermal imaging branch network. Output With audio-visual-thermal imaging subnetwork Output If there is an intersection, then the audio-visual-thermal imaging branch network is valid. The input includes an audio-thermal imaging subnetwork. The output of .

[0070] In summary, the second connection relationship includes: Visible-thermal imaging branch network With audio-visual-thermal imaging subnetwork Connected, and visible-thermal imaging branch network Previously, the audio-visual-thermal imaging subnetwork Following. Audio-visible branch network. With audio-visual-thermal imaging subnetwork Connectivity, and audio-visual branch network Previously, the audio-visual-thermal imaging subnetwork Following that. Audio-thermal imaging branch network. With audio-visual-thermal imaging subnetwork Connection, and audio-thermal imaging branch network Previously, the audio-visual-thermal imaging subnetwork After.

[0071] In the process of constructing the multimodal model for the i-th training stage based on the i-th combined block, the first i-1 combined blocks after training, and the second connection relationship, the first i-1 combined blocks after training, the i-th combined block, and the preset classification head can be connected according to the second connection relationship to obtain the multimodal model for the i-th training stage. Due to the similar concept, the construction process of the multimodal model for the i-th training stage can refer to the corresponding explanation section in the previous text "Constructing the multimodal model for the second training stage based on the second combined block, the first combined block after training, and the first connection relationship".

[0072] In one example, in the first training phase, all training objects are unimodal models, each consisting of one unimodal branch network from the first combination block. In the second training phase, all training objects are multimodal models, each generated based on the first and second combination blocks. In the third training phase, all training objects are multimodal models, each generated based on the first, second, and third combination blocks, and so on. In the i-th training phase, all training objects are multimodal models, each generated based on the first i-th combination block, and so on. In the N-th training phase, all training objects are multimodal models, each generated based on the first N combination blocks. i is greater than 1 and less than or equal to N.

[0073] As can be seen, through this embodiment, the connection relationship of each branch network in adjacent combination blocks is determined by utilizing modal characteristics, and based on this connection relationship, combined with the old combination blocks with fixed parameters and the new combination blocks to be trained, a multimodal model for the next training stage is constructed, and then staged iterative training is carried out.

[0074] In step S108 above, in the i-th training stage, the multimodal samples are input into the multimodal model of the i-th training stage for iterative training until the multimodal model of the N-th training stage is obtained, and the multimodal model of the N-th training stage is determined as the target multimodal model.

[0075] The target multimodal model refers to the final model to be implemented in the entire training process of this embodiment. All stage-by-stage block training, branch fusion, and parameter optimization are all centered around achieving the performance target of this model. In one example, the target multimodal model can be a multimodal classification model.

[0076] In one example, the complete training process for the target multimodal model is shown below: In the first training phase, multimodal samples are input into the first combination block for iterative training. Based on the second combined block and the first combined block after training, construct the multimodal model for the second training phase; In the second training phase, multimodal samples are input into the multimodal model for iterative training. Based on the third combined block and the first two combined blocks after training, a multimodal model for the third training phase is constructed. In the third training phase, multimodal samples are input into the multimodal model for iterative training. ... Based on the Nth combined block and the first (N-1) combined blocks after training, construct the multimodal model for the Nth training stage; In the Nth training phase, multimodal samples are input into the multimodal model of the Nth training phase for iterative training; After the Nth training phase, the entire training process ends, and the multimodal model after the Nth training phase is determined as the target multimodal model.

[0077] In one example, the branch network is visible in the first multimodal model during the second training phase. With visible-thermal imaging branch network Connection, thermal imaging branch network With visible-thermal imaging branch network Connections, and visible-thermal imaging branch networks Connect to the fourth category head.

[0078] In the second training phase, the multimodal samples are input into the first multimodal model for iterative training. Alternatively, the visible sub-samples from the multimodal samples can be input into the visible branch network. Feature extraction is performed to obtain sample features. Thermal imaging sub-samples from multimodal samples are input into the thermal imaging branch network. Feature extraction is performed to obtain sample features. For sample features With sample features The fusion process is performed to obtain fused feature 1, which is then input into the visible-thermal imaging branch network. Feature extraction is performed to obtain sample features. , sample features The fourth classification head is input for classification processing, resulting in classification result 1. Based on classification result 1, training loss 1 is generated. Based on training loss 1, the visible branch network is then processed. Thermal imaging subnetwork and visible-thermal imaging branch network Perform iterative training.

[0079] In one embodiment, the training method for the multimodal model further includes: recording the training result data of each training stage; and constructing a variant multimodal model associated with the target multimodal model based on the training result data.

[0080] At the end of the first training phase, record the training result data of the first training phase; at the end of the second training phase, record the training result data of the second training phase; ...; at the end of the Nth training phase, record the training result data of the Nth training phase.

[0081] For the i-th training phase, the training result data may include the model parameters of the first i combined blocks in the multimodal model of the i-th training phase after training, as well as the connection relationships between the first i combined blocks.

[0082] In this embodiment, the variant multimodal model can be a derived model of the target multimodal model, and the variant multimodal model can include at least one of the first to Nth combination blocks.

[0083] For example, with N=3, the target multimodal model includes a first combinatorial block X1, a second combinatorial block X2, and a third combinatorial block X3, while variant multimodal model 1 includes combinatorial block X1 and combinatorial block X4. Variant multimodal model 1 is constructed based on the training results data from the first training phase corresponding to the target multimodal model.

[0084] As can be seen, by recording the training results data at different training stages through this embodiment, some training results data can be reused to initialize the parameters of the improved model when the local structure of the model (such as fusion branches and classification heads) is improved, thereby improving training efficiency and reducing the consumption of computing resources and R&D cycle costs required for model training.

[0085] In one embodiment, the target multimodal model is a multimodal face recognition model, and the pre-set filler sub-samples in the missing training samples are labeled with unknown face tags.

[0086] The following example illustrates the concept of a target multimodal model involving three modalities: The multimodal samples include: visible sub-samples, thermal imaging sub-samples, and audio sub-samples. The visible sub-samples are photos containing faces, the thermal imaging sub-samples are thermal images containing faces, and the audio sub-samples are the voice data of the user corresponding to the face. The target multimodal model consists of N combination blocks and a face classification head. These N combination blocks are used to extract facial features from the multimodal samples, and the face classification head is used to classify the facial features to obtain the face recognition result.

[0087] The aforementioned multimodal samples may include S1 complete training samples and S2 missing training samples. The missing training samples contain pre-defined filler sub-samples labeled with unknown face tags. For example, missing training sample 1 includes: a valid photograph containing a face, a pre-defined filler sub-sample for the thermal imaging modality, and valid voice data of the user corresponding to the face. Specifically, the valid photograph containing the face is labeled with a pre-defined face tag, the pre-defined filler sub-sample for the thermal imaging modality is labeled with an unknown face tag, and the valid voice data of the user corresponding to the face is labeled with a pre-defined face tag.

[0088] Taking the first training phase as an example, the role of the unknown face label is explained: In the first training phase, the visible branch network, the thermal imaging branch network, and the audio branch network are trained separately. Specifically, the preset filled sub-samples of the thermal imaging modality are input into the thermal imaging branch network for feature extraction to obtain sample features. The sample features are then input into a preset classification head for classification to obtain classification results. Based on the classification results and the unknown face label, a training loss is generated, and the thermal imaging branch network is iteratively trained based on the training loss.

[0089] In the first training phase, the input data for the thermal imaging branch network only includes pre-filled sub-samples of the thermal imaging modality without real face information, lacking the necessary data for face recognition. By labeling these pre-filled sub-samples with unknown face tags, the pre-classifier head can avoid misclassifying meaningless filler sample features as a certain type of pre-classified face, thereby eliminating the inefficient learning problem caused by the lack of effective training data for the output head.

[0090] As can be seen, by using the unknown face label to mark the pre-filled sub-samples in the missing training samples, the inefficient learning caused by the lack of training data in the output head can be eliminated through this embodiment.

[0091] In summary, through the embodiments of this disclosure, each unimodal branch network in the first combination block is trained in the first training stage. Then, the trained old combination block is reused, and the multimodal model constructed based on the old combination block and the new combination block is put into the model training of the next training stage. The new combination block includes a multimodal branch network, and the number of modality types corresponding to the sample features received by the multimodal branch network matches the next training stage. This makes the final target multimodal model integrate the combination blocks that have been fully trained in each training stage. Therefore, in the scenario of random modality missing, the target multimodal model does not need to rely on invalid value filling. It can make full use of the low-order modality branches in each combination block that match the input data after the modality is missing to complete the inference, so that the accuracy of the multimodal model remains stable.

[0092] Based on the same technical concept, this disclosure also provides another method for training a multimodal model, which can be referred to below. Figures 2-6B .

[0093] Figure 2 This is a schematic diagram of the structure of a three-modal model provided in one embodiment of the present disclosure.

[0094] Figure 2 An exemplary model structure for a trimodal model is shown, which can be trained using the multimodal model training method provided in the above embodiments.

[0095] like Figure 2As shown, multimodal input refers to multimodal samples, which include: sub-sample 1 of the visible modality, sub-sample 2 of the thermal imaging modality, and sub-sample 3 of the audio modality. Among them, sub-sample 1 can be a valid or missing photograph, sub-sample 2 can be a valid or missing thermal image, and sub-sample 3 can be a valid or missing audio.

[0096] like Figure 2 As shown, the three-modal model includes: 1-combination branch, 2-combination branch, 3-combination branch, and personnel output head.

[0097] The first combinatorial block consists of three 1-combination branches, meaning it comprises three unimodal branch networks: visible branch networks. Thermal imaging subnetwork Audio branch network The second combiner block includes three 2-combination branches, meaning it comprises three multimodal branch networks: a visible-thermal imaging branch network. Audio-visible branch network Audio-thermal imaging subnetwork The third combinatorial block includes a 3-combination branch, meaning it comprises a multimodal branch network: an audio-visual-thermal imaging branch network. .

[0098] The personnel output head can be a preset classification head, which can be used to classify the features of the received samples to obtain the face recognition result, which includes personnel tags.

[0099] In one example, the training method for the multimodal model provided in the above-described method embodiment is used to train the model as follows: Figure 2 The specific process of the three-modal model shown is as follows: Based on image and audio data, a trimodal sample is constructed. In the first training phase, the trimodal sample is input into the first combined block for iterative training. The first combined block includes three single-modal branch networks for feature extraction. Based on the second combined block and the first combined block after training, a multimodal model for the second training stage is constructed. The second combined block includes three multimodal branch networks, each of which is used to extract features from the received sample features. The sample features correspond to two modalities. In the second training phase, the three-modal samples are input into the multimodal model for iterative training. Based on the third combined block and the first two combined blocks after training, a multimodal model for the third training stage is constructed. The third combined block includes a multimodal branch network, which is used to extract features from the received sample features. The sample features correspond to three modalities. The first two combined blocks after training are obtained based on the multimodal model for the second training stage after training. In the third training phase, the three-modal samples are input into the multimodal model for iterative training, resulting in the trained multimodal model for the third training phase. The trained multimodal model for the third training phase is then defined as follows: Figure 2 The three-modal model is shown.

[0100] Next, combined Figure 3A and Figure 3B This example illustrates the progressive learning framework used in this embodiment. Figure 3A This is a schematic diagram of the first training stage in a training method for a multimodal model provided in an embodiment of this disclosure; Figure 3B This is a schematic diagram of the k-th training stage in a multimodal model training method provided in an embodiment of this disclosure.

[0101] like Figure 3A As shown, in the first training phase, each single-modal branch in the first combined block is trained using multimodal samples and a pre-defined classification head. The multimodal samples include: input data 1, input data 2, and input data N, with each input data corresponding to one modality.

[0102] Input data 1 is input into branch 1 of the first combination block for feature extraction to obtain data features. The data features are then input into classification head 1 for classification to obtain classification result 1. Branch 1 is then iteratively trained based on classification result 1.

[0103] Input data 2 is input into branch 2 in the first combination block for feature extraction to obtain data features. The data features are then input into classification head 2 for classification to obtain classification result 2. Branch 2 is then iteratively trained based on classification result 2.

[0104] ... The input data N is fed into branch N in the first combination block for feature extraction to obtain data features. These data features are then fed into the classification head N for classification to obtain classification result N. Branch N is iteratively trained based on the classification result N. N represents the number of modalities in the multimodal samples.

[0105] A unimodal model consisting of branch 1 and classifier head 1 can be called a single-peaked model. In the subsequent second training phase, classifier head 1 in this unimodal model can be removed, and only the trained branch 1 can be retained. This trained branch 1 can then be used to construct the multimodal model for the second training phase.

[0106] In the first training phase, N unimodal models are trained. Following a similar concept, the multimodal model in the second training phase can be called a bimodal model.

[0107] like Figure 3B As shown, in the k-th training stage, the multimodal model of the (k-1)-th training stage is obtained after training. The classification head of this multimodal model is removed, and only the first (k-1) combination blocks are retained. The connection relationship between the (k-1)-th combination block and the k-th combination block is determined through an input selection mechanism. For details, please refer to the corresponding explanations of the first and second connection relationships above. Based on the first (k-1)-th combination blocks, the k-th combination block, the preset classification head, and the connection relationships, the multimodal model of the k-th training stage is constructed. Multimodal samples are input into the multimodal model for iterative training. Here, k can be an integer greater than 1.

[0108] Next, combine Figures 4A-4C The training process is illustrated through a specific example of a three-modal model. Figure 4A This is a schematic diagram of the first training phase of a trimodal model provided in an embodiment of the present disclosure; Figure 4B This is a schematic diagram of the second training phase of a trimodal model provided in an embodiment of the present disclosure; Figure 4C This is a schematic diagram of the third training phase of a trimodal model provided in one embodiment of the present disclosure.

[0109] In one example, the trimodal model is formulated for the following classification task: audio-video-thermal imaging personnel classification.

[0110] Use with Audio-video-thermal imaging dataset of 100 samples To train a trimodal model for audio-video-thermal imaging people classification to identify people, where... Indicates audio. This represents a video with j frames. This represents a thermal imaging video with j frames. A label representing a person. Audio input. Represented by a log-Melbourne spectrogram of size 128×889, obtained from the audio using a sampling rate of 44000Hz; video input Represented by three uniformly sampled frames of size (64×64×3); thermal imaging input It is represented by three uniformly sampled frames of size (64×64×1). The video and thermal imaging sequences are synchronized in the dataset.

[0111] The trimodal model consists of three combinatorial blocks: the first combinatorial block contains three 1-combination branches, the second combinatorial block contains three 2-combination branches, and the last combinatorial block contains one 3-combination branch.

[0112] In the first combination block, the visible and thermal imaging sequences are used as inputs to the standard embedding layers in the visible and thermal imaging branches. In the audio branch, an audio spectrogram is provided as input to the audio convolutional layer. The outputs of these layers are then fed as input to a single standard transformer layer.

[0113] In the second combination block, an input selection mechanism is used to concatenate the outputs of selected standard transformer layers and provide them as input to a specific standard transformer layer. For example, the outputs of the visible and thermal imaging branches are concatenated and used as input to the visible-thermal imaging branch. Similarly, the audio-visual and audio-thermal imaging branches are implemented using separate standard transformer layers, whose inputs are obtained through the selective concatenation of different branches in the first combination block.

[0114] Similarly, in the third combination block, the 3-combination branch is also implemented using a standard transformer layer. Here, the transformer input corresponds to the cascaded output of three 2-combination branches. The output of this block serves as the input to the classification head, which consists of a dense layer with 128 neurons, batch normalization, and a ReLU activation unit, followed by a dense layer with 143 neurons and a softmax activation function. This output head is referred to as the people classification head.

[0115] like Figure 4A As shown, three output heads are used to train the three 1-combination branches in the first combination block. A visible single-peak model is constructed based on the visible branch and output head 401; a thermal imaging single-peak model is constructed based on the thermal imaging branch and output head 402; and an audio single-peak model is constructed based on the audio branch and output head 403. Specifically, the visible single-peak model and the thermal imaging single-peak model both include a standard embedding layer, a standard transformer layer, and a person classification head; the audio single-peak model includes an audio convolutional layer, a standard transformer layer, and a person classification head.

[0116] In the first training phase, three single-peak models were trained respectively.

[0117] like Figure 4BAs shown, three output heads are used to train the three 2-combination branches in the second combination block. The 2-combination branches represent audio-visual, audio-thermal, and visible-thermal imaging data. Based on the visible branch, thermal imaging branch, visible-thermal imaging branch, and output head 404, a visible-thermal imaging bimodal model is constructed; based on the audio branch, visible branch, audio-visual branch, and output head 405, an audio-visual bimodal model is constructed; based on the audio branch, thermal imaging branch, audio-thermal imaging branch, and output head 406, an audio-thermal imaging bimodal model is constructed.

[0118] During the construction of the bimodal model, three trained unimodal models with their classification heads removed can be transferred into the bimodal model. Subsequently, each 2-combination branch, implemented by a standard transformer layer, accepts the output of a specific unimodal branch. The audio-visual branch accepts the outputs of the audio and visible unimodal branches, the audio-thermal imaging branch accepts the outputs of the audio and thermal imaging unimodal branches, and the visible-thermal imaging branch accepts the outputs of the visible and thermal imaging unimodal branches. Each 2-combination branch has a corresponding person classification head.

[0119] In the second training phase, three bimodal models were trained respectively.

[0120] like Figure 4C As shown, a 3-combination branch in the third combination block is trained using one output head. A trimodal model is constructed based on the visible branch, thermal imaging branch, audio branch, visible-thermal imaging branch, audio-visual branch, audio-thermal imaging branch, audio-visual-thermal imaging branch, and output head 407.

[0121] In constructing the trimodal model, the pre-trained bimodal model is first converted into a trimodal model with the three classifiers removed. Next, a 3-combination branch, implemented using standard transformer layers, accepts the outputs of all bimodal branches. Finally, a single-person classifier is used to train the 3-combination branch, as well as the transitioned 1-combination and 2-combination branches.

[0122] In the third training phase, a trimodal model is trained. The trained trimodal model can then be used for face classification.

[0123] Next, combine Figures 5-6B The training process is illustrated through a specific example of a two-modal model.

[0124] Figure 5 This is a schematic diagram of the structure of a dual-modal model provided in one embodiment of the present disclosure. Figure 6A This is a schematic diagram of the first training phase of a bimodal model provided in an embodiment of the present disclosure; Figure 6BThis is a schematic diagram of the second training phase of a bimodal model provided in an embodiment of the present disclosure.

[0125] In one example, the bimodal model is formulated for the following classification task: audio-video sentiment classification.

[0126] Using the audio-visible image dataset and Sample, in which Indicates audio. This indicates a video with j frames. Indicates emotional tags, Represents an emotion sample. Audio input. The video input is represented by a log-Mel spectrum image of size 128 × 889. It is represented by three frames of size (64×64×3).

[0127] like Figure 5 As shown, the bimodal model includes: a 1-combination branch, a 2-combination branch, and an emotion output head.

[0128] The first combination block includes two 1-combination branches, meaning it comprises two unimodal branch networks: a visible branch and an audio branch. The second combination block includes one 2-combination branch, meaning it comprises one multimodal branch network: an audio-visual branch network. The emotion output head can be a pre-defined classification head, used to classify the features of the received samples to obtain emotion recognition results, which include emotion labels.

[0129] In the visible branch of the first combined block, the visible sequence is provided as input to the standard embedding layer. In the audio branch, the audio spectrogram is provided as input to the audio convolutional layer. The outputs of these layers are then provided as input to a single standard transformer layer.

[0130] The second combination block is implemented using a standard transformer layer. The outputs from the two 1-combination branches are concatenated and used as input to the transformer. The output of the second block serves as input to the classification head, which consists of a dense layer with 128 neurons, batch normalization and ReLU activation units, followed by a dense layer with o neurons using the softmax activation function. The number of output neurons o varies depending on the dataset. This output head is called the sentiment classification head.

[0131] like Figure 6AAs shown, a visible single-peak model is constructed based on the visible branch and output head 601; an audio single-peak model is constructed based on the audio branch and output head 602. The visible single-peak model includes a standard embedding layer, a standard transformer layer, and a sentiment classification head; the audio single-peak model includes an audio convolutional layer, a standard transformer layer, and a sentiment classification head.

[0132] In the first training phase, two single-peak models are trained separately.

[0133] like Figure 6B As shown, an audio-visual bimodal model is constructed based on the audio branch, the visibility branch, the audio-visual branch, and output head 603. During the construction of the bimodal model, the classification heads of the two trained unimodal models are removed and transferred to the bimodal model. Subsequently, the audio-visual branch, implemented using a standard transformer layer, accepts the outputs of the audio and visibility unimodal branches, and the output of the transformer layer serves as the input to the sentiment classification head.

[0134] Since the technical concept is the same, the description in this embodiment is relatively simple. For the relevant parts, please refer to the corresponding descriptions of the method embodiments provided above.

[0135] Figure 7 This is a schematic diagram of the structure of a training device for a multimodal model provided in an embodiment of the present disclosure, as shown below. Figure 7 As shown, the device includes: The sample construction unit 701 is used to construct multimodal samples based on image material data and / or audio material data; The first training unit 702 is used to input the multimodal samples into the first combination block for iterative training in the first training phase; the first combination block includes N single-modal branch networks for feature extraction, where N is the number of modal types of the multimodal samples; Model building unit 703 is used to construct a multimodal model for the i-th training stage based on the i-th combination block and the first i-1 training combination blocks; where i is greater than 1; the i-th combination block includes at least one multimodal branch network, each of the multimodal branch networks is used to extract features from the received sample features, and the sample features correspond to i modalities; when i is greater than 2, the first i-1 training combination blocks are obtained based on the multimodal model for the (i-1)-th training stage after training; The second training unit 704 is used to input the multimodal samples into the multimodal model of the i-th training stage for iterative training until the multimodal model of the N-th training stage is obtained, and to determine the multimodal model of the N-th training stage as the target multimodal model.

[0136] Optionally, the multimodal samples include complete training samples and missing training samples; when constructing multimodal samples based on image material data and / or audio material data, the sample construction unit 701 performs the following steps: Based on image and / or audio source data, a first training sample and a second training sample are constructed; the first training sample includes sub-samples of N modalities, and the second training sample includes sub-samples of M modalities, where N is greater than M; The first training sample is determined as the complete training sample; The missing training sample is obtained by filling the second training sample with preset filler subsamples.

[0137] Optionally, if i equals 2, when the model building unit 703 constructs the multimodal model for the i-th training stage based on the i-th combination block and the first i-1 training combination blocks, it performs the following steps: Based on the two modes corresponding to the multimodal branch network in the second combination block and the one mode corresponding to the single-modal branch network in the first combination block after training, the first connection relationship between the multimodal branch network and the single-modal branch network is determined. Based on the second combined block, the trained first combined block, and the first connection relationship, a multimodal model for the second training phase is constructed.

[0138] Optionally, if i is greater than 2, when the model building unit 703 constructs the multimodal model for the i-th training stage based on the i-th combination block and the first i-1 training combination blocks, it performs the following steps: Based on the i-th modalities corresponding to the multimodal branch network in the i-th combination block and the i-1-th modalities corresponding to the multimodal branch network in the (i-1)-th combination block after training, a second connection relationship is determined between the multimodal branch network in the i-th combination block and the multimodal branch network in the (i-1)-th combination block. Based on the i-th combined block, the first i-1 combined blocks after training, and the second connection relationship, the multimodal model of the i-th training stage is constructed.

[0139] Optionally, the multimodal samples include visible sub-samples, thermal imaging sub-samples, and audio sub-samples; the first combination block includes: a visible branch network, a thermal imaging branch network, and an audio branch network; when the first training unit 702 inputs the multimodal samples into the first combination block for iterative training, it performs the following steps: The visible subsamples are input into the visible branch network for iterative training. The thermal imaging sub-samples are input into the thermal imaging branch network for iterative training. The audio subsamples are input into the audio branch network for iterative training.

[0140] Optionally, the training states of the multimodal model further include: A data recording unit is used to record the training result data of each training stage; Based on the training results data, a variant multimodal model associated with the target multimodal model is constructed.

[0141] Optionally, the target multimodal model is a multimodal face recognition model, and the preset filler sub-samples in the missing training samples are marked with unknown face labels.

[0142] In this embodiment, the single-modal branch networks in the first combination block are trained in the first training phase. Then, the trained old combination blocks are reused, and the multimodal model built based on the old and new combination blocks is put into the model training of the next training phase. The new combination block includes a multimodal branch network, and the number of modal types corresponding to the sample features received by the multimodal branch network matches the next training phase. This makes the final target multimodal model integrate the combination blocks that have been fully trained in each training phase. Therefore, in the scenario of random modality missing, the target multimodal model does not need to rely on invalid value filling. It can make full use of the low-order modal branches in each combination block that match the input data after the modality is missing to complete the inference, so that the accuracy of the multimodal model remains stable.

[0143] The multimodal model training device provided in one embodiment of this disclosure can implement the various processes in the foregoing method embodiments and achieve the same functions and effects, which will not be repeated here.

[0144] Furthermore, one embodiment of this disclosure also provides an electronic device, Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as... Figure 8 As shown, the device includes a memory 801, a processor 802, a bus 803, and a communication interface 804. The memory 801, processor 802, and communication interface 804 communicate via the bus 803. The communication interface 804 may include input / output interfaces, including but not limited to a keyboard, mouse, monitor, microphone, and loudspeaker.

[0145] Figure 8 In the processor 802, the memory 801 stores computer-executable instructions that can run on the processor 802. When the processor 802 executes the computer-executable instructions, the following process is implemented: Construct multimodal samples based on image and / or audio data; In the first training phase, the multimodal samples are input into the first combination block for iterative training; the first combination block includes N single-modal branch networks for feature extraction, where N is the number of modal types of the multimodal samples; Based on the i-th combined block and the first i-1 combined blocks after training, a multimodal model for the i-th training stage is constructed; where i is greater than 1; the i-th combined block includes at least one multimodal branch network, each of which is used to extract features from the received sample features, and the sample features correspond to i modalities; when i is greater than 2, the first i-1 combined blocks after training are obtained based on the multimodal model for the (i-1)-th training stage after training. In the i-th training phase, the multimodal samples are input into the multimodal model of the i-th training phase for iterative training until the multimodal model of the N-th training phase is obtained, and the multimodal model of the N-th training phase is determined as the target multimodal model.

[0146] In this embodiment, the single-modal branch networks in the first combination block are trained in the first training phase. Then, the trained old combination blocks are reused, and the multimodal model built based on the old and new combination blocks is put into the model training of the next training phase. The new combination block includes a multimodal branch network, and the number of modal types corresponding to the sample features received by the multimodal branch network matches the next training phase. This makes the final target multimodal model integrate the combination blocks that have been fully trained in each training phase. Therefore, in the scenario of random modality missing, the target multimodal model does not need to rely on invalid value filling. It can make full use of the low-order modal branches in each combination block that match the input data after the modality is missing to complete the inference, so that the accuracy of the multimodal model remains stable.

[0147] An electronic device provided in one embodiment of this disclosure can implement the various processes in the foregoing method embodiments and achieve the same functions and effects, which will not be repeated here.

[0148] Another embodiment of this disclosure also provides a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the following process: Construct multimodal samples based on image and / or audio data; In the first training phase, the multimodal samples are input into the first combination block for iterative training; the first combination block includes N single-modal branch networks for feature extraction, where N is the number of modal types of the multimodal samples; Based on the i-th combined block and the first i-1 combined blocks after training, a multimodal model for the i-th training stage is constructed; where i is greater than 1; the i-th combined block includes at least one multimodal branch network, each of which is used to extract features from the received sample features, and the sample features correspond to i modalities; when i is greater than 2, the first i-1 combined blocks after training are obtained based on the multimodal model for the (i-1)-th training stage after training. In the i-th training phase, the multimodal samples are input into the multimodal model of the i-th training phase for iterative training until the multimodal model of the N-th training phase is obtained, and the multimodal model of the N-th training phase is determined as the target multimodal model.

[0149] In this embodiment, the single-modal branch networks in the first combination block are trained in the first training phase. Then, the trained old combination blocks are reused, and the multimodal model built based on the old and new combination blocks is put into the model training of the next training phase. The new combination block includes a multimodal branch network, and the number of modal types corresponding to the sample features received by the multimodal branch network matches the next training phase. This makes the final target multimodal model integrate the combination blocks that have been fully trained in each training phase. Therefore, in the scenario of random modality missing, the target multimodal model does not need to rely on invalid value filling. It can make full use of the low-order modal branches in each combination block that match the input data after the modality is missing to complete the inference, so that the accuracy of the multimodal model remains stable.

[0150] The computer-readable storage medium includes read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc.

[0151] The computer-readable storage medium provided in one embodiment of this disclosure can implement the various processes in the foregoing method embodiments and achieve the same functions and effects, which will not be repeated here.

[0152] Another embodiment of this disclosure also provides a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the following process: Construct multimodal samples based on image and / or audio data; In the first training phase, the multimodal samples are input into the first combination block for iterative training; the first combination block includes N single-modal branch networks for feature extraction, where N is the number of modal types of the multimodal samples; Based on the i-th combined block and the first i-1 combined blocks after training, a multimodal model for the i-th training stage is constructed; where i is greater than 1; the i-th combined block includes at least one multimodal branch network, each of which is used to extract features from the received sample features, and the sample features correspond to i modalities; when i is greater than 2, the first i-1 combined blocks after training are obtained based on the multimodal model for the (i-1)-th training stage after training. In the i-th training phase, the multimodal samples are input into the multimodal model of the i-th training phase for iterative training until the multimodal model of the N-th training phase is obtained, and the multimodal model of the N-th training phase is determined as the target multimodal model.

[0153] In this embodiment, the single-modal branch networks in the first combination block are trained in the first training phase. Then, the trained old combination blocks are reused, and the multimodal model built based on the old and new combination blocks is put into the model training of the next training phase. The new combination block includes a multimodal branch network, and the number of modal types corresponding to the sample features received by the multimodal branch network matches the next training phase. This makes the final target multimodal model integrate the combination blocks that have been fully trained in each training phase. Therefore, in the scenario of random modality missing, the target multimodal model does not need to rely on invalid value filling. It can make full use of the low-order modal branches in each combination block that match the input data after the modality is missing to complete the inference, so that the accuracy of the multimodal model remains stable.

[0154] The computer program product in this disclosure embodiment can implement the various processes of the above-described multimodal model training method embodiment and achieve the same effect and function, which will not be repeated here.

[0155] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0159] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0160] Memory may include non-persistent storage in computer-readable storage media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable storage media.

[0161] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0162] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0163] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A training method for a multimodal model, characterized in that, include: Construct multimodal samples based on image and / or audio data; In the first training phase, the multimodal samples are input into the first combination block for iterative training; the first combination block includes N single-modal branch networks for feature extraction, where N is the number of modal types of the multimodal samples; Based on the i-th combined block and the first i-1 combined blocks after training, a multimodal model for the i-th training stage is constructed; where i is greater than 1; the i-th combined block includes at least one multimodal branch network, each of which is used to extract features from the received sample features, and the sample features correspond to i modalities; when i is greater than 2, the first i-1 combined blocks after training are obtained based on the multimodal model for the (i-1)-th training stage after training. In the i-th training phase, the multimodal samples are input into the multimodal model of the i-th training phase for iterative training until the multimodal model of the N-th training phase is obtained, and the multimodal model of the N-th training phase is determined as the target multimodal model.

2. The method according to claim 1, characterized in that, The multimodal samples include complete training samples and missing training samples; The construction of multimodal samples based on image and / or audio data includes: Based on image and / or audio source data, a first training sample and a second training sample are constructed; the first training sample includes sub-samples of N modalities, and the second training sample includes sub-samples of M modalities, where N is greater than M; The first training sample is determined as the complete training sample; The missing training sample is obtained by filling the second training sample with preset filler subsamples.

3. The method according to claim 1, characterized in that, If i equals 2, the construction of the multimodal model for the i-th training stage based on the i-th combined block and the first i-1 combined blocks after training includes: Based on the two modes corresponding to the multimodal branch network in the second combination block and the one mode corresponding to the single-modal branch network in the first combination block after training, the first connection relationship between the multimodal branch network and the single-modal branch network is determined. Based on the second combined block, the trained first combined block, and the first connection relationship, a multimodal model for the second training phase is constructed.

4. The method according to claim 1, characterized in that, If i is greater than 2, the construction of the multimodal model for the i-th training stage based on the i-th combined block and the first i-1 combined blocks after training includes: Based on the i-th modalities corresponding to the multimodal branch network in the i-th combination block and the i-1-th modalities corresponding to the multimodal branch network in the (i-1)-th combination block after training, a second connection relationship is determined between the multimodal branch network in the i-th combination block and the multimodal branch network in the (i-1)-th combination block. Based on the i-th combined block, the first i-1 combined blocks after training, and the second connection relationship, the multimodal model of the i-th training stage is constructed.

5. The method according to claim 1, characterized in that, The multimodal samples include visible sub-samples, thermal imaging sub-samples, and audio sub-samples; the first combination block includes: a visible branch network, a thermal imaging branch network, and an audio branch network; the step of inputting the multimodal samples into the first combination block for iterative training includes: The visible subsamples are input into the visible branch network for iterative training. The thermal imaging sub-samples are input into the thermal imaging branch network for iterative training. The audio subsamples are input into the audio branch network for iterative training.

6. The method according to claim 1, characterized in that, The method further includes: Record the training results data for each of the aforementioned training stages; Based on the training results data, a variant multimodal model associated with the target multimodal model is constructed.

7. The method according to claim 2, characterized in that, The target multimodal model is a multimodal face recognition model, and the preset filler sub-samples in the missing training samples are marked with unknown face labels.

8. A training device for a multimodal model, characterized in that, include: The sample construction unit is used to construct multimodal samples based on image and / or audio data. The first training unit is used to input the multimodal samples into the first combination block for iterative training in the first training phase; the first combination block includes N single-modal branch networks for feature extraction, where N is the number of modal types of the multimodal samples; A model building unit is used to construct a multimodal model for the i-th training stage based on the i-th combination block and the first i-1 training combination blocks; where i is greater than 1; the i-th combination block includes at least one multimodal branch network, each of the multimodal branch networks is used to extract features from the received sample features, and the sample features correspond to i modalities; when i is greater than 2, the first i-1 training combination blocks are obtained based on the multimodal model for the (i-1)-th training stage. The second training unit is used to input the multimodal samples into the multimodal model of the i-th training stage for iterative training until the multimodal model of the N-th training stage is obtained, and the multimodal model of the N-th training stage is determined as the target multimodal model.

9. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-executable instructions that, when executed on the processor, enable the implementation of the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, enable the implementation of the method described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images