Label constraint based multi-modal classification model training method and device

Abstract

The application discloses a kind of based on label constraint's multi-modal classification model training method and device, the method includes: determining the training data of target modality for training model and corresponding data label;The training data is input into training to convergence data classification model, obtain the training data corresponding training data feature;The data label is input into the label classification model trained, obtain the label feature corresponding to the data label;The training data and the data label are input into the data classification model and are trained, in training, according to target loss function value, the model parameter of the data classification model is optimized until convergence, obtains the data classification model trained;The target loss function value includes the feature difference degree between the training data feature and the label feature.It can be seen that the present application can make the feature extraction of model more label distinguishability, and then make the prediction effect of model better.

Description

Technical Field

[0001] This invention relates to the field of algorithm model training technology, and in particular to a method and apparatus for training a multimodal classification model based on label constraints. Background Technology

[0002] With the development of algorithm technology, more and more enterprises are beginning to use algorithmic models to perform data prediction tasks related to data classification, such as predicting the associated category or label for data of a specific modality. This type of task requires the algorithmic model to fully extract and process the features of the data. However, current technologies do not consider the difference between the introduced labels and the extracted features when training such models. Therefore, they cannot effectively improve the discriminative power of the extracted features and the correlation between labels during training, resulting in poor training performance. Clearly, current technologies have shortcomings that urgently need to be addressed. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method and apparatus for training and determining a multimodal classification model based on label constraints, which can make the feature extraction of the model more label-discriminative and improve the prediction effect of the model.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses a method for training a multimodal classification model based on label constraints, the method comprising:

[0005] Determine the training data and corresponding data labels for the target modality used to train the model;

[0006] The training data is input into a data classification model that has been trained to convergence to obtain the training data features corresponding to the training data; the data classification model is used to extract the features of the target modality data.

[0007] The data labels are input into a trained label classification model to obtain the label features corresponding to the data labels.

[0008] The training data and the data labels are input into the data classification model for training. During training, the model parameters of the data classification model are optimized according to the target loss function value until convergence, resulting in the trained data classification model. The target loss function value includes the feature difference between the training data features and the label features.

[0009] As an optional implementation, in the first aspect of the present invention, the target modality includes at least one of an audio modality, an image modality, and a text modality; and / or, the data classification model includes at least one of an audio classification model, an image classification model, and a text classification model.

[0010] As an optional implementation, in the first aspect of the present invention, the step of inputting the data label into a trained label classification model to obtain the label features corresponding to the data label includes:

[0011] Generate tag text that includes the data tags based on the data tags;

[0012] The labeled text is input into the trained label classification model to obtain the label features corresponding to the labeled text; the label classification model is trained using a training dataset that includes multiple training labeled texts and corresponding training data labels.

[0013] As an optional implementation, in a first aspect of the present invention, the target loss function value includes the feature difference degree and the label difference degree between the predicted label output by the data classification model and the data label.

[0014] As an optional implementation, in the first aspect of the present invention, the target loss function value is a weighted sum of the feature difference and the label difference; the weights of the feature difference or the label difference are used to reduce the size difference between their data values.

[0015] As an optional implementation, in the first aspect of the present invention, the label dissimilarity is a cross-entropy loss function; and / or, the feature dissimilarity is a KL divergence.

[0016] As an optional implementation, in the first aspect of the present invention, the label classification model is a classification model based on a Transformer network.

[0017] As an optional implementation, in the first aspect of the present invention, the audio classification model includes at least one of the SpeechTransformer model and the Conformer model, and / or the image classification model includes at least one of the CNN model, the ViT model and the CoTNet model, and / or the text classification model includes at least one of the BERT model, the XLNet model and the RoBERTa model.

[0018] A second aspect of this invention discloses a label-constrained multimodal classification model training device, the device comprising:

[0019] The data determination module is used to determine the training data and corresponding data labels for the target modality used to train the model;

[0020] The feature extraction module is used to input the training data into a data classification model that has been trained to convergence, and obtain the training data features corresponding to the training data; the data classification model is used to extract the features of the target modality data;

[0021] The label processing module is used to input the data labels into the trained label classification model to obtain the label features corresponding to the data labels;

[0022] The model training module is used to input the training data and the data labels into the data classification model for training. During training, the model parameters of the data classification model are optimized according to the target loss function value until convergence, so as to obtain the trained data classification model. The target loss function value includes the feature difference between the training data features and the label features.

[0023] As an optional implementation, in a second aspect of the invention, the target modality includes at least one of an audio modality, an image modality, and a text modality; and / or, the data classification model includes at least one of an audio classification model, an image classification model, and a text classification model.

[0024] As an optional implementation, in a second aspect of the present invention, the tag processing module includes:

[0025] A text generation unit is used to generate tag text including the data tags based on the data tags;

[0026] The feature extraction unit is used to input the labeled text into the trained label classification model to obtain the label features corresponding to the labeled text; the label classification model is trained using a training dataset that includes multiple training labeled texts and corresponding training data labels.

[0027] As an optional implementation, in a second aspect of the invention, the target loss function value includes the feature difference degree and the label difference degree between the predicted label output by the data classification model and the data label.

[0028] As an optional implementation, in a second aspect of the invention, the target loss function value is a weighted sum of the feature difference and the label difference; the weights of the feature difference or the label difference are used to reduce the size difference between their data values.

[0029] As an optional implementation, in a second aspect of the invention, the label dissimilarity is a cross-entropy loss function; and / or, the feature dissimilarity is a KL divergence.

[0030] As an optional implementation, in a second aspect of the present invention, the label classification model is a classification model based on a Transformer network.

[0031] As an optional implementation, in a second aspect of the invention, the audio classification model includes at least one of the SpeechTransformer model and the Conformer model, and / or the image classification model includes at least one of the CNN model, the ViT model, and the CoTNet model, and / or the text classification model includes at least one of the BERT model, the XLNet model, and the RoBERTa model.

[0032] A third aspect of the present invention discloses another label-constrained multimodal classification model training device, the device comprising:

[0033] Memory containing executable program code;

[0034] A processor coupled to the memory;

[0035] The processor calls the executable program code stored in the memory to execute some or all of the steps in the label-constrained multimodal classification model training method disclosed in the first aspect of the present invention.

[0036] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in the label-constrained multimodal classification model training method disclosed in the first aspect of the present invention.

[0037] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0038] This invention can use a training convergent classification model to extract features from training data, and then use a trained label classification model to extract label features. In subsequent training, the difference between the two is calculated so that the trained model can effectively extract data features with label discrimination. This effectively establishes the correlation between the extracted features of the model and the labels, which can make the model's feature extraction more label discriminative and improve the model's prediction performance. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating a label-constrained multimodal classification model training method disclosed in an embodiment of the present invention.

[0041] Figure 2 This is a flowchart illustrating another label-constrained multimodal classification model training method disclosed in an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the structure of a label-constrained multimodal classification model training device disclosed in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of another label-constrained multimodal classification model training device disclosed in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the structure of another label-constrained multimodal classification model training device disclosed in the embodiments of the present invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0047] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0048] This invention discloses a method and apparatus for training a multimodal classification model based on label constraints. It utilizes a converged classification model to extract features from training data, then uses a trained label classification model to extract label features. In subsequent training, the difference between the two models is calculated to ensure the trained model effectively extracts data features with label discriminative power. This results in more label-discriminative feature extraction and improved prediction performance. Detailed explanations follow.

[0049] Example 1

[0050] Please see Figure 1 , Figure 1 This is a flowchart illustrating a label-constrained multimodal classification model training method disclosed in an embodiment of the present invention. Figure 1 The described method is applied in a data processing device, which can be a corresponding processing terminal, processing equipment, or processing server. The server can be a local server or a cloud server; this embodiment of the invention does not impose any limitations. Figure 1 As shown, the label-constrained multimodal classification model training method can include the following operations:

[0051] 101. Determine the training data and corresponding data labels for the target modality used to train the model.

[0052] In embodiments of the present invention, the target modality may include at least one of an audio modality, an image modality, and a text modality. Correspondingly, the training data may also include at least one of audio data, image data, and text data. Optionally, when the training data is image data, it may also be image data of a specific frame from video data, thereby enabling the method of the present invention to be used for video data processing.

[0053] Optionally, data labels can be used to indicate the data category, data content, or importance of specific data in the training data. This invention does not limit this; the specific labels can be determined according to the classification purpose or classification requirements of the data classification model of this method in the actual application scenario.

[0054] Optionally, the training data may include one or more training data sets, and correspondingly, the data labels may also be one or more corresponding data labels, each corresponding to a training data set.

[0055] 102. Input the training data into the data classification model that has been trained to convergence, and obtain the training data features corresponding to the training data.

[0056] Specifically, the data classification model is used to extract features from the data of the target modality. Preferably, the data classification model may include at least one of an audio classification model, an image classification model, and a text classification model.

[0057] Optionally, the data classification model can be trained until convergence using training data and data labels in the pre-training steps, enabling it to accurately extract data features. Subsequently, the converged data classification model can be used to perform the data feature extraction function.

[0058] Optionally, when processing training data for audio modalities, the data classification model employs an audio classification model, which may include at least one of a Speech Transformer model and a Conformer model.

[0059] Optionally, when processing training data for image modalities, the data classification model may employ an image classification model, which may include at least one of a CNN model, a ViT model, and a CoTNet model.

[0060] Optionally, when processing training data for text modalities, the data classification model may employ a text classification model, which may include at least one of the BERT model, XLNet model, and RoBERTa model.

[0061] 103. Input the data labels into the trained label classification model to obtain the label features corresponding to the data labels.

[0062] Optionally, a label classification model that can extract label features can be pre-trained until it converges, and then used to extract label features.

[0063] 104. Input the training data and data labels into the data classification model for training. During training, optimize the model parameters of the data classification model according to the target loss function value until convergence, and obtain the trained data classification model.

[0064] Specifically, the target loss function value includes the feature difference between the training data features and the label features.

[0065] As can be seen, the method described in the embodiments of the present invention can use a training convergent classification model to extract features of training data, and then use a trained label classification model to extract label features. In subsequent training, the difference between the two is calculated so that the trained model can effectively extract data features with label discrimination. This makes the feature extraction of the model more label discriminative and improves the prediction effect of the model.

[0066] As an optional implementation, the target loss function value includes feature dissimilarity and label dissimilarity between the predicted label output by the data classification model and the data label.

[0067] Optionally, the label dissimilarity can be the cross-entropy loss function.

[0068] Optionally, the feature dissimilarity can be the KL divergence between the training data features and the label features.

[0069] As can be seen, through this optional implementation method, the target loss function value can be set to include feature difference and label difference, so that when training the data classification model, the classification prediction ability and feature extraction ability of the data classification model can be trained simultaneously according to the loss function. This can make the model's feature extraction more label-discriminative and improve the model's prediction performance.

[0070] As an optional implementation, the objective loss function is a weighted sum of feature dissimilarity and label dissimilarity. Specifically, the weights of feature dissimilarity or label dissimilarity are used to reduce the magnitude difference between their data values.

[0071] In one specific implementation, the total loss is equal to the sum of the first product and the second product, where the first product is the product of weight a and feature difference, and the second product is the product of weight b and label difference. During training, the model parameters are updated by backpropagating the gradient of the total loss function until the model converges and training is complete. Here, weights a and b are hyperparameters, and a = b = 1 can be set.

[0072] Preferably, since the total loss of the data classification model in this invention is the sum of multiple loss functions, the larger the output value of the loss function, the greater its impact on the model. Assuming the loss function calculating feature difference is 25 and the loss function calculating label difference is 1.5, the entire network will primarily use the loss function calculating feature difference for backpropagation. To amplify the effect of the loss function calculating label difference, it is usually multiplied by a weight 'a' less than 1, such as 0.1, making its output value 2.5. This reduces the difference between its output value and the loss function value calculating label difference. In practical applications, because different models have different loss functions, the values ​​of 'a' and 'b' can be adjusted based on the output value of the loss function to measure the importance of different loss functions.

[0073] As can be seen, by implementing this optional implementation method, the target loss function value can be set to the weighted sum of feature difference and label difference, and the weights corresponding to different differences can be adjusted. This allows the model to train classification prediction ability and feature extraction ability more evenly according to the loss function during training, which can make the model's feature extraction more label-discriminative and the model's prediction effect better.

[0074] Example 2

[0075] Please see Figure 2 , Figure 2 This is a flowchart illustrating another label-constrained multimodal classification model training method disclosed in an embodiment of the present invention. Figure 2 The described method is applied in a data processing device, which can be a corresponding processing terminal, processing equipment, or processing server. The server can be a local server or a cloud server; this embodiment of the invention does not impose any limitations. Figure 2 As shown, the label-constrained multimodal classification model training method can include the following operations:

[0076] 201. Determine the training data and corresponding data labels for the target modality used to train the model.

[0077] 202. Input the training data into the data classification model that has been trained to convergence, and obtain the training data features corresponding to the training data.

[0078] 203. Generate tag text that includes the data tags based on the data tags.

[0079] Optionally, label text including data labels can be generated based on a preset text template and data labels. Preferably, the text template can correspond to the modality of the training data, so the label text can be used to simultaneously indicate the content of the data labels and the modality information of the training data.

[0080] In one specific implementation, data labels can be constructed into a label text based on a text template. For example, when the data label is XXX, the label text for image modality training data can be constructed as: "This is an image of {XXX}", the label text for audio modality training data can be constructed as: "This is an audio of {XXX}", and the label text for text modality training data can be constructed as: "This is a text about {XXX}".

[0081] 204. Input the label text into the trained label classification model to obtain the label features corresponding to the label text.

[0082] Specifically, the label classification model is trained using a training dataset containing multiple training label texts and corresponding training data labels. In one specific implementation, a text data classification model for label classification is constructed. Its input data includes label text and data labels, such as the label text being "This is an audio clip of {sports and fitness}", and the corresponding data label being "sports and fitness". The input data is then fed into the text data classification model for training. Here, a classification model based on a Transformer network can be used because the data contains label information, so the classification accuracy of the model is 100%. The purpose of the trained label classification model is to act as a converter, receiving input labels and outputting the feature representation of the labels in the model.

[0083] Optionally, the label classification model can be an existing pre-trained text model, or when the data classification model in this invention is a data classification model for processing text modalities, the data classification model that has been trained to convergence can be directly used as the label classification model, as long as the vector dimension of the output label features and the training data features is consistent.

[0084] 205. Input the training data and data labels into the data classification model for training. During training, optimize the model parameters of the data classification model according to the target loss function value until convergence, and obtain the trained data classification model.

[0085] The specific technical details and explanations of technical terms for steps 201-202 and 205 above can be found in the descriptions of steps 101-102 and 104 in Implementation 1, and will not be repeated here.

[0086] As can be seen, the embodiments of the present invention can use the label classification model to accurately extract the features of the data labels, so that the difference between the feature and the training data features can be calculated in the subsequent training, so that the trained model can effectively extract data features with label discrimination, thereby making the feature extraction of the model more label discriminative and improving the prediction effect of the model.

[0087] Example 3

[0088] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a label-constrained multimodal classification model training device disclosed in an embodiment of the present invention. Figure 3 The described apparatus can be applied to a corresponding data processing device, which can be a corresponding processing terminal, processing equipment, or processing server. The server can be a local server or a cloud server; this embodiment of the invention does not limit the application. Figure 3 As shown, the device may include:

[0089] The data determination module 301 is used to determine the training data and corresponding data labels for the target modality used to train the model.

[0090] In embodiments of the present invention, the target modality may include at least one of an audio modality, an image modality, and a text modality. Correspondingly, the training data may also include at least one of audio data, image data, and text data. Optionally, when the training data is image data, it may also be image data of a specific frame from video data, thereby enabling the method of the present invention to be used for video data processing.

[0091] Optionally, data labels can be used to indicate the data category, data content, or importance of specific data in the training data. This invention does not limit this; the specific labels can be determined according to the classification purpose or classification requirements of the data classification model of this method in the actual application scenario.

[0092] Optionally, the training data may include one or more training data sets, and correspondingly, the data labels may also be one or more corresponding data labels, each corresponding to a training data set.

[0093] The feature extraction module 302 is used to input the training data into the data classification model that has been trained to convergence, and to obtain the training data features corresponding to the training data.

[0094] Specifically, the data classification model is used to extract features from the data of the target modality. Preferably, the data classification model may include at least one of an audio classification model, an image classification model, and a text classification model.

[0095] Optionally, the data classification model can be trained until convergence using training data and data labels in the pre-training steps, enabling it to accurately extract data features. Subsequently, the converged data classification model can be used to perform the data feature extraction function.

[0096] Optionally, when processing training data for audio modalities, the data classification model employs an audio classification model, which may include at least one of a Speech Transformer model and a Conformer model.

[0097] Optionally, when processing training data for image modalities, the data classification model may employ an image classification model, which may include at least one of a CNN model, a ViT model, and a CoTNet model.

[0098] Optionally, when processing training data for text modalities, the data classification model may employ a text classification model, which may include at least one of the BERT model, XLNet model, and RoBERTa model.

[0099] The label processing module 303 is used to input data labels into the trained label classification model to obtain the label features corresponding to the data labels.

[0100] Optionally, a label classification model that can extract label features can be pre-trained until it converges, and then used to extract label features.

[0101] The model training module 304 is used to input training data and data labels into the data classification model for training. During training, the model parameters of the data classification model are optimized according to the target loss function value until convergence, and the trained data classification model is obtained.

[0102] Specifically, the target loss function value includes the feature difference between the training data features and the label features.

[0103] As can be seen, the apparatus described in the embodiments of the present invention can use a training convergent classification model to extract features of training data, then use a trained label classification model to extract label features, and calculate the difference between the two in subsequent training so that the trained model can effectively extract data features with label discrimination. This makes the feature extraction of the model more label discriminative and improves the prediction effect of the model.

[0104] As an optional implementation, the target loss function value includes feature dissimilarity and label dissimilarity between the predicted label output by the data classification model and the data label.

[0105] Optionally, the label dissimilarity can be the cross-entropy loss function.

[0106] Optionally, the feature dissimilarity can be the KL divergence between the training data features and the label features.

[0107] As can be seen, through this optional implementation method, the target loss function value can be set to include feature difference and label difference, so that when training the data classification model, the classification prediction ability and feature extraction ability of the data classification model can be trained simultaneously according to the loss function. This can make the model's feature extraction more label-discriminative and improve the model's prediction performance.

[0108] As an optional implementation, the objective loss function is a weighted sum of feature dissimilarity and label dissimilarity. Specifically, the weights of feature dissimilarity or label dissimilarity are used to reduce the magnitude difference between their data values.

[0109] In one specific implementation, the total loss is equal to the sum of the first product and the second product, where the first product is the product of weight a and feature difference, and the second product is the product of weight b and label difference. During training, the model parameters are updated by backpropagating the gradient of the total loss function until the model converges and training is complete. Here, weights a and b are hyperparameters, and a = b = 1 can be set.

[0110] Preferably, since the total loss of the data classification model in this invention is the sum of multiple loss functions, the larger the output value of the loss function, the greater its impact on the model. Assuming the loss function calculating feature difference is 25 and the loss function calculating label difference is 1.5, the entire network will primarily use the loss function calculating feature difference for backpropagation. To amplify the effect of the loss function calculating label difference, it is usually multiplied by a weight 'a' less than 1, such as 0.1, making its output value 2.5. This reduces the difference between its output value and the loss function value calculating label difference. In practical applications, because different models have different loss functions, the values ​​of 'a' and 'b' can be adjusted based on the output value of the loss function to measure the importance of different loss functions.

[0111] As can be seen, by implementing this optional implementation method, the target loss function value can be set to the weighted sum of feature difference and label difference, and the weights corresponding to different differences can be adjusted. This allows the model to train classification prediction ability and feature extraction ability more evenly according to the loss function during training, which can make the model's feature extraction more label-discriminative and the model's prediction effect better.

[0112] As an optional implementation method, such as Figure 4 As shown, the label processing module 303 includes:

[0113] The text generation unit 3031 is used to generate tag text containing data tags based on the data tags.

[0114] The feature extraction unit 3032 is used to input the label text into the trained label classification model to obtain the label features corresponding to the label text.

[0115] Optionally, label text including data labels can be generated based on a preset text template and data labels. Preferably, the text template can correspond to the modality of the training data, so the label text can be used to simultaneously indicate the content of the data labels and the modality information of the training data.

[0116] In one specific implementation, data labels can be constructed into a label text based on a text template. For example, when the data label is XXX, the label text for image modality training data can be constructed as: "This is an image of {XXX}", the label text for audio modality training data can be constructed as: "This is an audio of {XXX}", and the label text for text modality training data can be constructed as: "This is a text about {XXX}".

[0117] Specifically, the label classification model is trained using a training dataset containing multiple training label texts and corresponding training data labels. In one specific implementation, a text data classification model for label classification is constructed. Its input data includes label text and data labels, such as the label text being "This is an audio clip of {sports and fitness}", and the corresponding data label being "sports and fitness". The input data is then fed into the text data classification model for training. Here, a classification model based on a Transformer network can be used because the data contains label information, so the classification accuracy of the model is 100%. The purpose of the trained label classification model is to act as a converter, receiving input labels and outputting the feature representation of the labels in the model.

[0118] Optionally, the label classification model can be an existing pre-trained text model, or when the data classification model in this invention is a data classification model for processing text modalities, the data classification model that has been trained to convergence can be directly used as the label classification model, as long as the vector dimension of the output label features and the training data features is consistent.

[0119] As can be seen, this optional implementation can use the label classification model to accurately extract the features of the data labels, so that the difference between the feature and the training data features can be calculated in the subsequent training, so that the trained model can effectively extract data features with label discrimination, thereby making the model's feature extraction more label discriminative and improving the model's prediction performance.

[0120] Example 4

[0121] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of another label-constrained multimodal classification model training device disclosed in an embodiment of the present invention. Figure 5 As shown, the device may include:

[0122] Memory 401 storing executable program code;

[0123] Processor 402 coupled to memory 401;

[0124] The processor 402 calls the executable program code stored in the memory 401 to execute some or all of the steps in the label-constrained multimodal classification model training method disclosed in Embodiment 1 or Embodiment 2 of the present invention.

[0125] Example 5

[0126] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute some or all of the steps in the label-constrained multimodal classification model training method disclosed in Embodiment 1 or Embodiment 2 of this invention.

[0127] The foregoing has described specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily have to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0128] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer-readable storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0129] The apparatus, device, non-volatile computer-readable storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and non-volatile computer storage medium will not be repeated here.

[0130] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0131] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0132] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0133] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0134] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0139] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0140] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0141] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0142] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0143] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0144] Finally, it should be noted that the label-constrained multimodal classification model training method and apparatus disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a multimodal classification model based on label constraints, characterized in that, The method includes: The training data and corresponding data labels for the target modality used to train the model are determined; the target modality includes at least one of audio modality, image modality, and text modality; the training data includes at least one of audio data, image data, and text data; the data labels include at least one of the data labels corresponding to the audio data, the data labels corresponding to the image data, and the data labels corresponding to the text data. The training data is input into a data classification model trained to convergence to obtain training data features corresponding to the training data; the data classification model is used to extract features of the target modality data; the data classification model includes at least one of an audio classification model, an image classification model, and a text classification model; the audio classification model includes at least one of a SpeechTransformer model and a Conformer model; the image classification model includes at least one of a CNN model, a ViT model, and a CoTNet model; the text classification model includes at least one of a BERT model, an XLNet model, and a RoBERTa model. The data labels are input into a trained label classification model to obtain the label features corresponding to the data labels. The training data and the data labels are input into the data classification model for training. During training, the model parameters of the data classification model are optimized according to the target loss function value until convergence, resulting in the trained data classification model. The target loss function value includes the feature difference between the training data features and the label features, as well as the label difference between the predicted label output by the data classification model and the data label. The trained data classification model can extract data for the target modality and data features with label discriminative power.

2. The multimodal classification model training method based on label constraints according to claim 1, characterized in that, The step of inputting the data labels into a trained label classification model to obtain the label features corresponding to the data labels includes: Generate tag text that includes the data tags based on the data tags; The labeled text is input into the trained label classification model to obtain the label features corresponding to the labeled text; the label classification model is trained using a training dataset that includes multiple training labeled texts and corresponding training data labels.

3. The multimodal classification model training method based on label constraints according to claim 1, characterized in that, The target loss function value is a weighted sum of the feature difference and the label difference; the weights of the feature difference or the label difference are used to reduce the size difference between their data values.

4. The multimodal classification model training method based on label constraints according to claim 1, characterized in that, The label dissimilarity is the cross-entropy loss function; and / or, the feature dissimilarity is the KL divergence.

5. The multimodal classification model training method based on label constraints according to claim 1, characterized in that, The label classification model is a classification model based on the Transformer network.

6. A training device for a multimodal classification model based on label constraints, characterized in that, The device includes: A data determination module is used to determine the training data and corresponding data labels for the target modality used to train the model; the target modality includes at least one of audio modality, image modality, and text modality; the training data includes at least one of audio data, image data, and text data; the data labels include at least one of the data labels corresponding to the audio data, the data labels corresponding to the image data, and the data labels corresponding to the text data. The feature extraction module is used to input the training data into a data classification model trained to convergence, and obtain the training data features corresponding to the training data; the data classification model is used to extract the features of the target modality data; the data classification model includes at least one of an audio classification model, an image classification model, and a text classification model; the audio classification model includes at least one of a Speech Transformer model and a Conformer model; the image classification model includes at least one of a CNN model, a ViT model, and a CoTNet model; and the text classification model includes at least one of a BERT model, an XLNet model, and a RoBERTa model. The label processing module is used to input the data labels into the trained label classification model to obtain the label features corresponding to the data labels; The model training module is used to input the training data and the data labels into the data classification model for training. During training, the model parameters of the data classification model are optimized according to the target loss function value until convergence, resulting in the trained data classification model. The target loss function value includes the feature difference between the training data features and the label features, as well as the label difference between the predicted label output by the data classification model and the data label. The trained data classification model can extract data for the target modality and data features with label discriminative power.

7. A training device for a multimodal classification model based on label constraints, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the label-constrained multimodal classification model training method as described in any one of claims 1-5.

Images