A multi-modal text classification method, device, equipment and storage medium

Abstract

The application discloses a multi-modal text classification method and device, equipment and a storage medium, and relates to the technical field of information, which comprises the following steps: determining a target input text based on an initial image, an initial input text and a predefined label, and obtaining a target text feature according to the target input text and a target language model; obtaining an initial image feature based on the initial image and a deep residual network model, and screening a target image feature by using an attention mechanism, the initial image feature and the target text feature; obtaining an intermediate layer classless text feature based on the target language model, inputting the intermediate layer classless text feature and an intermediate layer input text feature into a variational autoencoder to obtain a mean value and a variance corresponding to the intermediate layer input text feature and the intermediate layer classless text feature; and obtaining a weight value based on the mean value and the variance, so as to classify the initial input text according to the weight value, the target image feature and the target text feature by using a classifier. The application optimizes the introduction mechanism of the image feature.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a multimodal text classification method, apparatus, device, and storage medium. Background Technology

[0002] Compared to traditional text classification algorithms, multimodal text classification algorithms typically use image modalities as a supplement to text modalities to provide more contextual information and help the model identify the text category. Existing multimodal text classification algorithms can be divided into three main categories: The first category consists of attention-based algorithms, which primarily extract features from images using a visual pre-trained model and then fuse image and text features using an attention mechanism. Finally, the fused features are used for classification. However, this type of method does not consider whether the image contributes to category recognition when fusing image features, making it susceptible to noisy images. The second category consists of image translation-based algorithms, which primarily use existing image caption generation tools to convert images into text modal captions or use image parsing tools to extract text modal image attributes from images. The extracted text modal information is then used as a supplement to assist text classification. However, the effectiveness of this type of method heavily depends on the performance of existing image parsing tools, making it highly susceptible to error propagation. The third category consists of pre-training-based algorithms, which primarily improve the learning of multimodal representations by designing various pre-training tasks specifically for multimodal text classification. However, existing algorithms, when incorporating image information, do not consider whether the image truly contributes to text classification; noisy images will adversely affect model predictions. Their effectiveness depends on the performance of existing image parsing tools and is greatly affected by error propagation. Pre-training tasks rely on human experience for design, resulting in weak generalization, and require substantial data and equipment resources. Therefore, reducing the impact of noisy images on text classification is a pressing issue that needs to be addressed. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a multimodal text classification method, apparatus, device, and storage medium, capable of dynamically introducing image features and reducing interference from noisy images. The specific solution is as follows:

[0004] Firstly, this application discloses a multimodal text classification method, including:

[0005] Obtain initial input text and corresponding initial image; determine target input text based on initial image, initial input text and predefined labels corresponding to pre-trained target language model; and obtain target text features corresponding to target input text based on target input text and pre-trained target language model.

[0006] Based on the initial image and the pre-trained deep residual network model, the corresponding initial image features are obtained, and the target image features that meet the preset conditions are selected using the preset attention mechanism, the initial image features, and the target text features.

[0007] Select a set of unclassed texts from a predetermined training set, obtain intermediate unclassed text features corresponding to the set of unclassed texts based on the pre-trained target language model, and input the intermediate unclassed text features and the intermediate input text features corresponding to the target input text into a variational autoencoder to obtain the mean and variance of the intermediate input text features and the intermediate unclassed text features.

[0008] Based on the mean and variance, weight values ​​representing the uncertainty of the text category are obtained, so that a preset classifier can be used to classify the initial input text according to the weight values, the target image features, and the target text features, and obtain the corresponding classification results.

[0009] Optionally, after obtaining the initial input text and the corresponding initial image, the method further includes:

[0010] The initial image is converted into a text modal title using an image title generation tool.

[0011] Optionally, determining the target input text based on the initial image, the initial input text, and predefined labels corresponding to the pre-trained target language model includes:

[0012] Insert a first predefined label corresponding to the pre-trained target language model at the beginning of the sentence of the initial input text, and insert a second predefined label corresponding to the pre-trained target language model at the end of the sentence of the initial input text to obtain the input text after insertion;

[0013] The text modal title is concatenated with the inserted input text to obtain the corresponding concatenated input text. The second predefined tag is then concatenated to the end of the concatenated input text to obtain the target input text.

[0014] Optionally, obtaining the corresponding initial image features based on the initial image and the pre-trained deep residual network model includes:

[0015] The initial image is adjusted based on a preset image size to obtain an adjusted image. The adjusted image is then input into the pre-trained deep residual network model to obtain the corresponding initial image features.

[0016] Optionally, the step of using a preset attention mechanism, the initial image features, and the target text features to filter target image features that meet preset conditions includes:

[0017] The initial image features are processed using a preset linear transformation based on the target dimension to obtain the processed image features;

[0018] The processed image features and the target text features are multiplied by a matrix to obtain a correlation matrix;

[0019] Image features after attention filtering are obtained using the objective function, the correlation matrix, and the initial image features;

[0020] Max pooling is used to extract target image features that meet preset conditions from the attention-based filtered image features.

[0021] Optionally, before inputting the intermediate layer classless text features and the intermediate layer input text features corresponding to the target input text into the variational autoencoder, the method further includes:

[0022] The intermediate layer's unclassed text features and the intermediate layer's input text features are processed using average pooling.

[0023] Optionally, obtaining the weight value representing the uncertainty of the text category based on the mean and the variance includes:

[0024] The first KL divergence between the intermediate layer input text feature distribution and the intermediate layer unclassified text feature distribution, and the second KL divergence between the intermediate layer unclassified text feature distribution and the intermediate layer input text feature distribution are determined using the mean and the variance, respectively.

[0025] The weight values ​​representing the uncertainty of the text category are obtained based on the first KL divergence and the second KL divergence.

[0026] Secondly, this application discloses a multimodal text classification device, comprising:

[0027] The target text feature acquisition module is used to acquire the initial input text and the corresponding initial image, determine the target input text based on the initial image, the initial input text and the predefined labels corresponding to the pre-trained target language model, and acquire the target text features corresponding to the target input text according to the target input text and the pre-trained target language model;

[0028] The target image feature filtering module is used to obtain corresponding initial image features based on the initial image and the pre-trained deep residual network model, and to filter target image features that meet preset conditions using a preset attention mechanism, the initial image features and the target text features.

[0029] The mean and variance acquisition module is used to select a set of unclassed texts from a predetermined training set, obtain intermediate unclassed text features corresponding to the set of unclassed texts based on the pre-trained target language model, and input the intermediate unclassed text features and the intermediate input text features corresponding to the target input text into the variational autoencoder to obtain the mean and variance corresponding to the intermediate input text features and the intermediate unclassed text features.

[0030] The text classification module is used to obtain weight values ​​representing the uncertainty of text category based on the mean and the variance, so as to use a preset classifier to classify the initial input text according to the weight values, the target image features and the target text features, and obtain the corresponding classification results.

[0031] Thirdly, this application discloses an electronic device, including:

[0032] Memory, used to store computer programs;

[0033] A processor for executing the computer program to implement the aforementioned multimodal text classification method.

[0034] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned multimodal text classification method.

[0035] In classifying text, this application first obtains initial input text and a corresponding initial image. Based on the initial image, the initial input text, and predefined labels corresponding to a pre-trained target language model, a target input text is determined. Then, target text features corresponding to the target input text are obtained based on the target input text and the pre-trained target language model. Next, initial image features are obtained based on the initial image and a pre-trained deep residual network model. A preset attention mechanism, the initial image features, and the target text features are used to filter target image features that meet preset conditions. A set of unclassified texts is selected from a pre-determined training set. Intermediate layer unclassified text features corresponding to the unclassified texts are obtained based on the pre-trained target language model. These intermediate layer unclassified text features and the intermediate layer input text features corresponding to the target input text are input into a variational autoencoder to obtain the mean and variance of the intermediate layer input text features and the intermediate layer unclassified text features. Finally, weight values ​​representing the uncertainty of text category are obtained based on the mean and variance. A preset classifier is then used to classify the initial input text according to the weight values, the target image features, and the target text features to obtain the corresponding classification results. As can be seen, this application first obtains corresponding text and image features by encoding the text and image, performs multimodal fusion based on an attention mechanism, then learns the uncertainty of the text category, and finally performs text classification based on the uncertainty of the text category. In this way, when the text has obvious category features, this application minimizes the use of image information; when the text has no category features, it maximizes the use of image information. This optimizes the existing method's mechanism for utilizing image features, enabling dynamic introduction of image features and mitigating interference from noisy images. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 This is a flowchart of a multimodal text classification method disclosed in this application;

[0038] Figure 2 This is a schematic diagram of a multimodal fusion method based on an attention mechanism disclosed in this application;

[0039] Figure 3 Here is a flowchart of a specific multimodal text classification method disclosed in this application;

[0040] Figure 4 This is a framework diagram of a multimodal text classification algorithm disclosed in this application;

[0041] Figure 5 This is a schematic diagram of the structure of a multimodal text classification device disclosed in this application;

[0042] Figure 6 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Current multimodal text classification algorithms do not consider whether the image truly contributes to category recognition when fusing image information, and their ability to remove noisy images is weak. Their effectiveness depends on the performance of existing image analysis tools and is greatly affected by error propagation. Pre-training tasks rely on human experience for design, resulting in weak generalization, and pre-training requires substantial data and equipment resources. To address these technical problems, this application discloses a multimodal text classification method that enables the dynamic introduction of image features to reduce interference from noisy images.

[0045] See Figure 1 As shown, this embodiment of the invention discloses a multimodal text classification method, including:

[0046] Step S11: Obtain the initial input text and the corresponding initial image; determine the target input text based on the initial image, the initial input text, and the predefined labels corresponding to the pre-trained target language model; and obtain the target text features corresponding to the target input text based on the target input text and the pre-trained target language model.

[0047] In this embodiment, after obtaining the initial input text and the corresponding initial image, an image title generation tool is used to convert the initial image into a text modal title. Then, a first predefined tag corresponding to the pre-trained target language model is inserted at the beginning of the initial input text, and a second predefined tag corresponding to the pre-trained target language model is inserted at the end of the initial input text to obtain the inserted input text. The text modal title is then concatenated with the inserted input text to obtain the concatenated input text, and the second predefined tag is appended to the end of the concatenated input text to obtain the target input text. Specifically, for the input text, predefined [CLS] and [SEP] tags from BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model) are first inserted at the beginning and end of the sentence. To further enhance the contextual information of the text, an image caption generation tool is used to convert the image into a text modal caption, which is then concatenated with the original input text. Assuming the original input text is represented by S and the generated text modal caption by Caption, the final text T input to BERT can be represented as: T = [CLS]S[SEP]Caption[SEP]. Here, [CLS] and [SEP] are two special notations in BERT that play a special role in the input text. [CLS] is an abbreviation for "classification," and in text classification tasks, it usually represents the beginning of a sentence or document. In BERT, [CLS] corresponds to the word vector of the first word in the input text, and the first neuron in the output layer is usually used to predict the text category. [SEP] is an abbreviation for "separato," and it usually represents the end of a sentence or document. In BERT, [SEP] corresponds to the word vector of the last word in the input text, and its role is to segment different sentences. For example, when processing sentence pairs in BERT, a [SEP] is usually inserted between the two sentences to indicate their boundary.

[0048] After obtaining the target input text, the pre-trained BERT model is used to output text features. Where n is the sentence length, d T For feature dimensions.

[0049] Step S12: Based on the initial image and the pre-trained deep residual network model, obtain the corresponding initial image features, and use the preset attention mechanism, the initial image features and the target text features to filter target image features that meet the preset conditions.

[0050] In this embodiment, the initial image is adjusted based on a preset image size to obtain an adjusted image. The adjusted image is then input into the pre-trained deep residual network model to obtain the corresponding initial image features. Specifically, for the input image, it is first adjusted to a 128×128 image to fit the input of the pre-trained ResNet (Residual Network). Then, the adjusted image is input into the pre-trained ResNet model to obtain image features. Let the image features output by ResNet be... Among them, w, h, d V These represent the length, width, and feature dimensions, respectively. Furthermore, to facilitate subsequent fusion of text and image features, the image features are serialized into a shape m×d. V , where m = w × h.

[0051] After extracting text and image features, a multimodal fusion method based on an attention mechanism is used to select suitable parts from the image features for text classification. First, the initial image features are processed using a preset linear transformation based on the target dimension to obtain processed image features. Then, the processed image features are multiplied with the target text features to obtain a correlation matrix. Next, image features filtered based on attention are obtained using an objective function, the correlation matrix, and the initial image features. Finally, max pooling is used to extract target image features that meet preset conditions from the attention-filtered image features. Specifically, using text features as the query, image features are selected through an attention mechanism, as follows: Figure 2 As shown. The multimodal fusion process based on the attention mechanism can be expressed by the following formula:

[0052]

[0053]

[0054] in, W represents the image features output after max pooling. v Let b be the parameter matrix of the image feature transformation network. v For bias terms, The goal is to transform image features to have the same dimensionality as text features. Specifically, firstly, a linear transformation maps the image features to the same dimensionality as the text features. Then, matrix multiplication is performed between the mapped image features and the text features to obtain the correlation matrix between the image and the text. This correlation matrix is ​​then multiplied with the image features after passing through a softmax function, thus enabling the selection of important image features based on attention. Finally, the maximum pooling operation is used to... Extract an image feature for text classification.

[0055] Step S13: Select a set of classless texts from the predetermined training set, obtain intermediate classless text features corresponding to the set of classless texts based on the pre-trained target language model, and input the intermediate classless text features and the intermediate input text features corresponding to the target input text into the variational autoencoder to obtain the mean and variance of the intermediate input text features and the intermediate classless text features.

[0056] In this embodiment, traditional attention-based multimodal fusion methods do not consider whether the image truly contributes to text classification. They directly fuse extracted image features with text features for the final text classification, thus making them susceptible to the adverse effects of noisy images. This application optimizes the mechanism for utilizing image features, enabling the dynamic introduction of image features. When the text has obvious category features, the use of image information is minimized, as image information is not very meaningful for category recognition in this case, and introducing images will increase the risk of introducing noise. Conversely, when there are no obvious category features in the text, the use of image information is maximized, as the category features of the text may be hidden in the image. Therefore, an uncertainty learning mechanism for text categories is designed to model the above process. Specifically, a weight is dynamically learned to represent the uncertainty of the text category. When the uncertainty of the text category is high, the use of image information is increased, and vice versa. The dynamic weights are obtained by measuring the distance between the input text feature distribution and the unclassified text feature distribution. The first problem to solve is how to obtain the unclassified text. In multimodal text classification datasets, the category of a sample is determined by both text and image, making it difficult to determine whether a single text provides obvious category features. However, if a sample is labeled as unclassified, its text and image must also lack obvious category features. Therefore, this property is utilized to sample a portion of text from the unclassified samples in the training set to construct an unclassified text set. Subsequently, uncertainty learning of text categories will be performed by sampling from this unclassified text set. An unclassified text set is selected from a pre-determined training set, and intermediate unclassified text features corresponding to the unclassified text set are obtained based on the pre-trained target language model. Simultaneously, before inputting the intermediate unclassified text features and the intermediate input text features corresponding to the target input text into the variational autoencoder, average pooling is used to process the intermediate unclassified text features and the intermediate input text features. Then, the intermediate layer unclassed text features and the intermediate layer input text features corresponding to the target input text are input into the variational autoencoder to obtain the mean and variance of the intermediate layer input text features and the intermediate layer unclassed text features.

[0057] Step S14: Obtain weight values ​​representing the uncertainty of text category based on the mean and variance, so as to use a preset classifier to classify the initial input text according to the weight values, the target image features and the target text features, and obtain the corresponding classification results.

[0058] In this embodiment, to obtain the weight values, the first KL divergence between the intermediate layer input text feature distribution and the intermediate layer unclassified text feature distribution, and the second KL divergence between the intermediate layer unclassified text feature distribution and the intermediate layer input text feature distribution are first determined using the mean and the variance, respectively. Based on the first KL divergence and the second KL divergence, weight values ​​representing the uncertainty of the text category are obtained. Then, a preset classifier can be used to classify the initial input text according to the weight values, the target image features, and the target text features, and the corresponding classification results are obtained.

[0059] As can be seen from the above, when classifying text, this application first obtains the initial input text and the corresponding initial image. Based on the initial image, the initial input text, and predefined labels corresponding to the pre-trained target language model, the target input text is determined. Then, the target text features corresponding to the target input text are obtained based on the target input text and the pre-trained target language model. Next, based on the initial image and the pre-trained deep residual network model, the corresponding initial image features are obtained. Finally, a preset attention mechanism, the initial image features, and the target text features are used to filter target image features that meet preset conditions. Finally, target image features are selected from a predetermined training set. A set of unclassified texts is selected, and intermediate unclassified text features corresponding to the set of unclassified texts are obtained based on the pre-trained target language model. These intermediate unclassified text features, along with intermediate input text features corresponding to the target input text, are input into a variational autoencoder to obtain the mean and variance of the intermediate input text features and the intermediate unclassified text features. Finally, weight values ​​representing the uncertainty of text category are obtained based on the mean and variance, so that a preset classifier can be used to classify the initial input text according to the weight values, the target image features, and the target text features, and obtain the corresponding classification results. It can be seen that this application first obtains corresponding text and image features by encoding text and images, performs multimodal fusion based on an attention mechanism, then learns about the uncertainty of text category, and then performs text classification based on the uncertainty of text category. In this way, when the text has obvious category features, this application minimizes the use of image information; when the text has no category features, it maximizes the use of image information. The mechanism for utilizing image features in existing methods has been optimized to enable the dynamic introduction of image features, which can reduce the interference of noisy images.

[0060] As can be seen from the above embodiments, this application dynamically learns a weight to represent the uncertainty of the text category, thereby dynamically adjusting the utilization of image information. Next, the process of dynamically adjusting the utilization of image information through weight will be described in detail. See [link to documentation]. Figure 3 As shown in the figure, this application discloses a specific multimodal text classification method, including:

[0061] Step S21: Select a set of classless texts from a predetermined training set, obtain intermediate classless text features corresponding to the set of classless texts based on the pre-trained target language model, and input the intermediate classless text features and the intermediate input text features corresponding to the target input text into a variational autoencoder to obtain the mean and variance of the intermediate input text features and the intermediate classless text features.

[0062] In this embodiment, as Figure 4 As shown, a portion of text is first sampled from the unclassified samples in the training set to construct an unclassified text set. Subsequently, text uncertainty of the text category is learned by sampling from this unclassified text set. Since the text features output by BERT are fixed, the distribution of text features cannot be directly obtained. Therefore, a generative approach is used to obtain text features. Specifically, the intermediate layer features of BERT are first input into a variational autoencoder to obtain the mean and variance of the input text features and the unclassified text features, respectively.

[0063] Step S22: Determine the first KL divergence between the intermediate layer input text feature distribution and the intermediate layer unclassified text feature distribution, and the second KL divergence between the intermediate layer unclassified text feature distribution and the intermediate layer input text feature distribution using the mean and the variance, respectively.

[0064] In this embodiment, after obtaining the mean and variance, the input text feature distribution and the classless text feature distribution are calculated using the mean and variance, and then the Kullback-Leibler divergence (KL-Divergence) is calculated using the input text feature distribution and the classless text feature distribution. The KL divergence measures the distance between the input text feature distribution and the classless text feature distribution. Therefore, the first KL divergence between the intermediate layer input text feature distribution and the intermediate layer classless text feature distribution, and the second KL divergence between the intermediate layer classless text feature distribution and the intermediate layer input text feature distribution can be determined.

[0065] Step S23: Obtain weight values ​​representing text category uncertainty based on the first KL divergence and the second KL divergence.

[0066] In this embodiment, assuming a1 and a2 are the first KL divergence between the input text feature distribution and the classless text feature distribution, and the second KL divergence between the classless text feature distribution and the input text feature distribution, respectively, the weight representing the text category uncertainty can be calculated by the following formula:

[0067]

[0068] The utilization of image information is then dynamically adjusted using a weight 'a'. When 'a' is large, the utilization of image information is reduced, and vice versa.

[0069] Step S24: Use a preset classifier to classify the initial input text according to the weight value, target image features and target text features, and obtain the corresponding classification results.

[0070] In this embodiment, image features are dynamically introduced based on 'a' for text classification. This process can be described by the following formula:

[0071]

[0072]

[0073] in, and These are the image features output after max pooling and the features corresponding to [CLS] output by BERT, respectively. This represents the final classification result. c The data is input into a softmax classifier for classification, thereby obtaining the corresponding classification result.

[0074] In this way, this application employs an uncertainty learning algorithm for text categories. When the text has obvious category features, it minimizes the use of image information; when the text has no category features, it maximizes the use of image information. This optimizes the existing methods' mechanism for utilizing image features, enabling the dynamic introduction of image features and mitigating the interference of noisy images.

[0075] See Figure 5 As shown, an embodiment of the present invention discloses a multimodal text classification device, comprising:

[0076] The target text feature acquisition module 11 is used to acquire the initial input text and the corresponding initial image, determine the target input text based on the initial image, the initial input text and the predefined label corresponding to the pre-trained target language model, and acquire the target text features corresponding to the target input text according to the target input text and the pre-trained target language model;

[0077] The target image feature filtering module 12 is used to obtain corresponding initial image features based on the initial image and the pre-trained deep residual network model, and to filter target image features that meet preset conditions using a preset attention mechanism, the initial image features and the target text features.

[0078] The mean and variance acquisition module 13 is used to select a set of unclassed texts from a predetermined training set, obtain intermediate unclassed text features corresponding to the set of unclassed texts based on the pre-trained target language model, and input the intermediate unclassed text features and the intermediate input text features corresponding to the target input text into the variational autoencoder to obtain the mean and variance corresponding to the intermediate input text features and the intermediate unclassed text features.

[0079] The text classification module 14 is used to obtain a weight value representing the uncertainty of the text category based on the mean and the variance, so as to use a preset classifier to classify the initial input text according to the weight value, the target image features and the target text features, and obtain the corresponding classification result.

[0080] As can be seen from the above, when classifying text, this application first obtains the initial input text and the corresponding initial image. Based on the initial image, the initial input text, and predefined labels corresponding to the pre-trained target language model, the target input text is determined. Then, the target text features corresponding to the target input text are obtained based on the target input text and the pre-trained target language model. Next, based on the initial image and the pre-trained deep residual network model, the corresponding initial image features are obtained. Finally, a preset attention mechanism, the initial image features, and the target text features are used to filter target image features that meet preset conditions. Finally, target image features are selected from a predetermined training set. A set of unclassified texts is selected, and intermediate unclassified text features corresponding to the set of unclassified texts are obtained based on the pre-trained target language model. These intermediate unclassified text features, along with intermediate input text features corresponding to the target input text, are input into a variational autoencoder to obtain the mean and variance of the intermediate input text features and the intermediate unclassified text features. Finally, weight values ​​representing the uncertainty of text category are obtained based on the mean and variance, so that a preset classifier can be used to classify the initial input text according to the weight values, the target image features, and the target text features, and obtain the corresponding classification results. It can be seen that this application first obtains corresponding text and image features by encoding text and images, performs multimodal fusion based on an attention mechanism, then learns about the uncertainty of text category, and then performs text classification based on the uncertainty of text category. In this way, when the text has obvious category features, this application minimizes the use of image information; when the text has no category features, it maximizes the use of image information. The mechanism for utilizing image features in existing methods has been optimized to enable the dynamic introduction of image features, which can reduce the interference of noisy images.

[0081] In some specific embodiments, the apparatus can also be used to convert the initial image into a text modal title using an image title generation tool.

[0082] In some specific embodiments, the target text feature acquisition module 11 may specifically include:

[0083] The inserted input text acquisition unit is used to insert a first predefined label corresponding to the pre-trained target language model at the beginning position of the initial input text and insert a second predefined label corresponding to the pre-trained target language model at the end position of the initial input text to obtain the inserted input text;

[0084] The target input text acquisition unit is used to concatenate the text modal title with the inserted input text to obtain the corresponding concatenated input text, and to concatenate the second predefined tag to the end of the concatenated input text to obtain the target input text.

[0085] In some specific embodiments, the target image feature filtering module 12 may specifically include:

[0086] The initial image feature acquisition unit is used to adjust the initial image based on a preset image size, acquire the adjusted image, input the adjusted image into the pre-trained deep residual network model, and acquire the corresponding initial image features.

[0087] In some specific embodiments, the target image feature filtering module 12 may specifically include:

[0088] An initial image feature processing unit is used to process the initial image features based on the target dimension using a preset linear transformation to obtain processed image features;

[0089] The correlation matrix acquisition unit is used to perform matrix multiplication on the processed image features and the target text features to obtain a correlation matrix.

[0090] An image feature filtering unit is used to obtain attention-based image features using an objective function, the correlation matrix, and the initial image features;

[0091] The target image feature extraction unit is used to extract target image features that meet preset conditions from the attention-based filtered image features using max pooling operations.

[0092] In some specific embodiments, the apparatus can also be used to process the intermediate layer unclassed text features and the intermediate layer input text features using average pooling operations.

[0093] In some specific embodiments, the text classification module 14 may specifically include:

[0094] The divergence determination unit is used to determine the first KL divergence between the intermediate layer input text feature distribution and the intermediate layer unclassified text feature distribution, and the second KL divergence between the intermediate layer unclassified text feature distribution and the intermediate layer input text feature distribution, respectively, using the mean and the variance.

[0095] The weight value determination unit is used to obtain weight values ​​representing the uncertainty of text category based on the first KL divergence and the second KL divergence.

[0096] Furthermore, embodiments of this application also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0097] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the multimodal text classification method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0098] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0099] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0100] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the multimodal text classification method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0101] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed multimodal text classification method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0103] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0105] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0106] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal text classification method, characterized in that, include: Obtain initial input text and corresponding initial image; determine target input text based on initial image, initial input text and predefined labels corresponding to pre-trained target language model; and obtain target text features corresponding to target input text based on target input text and pre-trained target language model. Based on the initial image and the pre-trained deep residual network model, the corresponding initial image features are obtained, and the target image features that meet the preset conditions are selected using the preset attention mechanism, the initial image features, and the target text features. Select a set of unclassed texts from a predetermined training set, obtain intermediate unclassed text features corresponding to the set of unclassed texts based on the pre-trained target language model, and input the intermediate unclassed text features and the intermediate input text features corresponding to the target input text into a variational autoencoder to obtain the mean and variance of the intermediate input text features and the intermediate unclassed text features. Based on the mean and variance, weight values ​​representing the uncertainty of the text category are obtained, so that a preset classifier can be used to classify the initial input text according to the weight values, the target image features, and the target text features, and obtain the corresponding classification results.

2. The multimodal text classification method according to claim 1, characterized in that, After obtaining the initial input text and the corresponding initial image, the process further includes: The initial image is converted into a text modal title using an image title generation tool.

3. The multimodal text classification method according to claim 2, characterized in that, The step of determining the target input text based on the initial image, the initial input text, and predefined labels corresponding to the pre-trained target language model includes: Insert a first predefined label corresponding to the pre-trained target language model at the beginning of the sentence of the initial input text, and insert a second predefined label corresponding to the pre-trained target language model at the end of the sentence of the initial input text to obtain the input text after insertion; The text modal title is concatenated with the inserted input text to obtain the corresponding concatenated input text. The second predefined tag is then concatenated to the end of the concatenated input text to obtain the target input text.

4. The multimodal text classification method according to claim 1, characterized in that, The step of obtaining the corresponding initial image features based on the initial image and the pre-trained deep residual network model includes: The initial image is adjusted based on a preset image size to obtain an adjusted image. The adjusted image is then input into the pre-trained deep residual network model to obtain the corresponding initial image features.

5. The multimodal text classification method according to claim 1, characterized in that, The step of using a preset attention mechanism, the initial image features, and the target text features to filter target image features that meet preset conditions includes: The initial image features are processed using a preset linear transformation based on the target dimension to obtain the processed image features; The processed image features and the target text features are multiplied by a matrix to obtain a correlation matrix; Image features after attention filtering are obtained using the objective function, the correlation matrix, and the initial image features; Max pooling is used to extract target image features that meet preset conditions from the attention-based filtered image features.

6. The multimodal text classification method according to claim 1, characterized in that, Before inputting the intermediate layer classless text features and the intermediate layer input text features corresponding to the target input text into the variational autoencoder, the method further includes: The intermediate layer's unclassed text features and the intermediate layer's input text features are processed using average pooling.

7. The multimodal text classification method according to any one of claims 1 to 6, characterized in that, The step of obtaining the weight value representing the uncertainty of the text category based on the mean and the variance includes: The first KL divergence between the intermediate layer input text feature distribution and the intermediate layer unclassified text feature distribution, and the second KL divergence between the intermediate layer unclassified text feature distribution and the intermediate layer input text feature distribution are determined using the mean and the variance, respectively. The weight values ​​representing the uncertainty of the text category are obtained based on the first KL divergence and the second KL divergence.

8. A multimodal text classification device, characterized in that, include: The target text feature acquisition module is used to acquire the initial input text and the corresponding initial image, determine the target input text based on the initial image, the initial input text and the predefined labels corresponding to the pre-trained target language model, and acquire the target text features corresponding to the target input text according to the target input text and the pre-trained target language model; The target image feature filtering module is used to obtain corresponding initial image features based on the initial image and the pre-trained deep residual network model, and to filter target image features that meet preset conditions using a preset attention mechanism, the initial image features and the target text features. The mean and variance acquisition module is used to select a set of unclassed texts from a predetermined training set, obtain intermediate unclassed text features corresponding to the set of unclassed texts based on the pre-trained target language model, and input the intermediate unclassed text features and the intermediate input text features corresponding to the target input text into the variational autoencoder to obtain the mean and variance corresponding to the intermediate input text features and the intermediate unclassed text features. The text classification module is used to obtain weight values ​​representing the uncertainty of text category based on the mean and the variance, so as to use a preset classifier to classify the initial input text according to the weight values, the target image features and the target text features, and obtain the corresponding classification results.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the multimodal text classification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the multimodal text classification method as described in any one of claims 1 to 7.

Images