Model training method based on multi-modal feature fusion, electronic device, and medium

Abstract

Embodiments of the present application relate to the field of visual language, and disclose a model training method based on multi-modal feature fusion, an electronic device and a storage medium. The method comprises: obtaining multi-modal samples of public opinion information, the multi-modal samples comprising at least a first text sample describing the public opinion information, a picture sample describing the public opinion information, and a second text sample describing the picture sample in a text manner; extracting, by using a pre-training model, first text features of the first text sample, second text features of the second text sample, and image features of the picture sample; fusing the first text features and the second text features to obtain third text features; and training the pre-training model to perform contrast learning based on the third text features and the image features, so as to obtain a pre-training model meeting a training target. The present scheme can enhance the text features in the pre-training process, make the model learn the enhanced multi-modal features, and thus improve the performance of the model.

Description

Technical Field

[0001] This invention relates to the field of visual language, and in particular to a model training method, electronic device, and medium based on multimodal feature fusion. Background Technology

[0002] The main purpose of visual language pre-training is to use large-scale image and text datasets for joint pre-training to learn the correlation between multimodal information, thereby improving a series of downstream visual-language tasks such as image description generation, visual question answering, and image text retrieval.

[0003] Most current visual language pre-trained models either concatenate text and visual features together and feed them into a single Transformer block, or send text and visual features independently into two different Transformer blocks, and combine them with multiple pre-training objectives to perform subsequent representation fusion and other operations. They are then tested on different downstream tasks on some multimodal datasets (such as MSCOCO, Flickr30k, etc.).

[0004] However, influenced by the image-text pair paradigm of general multimodal datasets, current visual language pre-training work can only focus on the relationship between images and corresponding image description texts, focusing on using large-scale general datasets for pre-training, while ignoring other factors that can enhance multimodal representations. Summary of the Invention

[0005] The purpose of this invention is to provide a model training method, electronic device, and storage medium based on multimodal feature fusion. When extracting text features, the text of event information is introduced as the feature extraction object, which can enhance the text features in the pre-training process, and then form enhanced multimodal features with image features, allowing the model to learn the enhanced multimodal features, thereby improving the model's performance.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a model training method based on multimodal feature fusion, comprising:

[0007] Obtain multimodal samples of public opinion information, wherein the multimodal samples include at least a first text sample describing the public opinion information, an image sample describing the public opinion information, and a second text sample describing the image sample in a textual manner;

[0008] A pre-trained model is used to extract the first text features of the first text sample, the second text features of the second text sample, and the image features of the image sample; and,

[0009] The first text feature is fused with the second text feature to obtain the third text feature;

[0010] The pre-trained model is trained by comparing the third text features with the image features to obtain a pre-trained model that meets the training objectives.

[0011] Embodiments of the present invention also provide an electronic device, comprising:

[0012] At least one processor; and,

[0013] A memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the model training method based on multimodal feature fusion as described above.

[0015] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the model training method based on multimodal feature fusion as described above.

[0016] In some embodiments, the pre-trained model described above includes an image encoder and a text encoder; the step of using the pre-trained model to extract the first text feature of the first text sample, the second text feature of the second text sample, and the image feature of the image sample includes: inputting the first text sample and the second text sample into the text encoder respectively, and using the text encoder to extract the first text feature and the second text feature respectively; inputting the image sample into the image encoder, and using the image encoder to extract the image feature.

[0017] In some embodiments, the pre-trained model described above further includes a linear feature fusion layer; fusing the first text feature with the second text feature to obtain a third text feature includes: inputting the first text feature and the second text feature into the linear feature fusion layer, and fusing the first text feature and the second text feature through the linear feature fusion layer according to a preset fusion ratio corresponding to the text language type of the public opinion information to obtain the third text feature.

[0018] In some embodiments, the first text feature and the second text feature are fused according to the fusion ratio corresponding to the preset category of the public opinion information, as described above, to obtain the third text feature. The fusion formula is: w' = add(α*w + β*c); where w' is the third text feature, add is the tensor connection function, α is the proportion of the first text feature w1 in the fusion, and β is the proportion of the second text feature w2 in the fusion.

[0019] In some embodiments, the model training method based on multimodal feature fusion as described above further includes: the pre-trained model includes a shared embedding layer; training the pre-trained model to obtain a pre-trained model that meets the training objective by comparing the third text features with the image features, using the shared embedding layer to calculate the predicted similarity between the third text features and the image features, and constructing a contrastive loss function based on the normalized predicted similarity and its corresponding labeled similarity; and training the pre-trained model based on the constructed contrastive loss function to obtain the pre-trained model that meets the training objective.

[0020] In some embodiments, the contrastive loss function constructed as described above based on the normalized predicted similarity and its corresponding labeled similarity includes:

[0021] Among them, L itc For comparison, E is the expectation, D is the sample domain, H() is the cross-entropy, y(I) is the labeled similarity from the third text feature to the image feature, y(T') is the labeled similarity from the image feature to the third text feature, p(I) is the normalized predicted similarity from the third text feature to the image feature, and p(T') is the normalized predicted similarity from the image feature to the third text feature.

[0022] In some embodiments, the process of obtaining the second text sample as described above includes: performing semantic recognition on the image sample, and generating the second text sample based on the recognized semantic information.

[0023] In some embodiments, the fusion ratio corresponding to the text language type of the public opinion information as described above includes: when the text language type of the public opinion information is Chinese, the fusion ratio corresponding to the first text feature and the second text feature is 4:6; when the text language type of the public opinion information is English, the fusion ratio corresponding to the first text feature and the second text feature is 9:1.

[0024] Compared to existing technologies, the embodiments of this invention acquire multimodal samples of public opinion information. These multimodal samples include at least a first text sample describing the public opinion information, an image sample describing the public opinion information, and a second text sample describing the image sample using text. A pre-trained model is used to extract first text features from the first text sample, second text features from the second text sample, and image features from the image sample. The first text features and second text features are then fused to obtain a third text feature. The pre-trained model is trained by comparing and learning the third text feature with the image feature to obtain a pre-trained model that meets the training objectives. In other words, the solution of this application acquires image samples of public opinion information, first text samples describing the public opinion information, and second text samples describing the image sample using text when acquiring public opinion information samples. By fusing the first text features of the first text sample with the second text features of the second text sample, the text features are enhanced. This enhances the text features and forms multimodal features with the image features, allowing the model to learn the enhanced multimodal features and improve model performance. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the specific process of a model training method based on multimodal feature fusion according to an embodiment of the present invention.

[0026] Figure 2 This is a public opinion information dataset table according to an embodiment of the present invention;

[0027] Figure 3 Table 1 shows the comparison of experimental results between the pre-trained model according to the present invention and the existing technology model.

[0028] Figure 4 Table 2 shows the comparison of experimental results between the pre-trained model according to the present invention and the existing technology model.

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of the present invention to enable the reader to better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0031] Current visual language pre-training methods only focus on the relationship between images and their corresponding text descriptions, concentrating on using large-scale general datasets for pre-training, while neglecting other factors that can enhance multimodal features. This application's model training method based on multimodal feature fusion applies a visual language pre-trained model (hereinafter referred to as the pre-trained model) to the field of public opinion events, attempting to expand the application scope of pre-trained models and fill the gap in the field of public opinion events. Simultaneously, it considers other factors that enhance multimodal features, allowing the model to learn the enhanced multimodal features, thereby improving the model's performance after learning.

[0032] The following is a detailed description of the implementation details of the audio processing method in this embodiment. The following content is only for the convenience of understanding and is not necessary for implementing this solution.

[0033] One embodiment of the present invention relates to a model training method based on multimodal feature fusion, such as... Figure 1 As shown in the figure, the model training method based on multimodal feature fusion provided in this embodiment includes the following steps.

[0034] Step 101: Obtain multimodal samples of public opinion information. The multimodal samples include at least a first text sample describing the public opinion information, an image sample describing the public opinion information, and a second text sample describing the image sample in text form.

[0035] Public opinion information, broadly speaking, falls under the category of social information and encompasses all knowledge, messages, signals, and intelligence generated through human consciousness that can influence society. In a narrower sense, public opinion information is formed through the organized and purposeful collection and processing of public opinion; it is a carrier specifically designed to describe and reflect certain public opinion states and changes. The sources of public opinion information can include newspapers, periodicals, websites, forums, online communities, magazines, etc., and this application does not impose specific restrictions on these sources.

[0036] Specifically, most current visual language pre-training models use general large-scale datasets for training and evaluation, such as MSCOCO and Flickr30K. These datasets typically use images and corresponding image descriptions, allowing the model to learn the relationship between the images and the descriptions. However, datasets for the public opinion domain are currently lacking. To fill this gap, a large amount of image and text data related to public opinion can be crawled from the internet. Specific public opinion information includes, but is not limited to, categories such as military, sports, finance, and technology. Multimodal samples of public opinion information are obtained, including image samples, first text samples describing the public opinion information, and second text samples describing the image samples. The image samples are directly obtained from the crawled public opinion information. The first text samples are the public opinion text directly obtained from the crawled public opinion information, such as the main text or comments of the public opinion information. The second text samples are not directly obtained from the crawled public opinion information but are descriptive texts generated based on the image samples. The process of obtaining the second text sample can involve performing semantic recognition on the image sample and generating the second text sample based on the recognized semantic information. Specifically, deep learning can be used to automatically generate text descriptions for the image; this application does not impose any specific limitations on this.

[0037] Step 102: Use a pre-trained model to extract the first text features of the first text sample, the second text features of the second text sample, and the image features of the image sample.

[0038] Specifically, in this embodiment, the pre-trained model is a Vision-Language Pre-training (VLP) model (hereinafter referred to as the pre-trained model). Vision-Language Pre-training (VLP) can be understood from five aspects: feature extraction, model architecture, pre-training objectives, pre-training dataset, and downstream tasks. Feature extraction includes the preprocessing and representation methods for images, videos, and text in the Vision-Language Pre-trained VLP model. Model architecture: From the perspective of multimodal fusion, single-stream versus dual-stream; from the perspective of overall architecture design, encoder-only versus encoder-decoder. Pre-training objectives: Pre-training objectives are the core of VLP, mainly used to guide the model in learning visual language-related information. Typical and characteristic pre-training objectives are summarized, categorized into completion, matching, time, and specific types. Pre-training dataset: Data is crucial for VLP. Downstream tasks: Various tasks require collaborative knowledge of vision and language, and can be divided into five categories: classification, regression, retrieval, generation, and other tasks.

[0039] This embodiment uses a pre-trained model to extract the first text features of the first text sample, the second text features of the second text sample, and the image features of the image sample. The mainstream methods for extracting text features include: bag-of-words model + OneHot encoding, bag-of-words model + keyword weights (TF-IDF, TextRank, etc.), bag-of-words model + topic model (LSA, LDA, LDA2Vec, etc.), and embedding models. Methods for extracting image features include: SIFT (Scale Invariant Feature Transform), HOG (Histogram of Oriented Gradients), and neural network feature extraction. This application does not limit the specific methods used for extracting text and image features.

[0040] In one example, the pre-trained model includes an image encoder and a text encoder. Based on this, step 102 can be specifically described as follows:

[0041] The first text sample and the second text sample are input into the text encoder respectively, and the first text feature and the second text feature are extracted by the text encoder respectively; the image sample is input into the image encoder, and the image feature is extracted by the image encoder.

[0042] Specifically, image encoders can be used to extract image features from image samples. These encoders can employ a multi-layer Transformer architecture, such as Vision Transformer (ViT), to convert image pixels into sequential data. The Transformer encoder is then used to learn the image's feature representation. Text encoders are used to extract first and second text features from the first and second text samples respectively. These text encoders also employ a multi-layer Transformer architecture, such as BERT and RoBERTa. Specifically, different text encoders can be used for different language types of the public opinion information. For example, if the first and second text samples are in Chinese, RoBERTa can be used as the text encoder; if the first and second text samples are in English, BERT can be used.

[0043] It should be noted that both the image encoder and the text encoder in this embodiment are pre-trained encoders. At the same time, the text features (first text features and second text features) output by the image encoder and the text encoder are feature vectors of the same dimension as the image features.

[0044] Step 103: Fuse the first text feature with the second text feature to obtain the third text feature.

[0045] Specifically, the first text feature is the text feature describing public opinion information, and the second text feature is the feature of the descriptive text generated based on the image sample, which is essentially still a feature of the image sample. After fusing the first and second text features, the text and image of public opinion information can be closely integrated through text features, so that the resulting third text feature not only has features related to the image sample, but also features related to the descriptive text in the public opinion information, making the third text feature more relevant to public opinion information than the second text feature.

[0046] For example, for image sample I, the image features extracted after encoding by the image encoder can be represented as the vector {v|v=(v1, v2, …, v N For the first text sample T1, the first text feature extracted after encoding by the text encoder can be represented as the vector {w1|w1=(w 11 w 12 , ..., w 1N For the second text sample T2, the second text features extracted after encoding by the text encoder can be represented as the vector {w2|w2=(w 21 w 22 ,…,w 2N The third text feature, obtained by fusing the first and second text features, is represented as a vector {w'|w'=(w'1,w'2,…,w'}. N )}.

[0047] In one example, the pre-trained model also includes a linear feature fusion layer. Based on this, step 103, which fuses the first text features with the second text features, can be specifically described as follows:

[0048] The first text feature and the second text feature are input into the linear feature fusion layer. The linear feature fusion layer fuses the first text feature and the second text feature according to the fusion ratio corresponding to the text language type of the public opinion information, and obtains the third text feature.

[0049] Specifically, a linear feature fusion layer is added to the pre-trained model to fuse the first and second text features. Since public opinion information is diverse, different fusion ratios will result in different third text features, which will affect the final model's learning performance. Specifically, the first and second text features can be fused according to a pre-defined fusion ratio corresponding to the text language type of the public opinion information, or they can be fused according to a specific ratio based on the category of the public opinion information. This application does not impose specific restrictions on the fusion ratio of the first and second text features, or the basis for determining the fusion ratio.

[0050] In another example, the first text feature and the second text feature are fused according to the fusion ratio corresponding to the preset category of public opinion information to obtain the third text feature. The fusion formula is: w'=add(α*w1+β*w2);

[0051] Where w' is the third text feature, add is the tensor connection function, α is the proportion of the first text feature w1 in the fusion, and β is the proportion of the second text feature w2 in the fusion.

[0052] Specifically, the text fusion ratio coefficient in this embodiment can be 9:1, 8:2, 7:3, 6:4, 5:5, 4:6, 3:7, 2:8, or 1:9. This application does not impose any specific restrictions on this.

[0053] In another example, the fusion ratio corresponding to the text language type of public opinion information includes:

[0054] When the text language of public opinion information is Chinese, the optimal fusion ratio of the first text feature to the second text feature is α:β = 4:6; when the text language of public opinion information is English, the optimal fusion ratio of the first text feature to the second text feature is α:β = 9:1.

[0055] Step 104: Training the pre-trained model. Based on the comparison and learning of third-party text features and image features, a pre-trained model that meets the training objectives is obtained.

[0056] After obtaining the third textual and image features, the pre-trained model is trained using comparative learning based on these features. Comparative learning, as the training objective of the entire pre-trained model, focuses on learning the common features between similar instances and distinguishing the differences between dissimilar instances. It only needs to learn to differentiate data at the abstract semantic level of the feature space, thus simplifying the model and its optimization, and enhancing its generalization ability. For example, after training the pre-trained model using comparative learning to obtain a model that meets the training objectives, using this model to extract features from images and corresponding text, and calculating the similarity between the resulting image-text pairs, will yield significantly different similarities compared to calculating the same image-text pair before comparative learning training.

[0057] Since the third text feature and image feature (i.e., multimodal features) have a stronger correlation with public opinion information than the second text feature, meaning the text features of public opinion information are enhanced, the pre-trained model performs comparative learning based on the third text feature and image feature to obtain a pre-trained model that meets the training objective. The resulting pre-trained model performs better in downstream tasks. The training objective can be set according to the number of iterations, training duration, and contrastive loss value, etc., and this application does not impose specific restrictions on these.

[0058] In one example, the pre-trained model includes a shared embedding layer. Based on this, step 104 specifically includes the following steps:

[0059] A shared embedding layer is used to calculate the predicted similarity between the third text feature and the image feature, and a contrastive loss function is constructed based on the normalized predicted similarity and its corresponding labeled similarity; and a pre-trained model is trained based on the constructed contrastive loss function to obtain a pre-trained model that meets the training objective.

[0060] Specifically, the goal of contrastive learning is to enable the model to better learn the associations between images and text. Through contrastive learning, the pre-trained model is trained to map related images and text to similar representations, while mapping unrelated images and text to dissimilar representations. A shared embedding layer can be used to calculate the predicted similarity between third-party text features and image features. The function for calculating the predicted similarity, s, is as follows: s(I,T')=g v (v) Τ g w' (w'), s(T',I)=g w' (w') Τ g v (v).

[0061] Where s(I,T') is the predicted similarity from the third text feature to the image feature, s(T',I) is the predicted similarity from the image feature to the third text feature, and g is the normalization function.

[0062] After calculating the predicted similarity between the third text feature and the image feature for each image-text pair, the normalized softmax is then calculated, which is the normalized predicted similarity. Normalization can be calculated using the following formulas: p(I)=s(I,T') / τ, p(T'=s(T',I) / τ.

[0063] Where p(I) is the result of normalizing the predicted similarity s(I,T'), p(T') is the result of normalizing the predicted similarity s(T',I), and τ is a preset learnable parameter that can be set according to the actual needs of model training; this application does not impose specific restrictions on it. In another example, the contrastive loss function constructed based on the normalized predicted similarity and its corresponding labeled similarity includes:

[0064] Among them, L itc For comparison loss, E is expectation, D is the sample domain, H() is cross-entropy, y(I) is the labeled similarity from the third text feature to the image feature, y(T') is the labeled similarity from the image feature to the third text feature, p(I) is the normalized predicted similarity from the third text feature to the image feature, and p(T') is the normalized predicted similarity from the image feature to the third text feature.

[0065] Specifically, the labeled similarity is obtained based on the ground truth score. In the actual training process, assuming the batch size is 16, a batch of multimodal samples of public opinion information will contain 16 text samples and 16 image samples. After feature extraction and fusion, a two-dimensional vector is obtained. When calculating the predicted similarity, assuming the image features are in rows and the third text features are in columns, the similarity calculation will result in a 16*16 similarity matrix. This matrix is ​​the predicted similarity s(I,T') from the third text features to the image features in the overall multimodal samples of public opinion information. After normalization, p(I) is obtained. Transposing p(I) yields p(T'). Similarly, when calculating predicted similarity, assuming the third text feature is in rows and the image feature is in columns, the similarity calculation will also yield a 16*16 similarity matrix. This matrix is ​​the predicted similarity s(T',I) from the image feature to the third text feature. After normalization, we get p(T'), and the matrix transpose of p(T') gives p(I). However, in both similarity matrices p(I) and p(T'), the diagonal lines represent the similarity scores between images and texts from the same public opinion information. Ideally, the similarity scores on the diagonal of the similarity matrix should be the highest in each row or column. y(I) and y(T') are both vectors [0,1,2,……,15], representing that the first image sample corresponds to the first text sample (image and text from the same public opinion information), the second image corresponds to the second text, and so on. Using y(I) and p(I), y(T') and p(T') to calculate cross-entropy loss to train a pre-trained model for contrastive learning can improve the model's performance in performing downstream tasks.

[0066] The specific applications and effects of the above-described method embodiments of the present invention will be illustrated below. The following content is only intended to enable relevant personnel to better understand this application and does not constitute any limitation on this application.

[0067] Step 1: Select a pre-trained model as the base. This is usually a model pre-trained on a large-scale dataset, such as BERT or GPT. In this example, we choose CLIP, a classic model in the field of visual language pre-training, as the base model and then fine-tune it.

[0068] Step Two: Dataset Preparation, including cleaning and preprocessing. The dataset for this invention was obtained through web crawling, collecting a large amount of image and text data from the internet, including four categories: military, sports, finance, and technology. The text was cleaned to remove noise such as special symbols, and the images and text were aligned to ensure accuracy in subsequent experiments. In this embodiment, the downstream task of the model is image-text retrieval; therefore, corresponding image-text pairs need to be constructed to complete the bidirectional tasks of searching for images using text and searching text using images. The processed dataset is stored in .pkl file format, specifically including: event ID, image link, image description, and event text. The event ID is used to uniquely identify events. Next, the constructed dataset is divided into training and validation sets. The dataset statistics are as follows: Figure 2 As shown, in this embodiment, the training set (train) and the validation set (val) are split in a 4:1 ratio.

[0069] Step 3: Define the model structure. The downstream task evaluated in this invention is image-text retrieval, therefore the model structure needs to be modified. Specifically, the dataset processing class needs to be redefined. This class is used to read image-text pairs in the dataset, extract features from the text, and perform preprocessing operations on the images, including image segmentation and transformation. According to the model structure, in the feature extraction layer, the input is the corresponding text and image. The text includes two parts: image description text (second text sample) and public opinion event text (first text sample). The image and text encoders in this layer extract image features and text features respectively. Then, the extracted features are fed into the linear feature fusion layer customized in this invention. The extracted image description text (second text features) features and public opinion event text features (first text features) are fused together at a certain ratio to obtain a new fused and enhanced text representation (third text features). Finally, the image and text features are fed into the shared embedding layer, allowing the pre-trained model to learn by comparing the third text features with the image features.

[0070] For the fusion method, this embodiment uses a simple tensor connection to multiply the two text features by their respective scaling coefficients and then sum them to obtain the fused text representation. The fusion is then evaluated on an image-text retrieval task to obtain the accuracy of text-to-image search. The text fusion scaling coefficients used in this embodiment are 9:1, 8:2, 7:3, 6:4, 5:5, 4:6, 3:7, 2:8, and 1:9. The fusion performance under all scaling coefficients is verified. Experiments are conducted on a science and technology dataset to verify the effectiveness on datasets in different languages, Chinese and English.

[0071] Step 4: Define the loss function and set hyperparameters. The loss function used in this invention is a contrastive loss function. Specifically, it maps both image and text features to the same feature space, bringing similar image-text pairs closer together and distancing different text pairs further apart, thereby minimizing the contrastive loss and allowing the model to converge. On the other hand, this embodiment uses the Adam optimizer to optimize the model parameters, setting the learning rate to 5e-5, the decay factors for the first and second moments of the gradient to 0.9 and 0.98 respectively, and the weight decay coefficient to 0.2. Considering the balance between GPU memory and performance, the batch size is set to 16, and training is performed for 30 epochs.

[0072] Step 5: Train, evaluate, and save the model. After defining a series of hyperparameters, train and evaluate the model based on the divided training and evaluation sets, and save the trained model at a certain frequency for later use.

[0073] The pre-trained model that meets the training objectives is obtained using the method described in this application. Experiments are conducted on a constructed dataset to validate the downstream task of image and text retrieval, which is also the most widely used task for visual language pre-trained models. Specific experimental results are as follows: Figure 3 , Figure 4 As shown, where Figure 3 The results are from experiments conducted on an English dataset (the text of public opinion information is in English). Figure 4 The experimental results are presented on a Chinese dataset (the text language of public opinion information is Chinese). Due to the lack of Chinese pre-training checkpoints for some models, only the base model and the model proposed in this invention were compared on the Chinese dataset. The experimental results of this invention are shown in the last row (Opinion CLIP), achieving state-of-the-art performance across almost all metrics on nearly all datasets.

[0074] Compared with related technologies, the embodiments of the present invention, relative to the prior art, acquire multimodal samples of public opinion information. These multimodal samples include at least a first text sample describing the public opinion information, an image sample describing the public opinion information, and a second text sample describing the image sample using text. A pre-trained model is used to extract first text features from the first text sample, second text features from the second text sample, and image features from the image sample. The first and second text features are then fused to obtain a third text feature. The pre-trained model is trained by comparing and learning the third text feature with the image feature to obtain a pre-trained model that meets the training objectives. In other words, the solution of this application acquires image samples of public opinion information, first text samples describing the public opinion information, and second text samples describing the image sample using text when acquiring public opinion information samples. By fusing the first text features of the first text sample with the second text features of the second text sample, the text features are enhanced. This enhances the text features and forms multimodal features with the image features, allowing the model to learn the enhanced multimodal features and improve model performance.

[0075] Another embodiment of the present invention relates to an electronic device, such as Figure 5 As shown, it includes at least one processor 202; and a memory 201 communicatively connected to at least one processor 202; wherein the memory 201 stores instructions executable by at least one processor 202, the instructions being executed by at least one processor 202 to enable at least one processor 202 to execute any of the above method embodiments.

[0076] The memory 201 and processor 202 are connected via a bus, which can include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 202 and memory 201 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 202 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 202.

[0077] Processor 202 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 201 can be used to store data used by processor 202 during operation.

[0078] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements any of the above-described method embodiments.

[0079] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0080] Those skilled in the art will understand that the above embodiments are specific examples of implementing the present invention, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of the present invention.

Claims

1. A model training method based on multimodal feature fusion, characterized in that, include: The method involves acquiring multimodal samples of public opinion information, wherein the multimodal samples include at least a first text sample describing the public opinion information, an image sample describing the public opinion information, and a second text sample describing the image sample using text. The image sample is subjected to semantic recognition, and the second text sample is generated based on the recognized semantic information; A pre-trained model is used to extract the first text features of the first text sample, the second text features of the second text sample, and the image features of the image sample; and, The first text feature is fused with the second text feature to obtain the third text feature; wherein, The pre-trained model also includes a linear feature fusion layer; The step of fusing the first text feature and the second text feature to obtain the third text feature includes: inputting the first text feature and the second text feature into the linear feature fusion layer, and fusing the first text feature and the second text feature through the linear feature fusion layer according to the preset fusion ratio corresponding to the text language type of the public opinion information to obtain the third text feature; The pre-trained model is trained by comparing the third text features with the image features to obtain a pre-trained model that meets the training objectives.

2. The method according to claim 1, characterized in that, The pre-trained model includes an image encoder and a text encoder; The step of extracting the first text features of the first text sample, the second text features of the second text sample, and the image features of the image sample using a pre-trained model includes: The first text sample and the second text sample are respectively input into the text encoder, and the first text feature and the second text feature are extracted by the text encoder respectively; The image sample is input into the image encoder, and the image encoder is used to extract the image features.

3. The method according to claim 1, characterized in that, The first text feature and the second text feature are fused according to the preset fusion ratio corresponding to the category of the public opinion information, and the fusion formula for obtaining the third text feature is as follows: ; in, The third text feature, Tensor join function ， The first text feature The proportion of integration, β The second text feature The proportion that accounts for in the integration.

4. The method according to claim 1, characterized in that, The method further includes: the pre-trained model includes a shared embedding layer; The training of the pre-trained model is based on comparative learning between the third text features and the image features to obtain a pre-trained model that meets the training objective, including: The shared embedding layer is used to calculate the predicted similarity between the third text features and the image features, and a contrastive loss function is constructed based on the normalized predicted similarity and its corresponding labeled similarity; and, The pre-trained model is trained based on the constructed contrastive loss function to obtain the pre-trained model that meets the training objective.

5. The method according to claim 4, characterized in that, The contrastive loss function, constructed based on the normalized predicted similarity and its corresponding labeled similarity, includes: ； in, To compare the losses, E For the expectation, D For the sample domain, H () represents the cross-entropy. The labeled similarity is the similarity between the third text feature and the image feature. The annotation similarity from the image features to the third text features, The predicted similarity is the result of normalizing the third text features to the image features. The predicted similarity is the result of normalizing the image features to the third text features.

6. The method according to claim 1, characterized in that, The fusion ratios corresponding to the text language types of the public opinion information include: When the text language type of the public opinion information is Chinese, the fusion ratio of the first text feature and the second text feature is 4:6; When the text language type of the public opinion information is English, the fusion ratio of the first text feature and the second text feature is 9:

1.

7. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the model training method based on multimodal feature fusion as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the model training method based on multimodal feature fusion as described in any one of claims 1 to 6.

Images