Training method, feature extraction method, device, equipment, medium, product

Abstract

The present disclosure relates to a training method, a feature extraction method, an apparatus, a device, a medium and a product, and relates to the technical field of computer. The training method comprises: processing visual information in a plurality of samples by using a visual encoder of a multi-modal model to obtain visual features of each sample in the plurality of samples; processing text related to the content of the visual information in each sample by using a text encoder of the multi-modal model to obtain first text features of each sample, and processing classification information of each sample to obtain second text features of each sample; fusing the visual features and the first text features of each sample by using a first fusion module of the multi-modal model to obtain first fusion features of each sample; determining a contrast loss according to the first fusion features and the second text features of the plurality of samples; and adjusting parameters of the multi-modal model according to the contrast loss to train the multi-modal model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a training method, feature extraction method, apparatus, device, medium, and product. Background Technology

[0002] In the field of Artificial Intelligence (AI), multimodal learning is becoming increasingly important. AI models can learn through multimodal methods to understand and process various types of data, such as text, images, and videos.

[0003] Multimodal models can process multimodal data and extract its features, which can be used for various downstream tasks, such as classification and recommendation. Summary of the Invention

[0004] According to some embodiments of this disclosure, a training method is provided, comprising: processing visual information in multiple samples using a visual encoder of a multimodal model to obtain visual features of each sample; processing text related to the content of the visual information in each sample using a text encoder of the multimodal model to obtain first text features of each sample, and processing classification information of each sample to obtain second text features of each sample; fusing the visual features and first text features of each sample using a first fusion module of the multimodal model to obtain first fused features of each sample; determining a contrast loss based on the first fused features and second text features of multiple samples; and adjusting the parameters of the multimodal model based on the contrast loss to train the multimodal model.

[0005] According to some other embodiments of this disclosure, a feature extraction method is provided, comprising: processing visual information in multimedia content to be processed using a visual encoder of a multimodal model to obtain visual features of the multimedia content to be processed, wherein the multimodal model is trained using the training method of any embodiment of this disclosure; processing text related to the visual information in the multimedia content to be processed using a text encoder of a multimodal model to obtain text features of the multimedia content to be processed; and fusing the visual features and text features of the multimedia content to be processed using a first fusion module of the multimodal model to obtain fused features of the multimedia content to be processed.

[0006] According to further embodiments of this disclosure, a training apparatus is provided, comprising: a visual encoding unit configured to process visual information in multiple samples using a visual encoder of a multimodal model to obtain visual features of each sample; a text encoding unit configured to process text related to the content of the visual information in each sample using a text encoder of a multimodal model to obtain a first text feature of each sample, and to process classification information of each sample to obtain a second text feature of each sample; a first fusion unit configured to fuse the visual features and the first text features of each sample using a first fusion module of the multimodal model to obtain a first fusion feature of each sample; a determination unit configured to determine a contrast loss based on the first fusion feature and the second text feature of the multiple samples; and an adjustment unit configured to adjust the parameters of the multimodal model based on the contrast loss to train the multimodal model.

[0007] According to further embodiments of the present disclosure, a feature extraction apparatus is provided, comprising: a visual encoding unit configured to use a multimodal model visual encoder to process visual information in multimedia content to be processed to obtain visual features of the multimedia content to be processed, wherein the multimodal model is trained using the training method of any embodiment of the present disclosure; a text encoding unit configured to use a multimodal model text encoder to process text related to the visual information in the multimedia content to be processed to obtain text features of the multimedia content to be processed; and a fusion unit configured to use a first fusion module of the multimodal model to fuse the visual features and text features of the multimedia content to be processed to obtain fused features of the multimedia content to be processed.

[0008] According to further embodiments of the present disclosure, an electronic device is provided, including: a memory coupled to a processor for storing instructions, which, when executed by the processor, cause the processor to perform a training method as described in any embodiment of the present disclosure, and / or perform a feature extraction method as described in any embodiment of the present disclosure.

[0009] According to further embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, wherein, when executed by a processor, the program causes the processor to perform a training method as described in any embodiment of the present disclosure, and / or to perform a feature extraction method as described in any embodiment of the present disclosure.

[0010] According to further embodiments of the present disclosure, a computer program product is provided, comprising: instructions, wherein when the program is executed by a processor, the processor performs a training method as described in any embodiment of the present disclosure, and / or performs a feature extraction method as described in any embodiment of the present disclosure.

[0011] According to further embodiments of the present disclosure, a multimodal information processing system is provided, including: a training device according to any embodiment of the present disclosure and a feature extraction device according to any embodiment of the present disclosure.

[0012] Other features, aspects, and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0013] Embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the drawings described below are merely illustrative of some embodiments of this disclosure and are not intended to limit the scope of this disclosure. In the drawings:

[0014] Figure 1 A flowchart illustrating a training method according to some embodiments of this disclosure is shown;

[0015] Figure 2 A schematic diagram illustrating some embodiments of the dimensionality reduction method of this disclosure is shown;

[0016] Figure 3 A schematic diagram illustrating a method for determining contrast loss according to some embodiments of the present disclosure;

[0017] Figure 4 A schematic diagram illustrating a method for determining contrast loss according to other embodiments of this disclosure;

[0018] Figure 5 A schematic diagram illustrating multimodal model training according to some embodiments of this disclosure is shown;

[0019] Figure 6 A flowchart illustrating feature extraction methods according to some embodiments of this disclosure is shown;

[0020] Figure 7 A schematic diagram illustrating feature extraction using a multimodal model according to some embodiments of this disclosure is shown;

[0021] Figure 8 This invention discloses a schematic diagram of the structure of a training apparatus according to some embodiments of the present disclosure;

[0022] Figure 9 This diagram illustrates the structure of a feature extraction apparatus according to some embodiments of the present disclosure;

[0023] Figure 10 This diagram illustrates the structure of a multimodal information processing system according to some embodiments of the present disclosure;

[0024] Figure 11 This invention discloses schematic diagrams of the structure of electronic devices according to some embodiments of the present disclosure;

[0025] Figure 12 A schematic diagram of the structure of an electronic device according to other embodiments of the present disclosure is shown. Detailed Implementation

[0026] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. It should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0027] It should be understood that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the steps set forth in these embodiments should be interpreted as merely exemplary and do not limit the scope of this disclosure.

[0028] As used in this disclosure, the term "comprising" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". The term "based on" means "at least partially based on".

[0029] It should be noted that the concepts of "first," "second," etc., used in this disclosure are used only to distinguish different devices, modules, or units, and are not intended to define the order of functions performed by these devices, modules, or units or their interdependencies. Unless otherwise specified, the concepts of "first," "second," etc., are not intended to imply that the objects described herein must be in a given temporal, spatial, rank, or any other given order.

[0030] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0031] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0032] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings; however, this disclosure is not limited to these specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. Furthermore, in one or more embodiments, specific features, structures, or characteristics can be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0033] The training method for multimodal models is crucial for the accuracy of feature extraction from multimodal data, which in turn affects the accuracy of downstream tasks. In related technologies, the training process for visual and text-based multimodal models typically involves determining visual features based on visual information and text features based on relevant text such as titles. The multimodal model is then trained through comparative learning of visual and text features.

[0034] To further improve the accuracy of multimodal model training and feature extraction from multimodal data, this disclosure improves the training method of multimodal models. The visual encoder of the multimodal model processes visual information from multiple samples to obtain visual features for each sample. The text encoder of the multimodal model processes the text related to the visual information and the classification information of each sample to obtain first and second text features for each sample. Then, the first fusion module of the multimodal model fuses the visual features and first text features of each sample to obtain the first fused feature for each sample. Based on the first fused feature and second text features of multiple samples, a contrastive loss is determined. The multimodal model is then trained using the contrastive loss.

[0035] The training method disclosed herein fuses visual features and first text features during the training of a multimodal model. This allows the multimodal model to learn the semantic relationships between visual information and content-related text, as well as the relationships between different modal features. It integrates and complements visual information and text to make feature representation more comprehensive and complete. Furthermore, classification information is added during the training process. Since classification information can more accurately distinguish multimodal data with similar content, the second text features corresponding to the classification information are introduced and compared with the first fused features. This allows the multimodal model to more accurately learn the differences between different categories of multimodal data and to better extract key and detailed features from the multimodal data. Using this training method, the accuracy of feature extraction from multimodal data containing both visual and textual information can be improved, thereby enhancing the accuracy of downstream tasks. In addition, the fusion of visual information and content-related text during training also improves the generalization and transfer capabilities of the multimodal model, enhancing its performance on various downstream tasks.

[0036] The following is combined with Figures 1-5 The training method disclosed herein is described.

[0037] Figure 1 Flowcharts are shown for some embodiments of the training method disclosed herein. Figure 1 As shown, the method of this embodiment includes steps S102 to S110.

[0038] In step S102, the visual encoder of the multimodal model is used to process the visual information in multiple samples to obtain the visual features of each sample in the multiple samples.

[0039] In the training process of multimodal models, multiple samples can be identified first. When a multimodal model is applied to a specific domain, multiple samples within that specific domain can be identified. For example, multiple samples could be from categories such as lifestyle services or news and information. Training the multimodal model using multiple samples from a specific domain can improve the accuracy of the multimodal model in that domain.

[0040] Each sample is a multimodal sample, which may include visual and textual information. For example, visual information includes at least one of video and image. The multimodal model may include a visual encoder, a text encoder, and a first fusion module as its main components. The visual encoder can be used to extract visual features. For example, the visual information in each sample can be preprocessed to form an initial feature tensor, which is then input into the visual encoder to obtain the visual features of each sample as output. The visual features of each sample can be visual embeddings. For example, the visual encoder is a Vision Transformer (ViT), but is not limited to the examples given.

[0041] In step S104, the text encoder of the multimodal model is used to process the text related to the visual information in each sample to obtain the first text feature of each sample, and the classification information of each sample is processed to obtain the second text feature of each sample.

[0042] The text information for each sample includes text related to the content of the visual information and classification information. The text related to the content of the visual information can be carried within the visual information itself or within each sample. The classification information can be manually determined, such as a taxonomy title. The text encoder can use a BERT model, but is not limited to the examples given.

[0043] In some embodiments, text related to the content of visual information includes at least one of the following: title, text in an image, text obtained based on speech recognition, text obtained based on optical character recognition, caption, and descriptive text. Text in an image can be text within the visual information or text added to the visual information later, such as text in a sticker, text in the image background or foreground, etc. If each sample includes audio, the text can be obtained based on Automatic Speech Recognition (ASR). Text in the visual information can be recognized based on Optical Character Recognition (OCR). A description of the visual information can be a summary of the visual information, etc. Text related to the content of visual information can also include text corresponding to POIs (Points of Interest). Various types of information carried by visual information or samples that can be converted into text can be considered as text related to the content of the visual information, and are not limited to the examples mentioned above.

[0044] A text encoder can be used to extract text features. For each sample, if the text related to the visual information includes multiple texts, these multiple texts can be concatenated to form a first initial text tensor. This first initial text tensor is then input into the text encoder to obtain the first output text feature. After processing the text containing the classification information, a second initial text tensor is formed and input into the text encoder to obtain the second output text feature. The first text feature can be a first text embedding, and the second text feature can be a second text embedding.

[0045] In step S106, the first fusion module of the multimodal model is used to fuse the visual features and the first text features of each sample to obtain the first fused feature of each sample.

[0046] Since visual features and primary text features are inherent to each sample, fusing them can more comprehensively and accurately represent the features of each sample. Furthermore, during training, the multimodal model can learn the relationship between data from two different modalities, focusing on interrelated features. This allows for the extraction of more critical and accurate features from visual information and text related to the visual information, and also contributes to the generalization and transfer capabilities of the multimodal model.

[0047] In step S108, the contrast loss is determined based on the first fusion feature and the second text feature of multiple samples.

[0048] Contrastive loss includes the contrastive loss function and its value. Since the second text features are generated based on classification information, referencing classification information during the determination of contrastive loss and learning based on it enables the multimodal model to learn how to distinguish between samples of different categories but with similar content. This allows for the extraction of more accurate and detailed visual features and first text features from the samples, as well as obtaining more accurate first fusion features.

[0049] In step S110, the parameters of the multimodal model are adjusted according to the contrastive loss in order to train the multimodal model.

[0050] The parameters of the multimodal model can be adjusted iteratively using the contrastive loss function until an iteration stopping condition is met. This stopping condition could be reaching a preset number of iterations, the contrastive loss function value falling below a threshold, or performance on the validation set no longer improving. For example, in each iteration, the gradient is calculated based on the contrastive loss function, and an optimization algorithm is used to update the parameters of the multimodal model based on the gradient. Existing methods can be used for calculating the gradient and the optimization algorithm, which will not be elaborated upon here.

[0051] The method described in the above embodiments improves the training method for multimodal models. During the training process, visual features and first text features are fused, enabling the multimodal model to learn the semantic relationships between visual information and content-related text, as well as the relationships between different modal features. Visual information and text are integrated and complemented to make feature representation more comprehensive and complete. Furthermore, classification information is added during the training process. Since classification information can more accurately distinguish multimodal data with similar content, introducing the second text features corresponding to the classification information and comparing them with the first fused features allows the multimodal model to more accurately learn the differences between different categories of multimodal data, and to better extract key and detailed features from the multimodal data. Using the training method described in the above embodiments can improve the accuracy of feature extraction from multimodal data containing both visual and textual information, thereby improving the accuracy of downstream tasks. In addition, since visual information and content-related text are fused during training, the generalization and transfer capabilities of the multimodal model can also be improved, enhancing its performance on various downstream tasks.

[0052] The following describes how to use a visual encoder to determine the visual features of each sample.

[0053] Training samples can be divided into multiple batches, each containing multiple samples. One batch of samples can be used per iteration during training. If each sample includes multiple images or videos, a predetermined number of images can be extracted from these images or videos. Each image can be resized to a predetermined size. For example, if the number of channels in each image is represented by C, the height by H, and the width by W, then each image can be represented as a tensor [C, H, W], for example, with a predetermined size of 3. 224 224. If the number of samples in each batch is denoted as B, and the number of images in each sample is denoted as F, then the tensor of a batch of samples is [B, F, C, H, W]. Each image can be divided into multiple patches. For example, each image can be divided into 196 patches and input into a visual encoder.

[0054] If each sample includes multiple images (which can be multiple images extracted from a video), the amount of data processed by the multimodal model is large. In order to further improve processing efficiency, some improved methods for visual encoding of each sample are provided below.

[0055] In some embodiments, the visual information includes multiple images, each of which is divided into multiple blocks. The visual information in the multiple samples is processed using a visual encoder of a multimodal model to obtain the visual features of each sample, including: using the visual encoder to: determine the embedded encoding of multiple blocks of each image in each sample; and for each image in each sample, perform average pooling on the embedded encoding of the multiple blocks of the image to determine the embedded encoding of the image.

[0056] like Figure 2 As shown, dimensionality reduction methods can include two types. One is the average pooling method, which can utilize the average pooling module 21 in the multimodal model. The average pooling module 21 includes an average pooling layer 212. For each sample, the number of images in the sample is represented as F, each image is divided into P blocks, and the embedded encoding of each block is E. Then, the feature tensor 211 of each sample can be represented as [F, P, E]. The feature tensor 211 is input into the average pooling layer 212, and the average of the embedded encodings of the P blocks is taken to obtain the feature tensor 213, which can be represented as [F, E].

[0057] For example, the visual encoder is a Vision Transformer (ViT), but it is not limited to the examples given. In ViT, each block can be mapped to a vector space of a specific dimension through a linear projection layer, for example, a 768-dimensional vector. The feature vectors of multiple blocks in each image are added with the positional encoding and fed into the ViT encoder for further feature extraction and transformation. The final output feature vector of each block is 768-dimensional. A label vector cls can also be added to the ViT model. In the multimodal model, the model learns how to encode the overall information of the image into the feature vector of cls, and the output feature vector of cls can also be 768-dimensional. For example, if each image is divided into 196 blocks, each block can correspond to a 768-dimensional vector, and cls corresponds to a 768-dimensional vector, and each sample includes 5 images, then the feature tensor of each sample is [5, 197, 768]. After average pooling, the dimensionality is reduced to [5, 768], which greatly reduces the dimensionality of the features.

[0058] Average pooling can reduce the dimensionality of the embedded encoding of multiple blocks in each image, thereby reducing the computational cost and memory usage of the multimodal model and improving its operational efficiency. Furthermore, average pooling can aggregate features from multiple blocks to extract more representative features, helping the model to better learn the global information of the image.

[0059] In some embodiments, the visual information includes multiple images. A visual encoder using a multimodal model processes the visual information in the multiple samples to obtain the visual features of each sample. This includes, for each sample, using the visual encoder: determining the embedded codes of multiple images in the sample; concatenating the embedded codes of the multiple images of the sample with the initial embedded code corresponding to the sample to obtain the concatenated embedded code of the sample; fusing the concatenated embedded code and the initial embedded code of the sample based on a cross-attention mechanism to obtain a first fused code of the sample; and processing the first fused code of the sample based on a self-attention mechanism to determine the visual features of the sample.

[0060] The initial embedded encoding can be randomly generated and learned as a latent variable by the multimodal model, transforming the high-dimensional feature tensor of each sample into a low-dimensional feature tensor. For example... Figure 2As shown, another method for dimensionality reduction is to process the feature tensor of each sample using a perceptron 22. The perceptron 22 includes a cross-attention module 223, a self-attention module 224, and a feed-forward neural network (FFN) 225. The cross-attention module 223 may include one or more cross-attention layers, and the self-attention module 224 may include one or more self-attention layers. For each sample, the number of images in the sample is represented as F, each image is divided into P blocks, and the embedded encoding of each block is E. The feature tensor input to the cross-attention module 223 is the feature tensor [L+F*P,E] obtained by concatenating the embedded encodings of multiple images with the initial embedded encoding corresponding to the sample. L can be configured to be equal to F. The initial embedded encoding is used as a latent variable. After passing through the cross-attention module 223, the embedded encodings of multiple images of each sample can interact with the latent variable, updating the representation of the latent variable and fusing the features of multiple images of each sample to obtain the first fused encoding. The first fusion encoding, after passing through the self-attention module 224, can capture the long-term dependencies between latent variables, further optimizing the latent variables and better representing the features of multiple images for each sample. Furthermore, the FFN 225 can perform a non-linear transformation on the features output by the self-attention module 224, ultimately obtaining the visual features of each sample.

[0061] By using a perceptron to perform dimensionality reduction on the embedded encoding of multiple images for each sample, the computational cost and memory usage of the multimodal model are reduced, thus improving its operational efficiency. Furthermore, through cross-attention and self-attention mechanisms, the accuracy of the visual features obtained from dimensionality reduction can be improved, focusing on important features and enhancing the model's efficiency and generalization ability.

[0062] This disclosure provides several solutions for determining the comparative loss, which are described in detail below.

[0063] In some embodiments, determining the contrast loss based on the first fusion feature and the second text feature of multiple samples includes: for each sample, taking the sample as a positive sample and the other samples in the multiple samples as negative samples, determining the first similarity between the first fusion feature of the positive sample and the second text feature of the positive sample, and the second similarity between the first fusion feature of the positive sample and the second text feature of the negative sample; and determining the contrast loss based on the first similarity and the second similarity corresponding to the multiple samples.

[0064] The first and second similarity metrics can be cosine similarity or Euclidean distance similarity, and are not limited to the examples given. The method for calculating the contrastive loss and the training method based on this contrastive loss can be called the LabelDimension CLIP (Label Dimension Contrastive Language-Image Pre-training) method. Through contrastive loss, the first fused feature and second textual feature of positive samples can be made more similar, while the similarity between the first fused feature of positive samples and the second textual feature of negative samples is lower. The multimodal model can learn more detailed and key features to distinguish different categories, making the generated first fused feature more accurate. This, in turn, makes the multimodal model more accurate when used in downstream tasks. The specific form of the contrastive loss function and the similarity calculation method can be found in existing technologies and will not be elaborated here.

[0065] For example, based on the cross-attention mechanism, the visual features and the first text features of each sample are fused to obtain the first fused feature for each sample. Cross-attention can effectively fuse visual features and the first text features, allowing the multimodal model to interact with the visual features and the first text features to generate a more accurate and detailed first fused feature to represent each sample.

[0066] Further, more information can be added to the first fused feature to further improve the accuracy of multimodal model training. In some embodiments, the text encoder of the multimodal model is used to process the text related to the visual information content in each sample to obtain the first text feature of each sample, which includes: concatenating the text related to the visual information content and the classification information in each sample to obtain the concatenated text; and processing the concatenated text using the text encoder to obtain the first text feature of each sample.

[0067] like Figure 3 As shown, the visual information of each sample can be input into the visual encoder 301 to obtain the visual features of each sample, and the text information (the concatenated text related to the content of the visual information and the classification information of each sample) can be input into the text encoder 302 to obtain the first text features. Then, the visual features and the first text features of each sample are input into the first fusion module 303 to obtain the first fusion features. The contrast loss module 304 can determine the contrast loss based on the first fusion features and the second text features of each sample.

[0068] Because the first text feature incorporates categorical information features, the first fused feature also incorporates categorical information features. During training, the multimodal model learns categorical information features alongside visual features and the first text feature, as well as the relationships between these three types of features. This allows the multimodal model to focus more on extracting visual and first text features related to categorical information, thereby improving the accuracy of the multimodal model and the accuracy of feature extraction for each sample.

[0069] In some embodiments, the first fusion module of the multimodal model is used to fuse the visual features and the first text features of each sample to obtain the first fused feature of each sample. This includes using the first fusion module to fuse the visual features and the first text features of each sample based on a cross-attention mechanism to obtain the first fused feature of each sample.

[0070] For example, the first fusion module may include one or more cross-attention layers. The visual features of each sample can be used as the query matrix Q, and the first textual features of each sample as the key matrix K and value matrix V. For each sample, the similarity between the query matrix Q and the key matrix K is calculated to obtain an attention score; the attention score is normalized using a softmax function to obtain an attention distribution; and the first fusion feature of the sample is determined based on the product of the attention distribution and the value matrix V.

[0071] If the first fusion module includes multiple cross-attention layers, for each sample, the output of the previous cross-attention layer can be used as the query matrix of the next cross-attention layer, the first text features are used as the key matrix K and the value matrix V, and the output of the last cross-attention layer can take the features of the first layer as the first fusion feature of the sample. For example, if the feature tensor output after one or more cross-attention layers is [B, L, D], the features of the first layer are taken as [B, D].

[0072] Cross-attention can effectively fuse visual features and first text features, enabling multimodal models to interact with visual features and first text features to generate more accurate and detailed first fused features to represent each sample.

[0073] The following section introduces another method for determining contrast loss.

[0074] In some embodiments, determining the contrast loss based on the first fusion feature and the second text feature of multiple samples includes: fusing the visual features and the second text feature of each sample using the second fusion module of the multimodal model to obtain the second fusion feature of each sample; and determining the contrast loss based on the first fusion feature and the second fusion feature of multiple samples.

[0075] The second fusion feature combines visual information features and classification information features. During training, it enables the multimodal model to learn the relationship between the two types of visual and classification information, focusing on interrelated features, extracting more critical features that more accurately represent the samples from both visual and classification information, and contributing to the generalization and transfer capabilities of the subsequent multimodal model.

[0076] In some embodiments, determining the contrast loss based on the first fusion feature and the second fusion feature of multiple samples includes: for each sample, taking the sample as a positive sample and the other samples in the multiple samples as negative samples, determining the third similarity between the first fusion feature of the positive sample and the second fusion feature of the positive sample, and the fourth similarity between the first fusion feature of the positive sample and the second fusion feature of the negative sample; and determining the contrast loss based on the third similarity and the fourth similarity corresponding to the multiple samples.

[0077] The third and fourth similarity metrics, such as cosine similarity or Euclidean distance similarity, are not limited to the examples given. The method for calculating the contrastive loss described above, and the method for training based on this contrastive loss, can be called the CaseDimension CLIP (Case Dimension Contrastive Language-Image Pre-training) method. Through contrastive loss, the first and second fusion features of positive samples can be made more similar, while the similarity between the first fusion feature of positive samples and the second fusion feature of negative samples is lower. The multimodal model can learn more detailed and key features to distinguish different classifications, making the generated visual features, first text features, and first fusion features more accurate. This, in turn, makes the multimodal model more accurate when used in downstream tasks. The specific form of the contrastive loss function and the similarity calculation method can be found in existing technologies and will not be elaborated here.

[0078] like Figure 4 As shown, the visual information of each sample can be input into the visual encoder 401 to obtain the visual features of each sample. The text related to the content of the visual information of each sample can be input into the text encoder 402 to obtain the first text features. The classification information of each sample can be input into the text encoder 402 to obtain the second text features. Then, the visual features and the first text features of each sample are input into the first fusion module 403 to obtain the first fused features. The visual features and the second text features of each sample are input into the second fusion module 404 to obtain the second fused features. The contrast loss module 405 can determine the contrast loss based on the first fused features and the second fused features of each sample.

[0079] In some embodiments, the second fusion module of the multimodal model is used to fuse the visual features and the second text features of each sample to obtain the second fused feature of each sample. This includes using the second fusion module to fuse the visual features and the second text features of each sample based on a cross-attention mechanism to obtain the second fused feature of each sample.

[0080] For example, the second fusion module may include one or more cross-attention layers. The visual features of each sample can be used as the query matrix Q, and the second textual features of each sample as the key matrix K and value matrix V. For each sample, the similarity between the query matrix Q and the key matrix K is calculated to obtain an attention score; the attention score is normalized using a softmax function to obtain an attention distribution; and the second fusion feature of the sample is determined based on the product of the attention distribution and the value matrix V.

[0081] If the second fusion module includes multiple cross-attention layers, for each sample, the output of the previous cross-attention layer can be used as the query matrix of the next cross-attention layer, the first text features are used as the key matrix K and the value matrix V, and the output of the last cross-attention layer can take the features of the first layer as the second fusion feature of the sample.

[0082] Cross-attention can effectively fuse visual features and second text features, enabling multimodal models to interact with visual features and second text features, learn features in visual information that are more relevant to classification information, and thus extract more accurate and detailed visual features from each sample.

[0083] Another method for determining contrast loss is introduced below.

[0084] In some embodiments, determining the contrast loss based on the first fusion feature and the second text feature of multiple samples includes: determining a first contrast loss based on the first fusion feature and the second text feature of multiple samples; fusing the visual features and the second text feature of each sample using the second fusion module of the multimodal model to obtain a second fusion feature for each sample; determining a second contrast loss based on the first fusion feature and the second fusion feature of multiple samples; and determining a contrast loss based on the first contrast loss and the second contrast loss.

[0085] The method for calculating the contrast loss and the training method based on this contrast loss described above is a combination of the LabelDimension CLIP and CaseDimension CLIP methods. For example, the text and classification information related to the visual information in each sample are concatenated to obtain concatenated text; a text encoder is used to process the concatenated text to obtain the first text feature for each sample; then, based on the visual features and the first text feature of each sample, a first fused feature is obtained. The calculation methods for the first and second contrast losses can refer to the aforementioned embodiments and will not be repeated here. The first and second contrast losses can be weighted and summed, or directly summed, to obtain the total contrast loss.

[0086] Because the Label Dimension CLIP and Case Dimension CLIP methods add reference to classification information in the contrastive learning process in different ways, multimodal models can learn the relationship between visual information features, text features related to the content of visual information, and classification information features from different perspectives. This allows for a more comprehensive capture of detailed features related to classification information in visual and text features, thereby improving the accuracy of feature extraction for each sample.

[0087] In addition to using contrastive loss to train the multimodal model in the above embodiments, the training method can be further improved to further enhance the accuracy of the text encoder.

[0088] In some embodiments, a portion of the text related to the visual information content in each sample is replaced with a mask marker to obtain the mask text for each sample; the mask text for each sample is processed using a text encoder to obtain the prediction result corresponding to the mask marker; the mask language model loss is determined based on the prediction result and the portion of text corresponding to each sample; wherein, adjusting the parameters of the multimodal model based on the contrast loss includes: adjusting the parameters of the multimodal model based on the contrast loss and the mask language model loss.

[0089] For each sample, a mask is used to mask a portion of the text related to the visual information. This mask is then predicted by the text encoder. Based on the prediction result and the corresponding portion of text for each sample, the Masked Language Model Loss (MLM Loss) is determined. Introducing the Masked Language Model Loss into the training process of the multimodal model allows the text encoder to understand and predict the masked content through context, which helps improve the model's ability to understand natural language. This, in turn, improves the accuracy of encoding text related to the visual information, and overall improves the accuracy of feature extraction for each sample by the multimodal model.

[0090] like Figure 5 As shown, during training, the multimodal model includes a visual encoder 501, a text encoder 502, a first fusion module 503, a second fusion module 504, and a contrast loss module 505. Visual information from each sample is input to the visual encoder 501 to obtain visual features for each sample. Text related to the content of the visual information in each sample is input to the text encoder 502 to obtain first text features for each sample. Classification information from each sample is input to the text encoder 502 to obtain second text features for each sample. The concatenated text related to the content of the visual information and classification information in each sample is input to the text encoder 502 to obtain third text features for each sample (in the aforementioned embodiment, the first text features of classification information are added). The visual features and first text features of each sample are input to the first fusion module 503 to obtain first fused features. The visual features and second text features of each sample are input to the second fusion module 504 to obtain second fused features. The visual features and third text features of each sample are then processed by the first fusion module 503 to obtain third fused features (in the aforementioned embodiment, the first fused features of classification information are fused). In the contrast loss module 505, a first contrast loss is determined based on the third fusion feature and the second text feature of each sample, and a second contrast loss is determined based on the first fusion feature and the second fusion feature of each sample. Additionally, an MLM loss is determined. The first contrast loss, the second contrast loss, and the MLM loss are weighted and summed to obtain the total loss. The parameters of the multimodal model are adjusted based on the total loss to train the multimodal model. Specific details can be found in the aforementioned embodiments and will not be repeated here.

[0091] After the multimodal model is trained, it can be used to extract features from any multimedia content (multimodal data) that includes both visual and textual information. The feature extraction method of this disclosure is described below.

[0092] Figure 6 Flowcharts are shown for some embodiments of the feature extraction method of this disclosure. For example... Figure 6 As shown, the method of this embodiment includes steps S602 to S606.

[0093] In step 602, the visual encoder of the multimodal model is used to process the visual information in the multimedia content to be processed, and the visual features of the multimedia content to be processed are obtained.

[0094] The multimodal model is trained using the training method of any embodiment of this disclosure.

[0095] In step 604, a multimodal model text encoder is used to process the text related to visual information in the multimedia content to be processed, thereby obtaining the text features of the multimedia content to be processed.

[0096] In step 606, the first fusion module of the multimodal model is used to fuse the visual features and text features of the multimedia content to be processed to obtain the fused features of the multimedia content to be processed.

[0097] like Figure 7 As shown, the multimodal model mainly uses a visual encoder 701, a text encoder 702, and a first fusion module 703, which will not be described in detail here. Previously, the dimensionality reduction method in the aforementioned embodiments can still be used in the visual encoder 701, that is, the visual encoder 701 includes an average pooling module or a perceptron. The processing of visual features of the multimedia content to be processed can refer to the aforementioned embodiments, which will not be described in detail here.

[0098] Since the training process of multimodal models improves their accuracy, multimodal models can be used to extract fusion features more accurately for any multimedia content to be processed, and to more accurately represent the features of visual and textual information in the multimedia content to be processed, which helps to improve the accuracy of downstream tasks.

[0099] In some embodiments, classification information of the multimedia content to be processed is determined based on the fusion characteristics of the multimedia content to be processed.

[0100] A classifier can be added after the multimodal model to classify the multimedia content to be processed. Since the multimodal model focuses on learning the features associated with classification information in the multimodal data during training, using the multimodal model to extract the fused features of the multimedia content to be processed before classification improves the accuracy of the classification.

[0101] In some embodiments, response information is generated based on the fusion characteristics of the multimedia content to be processed and information input by the user.

[0102] For example, users can ask questions about the multimedia content to be processed, and the system can generate response information. Since the multimodal model learns how to extract key features from multimodal data during training, using the multimodal model to extract the fused features of the multimedia content to be processed, and then interacting with the user based on these fused features, can improve the accuracy of the interaction.

[0103] In some embodiments, it is determined whether to recommend multimedia content to the user based on the fusion characteristics of the multimedia content to be processed and the user's historical interaction data.

[0104] For example, a user's historical interaction data can include multimedia content viewed in the past, text of historical interaction behaviors, etc. The fusion characteristics of the multimedia content to be processed can be compared with the characteristics of historical interaction data to determine whether to recommend the multimedia content to be processed to the user.

[0105] After extracting the fusion features of the multimedia content to be processed using a multimodal model, the fusion features are compared with other features to determine whether to recommend the multimedia content to the user, which can improve the accuracy of the recommendation.

[0106] In some embodiments, it is determined whether the multimedia content to be processed matches the search term based on the fusion characteristics of the multimedia content to be processed and the search term.

[0107] For example, the similarity between the fusion features of the multimedia content to be processed and the features of the search term can be compared to determine whether the multimedia content to be processed should be used as the search result.

[0108] After extracting the fusion features of the multimedia content to be processed using a multimodal model, the fusion features are compared with the search terms to improve the accuracy of the search.

[0109] In some embodiments, introductory text or introductory audio for the multimedia content to be processed is generated based on the fusion characteristics of the multimedia content to be processed.

[0110] Multimodal models can be used to extract key features from the multimedia content to be processed. These features can then be used to summarize the multimedia content and generate introductory text or audio, thereby improving accuracy.

[0111] Furthermore, the trained multimodal model can be used for a variety of downstream tasks, not limited to the examples mentioned above. Multimodal models can also be used to process unimodal data. For example, for images, they can be input into a visual encoder to obtain visual features. In the first fusion module, these visual features can be processed based on a self-attention mechanism to obtain the features of the output image, which can then be used for subsequent downstream tasks. Similarly, features can also be extracted from plain text content using a multimodal model, which will not be elaborated further here.

[0112] This disclosure also provides a training device, which is described below in conjunction with... Figure 8 Describe it.

[0113] Figure 8 These are structural diagrams of some embodiments of the training device disclosed herein. Figure 8 As shown, the training device 80 of this embodiment includes: a visual encoding unit 810, a text encoding unit 820, a first fusion unit 830, a determination unit 840, and an adjustment unit 850.

[0114] The visual encoding unit 810 is configured to use a multimodal model visual encoder to process visual information in multiple samples to obtain visual features of each sample in the multiple samples.

[0115] The text encoding unit 820 is configured to use a multimodal model text encoder to process the text related to the visual information in each sample to obtain the first text feature of each sample, and to process the classification information of each sample to obtain the second text feature of each sample.

[0116] The first fusion unit 830 is configured to use the first fusion module of the multimodal model to fuse the visual features and the first text features of each sample to obtain the first fused feature of each sample.

[0117] The determination unit 840 is configured to determine the contrast loss based on the first fusion feature and the second text feature of multiple samples.

[0118] The adjustment unit 850 is configured to adjust the parameters of the multimodal model based on the contrastive loss in order to train the multimodal model.

[0119] The training apparatus described in the above embodiments fuses visual features and first text features during the training of the multimodal model. This allows the multimodal model to learn the semantic relationships between visual information and content-related text, as well as the relationships between different modal features. It integrates and complements visual information and text to make feature representation more comprehensive and complete. Furthermore, classification information is added during the training process. Since classification information can more accurately distinguish multimodal data with similar content, introducing the second text features corresponding to the classification information and comparing them with the first fused features allows the multimodal model to more accurately learn the differences between different categories of multimodal data, and to better extract key and detailed features from the multimodal data. Using the training apparatus described in the above embodiments can improve the accuracy of feature extraction from multimodal data containing both visual and textual information, thereby improving the accuracy of downstream tasks. In addition, because visual information and content-related text are fused during training, the generalization and transfer capabilities of the multimodal model can also be improved, enhancing its performance on various downstream tasks.

[0120] In some embodiments, the determining unit 840 is configured to, for each sample, take the sample as a positive sample and other samples among the multiple samples other than the sample as negative samples, determine a first similarity between the first fusion feature of the positive sample and the second text feature of the positive sample, and a second similarity between the first fusion feature of the positive sample and the second text feature of the negative sample; and determine a contrast loss based on the first similarity and the second similarity corresponding to the multiple samples.

[0121] In some embodiments, the training apparatus further includes a second fusion unit 860 configured to fuse the visual features and second text features of each sample using the second fusion module of the multimodal model to obtain a second fused feature for each sample; and a determination unit 840 configured to determine a contrast loss based on the first fused feature and the second fused feature of multiple samples.

[0122] In some embodiments, the determining unit 840 is configured to, for each sample, take the sample as a positive sample and the other samples among the plurality of samples other than the sample as negative samples, determine a third similarity between the first fusion feature of the positive sample and the second fusion feature of the positive sample, and a fourth similarity between the first fusion feature of the positive sample and the second fusion feature of the negative sample; and determine a contrast loss based on the third similarity and the fourth similarity corresponding to the plurality of samples.

[0123] In some embodiments, the training apparatus further includes a second fusion unit 860 configured to fuse the visual features and second text features of each sample using the second fusion module of the multimodal model to obtain a second fusion feature for each sample; and a determination unit 840 configured to determine a first contrast loss based on the first fusion feature and second text features of multiple samples; determine a second contrast loss based on the first fusion feature and second fusion feature of multiple samples; and determine a contrast loss based on the first contrast loss and the second contrast loss.

[0124] The text encoding unit 820 is configured to concatenate the text related to the visual information and the classification information in each sample to obtain the concatenated text; and to process the concatenated text using a text encoder to obtain the first text feature of each sample.

[0125] In some embodiments, the first fusion unit 830 is configured to utilize a first fusion module to fuse the visual features and first text features of each sample based on a cross-attention mechanism to obtain a first fused feature for each sample.

[0126] In some embodiments, the second fusion unit 860 is configured to fuse the visual features and second text features of each sample based on a cross-attention mechanism to obtain a second fused feature for each sample.

[0127] In some embodiments, the determining unit 840 is further configured to replace a portion of the text in the text related to the visual information content in each sample with a mask marker to obtain a mask text for each sample; process the mask text of each sample using a text encoder to obtain a prediction result corresponding to the mask marker; determine the mask language model loss based on the prediction result and the portion of text corresponding to each sample; and the adjusting unit 850 is further configured to adjust the parameters of the multimodal model based on the contrast loss and the mask language model loss.

[0128] In some embodiments, the visual information includes multiple images, each of which is divided into multiple blocks, and the visual encoding unit 810 is configured to use a visual encoder to: determine the embedded encoding of multiple blocks of each image in each sample; and, for each image in each sample, perform average pooling on the embedded encoding of multiple blocks of the image to determine the embedded encoding of the image.

[0129] In some embodiments, the visual information includes multiple images, and the visual encoding unit 810 is configured to use a visual encoder to, for each sample: determine the embedded encoding of multiple images in the sample; concatenate the embedded encoding of the multiple images of the sample with the initial embedded encoding corresponding to the sample to obtain the concatenated embedded encoding of the sample; fuse the concatenated embedded encoding and the initial embedded encoding of the sample based on a cross-attention mechanism to obtain a first fused encoding of the sample; and process the first fused encoding of the sample based on a self-attention mechanism to determine the visual features of the sample.

[0130] In some embodiments, the visual information includes images or videos, and the text related to the content of the visual information in each sample includes at least one of the following: a title, text in an image, text obtained based on speech recognition, text obtained based on optical character recognition, a caption, and descriptive text.

[0131] This disclosure also provides a feature extraction device, which is described below in conjunction with... Figure 9 Describe it.

[0132] Figure 9 These are structural diagrams of some embodiments of the feature extraction apparatus of this disclosure. Figure 9 As shown, the feature extraction device 90 of this embodiment includes: a visual encoding unit 910, a text encoding unit 920, and a fusion unit 930.

[0133] The visual encoding unit 910 is configured to use a multimodal model visual encoder to process the visual information in the multimedia content to be processed, and obtain the visual features of the multimedia content to be processed.

[0134] The multimodal model is trained using the training method of any embodiment of this disclosure;

[0135] The text encoding unit 920 is configured to use a multimodal model text encoder to process the text related to visual information in the multimedia content to be processed, and obtain the text features of the multimedia content to be processed.

[0136] The fusion unit 930 is configured to use a first fusion module of a multimodal model to fuse the visual and textual features of the multimedia content to be processed to obtain the fused features of the multimedia content to be processed.

[0137] Since the training process of multimodal models improves their accuracy, multimodal models can be used to extract fusion features more accurately for any multimedia content to be processed, and to more accurately represent the features of visual and textual information in the multimedia content to be processed, which helps to improve the accuracy of downstream tasks.

[0138] In some embodiments, the feature extraction device 90 further includes an execution unit 940, configured to perform at least one of the following: determining classification information of the multimedia content to be processed based on its fusion features; generating response information based on the fusion features of the multimedia content to be processed and information input by the user; determining whether to recommend the multimedia content to be processed to the user based on the fusion features of the multimedia content to be processed and the user's historical interaction data; determining whether the multimedia content to be processed matches the search term based on the fusion features of the multimedia content to be processed and the search term; and generating introductory text or introductory audio for the multimedia content to be processed based on its fusion features.

[0139] This disclosure also provides a multimodal information processing system. For example... Figure 10 As shown, the multimodal information processing system 10 includes a training device 80 and a feature extraction device 90.

[0140] This disclosed multimodal information processing system integrates visual features and first text features during the training process of a multimodal model. This allows the multimodal model to learn the semantic relationships between visual information and content-related text, as well as the relationships between different modal features. The system integrates and complements visual information and text to make feature representation more comprehensive and complete. Furthermore, classification information is added during the training process. Since classification information can more accurately distinguish multimodal data with similar content, the second text features corresponding to the classification information are introduced and compared with the first fused features. This allows the multimodal model to more accurately learn the differences between different categories of multimodal data and to better extract key and detailed features from the multimodal data. In addition, the integration of visual information and content-related text during training improves the generalization and transfer capabilities of the multimodal model, thereby enhancing its performance on various downstream tasks. Since the training process of multimodal models improves their accuracy, multimodal models can be used to extract fusion features more accurately for any multimedia content to be processed, and to more accurately represent the features of visual and textual information in the multimedia content to be processed, which helps to improve the accuracy of downstream tasks.

[0141] Figure 11 A block diagram of an electronic device according to some embodiments of the present disclosure is shown. The electronic device of the present disclosure includes: a processor 111; and a memory 112 coupled to the processor for storing instructions, which, when executed by the processor, cause the processor to perform a training method as described in any embodiment of the present disclosure, and / or a feature extraction method according to any embodiment of the present disclosure.

[0142] Memory 111 is used to store one or more computer-readable instructions. Memory 111 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Memory 111 may, for example, store operating systems, applications, bootloaders, databases, and other programs, as well as various applications and various data.

[0143] The processor 112 is configured to execute computer-readable instructions to implement the song selection method of any of the foregoing embodiments or the method of any of the foregoing embodiments. Specific implementations of each step of the method can be found in the above embodiments, and repeated details will not be elaborated here.

[0144] Processor 112 can be configured to execute Figures 1 to 9 The processor 112 can be various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The central processing unit (CPU) can be based on x86 or ARM architectures, etc.

[0145] The processor 112 and the memory 111 can communicate with each other directly or indirectly. For example, the processor 112 and the memory 111 can communicate via a network. The network can include a wireless network, a wired network, and / or any combination of wireless and wired networks. The processor 112 and the memory 111 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0146] It should be noted that Figure 11 The components of the electronic device 11 shown are merely exemplary and not limiting; the electronic device 11 may have other components as needed for the actual application. The processor 112 can control other components in the electronic device 11 to perform desired functions.

[0147] Electronic device 11 can be implemented by software, firmware and / or hardware, and can be integrated into a device with relevant applications installed.

[0148] The electronic device disclosed herein fuses visual features and first text features during the training process of a multimodal model. This allows the multimodal model to learn the semantic association between visual information and content-related text, as well as the relationships between different modal features. It integrates and complements visual information and text to make feature representation more comprehensive and complete. Furthermore, classification information is added during the training process. Since classification information can more accurately distinguish multimodal data with similar content, introducing the second text features corresponding to the classification information and comparing them with the first fused features allows the multimodal model to more accurately learn the differences between different categories of multimodal data, and to better extract key and detailed features of the multimodal data. In addition, the fusion of visual information and content-related text during training improves the generalization and transfer capabilities of the multimodal model, thereby enhancing its performance on various downstream tasks. Because the training process of the multimodal model improves its accuracy, for any multimedia content to be processed, the multimodal model can more accurately extract fused features, more accurately representing the features of visual and textual information in the multimedia content to be processed, thus contributing to improved accuracy in downstream tasks.

[0149] Figure 12 Block diagrams of electronic devices according to other embodiments of the present disclosure are shown.

[0150] Figure 12 The electronic device 12 shown can be a computer system with a dedicated hardware structure, which can perform corresponding functions when the relevant application is installed.

[0151] Electronic devices include, but are not limited to, mobile terminals such as smartphones, laptops, personal digital assistants (PDAs), tablet computers (PCs), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), wearable devices, and fixed terminals such as digital televisions and desktop computers.

[0152] like Figure 12As shown, the Central Processing Unit (CPU) 121 performs various processes based on a program stored in the Read-Only Memory (ROM) 122 or a program loaded from the storage section 128 into the Random Access Memory (RAM) 123. The RAM 123 stores data required as needed when the CPU 121 performs various processes, etc. The CPU is merely exemplary; it could also be other types of processors, such as the various processors described above. The ROM 122, RAM 123, and storage section 128 can be various forms of computer-readable storage media. It should be noted that although... Figure 12 The image shows ROM 122, RAM 123 and storage section 128, but one or more of them may be combined or located in the same or different memory or storage modules.

[0153] CPU 121, ROM 122 and RAM 123 are interconnected via bus 124. Input / output interface 125 is also connected to bus 124.

[0154] The following components are connected to the input / output interface 125: input section 126, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output section 127, including displays such as cathode ray tube (CRT), liquid crystal display (LCD), speakers, vibrators, etc.; storage section 128, including hard disk, magnetic tape, etc.; and communication section 129, including network interface cards such as LAN cards, modems, etc. The communication section 129 allows communication processing to be performed via a network such as the Internet. It is easy to understand that, although... Figure 12 The portion of the electronic device 12 shown communicates via bus 124, but it may also communicate via a network or other means, wherein the network may include a wireless network, a wired network, and / or any combination of wireless and wired networks.

[0155] As needed, drive 1210 is also connected to input / output interface 125. Removable media 1211, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 1210 as needed, so that computer programs read from them can be installed into storage section 128 as needed.

[0156] When the above series of processes are implemented through software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as a removable medium 1211.

[0157] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product that, when run on a computer, causes the computer to perform the methods of any of the foregoing embodiments. The computer program product includes computer instructions carried on a computer-readable medium, comprising program code for performing the methods shown in the flowcharts. In such embodiments, the computer instructions can be downloaded and installed from a network via communication section 129, or installed from storage section 128, or installed from ROM 122. When the computer program is executed by CPU 121, the methods of embodiments of this disclosure are performed.

[0158] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein, when executed by a processor, the program causes the processor to perform a training method as described in any embodiment of this disclosure, and / or a feature extraction method as described in any embodiment of this disclosure.

[0159] This disclosed computer-readable storage medium fuses visual features and first text features during the training process of a multimodal model. This enables the multimodal model to learn the semantic association between visual information and content-related text, as well as the relationships between different modal features. It integrates and complements visual information and text to make the feature representation more comprehensive and complete. Furthermore, classification information is added during the training process. Since classification information can more accurately distinguish multimodal data with similar content, introducing the second text features corresponding to the classification information and comparing them with the first fused features allows the multimodal model to more accurately learn the differences between different categories of multimodal data, and to better extract key and detailed features of the multimodal data. In addition, the fusion of visual information and content-related text during training improves the generalization and transfer capabilities of the multimodal model, thereby enhancing its performance on various downstream tasks. Because the training process of the multimodal model improves its accuracy, for any multimedia content to be processed, the multimodal model can more accurately extract fused features and more accurately represent the features of visual and textual information in the multimedia content to be processed, contributing to improved accuracy in downstream tasks.

[0160] It should be noted that, in the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0161] A computer-readable medium may be a computer-readable storage medium, a computer-readable signal medium, or any combination thereof.

[0162] Computer-readable storage media include, but are not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the methods of any of the foregoing embodiments.

[0163] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0164] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0165] This disclosure also provides a computer program product, comprising: instructions, wherein when the program is executed by a processor, the processor performs a training method as described in any embodiment of this disclosure, and / or a feature extraction method as described in any embodiment of this disclosure.

[0166] The computer program product disclosed herein fuses visual features and first text features during the training process of a multimodal model. This enables the multimodal model to learn the semantic association between visual information and content-related text, as well as the relationships between different modal features. It integrates and complements visual information and text to make feature representation more comprehensive and complete. Furthermore, classification information is added during the training process. Since classification information can more accurately distinguish multimodal data with similar content, introducing the second text features corresponding to the classification information and comparing them with the first fused features allows the multimodal model to more accurately learn the differences between different categories of multimodal data, and to better extract key and detailed features from the multimodal data. In addition, the fusion of visual information and content-related text during training improves the generalization and transfer capabilities of the multimodal model, thereby enhancing its performance on various downstream tasks. Because the training process of the multimodal model improves its accuracy, for any multimedia content to be processed, the multimodal model can more accurately extract fused features, more accurately representing the features of visual and textual information in the multimedia content to be processed, thus contributing to improved accuracy in downstream tasks.

[0167] In some embodiments, a computer program is also provided, comprising: instructions that, when executed by a processor, cause the processor to perform the methods described in any of the foregoing embodiments. For example, the instructions may be embodied in computer program code.

[0168] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0170] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0171] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A training method, comprising: A visual encoder based on a multimodal model is used to process visual information from multiple samples to obtain visual features for each sample. Using the text encoder of the multimodal model, the text related to the visual information in each sample is processed to obtain the first text feature of each sample, and the classification information of each sample is processed to obtain the second text feature of each sample. The first fusion module of the multimodal model is used to fuse the visual features and the first text features of each sample to obtain the first fused feature of each sample. The contrast loss is determined based on the first fusion feature and the second text feature of the multiple samples; The parameters of the multimodal model are adjusted based on the contrast loss to train the multimodal model.

2. The training method according to claim 1, wherein, The determination of the contrast loss based on the first fusion feature and the second text feature of the multiple samples includes: For each sample, the sample is designated as a positive sample, and the other samples in the plurality of samples are designated as negative samples. The first similarity between the first fusion feature of the positive sample and the second text feature of the positive sample, and the second similarity between the first fusion feature of the positive sample and the second text feature of the negative sample are determined. The contrast loss is determined based on the first similarity and the second similarity corresponding to the multiple samples.

3. The training method according to claim 1, wherein, The determination of the contrast loss based on the first fusion feature and the second text feature of the multiple samples includes: The second fusion module of the multimodal model is used to fuse the visual features and the second text features of each sample to obtain the second fused feature of each sample. The contrast loss is determined based on the first fusion feature and the second fusion feature of the multiple samples.

4. The training method according to claim 3, wherein, The step of determining the contrast loss based on the first fusion feature and the second fusion feature of the multiple samples includes: For each sample, the sample is designated as a positive sample, and the other samples in the plurality of samples are designated as negative samples. The third similarity between the first fusion feature of the positive sample and the second fusion feature of the positive sample, and the fourth similarity between the first fusion feature of the positive sample and the second fusion feature of the negative sample are determined. The contrast loss is determined based on the third and fourth similarities corresponding to the multiple samples.

5. The training method according to claim 1, wherein, The determination of the contrast loss based on the first fusion feature and the second text feature of the multiple samples includes: The first contrast loss is determined based on the first fusion feature and the second text feature of the multiple samples; The second fusion module of the multimodal model is used to fuse the visual features and the second text features of each sample to obtain the second fused feature of each sample. The second contrast loss is determined based on the first and second fusion features of the multiple samples; The contrast loss is determined based on the first contrast loss and the second contrast loss.

6. The training method according to claim 2 or 5, wherein, The text encoder using the multimodal model processes the text related to the visual information content in each sample to obtain the first text feature of each sample, including: The text and classification information related to the visual information in each sample are concatenated to obtain the concatenated text. The concatenated text is processed using the text encoder to obtain the first text feature of each sample.

7. The training method according to any one of claims 1 to 5, wherein, The first fusion module of the multimodal model fuses the visual features and the first text features of each sample to obtain the first fused feature of each sample, including using the first fusion module: Based on the cross-attention mechanism, the visual features and the first text features of each sample are fused to obtain the first fused feature of each sample.

8. The training method according to claim 3 or 5, wherein, The second fusion module of the multimodal model fuses the visual features and second text features of each sample to obtain the second fused feature of each sample, including using the second fusion module: Based on the cross-attention mechanism, the visual features and second text features of each sample are fused to obtain the second fused feature of each sample.

9. The training method according to any one of claims 1 to 5, further comprising: The text related to the visual information content in each sample is replaced with a mask mark to obtain the mask text of each sample; The text encoder is used to process the masked text of each sample to obtain the prediction result corresponding to the masked marker. The masked language model loss is determined based on the prediction results and partial text corresponding to each sample. The step of adjusting the parameters of the multimodal model based on the contrast loss includes: The parameters of the multimodal model are adjusted based on the contrast loss and the masked language model loss.

10. The training method according to any one of claims 1 to 5, wherein, The visual information includes multiple images, each of which is divided into multiple blocks. The visual encoder using a multimodal model processes the visual information from the multiple samples to obtain the visual features of each sample, including... Using the visual encoder: Determine the embedded encoding of multiple blocks in each image within each sample; For each image in each sample, the embedded codes of multiple blocks of the image are averaged and pooled to determine the embedded codes of the image.

11. The training method according to any one of claims 1 to 5, wherein, The visual information includes multiple images. The visual encoder using a multimodal model processes the visual information from the multiple samples to obtain the visual features of each sample, including... Using the visual encoder, for each sample: Determine the embedded encoding of multiple images in the sample; The embedded codes of multiple images of the sample are concatenated with the initial embedded codes corresponding to the sample to obtain the concatenated embedded codes of the sample. Based on the cross-attention mechanism, the concatenated embedded code and the initial embedded code of the sample are fused to obtain the first fused code of the sample; Based on the self-attention mechanism, the first fusion encoding of the sample is processed to determine the visual features of the sample.

12. The training method according to any one of claims 1 to 5, wherein, The visual information includes images or videos, and the text related to the content of the visual information in each sample includes at least one of the following: title, text in the image, text obtained based on speech recognition, text obtained based on optical character recognition, subtitle, and descriptive text.

13. A feature extraction method, comprising: A visual encoder based on a multimodal model is used to process visual information in multimedia content to be processed, thereby obtaining visual features of the multimedia content to be processed. The multimodal model is trained using the training method described in any one of claims 1 to 12. The text encoder of the multimodal model is used to process the text related to the visual information in the multimedia content to be processed, so as to obtain the text features of the multimedia content to be processed. The first fusion module of the multimodal model is used to fuse the visual features and text features of the multimedia content to be processed to obtain the fused features of the multimedia content to be processed.

14. The feature extraction method according to claim 13, further comprising at least one of the following: Based on the fusion characteristics of the multimedia content to be processed, the classification information of the multimedia content to be processed is determined; Based on the fusion characteristics of the multimedia content to be processed and the information input by the user, a response message is generated; Based on the fusion characteristics of the multimedia content to be processed and the user's historical interaction data, determine whether to recommend the multimedia content to be processed to the user; Based on the fusion characteristics of the multimedia content to be processed and the search terms, determine whether the multimedia content to be processed matches the search terms; Based on the fusion characteristics of the multimedia content to be processed, an introductory text or audio for the multimedia content to be processed is generated.

15. A training device, comprising: The visual encoding unit is configured to use a multimodal model visual encoder to process visual information in multiple samples to obtain visual features of each sample in the multiple samples. The text encoding unit is configured to use the text encoder of the multimodal model to process the text related to the visual information in each sample to obtain the first text feature of each sample, and to process the classification information of each sample to obtain the second text feature of each sample. The first fusion unit is configured to use the first fusion module of the multimodal model to fuse the visual features and the first text features of each sample to obtain the first fused feature of each sample; The determining unit is configured to determine a contrast loss based on a first fusion feature and a second text feature of the plurality of samples; The adjustment unit is configured to adjust the parameters of the multimodal model according to the contrastive loss in order to train the multimodal model.

16. A feature extraction apparatus, comprising: A visual encoding unit is configured to use a multimodal model visual encoder to process visual information in multimedia content to be processed, thereby obtaining visual features of the multimedia content to be processed, wherein the multimodal model is trained using the training method described in any one of claims 1 to 12. The text encoding unit is configured to use the text encoder of the multimodal model to process the text in the multimedia content to be processed that is related to the visual information, so as to obtain the text features of the multimedia content to be processed. The fusion unit is configured to use the first fusion module of the multimodal model to fuse the visual features and text features of the multimedia content to be processed to obtain the fused features of the multimedia content to be processed.

17. An electronic device comprising: processor; as well as A memory coupled to the processor is used to store instructions that, when executed by the processor, cause the processor to perform the training method as described in any one of claims 1 to 12, and / or the feature extraction method as described in any one of claims 13 to 14.

18. A computer-readable storage medium having a computer program stored thereon, wherein, When the program is executed by a processor, the processor performs the training method as described in any one of claims 1 to 12, and / or the feature extraction method as described in any one of claims 13 to 14.

19. A computer program product comprising: Instructions, wherein when the program is executed by a processor, the processor performs the training method as described in any one of claims 1 to 12, and / or the feature extraction method as described in any one of claims 13 to 14.

20. A multimodal information processing system, comprising: The training apparatus of claim 16 and the feature extraction apparatus of claim 17.

Images