Method for training a feature extraction model, feature extraction method and apparatus

Abstract

Embodiments of the present specification provide a method for training a feature extraction model, a feature extraction method and apparatus. In the method for training the feature extraction model, the following model training process is performed cyclically using a training sample set until a training end condition is met: image data and text data of each current training sample are provided to a current image feature extraction model and a current text vectorization model respectively to obtain corresponding image features and corresponding text vectors of each current training sample; the corresponding text vectors of each current training sample and a result of splicing the corresponding image features and the corresponding text vectors are provided to a current multi-modal feature fusion model to obtain text features and multi-modal features of each current training sample; first and second loss values are determined according to the multi-modal features and the text features respectively to determine a total loss value of the current model training process; and model parameters of the current feature extraction model are adjusted according to the total loss value.

Description

Technical Field

[0001] The embodiments in this specification generally relate to the field of computer technology, and in particular to methods, feature extraction methods and apparatus for training feature extraction models. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal pre-trained models for video-text pair matching have gradually gained increasing attention. In various tasks involving matching between images, videos, and text (such as multimodal video-text matching and video question answering), how to extract features from image-text pairs to better represent the semantic information of images and text has become an important research problem. Existing technologies typically require three different encoding modules to perform the aforementioned video-text pair feature extraction task; that is, different encoding modules are used to encode text and video separately, and then a fusion encoding module is used to further extract features from the obtained text and video encodings. This results in a cumbersome model due to the large number of encoding modules, and the fusion encoding module often suffers from difficulties in parameter initialization, leading to slow model convergence or unsatisfactory feature extraction performance after training. Therefore, a method for training feature extraction models and for image-text pair feature extraction that can at least partially overcome the above shortcomings is needed. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide a method, a feature extraction method, and an apparatus for training a feature extraction model. Using this method and apparatus, the number of model parameters can be effectively reduced by reasonably sharing encoding modules. On the one hand, this lightweighting of the model improves the efficiency of extracting image-text pairs of features; on the other hand, it reduces the difficulty of model convergence, shortens training time, and helps the features extracted by the model to have better representational effects.

[0004] According to one aspect of an embodiment of this specification, a method for training a feature extraction model is provided, wherein the feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model. The method includes: cyclically executing the following model training process using a training sample set until a training termination condition is met, wherein each training sample in the training sample set includes an image-text pair composed of image data and text data; providing the image data of each current training sample in the current training sample set to a current image feature extraction model to obtain corresponding image features for each current training sample; providing the text data of each current training sample to a current text vectorization model to obtain corresponding text vectors for each current training sample; and for each current training sample... The corresponding image features and corresponding text vector of the current training sample are concatenated to obtain a concatenation result; for each current training sample, the corresponding text vector and the concatenation result are provided to the current multimodal feature fusion model to obtain the text features and multimodal features of the current training sample; a first loss value and a second loss value are determined based on the multimodal features and text features of each current training sample; the total loss value of the current model training process is determined based on the first loss value and the second loss value; and in response to the failure to meet the training termination condition, the model parameters of the current feature extraction model are adjusted based on the total loss value, wherein the feature extraction model after the model parameter adjustment serves as the current feature extraction model for the next model training process.

[0005] According to another aspect of the embodiments of this specification, a feature extraction method based on a feature extraction model is provided, wherein the feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model. The feature extraction method includes: providing image data of an acquired image-text pair to the image feature extraction model to obtain corresponding image features of the image-text pair; providing text data of the image-text pair to the text vectorization model to obtain corresponding text vectors of the image-text pair; concatenating the corresponding image features and the corresponding text vectors to obtain a concatenation result; and providing the concatenation result to the multimodal feature fusion model to obtain multimodal features of the image-text pair, wherein the multimodal feature fusion model includes a sequence model for fusing multimodal features.

[0006] According to another aspect of the embodiments of this specification, an image processing method based on an image processing model is provided, wherein the image processing model includes a feature extraction model and a classification network. The image processing method includes: extracting features from an acquired image-text pair using the feature extraction method described above to obtain multimodal features of the image-text pair; providing the multimodal features of the image-text pair to the classification network to obtain a classification result matching the classification task corresponding to the classification network, wherein the classification result is used to indicate at least one of the following: the answer to the question indicated by the text data of the image-text pair, and whether there is a match between the image data and the text data of the image-text pair.

[0007] According to another aspect of the embodiments of this specification, an apparatus for training a feature extraction model is provided, wherein the feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model. The apparatus is configured to cyclically execute a model training process using a training sample set via a training unit until a training termination condition is met. Each training sample in the training sample set includes an image-text pair consisting of image data and text data. The training unit includes: an image feature extraction module configured to provide image data of each current training sample in the current training sample set to a current image feature extraction model to obtain corresponding image features for each current training sample; a text vectorization module configured to provide text data of each current training sample to a current text vectorization model to obtain corresponding text vectors for each current training sample; and a vector concatenation module configured to perform vector concatenation on each current training sample. The device further includes: a concatenated image feature and text vector of the current training sample; a feature generation module configured to provide the corresponding text vector and concatenated result of each current training sample to the current multimodal feature fusion model to obtain the text feature and multimodal feature of the current training sample; a loss value determination module configured to determine a first loss value and a second loss value based on the multimodal feature and text feature of each current training sample; a total loss value determination module configured to determine the total loss value of the current model training process based on the first loss value and the second loss value; and the device further includes: a parameter adjustment unit configured to adjust the model parameters of the current feature extraction model according to the total loss value in response to the failure to meet the training termination condition, wherein the feature extraction model after model parameter adjustment serves as the current feature extraction model for the next model training process.

[0008] According to another aspect of the embodiments of this specification, a feature extraction apparatus based on a feature extraction model is provided, wherein the feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model. The feature extraction apparatus includes: an image feature extraction unit configured to provide image data of an acquired image-text pair to the image feature extraction model to obtain corresponding image features of the image-text pair; a text vectorization unit configured to provide text data of the image-text pair to the text vectorization model to obtain corresponding text vectors of the image-text pair; a vector concatenation unit configured to concatenate the corresponding image features and the corresponding text vectors to obtain a concatenation result; and a multimodal feature generation unit configured to provide the concatenation result to the multimodal feature fusion model to obtain multimodal features of the image-text pair, wherein the multimodal feature fusion model includes a sequence model for fusing multimodal features.

[0009] According to another aspect of the embodiments of this specification, an image processing apparatus based on an image processing model is provided, wherein the image processing model includes a feature extraction model and a classification network. The image processing apparatus includes: a feature extraction device as described above, configured to extract features from acquired image-text pairs to obtain multimodal features of the image-text pairs; and a classification unit, configured to provide the multimodal features of the image-text pairs to the classification network to obtain a classification result matching a classification task corresponding to the classification network, wherein the classification result is used to indicate at least one of the following: the answer to the question indicated by the text data of the image-text pair, and whether there is a match between the image data and the text data of the image-text pair.

[0010] According to another aspect of the embodiments of this specification, an apparatus for training a feature extraction model is provided, comprising: at least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the method for training a feature extraction model as described above.

[0011] According to another aspect of the embodiments of this specification, a feature extraction apparatus based on a feature extraction model is provided, comprising: at least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the feature extraction method based on the feature extraction model as described above.

[0012] According to another aspect of the embodiments of this specification, an image processing apparatus based on an image processing model is provided, comprising: at least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the image processing method based on the image processing model as described above.

[0013] According to another aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the method for training a feature extraction model as described above, or implements a feature extraction method based on a feature extraction model as described above, or implements an image processing method based on an image processing model as described above.

[0014] According to another aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program that is executed by a processor to implement the method for training a feature extraction model as described above, or to implement the feature extraction method based on the feature extraction model as described above, or to implement the image processing method based on the image processing model as described above. Attached Figure Description

[0015] A further understanding of the nature and advantages of this specification can be achieved by referring to the following figures. In the figures, similar components or features may have the same reference numerals.

[0016] Figure 1 Exemplary architectures of methods and apparatus for training feature extraction models, feature extraction methods and apparatus based on feature extraction models, and image processing methods and apparatus based on image processing models, according to embodiments of this specification, are shown.

[0017] Figure 2 A flowchart illustrating an example of a method for training a feature extraction model according to an embodiment of this specification is shown.

[0018] Figure 3 A flowchart illustrating an example of a keyword masking process for raw text according to an embodiment of this specification is shown.

[0019] Figure 4 A flowchart illustrating an example of a process for obtaining a keyword dictionary according to an embodiment of this specification is shown.

[0020] Figure 5 A schematic diagram is shown as yet another example of a method for training a feature extraction model according to an embodiment of this specification.

[0021] Figure 6A flowchart illustrating an example of a feature extraction method based on a feature extraction model according to an embodiment of this specification is shown.

[0022] Figure 7 A flowchart illustrating an example of an image processing method based on an image processing model according to an embodiment of this specification is shown.

[0023] Figure 8 A block diagram of an example apparatus for training a feature extraction model according to an embodiment of this specification is shown.

[0024] Figure 9 A block diagram of yet another example of an apparatus for training a feature extraction model according to an embodiment of this specification is shown.

[0025] Figure 10 A block diagram of an example feature extraction apparatus based on a feature extraction model according to an embodiment of this specification is shown.

[0026] Figure 11 A block diagram illustrating an example of an image processing apparatus based on an image processing model according to an embodiment of this specification is shown.

[0027] Figure 12 A block diagram of an example apparatus for training a feature extraction model according to an embodiment of this specification is shown.

[0028] Figure 13 A block diagram of an example feature extraction apparatus based on a feature extraction model according to an embodiment of this specification is shown.

[0029] Figure 14 A block diagram illustrating an example of an image processing apparatus based on an image processing model according to an embodiment of this specification is shown. Detailed Implementation

[0030] The subject matter described herein will be discussed below with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of the embodiments described herein. Various processes or components may be omitted, substituted, or added as needed in the various examples. Furthermore, features described in some examples may be combined in other examples.

[0031] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. Unless explicitly indicated by the context, the definition of a term shall remain consistent throughout the specification.

[0032] In this specification, the term "BERT (Bidirectional Encoder Representations from Transformer)" refers to a high-performance text feature extraction model. The BERT model is designed based on the Transformer architecture and has achieved high performance levels on multiple language understanding tasks.

[0033] In this specification, the term "Transformer" refers to a common deep learning neural network widely used in text processing and pre-training techniques, characterized by its scalability, low bias, and high performance. A Transformer consists of multiple encoder blocks and decoder blocks, each of which comprises a multi-head attention network and a feedforward network (FFN).

[0034] In this specification, the term "contrastive learning" refers to a type of self-supervised learning. Contrastive learning focuses on learning the common features among similar instances and distinguishing the differences between dissimilar instances. Its goal is to learn an encoder that encodes similar data for similar classes and makes the encoding results for different classes as different as possible.

[0035] The following describes in detail, with reference to the accompanying drawings, a method and apparatus for training a feature extraction model, a feature extraction method and apparatus based on a feature extraction model, and an image processing method and apparatus based on an image processing model, according to embodiments of this specification.

[0036] Figure 1 An exemplary architecture 100 is shown for a method and apparatus for training a feature extraction model, a feature extraction method and apparatus based on a feature extraction model, and an image processing method and apparatus based on an image processing model, according to embodiments of this specification.

[0037] exist Figure 1 In this context, network 110 is used to interconnect terminal device 120 and application server 130.

[0038] Network 110 can be any type of network capable of interconnecting network entities. Network 110 can be a single network or a combination of various networks. In terms of coverage, network 110 can be a local area network (LAN), a wide area network (WAN), etc. In terms of the carrying medium, network 110 can be a wired network, a wireless network, etc. In terms of data switching technology, network 110 can be a circuit-switched network, a packet-switched network, etc.

[0039] Terminal device 120 can be any type of electronic computing device capable of connecting to network 110, accessing servers or websites on network 110, processing data or signals, etc. For example, terminal device 120 can be a desktop computer, laptop computer, tablet computer, smartphone, etc. Although in Figure 1 Only one terminal device is shown in the diagram, but it should be understood that a different number of terminal devices may be connected to network 110.

[0040] In one implementation, terminal device 120 can be used by a user. Terminal device 120 may include an application client (e.g., application client 121) that can provide various services to the user. In one example, the application client may be a video application, a news application, etc. In some cases, application client 121 may interact with application server 130. For example, application client 121 may transmit a message input by the user to application server 130 and receive a response associated with the message from application server 130. However, it should be understood that in other cases, application client 121 may also generate a response to a message input by the user locally, instead of interacting with application server 130. In this document, "message" can refer to any input information, such as text input by the user for matching candidate information.

[0041] Application server 130 can be connected to candidate information database 140. Candidate information database 140 may include a set of candidate images or a set of candidate videos. In one example, application server 130 can combine text from application client 121 with various candidate images or videos from the candidate information database to form corresponding text-image pairs, and use a feature extraction model to obtain the fused features of these text-image pairs. Optionally, application server 130 can also use the obtained fused features to perform corresponding classification tasks (e.g., text-image matching or video question answering). The feature extraction model can be trained using the method described below.

[0042] It should be understood that Figure 1 All network entities shown are exemplary, and any other network entities may be involved in Architecture 100 depending on the specific application requirements.

[0043] Figure 2 A flowchart of a method 200 for training a feature extraction model according to an embodiment of this specification is shown.

[0044] In this embodiment, the feature extraction model may include an image feature extraction model, a text vectorization model, and a multimodal feature fusion model.

[0045] like Figure 2 As shown in 210, the following model training process 220-270 is executed iteratively using the training sample set until the training termination condition is met.

[0046] In this embodiment, each training sample in the training sample set may include an image-text pair consisting of image data and text data. The image data may include a single image or multiple images associated with the content. In one example, the image-text pair may be an image or video and its corresponding descriptive text. In another example, the training sample may be a video and the corresponding audio, processed by speech recognition to obtain the text. In one example, a batch of training samples may be selected from the training sample set as the current training sample set to perform the following model training iteration.

[0047] In step 220, the image data of each current training sample in the current training sample set is provided to the current image feature extraction model to obtain the corresponding image features of each current training sample.

[0048] In this embodiment, the aforementioned current image feature extraction model may include various models for extracting image features. The aforementioned current image feature extraction model may include, but is not limited to, at least one of the following: ResNet (Residual Neural Network), VST (Video Swin Transformer) model, PVT (Pyramid Vision Transformer) model, ViT (Vision Transformer) model, etc.

[0049] In step 230, the text data of each current training sample is provided to the current text vectorization model to obtain the corresponding text vector of each current training sample.

[0050] In this embodiment, the text vectorization model described above can include various models for converting text into vectors. This allows text to be transformed into corresponding text vectors. In one example, the model parameters of the current text vectorization model can be fine-tuned during the training process, or they can remain unchanged (i.e., the trained text vectorization model can be used).

[0051] In some optional implementations of this embodiment, the text vectorization model described above may include a BERT-based word vector model. In one example, the word vector model may be, for example, BERT, DistillBERT, etc. In one example, for each current training sample, the text data of the current training sample can be provided to the BERT-based word vector model to obtain the word vectors of each token of the text data. Then, the corresponding text vector can be obtained based on the word vectors of each token of the text data. For example, the word vectors of each token can be concatenated to obtain the corresponding text vector. Another example is that the word vectors of each token can be weighted and summed (provided that the dimensions match) to obtain the corresponding text vector. Yet another example is that the features corresponding to the [cls] token obtained after the text data has been processed by word segmentation can be determined as the corresponding text vector.

[0052] In step 240, for each current training sample, the corresponding image features and corresponding text vector of the current training sample are concatenated to obtain the concatenation result.

[0053] In this embodiment, the concatenation result of each current training sample includes the corresponding image features and the corresponding text vector of that current training sample. The order in which the corresponding image features and the corresponding text vector are concatenated is not limited here, but it must remain consistent throughout the entire training process and the application of the model.

[0054] In step 250, for each current training sample, the corresponding text vector and the concatenation result of the current training sample are provided to the current multimodal feature fusion model to obtain the text features and multimodal features of the current training sample.

[0055] In this embodiment, the aforementioned current multimodal feature fusion model may include various models for performing multimodal feature fusion, such as models for processing sequence data. Optionally, the aforementioned multimodal feature fusion model may include a Transformer-based encoder. For example, the current multimodal feature fusion model may include, but is not limited to, at least one of the following: BERT, DistillBERT, etc.

[0056] In this embodiment, for each current training sample, the corresponding text vector of the current training sample is processed by the aforementioned multimodal feature fusion model to obtain the text features of the current training sample. The concatenated result of the current training sample is processed by the aforementioned multimodal feature fusion model to obtain the multimodal features of the current training sample.

[0057] In step 260, the first loss value and the second loss value are determined based on the multimodal features and text features of each current training sample.

[0058] In this embodiment, a first loss value can be determined based on the multimodal features of each current training sample. A second loss value can be determined based on the text features of each current training sample.

[0059] In some optional implementations of this embodiment, the text data of each training sample in the training sample set may include text after keyword masking of the original text. In one example, the masked keywords typically belong to at least one of nouns, verbs, and adjectives. In one example, an open-source grammar analysis tool (such as Spacy) can be used to extract at least one of nouns, verbs, and adjectives from the original text of the current training sample to form a keyword set. Then, words belonging to the keyword set can be selected from the original text of the current training sample and masked to obtain the keyword-masked text of each current training sample.

[0060] In one example, the training sample text data could be "a [MASK] surfing the sea," and the corresponding blocked keyword could be "man." In another example, the training sample text data could be "a man [MASK] the sea," and the corresponding blocked keyword could be "surfing." In yet another example, the training sample text data could be "a man surfing the [MASK]," and the corresponding blocked keyword could be "sea."

[0061] Optionally, refer to Figure 3 , Figure 3 A flowchart illustrating an example of a keyword masking process 300 for raw text according to an embodiment of this specification is shown.

[0062] like Figure 3 As shown in step 310, the original text of each current training text is segmented into words to obtain the segmentation results.

[0063] In this embodiment, for each current training text in the current training text set, various methods can be used to process the current training text (e.g., Captcha). iThe original text (represented by) is segmented into words to obtain the segmentation result (for example, it can be represented by...). (This is represented as a block of text). This allows us to obtain word segmentation results corresponding to each current training text. Each word segmentation result can include at least one segment of the original text of the corresponding current training text.

[0064] In step 320, each word in the word segmentation result is matched with the keyword dictionary to obtain the matching result.

[0065] In this embodiment, each word in the above word segmentation results can be matched with a keyword dictionary to obtain matching results. The keyword dictionary can be determined based on the frequency of words in the original text of the acquired training samples. In one example, the keyword dictionary can be composed of words with high frequency (e.g., the top 10% or top 500) in the original text of each training sample in the acquired training sample set that are not function words (e.g., prepositions, conjunctions, auxiliary words, modal particles, etc.). In another example, the keyword dictionary can be composed by combining keywords determined by TF-IDF (term frequency–inverse document frequency) from each training sample in the acquired training sample set.

[0066] Optionally, refer to Figure 4 , Figure 4 A flowchart illustrating an example of a keyword dictionary acquisition process 400 according to an embodiment of this specification is shown.

[0067] like Figure 4 As shown, in step 410, the text data of each training sample obtained are segmented into words to obtain the total segmentation result.

[0068] In this embodiment, the text data (usually the original text) of each training sample in the obtained training sample set can be segmented into words to obtain the total segmentation result. Optionally, BERT can be used for word segmentation.

[0069] In step 420, words that meet the part-of-speech requirements are selected from the overall word segmentation results to form a candidate word set.

[0070] In this embodiment, the part-of-speech requirement mentioned above can be, for example, belonging to a noun, verb, or adjective.

[0071] In step 430, based on the frequency of each word in the candidate word set in the original text of the obtained training samples, words that meet the word frequency requirements are selected from the candidate word set to form a keyword dictionary.

[0072] In this embodiment, as an example, the above-mentioned word frequency requirement can be a high word frequency (e.g., the top 10% or the top 2000). As another example, the above-mentioned word frequency requirement can also be a low word frequency (e.g., the bottom 10% or the bottom 2000).

[0073] Based on this, this solution can determine the keyword dictionary offline, thus providing a basis for keyword masking operations for each training sample. Furthermore, this process does not require the introduction of new model parameters, making it easy to port and lightweight.

[0074] Back Figure 3 In step 330, based on each successfully matched word, the keyword set corresponding to each current training text is obtained.

[0075] In this embodiment, as an example, for each current training text, the successfully matched words can be combined to form a keyword set corresponding to that current training text.

[0076] In step 340, for each current training sample, the target words in the original text of the current training sample are masked based on the keywords in the keyword set corresponding to that current training sample.

[0077] In this embodiment, for each current training sample, target words can be selected from the original text of that current training sample for masking. These target words include words belonging to the keyword set corresponding to that current training sample. In one example, the number of masked keywords corresponding to that current training sample can be determined first. For example, the number of word segments contained in the original text of that current training sample can be multiplied by 20%. If the number of keywords contained in the keyword set corresponding to that current training sample is less than the determined number of masked keywords, then all values ​​in the keyword set are used. Optionally, the portion insufficient for the number of masked keywords can be replaced with other non-keywords. If the number of keywords contained in the keyword set corresponding to that current training sample is not less than the determined number of masked keywords, then a value matching the number of masked keywords is randomly selected from the keyword set. This yields the keyword-masked text for each current training sample.

[0078] Based on this, by selecting blocked keywords based on the keyword set and selecting a certain proportion of blocked keywords, this scheme can ensure both the quality and quantity of blocked keywords corresponding to each training sample, thereby improving the model training effect.

[0079] Back Figure 2In the optional implementation of step 260, the feature extraction model may further include a first multi-class classifier and a second multi-class classifier. In one example, the first and second multi-class classifiers may employ a "fully connected layer + softmax" structure. In another example, the first and second multi-class classifiers may employ a "multi-layer perceptron (MLP)" structure.

[0080] The first loss value mentioned above may include a multimodal feature prediction loss value. The second loss value mentioned above may include a text feature prediction loss value. The multimodal feature prediction loss value and the text feature prediction loss value can be determined based on the differences between the first prediction result and the second prediction result, respectively, and the blocked keywords corresponding to the current training samples. The first prediction result and the second prediction result can be the prediction results of the blocked keywords obtained by the first multi-class classifier based on the multimodal features and the second multi-class classifier based on the text features, respectively.

[0081] In one example, for each current training sample, the multimodal feature prediction loss value can be calculated using the following formula:

[0082]

[0083] Where Q represents the set of blocked keywords. |Q| represents the number of words contained in the set of blocked keywords. q represents the value of the blocked keyword. It can be used to represent cross-entropy loss. q It can be used to represent the ground truth of blocked keywords. q This can be used to represent the multimodal features of the training samples corresponding to the masked keyword taking the value q. Θ1(·) can be used to represent the first multi-class classifier mentioned above. The parameters of the first multi-class classifier can be determined empirically or through machine learning training.

[0084] For example, Q can be {man, surfing, sea}. |Q| can be 3. When q is "man", The cross-entropy loss can be calculated based on the first prediction result obtained by the first multi-class classifier based on the multimodal features of "a[MASK]surfing the sea" and "man". When q is "surfing", The cross-entropy loss can be calculated based on the first prediction result obtained by the first multi-class classifier based on the multimodal features of "a man [MASK]thesea" and "surfing". When q is "sea", The cross-entropy loss can be calculated based on the first prediction result obtained by the first multi-class classifier based on the multimodal features of "sea" and "a man surfing the [MASK]".

[0085] In one example, for each current training sample, the text feature prediction loss value can be calculated using the following formula:

[0086]

[0087] Among them, t q It can be used to represent the text features of the training samples when the masked keyword is q. Θ2(·) can be used to represent the second multi-class classifier mentioned above. The meanings of the other symbols can be found in the previous examples.

[0088] For example, Q can be {man, surfing, sea}. |Q| can be 3. When q is "man", The cross-entropy loss can be calculated based on the second prediction result obtained by the second multi-class classifier based on the text features of "a[MASK]surfing the sea" and "man". When q is "surfing", The cross-entropy loss can be calculated based on the second prediction result obtained by the second multi-class classifier based on the text features of "surfing" and "a man [MASK] the sea". When q is "sea", The cross-entropy loss can be calculated based on the second prediction result obtained by the second multi-class classifier based on the text features of “sea” and “a man surfing the [MASK]”.

[0089] In one example, the first loss value for the current training sample set can be obtained by combining the multimodal feature prediction loss values ​​of each current training sample. In another example, the second loss value for the current training sample set can be obtained by combining the text feature prediction loss values ​​of each current training sample.

[0090] Based on this, compared to existing technologies that mask some unimportant words in a given text, this approach can perform targeted MLM (Masking Language Modeling) training on keywords in the training samples. This promotes the fusion of features from both visual and textual modalities and helps the model learn high-quality cross-modal features. For example, existing technologies, for the masked text "a boy sitting [MASK] the sofa with [MASK] dog around," can directly deduce that the masked text is "on" and "a" based solely on the contextual grammatical rules, without relying on any visual information. However, the model training in this approach focuses on the features of keywords that fully interact with the image content, such as "boy," "sitting," "sofa," and "dog," and combines textual features with multimodal features for joint supervision during model training, thereby effectively improving the feature extraction performance of the trained model.

[0091] In some optional implementations of this embodiment, the second loss value may further include a local text feature contrast loss value. This local text feature contrast loss value can be determined based on the similarity between the corresponding image features and local text features of the current training sample. The local text features can be determined based on the text features corresponding to the blocked keywords of the current training sample.

[0092] The aforementioned local text features can be determined in various ways. In one example, the masked keywords corresponding to the current training sample can be provided to the current text vectorization model, and the resulting corresponding text vectors can be provided to the current multimodal feature fusion model. The text features corresponding to each masked keyword can then be used as local text features. In another example, if the current multimodal feature fusion model is a model where text features and word segmentation correspond one-to-one (e.g., a Transformer-based encoder), the text features of the corresponding masked keywords can be directly extracted from the corresponding text features of the current training sample based on the position of the masked keywords in the text data.

[0093] In one example, the local text feature contrast loss value mentioned above can be calculated using the following formula:

[0094]

[0095] in, This can be used to represent the batch size of the current training sample set. For the i-th current training sample, N can be used to represent the corresponding image features of the k-th image in the image data of the current training sample.v N can be used to represent the number of images contained in the image data of the current training sample. l This can be used to represent the number of masked keywords in the calculation of the local text feature contrast loss value for the current training sample. This can be used to represent the local text features of the current training sample when the l-th blocked keyword is selected (e.g., when the l-th blocked keyword is "sea"). (This can be the text feature corresponding to "sea"). It can be used to represent the corresponding image feature of the k-th image in the image data of the j-th current training sample. <,> can be used to represent the cross product between matrices, for example...<a,b> =a*b T (T can be used to denote the transpose of a matrix). Optionally, the image data of the current training sample can be represented by I. i To represent. I i It can include data from the target video V. i N obtained from random sampling v One video frame.

[0096] In one example, the number of masked keywords in the local text feature contrast loss calculation process for each current training sample can be determined first. For example, this could be a preset number (e.g., 4), or the number of words in the original text of the current training sample multiplied by a preset percentage (e.g., 15%). As an example, for each current training sample, if the number of keywords in the keyword set corresponding to that current training sample (e.g., 8) is not less than the number of masked keywords in the aforementioned local text feature contrast loss calculation process, then keywords can be randomly selected from the corresponding keyword set (e.g., 4 keywords can be selected from 8). If the number of keywords in the keyword set corresponding to that current training sample (e.g., 3) is less than the number of masked keywords in the aforementioned local text feature contrast loss calculation process, then repeated selection from the corresponding keyword set is allowed (e.g., one of the 3 keywords can be selected twice).

[0097] Based on this, by constructing a matching mechanism between local text features of keywords based on text data and corresponding image features, the local important information of the text data is effectively utilized, thereby helping to enhance the feature extraction capability of the trained model.

[0098] In some optional implementations of this embodiment, the text data of each training sample in the training sample set may include the original text without keyword masking. In one example, the text data of the training sample may be "a man surfing the sea".

[0099] The aforementioned feature extraction model may further include a feature mapping layer. This feature mapping layer may have a structure similar to the first and second multi-class classifiers described above. The aforementioned first loss value may include a multimodal feature contrast loss value. This multimodal feature contrast loss value can be obtained based on the mapping results obtained by passing the multimodal features corresponding to the current training sample and the swapped current training sample through the feature mapping layer. The swapped current training sample may consist of image data belonging to different current training samples and original text. In one example, current training sample 1 may include a surfing image set and the word "surfing". Current training sample 2 may include a horseback riding image set and the word "horseback riding". Then, swapped training sample 3 may include a surfing image set and the word "horseback riding". Swapped training sample 4 may include a horseback riding image set and the word "surfing".

[0100] In one example, the multimodal feature contrast loss value mentioned above can be calculated using the following formula:

[0101]

[0102] in, This can be used to represent the batch size of the current training sample set. Θ3(·) can be used to represent the feature mapping layer described above. For the i-th current training sample, m i,i It can be used to represent the multimodal features of the current training samples, m j,i It can be used to represent the multimodal features of the current training sample after the exchange process.

[0103] The second loss value mentioned above can include a global text feature contrast loss value. This global text feature contrast loss value can be determined based on the similarity between the text features of the original text of the current training sample and the corresponding image features.

[0104] In one example, the global text feature contrast loss value mentioned above can be calculated using the following formula:

[0105]

[0106] in, This can be used to represent the batch size of the current training sample set. For the i-th current training sample, N can be used to represent the corresponding image feature of the k-th image data of the current training sample. v It can be used to represent the number of images contained in the image data of the current training sample. These can be used to represent the text features (i.e., global text features) of the original text of the current training sample. This can be used to represent the corresponding image feature of the k-th image data of the j-th current training sample. Optionally, For example, it could be the feature corresponding to the [cls] tag obtained after the original text has been segmented (e.g., it could be used as a feature). express).

[0107] Based on this, the global text features and multimodal features extracted by the aforementioned lighter feature extraction model can be combined to train the model, enabling it to learn the matching relationship between global text and images, which helps to improve the model's feature extraction performance.

[0108] In some optional implementations of this embodiment, the text data of each training sample in the training sample set may include the original text without keyword masking and the text after keyword masking. The feature extraction model may further include a first multi-class classifier, a second multi-class classifier, and a feature mapping layer. The first loss value may include a multimodal feature contrast loss value and a multimodal feature prediction loss value, and the second loss value may include a global text feature contrast loss value and a text feature prediction loss value.

[0109] Optionally, the second loss value may include a global text feature contrast loss value, a text feature prediction loss value, and a local text feature contrast loss value.

[0110] For a detailed description of the above-mentioned multimodal feature contrast loss, multimodal feature prediction loss, global text feature contrast loss, text feature prediction loss, and local text feature contrast loss, please refer to the foregoing.

[0111] Based on this, a loss function can be constructed by combining global text features, matching local text features with corresponding image features, and using contrastive learning and keyword-based missing character filling modeling, thereby improving the training effect of the model.

[0112] At 270, the total loss value for the current model training process is determined based on the first loss value and the second loss value.

[0113] In this embodiment, the total loss value of the current model training process can be determined by combining the first loss value and the second loss value. In one example, the sum of the first loss value and the second loss value can be determined as the total loss value of the current model training process. In another example, the weighted sum of the first loss value and the second loss value can be determined as the total loss value of the current model training process.

[0114] At 280, determine whether the training termination condition is met.

[0115] In one example, the training termination condition can be determined by checking whether the number of iterations has reached the preset number, the training duration has reached the preset duration, and the loss value has converged.

[0116] At 290, in response to the failure to meet the training termination condition, the model parameters of the current feature extraction model are adjusted based on the total loss value.

[0117] In this embodiment, in response to the failure to meet the training termination condition, in one example, the gradient can be determined based on the determined total loss value, and then combined with the learning rate to determine the adjustment value, thereby adjusting the network parameters of the current feature extraction model. For example, the model parameters of the image feature extraction model and the multimodal feature fusion model can be adjusted. Another example is that the model parameters of the image feature extraction model, the text vectorization model, and the multimodal feature fusion model can be adjusted.

[0118] The feature extraction model, after parameter adjustments, can serve as the current feature extraction model for the next model training process. Then, the current training samples can be redefined using the aforementioned training sample set, and the model training process (steps 220-270) can continue until the training termination condition is met.

[0119] Continue to refer to Figure 5 , Figure 5 A schematic diagram is shown of yet another example of a method 500 for training a feature extraction model according to an embodiment of this specification.

[0120] In this embodiment, each training sample in the training sample set can be as shown in Figure 520. The training sample 520 can consist of image data 521 and text data. Optionally, the image data 521 can be a sequence of video frames extracted from the target video 510. Optionally, the text data can be the original text as shown in Figure 522, or it can be text processed by keyword masking (e.g., replacing "girl", "riding", or "horse" with [mask]). The following describes the model training process using a training sample (i.e., training sample 520) in the current training sample set as an example. The video frame sequence 521 can be provided to the image feature extraction model 530 to obtain the corresponding image features 531 of the training sample 520. The corresponding image features 531 can include the image features of each video frame in the video frame sequence 521. The text data (such as text 522) can be provided to the text vectorization model 540 to obtain the corresponding text vector 541 of the training sample 520. The corresponding text vector 541 can include the vectors corresponding to each word segment of text 522. Then, the corresponding image features 531 of training sample 520 can be concatenated with the corresponding text vector 541 to obtain the concatenated result 550 of training sample 520. The concatenated result 550 is then provided to the multimodal feature fusion model 560 to obtain the multimodal features 561 of training sample 520. The corresponding text vector 541 of training sample 520 is then provided to the multimodal feature fusion model 560 to obtain the text features 562 of training sample 520.

[0121] Next, a first loss value 571 can be determined based on the multimodal features of each current training sample in the current training sample set (including multimodal feature 561 of training sample 520). A second loss value 572 is determined based on the text features of each current training sample (including text feature 562 of training sample 520). Then, a total loss value 580 is determined based on the first loss value 571 and the second loss value 572. In response to the failure to meet the training termination condition, the model parameters of the current feature extraction model (e.g., it may include an image feature extraction model and a multimodal feature fusion model) are adjusted based on the total loss value 580. The feature extraction model with adjusted model parameters is used as the current feature extraction model for the next model training process, and the above model training process is repeated until the training termination condition is met.

[0122] use Figures 1-5The method disclosed in this paper for training a feature extraction model, compared to existing technologies that require an additional text feature extraction model to convert text vectors into text features, can simultaneously extract text features and fuse multimodal features (e.g., text features and image features) by sharing a multimodal feature fusion model, thus effectively reducing the number of model parameters. Furthermore, research has revealed that since both the aforementioned text feature extraction model and the multimodal feature fusion model process sequential vector features (e.g., the former's input is word vectors, and the latter's input is a concatenation of image features and word vectors), and their structures are typically similar (e.g., both can use the BERT structure, differing only in the number of Transformer models used), this provides a theoretical basis for simultaneously performing multimodal feature fusion and text feature extraction using a multimodal feature fusion model. This makes the feature extraction model more lightweight while still ensuring good feature extraction performance. Moreover, since the multimodal feature fusion model of this scheme has the functions of both text feature extraction and multimodal feature fusion, the difficulty of finding suitable initialization parameters for the multimodal feature fusion model can be reduced by relying on the parameter initialization experience of existing text feature extraction models. This helps to accelerate the convergence of the model and save the time cost and computing resources of model training.

[0123] The following is for reference. Figure 6 , Figure 6 A flowchart of a feature extraction method 600 based on a feature extraction model according to an embodiment of this specification is shown.

[0124] In this embodiment, the feature extraction model may include an image feature extraction model, a text vectorization model, and a multimodal feature fusion model.

[0125] like Figure 6 As shown, in step 610, the image data of the obtained image-text pair is provided to the image feature extraction model to obtain the corresponding image features of the image-text pair.

[0126] In this embodiment, image-text pairs can be obtained in various ways. These image-text pairs may include image data and text data. In one example, an image-text pair may include an image and corresponding descriptive text. In another example, an image-text pair may include an image sequence and corresponding descriptive text. Optionally, the image sequence may be a video frame sequence. The image feature extraction model may include various pre-trained feature extraction models for extracting image features. These image feature extraction models may include, but are not limited to, at least one of the following: ResNet (Residual Neural Network), VST model, PVT model, ViT model, etc.

[0127] In step 620, the text data of the image-text pair is provided to the text vectorization model to obtain the corresponding text vector of the image-text pair.

[0128] In this embodiment, the text data of the aforementioned image-text pair can be provided to a text vectorization model to obtain the corresponding text vectors for the image-text pair. The text vectorization model can include various models for converting text into vectors. Thus, text can be converted into corresponding text vectors.

[0129] In step 630, the corresponding image features and the corresponding text vector are concatenated to obtain the concatenation result.

[0130] In this embodiment, the corresponding image features and corresponding text vectors of the above-mentioned image-text pair can be concatenated to obtain the concatenated result. The concatenation order of the corresponding image features and corresponding text vectors is generally consistent with the corresponding training process of the feature extraction model described above.

[0131] At 640, the splicing result is provided to the multimodal feature fusion model to obtain the multimodal features of the image-text pair.

[0132] In this embodiment, the multimodal feature fusion model described above can include various sequence models for fusing multimodal features. The multimodal feature fusion model can perform fusion processing on serialized vectors (e.g., vector sequences), so that each vector in the processed multimodal feature (usually also in the form of a vector sequence) can incorporate the original information of that vector and the information of other vectors in the original vector sequence. As an example, the multimodal feature fusion model may include, but is not limited to, at least one of the following: BERT, DistillBERT, etc. As another example, the multimodal feature fusion model may also include attention-based recurrent neural networks (RNNs), long short-term memory (LSTM) networks, gated recurrent units (GRUs), etc.

[0133] In some optional implementations of this embodiment, the feature extraction model described above can be implemented as described in the foregoing. Figure 1-5 The method described is used to train the feature extraction model.

[0134] use Figure 6The feature extraction method based on the feature extraction model disclosed in this paper, compared with the existing technology which requires an additional text feature extraction model to convert text vectors into text features, creatively achieves the extraction of text features and the fusion of multimodal features (such as text features and image features) through a multimodal feature fusion model, thereby effectively reducing the model parameters and realizing the extraction of multimodal features of image-text pairs through a lightweight feature extraction model.

[0135] Continue to refer to Figure 7 , Figure 7 A flowchart illustrating an example of an image processing method 700 based on an image processing model according to an embodiment of this specification is shown.

[0136] In this embodiment, the image processing model includes a feature extraction model and a classification network.

[0137] like Figure 7 As shown in Figure 710, feature extraction methods are used to extract features from the obtained image-text pairs to obtain multimodal features of the image-text pairs.

[0138] In this embodiment, the above feature extraction method can be referred to as follows: Figure 6 The feature extraction method based on the feature extraction model described herein will not be elaborated upon here.

[0139] In 720, the multimodal features of the image-text pair are provided to the classification network to obtain a classification result that matches the classification task corresponding to the classification network.

[0140] In this embodiment, the classification network described above can be used to perform corresponding classification tasks. In one example, the classification task may include a video-question answering (VQA) task. In another example, the classification task may include a text-to-video matching task. The text-to-video matching task may, for example, include a text-to-video retrieval (TVR) task. Accordingly, the classification result can be used to indicate at least one of the following: the answer to the question indicated by the text data of the image-text pair, and whether there is a match between the image data and text data of the image-text pair.

[0141] In one example, the classification network described above can employ a "fully connected layer + softmax" structure. The classification network can be adapted to different classification tasks. Figure 2 The description of the "first multi-class classifier" or "feature mapping layer" in the optional implementation of the embodiments is provided. In one example, the above classification network can be trained in a subsequent fine-tuning process.

[0142] Based on this, this solution provides a method to perform corresponding downstream tasks by utilizing the multimodal features of image-text pairs extracted by the feature extraction method based on the above feature extraction model, thereby improving task performance in application scenarios such as video text matching and video question answering.

[0143] The following is for reference. Figure 8 , Figure 8 A block diagram of an example of an apparatus 800 for training a feature extraction model according to an embodiment of this specification is shown. This apparatus embodiment can be used with... Figures 2-5 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0144] like Figure 8 As shown, the apparatus 800 for training a feature extraction model may include a training unit 810 and a parameter adjustment unit 820. The feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model. The training unit 810 can be configured to iteratively execute the model training process using a training sample set until the training termination condition is met. Each training sample in the training sample set includes an image-text pair composed of image data and text data. The training unit 810 may include an image feature extraction module 811, a text vectorization module 812, a vector concatenation module 813, a feature generation module 814, a partial loss value determination module 815, and a total loss value determination module 816.

[0145] The image feature extraction module 811 is configured to provide the image data of each current training sample in the current training sample set to the current image feature extraction model to obtain the corresponding image features of each current training sample. The operation of the image feature extraction module 811 can be referred to above. Figure 2 The operation described in section 220.

[0146] The text vectorization module 812 is configured to provide the text data of each current training sample to the current text vectorization model, thereby obtaining the corresponding text vector for each current training sample. The operation of the text vectorization module 812 can be referenced above. Figure 2 The operation described in section 230.

[0147] In one example, the text vectorization model includes a BERT-based word vector model, and the multimodal feature fusion model includes a Transformer-based encoder.

[0148] The vector concatenation module 813 is configured to concatenate the corresponding image features and the corresponding text vector of each current training sample to obtain the concatenated result. The operation of the vector concatenation module 813 can be found above. Figure 2The operation described in section 240.

[0149] The feature generation module 814 is configured to provide the corresponding text vector and concatenation result of each current training sample to the current multimodal feature fusion model to obtain the text features and multimodal features of the current training sample. The operation of the feature generation module 814 can be referenced above. Figure 2 The operation described is 250.

[0150] The sub-loss determination module 815 is configured to determine the first loss value and the second loss value based on the multimodal features and text features of each current training sample, respectively. The operation of the sub-loss determination module 815 can be referenced above. Figure 2 The operation described in section 260.

[0151] In one example, the text data of each training sample in the training sample set includes text after keyword masking of the original text. The feature extraction model further includes a first multi-class classifier and a second multi-class classifier. The first loss value includes a multimodal feature prediction loss value, and the second loss value includes a text feature prediction loss value. The multimodal feature prediction loss value and the text feature prediction loss value are determined based on the differences between the first prediction result and the second prediction result and the masked keyword corresponding to the current training sample, respectively. The first prediction result and the second prediction result are the prediction results of the masked keyword obtained by the first multi-class classifier based on the multimodal features and the second multi-class classifier based on the text features, respectively.

[0152] In one example, the second loss value also includes a local text feature contrast loss value, which is determined based on the similarity between the corresponding image features and local text features of the current training sample, wherein the local text features are determined based on the text features corresponding to the masked keywords of the current training sample.

[0153] In one example, the text data of each training sample in the training sample set includes the original text without keyword masking. The first loss value includes a multimodal feature contrast loss value. The feature extraction model further includes a feature mapping layer. The multimodal feature contrast loss value is obtained based on the mapping results of the multimodal features corresponding to the current training sample and the swapped current training sample after passing through the feature mapping layer. The swapped current training sample consists of image data and original text belonging to different current training samples. The second loss value includes a global text feature contrast loss value, which is determined based on the similarity between the text features of the original text and the corresponding image features of the current training sample.

[0154] In one example, the text data of each training sample in the training sample set also includes the original text without keyword masking. The first loss value also includes a multimodal feature contrast loss value. The feature extraction model further includes a feature mapping layer. The multimodal feature contrast loss value is obtained based on the mapping results obtained by passing the multimodal features corresponding to the current training sample and the swapped current training sample through the feature mapping layer. The swapped current training sample consists of image data and original text belonging to different current training samples. The second loss value includes a global text feature contrast loss value, which is determined based on the similarity between the text features of the original text and the corresponding image features of the current training sample.

[0155] The total loss determination module 816 is configured to determine the total loss value of the current model training process based on the first loss value and the second loss value. The operation of the total loss determination module 816 can be referenced above. Figure 2 The operation described in section 270.

[0156] The parameter adjustment unit 820 is configured to adjust the model parameters of the current feature extraction model based on the total loss value in response to the failure to meet the training termination condition. The feature extraction model after parameter adjustment serves as the current feature extraction model for the next model training process. The operation of the parameter adjustment unit 820 can be referenced above. Figure 2 The operation described in section 280.

[0157] The following is for reference. Figure 9 , Figure 9 A block diagram of yet another example of an apparatus 900 for training a feature extraction model according to an embodiment of this specification is shown. This apparatus embodiment can be used with... Figures 2-5 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0158] like Figure 9 As shown, the apparatus 900 for training the feature extraction model may include a training unit 910, a parameter adjustment unit 920, and a keyword masking unit 930. The feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model.

[0159] In this embodiment, the training unit 910 and parameter adjustment unit 920 described above can be used in conjunction with the aforementioned... Figure 8 The training unit 810 and the parameter adjustment unit 820 are described in the same way in the embodiment, and will not be repeated here.

[0160] The keyword masking unit 930 is configured to segment the original text of each current training text to obtain segmentation results; match each word in the segmentation results with a keyword dictionary to obtain matching results, wherein the keyword dictionary is determined based on the frequency of word occurrences in the original text of the acquired training samples; obtain a keyword set corresponding to each current training text based on each successfully matched word; and for each current training sample, mask the target words in the original text of that current training sample based on the keywords in the keyword set corresponding to that current training sample. The operation of the keyword masking unit 930 can be referred to above. Figure 3 The relevant operations described in the examples.

[0161] In this embodiment, the text data of each training sample in the training sample set includes the text after keyword masking of the original text.

[0162] In one example, the keyword dictionary is obtained through the following steps: segmenting the text data of each training sample to obtain a total segmentation result; selecting words that meet the part-of-speech requirements from the total segmentation result to form a candidate word set; and selecting words that meet the word frequency requirements from the candidate word set to form the keyword dictionary based on the frequency of occurrence of each word in the candidate word set in the original text of the training samples.

[0163] The following is for reference. Figure 10 , Figure 10 A block diagram illustrating an example of a feature extraction apparatus 1000 based on a feature extraction model according to an embodiment of this specification is shown. This apparatus embodiment can be used with... Figure 6 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0164] like Figure 10 As shown, the feature extraction device 1000 based on the feature extraction model may include an image feature extraction unit 1010, a text vectorization unit 1020, a vector concatenation unit 1030, and a multimodal feature generation unit 1040. The feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model.

[0165] The image feature extraction unit 1010 is configured to provide the acquired image data of the image-text pair to the image feature extraction model to obtain the corresponding image features of the image-text pair. The operation of the image feature extraction unit 1010 can be referred to above. Figure 6 The operation of 610 is described.

[0166] The text vectorization unit 1020 is configured to provide the text data of the image-text pair to the text vectorization model to obtain the corresponding text vector of the image-text pair. The operation of the text vectorization unit 1020 can be referred to above. Figure 6 The operation described in section 620.

[0167] The vector concatenation unit 1030 is configured to concatenate the corresponding image features and the corresponding text vector to obtain a concatenation result. The operation of the vector concatenation unit 1030 can be referred to above. Figure 6 The operation described in section 630.

[0168] The multimodal feature generation unit 1040 is configured to provide the concatenation result to the multimodal feature fusion model to obtain the multimodal features of the image-text pair, wherein the multimodal feature fusion model includes a sequence model for fusing multimodal features. The operation of the multimodal feature generation unit 1040 can be referred to above. Figure 6 The operation described in section 640.

[0169] In one example, the feature extraction model uses, for example... Figures 2-5 The method described in [the document] for training the feature extraction model is used to train it.

[0170] Further reference Figure 11 , Figure 11 A block diagram illustrating an example of an image processing apparatus 1100 based on an image processing model according to an embodiment of this specification is shown. This apparatus embodiment can be used with... Figure 7 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0171] like Figure 11 As shown, the image processing device 1100 based on the image processing model may include a feature extraction device 1110 and a classification unit 1120. The image processing model includes a feature extraction model and a classification network.

[0172] Feature extraction device 1110 is configured to extract features from the acquired image-text pairs to obtain the multimodal features of the image-text pairs. The operation of feature extraction device 1110 can be referred to above. Figure 7 The operation of 710 is described.

[0173] Classification unit 1120 is configured to provide the multimodal features of the image-text pair to the classification network to obtain a classification result that matches the classification task corresponding to the classification network. The classification result indicates at least one of the following: the answer to the question indicated by the text data of the image-text pair, and whether there is a match between the image data and text data of the image-text pair. The operation of classification unit 1120 can be referred to above. Figure 7The operation described in section 720.

[0174] Reference above Figures 1 to 11 Embodiments of methods and apparatus for training feature extraction models, feature extraction methods and apparatus based on feature extraction models, and image processing methods and apparatus based on image processing models, according to embodiments of this specification, have been described.

[0175] The apparatus for training a feature extraction model, the feature extraction apparatus based on the feature extraction model, and the image processing apparatus based on the image processing model described in this specification can be implemented in hardware, software, or a combination of hardware and software. Taking software implementation as an example, as a logical device, it is formed by the processor of its host device reading the corresponding computer program instructions from the memory into memory and executing them. In the embodiments of this specification, the apparatus for training a feature extraction model, the feature extraction apparatus based on the feature extraction model, and the image processing apparatus based on the image processing model can, for example, be implemented using electronic devices.

[0176] Figure 12 A block diagram of an example of an apparatus 1200 for training a feature extraction model according to an embodiment of this specification is shown.

[0177] like Figure 12 As shown, the apparatus 1200 for training a feature extraction model may include at least one processor 1210, a memory (e.g., non-volatile memory) 1220, a RAM 1230, and a communication interface 1240, and the at least one processor 1210, memory 1220, RAM 1230, and communication interface 1240 are connected together via a bus 1250. At least one processor 1210 executes at least one computer-readable instruction (i.e., the elements implemented in software described above) stored or encoded in the memory.

[0178] In one embodiment, computer-executable instructions are stored in memory, which, when executed, cause at least one processor 1210 to: cyclically execute the following model training process using a training sample set until a training termination condition is met, wherein each training sample in the training sample set comprises an image-text pair consisting of image data and text data; provide the image data of each current training sample in the current training sample set to a current image feature extraction model to obtain corresponding image features for each current training sample; provide the text data of each current training sample to a current text vectorization model to obtain corresponding text vectors for each current training sample; for each current training sample, concatenate the corresponding image features and corresponding text vectors of that current training sample to obtain a concatenation result; for each... Given a current training sample, the corresponding text vector and concatenation result of the current training sample are provided to the current multimodal feature fusion model to obtain the text features and multimodal features of the current training sample. A first loss value and a second loss value are determined based on the multimodal features and text features of each current training sample. The total loss value of the current model training process is determined based on the first loss value and the second loss value. In response to the failure to meet the training termination condition, the model parameters of the current feature extraction model are adjusted based on the total loss value. The feature extraction model after the model parameter adjustment serves as the current feature extraction model for the next model training process. The feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model.

[0179] It should be understood that the computer-executable instructions stored in the memory, when executed, cause at least one processor 1210 to perform the above-described combinations in the various embodiments of this specification. Figures 2-5 The description includes various operations and functions.

[0180] Figure 13 A block diagram of an example feature extraction apparatus 1300 based on a feature extraction model according to an embodiment of this specification is shown.

[0181] like Figure 13 As shown, the feature extraction apparatus 1300 based on the feature extraction model may include at least one processor 1310, a memory (e.g., non-volatile memory) 1320, a main memory 1330, and a communication interface 1340, and the at least one processor 1310, memory 1320, main memory 1330, and communication interface 1340 are connected together via a bus 1350. At least one processor 1310 executes at least one computer-readable instruction (i.e., the elements implemented in software above) stored or encoded in the memory.

[0182] In one embodiment, computer-executable instructions are stored in memory, which, when executed, cause at least one processor 1310 to: provide image data of the acquired image-text pair to the image feature extraction model to obtain corresponding image features of the image-text pair; provide text data of the image-text pair to the text vectorization model to obtain corresponding text vectors of the image-text pair; concatenate the corresponding image features and the corresponding text vectors to obtain a concatenation result; and provide the concatenation result to the multimodal feature fusion model to obtain multimodal features of the image-text pair, wherein the multimodal feature fusion model includes a sequence model for fusing multimodal features. The feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model.

[0183] It should be understood that the computer-executable instructions stored in memory, when executed, cause at least one processor 1310 to perform the above-described combinations in the various embodiments of this specification. Figure 6 The description includes various operations and functions.

[0184] Figure 14 A block diagram illustrating an example of an image processing apparatus 1400 based on an image processing model according to an embodiment of this specification is shown.

[0185] like Figure 14 As shown, the image processing apparatus 1400 based on the image processing model may include at least one processor 1410, a memory (e.g., non-volatile memory) 1420, a main memory 1430, and a communication interface 1440, and the at least one processor 1410, memory 1420, main memory 1430, and communication interface 1440 are connected together via a bus 1450. At least one processor 1410 executes at least one computer-readable instruction (i.e., the elements implemented in software described above) stored or encoded in the memory.

[0186] In one embodiment, computer-executable instructions are stored in memory, which, when executed, cause at least one processor 1410 to: extract features from the acquired image-text pair using the feature extraction method described above, to obtain multimodal features of the image-text pair; provide the multimodal features of the image-text pair to the classification network to obtain a classification result matching the classification task corresponding to the classification network, wherein the classification result indicates at least one of the following: the answer to the question indicated by the text data of the image-text pair, and whether there is a match between the image data and the text data of the image-text pair. The image processing model includes a feature extraction model and a classification network.

[0187] It should be understood that the computer-executable instructions stored in memory, when executed, cause at least one processor 1410 to perform the above-described combinations in the various embodiments of this specification. Figure 7The description includes various operations and functions.

[0188] According to one embodiment, a program product, such as a computer-readable medium, is provided. The computer-readable medium may have instructions (i.e., the elements implemented in software as described above), which, when executed by a computer, cause the computer to perform the above-described combinations of the various embodiments of this specification. Figure 1-7 The description includes various operations and functions.

[0189] Specifically, a system or apparatus equipped with a readable storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system or apparatus can read and execute the instructions stored in the readable storage medium.

[0190] In this case, the program code itself, which can be read from a readable medium, can perform the functions of any of the above embodiments. Therefore, the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0191] The computer program code required for the operation of each part of this manual can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB, .NET, and Python; conventional procedural programming languages ​​such as C, Visual Basic 2003, Perl, COBOL 2002, PHP, and ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service, such as Software as a Service (SaaS).

[0192] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0193] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0194] Not all steps and units in the above process and system structure diagrams are mandatory; some steps or units can be omitted as needed. The execution order of each step is not fixed and can be determined as required. The device structure described in the above embodiments can be a physical structure or a logical structure. That is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0195] The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply that it is "preferred" or "advantageous" over other embodiments. Detailed descriptions are included for the purpose of providing an understanding of the described techniques. However, these techniques may be practiced without these detailed descriptions. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0196] The optional embodiments of the present specification have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present specification are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present specification, various simple modifications can be made to the technical solutions of the embodiments of the present specification, and these simple modifications all fall within the protection scope of the embodiments of the present specification.

[0197] The foregoing description of this specification is provided to enable any person skilled in the art to implement or use the content of this specification. Various modifications to the content of this specification will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the scope of protection of this specification. Therefore, this specification is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.

Claims

1. A method for training a feature extraction model, wherein, The feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model. The feature extraction model is used to extract features from image-text pairs. The method includes: The following model training process is performed iteratively using the training sample set until the training termination condition is met. Each training sample in the training sample set consists of an image-text pair composed of image data and text data: The image data of each current training sample in the current training sample set is provided to the current image feature extraction model to obtain the corresponding image features of each current training sample. The text data of each current training sample is provided to the current text vectorization model to obtain the corresponding text vector of each current training sample. For each current training sample, the corresponding image features and corresponding text vector of the current training sample are concatenated to obtain the concatenation result; For each current training sample, the corresponding text vector and the concatenation result of the current training sample are provided to the current multimodal feature fusion model to obtain the text features and multimodal features of the current training sample. The first loss value and the second loss value are determined based on the multimodal features and text features of each current training sample, respectively; Based on the first loss value and the second loss value, determine the total loss value for the current model training process; and In response to the failure to meet the training termination condition, the model parameters of the current feature extraction model are adjusted according to the total loss value, wherein the feature extraction model after the model parameter adjustment serves as the current feature extraction model in the next model training process.

2. The method as described in claim 1, wherein, The text data for each training sample in the training sample set includes the original text after keyword masking. The feature extraction model further includes a first multi-class classifier and a second multi-class classifier. The first loss value includes a multimodal feature prediction loss value, and the second loss value includes a text feature prediction loss value. The multimodal feature prediction loss value and the text feature prediction loss value are determined based on the differences between the first prediction result and the second prediction result and the masked keyword corresponding to the current training sample, respectively. The first prediction result and the second prediction result are the prediction results of the masked keywords obtained by the first multi-class classifier based on the multimodal features and the second multi-class classifier based on the text features, respectively.

3. The method as described in claim 2, wherein, The second loss value also includes a local text feature contrast loss value, which is determined based on the similarity between the corresponding image features and local text features of the current training sample. The local text features are determined based on the text features corresponding to the blocked keywords of the current training sample.

4. The method as described in claim 2 or 3, wherein, Before providing the text data of each current training sample to the current text vectorization model to obtain the corresponding text vector of each current training sample, the keyword masking process of the original text includes: The original text of each current training text is segmented into words to obtain the segmentation results; Each word in the word segmentation result is matched with a keyword dictionary to obtain a matching result. The keyword dictionary is determined based on the frequency of word occurrences in the original text of the obtained training samples. Based on each successfully matched word, we obtain the keyword set corresponding to each current training text; For each current training sample, target words in the original text of that current training sample are masked based on keywords from the keyword set corresponding to that current training sample.

5. The method of claim 4, wherein, The keyword dictionary is obtained through the following steps: The text data of each training sample obtained is segmented into words to obtain the total segmentation result; From the overall word segmentation results, words that meet the part-of-speech requirements are selected to form a candidate word set; Based on the frequency of occurrence of each word in the candidate word set in the original text of the obtained training samples, words that meet the word frequency requirements are selected from the candidate word set to form the keyword dictionary.

6. The method of claim 1, wherein, The text data of each training sample in the training sample set includes the original text without keyword masking. The first loss value includes a multimodal feature contrast loss value. The feature extraction model further includes a feature mapping layer. The multimodal feature contrast loss value is obtained based on the mapping results of the multimodal features corresponding to the current training sample and the swapped current training sample after passing through the feature mapping layer. The swapped current training sample consists of image data and original text belonging to different current training samples. The second loss value includes a global text feature contrast loss value, which is determined based on the similarity between the text features of the original text of the current training sample and the corresponding image features.

7. The method of claim 2, wherein, The text data of each training sample in the training sample set also includes the original text without keyword masking. The first loss value also includes a multimodal feature contrast loss value. The feature extraction model also includes a feature mapping layer. The multimodal feature contrast loss value is obtained based on the mapping results obtained by the feature mapping layer for the multimodal features corresponding to the current training sample and the swapped current training sample, respectively. The swapped current training sample consists of image data and original text belonging to different current training samples. The second loss value also includes a global text feature contrast loss value, which is determined based on the similarity between the text features of the original text of the current training sample and the corresponding image features.

8. The method of claim 1, wherein, The text vectorization model includes a BERT-based word vector model, and the multimodal feature fusion model includes a Transformer-based encoder.

9. A feature extraction method based on a feature extraction model, wherein, The feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model. The feature extraction method includes: The image data of the obtained image-text pair is provided to the image feature extraction model to obtain the corresponding image features of the image-text pair; The text data of the image-text pair is provided to the text vectorization model to obtain the corresponding text vector of the image-text pair; The corresponding image features and the corresponding text vector are concatenated to obtain the concatenation result; and The splicing result is provided to the multimodal feature fusion model to obtain the multimodal features of the image-text pair, wherein the multimodal feature fusion model includes a sequence model for fusing multimodal features, and the feature extraction model is trained by the method for training the feature extraction model as described in any one of claims 1 to 8.

10. An image processing method based on an image processing model, wherein, The image processing model includes a feature extraction model and a classification network, and the image processing method includes: The feature extraction method described in claim 9 is used to extract features from the obtained image-text pairs to obtain the multimodal features of the image-text pairs; The multimodal features of the image-text pair are provided to the classification network to obtain a classification result that matches the classification task corresponding to the classification network. The classification result is used to indicate at least one of the following: the answer to the question indicated by the text data of the image-text pair, and whether there is a match between the image data and the text data of the image-text pair.

11. An apparatus for training a feature extraction model, wherein, The feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model. The feature extraction model is used to extract features from image-text pairs. The device is configured to cyclically execute the model training process using a training sample set via a training unit until the training termination condition is met. Each training sample in the training sample set includes an image-text pair consisting of image data and text data. The training unit includes: The image feature extraction module is configured to provide the image data of each current training sample in the current training sample set to the current image feature extraction model to obtain the corresponding image features of each current training sample. The text vectorization module is configured to provide the text data of each current training sample to the current text vectorization model to obtain the corresponding text vector of each current training sample. The vector concatenation module is configured to concatenate the corresponding image features and the corresponding text vector of each current training sample to obtain the concatenation result. The feature generation module is configured to provide the corresponding text vector and concatenation result of each current training sample to the current multimodal feature fusion model in order to obtain the text features and multimodal features of the current training sample. The loss value determination module is configured to determine the first loss value and the second loss value based on the multimodal features and text features of each current training sample, respectively. The total loss value determination module is configured to determine the total loss value of the current model training process based on the first loss value and the second loss value; and The device further includes: The parameter adjustment unit is configured to adjust the model parameters of the current feature extraction model according to the total loss value in response to the failure to meet the training termination condition, wherein the feature extraction model after the model parameter adjustment serves as the current feature extraction model in the next model training process.

12. The apparatus of claim 11, wherein, The text data for each training sample in the training sample set includes the original text after keyword masking. The device further includes: The keyword masking unit is configured to segment the original text of each current training text to obtain segmentation results; match each word in the segmentation results with a keyword dictionary to obtain matching results, wherein the keyword dictionary is determined based on the frequency of occurrence of words in the original text of the obtained training samples; obtain a keyword set corresponding to each current training text based on each successfully matched word; and mask the target words in the original text of each current training sample based on the keywords in the keyword set corresponding to that current training sample for each current training sample.

13. A feature extraction device based on a feature extraction model, wherein, The feature extraction model includes an image feature extraction model, a text vectorization model, and a multimodal feature fusion model; the feature extraction device includes: The image feature extraction unit is configured to provide the image data of the acquired image-text pair to the image feature extraction model to obtain the corresponding image features of the image-text pair; The text vectorization unit is configured to provide the text data of the image-text pair to the text vectorization model to obtain the corresponding text vector of the image-text pair; The vector concatenation unit is configured to concatenate the corresponding image features and the corresponding text vector to obtain a concatenation result; and A multimodal feature generation unit is configured to provide the splicing result to the multimodal feature fusion model to obtain the multimodal features of the image-text pair, wherein the multimodal feature fusion model includes a sequence model for fusing multimodal features, and the feature extraction model is trained by the method for training the feature extraction model as described in any one of claims 1 to 8.

14. An image processing apparatus based on an image processing model, wherein, The image processing model includes a feature extraction model and a classification network, and the image processing device includes: The feature extraction apparatus as described in claim 13 is configured to extract features from the acquired image-text pairs to obtain the multimodal features of the image-text pairs; A classification unit is configured to provide the multimodal features of the image-text pair to the classification network to obtain a classification result that matches the classification task corresponding to the classification network, wherein the classification result is used to indicate at least one of the following: the answer to the question indicated by the text data of the image-text pair, and whether there is a match between the image data and the text data of the image-text pair.

15. An apparatus for training a feature extraction model, comprising: At least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the method for training a feature extraction model as described in any one of claims 1 to 8.

16. A feature extraction device based on a feature extraction model, comprising: At least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the feature extraction method based on the feature extraction model as described in claim 9.

17. An image processing apparatus based on an image processing model, comprising: At least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the image processing method based on the image processing model as described in claim 10.

18. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for training a feature extraction model as described in any one of claims 1 to 8, or the feature extraction method based on the feature extraction model as described in claim 9, or the image processing method based on the image processing model as described in claim 10.

19. A computer program product comprising a computer program executed by a processor to implement the method for training a feature extraction model as described in any one of claims 1 to 8, or the feature extraction method based on the feature extraction model as described in claim 9, or the image processing method based on the image processing model as described in claim 10.

Images