Method and device for updating two-way multi-modal model and computer readable storage medium
By introducing a priori reserve loss function and deviation loss function in the dual-channel multimodal model, only fine-tuning of the text encoder and freezing the image encoder, the problems of catastrophic forgetting and feature distribution deviation during the fine-tuning process are solved, and classification performance is improved And realizes unbiased merger of categories.
Patent Information
- Application Number
- CN202310524691.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-12
AI Technical Summary
Existing dual-channel multimodal models are prone to catastrophic forgetting and feature distribution deviation problems during fine-tuning, resulting in degradation of classification performance, especially poor performance in open category problems.
By introducing a priori reserve loss function and deviation loss function, only text encoders to update categories are fine-tuned, and the image encoder is frozen, ensuring that the feature distribution of unfine-tuned categories remains unchanged, and the feature distribution of fine-tuned categories is fine-tuned through deviation loss function and The feature distributions of non-fine-tuning categories are distinguished to avoid feature distribution deviations.
Without changing the untouched categories, the performance of the fine-tuned categories is improved, and the new fine-tuned categories can be merged into the original feature space without bias, improving the performance of mixed classification and avoiding catastrophic forgetting and feature distribution bias.
Smart Images

Figure CN118940791A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to the field of neural networks, and more particularly to methods of updating a two-way multimodal model. Background Art
[0002] In recent years, large-scale two-way multimodal pre-training models such as CLIP (Contrastive Language–Image Pre-training) and ALIGN have achieved great success and attracted a lot of attention and applications.
[0003] Figure 1 The schematic diagram shows the principle block diagram of CLIP. CLIP is a large-scale language-image pre-training model released by OpenAI. It uses 400 million pairs of image-text data for training, and its model framework includes a text encoder, an image encoder, etc. Figure 1 As shown in part (1) of , during training, CLIP inputs these image-text pairs into the text encoder and image encoder of the model to encode the image and text into image features and text features respectively, and inputs the cosine similarity matrix between the image features and the text features, e.g. Figure 7 The optimization goal of model training is to maximize the cosine similarity between the features of each image and their corresponding text features, and minimize the cosine similarity between the features of each image and the features of the corresponding text of other images. The CLIP model was trained using this strategy and has been applied to many downstream tasks and has achieved good results in more than 30 visual datasets.
[0004] The most striking thing is CLIP's powerful zero-shot learning capability, which does not require any training or fine-tuning and can be used directly in these downstream datasets, and can achieve good results with strong robustness. Figure 1 As shown in parts (2) and (3) of the paper, CLIP does not require any training or fine-tuning of the model when performing zero-shot image classification. Instead, it only needs to convert the name of each category into a specific prompt according to a certain text template or text conversion technique, and then calculate the cosine similarity between each category prompt and the classified image, and finally take the category with the maximum similarity as the predicted value.
[0005] Specifically, Figure 1 Parts (2) and (3) of the paper show the principle of the process of CLIP when performing classification task reasoning after pre-training: Assume that there are N categories in total, and the name of each category (such as ship, cat, etc. in the figure) is substituted into the template of CLIP (such as Figure 1After the N prompt words are input into the text encoder of CLIP, N text features will be generated, representing the corresponding categories. At this time, if any image is input into the image encoder, the image will be converted into the corresponding image feature I1. After calculating the cosine similarity between this image feature and the text feature of each category, the category with the largest similarity is selected as the prediction result.
[0006] Although CLIP has good zero-shot performance on more than 30 datasets, it still performs poorly on some categories. ImageNet-1k dataset is a large public image dataset organized according to the WordNet hierarchy, which contains 1000 categories, each with about 1300 training set images and 50 validation set images. Figure 2 As shown in the figure, CLIP is used to test the validation set of the ImageNet-1k dataset and predict 50 validation set images of each category. When predicting each image, CLIP arranges the images from large to small according to the cosine similarity of 1000 categories. If the result ranked first is equal to the true value, it is recorded as the top-1 correct image. The number of top-1 correct images is divided by the total number of images of each category, 50, which is the top-1 recall rate of each category. The recall rate of each of these 1000 categories is calculated to obtain Figure 2 Statistical histogram of .
[0007] pass Figure 2 It can be seen that although CLIP has good classification performance for many categories, it still has poor classification performance for a considerable number of categories, and the performance of these categories obviously needs to be improved. It can be seen that in real-world applications, CLIP's zero-shot classification ability inevitably has shortcomings, that is, the classification performance for some categories is poor. For these categories with poor classification performance, there is no other way except to collect samples for training. However, OpenAI used 400 million pairs of images and texts when training CLIP, which consumed a lot of time and resources. For example, 592 V100 GPUs were used for 18 days to train the CLIP RN50x64 model; 256 V100 GPUs were used for 12 days to train the CLIP ViT-L / 14@336px model. Therefore, it is unwise to train CLIP from scratch by only collecting a small amount of sample data that supplements a small number of categories. The effect will not be good, and there is a great risk. Fine-tuning CLIP is a more practical and reliable strategy.
[0008] Model fine-tuning is a common model transfer learning strategy in the field of deep learning. However, when implementing common fine-tuning methods on CLIP, two problems are encountered: catastrophic forgetting; and feature distribution bias of the fine-tuning class.
[0009] Catastrophic forgetting refers to the fact that during the fine-tuning process, the new model gradually forgets previously learned information when learning new information. For example, suppose CLIP is fine-tuned based on 1,300 training images of the "tights" class in ImageNet-1k. Although the top-1 recall rate of the "tights" class increases from 0% to 100%, the top-1 recall rate of the remaining 999 classes before and after fine-tuning is statistically analyzed. Figure 3 As shown in the figure, the top-1 recall of 809 categories has decreased. This means that the fine-tuned CLIP model has forgotten most of the knowledge learned during the previous OpenAI training. This affects the practical application of CLIP, which is obviously a situation that should be avoided as much as possible.
[0010] The feature distribution deviation of the fine-tuned class means that after fine-tuning, the feature distribution of the fine-tuned class has changed significantly. In other words, the feature distribution of the fine-tuned class is significantly different from that of most other un-fine-tuned classes, so it becomes an outlier in the overall feature space of CLIP, and can no longer be jointly classified with other un-fine-tuned classes. This seriously affects the classification advantage of CLIP for open category problems. The reason is that the image distribution used in fine-tuning training is inevitably different from the image distribution of the category used by OpenAI to train CLIP. This difference will cause the feature points of the category to become outliers in the CLIP feature space after fine-tuning. This problem is basically unavoidable.
[0011] Table 1 below shows the performance changes of CLIP on the ImageNet-1k dataset after simple fine-tuning of the category "tights". As shown in Table 1, if only the class "tights" in the ImageNet-1k dataset is fine-tuned, although the classification accuracy of this category will increase from 74% to 97%, due to the problem of catastrophic forgetting, the top-1 classification accuracy of the remaining 999 categories will drop from 75.63% to 72.18%. Worse, due to the deviation between the fine-tuned class "tights" and the remaining 999 classes, when they are mixed for classification of 1000 categories, the performance drops sharply from 75.54% to 67.14%. In other words, the fine-tuned class "tights" cannot be jointly classified with other unfine-tuned classes, making the newly trained model unable to handle open-set problems.
[0012]
[0013] Table 1
[0014] The best accuracy and best F1 score in Table 1 mean that a threshold needs to be selected during the experiment. When the cosine similarity between the image feature and the mean text feature of "[V] Tights" exceeds the threshold, the image is classified as a positive sample of "tights", otherwise it is classified as a negative sample of non-"tights". For example, the threshold can be searched by brute force search to maximize its classification accuracy, which is recorded as the best classification accuracy. Similarly, the best F1 score can also be found. Summary of the invention
[0015] A brief overview of the disclosure is provided below in order to provide a basic understanding of certain aspects of the disclosure. It should be understood that this overview is not an exhaustive overview of the disclosure. It is not intended to identify key or important parts of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is simply to provide certain concepts in a simplified form as a prelude to a more detailed description discussed later.
[0016] According to one aspect of the present invention, a method for updating a two-way multimodal model is provided, comprising: for one or more categories to be updated, obtaining a first text set including a unique identifier and the name of each category to be updated, a second text set including only the name of each category to be updated, and a third text set including the name of each non-updated category in one or more non-updated categories; constructing a first loss function by minimizing the distance between the mean of the updated text features obtained based on the first text set and belonging to each category to be updated and the features of the training samples belonging to the category to be updated; constructing a second text set including only the name of each category to be updated and the name of each non-updated category in the one or more non-updated categories; constructing a third ... A second loss function is constructed by minimizing the distance between new text features; a third loss function is constructed by maximizing the distance between the mean of original text features obtained based on the third text set and belonging to each non-updated category and the mean of updated text features obtained based on the first text set and belonging to each to-be-updated category; and a two-way multimodal model is updated by minimizing the total loss function consisting of the weighted sum of the first loss function, the second loss function and the third loss function, wherein the original text features are features obtained by the original two-way multimodal model, and the updated text features are features obtained by the updated two-way multimodal model.
[0017] Preferably, in the case of updating for a specific category, the first loss function minimizes the sum or mean of the distances between the mean of the updated text features belonging to the specific category obtained based on the first text set and the features of each training sample calculated for each batch belonging to the specific category.
[0018] Preferably, when updating for a specific category, the first loss function minimizes the distance between the mean of updated text features obtained based on the first text set and belonging to the specific category and the mean of features of training samples belonging to the specific category.
[0019] Preferably, in the case of updating for more than two categories, a similarity matrix between the updated text features obtained based on the first text set and belonging to each category to be updated and the mean of the training sample features belonging to each category to be updated is obtained by calculation, wherein the first loss function makes the diagonal elements of the similarity matrix maximized and the remaining elements minimized.
[0020] Preferably, the second loss function is a cosine prior preserving loss function or a two-norm prior preserving loss function.
[0021] Preferably, the cosine prior preservation loss function is constructed by minimizing the sum of cosine distances between the original text features and the updated text features belonging to each category to be updated obtained based on the second text set.
[0022] Preferably, the two-norm prior-preserving loss function is constructed by minimizing the sum of the Euclidean distances between the normalized unit vectors of the original text features and the normalized unit vectors of the updated text features obtained based on the second text set and belonging to each category to be updated.
[0023] Preferably, the third loss function is a deviation loss function.
[0024] Preferably, the deviation loss function is constructed by maximizing the sum of the cosine distances between the means of the original text features belonging to each non-updated category obtained based on the third text set and the means of the updated text features belonging to each to-be-updated category obtained based on the first text set.
[0025] Preferably, the unique identifier is an identifier whose meaning cannot be discerned by the two-way multimodal model.
[0026] Preferably, different update categories may use the same unique identifier, or respectively use different unique identifiers.
[0027] Preferably, a plurality of text sets belonging to each category are obtained by expanding the name of the category using one or more text templates or synonym conversions.
[0028] Preferably, the training samples include images, videos or audios.
[0029] According to another aspect of the present invention, there is provided an apparatus for updating a two-way multimodal model, comprising: an obtaining unit, which is configured to obtain, for one or more categories to be updated, a first text set including a unique identifier and the name of each category to be updated, a second text set including only the name of each category to be updated, and a third text set including the name of each non-updated category in one or more non-updated categories; a first construction unit, which is configured to construct a first loss function by minimizing the distance between the mean of the updated text features belonging to each category to be updated obtained based on the first text set and the features of the training samples belonging to the category to be updated; a second construction unit, which is configured to construct a first loss function by minimizing the distance between the mean of the updated text features belonging to each category to be updated obtained based on the first text set and the features of the training samples belonging to the category to be updated; a third construction unit, which is configured to construct a third loss function by maximizing the distance between the mean of the original text features obtained based on the third text set and belonging to each non-updated category and the mean of the updated text features obtained based on the first text set and belonging to each to-be-updated category; and an updating unit, which is configured to update the two-way multimodal model by minimizing the total loss function consisting of the weighted sum of the first loss function, the second loss function and the third loss function, wherein the original text features are features obtained by the original two-way multimodal model, and the updated text features are features obtained by the updated two-way multimodal model.
[0030] According to other aspects of the present invention, corresponding computer program codes, computer readable storage media and computer program products are also provided.
[0031] By means of the method and device for updating the dual-path multimodal model of the present invention, the performance of the fine-tuned class is improved without changing the remaining non-fine-tuned classes, and the new fine-tuned class can be smoothly merged into the original feature space without bias, thereby improving the classification performance when the fine-tuned class and the non-fine-tuned class are mixed for classification.
[0032] These and other advantages of the present invention will become more apparent through the following detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to further illustrate the above and other advantages and features of the present disclosure, the specific embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings. The drawings together with the following detailed description are included in this specification and form a part of this specification. Elements with the same function and structure are represented by the same reference numerals. It should be understood that these drawings only describe typical examples of the present disclosure and should not be regarded as limiting the scope of the present disclosure. In the drawings:
[0034] Figure 1 The principle block diagram of CLIP is schematically shown;
[0035] Figure 2 It is a statistical histogram of the first recall rate of each category in the validation set of CLIP for the ImageNet-1k dataset;
[0036] Figure 3 It is a statistical histogram of the change in the first recall rate of each category in the ImageNet-1k validation set after a simple fine-tuning of the "tights" category of CLIP;
[0037] Figure 4 Schematically shows a principle block diagram of a method according to the present disclosure;
[0038] Figure 5 is a flow chart of a method for updating a two-way multimodal model according to an embodiment of the present disclosure;
[0039] Figure 6 Schematically shows Figure 4 The loss function l is used for fine-tuning f Related sub-block diagrams;
[0040] Figure 7 The calculation block diagram of the similarity matrix S is shown;
[0041] Figure 8 Schematically shows Figure 4 The loss function l is retained with the prior p Related sub-block diagrams;
[0042] Fig. 9 Schematically shows Figure 4 The deviation loss function l b Related sub-block diagrams;
[0043] Fig.10 is a block diagram of an apparatus for updating a dual-path multimodal model according to an embodiment of the present disclosure;
[0044] Fig.11 is a block diagram of an exemplary structure of a general-purpose personal computer in which the methods and / or apparatuses according to embodiments of the present invention may be implemented. DETAILED DESCRIPTION
[0045] Exemplary embodiments of the present disclosure will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual implementation are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the process of developing any such actual implementation in order to achieve the developer's specific goals, such as meeting those constraints related to the system and business, and these constraints may vary from implementation to implementation. In addition, it should be understood that although the development work may be very complex and time-consuming, it is only a routine task for those skilled in the art who benefit from the content of this disclosure.
[0046] It is also necessary to explain here that, in order to avoid obscuring the present disclosure due to unnecessary details, only the device structure and / or processing steps closely related to the scheme according to the present disclosure are shown in the accompanying drawings, while other details that are not very relevant to the present disclosure are omitted.
[0047] As mentioned above, there are two problems encountered when implementing common fine-tuning methods for dual-path multimodal models such as CLIP: catastrophic forgetting; and feature distribution bias of the fine-tuning class. In response to these two problems, the present disclosure proposes a new fine-tuning method for language-image pre-trained models. This method only fine-tunes the text encoder of CLIP while freezing the image encoder of CLIP. By introducing a prior preservation loss function and a bias loss function in the common fine-tuning loss, this method not only solves the catastrophic forgetting problem, but also solves the bias problem of the fine-tuning class.
[0048] Specifically, for the problem of catastrophic forgetting, the solution of the method of the present disclosure is as follows.
[0049] Assuming that only the category "car" is fine-tuned among the N categories, in addition to the original N prompt words, a new prompt word "[V] photo of car" is added, where [V] is a word that CLIP cannot recognize before fine-tuning, such as skszmj. This new prompt word represents the new knowledge that CLIP has learned about the category "car" after fine-tuning. The new knowledge is obtained by collecting K training images of "car" and fine-tuning CLIP's old knowledge of "car" as the initial value based on these training images to obtain the new knowledge. In other words, the new knowledge is a fine-tuning correction of the old knowledge based on the collected training images.
[0050] While learning new knowledge, the method disclosed in the present invention also needs to avoid the problem of catastrophic forgetting, that is: keep the CLIP text features of all other texts without [V], so that old knowledge will not be forgotten after fine-tuning. Based on simple mathematical knowledge, this goal can be equivalent to: the angle of the text feature vector of all CLIP text features without [V] will not change after fine-tuning. When the text feature angle remains unchanged, the cosine similarity between these prompt words without [V] and the image features of any image will not change before and after fine-tuning. In other words, CLIP will not forget the knowledge of all texts without [V] before and after fine-tuning.
[0051] Obviously, the number of texts without [V] is infinite and cannot be enumerated one by one. The strategy adopted by the method disclosed in the present invention is that this optimization goal can be simplified to be equivalent to: it is only necessary to ensure that the angle of the text feature of the prompt word "[fine-tuned category name]" remains unchanged before and after fine-tuning. When this optimization goal is met, the feature angle of any text without [V] will not change much.
[0052] This method does not affect the results of other original prompt words of CLIP, and can also call the fine-tuned model results by using the prompt word "[V][fine-tuning class name]", thereby improving the classification performance of the fine-tuning class in the test set. Therefore, the method of the present disclosure is more flexible in use.
[0053] For the problem of feature distribution deviation, the method of the present disclosure solves the problem as follows.
[0054] Although the fine-tuned prompt word "[V] Photos of cars" will learn the knowledge of the fine-tuning training set under the action of the fine-tuning function, it will bring new deviations. Specifically, compared with the old prompt word "photos of cars", the cosine similarity of many other un-fine-tuned categories of images to the new prompt word "[V] Photos of cars" will become larger. When this cosine similarity becomes large to a certain extent, it is likely to exceed the cosine similarity between the images and their own category prompt words. At this time, if the new prompt words are used to classify N categories, then CLIP will misclassify these other categories of images as "cars", and these errors will not occur before fine-tuning.
[0055] The most direct solution is to continue to collect some of these wrong unfine-tuned images and fine-tune them together, so that they can be distinguished from the fine-tuned "car". However, this is unrealistic because it takes more time and cost to collect and annotate more samples, and there are infinite categories other than "car" in practical applications. Users may enter or be interested in any category, and it is laborious and unrealistic to exhaustively collect and annotate samples one by one.
[0056] Therefore, the strategy of the present disclosure is to assume that CLIP already has a good knowledge of all texts and images of the non-fine-tuned class, so their text features can be directly used instead of the process of collecting and annotating image samples, and their text features can be used instead of image features as category features. With this idea, the fine-tuned class and the non-fine-tuned class can be jointly classified, and the classification performance is also improved. In other words, the feature distribution extracted by the fine-tuned model for the prompt word "[V][fine-tuned class name]" has no deviation from the original CLIP feature space.
[0057] For the problem that cannot be exhaustively enumerated, it can be solved by appropriately selecting reference categories. Compared with the collection and annotation of images, text acquisition is quite easy, flexible and has no boundaries. Therefore, through the method of the present disclosure, a large amount of text far exceeding N categories can be easily written as reference categories, and even categories that are semantically close to "car" but different can be deliberately selected as reference categories. This solves the problem of the infinite number of categories to a certain extent. The fine-tuned CLIP will not be limited to these N categories, and will be more robust.
[0058] Compared with other CLIP-based fine-tuning methods, the method of the present disclosure has the following advantages:
[0059] -A prior preservation loss function is proposed to freeze other previously learned knowledge that does not need to be fine-tuned while fine-tuning the features of certain prompt words in CLIP. In other words, the angle of the CLIP text features of all texts without [V] will not change after fine-tuning. Based on this loss function, the catastrophic forgetting problem when fine-tuning CLIP can be solved.
[0060] -Propose a deviation loss function to eliminate the deviation of the fine-tuned cue word features from the original feature space distribution of CLIP. For example, let the text features of the fine-tuned category "car" get closer to the fine-tuned training image set of "car" while moving away from the text features of other reference categories. Based on this loss function, new fine-tuned classes can be inserted into the CLIP feature space without any performance degradation.
[0061] - Experimental results on the ImageNet-1k dataset show that the proposed method can improve the performance of fine-tuned classes without changing the remaining non-fine-tuned classes, and the new fine-tuned classes can be merged into the original CLIP feature space without bias and smoothly. Therefore, the classification performance of combining fine-tuned and non-fine-tuned classes is also improved.
[0062] Figure 4 It should be understood that although the present disclosure is described by taking CLIP as an example, the present disclosure is not limited thereto, but can be applied to any other suitable two-way multi-modal model.
[0063] It should be pointed out that Figure 4 The left dashed box and the right dashed box in represent the same text encoder 40. Figure 4 The two dotted boxes in the figure indicate that the same text encoder will produce different responses when different inputs are given. Specifically, the black text encoder network in the left dotted box is the parameter state before fine-tuning. After fine-tuning begins, the network structure of the text encoder will not change, but the parameters will change. The gray text encoder network in the right dotted box is the parameter state during or after fine-tuning. That is, after fine-tuning begins or ends, the parameters of the black text encoder network in the left dotted box will change, thus becoming the gray text encoder network in the right dotted box. In terms of the specific implementation of the code, the two dotted boxes are the same piece of code, the left box represents the network before fine-tuning begins, and the right box represents the network during or after fine-tuning.
[0064] It should also be noted that Figure 4 The "shared weights" in the figure means that the two networks inside the dotted box refer to the same network in the specific implementation of the code, rather than two different networks.
[0065] It should also be noted that Figure 4 The “frozen image encoder 41 ” here means that the method of the present disclosure only fine-tunes the text encoder of CLIP, while the parameters of the image encoder of CLIP remain unchanged.
[0066] Combination Figure 4 The principle block diagram shown in the figure requires the following fine-tuning training data:
[0067] - Several fine-tuning training images corresponding to the categories that need fine-tuning. The number of images p for each category can be a few, dozens or more;
[0068] -The names of the Q categories that need fine-tuning, e.g. Figure 4 "Jeep" in , and some other K categories that do not require fine-tuning ( Figure 4 The name of the reference category in Figure 4 The number of reference categories depends on the needs. Generally speaking, the more the better, and the wider the distribution, the better;
[0069] - a unique identifier representing the fine-tuned sample: [V], which can be any rare identifier of CLIP. For example, [V] used in the experiments of this disclosure is: "skszmj";
[0070] -(Optional) M text templates, used to expand the category names into sentences to better match the original settings of CLIP. For example, the template commonly used by CLIP during training is "Photos of [category name]", so continuing to use this template during prediction or fine-tuning will slightly improve performance. The number of templates can be 0, that is, no templates are used, or it can be a few, dozens, or even more. Similarly, the category names can be converted into synonyms, antonyms, or related phrases with better classification performance according to GPT-3, GPT-3.5, GPT-4, Chat-GPT, or dictionaries, thereby improving the classification performance of CLIP.
[0071] It should be understood that although the training images are used as training samples above, the present disclosure is not limited thereto, and other suitable samples, such as video or audio, may be used.
[0072] Combine the following Figure 4 and Figures 6 to 9 To describe in detail Figure 5 A method 500 of updating a two-way multimodal model according to the present disclosure.
[0073] It should be noted that in the context of the present disclosure, the terms "fine-tune" and "update" are used interchangeably.
[0074] like Figure 5 As shown, first, in step 501, for one or more categories to be updated, a first text set including a unique identifier and the name of each category to be updated, a second text set including only the name of each category to be updated, and a third text set including the name of each non-updated category in one or more non-updated categories are obtained.
[0075] Specifically, in this embodiment, reference Figure 4 , for the category to be updated "Jeep", a first text set "[V] Jeep" including a unique identifier and a category name can be obtained. Optionally, the first text set can also include M texts containing "[V] Jeep" generated using M templates. The second text set is "Jeep", or optionally, M texts containing "Jeep" generated using M templates. The third text set is K reference category names such as dog, cat, etc., or optionally, M×K texts containing K reference category names generated using M templates.
[0076] It should be noted that the unique identifier [V] refers to an identifier whose meaning cannot be recognized by the dual-channel multimodal model.
[0077] Optionally, different update categories can use the same unique identifier or use different unique identifiers respectively.
[0078] Next, in step 502, a first loss function is constructed by minimizing the distance between the mean of the updated text features belonging to each category to be updated obtained from the first text set and the features of the training samples belonging to the category to be updated.
[0079] Specifically, in this embodiment, referring to Figure 6 , p (where p is an integer greater than or equal to 3) images of the category "Jeep" are input into the frozen (unchanged) CLIP image encoder 41, and p N-dimensional image feature vectors I i ∈R N can be obtained, where 0 ≤ i < p. After combining "[V] Jeep" with M text templates, M text phrases of the category "Jeep" can be obtained, and then they are input into the text encoder 40 of the fine-tuned CLIP model, and M N-dimensional text feature vectors of the iteratively updated category "Jeep" can be obtained.
[0080] And so on, performing the above operations for each category to be updated, and the iteratively updated text features T V ∈R Q×(M)×N can be obtained, which is composed of Q × M N-dimensional text feature vectors, where Q is the number of categories to be updated and M is the number of templates. After normalizing and taking the mean of the M vectors for each category, the mean text feature corresponding to each category is obtained The optimization objective of the first loss function called the fine-tuning loss function is to make the two feature vectors I and as close to each other in terms of angle as possible.
[0081] In practice, there are two cases.
[0082] In the first case, that is, when only one category needs to be updated, the fine-tuning loss function l f is constructed as follows:
[0083]
[0084] Among them,
[0085] The two formulas in formula (1) are mathematically equivalent, except for slight differences in the specific algorithm code implementation. The left formula is to calculate the distance between the features of each image and Calculate the cosine similarity. At this time, the fine-tuned loss function is the sum of the cosine similarities of each batch (or equivalently the mean), and the batch size b during training is less than the total number of images. When the training batch size b is equal to the total number of images of this category in the training set p, the formula on the left can be equivalently simplified to the formula on the right, that is, first calculate the feature I of each image, and then obtain its normalized mean feature in advance Finally, we only need to calculate the mean image feature and The cosine similarity between them is sufficient.
[0086] In the second case, that is, when more than one category needs to be fine-tuned, it can be based on Figure 7 The similarity matrix S in the multi-category case shown is used for calculation. There are also two cases for constructing the fine-tuning loss function.
[0087] The first case is similar to the loss function used during CLIP training, and the training images are input in batches according to the training. Figure 7 The i-th row of the S matrix in i Represents the image features of the i-th image in the batch, which is randomly selected from the i-th category training set. is the mean text feature of the qth category, and the (i, q)th element in the S matrix is The goal of fine-tuning the loss function at this time is to make the diagonal elements of the S matrix as large as possible and the remaining elements as small as possible. Therefore, the loss function used in CLIP training can be used to calculate the sum (or mean) of the cross entropy by row and the cross entropy by column:
[0088] l f =[CE(S,axis=0)+CE(S,aix=1)] / 2 (2)
[0089] In the second case, Figure 7 The i-th row of the S matrix in i It represents the image features of the mean after all the training images of the ith category are normalized, rather than the features of a specific image. At this time, the training images are no longer input in training batches. Instead, the mean image features of each category are calculated before training begins, and then the network parameters are iteratively updated accordingly. The (i, q)th element in the S matrix is the cosine similarity between the mean image features of the ith category and the mean text features of the qth category. From this, we can calculate the similarity matrix S between the mean image features and the mean text features of each category. Similar to formula 2, we can also calculate the cross entropy by row and the sum (or mean) of the cross entropy by column for the matrix S to construct a fine-tuning loss function.
[0090] Next, in step 503, a second loss function is constructed by minimizing the distance between the original text features obtained based on the second text set and the updated text features.
[0091] Specifically, in this embodiment, referring to Figure 8 , the category name to be updated without the unique identifier [V] (such as Figure 8 "Jeep" in 1,j ∈R N , 0 ≤ j < QM. Then, when fine-tuning the CLIP model, these QM text phrases are re-input into the fine-tuned CLIP text encoder 40 for each iterative update, so as to obtain the updated Q×M N-dimensional CLIP text features T 2,j ∈R N , 0 ≤ j < QM. The optimization objective of the second loss function called the prior retention loss function l p is to keep the text features T 1,j and T 2,j consistent in angle before and after the update:
[0092]
[0093] where Q is the number of categories to be updated. These two formulas are roughly equivalent in mathematics except for a square root difference, and the latter is more sensitive in terms of accuracy. The former formula is called the cosine prior retention loss function, and the latter formula is the two-norm prior retention loss function.
[0094] Optionally, the normalized mean text features and of each category can also be calculated, and then the mean text features of each category are kept unchanged in angle before and after fine-tuning:
[0095]
[0096] It should be noted that formula (4) is inferior to formula (3) both in terms of accuracy and the performance of the fine-tuned model.
[0097] Next, in step 504, a third loss function is constructed by maximizing the distance between the mean of the original text features belonging to each non-updated category obtained based on the third text set and the mean of the updated text features belonging to each category to be updated obtained based on the first text set.
[0098] Specifically, in this embodiment, referring to Fig. 9First, K reference category names without unique identifiers [V] such as "dog", "cat", etc. are combined with M text templates to obtain M text phrases of K categories. Then, the K×M text phrases are input into the original CLIP text encoder 40 to obtain K×M text features. The K×M text features are normalized and the mean of each category is calculated to obtain the original CLIP mean text features of the K reference categories:
[0099]
[0100] Next, the name of the category to be updated (e.g. Fig. 9 After combining the "[V] Jeep" in the text template with M text templates, M text phrases can be obtained. Assuming that there are Q categories to be updated, Q×M text phrases are obtained, and they are input into the fine-tuned CLIP text encoder 40 to obtain updated Q×M text features. The updated Q×M text features are normalized and the mean of each category is calculated to obtain the updated mean text features of the Q categories to be updated:
[0101]
[0102] Assuming that CLIP performs well on the reference categories, the image features of these reference categories must be close to their category mean text features. If the mean text features of the updated category can be made far away from the mean text features of these reference categories, then the images of the reference category will not be misclassified as the category to be updated. Therefore, when the reference category is distributed widely enough, in theory, the category to be updated and any non-updated category can be mixed together for classification. Therefore, the deviation loss function l is constructed as follows b :
[0103]
[0104] That is to say, the text mean features of any reference category and any category to be updated should be far enough apart.
[0105] Finally, in step 505, the two-way multimodal model is updated by minimizing the total loss function consisting of the weighted sum of the first loss function, the second loss function and the third loss function.
[0106] Specifically, in this embodiment, the total loss function arg min θ loss is composed of the fine-tuning loss function l f , deviation loss function l b and the prior preservation loss function l p Weighted and composed:
[0107] arg min θ loss = l f +w b l b +w p l p (8)
[0108] Among them, w b ≥0 and w p ≥0 are the deviation loss function l b and the prior preservation loss function l p weights. Empirically, they can range from 2 to 1000. However, w b and w p The value of can be set arbitrarily according to the experimental object and experimental requirements, and is not limited to the range of 2-1000, but can also be 0.001, 1, 10000, etc.
[0109] By making the total loss function arg min θ The loss is minimal, so fine-tune Figure 4 The CLIP model is shown.
[0110] Ablation experiment
[0111] In order to verify the advantages of the method of the present disclosure, a large number of ablation experiments have been conducted. The experiment was carried out in the public ImageNet-1k data. The data set has a total of 1000 categories, each category has about 1300 training set images and 50 validation set images. In this experiment, only the "tights" category, which CLIP is not good at, is fine-tuned, so the fine-tuning training set is only 1300 training images of "tights", and the test set is 50×1000 validation set images of 1000 categories. In this experiment, the 80 templates used in the ImageNet data given in the original CLIP code are used, the learning rate is 1e-6, the optimizer is the Adam optimizer, the unique identifier [V] is set to "skszmj", and the reference category is the remaining 999 categories.
[0112] As shown in Table 2 below, for each fine-tuned CLIP model, three different test experiments were conducted, namely: a single classification experiment for the category "tights" only, to evaluate the performance change of the fine-tuned category "tights"; a classification experiment of 999 categories of the remaining 999 un-fine-tuned categories, to evaluate the degree of catastrophic forgetting of the fine-tuned model; a classification experiment of 1000 categories of mixed classification of the fine-tuned category "tights" and the remaining 999 un-fine-tuned categories, to evaluate the degree of deviation of the fine-tuned model. The above three experiments, that is, each row in Table 2, use the fine-tuned CLIP model obtained under the same loss function and the same number of iterations.
[0113]
[0114]
[0115] Table 2
[0116] In the first experiment (corresponding to the 2nd and 3rd columns in Table 2), not only 50 images of the "tights" validation set were used as positive samples, but also 50 categories were randomly selected from the remaining 999 categories in the validation set, and 1 image was randomly selected from each category to form 50 negative samples. The classification accuracy of these 100 samples is used as the classification accuracy of the category "tights". These 50 negative samples cannot appear in the second experiment. By observing the ablation experiment of the first experiment, it can be found that the classification accuracy of the category "tights" can be greatly improved from 74% to 97% by training with the fine-tuning loss function, and whether the prior retention loss function or the deviation loss function is added, it will not affect the performance of the fine-tuning loss function, nor will it affect the improvement of the fine-tuned category "tights".
[0117] In the second experiment (corresponding to the 4th and 5th columns in Table 2), 491,100 images in the ImageNet-1k validation set, except for the 100 images used in the first experiment, were analyzed and classified into 999 categories except for the category "tights". The top-n accuracy in the evaluation index means that each image is predicted using the model. The similarity score of each category is given in the model prediction result, and it is arranged from large to small. If the top n most likely categories contain the correct category, then the prediction result of this image is considered correct, and the number of correctly predicted images divided by the total number of images is the top-n accuracy of the model in the image test set. According to the 4th and 5th columns of Table 2, when there is only a simple fine-tuning loss function, the top-1 accuracy is reduced from 75.63% to 72.18% due to the catastrophic forgetting problem during the fine-tuning process; if the bias loss function is introduced, although this problem will be alleviated, the top-1 accuracy still drops to 75.05%; and this problem can only be solved when the prior-preserving loss function is introduced, among which the two-norm prior-preserving loss function is better than the cosine prior-preserving loss function in terms of accuracy. When the two-norm prior-preserving loss function is used, not only does the catastrophic forgetting problem disappear, but the remaining 999 unfine-tuned categories also benefit from the learning update of the irrelevant fine-tuning category "tights", which slightly improves their performance by about 0.1% to 0.4%.
[0118] In the third experiment (corresponding to the 6th and 7th columns in Table 2), the fine-tuned category "tights" and the remaining 999 categories without fine-tuning are jointly classified, that is, the entire 50,000 validation set images of ImageNet-1k are tested and divided into 1,000 categories, where the category "tights" is classified using the prompt word "[V] tights" used in fine-tuning. Comparing the performance of the same model in each row of Table 2 in the first and second experiments, it can be found that the unfine-tuned CLIP does not have the problem of feature distribution deviation, and the performance does not change significantly after joint classification; however, once a simple fine-tuning is performed using the fine-tuning loss function, a deviation occurs between the fine-tuned "tights" and the other 999 categories without fine-tuning, and the performance of the joint classification will be significantly reduced compared with the single "tights" classification and the classification of the 999 categories; and after the introduction of the prior preservation loss function, this deviation problem is aggravated, and the performance of the joint classification has dropped sharply. The more thoroughly the catastrophic forgetting problem is solved, the more serious the deviation problem is. Once the bias loss function is introduced, this problem is solved and the performance of the joint classification is no different than that of the separate “tights” classification and the 999-class classification.
[0119] In summary, the ablation experiment shows that the three loss functions used in the method 500 according to the embodiment of the present disclosure have achieved their respective intended purposes, have their own functions, are indispensable, and when the three are used simultaneously, the performance is optimal. Not only can the classification performance of the category "tights" be improved by collecting "tights" training data, but the other irrelevant un-fine-tuned categories will not suffer from catastrophic forgetting problems, and the performance will also benefit from it, with a slight improvement of about 0.4%. The features of the new prompt word "[V] Tights" after fine-tuning are also well integrated with the CLIP feature space, and there is no deviation problem. When performing joint classification with un-fine-tuned categories, the performance will not only not decrease, but also improve by about 0.3% compared with the un-fine-tuned ones. It should be understood that although this improvement seems not high, in fact, the category "tights" only accounts for 0.1% of the number in the entire validation set. Therefore, in essence, the improvement is already very high because it means the performance improvement of the entire feature space rather than just the category "tights".
[0120] The methods discussed above can be implemented entirely by a computer executable program, or partially or completely by hardware and / or firmware. When it is implemented by hardware and / or firmware, or when a computer executable program is loaded into a hardware device that can run the program, the device for updating the dual-path multimodal model described below is implemented. Below, an overview of these devices is given without repeating some of the details discussed above, but it should be noted that although these devices can perform the methods described above, the methods do not necessarily use those components of the described devices or are not necessarily performed by those components.
[0121] Fig.10A device 100 for updating a dual-path multimodal model according to an embodiment is shown. The device 100 includes an obtaining unit 101, a first construction unit 102, a second construction unit 103, a third construction unit 104, and an updating unit 105. The obtaining unit 101 is used to obtain, for one or more categories to be updated, a first text set including a unique identifier and the name of each category to be updated, a second text set including only the name of each category to be updated, and a third text set including the name of each non-updated category in one or more non-updated categories. The first construction unit 102 is used to construct a first loss function by minimizing the distance between the mean of the updated text features obtained based on the first text set and belonging to each category to be updated and the features of the training samples belonging to the category to be updated. The second construction unit 103 is used to construct a second loss function by minimizing the distance between the original text features obtained based on the second text set and the updated text features. The third construction unit 104 is used to construct a third loss function by maximizing the distance between the mean of the original text features obtained based on the third text set and belonging to each non-updated category and the mean of the updated text features obtained based on the first text set and belonging to each category to be updated. The updating unit 105 is used to update the two-way multimodal model by minimizing the total loss function composed of the weighted sum of the first loss function, the second loss function and the third loss function. The original text features are features obtained by the original two-way multimodal model, and the updated text features are features obtained by the updated two-way multimodal model.
[0122] Preferably, in the case of updating for a specific category, the first loss function minimizes the sum or mean of the distances between the mean of the updated text features belonging to the specific category obtained based on the first text set and the features of each training sample calculated for each batch belonging to the specific category.
[0123] Preferably, when updating for a specific category, the first loss function minimizes the distance between the mean of updated text features obtained based on the first text set and belonging to the specific category and the mean of features of training samples belonging to the specific category.
[0124] Preferably, in the case of updating for more than two categories, a similarity matrix between the updated text features obtained based on the first text set and belonging to each category to be updated and the mean of the training sample features belonging to each category to be updated is obtained by calculation, wherein the first loss function makes the diagonal elements of the similarity matrix maximized and the remaining elements minimized.
[0125] Preferably, the second loss function is a cosine prior preserving loss function, and the cosine prior preserving loss function is constructed by minimizing the sum of cosine distances between original text features and updated text features belonging to each category to be updated obtained based on the second text set.
[0126] Preferably, the second loss function is a two-norm prior-preserving loss function, and the two-norm prior-preserving loss function is constructed by minimizing the sum of the Euclidean distances between the normalized unit vectors of the original text features belonging to each category to be updated obtained based on the second text set and the normalized unit vectors of the updated text features.
[0127] Preferably, the third loss function is a deviation loss function, and the deviation loss function is constructed by maximizing the sum of the cosine distances between the means of the original text features belonging to each non-updated category obtained based on the third text set and the means of the updated text features belonging to each category to be updated obtained based on the first text set.
[0128] Fig.10 The apparatus 100 for updating the dual-path multimodal model shown corresponds to Figure 5 Therefore, the relevant details of each component in the apparatus 100 for updating the dual-path multimodal model have been described in detail. Figure 5 The method 500 for updating the dual-path multimodal model is described in detail and will not be repeated here.
[0129] Each component module or unit in the above device can be configured by software, firmware, hardware or a combination thereof. The specific means or methods that can be used for configuration are well known to those skilled in the art and will not be described in detail here. In the case of software or firmware implementation, the data is transmitted from a storage medium or a network to a computer (e.g., a computer) with a dedicated hardware structure. Fig.11 The general-purpose computer 1100 shown in the figure is installed with the programs constituting the software, and when the various programs are installed, the computer can execute various functions, etc.
[0130] Fig.11 is a block diagram of an exemplary structure of a general-purpose personal computer in which the method and / or apparatus according to an embodiment of the present invention can be implemented. Fig.11 As shown, a central processing unit (CPU) 1101 performs various processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage device 1108 to a random access memory (RAM) 1103. In the RAM 1103, data required when the CPU 1101 performs various processes and the like is also stored as needed. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output interface 1105 is also connected to the bus 1104.
[0131] The following components are connected to the input / output interface 1105: an input device 1106 (including a keyboard, a mouse, etc.), an output device 1107 (including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage device 1108 (including a hard disk, etc.), a communication device 1109 (including a network interface card such as a LAN card, a modem, etc.). The communication device 1109 performs communication processing via a network such as the Internet. A drive 1110 may also be connected to the input / output interface 1105 as needed. A removable medium 1111 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is installed on the drive 1110 as needed, so that a computer program read therefrom is installed in the storage device 1108 as needed.
[0132] In the case where the above-described series of processing is realized by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 1111 .
[0133] It should be understood by those skilled in the art that such storage media is not limited to Fig.11 The removable medium 1111 shown has a program stored therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 1111 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including minidiscs (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be a ROM 1102, a hard disk included in the storage device 1108, or the like, in which the program is stored and distributed to the user together with the device containing them.
[0134] The present invention also provides corresponding computer program codes and a computer program product storing machine-readable instruction codes. When the instruction codes are read and executed by a machine, the method according to the embodiment of the present invention can be executed.
[0135] Accordingly, storage media configured to carry the program product storing the machine-readable instruction code are also included in the disclosure of the present invention, including but not limited to floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, and the like.
[0136] Through the above description, the embodiments of the present disclosure provide the following technical solutions, but are not limited thereto.
[0137] Note 1. A method for updating a dual-path multimodal model, comprising:
[0138] For one or more to-be-updated categories, obtaining a first text set including a unique identifier and a name of each to-be-updated category, a second text set including only the name of each to-be-updated category, and a third text set including the name of each non-updated category in one or more non-updated categories;
[0139] Constructing a first loss function by minimizing the distance between the mean of the updated text features obtained based on the first text set and belonging to each category to be updated and the features of the training samples belonging to the category to be updated;
[0140] Constructing a second loss function by minimizing the distance between the original text features obtained based on the second text set and the updated text features;
[0141] constructing a third loss function by maximizing the distance between the mean of the original text features obtained based on the third text set and belonging to each non-updated category and the mean of the updated text features obtained based on the first text set and belonging to each to-be-updated category; and
[0142] The two-way multimodal model is updated by minimizing the total loss function consisting of the weighted sum of the first loss function, the second loss function, and the third loss function,
[0143] Among them, the original text features are features obtained by the original two-way multimodal model, and the updated text features are features obtained by the updated two-way multimodal model.
[0144] Note 2. The method according to Note 1, wherein, in the case of updating for a specific category, the first loss function minimizes the sum or mean of the distances between the mean of the updated text features obtained based on the first text set and belonging to the specific category and the features of each training sample calculated for each batch belonging to the specific category.
[0145] Note 3. According to the method of 1, when updating for a specific category, the first loss function minimizes the distance between the mean of the updated text features belonging to the specific category obtained based on the first text set and the feature mean of the training samples belonging to the specific category.
[0146] Note 4. A method according to Note 1, wherein, in the case of updating for more than two categories, a similarity matrix is obtained by calculating the means of updated text features of each category to be updated and training sample features of each category to be updated obtained based on the first text set, and wherein the first loss function makes the diagonal elements of the similarity matrix as large as possible and the remaining elements as small as possible.
[0147] Note 5. A method according to any one of Notes 1 to 4, wherein the second loss function is a cosine prior preserving loss function or a bi-norm prior preserving loss function.
[0148] Note 6. The method according to Note 5, wherein a cosine prior preservation loss function is constructed by minimizing the sum of cosine distances between original text features and updated text features belonging to each category to be updated obtained based on the second text set.
[0149] Note 7. A method according to Note 5, wherein a two-norm prior-preserving loss function is constructed by minimizing the sum of the Euclidean distances between the normalized unit vectors of the original text features belonging to each category to be updated obtained based on the second text set and the normalized unit vectors of the updated text features.
[0150] Note 8. A method according to any one of Notes 1 to 4, wherein the third loss function is a deviation loss function.
[0151] Note 9. A method according to Note 8, wherein a deviation loss function is constructed by maximizing the sum of cosine distances between the means of original text features belonging to each non-updated category obtained based on the third text set and the means of updated text features belonging to each category to be updated obtained based on the first text set.
[0152] Note 10. A method according to any one of Notes 1 to 4, wherein the unique identifier is an identifier whose meaning cannot be recognized by the two-way multimodal model.
[0153] Note 11. The method according to Note 10, wherein different update categories can use the same unique identifier, or use different unique identifiers respectively.
[0154] Note 12. The method according to any one of Notes 1 to 4, wherein a plurality of text sets belonging to each category are obtained by expanding the name of the category using one or more text templates or synonym conversions.
[0155] Note 13. A method according to any one of Notes 1 to 4, wherein the training samples include images, videos or audios.
[0156] Note 14. A device for updating a dual-path multimodal model, comprising:
[0157] an obtaining unit configured to obtain, for one or more to-be-updated categories, a first text set including a unique identifier and a name of each to-be-updated category, a second text set including only the name of each to-be-updated category, and a third text set including the name of each non-updated category in one or more non-updated categories;
[0158] A first construction unit configured to construct a first loss function by minimizing the distance between a mean of updated text features obtained based on the first text set and belonging to each category to be updated and features of training samples belonging to the category to be updated;
[0159] A second construction unit configured to construct a second loss function by minimizing the distance between the original text features obtained based on the second text set and the updated text features;
[0160] a third construction unit configured to construct a third loss function by maximizing the distance between a mean of original text features obtained based on the third text set and belonging to each non-updated category and a mean of updated text features obtained based on the first text set and belonging to each to-be-updated category; and
[0161] an updating unit configured to update the two-way multimodal model by minimizing a total loss function consisting of a weighted sum of the first loss function, the second loss function, and the third loss function,
[0162] Among them, the original text features are features obtained by the original two-way multimodal model, and the updated text features are features obtained by the updated two-way multimodal model.
[0163] Note 15. An apparatus according to Note 14, wherein, in the case of updating for a specific category, the first loss function minimizes the sum or mean of the distances between the mean of the updated text features belonging to the specific category obtained based on the first text set and the features of each training sample calculated for each batch belonging to the specific category.
[0164] Note 16. An apparatus according to Note 14, wherein, when updating for a specific category, the first loss function minimizes the distance between the mean of updated text features belonging to the specific category obtained based on the first text set and the mean of features of training samples belonging to the specific category.
[0165] Note 17. An apparatus according to Note 14, wherein, in the case of updating for more than two categories, a similarity matrix between updated text features obtained based on the first text set and belonging to each category to be updated and the mean of training sample features belonging to each category to be updated is obtained by calculation, and wherein the first loss function makes the diagonal elements of the similarity matrix maximized and the remaining elements minimized.
[0166] Note 18. The device according to any one of Notes 14 to 17, wherein:
[0167] The second loss function is a cosine prior preservation loss function, which is constructed by minimizing the sum of cosine distances between the original text features and the updated text features belonging to each category to be updated obtained based on the second text set; or
[0168] The second loss function is a two-norm prior-preserving loss function, which is constructed by minimizing the sum of the Euclidean distances between the normalized unit vectors of the original text features belonging to each category to be updated and the normalized unit vectors of the updated text features obtained based on the second text set.
[0169] Note 19. An apparatus according to any one of Notes 14 to 17, wherein the third loss function is a deviation loss function, and the deviation loss function is constructed by maximizing the sum of the cosine distances between the means of the original text features belonging to each non-updated category obtained based on the third text set and the means of the updated text features belonging to each category to be updated obtained based on the first text set.
[0170] Note 20. A computer-readable storage medium having a program stored thereon, the program causing a computer to perform a method according to any one of Notes 1 to 13 when executed by a processor.
[0171] Finally, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or device. In addition, in the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or device that includes the elements.
[0172] Although the embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, it should be understood that the embodiments described above are only configured to illustrate the present invention and do not constitute a limitation of the present invention. For those skilled in the art, various modifications and changes can be made to the above embodiments without departing from the essence and scope of the present invention. Therefore, the scope of the present invention is limited only by the attached claims and their equivalent meanings.
Claims
1. A method for updating a dual-path multimodal model, comprising: For one or more to-be-updated categories, obtaining a first text set including a unique identifier and a name of each to-be-updated category, a second text set including only the name of each to-be-updated category, and a third text set including the name of each non-updated category in one or more non-updated categories; Constructing a first loss function by minimizing the distance between the mean of the updated text features obtained based on the first text set and belonging to each category to be updated and the features of the training samples belonging to the category to be updated; Constructing a second loss function by minimizing the distance between the original text features obtained based on the second text set and the updated text features; constructing a third loss function by maximizing the distance between the mean of the original text features obtained based on the third text set and belonging to each non-updated category and the mean of the updated text features obtained based on the first text set and belonging to each to-be-updated category; and updating the two-way multimodal model by minimizing a total loss function consisting of a weighted sum of the first loss function, the second loss function, and the third loss function, The original text features are features obtained through the original two-way multimodal model, and the updated text features are features obtained through the updated two-way multimodal model.
2. The method according to claim 1, wherein: When updating for a specific category, the first loss function minimizes the sum or mean of the distances between the mean of the updated text features belonging to the specific category obtained based on the first text set and the features of each training sample calculated for each batch belonging to the specific category.
3. The method according to claim 1, wherein: When updating for a specific category, the first loss function minimizes the distance between the mean of updated text features obtained based on the first text set and belonging to the specific category and the mean of features of training samples belonging to the specific category.
4. The method according to claim 1, wherein: In the case of updating for more than two categories, a similarity matrix between the updated text features belonging to each category to be updated and the means of the training sample features belonging to each category to be updated obtained based on the first text set is obtained by calculation, and wherein the first loss function makes the diagonal elements of the similarity matrix maximized and the remaining elements minimized.
5. The method according to any one of claims 1 to 4, wherein: The second loss function is a cosine prior-preserving loss function, and wherein the cosine prior-preserving loss function is constructed by minimizing the sum of cosine distances between original text features and updated text features belonging to each category to be updated obtained based on the second text set.
6. The method according to any one of claims 1 to 4, wherein: The second loss function is a two-norm prior-preserving loss function, and wherein the two-norm prior-preserving loss function is constructed by minimizing the sum of the Euclidean distances between the normalized unit vectors of the original text features belonging to each category to be updated obtained based on the second text set and the normalized unit vectors of the updated text features.
7. The method according to any one of claims 1 to 4, wherein: The third loss function is a deviation loss function, and wherein the deviation loss function is constructed by maximizing the sum of the cosine distances between the means of the original text features belonging to each non-updated category obtained based on the third text set and the means of the updated text features belonging to each to-be-updated category obtained based on the first text set.
8. The method according to any one of claims 1 to 4, wherein: The unique identifier is an identifier whose meaning cannot be recognized by the two-way multimodal model.
9. A device for updating a dual-path multimodal model, comprising: an obtaining unit configured to obtain, for one or more to-be-updated categories, a first text set including a unique identifier and a name of each to-be-updated category, a second text set including only the name of each to-be-updated category, and a third text set including the name of each non-updated category in one or more non-updated categories; A first construction unit configured to construct a first loss function by minimizing the distance between a mean of updated text features obtained based on the first text set and belonging to each category to be updated and features of training samples belonging to the category to be updated; A second construction unit is configured to construct a second loss function by minimizing the distance between the original text features obtained based on the second text set and the updated text features; a third construction unit configured to construct a third loss function by maximizing the distance between a mean of original text features obtained based on the third text set and belonging to each non-updated category and a mean of updated text features obtained based on the first text set and belonging to each to-be-updated category; and an updating unit configured to update the two-way multimodal model by minimizing a total loss function consisting of a weighted sum of the first loss function, the second loss function, and the third loss function, The original text features are features obtained through the original two-way multimodal model, and the updated text features are features obtained through the updated two-way multimodal model.
10. A computer-readable storage medium having stored thereon a program which, when executed by a processor, causes a computer to perform the method according to any one of claims 1 to 8.
