A partial label oriented multi-modal learning method and system, and a storage medium

Abstract

The application discloses a kind of partial label-oriented multimodal learning method, system and storage medium, comprising the following steps: S1: processing including image and corresponding text description training dataset, obtain training dataset with partial label;S2: according to the training dataset after processing, construct partial label-oriented multimodal learning model for cross-modal retrieval;S3: input the data modality that needs to retrieve to multimodal learning model, and calculate its similarity with the data in retrieval library;S4: according to the similarity of data similarity calculation Similarity ranking, output cross-modal retrieval result;The application solves the label ambiguity and modal difference problem faced by cross-modal retrieval with partial label, improves the accuracy of cross-modal retrieval.

Description

Technical Field

[0001] This invention relates to the field of multimodal learning, and more specifically to a multimodal learning method, system, and storage medium for partial labeling. Background Technology

[0002] Multimodal learning is a method of learning using data from different senses or interaction methods, such as text, images, audio, and video. By fusing multiple data modalities to train models, multimodal learning can improve the model's perception and understanding capabilities, enabling cross-modal information interaction and fusion. Cross-modal retrieval is a classic multimodal learning task, capable of finding relevant data from different modalities. In partial-label cross-modal retrieval tasks, each instance is associated with a set of candidate labels, where only one is the true label, and the others are erroneous labels, i.e., label ambiguity. This labeling method can significantly reduce labeling costs, especially when dealing with large-scale data.

[0003] In recent years, researchers have proposed numerous cross-modal retrieval methods, which can generally be categorized into three types based on their data annotation formats: supervised cross-modal retrieval methods, unsupervised cross-modal retrieval methods, and semi-supervised cross-modal retrieval methods. Specifically: 1) Supervised cross-modal retrieval methods utilize labeled data to learn common representations of different modalities. While they can theoretically provide accurate retrieval results, these methods typically require a large amount of unambiguous labeled data, resulting in high annotation costs and an inability to handle scenarios with partially labeled data. 2) Unsupervised cross-modal retrieval methods learn modality-specific features by maximizing the correlation between different modalities. Although they can operate without relying on label information, they often suffer from performance issues due to a lack of supervised information. 3) Semi-supervised cross-modal retrieval methods, to some extent, balance the pain points of high annotation costs and insufficient supervised information. While these methods reduce the reliance on a large number of clean labels, they still require a certain amount of clean label information, cannot handle the label ambiguity issues faced by cross-modal retrieval with partially labeled data, and require further reduction of modality differences.

[0004] In summary, cross-modal retrieval based on partial tags faces the problems of tag ambiguity and modal differences. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention proposes a multimodal learning method, system, and storage medium for partial labeling, which solves the problems of label ambiguity and modal differences in cross-modal retrieval with partial labels, thereby improving the accuracy of cross-modal retrieval.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] This solution provides a multimodal learning method, system, and storage medium for partial labeling, including the following steps:

[0008] S1: Process the training dataset, which includes images and corresponding text descriptions, to obtain a training dataset with partial labels;

[0009] S2: Construct a multimodal learning model oriented towards partial labels based on the processed training dataset for cross-modal retrieval;

[0010] S3: Input the data modality to be retrieved into the multimodal learning model and calculate its similarity with the data in the retrieval database;

[0011] S4: Sort the data based on the calculated similarity and output the cross-modal search results.

[0012] The beneficial effects of this invention are that it learns discriminative representations from ambiguous partial labels, while eliminating differences between modalities, thus solving the problems of label ambiguity and modal differences in cross-modal retrieval with partial labels and improving the accuracy of cross-modal retrieval.

[0013] Further, step S1 includes the following sub-steps:

[0014] Furthermore, the input space is Tag space is represented as Where K represents the number of categories in the dataset. The expressions for the processed image training set and text description training set are:

[0015]

[0016] Where N represents the number of sample pairs in the training set. 和 表示 The i-th image sample and the set of candidate labels. 和 表示 The first text description sample and candidate label set.

[0017] The beneficial effect of the above further scheme is that the cross-modal retrieval training set can be processed to obtain a cross-modal retrieval dataset with partial labels, which can be used to study cross-modal retrieval methods oriented towards partial labels.

[0018] Furthermore, step S2 includes the following sub-steps:

[0019] S21: Use the encoder to extract the feature vectors of samples in the input model;

[0020] S22: Use the softmax function to obtain the probability distribution of a sample belonging to each label;

[0021] S23: Extract the disambiguated label of each sample during the training process using the obtained sample label probability distribution;

[0022] S24: By considering the loss on both the candidate label set and the non-candidate label set, the model is encouraged to reduce the prediction probability of non-candidate labels while disambiguating the candidate label set.

[0023] S25: Use the labels obtained by deambiguation during model training as pseudo-labels to perform instance-level cross-modal contrastive learning to eliminate modal differences;

[0024] S26: Further reduce cross-modal differences by maintaining the same number of category prototype vectors for image data and text data respectively, and then performing cross-modal alignment at the prototype level.

[0025] S27: By repeatedly executing the above process, the overall loss function guides the updating of model parameters on the training dataset.

[0026] Furthermore, the formula for extracting the feature vector in step S21 is as follows:

[0027] q = f(x)

[0028] Where q represents the extracted sample feature vector, x represents the sample input to the model, and f represents the feature encoder.

[0029] Furthermore, the formula for the probability distribution of the sample belonging to each label obtained in step S22 is as follows:

[0030] z = g(q)

[0031] Where q represents the extracted sample feature vector, g(·) represents the softmax function, and z represents the probability distribution of the obtained sample belonging to each label.

[0032] Furthermore, the formula for removing the demisting label in step S23 is as follows:

[0033]

[0034] Among them, z I and z T Representing image samples x respectively I and text description sample x T The probabilities of belonging to the corresponding categories in the image and text modalities, respectively. Represents the Hartmann product, y I and y TThese represent the candidate label sets for image and text modalities, respectively. and These represent the label confidence scores after deambiguation of the image and text modalities, respectively.

[0035] Furthermore, the disambiguation loss expression for the image and text modalities in step S24 is as follows:

[0036]

[0037] Where ω is a hyperparameter used to balance the disambiguation loss on the candidate label set and the non-candidate label set. f(x) represents the output of the model. p(y=c|x) represents the confidence level of the model in assigning the label that sample x belongs to class c. p(y=c|x, f(x)) represents the predicted probability after applying the softmax function.

[0038] Therefore, the overall disambiguation loss expression for the model is:

[0039]

[0040] in, and These represent the disambiguation losses for image and text modalities, respectively. This represents the total disambiguation loss used in the model.

[0041] The beneficial effect of the above-mentioned further scheme is that it utilizes information from both the candidate label set and the non-candidate label set, projects data from different modalities into a common space, effectively solves the label ambiguity problem, and reduces the differences between modalities.

[0042] Furthermore, the instance-level cross-modal contrastive learning in step S25 first requires defining pseudo-labels to serve as supervision information. The calculation expression for the pseudo-labels is:

[0043]

[0044] in, and These represent the lists of label confidence scores obtained from the image and text modalities, respectively. argmax(·) represents the operation of finding the label with the highest confidence score. and These represent the pseudo-labels for the acquired image and text modalities, respectively.

[0045] To maximize consistency among different modalities within a common space, this is achieved by maximizing the number of modalities with the same pseudo-label. The distance between instances, samples The probability of belonging to the j-th instance among m modes can be defined as:

[0046]

[0047] Where m represents the number of modalities, with a value of 2, l represents the modality selection, and l takes the values ​​1 and 2 to represent the image and text modalities, respectively. N represents the total number of sample pairs. This indicates the j-th instance belonging to m modes. Indicates sample The probability of belonging to the j-th instance among m modalities. This indicates that the similarity between the features of two samples is calculated, where τ is the temperature hyperparameter. This indicates the operation of selecting samples from all modalities l that have the same pseudo-label as the sample. exp indicates performing an exponential operation.

[0048] Therefore, the expression for the instance-level cross-modal contrastive learning loss is:

[0049]

[0050] Where m represents the number of modes, with a size of 2, and N represents the total number of sample pairs. Indicates sample The probability of belonging to the j-th instance among m modalities, where log represents the logarithmic operation.

[0051] The beneficial effect of the above-mentioned further scheme is that it performs instance-level cross-modal contrastive learning in the case of label ambiguity and significantly reduces the heterogeneity differences between different modalities.

[0052] Furthermore, the cross-modal alignment module in step S26 first needs to maintain a category prototype for each category in the image and text modalities, the expression of which is as follows:

[0053]

[0054] in, and V represents the class prototype vector of the Kth class in the image and text modalities, respectively. I and V T These represent the category prototype lists for image and text modalities, respectively.

[0055] The category prototype is updated as follows:

[0056]

[0057] λ(t)=1-t / E

[0058] in, and Let q represent the class prototype vectors of the Kth class in the image and text modalities, respectively.I and q T Let represent the feature vectors of the image and text modalities, respectively; y represents the current training epoch of the model; E is the total number of iterations; and λ(t) is the dynamic parameter that balances the current prototype vector and feature vectors. and Let represent the pseudo-labels for image and text modal samples, respectively, and k represent the k-th category.

[0059] The expression for the cross-modal alignment loss at the category prototype level for image and text modalities is:

[0060]

[0061] Where, q I and q T These represent the feature vectors of the image and text modalities, respectively. and These represent the prototype vectors of the k-th category for image and text modalities, respectively. and These represent the cross-modal alignment loss for image and text modalities, respectively.

[0062] The overall cross-modal alignment loss expression is:

[0063]

[0064] in This represents the total cross-modal alignment loss. and These represent the cross-modal alignment loss for image and text modalities, respectively.

[0065] The beneficial effect of the above-mentioned further scheme is that it performs cross-modal alignment at the category level, further reducing cross-modal differences.

[0066] Furthermore, the overall loss function expression for the model in step S27 is as follows:

[0067]

[0068] in, Let be the total loss function of the modes. To mitigate the loss of misinterpretation, For cross-modal contrastive learning loss, This represents the cross-modal alignment loss. α and β are hyperparameters.

[0069] The beneficial effect of the above-mentioned further scheme is that by guiding the optimization of the cross-modal retrieval model through disambiguation loss, cross-modal contrastive learning loss and cross-modal alignment loss, the model solves the problems of label ambiguity and modal differences, enabling the trained model to effectively solve the cross-modal retrieval problem in some label cases.

[0070] Further, step S3 involves inputting an image or text modality test sample for querying into the retrieval model trained in the previous step, and calculating the similarity between another modality data in the retrieval database and the query sample.

[0071] Furthermore, the method of step S4 is as follows: sort the data according to the calculated data similarity, and output the retrieved data with the highest similarity as the retrieval result.

[0072] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method described above.

[0073] A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the above-described method.

[0074] The beneficial effects of this invention are that it learns discriminative representations from ambiguous partial labels, while eliminating intermodal differences, thus solving the problems of label ambiguity and modal differences in cross-modal retrieval with partial labels and improving the accuracy of cross-modal retrieval. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. It should be understood that the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 A step diagram illustrating the multimodal learning method, system, and storage medium for partial labeling provided in this embodiment of the invention. Detailed Implementation

[0077] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0078] like Figure 1 As shown, this invention provides a multimodal learning method, system, and storage medium for partial labeling, the implementation of which is as follows:

[0079] S1: Process the training dataset, which includes images and corresponding text descriptions, to obtain a training dataset with partial labels;

[0080] S2: Construct a multimodal learning model oriented towards partial labels based on the processed training dataset for cross-modal retrieval;

[0081] S3: Input the data modality to be retrieved into the multimodal learning model and calculate its similarity with the data in the retrieval database;

[0082] S4: Sort the data based on the calculated similarity and output the cross-modal search results.

[0083] In this embodiment, step S2 includes the following sub-steps:

[0084] S21: Use the encoder to extract the feature vectors of samples in the input model;

[0085] S22: Use the softmax function to obtain the probability distribution of a sample belonging to each label;

[0086] S23: Extract the disambiguated label of each sample during the training process using the obtained sample label probability distribution;

[0087] S24: By considering the loss on both the candidate label set and the non-candidate label set, the model is encouraged to reduce the prediction probability of non-candidate labels while disambiguating the candidate label set.

[0088] S25: Use the labels obtained by deambiguation during model training as pseudo-labels to perform instance-level cross-modal contrastive learning to eliminate modal differences;

[0089] S26: Further reduce cross-modal differences by maintaining the same number of category prototype vectors for image data and text data respectively, and then performing cross-modal alignment at the prototype level;

[0090] S27: By repeatedly executing the above process, the overall loss function guides the updating of model parameters on the training data.

[0091] In this embodiment, the formula for extracting the feature vector in step S21 is as follows:

[0092] q = f(x)

[0093] Where q represents the extracted sample feature vector, x represents the sample input to the model, and f represents the feature encoder.

[0094] The formula for the probability distribution of a sample belonging to each label obtained in step S22 is as follows:

[0095] z = g(q)

[0096] Where q represents the extracted sample feature vector, g(·) represents the softmax function, and z represents the probability distribution of the obtained sample belonging to each label.

[0097] The formula for removing the deambiguity label in step S23 is as follows:

[0098]

[0099] Among them, z I and z T Representing image samples x respectively I and text description sample x T The probabilities of belonging to the corresponding categories in the image and text modalities, respectively. Represents the Hartmann product, y I and y T These represent the candidate label sets for image and text modalities, respectively. and These represent the label confidence scores after deambiguation of the image and text modalities, respectively.

[0100] The disambiguation loss expression for the image and text modalities in step S24 is:

[0101]

[0102] Where ω is a hyperparameter used to balance the disambiguation loss on the candidate label set and the non-candidate label set. f(x) represents the output of the model. p(y=c|x) represents the confidence level of the model in assigning the label that sample x belongs to class c. p(y=c|x, f(x)) represents the predicted probability after applying the softmax function.

[0103] The overall disambiguation loss expression for the model is:

[0104]

[0105] in, and These represent the disambiguation losses for image and text modalities, respectively. This represents the total disambiguation loss used in the model.

[0106] The instance-level cross-modal contrastive learning in step S25 first requires defining pseudo-labels to be used as supervision information. The pseudo-labels are obtained as follows:

[0107]

[0108] in, and These represent the lists of label confidence scores obtained from the image and text modalities, respectively. argmax(·) represents the operation of finding the label with the highest confidence score. and These represent the pseudo-labels for the acquired image and text modalities, respectively.

[0109] To maximize consistency among different modalities within a common space, this is achieved by maximizing the number of modalities with the same pseudo-label. The distance between instances, samples The probability of belonging to the j-th instance among m modes can be defined as follows:

[0110]

[0111] Where m represents the number of modalities, with a value of 2, l represents the modality selection, and l takes the values ​​1 and 2 to represent the image and text modalities, respectively. N represents the total number of sample pairs. This indicates the j-th instance belonging to m modes. Indicates sample The probability of belonging to the j-th instance among m modalities. This indicates that the similarity between the features of two samples is calculated, where τ is the temperature hyperparameter. This indicates the operation of selecting samples from all modalities l that have the same pseudo-label as the sample. exp indicates performing an exponential operation.

[0112] Therefore, the expression for instance-level cross-modal contrastive learning loss is as follows:

[0113]

[0114] Where m represents the number of modes, with a size of 2, and N represents the total number of sample pairs. Indicates sample The probability of belonging to the j-th instance among m modalities, where log represents the logarithmic operation.

[0115] The cross-modal alignment module in step S26 first needs to maintain a category prototype for each category in the image and text modalities, with the following expression:

[0116]

[0117] in, and V represents the class prototype vector of the k-th class in the image and text modalities, respectively. I and V T These represent the category prototype lists for image and text modalities, respectively.

[0118] The update method for category prototypes is as follows;

[0119]

[0120] in, and Let q represent the class prototype vectors of the k-th class in the image and text modalities, respectively. I and qT Let represent the feature vectors of the image and text modalities, respectively; t represent the current training epoch of the model; E is the total number of iterations; and λ(t) is the dynamic parameter that balances the current prototype vector and feature vectors. and Let represent the pseudo-labels for image and text modal samples, respectively, and k represent the k-th category.

[0121] Therefore, the cross-modal alignment loss expression at the category prototype level for image and text modalities is:

[0122]

[0123] Where, q I and q T These represent the feature vectors of the image and text modalities, respectively. and These represent the prototype vectors of the k-th category for image and text modalities, respectively. and These represent the cross-modal alignment loss for image and text modalities, respectively.

[0124] The overall cross-modal alignment loss expression is as follows:

[0125]

[0126] in This represents the total cross-modal alignment loss. and These represent the cross-modal alignment loss for image and text modalities, respectively.

[0127] The overall loss function expression for the model in step S27 is:

[0128]

[0129] in, Let be the total loss function of the modes. To mitigate the loss of misinterpretation, For cross-modal contrastive learning loss, This represents the cross-modal alignment loss. α and β are hyperparameters.

[0130] In this embodiment, the method of step S3 is as follows: input the image or text modality test sample for query into the retrieval model trained in the previous step, and calculate the similarity between the other modality data in the retrieval library and the query sample.

[0131] In this embodiment, the method of step S4 is as follows: sort the data according to the calculated data similarity, and output the retrieved data with the highest similarity as the retrieval result.

[0132] This embodiment provides a computer device including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the above-described multimodal learning method for partial labeling.

[0133] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0134] The memory includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D-interface display memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the memory may include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code of the partial label-oriented multimodal learning method. In addition, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0135] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of the computer device. In this embodiment, the processor is used to run program code stored in the memory or process data, for example, to run the program code for the partial label-oriented multimodal learning method.

[0136] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it causes the processor to perform the steps of the above-described multimodal learning method for partial labeling.

[0137] The computer-readable storage medium stores an interface display program that can be executed by at least one processor to cause the at least one processor to perform the steps of the partial label-oriented multimodal learning method described above.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the multimodal learning method for partial labeling described in the embodiments of this application.

[0139] This invention provides a multimodal learning method, system, and storage medium for partial labeling, which solves the problems of label ambiguity and modal differences in cross-modal retrieval with partial labels, and improves the accuracy of cross-modal retrieval.

Claims

1. A multimodal learning method, system, and storage medium for partial labeling, characterized in that, The method includes: S1: Process the training dataset, which includes images and corresponding text descriptions, to obtain a training dataset with partial labels; S2: Construct a multimodal learning model oriented towards partial labels based on the processed training dataset for cross-modal retrieval; S3: Input the data modality to be retrieved into the multimodal learning model and calculate its similarity with the data in the retrieval database; S4: Sort the data based on the calculated similarity and output the cross-modal search results; Step S2 specifically includes: S21: Use the encoder to extract the feature vectors of samples in the input model; S22: Use the softmax function to obtain the probability distribution of a sample belonging to each label; S23: Extract the disambiguated label of each sample during the training process using the obtained sample label probability distribution; S24: By considering the loss on both the candidate label set and the non-candidate label set, the model is encouraged to reduce the prediction probability of non-candidate labels while disambiguating the candidate label set. S25: Use the labels obtained by deambiguation during model training as pseudo-labels to perform instance-level cross-modal contrastive learning to eliminate modal differences; S26: Further reduce cross-modal differences by maintaining the same number of category prototype vectors for image data and text data respectively, and then performing cross-modal alignment at the prototype level; S27: By repeatedly executing the above process, the overall loss function guides the updating of model parameters on the training dataset; The disambiguation loss expression for the image and text modalities in step S24 is as follows: Where ω is a hyperparameter used to balance the disambiguation loss on the candidate label set and the non-candidate label set, f(x) represents the model output, p(y = c|x) represents the model's confidence in labeling sample x as belonging to class c, and p(y = c|x, f(x)) represents the predicted probability after applying the softmax function. Therefore, the overall disambiguation loss expression for the model is: in, and These represent the disambiguation losses for image and text modalities, respectively. This represents the total disambiguation loss used by the model; S3: Input the data modality to be retrieved into the multimodal learning model and calculate its similarity with the data in the retrieval database; S4: Sort the data based on the calculated similarity and output the cross-modal search results.

2. The multimodal learning method, system, and storage medium for partial labeling as described in claim 1, characterized in that, The instance-level cross-modal contrastive learning in step S25 first requires defining pseudo-labels to serve as supervision information. The calculation expression for the pseudo-labels is as follows: in, and These represent the lists of label confidence scores obtained from the image and text modalities, respectively. `argmax()` represents the operation to find the label with the highest confidence score. and Let represent the pseudo-labels for the acquired image and text modalities, respectively. The expression for the instance-level cross-modal contrastive learning loss is: Where m represents the number of modalities, with a size of 2, N represents the total number of sample pairs, and log represents the logarithm operation. Indicates sample The probability of belonging to the j-th instance among m modalities is calculated as follows: Where m represents the number of modalities, with a size of 2; l represents the modality selection, where l = 1 and l = 2 to represent the image and text modalities, respectively; and N represents the total number of sample pairs. This indicates the j-th instance belonging to m modes. Indicates sample The probability of belonging to the j-th instance among m modalities. This indicates that the similarity between the features of two samples is calculated, where τ is the temperature hyperparameter. This indicates the operation of selecting samples from all modalities l that have the same pseudo-label as the sample, and exp indicates performing an exponential operation.

3. The multimodal learning method, system, and storage medium for partial labeling as described in claim 1, characterized in that, The cross-modal alignment module in step S26 first needs to maintain a category prototype for each category in the image and text modalities, with the following expression: in, and V represents the class prototype vector of the Kth class in the image and text modalities, respectively. I and V T These represent the category prototype lists for image and text modalities, respectively. The category prototype is updated as follows: λ(t)=1-t / E in, and Let q represent the class prototype vectors of the Kth class in the image and text modalities, respectively. I and q T Let represent the feature vectors of the image and text modalities, respectively; t represent the current training epoch of the model; E is the total number of iterations; and λ(t) is the dynamic parameter that balances the current prototype vector and feature vectors. and Let represent the pseudo-labels of the image and text modal samples, respectively, and k represent the k-th class. The cross-modal alignment loss at the class prototype level for the image and text modalities is: Where, q I and q T These represent the feature vectors of the image and text modalities, respectively. and These represent the prototype vectors of the k-th category for the image and text modalities, respectively. and Let represent the cross-modal alignment loss for image and text modalities respectively. The overall cross-modal alignment loss is expressed as: in This represents the total cross-modal alignment loss. and These represent the cross-modal alignment loss for image and text modalities, respectively.

4. A computer device, characterized in that: It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that: The system contains a computer program that, when executed by a processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 3.

Images