Multimodal model training methods, devices and electronic equipment

Abstract

This application provides a multimodal model training method, apparatus, and electronic device. The method includes: acquiring category text information of N categories in a training image set; acquiring N first text feature vectors obtained by a text encoder in a multimodal model based on the category text information of the N categories; determining the polar angle and auxiliary angle of the N first text feature vectors in a spherical coordinate system based on the N first text feature vectors; increasing the polar angle of the N first text feature vectors based on the maximum value among the polar angles; determining N second text feature vectors in a Cartesian coordinate system based on the N first text feature vectors, the auxiliary angle, and the adjusted polar angle; training the image encoder in the multimodal model based on the N second text feature vectors and the training image set to obtain a trained multimodal model, which is then used for image classification. This improves the performance of the multimodal model after training.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a multimodal model training method, apparatus and electronic device. Background Technology

[0002] In related technologies, in multimodal learning, specific anchor feature data (e.g., text anchors (such as text features of category text information)) can be used to enhance the robustness and generalization ability of the model. These anchor feature data usually contain representative multimodal information, which can help the model better understand and handle the complex relationships between different modalities, thereby improving the reliability of the model in practical applications.

[0003] However, in related technologies, the distance between different text anchors is small, that is, the similarity is high, and they are relatively concentrated in the feature space and are not easy to distinguish. This can easily lead to poor performance of the trained multimodal model. Summary of the Invention

[0004] This application provides a multimodal model training method, apparatus, and electronic device to address the problem of poor performance of existing multimodal models.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a multimodal model training method, the method comprising:

[0007] Obtain the category text information for N categories in the training image set;

[0008] Obtain N first text feature vectors from the text encoder in the multimodal model based on the category text information of the N categories. The first text feature vectors are vectors in the Cartesian coordinate system, and N is an integer greater than 1.

[0009] Based on the N first text feature vectors, determine the polar angle and auxiliary angle of the N first text feature vectors in the spherical coordinate system;

[0010] Based on the maximum value among the polar angles of the N first text feature vectors, the polar angles of the N first text feature vectors are increased to obtain the adjusted polar angles of the N first text feature vectors;

[0011] Based on the N first text feature vectors, the auxiliary angle, and the adjusted polar angle, determine the N second text feature vectors in the Cartesian coordinate system;

[0012] Based on the N second text feature vectors and the training image set, the image encoder in the multimodal model is trained to obtain the trained multimodal model, which is used for image classification.

[0013] Secondly, embodiments of this application provide a multimodal model training device, the device comprising:

[0014] The first acquisition module is used to acquire the category text information of N categories in the training image set;

[0015] The second acquisition module is used to acquire N first text feature vectors obtained by the text encoder in the multimodal model based on the category text information of the N categories. The first text feature vectors are vectors in the Cartesian coordinate system, and N is an integer greater than 1.

[0016] The first determining module is used to determine the polar angle and auxiliary angle of the N first text feature vectors in the spherical coordinate system based on the N first text feature vectors.

[0017] The first adjustment module is used to increase the polar angle of the N first text feature vectors according to the maximum value among the polar angles of the N first text feature vectors, so as to obtain the adjusted polar angles of the N first text feature vectors.

[0018] The second determining module is used to determine N second text feature vectors in the Cartesian coordinate system based on the N first text feature vectors, the auxiliary angle, and the adjusted polar angle.

[0019] The training module is used to train the image encoder in the multimodal model based on the N second text feature vectors and the training image set to obtain the trained multimodal model, which is used for image classification.

[0020] Thirdly, embodiments of this application provide an electronic device, including a transceiver and a processor.

[0021] The processor is used for:

[0022] Obtain the category text information for N categories in the training image set;

[0023] Obtain N first text feature vectors from the text encoder in the multimodal model based on the category text information of the N categories. The first text feature vectors are vectors in the Cartesian coordinate system, and N is an integer greater than 1.

[0024] Based on the N first text feature vectors, determine the polar angle and auxiliary angle of the N first text feature vectors in the spherical coordinate system;

[0025] Based on the maximum value among the polar angles of the N first text feature vectors, the polar angles of the N first text feature vectors are increased to obtain the adjusted polar angles of the N first text feature vectors;

[0026] Based on the N first text feature vectors, the auxiliary angle, and the adjusted polar angle, determine the N second text feature vectors in the Cartesian coordinate system;

[0027] Based on the N second text feature vectors and the training image set, the image encoder in the multimodal model is trained to obtain the trained multimodal model, which is used for image classification.

[0028] Fourthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the multimodal model training method described in the first aspect.

[0029] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multimodal model training method described in the first aspect.

[0030] In a sixth aspect, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in the first aspect above.

[0031] In this embodiment, the polar angles of the N first text feature vectors can be increased by maximizing the polar angles among them. This allows the determination of N second text feature vectors in Cartesian coordinates using the N first text feature vectors, auxiliary angles, and the increased polar angles. This increases the distance between the feature vectors in the resulting N second text feature vectors, meaning that the similarity between the feature vectors in the N second text feature vectors is reduced compared to the similarity between the feature vectors in the N first text feature vectors, making it easier to distinguish between different categories. Subsequently, based on the N second text feature vectors and the training image set, the image encoder in the multimodal model is trained to obtain the trained multimodal model, thereby improving the image classification performance of the trained multimodal model. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart of a multimodal model training method provided in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram of a multimodal model training method provided in an embodiment of this application;

[0035] Figure 3 This is a schematic diagram of the structure of a multimodal model training device provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] See Figure 1 , Figure 1 This is a flowchart illustrating a multimodal model training method provided in an embodiment of this application. It can be applied to electronic devices, which may be terminals or servers, etc., without specific limitations. Figure 1 As shown, the multimodal model training method provided in this embodiment includes the following steps:

[0039] Step 101: Obtain the category text information of N categories in the training image set.

[0040] The training image set may include multiple images, each with its corresponding category. The number of categories in the training image set is N. Category text information can also be understood as category prompt text or prompt words, which may include the category name. Since the category is the image's class and is related to the image, the category text information can also be associated with the image. For example, as an example, category text information in a fixed format including the category name can be obtained. For example, the fixed format could be "an image with a category name." For instance, for the lion category, the category text information could be an image of a lion. There are various ways to obtain category text information, and this application embodiment does not limit this. For example, as an example, a prompt generator can generate corresponding category text information for each category for use in the subsequent text encoding process.

[0041] Step 102: Obtain N first text feature vectors obtained by the text encoder in the multimodal model based on the category text information of N categories. The first text feature vectors are vectors in the Cartesian coordinate system, and N is an integer greater than 1.

[0042] Multimodal models can include text encoders and image encoders. Text encoders process text information, while image encoders process images. Multimodal models can handle information from different modalities. As an example, a pre-trained text encoder can be used. After obtaining the category text information for each category, this information can be input into the text encoder. The text encoder performs feature encoding (feature extraction) to determine the first text feature vector corresponding to each category's text information, resulting in N first text feature vectors in Cartesian coordinates. As an example, the first text feature vector is the normalized version of the text feature vector output by the text encoder. The geometric length of the first text feature vector is 1. Various normalization methods are available and are not limited; for example, L2 normalization can be used.

[0043] Step 103: Based on the N first text feature vectors, determine the polar angle and auxiliary angle of the N first text feature vectors in the spherical coordinate system.

[0044] The spherical coordinate system, also known as the spherical polar coordinate system, represents the first text feature vector, which is a vector in the Cartesian coordinate system. This first text feature vector can be converted to a representation in the spherical coordinate system, which uses a radius (…). The first text feature vector is represented by radial distance and angular coordinates (which may include polar angles and auxiliary angles), i.e., using r and angle parameters. Therefore, based on the first text feature vector, the angular coordinates of the first text feature vector in the spherical coordinate system can be determined. It should be noted that the number of auxiliary angles can be one or more, and the number is related to the dimension of the first text feature vector. n(Greater than 2) The number of auxiliary angles is n-2. It should be noted that when n is three, the auxiliary angle is the azimuth angle. When n is greater than 3, the auxiliary angle is not the azimuth angle, but an angle used to control rotation. It can also be understood as a rotation parameter, that is, an auxiliary angle for layered rotation, used to decompose the higher-dimensional space into two-dimensional subspaces.

[0045] For example, the first text feature vector of a category is ( , ,..., ), which are Cartesian coordinates, represented in a spherical coordinate system as follows:

[0046] ;

[0047] ;

[0048] ;

[0049] The polar angle is n-2, and the auxiliary angles include n-2. , , If n is 3, then This represents the azimuth angle. If n is greater than 3, For use in control - A plane (i.e., a plane defined by the coordinate axes in an n-dimensional Cartesian coordinate system) and The auxiliary angle of rotation of the formed plane (i.e., the two-dimensional subspace). Used for control - A plane (i.e., a plane defined by the coordinate axes in an n-dimensional Cartesian coordinate system) and The auxiliary angle of rotation of the plane formed. To control - A plane (i.e., a plane defined by the coordinate axes in an n-dimensional Cartesian coordinate system) and The auxiliary angle of rotation of the plane formed. Taking three dimensions as an example, the first text feature vector is a three-dimensional feature vector, which is a three-dimensional Cartesian coordinate system. The corresponding spherical coordinate system is a three-dimensional spherical coordinate system, and the polar angle is the angle relative to the z-axis. Through the above, the first text feature vector ( , ,..., Given that r is the first text feature vector ( , ,..., The length of the angle r is known, so the corresponding polar angle and auxiliary angle can be calculated.

[0050] Step 104: Based on the maximum value among the polar angles of the N first text feature vectors, increase the polar angles of the N first text feature vectors to obtain the adjusted polar angles of the N first text feature vectors.

[0051] Step 105: Determine N second text feature vectors in the Cartesian coordinate system based on the N first text feature vectors, auxiliary angles, and adjusted polar angles.

[0052] Step 106: Based on N second text feature vectors and the training image set, train the image encoder in the multimodal model to obtain the trained multimodal model, which is then used for image classification.

[0053] Since each category's first text feature vector has a corresponding polar angle, and there are N categories, there are N polar angles. The largest polar angle among these N polar angles can be used to increase the value of the remaining N polar angles, resulting in N adjusted polar angles. It should be noted that in this embodiment, the auxiliary angles may not be adjusted. A larger distance between feature vectors indicates lower similarity, and vice versa. Regarding the feature representation of category text information, if the distance between the first text feature vectors of different categories is small, it becomes difficult to distinguish between different categories, potentially leading to weak performance of the trained multimodal model. Therefore, in this embodiment, the polar angle of the N first text feature vectors can be increased by maximizing the polar angle of the N first text feature vectors. In this way, the N first text feature vectors, auxiliary angles, and increased polar angles can be used to determine the N second text feature vectors in the Cartesian coordinate system. This increases the distance between the feature vectors in the N second text feature vectors, that is, the similarity between the feature vectors in the N second text feature vectors is reduced compared to the similarity between the feature vectors in the N first text feature vectors, making it easier to distinguish between different categories. Subsequently, based on the N second text feature vectors and the training image set, the image encoder in the multimodal model is trained to obtain the trained multimodal model, thereby improving the image classification performance of the trained multimodal model.

[0054] In some embodiments, before determining the polar angle and auxiliary angle of the N first text feature vectors in the spherical coordinate system based on the N first text feature vectors, the method further includes:

[0055] Determine the average vector of the N first text feature vectors;

[0056] Determine the rotation matrix based on the average vector and the preset target vector;

[0057] Based on the rotation matrix, rotate the N first text feature vectors to update the N first text feature vectors.

[0058] It should be noted that after updating the first text feature vector, all subsequent uses of the first text feature vector will use the updated first text feature vector. It should be understood that the result of rotating the average vector using a rotation matrix is ​​the preset target vector. The dimension of the preset target vector is the same as the dimension of the first text feature vector. The preset target vector can also be understood as the coordinate vector of the pole in the Cartesian coordinate system, or as the direction of the pole. The rotation matrix is ​​calculated to rotate the average vector to the preset target vector. There are various algorithms for calculating the rotation matrix, and no limitation is made. For example, the Aguilera and Pérez-Aguila algorithms can be used to calculate the rotation matrix. For example, the average vector is... v The preset target vector is p ,pass Rv = p Determine the rotation matrix in this way R。

[0059] By rotating the N first text feature vectors using a rotation matrix, the N first text feature vectors are updated, thus concentrating them around the poles. Subsequently, the polar angles and auxiliary angles of the updated N first text feature vectors are determined, and the polar angles are increased based on the maximum value among the N polar angles, resulting in N adjusted polar angles. This expands the concentration area from the original region to the entire hemisphere. Next, based on the N first text feature vectors, the auxiliary angles, and the adjusted polar angles, N second text feature vectors are determined in the Cartesian coordinate system, dispersing them within the Cartesian coordinate system. Finally, the image encoder in the multimodal model is trained using the training image set containing the N second text feature vectors in the Cartesian coordinate system to improve the performance of the trained model.

[0060] As an example, a preset target vector is used. p =[1,0,…,0] T The first element is 1, and the rest are 0.

[0061] In some embodiments, before training the image encoder in the multimodal model based on N second text feature vectors and a training image set to obtain the trained multimodal model, the method further includes:

[0062] Rotate the N second text feature vectors according to the transpose of the rotation matrix to update the N second text feature vectors.

[0063] It should be noted that after updating the second text feature vector, all subsequent uses of the second text feature vector will use the updated second text feature vector. Since rotating the N second text feature vectors according to the transpose of the rotation matrix can be understood as follows: the N first text feature vectors were previously rotated using the rotation matrix; to improve accuracy, the N second text feature vectors are rotated inversely, that is, the N second text feature vectors are rotated using the transpose of the rotation matrix, and then the N second text feature vectors are updated.

[0064] In some embodiments, the adjusted polar angle of the first text feature vector is M times the polar angle of the first text feature vector, where M is... The result of dividing by the maximum value.

[0065] It should be understood that by adjusting the polar angle to M times the original polar angle, where M is greater than 1, the polar angle is enlarged, thereby increasing the distance between subsequent second text feature vectors. This facilitates category differentiation, and training using the second feature vectors improves the performance of the trained model. Meanwhile, the maximum value among the N polar angles is less than... Making M greater than 1 allows the points corresponding to the second text feature vector on the sphere to be extended to the entire hemisphere, while ensuring that the extension does not exceed [a certain value]. This avoids points approaching from the other hemisphere, ensuring consistency in feature semantics.

[0066] In some embodiments, the training image set includes a first image set and an adversarial image set of the first image set, wherein the adversarial image set is an image obtained by perturbing the images in the first image set; based on N second text feature vectors and the training image set, the image encoder in the multimodal model is trained to obtain the trained multimodal model, including:

[0067] The first image set and the adversarial image set are input into the image encoder for feature extraction to obtain the first image feature vector of the first image set and the second image feature vector of the adversarial image in the adversarial image set.

[0068] The target loss is determined based on the first image feature vector, the second image feature vector, and N second text feature vectors.

[0069] Based on the target loss, the image encoder is trained to obtain the trained multimodal model.

[0070] By perturbing the original first image set, an adversarial image set can be obtained. Using the first image set and its adversarial image set, the image encoder can be trained adversarially, which can improve the image encoder's ability to cope with adversarial attacks and improve its performance.

[0071] In some embodiments, the target loss is determined based on a first image feature vector, a second image feature vector, and N second text feature vectors, including:

[0072] Determine the first similarity between the second image feature vector of each adversarial image in the adversarial image set and N second text feature vectors;

[0073] Based on the first similarity, determine the first loss;

[0074] Determine the second similarity between the first image feature vector and the second image feature vector;

[0075] Based on the second similarity, determine the second loss;

[0076] The target loss is determined based on the first loss and the second loss.

[0077] The first loss represents the difference between the second image feature vector of the adversarial image and the second text feature vector of the category to which the adversarial image belongs. The first loss is positively correlated with the sum of the first similarities between the second image feature vector of the adversarial image and N second text feature vectors, and negatively correlated with the first similarity between the second feature vector of the adversarial image and the second text feature vector of the category to which the adversarial image belongs. The second loss represents the difference between the first image feature vector and the second image feature vector, and the second loss is negatively correlated with the second similarity. As an example, the target loss can be a weighted sum of the first loss and the second loss.

[0078] In this embodiment, the second loss can constrain the similarity between the first image feature vector and the second image feature vector to mitigate the impact of adversarial perturbation and improve the ability to cope with adversarial perturbation attacks. Based on the second loss, the model is also trained by the first loss to ensure classification accuracy and robustness under adversarial attacks.

[0079] There are various forms of the first and second losses, and no limitation is made. For example, the first loss can be the aligned cross-entropy loss (A-CE), and the second loss can be the smoothing loss.

[0080] In some embodiments, after training the image encoder in the multimodal model based on N second text feature vectors and a training image set to obtain the trained multimodal model, the method further includes:

[0081] Obtain test images for image classification scenarios;

[0082] The test image is input into the image encoder of the trained multimodal model to extract features and obtain the image feature vector of the test image.

[0083] Determine the third similarity between the image feature vector of the test image and the first text feature vectors of N categories to obtain N third similarities;

[0084] Based on N third similarity scores, determine the target category of the test image from N categories.

[0085] The training image set above is also the training image for the image classification scenario. After training, test images (also called images to be classified, images to be tested, or test images) for the actual application scenario, i.e., the image classification scenario, can be obtained. The trained image encoder is used to extract features from the test images to obtain image feature vectors (also called test features) to improve the accuracy of the image feature vectors. Then, the third similarity between the image feature vector of the test image and the first text feature vectors of N categories is calculated, resulting in N third similarities. Using these N third similarities, the target category of the test image is determined from the N categories, improving image classification accuracy. There are multiple ways to determine the target category using the N third similarities; no limitation is imposed. As an example, the category corresponding to the largest third similarity among the N third similarities can be used as the target category of the test image.

[0086] The method of this application will be described in detail below with reference to some specific embodiments.

[0087] like Figure 2 As shown, the images in the first image set of the training image set can also be called benign images, and the first image feature vector can also be called benign features. The images in the adversarial image set of the training image set can also be called adversarial images, and the second image features can also be called adversarial features. The test image can also be called the test image, the category text information can also be called the prompt word, and the image feature vector of the test image can also be called the test feature. The first loss is... The second loss is The first text feature can be called the anchor point or text anchor point or anchor point feature, and the second text feature vector can be called the extended anchor point or extended anchor point feature. It is the result of extending the anchor point through an extension algorithm. The extension algorithm corresponds to the process of determining the polar angle and auxiliary angle, increasing the polar angle, and using the auxiliary angle and the increased polar angle to determine the second text feature vector.

[0088] The process of the method in this embodiment is as follows:

[0089] 1) Generation of prompts (category text information / prompt sentences) for known and new categories

[0090] The system lists known and new categories. For example, known categories might include lion, tiger, bear, and wolf, while new categories might include fox and leopard. Then, a prompt generator produces a fixed prompt sentence for each category, such as "A photo of a lion." These prompt sentences will be used in the subsequent text encoding process.

[0091] For example, suppose there are N categories in the training set. Each category name will be inserted into a fixed-format prompt sentence, such as "A photo of Lion". "A photo of {}" is a placeholder, and {} will be replaced with the category name. The generated sentences will be encoded by a pre-trained text encoder (e.g., CLIP). The text encoder converts these natural language sentences into feature embeddings (i.e., anchors). These embeddings are high-dimensional vectors that represent the semantic information of the sentence. In this process, each generated anchor... They are all normalized vectors, meaning their L2 norm is 1. This means that each anchor vector has a length of 1. To ensure that the anchors can correctly represent the semantic relationships between categories, an architecture with semantic consistency was chosen. The CLIP text encoder, which is pre-trained on a large scale, is used and is able to capture and represent the semantic information of the text well.

[0092] 2) Text Encoder and Extended Algorithm

[0093] The generated prompt sentences are encoded using a pre-trained text encoder (such as CLIP) and transformed into feature embeddings (anchors). Since directly using CLIP anchors results in high cosine similarity, an extension algorithm is required. This extension algorithm maps anchors to new locations to increase the distance between them while maintaining semantic consistency.

[0094] Directly using CLIP-generated anchors suffers from high cosine similarity (CoS), meaning that the angle between anchor vectors of different classes is small, and the distance between anchors is close. This leads to anchors being overly concentrated in the feature space, affecting the effectiveness of adversarial training. Due to this problem, CLIP anchors cannot be used directly and must be processed to reduce the cosine similarity between anchors and increase the distance between them. An expansion algorithm is used to increase the distance between anchors (reduce cosine similarity). The expansion algorithm can remap anchors to increase the distance between them while maintaining their semantic consistency. High cosine similarity means that the normalized anchors are scattered in a cluster on a unit hypersphere. Since the anchors to be remapped are also located on a unit hypersphere, the expansion algorithm should be designed in a spherical coordinate system. Assume a first text feature vector is ( , ,..., ), which are Cartesian coordinates, represented in a spherical coordinate system as follows:

[0095] ;

[0096] ;

[0097] ;

[0098] in It is a radial coordinate. ,..., These are angular coordinates. Here, It is the polar angle (taking a three-dimensional spherical coordinate system as an example, the polar angle is relative to...). (Angle of the axis). If the anchor points are concentrated around the pole, i.e. ,in Indicates anchor point If the polar angle is such that a natural extension method is to use the polar angle... Expand to · And keep other angular coordinates unchanged. The cluster can then be extended to the entire hemisphere. It should be noted that it cannot be extended beyond... This is because the anchor point might approach from another hemisphere, potentially compromising its semantic consistency. Therefore, given the original anchor point... The first step in the extension is to compute a rotation matrix. ,make ,in And calculate the anchor point after rotation. , Calculate the distance between two unit vectors. There are multiple methods, and no specific method is specified. As an example, the Aguilera and Pérez-Aguila algorithms can be used. Because... Gathering at the poles Next, expand the surrounding area using the method described above. The polar angle. The j-th element is represented as Anchor point and The maximum polar angle between them can be calculated as follows: The polar angle is adjusted using the maximum polar angle, and the adjusted polar angle is... · / Cartesian coordinates of extended anchor points for:

[0099] ;

[0100] ;

[0101] The first element is represented as , The j-th element is represented as , To determine the next element after rotating the anchor point (the first document feature vector) and updating the anchor point (the rotated anchor point, i.e., the updated first text feature vector), the element after the first element is... i One auxiliary angle, finally, use transpose matrix For extended anchor points Perform mapping (rotation) to obtain the final anchor point (i.e., the updated second text feature vector): .

[0102] 3) Image encoder

[0103] The input images (including benign and adversarial images) are encoded into image features. The input images may contain natural images and adversarial images generated by adding small perturbations. The image encoder transforms these images into feature embeddings, which will be used for subsequent adversarial training and classification tasks.

[0104] The primary function of an image encoder is to transform an input image into high-dimensional feature embeddings. These embeddings represent the visual information of the image and are used for alignment with text anchors, enabling classification and adversarial training. In adversarial training, the image encoder needs to handle both benign and adversarial images to ensure the model maintains high accuracy when facing adversarial attacks. Pre-trained deep neural networks (such as ResNet or ViT) are used as the underlying architecture. These models are pre-trained on large-scale image datasets (such as ImageNet) and possess excellent feature extraction capabilities. The feature vectors output by the image encoder are L2 normalized to ensure that the length of the feature vector is 1 (…). This treatment helps improve the effectiveness of adversarial training.

[0105] The image encoder receives two types of input images: benign images and adversarial images. Benign images are the original, unperturbed images, while adversarial images are generated by adding small perturbations to the original images. For each input image, the image encoder extracts its feature embeddings. These embeddings are used to calculate cosine similarity with text anchors, thus enabling classification and adversarial training. In adversarial training, small perturbations are added to the images (… Generate adversarial images. The magnitude and direction of the perturbation are determined by the gradient of the loss function ( The decision is made to maximize the classification loss of the model.

[0106] To further improve anchor-based adversarial training methods, Aligned Cross-Entropy (A-CE) loss is introduced into the supervision objective. A-CE loss combines cross-entropy loss with the anchor alignment process (cosine similarity between the image feature vector of the adversarial image and the anchor point). This can be seen as a combination of L2-normalized features in InfoNCE. In standard adversarial training settings, traditional CE loss is often inferior to anchor-based targets (such as the angle between the image feature vector of the adversarial image and the anchor point). and (This is effective. However, the A-CE loss is affected by the softmax function.) The target was relaxed to improve its performance in high cosine similarity problems. The formula for the A-CE loss function is as follows:

[0107] ;

[0108] in, To find the average function, Through The generated adversarial perturbation can be used to perturb the images in the first image set. It should be noted that... It is a disturbance, also known as a disturbance parameter, disturbance. The initial value can be preset, for example, set to 0, and subsequently adjusted according to the gradient. Update perturbation The updated perturbation can then be utilized. The images in the first image set are perturbed to generate an updated adversarial image set. This allows subsequent training rounds to be performed using both the first image set and the corresponding updated adversarial image set, until a training stopping condition is met (e.g., reaching a preset maximum number of training iterations with minimal loss variation, such as the loss difference between adjacent training iterations being less than a preset difference, etc., without specific limitations). Training then stops, resulting in the trained multimodal model. This formula indicates that for each input image x and its corresponding category... After calculating the counter-disturbance With all anchor points The cosine similarity is calculated, then normalized using the softmax function, and the negative logarithm is used as the loss. As an example, during training, the parameters of the multimodal model can be fixed first, and the perturbation optimized. Then fix the disturbance Optimize the parameters of the multimodal model.

[0109] 4) Alignment and testing of adversarial features and text anchors

[0110] Adversarial training is used to optimize the similarity between adversarial features and text anchors. The Alignment Cross-Entropy (A-CE) loss function is employed to maximize the similarity between adversarial features and their corresponding text anchors, while simultaneously using... A loss function is used to enhance the consistency between benign and adversarial features. Finally, in the testing phase, an image encoder is used to generate features for the test image, and classification prediction is performed by calculating the similarity with text anchors.

[0111] To further improve the adversarial robustness of adversarial training on unseen categories, a smoothing loss is proposed. , The loss mitigates the impact of adversarial perturbations by constraining the cosine similarity between benign and adversarial features. Specifically, the smoothing loss is defined as:

[0112] ;

[0113] in, This represents the benign features of the input image x. This represents the adversarial features. Maximizing the cosine similarity between benign and adversarial features encourages the model to generate more consistent feature embeddings during adversarial training. By maximizing the cosine similarity between benign and adversarial features, smoothing loss encourages the model to generate consistent feature embeddings during adversarial training. This consistency helps improve the model's robustness to adversarial attacks, especially in unseen categories.

[0114] During the testing phase, the cosine similarity between the test features of the test image and all extended text anchors (corresponding to the updated second text feature vectors) is calculated, and the final category is determined using the Argmax function. Specifically, for each test image, the cosine similarity between its feature vector and each extended text anchor is calculated. Using the backpropagation algorithm, the parameters of the image encoder are optimized based on the A-CE loss value to maximize the cosine similarity between the adversarial image features and the correct text anchors, thus achieving the alignment process. This process ensures that the test image can be correctly mapped to the corresponding text anchors, thereby achieving classification. For example, in... Figure 3 In the test, the target category of the determined test image was the fox category.

[0115] This application proposes a language-driven anchor expansion algorithm, a joint loss optimization strategy, and a dynamic alignment mechanism, which can solve problems such as insufficient adversarial capability, robustness deficiency, and transferability limitations in related technologies. By introducing mathematically modeled anchor expansion, cross-modal loss function design, and dynamic feature alignment, the accuracy and generalization ability of the model under complex adversarial attacks are significantly improved.

[0116] Language-driven multimodal anchor generation and expansion algorithm:

[0117] Semantic anchors for category-hint sentences are generated using a pre-trained text encoder (such as CLIP), and a spherical coordinate system expansion algorithm is designed to address the high cosine similarity problem caused by directly using CLIP anchors. The algorithm expands the anchor spacing by adjusting the polar angle while maintaining semantic consistency, making the anchors more dispersed and distinguishable in the feature space.

[0118] The specific implementation of the anchor point generation and expansion algorithm (including polar angle calculation, rotation matrix mapping, hyperspherical coordinate transformation, etc.).

[0119] Based on A-CE loss and Adversarial training framework with smooth loss:

[0120] An alignment cross-entropy loss (A-CE) is introduced, and combined with the cross-entropy loss and anchor alignment target to optimize the similarity between adversarial features and text anchors; a design is implemented. Smoothing loss constrains the cosine similarity between benign and adversarial features, improving the model's robustness to perturbations.

[0121] A-CE and The joint optimization strategy for smoothing loss and its specific application process in multimodal adversarial training.

[0122] Dynamic alignment mechanism between adversarial features and text anchors:

[0123] During the training phase, backpropagation is used to maximize the cosine similarity between adversarial features and corresponding text anchors, and the expanded anchors are used to optimize the classification boundary. During the testing phase, classification is achieved by calculating the similarity between image features and anchors, ensuring semantic consistency under adversarial perturbations.

[0124] The dynamic alignment method based on anchor point similarity includes gradient optimization during the training phase and similarity matching rules during the testing phase.

[0125] See Figure 3 , Figure 3 This is a schematic diagram of the structure of a multimodal model training device 300 provided in an embodiment of this application, which can be applied to electronic devices, such as... Figure 3 As shown, the multimodal model training device 300 includes:

[0126] The first acquisition module 301 is used to acquire category text information of N categories in the training image set;

[0127] The second acquisition module 302 is used to acquire N first text feature vectors obtained by the text encoder in the multimodal model based on the category text information of N categories. The first text feature vectors are vectors in the Cartesian coordinate system, and N is an integer greater than 1.

[0128] The first determining module 303 is used to determine the polar angle and auxiliary angle of the N first text feature vectors in the spherical coordinate system based on the N first text feature vectors.

[0129] The first adjustment module 304 is used to increase the polar angle of the N first text feature vectors according to the maximum value among the polar angles of the N first text feature vectors, so as to obtain the adjusted polar angle of the N first text feature vectors.

[0130] The second determining module 305 is used to determine N second text feature vectors in the Cartesian coordinate system based on N first text feature vectors, auxiliary angles, and adjusted polar angles.

[0131] Training module 306 is used to train the image encoder in the multimodal model based on N second text feature vectors and the training image set, so as to obtain the trained multimodal model, which is used for image classification.

[0132] In some embodiments, the multimodal model training device 300 further includes:

[0133] The third determining module is used to determine the average vector of the N first text feature vectors;

[0134] The fourth determining module is used to determine the rotation matrix based on the average vector and the preset target vector;

[0135] The first rotation module is used to rotate the N first text feature vectors according to the rotation matrix in order to update the N first text feature vectors.

[0136] In some embodiments, the multimodal model training device 300 further includes:

[0137] The second rotation module is used to rotate the N second text feature vectors according to the transpose of the rotation matrix, so as to update the N second text feature vectors.

[0138] In some embodiments, the adjusted polar angle of the first text feature vector is M times the polar angle of the first text feature vector, where M is... The result of dividing by the maximum value.

[0139] In some embodiments, the training image set includes a first image set and an adversarial image set of the first image set, wherein the adversarial image set is an image obtained by perturbing the images in the first image set; based on N second text feature vectors and the training image set, the image encoder in the multimodal model is trained to obtain the trained multimodal model, including:

[0140] The first image set and the adversarial image set are input into the image encoder for feature extraction to obtain the first image feature vector of the first image set and the second image feature vector of the adversarial image in the adversarial image set.

[0141] The target loss is determined based on the first image feature vector, the second image feature vector, and N second text feature vectors.

[0142] Based on the target loss, the image encoder is trained to obtain the trained multimodal model.

[0143] In some embodiments, the target loss is determined based on a first image feature vector, a second image feature vector, and N second text feature vectors, including:

[0144] Determine the first similarity between the second image feature vector of each adversarial image in the adversarial image set and N second text feature vectors;

[0145] The first loss is determined based on the first similarity.

[0146] Determine the second similarity between the feature vectors of the first image and the feature vectors of the second image;

[0147] The second loss is determined based on the second similarity.

[0148] Based on the first loss and the second loss, the target loss is determined.

[0149] In some embodiments, the multimodal model training device 300 further includes:

[0150] The image acquisition module is used to acquire test images for image classification scenarios;

[0151] The feature extraction module is used to extract features from the image encoder in the trained multimodal model by inputting the test image into it, and obtain the image feature vector of the test image.

[0152] The fifth determination module is used to determine the third similarity between the image feature vector of the test image and the first text feature vector of N categories, so as to obtain N third similarities;

[0153] The category determination module is used to determine the target category of a test image from N categories based on N third similarities.

[0154] The multimodal model training device 300 provided in this embodiment can realize the various processes of the above-described multimodal model training methods, with one-to-one correspondence of technical features and the same technical effect. To avoid repetition, it will not be described again here.

[0155] This application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described multimodal model training method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0156] For details, see Figure 4 This application also provides an electronic device, including a bus 401, a transceiver 402, an antenna 403, a bus interface 404, a processor 405, and a memory 406.

[0157] The processor 405 is used for:

[0158] Obtain the category text information for N categories in the training image set;

[0159] Obtain N first text feature vectors from the text encoder in the multimodal model, which are obtained based on the category text information of N categories. The first text feature vectors are vectors in the Cartesian coordinate system, and N is an integer greater than 1.

[0160] Based on the N first text feature vectors, determine the polar angle and auxiliary angle of the N first text feature vectors in the spherical coordinate system;

[0161] Based on the maximum value among the polar angles of the N first text feature vectors, the polar angles of the N first text feature vectors are increased to obtain the adjusted polar angles of the N first text feature vectors.

[0162] Based on the N first text feature vectors, auxiliary angles, and adjusted polar angles, determine the N second text feature vectors in the Cartesian coordinate system;

[0163] Based on N second text feature vectors and the training image set, the image encoder in the multimodal model is trained to obtain the trained multimodal model, which is then used for image classification.

[0164] exist Figure 4 In this context, a bus architecture (represented by bus 401) is used. Bus 401 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 405 and memory represented by memory 406. Bus 401 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 404 provides an interface between bus 401 and transceiver 402. Transceiver 402 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 405 is transmitted over a wireless medium via antenna 403, which further receives data and transmits data to processor 405.

[0165] Processor 405 is responsible for managing bus 401 and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 406 can be used to store data used by processor 405 during operation.

[0166] The processor 405 in the electronic device provided in this embodiment can implement each process of each embodiment of the above-described multimodal model training method. The technical features correspond one-to-one and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0167] Optionally, the processor 405 can be a CPU, ASIC, FPGA, or CPLD.

[0168] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described multimodal model training method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0169] This application provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the method described in the embodiment. The technical features are one-to-one and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0170] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0171] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0172] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A multimodal model training method, characterized in that, The method includes: Obtain the category text information for N categories in the training image set; Obtain N first text feature vectors from the text encoder in the multimodal model based on the category text information of the N categories. The first text feature vectors are vectors in the Cartesian coordinate system, and N is an integer greater than 1. Based on the N first text feature vectors, determine the polar angle and auxiliary angle of the N first text feature vectors in the spherical coordinate system; Based on the maximum value among the polar angles of the N first text feature vectors, the polar angles of the N first text feature vectors are increased to obtain the adjusted polar angles of the N first text feature vectors; Based on the N first text feature vectors, the auxiliary angle, and the adjusted polar angle, determine the N second text feature vectors in the Cartesian coordinate system; Based on the N second text feature vectors and the training image set, the image encoder in the multimodal model is trained to obtain the trained multimodal model, which is used for image classification. Wherein, the adjusted polar angle of the first text feature vector is M times the polar angle of the first text feature vector, where M is... The result of dividing by the maximum value.

2. The method according to claim 1, characterized in that, Before determining the polar angle and auxiliary angle of the N first text feature vectors in the spherical coordinate system based on the N first text feature vectors, the method further includes: Determine the average vector of the N first text feature vectors; The rotation matrix is ​​determined based on the average vector and the preset target vector; The N first text feature vectors are rotated according to the rotation matrix to update the N first text feature vectors.

3. The method according to claim 2, characterized in that, Before training the image encoder in the multimodal model based on the N second text feature vectors and the training image set to obtain the trained multimodal model, the process further includes: The N second text feature vectors are rotated according to the transpose of the rotation matrix to update the N second text feature vectors.

4. The method according to any one of claims 1-3, characterized in that, The training image set includes a first image set and an adversarial image set of the first image set, wherein the adversarial image set consists of images obtained by perturbing the images in the first image set; the step of training the image encoder in the multimodal model based on the N second text feature vectors and the training image set to obtain the trained multimodal model includes: The first image set and the adversarial image set are input into the image encoder for feature extraction to obtain the first image feature vector of the image in the first image set and the second image feature vector of the adversarial image in the adversarial image set. Based on the first image feature vector, the second image feature vector, and the N second text feature vectors, the target loss is determined; The image encoder is trained based on the target loss to obtain a trained multimodal model.

5. The method according to claim 4, characterized in that, The step of determining the target loss based on the first image feature vector, the second image feature vector, and the N second text feature vectors includes: Determine the first similarity between the second image feature vector of each adversarial image in the adversarial image set and N second text feature vectors; Based on the first similarity, determine the first loss; Determine the second similarity between the first image feature vector and the second image feature vector; Based on the second similarity, determine the second loss; The target loss is determined based on the first loss and the second loss.

6. The method according to claim 1, characterized in that, After training the image encoder in the multimodal model based on the N second text feature vectors and the training image set to obtain the trained multimodal model, the process further includes: Obtain test images for image classification scenarios; The test image is input into the image encoder of the trained multimodal model for feature extraction to obtain the image feature vector of the test image; Determine the third similarity between the image feature vector of the test image and the first text feature vectors of the N categories to obtain N third similarities; Based on the N third similarities, the target category of the test image is determined from the N categories.

7. A multimodal model training device, characterized in that, The device includes: The first acquisition module is used to acquire the category text information of N categories in the training image set; The second acquisition module is used to acquire N first text feature vectors obtained by the text encoder in the multimodal model based on the category text information of the N categories. The first text feature vectors are vectors in the Cartesian coordinate system, and N is an integer greater than 1. The first determining module is used to determine the polar angle and auxiliary angle of the N first text feature vectors in the spherical coordinate system based on the N first text feature vectors. The first adjustment module is used to increase the polar angle of the N first text feature vectors according to the maximum value among the polar angles of the N first text feature vectors, so as to obtain the adjusted polar angles of the N first text feature vectors. The second determining module is used to determine N second text feature vectors in the Cartesian coordinate system based on the N first text feature vectors, the auxiliary angle, and the adjusted polar angle. The training module is used to train the image encoder in the multimodal model based on the N second text feature vectors and the training image set to obtain the trained multimodal model, which is used for image classification. Wherein, the adjusted polar angle of the first text feature vector is M times the polar angle of the first text feature vector, where M is... The result of dividing by the maximum value.

8. An electronic device, characterized in that, The electronic device is a network device, including a transceiver and a processor. The processor is used for: Obtain the category text information for N categories in the training image set; Obtain N first text feature vectors from the text encoder in the multimodal model based on the category text information of the N categories. The first text feature vectors are vectors in the Cartesian coordinate system, and N is an integer greater than 1. Based on the N first text feature vectors, determine the polar angle and auxiliary angle of the N first text feature vectors in the spherical coordinate system; Based on the maximum value among the polar angles of the N first text feature vectors, the polar angles of the N first text feature vectors are increased to obtain the adjusted polar angles of the N first text feature vectors; Based on the N first text feature vectors, the auxiliary angle, and the adjusted polar angle, determine the N second text feature vectors in the Cartesian coordinate system; Based on the N second text feature vectors and the training image set, the image encoder in the multimodal model is trained to obtain the trained multimodal model, which is used for image classification. Wherein, the adjusted polar angle of the first text feature vector is M times the polar angle of the first text feature vector, where M is... The result of dividing by the maximum value.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

11. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-6.

Images