A progressive fine-tuning method and system of a multi-modal pre-training model

Abstract

The application discloses a kind of progressive fine-tuning method and system of multimodal pre-training model, belong to the field of depth learning, the method includes from the visual encoder of multimodal large model that has completed pre-training obtains high-dimensional visual feature vector, and obtains text feature vector from text encoder, and heteromodal feature is constructed;The contribution degree of different modalities is analyzed by resource allocation strategy, and the whole fine-tuning process is dynamically guided, and the limited computing resources are allocated to the multimodal large model component that contributes most to the current task;The final optimization target is obtained by processing the heteromodal feature by cross-modal contrast consistency model.The application can train larger or use larger batch size for fine-tuning training under limited hardware conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, and in particular to a progressive fine-tuning method and system for a multi-modal pre-training model. BACKGROUND

[0002] With the development of deep learning technology, pre-training models based on the Transformer architecture have been widely used in natural language processing and computer vision, especially in visual-linguistic tasks. By pre-training on large-scale data, these models learn general feature representations, and then fine-tune on specific tasks to improve overall performance. However, existing multi-modal pre-training models often lack generalization ability when dealing with data from different domains. Furthermore, in practical applications, models often need to be transferred between different domains. However, existing multi-modal pre-training models often need to be retrained or fine-tuned with a large number of parameters when facing new domain data, which not only consumes time and effort, but also may cause the model to lose its original knowledge. Therefore, how to efficiently transfer between domains and adapt to tasks while maintaining the original structure and parameters of the model has become an important research direction in the field of multi-modal learning. SUMMARY

[0003] One of the objectives of the present application is to provide a progressive fine-tuning method for a multi-modal pre-training model to solve the problem of needing to retrain or fine-tune a large number of parameters when facing new domain data in the prior art.

[0004] The present application is achieved by the following technical solution: a progressive fine-tuning method for a multi-modal pre-training model, comprising the following steps: S100, obtaining high-dimensional visual feature vectors from the visual encoder of a pre-trained multi-modal large model, and obtaining text feature vectors from the text encoder to construct heterogeneous modal features; S200, analyzing the contribution of different modalities through a resource allocation strategy to dynamically guide the entire fine-tuning process, and allocating limited computing resources to the multi-modal large model component that contributes most to the current task; S300, processing the heterogeneous modal features through a cross-modal contrast consistency model to obtain the final optimization target, wherein the cross-modal contrast consistency model does not change the original encoder under the premise of recognizing that different modalities have great differences, but constructs a projection layer and dynamically defines a good geometric structure through a contrast method

[0005] Further, the progressive fine-tuning method further comprises: S400, in the fine-tuning training process, using an automatic mixed precision technology to automatically convert model parameters and activation values in calculation from 32-bit floating point numbers to 16-bit floating point numbers for calculation, and then converting back to 32-bit floating point numbers for gradient update, thereby reducing GPU memory occupation overhead; and for the backbone of a visual or text encoder in the model, by using a gradient checkpoint technology, in the forward propagation process, the activation values of the intermediate layers are not saved, and the activation values are recalculated only when the gradient needs to be calculated in the backward propagation, thereby reducing the memory peak value caused by a long sequence or a deep network.

[0006] Further, the modality interaction layer of the multi-modal large model in S100 comprises: one or more cross-modal adapter bridge modules inserted for a multi-head self-attention module or a feedforward network module; the cross-modal adapter bridge module adopts a low-rank decomposition structure.

[0007] Further, the low-rank decomposition structure

[0008] ,

[0009] wherein, is a dimension reduction matrix that maps original high-dimensional features to a low-dimensional hidden space; is a dimension increase matrix that maps features from the low-dimensional hidden space back to the original dimension; is a nonlinear activation function, which can be a GELU function or a ReLU function; is the rank of the low-rank decomposition, and in the embodiment, preferably, is set to 64, at which time the amount of trainable parameters introduced is only about 0.3% of the original model backbone parameters. After the adapter is implanted, the backbone parameters of the pre-trained model are all frozen, and only the parameters of the adapter module are set to be trainable.

[0010] Further, the resource allocation strategy calculates the gradient of the weight parameters of each cross-modal attention layer in the model after forward and backward propagation in one or more training iterations, and calculates the L2 norm of the gradient matrix, takes the calculated L2 norm of the gradient matrix as the contribution score, and allocates different learning rates to different cross-modal attention layers according to the contribution score.

[0011] Further, S200 further comprises a progressive unfreezing scheduler, which makes the model adapt to the task quickly in the early stage of training by using very few parameters, and then fine-tunes the performance of the model by unfreezing more parameters in the later stage. The progressive unfreezing scheduler monitors one or more preset performance indicators of the model on the validation set at the end of each training cycle to maintain a priority list of a to-be-unfrozen parameter layer of the progressive unfreezing scheduler. By setting a performance stagnation threshold, when the performance improvement on the validation set in the last N cycles is less than the threshold, the scheduler is triggered. After the trigger, the scheduler selects a part of the currently frozen parameters for unfreezing according to the preset priority strategy.

[0012] Further, the resource allocation strategy further comprises: setting a contribution score threshold, setting layers with scores higher than the threshold as important layers, and setting layers with scores lower than the threshold as secondary layers; and allocating a learning rate higher than that of the secondary layers to the important layers, thereby accelerating the convergence of the model to obtain better performance.

[0013] Further, the cross-modal contrast consistency model is constructed by the following steps: S310, converting and mapping the heterogeneous modal features into a unified and standardized shared semantic space, so that the image features and the text features can be measured and compared in the space; S320, constructing a structured optimization objective function according to the constructed shared semantic space to realize accurate cross-modal alignment and guide the training process of the projection function. The optimization objective function is a multi-objective composite loss function, which includes: a core cross-modal contrast loss that pulls close the matching positive samples and pushes away the mismatched negative samples in the global range; an intra-modal clustering loss that forms compact clusters of samples of the same semantic category in their respective modal spaces, thereby enhancing the quality of feature expression within a single modal; and an orthogonal regularization term that encourages each feature dimension learned by the projection transformation to be mutually independent, thereby improving the expression efficiency of the feature space and avoiding dimension redundancy. The multi-objective composite loss function is as follows: wherein, is a cross-modal contrast loss; is an intra-modal clustering loss, is an orthogonal regularization term, is a hyperparameter for balancing the weight of the intra-modal clustering loss in the total loss; is a hyperparameter for balancing the weight of the orthogonal regularization term loss in the total loss; S330, taking the composite loss function as the final optimization objective, applying the model to the fine-tuning task through the standard model training and inference process.

[0014] Further, the shared semantic space linearly transforms the original feature vectors of the visual modality and the text modality into a vector space of the same target dimension by respectively designing a separate projection function for the visual modality and the text modality, and on this basis, normalizes the obtained vectors after projection by L2 norm, so that the feature vectors of all modalities are constrained on the same unit hypersphere, providing a standardized geometric basis for subsequent contrastive learning.

[0015] Further, the process of linearly transforming the original feature vectors of the visual modality and the text modality into a vector space of the same target dimension can be achieved by defining the visual feature vector from the visual encoder as: ; and defining the text feature vector from the text encoder as: , mapping into a shared semantic space of a unified dimension through respective independent and trainable projection functions and , and performing L2 norm normalization to obtain unit feature vectors and , which can be specifically represented by the following formula:

[0016] ,

[0017] wherein, is the original visual feature vector extracted from the visual encoder (such as ResNet), with a dimension of ; is the original text feature vector extracted from the text encoder (such as BERT), with a dimension of ; is the projection function acting on the visual feature, which can be: ; is the projection function acting on the text feature, which can be: ; is the trainable parameter weight matrix of the visual feature vector, is the trainable parameter weight matrix of the text feature vector; is the bias term of the visual feature vector, is the bias term of the text feature vector; represents the L2 norm of the vector, representing the length of the vector in space; is the visual feature vector after projection and L2 normalization, with a dimension of ; is the text feature vector after projection and L2 normalization, also with a dimension of .

[0018] Further, the cross-modal contrastive loss term is constructed by an InfoNCE-based loss function. For a given visual feature, its corresponding text feature is the positive sample, while all other text features in a batch form the negative sample set. By maximizing the similarity between the anchor and the positive sample and minimizing the similarity between the anchor and all negative samples, the similarity score is converted into a probability distribution using a Softmax function with a temperature parameter, thereby driving the alignment of features at the probability level. The intra-modal clustering loss term is obtained by projecting all samples belonging to the same class in the data set to the shared semantic space, so that their vector representations are also close. The Euclidean distance between all same-class sample pairs is penalized to encourage the model to learn more discriminative single-modal feature representations, so that expressions of the same concept form a compact cluster. The orthogonal regularization term is obtained by imposing constraints on the image weight matrix and the text weight matrix from the perspective of linear algebra. When the column vectors of a matrix are orthogonal unit vectors, the matrix is called an orthogonal matrix, which satisfies where is the identity matrix, is the matrix transpose symbol. By adding a regularization term to penalize the difference between and the identity matrix , the two projection matrices gradually tend to orthogonal matrices during the training process, prompting the model to learn a set of orthogonal feature bases, achieving feature decoupling, and improving the generalization ability and expression efficiency of the model.

[0019] Further, the InfoNCE-based cross-modal contrastive loss function can be expressed as follows:

[0020] where is the index set of all positive sample pairs (i.e., matched image-text pairs) in a batch; is the number of positive sample pairs; indicates the th visual sample and the th text sample is a positive sample pair; is the index set of all negative samples corresponding to the th visual sample; is a similarity calculation function, which is defined as cosine similarity in this formula; and are the projected and normalized visual and text feature vectors, respectively; is the temperature parameter; is the natural exponential function; is the logarithmic function.

[0021] Further, the intra-modality clustering loss can be represented by the following formula:

[0022] wherein, denotes mathematical expectation, which is usually approximated by the sample mean within a batch in practical calculation; denotes and are two different visual samples belonging to the same semantic category extracted from the data set; denotes and are two different text samples belonging to the same semantic category; and are the feature vectors of the projected and normalized visual and text features, respectively.

[0023] Further, the orthogonal regularization term can be represented by the following formula:

[0024] wherein, and are the learnable weight matrices in the visual and text projection functions and respectively; denotes the transpose of the matrix ; is an identity matrix (a square matrix with 1 on the main diagonal and 0 elsewhere); is the product of the matrix and its transpose; is the Frobenius Norm of the matrix, which can be regarded as the L2 norm of the matrix in function.

[0025] Another aspect of the present application provides a progressive fine-tuning system for a multi-modal pre-training model, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements any of the progressive fine-tuning methods for the multi-modal pre-training model as described above when executing the program.

[0026] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0027] 1. The present application inserts a cross-modal adapter bridge module in the modality interaction layer of the model and adopts a low-rank decomposition structure, thereby achieving efficient adaptation and migration of data in different fields, and introducing a dynamic differentiated learning rate allocation strategy based on modality contribution degree analysis, dynamically adjusting the learning rate by calculating the gradient norm of the weight parameters of each cross-modal attention layer, thereby intelligently allocating computing resources according to the contribution of different modality layers to the final loss, accelerating the model convergence process, and significantly improving the training efficiency of the model.

[0028] 2、The application realizes intelligent unfreezing and training of model parameters by introducing a progressive hierarchical unfreezing scheduler, the scheduler in the application dynamically selects the parameter layer to be unfreezed according to the performance index of the model on the validation set, and adopts an on-demand unfreezing mode, which not only ensures the rapid adaptation of the model in the early training stage, but also avoids unnecessary parameter updating, effectively balancing the relationship between the training efficiency and the performance of the model.

[0029] 3、The application realizes deep interaction and semantic alignment of features of different modalities by introducing a cross-modal contrast consistency regularization model and a multi-objective composite loss function, which not only ensures the semantic consistency of features between different modalities, but also avoids feature dimension redundancy through orthogonal regularization terms, improving the expression efficiency of the feature space.

[0030] 4、The application significantly reduces the GPU memory occupation overhead in the model fine-tuning training process by combining automatic mixed precision technology and gradient checkpoint technology, converts model parameters and activation values between different precisions, and recalculates intermediate layer activation values during back propagation, effectively releasing GPU memory resources, so that larger or larger batch size can be trained for fine-tuning training under limited hardware conditions. BRIEF DESCRIPTION OF DRAWINGS

[0031] The drawings described herein are used to provide further understanding of the embodiments of the application, constitute a part of the application, and do not constitute a limitation on the embodiments of the application. In the drawings:

[0032] Figure 1 The method flowchart is provided for the embodiment 1 of the application.

[0033] Figure 2 The timing diagram of the cross-modal contrast consistency model is provided for the embodiment 1 of the application.

[0034] Figure 3 The flowchart of the cross-modal contrast consistency model is provided for the embodiment 1 of the application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. The components of the embodiments of the application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0036] Embodiment 1

[0037] The embodiment discloses a progressive fine-tuning method of a multi-modal pre-training model. The progressive fine-tuning method will be described in detail below in combination with the accompanying drawings. It should be noted that the execution environment of the embodiment can be one or more servers, and the hardware environment thereof can include a processor (CPU), a graphics processor (GPU), a memory, a hard disk, and the like, and the software environment thereof can include an operating system and a deep learning framework (such as PyTorch or TensorFlow).

[0038] Figure 1 The method described in the embodiment is shown in a flowchart. As can be seen from the figure, the embodiment includes the following steps:

[0039] Step 1: First, a large multi-modal pre-training model that has been pre-trained is obtained. In the embodiment, this model can be a visual-textual model based on a Transformer architecture. The structure inside the model includes a visual encoder (such as ViT) for extracting visual features from images, a textual encoder (such as BERT) for extracting textual features from texts, and a modal interaction layer for information interaction and fusion of features of the two modalities.

[0040] Generally speaking, traditional full-parameter fine-tuning not only requires huge computing resources and storage space, but also needs to save a complete model copy for each new task adaptation, and can also cause the model to forget the general knowledge learned in the pre-training stage, that is, the catastrophic forgetting problem.

[0041] To solve this problem, the core strategy adopted in the embodiment is to freeze the backbone and inject new modules. Specifically, in the embodiment, after obtaining the model, to achieve efficient fine-tuning of parameters, the entire parameters of the model are not directly trained. By freezing all the parameters (such as the backbone of the visual and textual encoders) that account for the vast majority of the pre-training model, the parameters remain unchanged during training. Then, one or more lightweight, trainable cross-modal adapter bridge modules are inserted at the key positions of the model, which are specifically the Multi-Head Self-Attention module or the Feed-Forward Network module in the modal interaction layer. The core role of the adapter module is to serve as a customizable and pluggable modifier or adjuster for new tasks. This adapter does not change the original knowledge structure of the model, but learns to make small but critical task-related adjustments to the features on the basis of retaining the strong general ability of the backbone model.

[0042] Specifically, the cross-modal adapter bridge module can adopt a low-rank decomposition structure. That is, for an input feature vector The calculation process thereof can be represented as:

[0043] ,

[0044] wherein, is a dimension reduction matrix that maps the original high-dimensional features to a low-dimensional hidden space; is a dimension increase matrix that maps the features from the low-dimensional hidden space back to the original dimension. is a nonlinear activation function, which can be a GELU function or a ReLU function. is the rank of the low-rank decomposition, and in the embodiment, preferably, can be set to 64, at which time the amount of trainable parameters introduced is only about 0.3% of the original model backbone parameters. After implanting the adapter, the backbone parameters of the pre-trained model are all frozen, and only the parameters of the adapter module are set to be trainable; is the fixed size of the core feature vector.

[0045] It should be noted that this output feature corrected by the adapter is the key link connecting all the steps. The can continue to propagate in the model as the input of the next layer. After multiple layers containing adapters, the visual and text feature vectors finally output by the model have been completely affected by these trainable adapter modules. These final output feature vectors that have been deeply adjusted by the adapter are the direct input for calculating the total loss function, especially the core loss, in step 3. That is, the goal of subsequent step 3 is to judge whether all adapter modules have done a good job of modification by measuring the degree of alignment of these features. If the image-text features are well aligned, the loss is low; otherwise, it is high. At the same time, in order to reduce the loss calculated in step 3, the gradient needs to be calculated through backpropagation. However, since the backbone parameters are frozen, this gradient will only flow to and act on those trainable parameters - that is, all the adapter modules we inserted. Therefore, in the next step, the gradient norm of each adapter module parameter can be analyzed. The adapter with a larger gradient means that it contributes more to the final alignment task and should be given a higher learning rate for focused optimization.

[0046] Step 2: In each iteration of fine-tuning, the model receives a pair of image and text as input. The image data flows through the visual encoder (e.g. ViT) mentioned in Step 1, finally generating a high-dimensional visual feature vector; similarly, the text data is processed by the text encoder (e.g. BERT), generating a text feature vector. Since these two feature vectors originate from different modalities, are in different vector spaces, and are not aligned, they are referred to as heterogeneous modal features. These features are the starting point for all subsequent alignment tasks and are the core processing objects to be optimized in Step 3. Through a carefully designed loss function, the alignment degree of these heterogeneous features is evaluated. In order to minimize the loss of heterogeneous feature alignment, the gradient of the loss function with respect to the trainable parameters of the model needs to be calculated. Therefore, in this embodiment, a more intelligent resource allocation strategy is used, which is not the same for all parameters, but dynamically guides the entire fine-tuning process by analyzing the contribution of different modalities. The core idea is to allocate limited computing resources more intelligently to the model components that contribute most to the current task.

[0047] Specifically, in this embodiment, a dynamic, differentiated learning rate allocation strategy based on modal contribution analysis is used to achieve dynamic allocation of fine-tuning resources. The criterion for modal contribution analysis can be based on the gradient norm of the parameters. Generally, a larger gradient norm indicates that the layer contributes more to the reduction of the final loss, so its modal contribution score will be higher. Therefore, the gradient of the weight parameters of each cross-modal attention layer in the model can be calculated after one or more forward and backward propagations during training iterations, and then the L2 norm of the gradient matrix is calculated. At the same time, the calculated L2 norm of the gradient matrix is used as the contribution score. Different cross-modal attention layers are allocated different learning rates based on the contribution score.

[0048] For example, in this embodiment, by setting a contribution score threshold, layers with a score higher than the threshold are set as important layers and are allocated a higher learning rate (e.g. ); while layers with a score lower than the threshold are set as secondary layers and are allocated a lower learning rate (e.g. ). Through this non-uniform resource allocation method, the convergence of the model can be effectively accelerated to achieve better performance.

[0049] In this embodiment, in order to further balance the parameter efficiency and model performance, a gradual unfreezing scheduler can also be introduced. This scheduler can enable the model to use very few parameters to quickly adapt to the task in the early stages of training through a gradual, on-demand unfreezing approach; while in the later stages, more parameters can be unfrozen to further fine-tune the performance of the model, thereby achieving automation and intelligentization of the training process.

[0050] Specifically, only the parameters of the adapter modules are trainable at the beginning of training. The progressive unfreezing scheduler introduced in this embodiment maintains a priority list of parameter layers to be unfrozen by monitoring one or more preset performance indicators of the model on the validation set at the end of each training cycle. By setting a performance stagnation threshold , if the performance improvement on the validation set is less than the threshold in the last N cycles (e.g. N = 3), the scheduler is triggered. After being triggered, the scheduler selects a portion of the currently frozen parameters to be unfrozen (i.e. set to a trainable state) according to a preset priority strategy. The strategy prioritizes unfreezing parameter layers that have a greater impact on the final output of the model, for example, first unfreezing the top few modality fusion layers, and then the high-level representation layers of the visual and text encoders.

[0051] Exemplarily, in this embodiment, it is assumed that the multi-modal model in this embodiment consists of the following three parts: a 12-layer ViT model for the visual encoder (Vision Encoder); the layers are named Vision-L1 (the bottom layer) to Vision-L12 (the top layer). A 12-layer BERT model for the text encoder (Text Encoder); the layers are also named Text-L1 (the bottom layer) to Text-L12 (the top layer). A 6-layer Transformer decoder structure for modality interaction layers (Modality Interaction Layers) to fuse visual and textual information. We name the layers Fusion-L1 (the bottom layer) to Fusion-L6 (the top layer). According to the strategy of prioritizing unfreezing parameter layers that have a greater impact on the final output of the model, the following priority list of parameter layers to be unfrozen can be constructed, as shown in Table 1:

[0052] Table 1. Priority list of parameter layers to be unfrozen

[0053]

[0054] Step 3: Process the heterogeneous modality features obtained in Step 1 through the cross-modal consistency model to obtain the final optimization target.

[0055] In this embodiment, the total training objective function is composed of the basic task loss (e.g. an optional classification task loss) and the contrast consistency regularization loss . Wherein, is the core of realizing modality depth alignment. Figure 2A timing diagram of the cross-modal contrast consistency model in the embodiment is shown.

[0056] Specifically, in the embodiment, the cross-modal contrast consistency model is constructed by the following steps:

[0057] 1) First, the heterogeneous modal features need to be transformed and mapped into a unified, standardized shared semantic space. In order to effectively align and compare data from different sources, in the embodiment, a common semantic platform is constructed, so that image features and text features can be meaningfully measured and compared in this space, thereby solving the problem that data from different sources exist in different vector spaces in mathematics (i.e. dimensions, scales and semantic densities are all different).

[0058] Specifically, in the embodiment, an independent projection function can be designed for the visual modality and the text modality respectively, so as to linearly transform the original feature vectors of the visual modality and the text modality into a vector space of the same target dimension. And on this basis, in order to eliminate the interference caused by the different lengths of the original features, and ensure that the subsequent similarity calculation focuses on the direction of feature semantic content rather than its size. Therefore, by performing L2 norm normalization on the projected vectors, L2 norm normalization can constrain all modal feature vectors on the same unit hypersphere, thereby providing a standardized geometric basis for subsequent contrast learning.

[0059] Exemplarily, in the embodiment, the projection and alignment process of the heterogeneous feature vector to the shared semantic space can define the visual feature vector from the visual encoder as: ; the text feature vector from the text encoder is defined as: , through the respective independent and trainable projection functions and mapped to a unified dimension shared semantic space, and L2 norm normalization is performed, so as to obtain unit feature vectors and . Specifically, it can be represented by the following formula:

[0060] ,

[0061] wherein, is the original visual feature vector extracted from the visual encoder (such as ResNet), and the dimension is ; is the original text feature vector extracted from the text encoder (such as BERT), and the dimension is ; is the projection function acting on the visual feature, which can be: ; For the projection function acting on the text feature, we can have: ; is the trainable parameter weight matrix of the visual feature vector, is the trainable parameter weight matrix of the text feature vector; is the bias term of the visual feature vector, is the bias term of the text feature vector; represents the L2 norm of a vector, which represents the length of the vector in space; is the visual feature vector after projection and L2 normalization, and its dimension is ; is the text feature vector after projection and L2 normalization, and its dimension is also .

[0062] It should be noted that the formula disclosed in the present embodiment fully considers that the vector length (i.e., the norm) of the features extracted by different encoders may differ greatly, and if not normalized, the model may rely too much on the features with large norms when calculating the similarity. Through L2 normalization, all feature vectors are converted into points on a hypersphere with a radius of 1. Thus, the distance between the features extracted by different encoders is determined only by their directions (angles), and the inner product (dot product) of the vectors is equivalent to the cosine similarity. Thus, the model can more purely focus on the semantic direction of the features, rather than being disturbed by the inherent norms of irrelevant modalities.

[0063] 2) After a shared semantic space is constructed, a structured optimization objective function is constructed to achieve precise cross-modal alignment, thereby finely guiding the training process of the projection function. In the actual training process, it is not enough to only map the features to the same space, but also to ensure that the space has the correct semantic topological structure, that is, the image-text pairs related to semantics are close to each other in the space, while the unrelated image-text pairs are far away from each other. In order to achieve this goal, we cannot rely on a single optimization criterion, but should constrain and regularize the feature space from multiple dimensions.

[0064] Specifically, in the present embodiment, a multi-objective composite loss function can be constructed: first, a core cross-modal contrast loss is introduced, which pulls together the matching image-text pairs (i.e., positive samples) in the global range, while pushing away the unmatched image-text pairs (i.e., negative samples); second, an intra-modal clustering loss is introduced, which makes the samples of the same semantic category (e.g., all pictures or texts about "dog") form a compact cluster in their respective modal space, thereby enhancing the quality of feature expression within a single modality; finally, an orthogonal regularization term is introduced, which encourages each feature dimension learned by the projection transformation to be independent of each other, thereby improving the expression efficiency of the feature space and avoiding dimension redundancy.

[0065] Exemplarily, in the embodiment, the structured optimization objective function constructed may be represented by the following formula:

[0066] ,

[0067] wherein, is a cross-modal contrastive loss, which aims to pull closer the matching image-text pairs and push away the non-matching image-text pairs; is an intra-modal clustering loss, which aims to let the samples of the same modality that are semantically similar also be close to each other in the feature space; is an orthogonal regularization term, which aims to let the dimensions (basis vectors) of the projection matrix be as independent as possible; and are hyperparameters, which are scalar coefficients for balancing the weights of the intra-modal clustering loss and the orthogonal regularization term loss in the total loss, respectively.

[0068] Specifically, the cross-modal contrastive loss is actually a probability classification task for the model to learn to identify matching image-text pairs. In the embodiment, the problem can be transformed into a probability classification task of identifying a unique positive sample from a large number of negative samples by referring to the idea in information theory.

[0069] In the embodiment, the cross-modal contrastive loss term can be constructed by an InfoNCE-based loss function. For a given visual feature (i.e., anchor), its corresponding text feature is the positive sample, and all other text features in a batch form the negative sample set. The task of the model is to maximize the similarity between the anchor and the positive sample, while minimizing the similarity with all negative samples. A Softmax function with a temperature parameter can be used to convert the similarity score into a probability distribution, and the optimization objective is to maximize the log-likelihood of selecting the true sample. The essence is to maximize the lower bound of mutual information between matching image-text pairs, thereby driving the alignment of features at the probability level.

[0070] Exemplarily, in the embodiment, the InfoNCE-based cross-modal contrastive loss function can be expressed as follows:

[0071]

[0072] wherein, is the index set of all positive sample pairs (i.e., matching image-text pairs) in a batch; is the number of positive sample pairs. represents the th visual sample and the th text sample is a positive sample pair; the index set of all negative samples (i.e. mismatched text samples) corresponding to the first visual sample; the index set of all negative samples (i.e. mismatched text samples) corresponding to the first visual sample; is a similarity computation function, which is defined as cosine similarity in the following formula: and are the projected and normalized visual and text feature vectors, respectively; is a temperature parameter; is the natural exponential function; is the logarithmic function.

[0073] The intra-modal clustering loss term is to prevent the model from ignoring the semantic structure within a single modality when aligning cross-modal information (e.g. two different pictures of a cat are far apart in the feature space), in the present embodiment, an additional constraint is introduced to enhance the semantic consistency within the modality.

[0074] Specifically, the construction of the intra-modal clustering loss term in the present embodiment can be achieved by explicitly minimizing the distance between samples of the same semantic category and the same modality. That is, for all samples in the data set belonging to the same category, it is expected that their vector representations in the shared semantic space after the transformation of the projection function should also be as close as possible. By penalizing the Euclidean distance between all pairs of samples of the same category, the model can be prompted to learn more discriminative single-modal feature representations, so that the expression of the same concept (whether visual or textual) forms a compact cluster.

[0075] Exemplarily, in the present embodiment, the intra-modal clustering loss can be represented by the following formula:

[0076]

[0077] wherein, denotes the mathematical expectation, which is usually approximated by the sample mean within a batch in actual calculation; denotes and are two different visual samples extracted from the data set belonging to the same semantic category; denotes and are two different text samples belonging to the same semantic category; are the projected and normalized visual and text feature vectors, respectively.

[0078] The orthogonal regular term is to make the learned shared semantic space have higher expression efficiency. Therefore, it is necessary to reduce the information redundancy between the feature dimensions, and an ideal efficient feature space should have independent basis vectors (i.e. each dimension), and only the independent basis vectors can ensure that each dimension can capture the unique and irrelevant change pattern of other dimensions in the data.

[0079] Specifically, in the embodiment, the construction of the orthogonal regular term can impose constraints on the weight matrix and for realizing feature projection from the perspective of linear algebra. If the column vectors of a matrix are unit vectors that are orthogonal to each other, the matrix is called an orthogonal matrix, which satisfies (where is an identity matrix, and is a matrix transpose symbol). By increasing a regular term to punish the difference between and the identity matrix , the two projection matrices can be driven to gradually tend to an orthogonal matrix during the training process. This encourages the model to learn a set of orthogonal feature bases, thereby realizing feature decoupling, improving the generalization ability and expression efficiency of the model.

[0080] Exemplarily, in the embodiment, the orthogonal regular term can be as follows:

[0081] ,

[0082] wherein and are the learnable weight matrices in the visual and text projection functions and respectively; denotes the transpose of the matrix ; is an identity matrix (a square matrix whose main diagonal is 1 and the rest is 0); is the product of the matrix and its transpose; is the Frobenius norm of the matrix, which can be regarded as the L2 norm of the matrix in function.

[0083] It should be noted that the column vectors of a matrix can be regarded as a set of basis vectors under the linear transformation defined by the matrix. If the set of basis vectors is orthogonal, it means that the set of basis vectors is perpendicular to each other in space, and each basis vector (i.e. each dimension of the feature space) captures an independent and irrelevant change direction in the input data. When the feature dimensions are orthogonal to each other, the information correlation between them is the lowest, and each dimension provides new and irreplaceable information. On the contrary, if the dimensions are not orthogonal, it means that there is correlation between them, and there is information redundancy between them. By minimizing , the model can be encouraged to learn a set of efficient basis vectors, so that each feature dimension can perform its own function, and thus encode the richest information in a limited dimension d.

[0084] 3) Finally, the constructed optimization objective function is used as the final optimization objective, and the model is applied to downstream tasks through the standard model training and inference process.

[0085] After obtaining the feature space and the optimization objective, the model parameters can be optimized (fine-tuning stage) and the trained model can be used for prediction (inference stage) using data.

[0086] Specifically, in the fine-tuning stage, the model receives pairs of image-text data and their matching relationship (positive or negative samples), and iteratively updates the learnable parameters in the projection function by minimizing the optimization objective function defined above using optimization algorithms such as gradient descent, including weight matrix , bias term and temperature parameter . In the inference stage, the model is frozen, and its core function is to calculate the alignment score of any given image and text in the shared semantic space, which can be directly used for cross-modal retrieval, matching or classification in various application scenarios.

[0087] It should be noted that a flowchart is drawn according to the above description of the cross-modal contrast consistency model as Figure 3As shown, the cross-modal contrast consistency model in the embodiment is under the premise of recognizing that the original data of different modalities (such as vision and language) has great differences, without trying to change the original encoder, but by constructing a lightweight and trainable translation hub (i.e. projection layer). Then, instead of setting a specific and predefined target position for this hub, a good geometric structure is dynamically defined through contrast. And this defined geometric structure needs to meet three conditions: 1. Semantically matched cross-modal samples attract each other; 2. Semantically similar samples of the same modality form their own groups; 3. The coordinate axes expressing this space are perpendicular to each other and information is not redundant. Through a multi-objective loss function, these three seemingly scattered geometric constraints are unified to jointly shape a high-quality shared semantic space. A general and quantifiable semantic measurement is established for heterogeneous multi-modal data. In this space, the coordinates of the picture of 'kitten' and the sentence describing 'kitten' will be very close, while the picture or sentence of 'car' will be far away. Thus, cross-modal retrieval (text-to-image), data alignment, multi-modal fusion and other downstream applications become possible.

[0088] Step 4: In the embodiment, in order to further improve the efficiency of the multi-modal pre-training model and reduce the hardware resource requirement, the following content can also be included:

[0089] During the fine-tuning training process, the Automatic Mixed Precision (AMP) technology can be used. For example, using the torch.cuda.amp library of PyTorch, the model parameters and activation values in the calculation are automatically converted from FP32 (32-bit floating point number) to FP16 (16-bit floating point number) for calculation, and then converted back to FP32 when the gradient is updated, so that the GPU memory occupation overhead can be reduced without significantly affecting the accuracy, and the calculation can be accelerated using the Tensor Cores of the GPU.

[0090] And for the part of the calculation graph in the model that is deep (such as the backbone of the vision or text encoder), the Gradient Checkpointing technology can be used. By not saving the activation values of the intermediate layers during forward propagation, and only recalculating these activation values when the gradient needs to be calculated during back propagation, the memory peak value caused by long sequences or deep networks can be greatly reduced, so that larger models can be trained or larger batch sizes can be used for training under the condition of limited GPU memory.

[0091] By supplementing optimization techniques such as mixed precision training and gradient checkpointing, and combining the cross-modal contrast consistency model in the embodiment, an efficient, automatic and deep alignment achieving multi-modal pre-training model progressive fine-tuning method is realized. The method disclosed in the embodiment only needs to fine-tune a small amount of parameters, which can significantly reduce the consumption of computing resources while achieving or even exceeding the performance of full parameter fine-tuning.

[0092] The above specific embodiments further illustrate the purposes, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A progressive fine-tuning method of a multi-modal pre-training model, characterized in that, The progressive fine-tuning method comprises: S100, obtaining a high-dimensional visual feature vector from a visual encoder of a pre-trained multi-modal large model, and obtaining a text feature vector from a text encoder, to construct a heterogeneous modal feature; S200, analyzing the contribution degrees of different modalities through a resource allocation strategy, dynamically guiding the entire fine-tuning process, and allocating limited computing resources to the multi-modal large model component that contributes most to the current task; S300, processing the heterogeneous modal feature through a cross-modal contrast consistency model to obtain a final optimization target, The cross-modal contrast consistency model does not change the original encoder under the premise of acknowledging that the original data of different modalities has great differences, but constructs a projection layer and dynamically defines a good geometric structure through a contrast method; The S200 further comprises a progressive unfreezing scheduler, The progressive unfreezing scheduler uses a progressive and on-demand unfreezing method to enable the model to quickly adapt to the task using few parameters in the early training stage, and to fine-tune the performance of the model by unfreezing more parameters in the later stage, The progressive unfreezing scheduler monitors one or more preset performance indicators of the model on the validation set at the end of each training cycle to maintain a priority list of a to-be-unfrozen parameter layer of the progressive unfreezing scheduler; by setting a performance stagnation threshold, when the performance improvement on the validation set in the last N cycles is less than the threshold, the scheduler is triggered, The triggered scheduler selects a part of the currently frozen parameters for unfreezing according to a preset priority strategy; The cross-modal contrast consistency model is constructed through the following steps: S310, the heterogeneous modal feature is transformed and mapped into a unified and standardized shared semantic space, so that the image feature and the text feature can be measured and compared in the space; S320, a structured optimization objective function is constructed according to the constructed shared semantic space, to realize accurate cross-modal alignment and guide the training process of the projection function, The optimization objective function is a multi-objective composite loss function, comprising: a core cross-modal contrast loss that narrows the matching positive samples and pushes away the negative samples in the global range; an intra-modal clustering loss that forms a compact cluster of samples of the same semantic category in the respective modal space, thereby enhancing the feature expression quality of a single modal; and an orthogonal regularization term that encourages each feature dimension learned by the projection transformation to be mutually independent, thereby improving the expression efficiency of the feature space and avoiding dimension redundancy; The multi-objective composite loss function is shown in the following formula: ； wherein, is a cross-modal contrastive loss; is an intra-modal clustering loss, is an orthogonal regularizer term, is a hyper-parameter for balancing the weight of the intra-modal clustering loss in the total loss; is a hyper-parameter for balancing the weight of the orthogonal regularizer term loss in the total loss; S330, the composite loss function is used as the final optimization target, and the model is applied to the fine-tuning task through a standard model training and inference process.

2. The progressive fine-tuning method of a multi-modal pre-trained model according to claim 1, wherein, The progressive fine-tuning method further comprises: S400, in the fine-tuning training process, an automatic mixed precision technology is used to automatically convert the model parameters and the activation values in the calculation from 32-bit floating point numbers to 16-bit floating point numbers for calculation, and then convert back to 32-bit floating point numbers for gradient update, thereby reducing the GPU memory occupation overhead; For the backbone of the visual or text encoder in the model, the gradient checkpointing technique is used, so that during the forward propagation process, the activation values of the intermediate layers are not saved, and these activation values are recalculated only when the gradient needs to be calculated during the backward propagation, thereby reducing the memory peak caused by long sequences or deep networks.

3. The progressive fine-tuning method of a multi-modal pre-trained model according to claim 1, wherein, The modality interaction layer of the multi-modal large model in the S100 comprises: For the multi-head self-attention module or the feedforward network module, one or more cross-modal adapter bridge modules are inserted. The cross-modal adapter bridge module adopts a low-rank decomposition structure.

4. The progressive fine-tuning method of a multi-modal pre-trained model according to claim 1, wherein, The resource allocation strategy calculates the gradient of the weight parameters of each cross-modal attention layer in the model after the forward and backward propagation of one or more training iterations, calculates the L2 norm of the gradient matrix, takes the calculated L2 norm of the gradient matrix as the contribution score, and allocates different learning rates to different cross-modal attention layers according to the contribution score.

5. The progressive fine-tuning method of a multi-modal pre-trained model according to claim 4, wherein, The resource allocation strategy further comprises: setting a contribution score threshold, setting the layers with scores higher than the threshold as important layers, and setting the layers with scores lower than the threshold as secondary layers; An important layer is allocated a learning rate higher than that of a secondary layer, thereby accelerating the convergence of the model to obtain better performance.

6. The progressive fine-tuning method of a multi-modal pre-trained model according to claim 1, wherein, The shared semantic space linearly transforms the original feature vectors of the visual modality and the text modality into a vector space of the same target dimension by designing an independent projection function for the visual modality and the text modality, and on this basis, the L2 norm of the projected vector is normalized, so that the feature vectors of all modalities are constrained on the same unit hypersphere, providing a standardized geometric basis for subsequent contrastive learning.

7. The progressive fine-tuning method of a multi-modal pre-trained model according to claim 1, wherein, The cross-modal contrastive loss is constructed by an InfoNCE-based loss function, For a given visual feature, the corresponding text feature is the positive sample, and all other text features in a batch form a negative sample set, By maximizing the similarity between the anchor point and the positive sample and minimizing the similarity between the anchor point and all negative samples, a Softmax function with a temperature parameter is used to convert the similarity score into a probability distribution, thereby driving the alignment of features at the probability level; The intra-modal clustering loss makes the vector representations of all samples belonging to the same class in the shared semantic space close to each other after being transformed by the projection function, The Euclidean distance between all same-class sample pairs is penalized to promote the model to learn more discriminative single-modal feature representations, so that the expressions of the same concept form a compact cluster; The orthogonal regular term imposes a constraint from the perspective of linear algebra on the image weight matrix that realizes feature projection and the text weight matrix A matrix is called an orthogonal matrix when the column vectors of the matrix are unit vectors that are pairwise orthogonal, which satisfies where, is the identity matrix, is the matrix transpose symbol, By adding a regularization term to penalize the difference between the identity matrix and the two projection matrices, the two projection matrices are driven to gradually tend to orthogonal matrices during the training process, prompting the model to learn an orthogonal feature basis, achieving feature decoupling, improving the generalization ability and expression efficiency of the model.

8. A progressive fine-tuning system of a multi-modal pre-trained model, characterized in that, The progressive fine-tuning system comprises: a processor; a memory storing a computer program, when the computer program is executed by the processor, the progressive fine-tuning method of the multi-modal pre-training model in any one of claims 1-7 is realized.

Images