Training method and system of vision-text pre-training model during open category test based on double low rank adaptation

Abstract

The invention provides a training method and system for a vision-text pre-training model during open category testing based on double low-rank adaptation, and the method comprises the steps: embedding a double low-rank adaptation matrix in an upper sampling layer of a feedforward network in the vision-text pre-training model, namely, a base-class low-rank adaptation matrix and a new-class low-rank adaptation matrix, the low-rank adaptation technology utilizes a low-rank matrix decomposition method; training the model on base class data by using a base class low-rank adaptation matrix, and performing fine tuning through a cross entropy loss function; in the open class test stage, the trained base class low-rank adaptation model is used for predicting test data, and low-quality prediction data and potential new class data are screened out; gradually adapting to the data of the new category by adjusting the low-rank adaptation matrix of the new category; and completing a classification decision according to a joint prediction result of the base class and new class low-rank adaptation models. According to the method, the adaptability and accuracy of the model in an open category environment are improved, and classification tasks of known categories and classification tasks of unseen categories can be processed at the same time.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and natural language processing, and particularly relates to a training method and system for a vision-text pre-training model based on double low-rank adaptation during open category testing. Background Art

[0002] In the real world, the distribution of data is usually unknown, and due to diverse actual scenarios, machine learning models often encounter new category data outside the training set during testing. This makes the task of applying machine learning models to open category environments (i.e., the mixed appearance of seen and unseen categories) a challenging problem. Traditional machine learning methods usually rely on closed label sets, that is, the model is only trained and predicted on the categories encountered during the training phase, which has poor performance in the face of open categories.

[0003] In recent years, the vision-text pre-training model CLIP (Contrastive Language-Image Pretraining) has achieved cross-modal alignment through contrastive learning and demonstrated strong zero-shot inference capabilities. CLIP pairs images with text descriptions and uses a text prompt (such as "a picture of a [category]") input to the text encoder as a semantic query signal, rather than traditional closed one-hot labels. This method enables CLIP to recognize various different concepts in the open world and is no longer limited to predefined categories. Therefore, CLIP and similar vision-text pre-training models are considered powerful tools for solving the open category detection problem.

[0004] However, when migrating the CLIP model to downstream tasks, some parameter-efficient methods based on prompt fine-tuning have emerged. Although these methods effectively reduce the adjustment of model parameters and alleviate the problem of catastrophic forgetting to a certain extent, they are usually only trained and tuned on known category data. Although this method enhances the discriminative ability of the model for known category data, it is prone to overfitting of the model during the training phase, seriously affecting the discriminative ability of the model for unseen categories in the open environment. As a result, the prediction effect on unseen categories during inference is poor, and even worse than the original zero-shot inference performance of CLIP. Summary of the Invention

[0005] In view of the deficiencies of the prior art, the present invention provides a training method and system for a vision-text pre-trained model based on double low-rank adaptation during open-category testing. By introducing low-rank adaptation technology into the vision-text pre-trained model and performing online adjustment during testing, the pre-trained model can adapt to the knowledge of unseen categories in the test data. The present invention can effectively achieve a better performance balance between seen categories and unseen categories, thereby improving the adaptability and classification accuracy of the vision-text model in an open-category environment.

[0006] In order to achieve the above invention objectives, the technical solutions adopted by the present invention are as follows:

[0007] A training method for a vision-text pre-trained model based on double low-rank adaptation during open-category testing, comprising the following steps:

[0008] 1) Embed a double low-rank adaptation matrix in the upsampling W layer of the feed-forward network (FFN) of the vision-text pre-trained model, including a base-class low-rank adaptation matrix ΔW and a new-class low-rank adaptation matrix ΔW b , and the adaptation technology is parameterized in the form of low-rank matrix factorization as ΔW = B·A, where n and r << min(d1, d2);

[0009] 2) Use the base-class low-rank adaptation matrix ΔW b to train the model with base-class data, freeze other parameters in the model, and fine-tune the parameters of the model through the cross-entropy loss function to make the model fully adapt to the base-class data distribution provided during training;

[0010] 3) In the open-category testing phase, use the trained base-class low-rank adaptation model (including the base-class low-rank adaptation matrix ΔW b ) to make a preliminary prediction on the test data, and compare the prediction confidence S(x|c = b) with a preset threshold γ, and screen out the samples with confidence lower than γ as low-confidence data;

[0011] 4) Further use the model containing the base-class low-rank adaptation matrix as a base-class detector to screen out potential new-class data according to the maximum concept matching score;

[0012] 5) For the low-confidence data and potential new-class data screened out in steps 3) and 4), for the new-class low-rank adaptation matrix ΔW n ​Online adjustment is implemented. The initial setting of the new-class low-rank adaptation matrix is zero, making its initial state equivalent to zero-sample inference. Subsequently, through online learning of pseudo-labeled data, it gradually adapts to the knowledge of new classes;

[0013] 6) Finally, based on the combined prediction results of the base-class low-rank adaptation model and the new-class low-rank adaptation model, the classification decision in an open-class environment is completed.

[0014] Furthermore, the vision-text pre-training model adopts the CLIP architecture, including a vision encoder Φ V and a text encoder Φ T , and the text encoder generates text input by embedding the class name into a predefined prompt template (such as "a picture of [class]") The extracted text features are used to perform cosine similarity matching with the image features. In the upsampling W up layer in the feed-forward network FFN of the vision-text pre-training model, a double low-rank adaptation matrix is embedded, including a base-class low-rank adaptation matrix ΔW b and a new-class low-rank adaptation matrix ΔW n , and the vision-text pre-training model with base-class low-rank adaptation and new-class low-rank adaptation added is denoted as

[0015] Furthermore, the training of the base-class low-rank adaptation model adopts the cross-entropy loss function, and its calculation formula is:

[0016]

[0017] where, K b is the total number of base-class categories, is the indicator function, K b represents the number of base-class categories, k represents the index of the current category, represents the probability that the sample is predicted as the k-th category, and x b represents the input base-class sample.

[0018] Furthermore, during the test training phase, the model containing the base-class low-rank adaptation matrix is used for preliminary prediction, and samples with a confidence level S(x|c = b) lower than γ are regarded as low-confidence data

[0019] Furthermore, the Maximum Concept Matching (MCM) strategy is adopted for detecting potential new-class data , and its calculation formula is:

[0020]

[0021] Among them, S MCM(x) When it is less than the set threshold λ, the data is determined to be a new category, and exp(.) is the exponential function. represents the text feature of the k-th category. represents the visual feature of the input image x, and τ represents the temperature coefficient.

[0022] Furthermore, the new class low-rank adaptation matrix ΔW n adopts an online update strategy in the test stage. Its initial value is set to zero. By detecting the low-confidence data and the new class data of the pseudo-label data for online learning, and continuously updating using the gradient descent method, gradually realizing the adaptation to the data distribution of the new category. The new class low-rank adaptation model The online update objective includes the joint optimization of the following loss functions:

[0023] a) The cross-entropy loss adopted for the pseudo-label data

[0024] b) The consistency distillation loss used to constrain the consistency between the base class and the new class text features

[0025] c) The L2 regularization loss imposed on the parameters of the new class low-rank adaptation matrix

[0026] Its total optimization objective is:

[0027]

[0028] Among them, β is a regularization coefficient adaptively adjusted according to the quality of the pseudo-label. Since the quality of the pseudo-label affects the training during testing of the new class low-rank adaptation matrix, β can be adaptively adjusted by the proportion of high-quality pseudo-labels. If the proportion of evaluated high-quality pseudo-labels is large, it indicates that the pseudo-labels are relatively reliable, then the regularization intensity can be reduced to allow the model to fully utilize this part of the pseudo-labels for training; if the proportion of high-quality pseudo-labels is small, it indicates that the credibility of the pseudo-labels is low, then increase the regularization coefficient to limit the model's dependence on this part of the pseudo-labels and alleviate the accumulation of incorrect knowledge. Accordingly, the evaluation of high-quality pseudo-labels is determined by the consistency of the two low-rank adaptation prediction results, specifically expressed as,

[0029]

[0030] Among them, β ∈ [β0, 10β0], and β0 is fixed at 0.05 during testing.

[0031] Furthermore, the consistency distillation loss L ConsistencyTo minimize the Euclidean distance between the output text features of the base-class low-rank adaptation matrix and the new-class low-rank adaptation matrix, the calculation formula is as follows:

[0032]

[0033] where, are the text features corresponding to the base-class low-rank adaptation matrix and the new-class low-rank adaptation matrix respectively, and K is the total number of categories.

[0034] Furthermore, the final classification decision in the test stage is jointly determined based on the prediction results of the base-class low-rank adaptation and the new-class low-rank adaptation, and its decision rule is:

[0035]

[0036] where, is the predicted category of the base-class low-rank adaptation matrix, is the predicted category of the new-class low-rank adaptation matrix. S(x|c = b) represents the maximum prediction probability of the model containing the base-class low-rank adaptation matrix for the sample x.

[0037] Furthermore, the method is applicable to open-category visual recognition tasks, such as biomedical image classification visual recognition and other fields.

[0038] The present invention also discloses a training system for a visual-text pre-training model based on dual low-rank adaptation during open-category testing, including:

[0039] A visual encoder module, configured to receive and process image inputs and generate image feature representations;

[0040] A text encoder module, configured to receive and process text inputs and generate text feature representations;

[0041] The base-class low-rank adaptation module includes the base-class low-rank adaptation matrix inserted at the upsampling W up layer of each FFN. By adjusting the weights of the base-class low-rank adaptation matrix in the visual encoder and the text encoder, the visual-text pre-training model can be adapted to the base-class data, perform training on the base-class data, and freeze the weights of the visual text encoder during the training process;

[0042] The new-class low-rank adaptation module includes the new-class low-rank adaptation matrix inserted at the upsampling W up layer of each FFN. During the test stage, the new-class low-rank adaptation matrix is adjusted online according to the data screened by the prediction, and its weights are updated to achieve the adaptation of the model to unseen categories;

[0043] A low-confidence data detection module, configured to initially predict the test data and screen out the low-confidence data according to the prediction confidence And use the low-confidence data for training processing during subsequent tests.

[0044] A base class detection module, which is used to classify the input data according to the maximum concept matching (MCM) method, determine whether it is new category data, and filter the data with MCM lower than the set threshold as new category data For subsequent processing;

[0045] A decision-making module, which makes a final classification decision according to the prediction results of the base class low-rank adaptation model and the new class low-rank adaptation model, and outputs the prediction results of the model in an open category environment.

[0046] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it realizes the training of the above-mentioned vision-text pre-training model during open category testing.

[0047] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it realizes the training of the above-mentioned vision-text pre-training model during open category testing.

[0048] Compared with the prior art, the advantages of the present invention are as follows:

[0049] 1. Improve the classification performance in an open category environment: By introducing a double low-rank adaptation matrix, which is responsible for adapting to the base class data during training and the unseen new class data during testing respectively, an adaptive adjustment of open category data is realized in the vision-text pre-training model, thereby effectively improving the balance performance of the model between the seen categories and the unseen categories. Especially in an open category environment, the model can better identify and adapt to unknown categories, avoiding the performance degradation caused by out-of-training-set data in traditional models.

[0050] 2. Reduce the interference to the original pre-training knowledge of the model: By using the low-rank adaptation technology for parameter fine-tuning, only the key parts of the model are adjusted, avoiding excessive modification of the entire model parameters. This enables the model to maintain the original pre-training knowledge, reduces the occurrence of catastrophic forgetting, and thus improves the generalization ability of the model.

[0051] 3. Efficient online adaptation ability: The new class low-rank adaptation matrix proposed by the present invention is adjusted online during testing, enabling the model to dynamically adjust according to the characteristics of the input data and gradually adapt to the distribution of new category data. This method improves the prediction ability of the model for unseen categories in the open world, avoiding the performance bottleneck caused by the inability of the static training process to adapt to unknown data.

[0052] 4. Reduce the risk of overfitting during the training phase: By independently optimizing the base-class low-rank adaptation matrix and the new-class low-rank adaptation matrix respectively, and combining the joint prediction of the base class and the new class, the overfitting problem during training can be effectively alleviated, ensuring the robustness and reliability of the model in a real open-class environment.

[0053] 5. Improve the applicability and flexibility of the multimodal model: By introducing the vision-text pre-training model CLIP and combining low-rank adaptation techniques for efficient adjustment, the present invention gives full play to the advantages of CLIP in cross-modal (image and text) alignment, improving the adaptability and flexibility of the model in various practical tasks, especially in complex open-world tasks.

[0054] 6. Efficient computational and storage overhead: By using the low-rank adaptation technique for fine-tuning, compared with the traditional full-model fine-tuning method, the computational and storage overhead can be significantly reduced, making this method more suitable for large-scale data processing and real-time application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is the flowchart of the training method of the vision-text pre-training model based on double low-rank adaptation in an open-class environment during testing. DETAILED DESCRIPTION OF THE INVENTION

[0056] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the following further describes the present invention in detail with reference to the drawings and by way of examples.

[0057] A training method for a vision-text pre-training model based on double low-rank adaptation during open-class testing, characterized by comprising the following steps:

[0058] 1) Embed double low-rank adaptation matrices in the upsampling W layer of the feed-forward network (FFN) of the vision-text pre-training model, including the base-class low-rank adaptation matrix ΔW up and the new-class low-rank adaptation matrix ΔW b The adaptation technique is parameterized in the form of low-rank matrix factorization as ΔW = B·A, where n and r << min(d1, d2);

[0059] 2) Use the base-class low-rank adaptation matrix ΔW b to train the model with base-class data, freeze other parameters in the model, and fine-tune the parameters of the model through the cross-entropy loss function to make the model fully adapt to the base-class data distribution provided during training;

[0060] 3) In the open-class testing phase, use the trained base-class low-rank adaptation model (including the base-class low-rank adaptation matrix ΔW b ) to make a preliminary prediction on the test data, and compare the prediction confidence S(x|c = b) with the preset threshold γ, and screen out the samples with confidence lower than γ as low-confidence data

[0061] 4) Further use the model containing the base-class low-rank adaptation matrix as the base-class detector, and screen out potential new-class data according to the maximum concept matching score

[0062] 5) For the low-confidence data and potential new-class data screened out in steps 3) and 4), perform online adjustment on the new-class low-rank adaptation matrix ΔW n The new-class low-rank adaptation matrix is initially set to zero, making its initial state equivalent to zero-sample inference. Subsequently, through online learning of the pseudo-label data, gradually adapt to the knowledge of the new class;

[0063] 6) Finally, based on the joint prediction results of the base-class low-rank adaptation model and the new-class low-rank adaptation model, complete the classification decision in the open-class environment.

[0064] In the open-world environment, base-class (known class) data is usually provided during training where y b is the label class and During open-environment testing, new-class (unseen class) data and base-class data appear as test data at the same time, where and We use the vision-text pre-training model {Φ V , Φ T} to adapt to this open-class scenario, expecting the model to achieve a better balance between the known classes and the unseen classes during testing. Among them, the vision-text pre-training model consists of a vision encoder Φ V and a text encoder Φ T . When using the CLIP model for inference, assume that all classes are filled into a manually designed prompt template (such as "a picture of [class]") to obtain the input text After being encoded by Φ T to obtain the text feature as the classifier weights for K candidate classes. Given an image and its corresponding label Extract the visual feature of the image After that, the probability of classifying the image into the k-th category can be calculated as

[0065]

[0066] where cos(·,·) represents the calculation of cosine similarity, and τ is the temperature coefficient fixed by the CLIP pre-trained model.

[0067] To enhance the adaptability of the vision-language pre-trained model in the open-world scenario and minimize the impact on the knowledge learned in the vision-language pre-trained model, first, the method of adding Low rank of adaptation (LoRA) to the Feed-Forward Network (FFN) layer is used to fully adjust the pre-trained encoders of the vision and language modalities.

[0068] Since the FFN layer in the Transformer model is a non-linear transformation layer, it can be expressed as

[0069] FFN(f) = GELU(fW up +b up )W down +b down (2)

[0070] where is the input feature, is the upsampling layer, responsible for projecting the feature into a high-dimensional space (usually d out >d in ), and after passing through the activation function GELU(·), it is then mapped back to the original dimension through , and b up and b down are its bias terms respectively. Through the FFN layer, the model can learn more complex feature representations, thus helping the model capture more information. This process is similar to the look-up table mechanism of features, where the input feature f is used to look up the mapping of the knowledge learned by the model. That is to say, the input feature f is used as a query, and W up and W down of the FFN layer are used as keywords and query results respectively to query the knowledge learned by the model using the input feature. Therefore, to reduce the impact on the parameters of the vision-language pre-trained model, we use the low-rank adaptation matrix as a bypass of W up in the FFN and freeze W up . By adjusting the low-rank adaptation matrix ΔW, W up and ΔW span a new subspace, thereby realizing the adjustment of the upsampling layer and making the W upAdapt to the characteristics of the input data and map these characteristics to the pre-trained knowledge learned by the visual-text model. For each FFN layer W up Layers have,

[0071] f out =W up f in +αΔWf in =W up f in +αBAf in (3)

[0072] Where α is a scaling factor. Before training, A is Kaiming initialized and B is zero initialized, so that ΔW is initially 0, so that the visual-text pre-training model starts training from a zero-sample inference state. Therefore, adding a low-rank adaptation matrix to the visual-text pre-training model can achieve full adaptation to the input data by adjusting the bimodal encoder, and has little impact on the original pre-training knowledge.

[0073] During training, in each FFN layer of the vision-text pre-trained model, W up Introduce a base class low-rank adaptation matrix ΔW b As an adjustable bypass, the model can be fully adapted to the base class data. The optimization loss of the training process is as follows,

[0074]

[0075] in, represents the indicative function, A one hot vector representing the predictions.

[0076] Furthermore, in order to reduce the mutual interference between the knowledge learned from the base class and the knowledge from the new class, during the test, each layer W of the bimodal encoder is up Add a new class of low-rank adaptation matrix ΔW n and ΔW b In parallel. At this time, ΔW b The base class data information has been learned, so it is frozen, and ΔW n As the test data is input, it is predicted and adjusted, and gradually adapted to the new category data during the test. Based on this, the visual-text pre-training model with added base class low-rank adaptation and new class low-rank adaptation can be expressed as

[0077] During testing, since the input data category is uncertain, a model with a base class low-rank adaptation matrix is ​​first used. Perform a preliminary prediction on the incoming data, filter out the data whose maximum prediction probability S(x|c = b) is lower than the set threshold γ, and send the filtered data into the new class low-rank adaptation matrix for further prediction. Initially, since the weight of the new class low-rank adaptation matrix is 0, it is equivalent to directly performing zero-shot inference using the vision-text pre-trained model. Therefore, the prediction process of the vision-text pre-trained model during testing is as follows:

[0078]

[0079]

[0080]

[0081] To enable the new class low-rank adaptation matrix to fully utilize the test data knowledge during testing, the new class low-rank adaptation matrix is set to be adjusted and predicted during testing. Specifically, further use as the base class detector, and the newly detected class data is sent to for training during testing. The detection criterion will follow the commonly used detection metrics in the out-of-distribution detection method of the multi-modal pre-trained model, that is, Maximum concept matching (MCM), and the output result of is used as the detection evaluation of out-of-distribution (new class) data. The MCM score is calculated as follows:

[0082]

[0083] Then, the base class detector can be implemented through the following definition:

[0084]

[0085] where it is set that D(x) = 1 represents the base class, D(x) = 0 represents the new class, and λ is the discrimination threshold. According to the experience of predecessors in the out-of-distribution detection method of multi-modal models, λ is set to 0.95. Accordingly, Table 1 shows the results of using the vision-text pre-trained model trained on different base class datasets for new class detection, and also shows the approximate False Positive Ratio (FPR) results at this threshold. It can be seen that on most datasets, using the trained model as the base class detector, there are some false positives in the detected new classes, that is, there is some base class data mixed in. These false positive data are somewhat similar to the new class data and have certain value for the training of the new class low-rank adaptation matrix. Therefore, the detected new class data and the filtered low-confidence data are used together in the testing During the optimized adjustment process. When the number of these filtered data is not zero The specific optimization loss during training for testing is as follows

[0086]

[0087] Among them is the one-hot pseudo-label vector, and the selection of the pseudo-label is determined by the maximum predicted probability output

[0088] Table 1 Results of visual-text pre-training with a base-class low-rank adaptation matrix for detecting new classes using MCM. Among them, FPR is the estimated value when λ = 0.95

[0089]

[0090]

[0091] However, adapting to these detected data in the long term may lead to the accumulation of incorrect knowledge, seriously damaging the original zero-shot inference performance of the model. To alleviate the accumulation of incorrect knowledge, a consistency distillation loss is added to minimize the square of the Euclidean distance between the text features of the base-class output and the text features of the new-class output, restricting the differences between the text knowledge in n To alleviate catastrophic forgetting, calculate the L2-norm regularization loss for the ΔW weights of layer L to penalize the complexity of the new-class low-rank adaptation matrix. These two regularization losses are expressed as follows

[0092]

[0093] The total optimization loss function of the new-class low-rank adaptation matrix during testing is expressed as

[0094]

[0095] where β is the penalty coefficient, controlling the intensity of these two regularization losses

[0096] Since the quality of the pseudo-labels affects the training during testing of the new-class low-rank adaptation matrix, β can be adaptively adjusted by the proportion of high-quality pseudo-labels. If the proportion of high-quality pseudo-labels evaluated is large, indicating that the pseudo-labels are relatively reliable, the regularization strength can be reduced to allow the model to fully utilize these pseudo-labels for training; if the proportion of high-quality pseudo-labels is small, indicating that the credibility of the pseudo-labels is low, the regularization coefficient is increased to limit the model's dependence on these pseudo-labels and mitigate the accumulation of incorrect knowledge. Accordingly, the evaluation of high-quality pseudo-labels is determined by the consistency of the two low-rank adaptation prediction results, specifically expressed as,

[0097]

[0098] where β ∈ [β0, 10β0], and in the test, β0 is fixed at 0.05.

[0099] The training process during testing of the entire visual-text pre-training model based on dual low-rank adaptation in the open-category world is as Figure 1 shown.

[0100] In another embodiment of the present invention, a training system for a visual-text pre-training model based on dual low-rank adaptation during testing in an open-category environment is provided. This system can be used to implement the training method for the visual-text pre-training model during open-category testing. Specifically, it includes:

[0101] A visual encoder module for receiving and processing image inputs and generating image feature representations;

[0102] A text encoder module for receiving and processing text inputs and generating text feature representations;

[0103] A base-class low-rank adaptation module including base-class low-rank adaptation matrices inserted at the upsampling W up layer of each FFN. By adjusting the weights of the base-class low-rank adaptation matrices in the visual encoder and the text encoder, the visual-text pre-training model can adapt to the base-class data, perform training on the base-class data, and freeze the weights of the visual-text encoder during the training process;

[0104] A new-class low-rank adaptation module including new-class low-rank adaptation matrices inserted at the upsampling W up layer of each FFN. During the test phase, the new-class low-rank adaptation matrices are adjusted online according to the data screened by the prediction to update their weights to achieve the adaptation of the model to unseen categories;

[0105] A low-confidence data detection module for preliminarily predicting test data, screening out low-confidence data according to the prediction confidence and using the low-confidence data for subsequent training during testing.

[0106] A base class detection module, which is used to classify input data according to the maximum concept matching (MCM) method, determine whether it is new category data, and filter the data with an MCM lower than a set threshold as new category data for subsequent processing;

[0107] A decision-making module, which makes a final classification decision based on the prediction results of the base class low-rank adaptation model and the new class low-rank adaptation model, and outputs the prediction results of the model in an open category environment.

[0108] In another embodiment of the present invention, a terminal device is provided. The terminal device includes a processor and a memory. The memory is used to store a computer program. The computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the training method for open category testing of the vision-text pre-training model based on dual low-rank adaptation.

[0109] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in the terminal device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, and the operating system of the terminal is stored in this storage space. And, one or more instructions suitable for being loaded and executed by the processor are also stored in this storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.

[0110] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the training method during the open-class testing of the visual-text pre-training model in the above embodiments; the one or more instructions in the computer-readable storage medium are loaded and executed by the processor.

[0111] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0112] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0113] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0115] Those of ordinary skill in the art will realize that the embodiments described herein are provided to assist the reader in understanding the implementation methods of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present invention based on these technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the present invention.

Claims

1. A training method for a visual-text pre-trained model based on dual low-rank adaptation in an open category test, characterized in that: The following steps are involved: 1) Upsampling W in the Feedforward Network FFN of the Vision-Text Pre-training Model up The layer embeds a dual low-rank adaptation matrix, which includes the base class low-rank adaptation matrix ΔW b And the new class of low-rank adaptation matrix ΔW n , the adaptation technique adopts low-rank matrix factorization and is parameterized as ΔW = B·A, where and And r< <min(d1,d2); 2) Using the base class low-rank adaptation matrix ΔW b Train the model on base class data, freeze other parameters in the model, and use the cross entropy loss function Fine-tune the model parameters to make the model fully adapt to the base class data distribution provided during training; 3) In the open category test phase, the trained low-rank adaptation matrix ΔW containing the base class is used b The base class adaptation model makes a preliminary prediction on the test data, and compares the prediction confidence S(x|c=b) with the preset threshold γ, and screens the samples with confidence lower than γ as low confidence data; 4) Further using the model containing the base class low-rank adaptation matrix as a base class detector to screen out potential new class data according to the maximum concept matching score; 5) For the low confidence data screened out in steps 3) and 4) and potential new types of data For the new class low-rank adaptation matrix ΔW n Implement online adjustment, where the low-rank adaptation matrix of the new class is initially set to zero, making its initial state equivalent to zero-sample reasoning, and then gradually adapt to the knowledge of the new class through online learning of pseudo-labeled data; 6) Finally, the classification decision in the open category environment is completed based on the joint prediction results of the base class low-rank adaptation matrix and the new class low-rank adaptation matrix.

2. The training method according to claim 1, characterized in that: The visual-text pre-training model adopts the CLIP architecture, including a visual encoder Φ V and text encoder Φ T , the text encoder generates text input by embedding the category name into a predefined prompt template Extracted text features Used for cosine similarity matching with image features. Upsampling W in the feedforward network FFN of the vision-text pre-training model up The layer embeds a dual low-rank adaptation matrix, which includes the base class low-rank adaptation matrix ΔW b And the new class of low-rank adaptation matrix ΔW n , the visual-text pre-trained model with added base class low-rank adaptation and new class low-rank adaptation is represented as 3. The training method according to claim 2, characterized in that: The base class low-rank adaptation model The training adopts the cross entropy loss function, which is calculated as follows: Among them, K b is the total number of base class categories, is the indicative function, K b represents the number of categories of the base class, k represents the index of the current category, represents the probability that the sample is predicted to be the kth category, x b Represents the base class sample of the input.

4. The training method according to claim 1, characterized in that: During the test training phase, a model containing a base class low-rank adaptation matrix is ​​used for preliminary prediction, and samples with confidence S(x|c=b) lower than γ are regarded as low-confidence data.

5. According to the training method of claim 1, the maximum concept matching (MCM) strategy is used to train potential new class data Detection, the calculation formula is: in, S MCM(x) When it is less than the set threshold λ, the data is judged as a new category, and exp(.) is an exponential function. represents the text feature of the kth category, represents the visual features of the input image x, and τ represents the temperature coefficient.

6. The training method according to claim 5, characterized in that: The new class of adapters ΔW n In the test phase, an online update strategy is adopted, with its initial value set to zero. Through online learning of pseudo-label data from test data, the gradient descent method is used to continuously update and gradually achieve adaptation to the distribution of new category data. New low-rank adaptive model The online update objective of consists of the joint optimization of the following loss functions: a) Cross entropy loss for pseudo-labeled data b) Consistency distillation loss used to constrain the consistency of base class and new class text features c) L2 regularization loss applied to the parameters of the new class low-rank adaptation matrix The overall optimization goal is: Among them, β is the regularization coefficient that is adaptively adjusted according to the quality of the pseudo-label. The evaluation of high-quality pseudo-labels is determined by the consistency of the two low-rank adaptive prediction results, which is specifically expressed as, where β∈[β0,10β0] and β0 is fixed to 0.05 for the test. The consistency distillation loss L Consistency The Euclidean distance between the text features output by the base class low-rank adaptation matrix and the new class low-rank adaptation matrix is ​​calculated as follows: in, and are the text features corresponding to the base class low-rank adaptation matrix and the new class low-rank adaptation matrix, and K is the total number of categories.

7. The training method according to claim 1, characterized in that: The final classification decision in the test phase is determined jointly based on the prediction results of the base class low-rank adaptive model and the new class low-rank adaptive model, and the decision rule is: in, is the predicted category of the base class low-rank adaptation model, is the predicted category of the new class low-rank adaptation model, and S(x|c=b) represents the maximum predicted probability of the model containing the base class low-rank adaptation matrix for sample x.

8. The training method according to any one of claims 1 to 7, characterized in that: The method of training the visual-textual pre-trained model when tested in an open category environment is suitable for open category visual recognition tasks.

9. A system for training a visual-text pre-trained model based on dual low-rank adaptation when tested in an open category environment, the system being capable of implementing the training method of one of claims 1 to 7, comprising: The visual encoder module is used to receive and process image input and generate image feature representation; The text encoder module is used to receive and process text input and generate text feature representation; Base low-rank adaptation module, including upsampling W of each FFN up The base class low-rank adaptation matrix inserted in the layer adjusts the weights of the base class low-rank adaptation matrix in the visual encoder and the text encoder, so that the visual-text pre-trained model can adapt to the base class data, perform training on the base class data, and freeze the weights of the visual-text encoder during the training process; New class of low-rank adaptation modules, including upsampling W for each FFN up The low-rank adaptation matrix of the new class inserted in the layer is adjusted online according to the predicted and filtered data in the test phase, and its weight is updated to achieve the adaptation of the model to unseen categories; Low confidence data detection module, used to preliminarily predict test data and filter out low confidence data based on prediction confidence And use low confidence data for training processing during subsequent testing. The base class detection module is used to classify the input data according to the maximum concept matching (MCM) method, determine whether it is new category data, and filter the data with MCM below the set threshold as new category data. Carry out subsequent processing; The decision module makes the final classification decision based on the prediction results of the base class low-rank adaptation matrix and the new class low-rank adaptation matrix, and outputs the prediction results of the model in an open category environment.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the training method described in one of claims 1 to 7 is implemented.

Images