A knowledge coordination-based efficient fine-tuning method and system for VLP model parameters

Abstract

The application discloses a kind of based on knowledge coordination VLP model parameter efficient fine-tuning method, it is characterized in that, based on given image text pair and downstream task, select multiple network modules from the VLP model of pre-training completion, construct VLP basic model;Knowledge coordination network adapter is constructed with each network module in VLP basic model;Corresponding network adapter is inserted into each network module in VLP basic model, constitutes VLP extension model, when VLP extension model is trained and optimized model parameter, the parameter of original network module in VLP basic model is not adjusted, only the parameter of network adapter and subsequent regularization layer is adjusted.The application adopts network adapter and VLP basic model combination, obtains the VLP extension model satisfying the demand of different downstream tasks, can be plugged in and used, is suitable for the efficient fine-tuning of light-weight parameter of different VLP model.

Description

Technical Field

[0001] This invention belongs to the technical field of machine learning, and specifically relates to an efficient method and system for fine-tuning VLP model parameters based on knowledge coordination. Background Technology

[0002] Currently, the pre-training-fine-tuning paradigm for large models has achieved significant success in fields such as natural language processing (NLP), computer vision (CV), and multimodal computing. In this paradigm, the model is first pre-trained on large-scale data, and then fine-tuned through full parameter tuning to adapt to various downstream tasks. However, with the continuous increase in model size (e.g., GPT-3 has up to 175 bytes of parameters) and the increasing variety of downstream tasks, especially in multimodal scenarios, the traditional full parameter tuning method is becoming increasingly impractical, mainly due to the significantly increased computational and storage requirements it brings.

[0003] To address these challenges, researchers have actively explored alternative methods. For example, linear probe methods adjust a lightweight head for each task to reduce the scale of fine-tuning; methods such as Adapter and Prompt Learning have demonstrated significant generalization performance in NLP by introducing lightweight structures into the model's input and between modules. While these methods alleviate the computational and storage costs associated with model parameter fine-tuning to some extent, they typically only consider a single modality or a single downstream task, lacking support for single / cross-modal and diverse downstream tasks. Therefore, given the increasingly complex visual-language pre-trained models and diverse cross-modal downstream tasks, there is an urgent need to research a plug-and-play lightweight fine-tuning method applicable to different VLP models, making the adaptation of pre-trained models to specific downstream tasks more flexible and efficient. Summary of the Invention

[0004] This invention provides a plug-and-play, knowledge-coordinated method and system for efficient fine-tuning of VLP model parameters, applicable to different VLP models, to solve the technical problems existing in the prior art.

[0005] The technical solution adopted by this invention to solve the technical problems existing in the prior art is as follows:

[0006] A knowledge-coordinated method for efficient fine-tuning of VLP model parameters is proposed. Based on a given image-text pair and downstream task, multiple network modules are selected from a pre-trained VLP model to construct a VLP base model. A network adapter is constructed to coordinate with each network module in the VLP base model using knowledge. The corresponding network adapter is inserted into each network module in the VLP base model to form a VLP extension model. When training and optimizing the parameters of the VLP extension model, the parameters of the original network modules in the VLP base model are not adjusted; only the parameters of the network adapter and its subsequent regularization layer are adjusted.

[0007] Furthermore, the network adapter includes an input fully connected layer, an activation function layer, a routing policy layer, and multiple parallel output fully connected layers, all connected to the routing policy layer, connected in sequence.

[0008] Furthermore, let f be the knowledge coordination function for knowledge coordination between the network adapter and each network module in the VLP basic model. m (z m ), f m (z m The expression for ) is as follows:

[0009]

[0010] in,

[0011] In the formula:

[0012] m is the network adapter serial number;

[0013] N is the number of network adapters;

[0014] The weights of the downsampling layer are shared by all network adapters;

[0015] Let be the weights of the upsampling layer for the m-th network adapter;

[0016] z m This involves knowledge of the internal workings of the network module into which the m-th network adapter is inserted.

[0017] σ(·) is the ReLU activation function;

[0018] d m Input the dimension for the m-th network adapter;

[0019] r m Let m be the dimension of the intermediate hidden layer of the m-th network adapter;

[0020] Set r m <<d mTo ensure a lightweight design of the model, the downsampling layer of the network adapter uses parameter sharing to effectively reduce parameters.

[0021] Furthermore, the VLP basic model includes a single-modal encoder, a cross-modal encoder, and / or a cross-modal decoder.

[0022] Furthermore, the single-modal encoder includes a single-modal multi-head attention layer and a single-modal forward propagation network; the cross-modal encoder includes a cross-modal multi-head self-attention layer and a cross-modal forward propagation network; and the cross-modal decoder includes a decoder multi-head self-attention layer and a decoder forward propagation network.

[0023] Furthermore, a network adapter is inserted after both the unimodal multi-head self-attention layer and the unimodal forward propagation network. This network adapter is called the unimodal network adapter. Let the input of the unimodal network adapter be x. S , Input feature x S The range of the set to which it belongs; r s x is the hidden layer dimension of the single-modal network adapter; the output of the single-modal network adapter is: S +router(f s (x S ), where router(·) is a routing planning strategy based on single-modal knowledge; x S For single-modal internal features; f s (x S This refers to pre-set knowledge information from a single modality.

[0024] Furthermore, the unimodal encoder includes a visual encoder and a text encoder; the visual encoder includes a visual multi-head self-attention layer and a visual forward propagation network; the text encoder includes a text multi-head self-attention layer and a text forward propagation network.

[0025] Furthermore, network adapters are inserted after both the cross-modal multi-head self-attention layer and the cross-modal forward propagation network. The network adapters inserted after the cross-modal multi-head self-attention layer and the cross-modal forward propagation network are called cross-modal network adapters.

[0026] Let the input of the cross-modal network adapter be x. C x C =x C1 +x C2 +x C3 x C For cross-modal internal features, x C1 For cross-modal intra-visual features, x C2 For features of the corresponding text within a cross-modal context, x C3Features that correspond to multiple modes within a cross-modal context; Input feature x C The range of the set to which it belongs; r c For the hidden layer dimension of the cross-modal network adapter;

[0027] The output of the cross-modal network adapter is: x C +router(f1(x C1 ),f2(x C2 ),f3(x C3 );f t (t)), where router(·) is the routing planning strategy based on modal knowledge, f1(x C1 f2(x) represents pre-defined visual knowledge information. C2 f3(x) represents pre-defined knowledge information from the text. C3 f represents pre-defined knowledge information from multiple modalities. t (t) assigns weights to the corresponding visual, textual, and multimodal features.

[0028] Furthermore, make router(f1(x) C1 ),f2(x C2 ),f3(x C3 );f t (t))=w1·f1(x C1 )+w2·f2(x C2 )+w3·f3(x C3 ), where w1 is the weight assigned to the corresponding visual feature; w2 is the weight assigned to the corresponding text feature; and w3 is the weight assigned to the corresponding multimodal feature. The formulas for calculating w1, w2, and w3 are as follows:

[0029] Let x′ ci For x ci Features after transformation by the cross-modal network adapter; i = 1, 2, 3;

[0030]

[0031]

[0032] In the formula:

[0033] d c For the input dimension of the cross-modal network adapter;

[0034] τ is the temperature coefficient of the softmax function;

[0035] b1 represents the bias of the corresponding visual feature in the cross-modal network adapter routing planning strategy;

[0036] b2 represents the bias of the corresponding text feature in the cross-modal network adapter routing planning strategy;

[0037] b3 represents the bias corresponding to the multimodal feature in the cross-modal network adapter routing planning strategy.

[0038] The present invention also provides a knowledge-coordinated VLP model parameter high-efficiency fine-tuning system, including a memory and a processor. The memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the steps of the knowledge-coordinated VLP model parameter high-efficiency fine-tuning method as described above.

[0039] The advantages and positive effects of this invention are:

[0040] (1) This invention provides a plug-and-play, lightweight, parameter-efficient fine-tuning network adapter suitable for different visual language pre-trained models. It not only has significant research value in academics but also demonstrates excellent practicality in real-world applications. This invention combines the network adapter with a VLP base model to obtain an extended VLP model that meets the needs of different downstream tasks. This makes the adaptation of the pre-trained model to specific downstream tasks more flexible and efficient.

[0041] (2) This invention enables support for single / cross-modal and different downstream tasks, solving the technical problem that traditional methods often only consider a single modality or a single downstream task.

[0042] (3) This invention significantly reduces the number of model training parameters while maintaining performance, thereby reducing the computational and storage requirements of the visual language pre-trained model when adapting it to downstream tasks. This technological breakthrough provides a feasible approach for more efficient model adaptation and offers strong support for dealing with the increasing number of downstream tasks. Attached Figure Description

[0043] Figure 1 This is a structural diagram of a classic visual language pre-training model.

[0044] Figure 2 A schematic diagram of inserting a cross-modal encoder into a network adapter.

[0045] Figure 3 A schematic diagram of inserting a single-mode encoder into a network adapter.

[0046] Figure 4 This is a diagram of a network adapter structure.

[0047] Figure 5 This is a flowchart illustrating the efficient fine-tuning method for VLP model parameters based on knowledge coordination, as described in this invention. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0049] The Chinese translations of the following English words in this application are as follows:

[0050] VLP model: Visual Language Pre-trained Model.

[0051] BLIP pre-trained model: an abbreviation for Bootstrapping Language-Image Pre-training, is a guided language and image pre-training model used for unified visual language understanding and generation; BLIP (Bootstrapping Language-Image Pre-training) is a multimodal framework proposed by Salesforce in 2022, which unifies understanding and generation, introduces cross-modal encoders and decoders, and realizes cross-modal information flow.

[0052] Please see Figures 1 to 5 A knowledge-coordinated method for efficient fine-tuning of VLP model parameters is proposed. Based on a given image-text pair and downstream task, multiple network modules are selected from a pre-trained VLP model to construct a VLP base model. A network adapter is constructed to coordinate with each network module in the VLP base model using knowledge. The corresponding network adapter is inserted into each network module in the VLP base model to form a VLP extension model. When training and optimizing the parameters of the VLP extension model, the parameters of the original network modules in the VLP base model are not adjusted; only the parameters of the network adapter and its subsequent regularization layer are adjusted.

[0053] Preferably, the network adapter may include an input fully connected layer, an activation function layer, a routing policy layer, and multiple parallel output fully connected layers, all connected to the routing policy layer, connected in sequence.

[0054] Preferably, the knowledge coordination function for knowledge coordination between the network adapter and each network module in the VLP basic model can be set as f. m (z m ), f m (z m The expression for ) can be as follows:

[0055]

[0056] in,

[0057] In the formula:

[0058] m is the network adapter serial number;

[0059] N is the number of network adapters;

[0060] The weights of the downsampling layer are shared by all network adapters;

[0061] Let be the weights of the upsampling layer for the m-th network adapter;

[0062] z m This involves knowledge of the internal workings of the network module into which the m-th network adapter is inserted.

[0063] σ(·) is the ReLU activation function;

[0064] d m Input the dimension for the m-th network adapter;

[0065] r m Let m be the dimension of the intermediate hidden layer of the m-th network adapter;

[0066] r can be set m <<d m To ensure a lightweight design of the model; the downsampling layer of the network adapter can choose to share parameters to effectively reduce the number of parameters.

[0067] Preferably, the VLP base model may include a single-modal encoder, a cross-modal encoder, and / or a cross-modal decoder.

[0068] Preferably, the single-modal encoder may include a single-modal multi-head attention layer and a single-modal forward propagation network; the cross-modal encoder may include a cross-modal multi-head self-attention layer and a cross-modal forward propagation network; and the cross-modal decoder may include a decoder multi-head self-attention layer and a decoder forward propagation network.

[0069] Preferably, network adapters can be inserted after both the single-modal multi-head self-attention layer and the single-modal forward propagation network. These network adapters can be referred to as single-modal network adapters. The input to the single-modal network adapter can be set to x. S , Input feature x S The range of the set to which it belongs; r s Let x be the hidden layer dimension of the single-modal network adapter; the output of the single-modal network adapter can be: x S +router(f s (x S ), where router(·) is a routing planning strategy based on single-modal knowledge; x S For single-modal internal features; f s (x S This refers to pre-set knowledge information from a single modality.

[0070] Preferably, the unimodal encoder may include a visual encoder and a text encoder; the visual encoder may include a visual multi-head self-attention layer and a visual forward propagation network; the text encoder may include a text multi-head self-attention layer and a text forward propagation network.

[0071] Preferably, a network adapter can be inserted after both the cross-modal multi-head self-attention layer and the cross-modal forward propagation network. The network adapter inserted after the cross-modal multi-head self-attention layer and the cross-modal forward propagation network can be called a cross-modal network adapter.

[0072] The input of the cross-modal network adapter can be set to x. C x C =x C1 +x C2 +x C3 x C For cross-modal internal features, x C1 For cross-modal intra-visual features, x C2 For features of the corresponding text within a cross-modal context, x C3 Features that correspond to multiple modes within a cross-modal context; Input feature x C The range of the set to which it belongs; r c For the hidden layer dimension of the cross-modal network adapter;

[0073] The output of a cross-modal network adapter can be: x C +router(f1(x C1 ),f2(x C2 ),f3(x C3 );f t (t)), where router(·) is the routing planning strategy based on modal knowledge, f1(x C1 f2(x) represents pre-defined visual knowledge information. C2 f3(x) represents pre-defined knowledge information from the text. C3 ) represents pre-set knowledge information from multimodal sources, and ft(t) represents the weights assigned to the corresponding visual, textual, and multimodal features.

[0074] Preferably, router(f1(x)) can be configured to... C1 ),f2(x C2 ),f3(x C3 );f t (t))=w1·f1(x C1 )+w2·f2(x C2 )+w3·f3(x C3), where w1 is the weight assigned to the corresponding visual feature; w2 is the weight assigned to the corresponding text feature; w3 is the weight assigned to the corresponding multimodal feature; the calculation formulas for w1, w2, and w3 are as follows:

[0075] Let x′ ci For x ci Features after transformation by the cross-modal network adapter; i = 1, 2, 3;

[0076]

[0077]

[0078] In the formula:

[0079] d c For the input dimension of the cross-modal network adapter;

[0080] τ is the temperature coefficient of the softmax function;

[0081] b1 represents the bias of the corresponding visual feature in the cross-modal network adapter routing planning strategy;

[0082] b2 represents the bias of the corresponding text feature in the cross-modal network adapter routing planning strategy;

[0083] b3 represents the bias corresponding to the multimodal feature in the cross-modal network adapter routing planning strategy.

[0084] The present invention also provides an embodiment of a knowledge-coordinated VLP model parameter efficient fine-tuning system, the system comprising a memory and a processor, the memory being used to store a computer program; the processor being used to execute the computer program and, when executing the computer program, to implement the steps of the knowledge-coordinated VLP model parameter efficient fine-tuning method as described above.

[0085] The workflow and working principle of the present invention will be further described below with reference to a preferred embodiment:

[0086] A knowledge-coordinated method for efficient fine-tuning of VLP model parameters is proposed. Based on a given image-text pair and downstream task, multiple network modules are selected from a pre-trained VLP model to construct a VLP base model. A network adapter is constructed to coordinate with each network module in the VLP base model using knowledge. The corresponding network adapter is inserted into each network module in the VLP base model to form a VLP extension model. When training and optimizing the parameters of the VLP extension model, the parameters of the original network modules in the VLP base model are not adjusted; only the parameters of the network adapter and its subsequent regularization layer are adjusted.

[0087] This invention first constructs a network framework for the downstream task based on a given image-text pair and specific downstream task requirements, utilizing multiple network modules provided in existing Visual Language Pre-trained Models (VLP models), such as visual encoders, text encoders, cross-modal encoders, and decoders. Figure 1 As shown, a VLP base model is constructed; a network adapter for knowledge coordination is designed to enable knowledge coordination of different types of network modules in the visual language pre-trained model. The network adapter is inserted into different network modules in the VLP base model, including: visual multi-head self-attention layer, visual forward propagation network, text multi-head self-attention layer, text forward propagation network, cross-modal multi-head self-attention layer, and cross-modal forward propagation network; the original pre-training parameters in the network modules of the visual language pre-trained model from the prior art are fixed, so that the newly added lightweight network adapter and regularization layer can be fine-tuned. An optimization objective is constructed according to the downstream task, and the VLP extended model network is trained.

[0088] The following uses image and text retrieval as an example to introduce the specific implementation steps of a preferred embodiment of the efficient fine-tuning method for VLP model parameters based on knowledge coordination of the present invention:

[0089] Step 1: Select the BLIP pre-trained model as the base model for VLP and construct the backbone network for downstream tasks. Utilize the visual encoder, text encoder, and cross-modal encoder provided by the BLIP pre-trained model to build the network architecture for the image-text retrieval task.

[0090] Step 2: Insert the corresponding network adapter according to the different network module categories. In the image and text retrieval task, the network module categories include unimodal encoders and cross-modal encoders, both of which include multi-head attention layers and forward propagation networks.

[0091] Step 3: For the cross-modal multi-head attention layer and the forward propagation network, insert network adapters after them respectively. Insert the corresponding cross-modal network adapter after both the cross-modal multi-head self-attention layer and the cross-modal forward propagation network. Let the input of the cross-modal network adapter be x. C x C =x C1 +x C2 +x C3 x C For cross-modal internal features, x C1 For cross-modal intra-visual features, x C2 For features of the corresponding text within a cross-modal context, x C3 Features that correspond to multiple modes within a cross-modal context; Input feature x C The range of the set to which it belongs; r cFor the hidden layer dimension of the cross-modal network adapter;

[0092] The output of the cross-modal network adapter is: x C +router(f1(x C1 ),f2(x C2 ),f3(x C3 );f t (t)), where router(·) is the routing planning strategy based on modal knowledge, f1(x C1 f2(x) represents pre-defined visual knowledge information. C2 f3(x) represents pre-defined knowledge information from the text. C3 f represents pre-defined knowledge information from multiple modalities. t (t) assigns weights to the corresponding visual, textual, and multimodal features.

[0093] Specifically, router(f1(x) C1 ),f2(x C2 ),f3(x C3 );f t (t))=w1·f1(x C1 )+w2·f2(x C2 )+w3•f3(x C3 ), where w1 is the weight assigned to the corresponding visual feature; w2 is the weight assigned to the corresponding text feature; and w3 is the weight assigned to the corresponding multimodal feature. The formulas for calculating w1, w2, and w3 are as follows:

[0094] Let x′ ci For x ci Features after transformation by the cross-modal network adapter; i = 1, 2, 3;

[0095]

[0096]

[0097] In the formula:

[0098] d c d represents the input dimension of the cross-modal network adapter. c It can be set to 16 or 32;

[0099] This indicates the range of sets to which the weights of the cross-modal network adapter routing planning strategy belong;

[0100] This indicates the range of sets to which the biases of cross-modal network adapter routing planning strategies belong;

[0101] τ is the temperature coefficient of the softmax function;

[0102] b1 represents the bias of the corresponding visual feature in the cross-modal network adapter routing planning strategy;

[0103] b2 represents the bias of the corresponding text feature in the cross-modal network adapter routing planning strategy;

[0104] b3 represents the bias corresponding to the multimodal feature in the cross-modal network adapter routing planning strategy.

[0105] For the unimodal multi-head attention layer and the forward propagation network, network adapters are also inserted after them. A corresponding unimodal network adapter is inserted after both the unimodal multi-head self-attention layer and the unimodal forward propagation network. Let the input of the unimodal network adapter be x. S , Input feature x S The range of the set to which it belongs; r s x is the hidden layer dimension of the single-modal network adapter; the output of the single-modal network adapter is: S +router(f s (x S ), where router(·) is a routing planning strategy based on single-modal knowledge; x S For single-modal internal features; f s (x S This refers to pre-set knowledge information from a single modality.

[0106] At this point, only the knowledge within a single modality is selected as the information source, i.e. These are the weights of the downsampling layer in a single-modal network adapter; These are the weights of the upsampling layer in a single-mode network adapter.

[0107] Step 5: After constructing the entire network and inserting the network adapter, fix the original pre-training parameters in the network module from the existing visual language pre-training model, and only fine-tune the parameters of the network adapter and the regularization layer.

[0108] Step 6: Construct the loss function. In the image-text retrieval task, image-text contrast learning loss and image-text matching loss are used to train the entire network.

[0109] Step 7: Set an appropriate learning rate and start training the VLP extended model.

[0110] The aforementioned visual language pre-trained model (VLP model), input fully connected layer, activation function layer, routing strategy layer, unimodal encoder, cross-modal encoder, cross-modal decoder, unimodal multi-head attention layer, unimodal forward propagation network, cross-modal multi-head self-attention layer, cross-modal forward propagation network, decoder multi-head self-attention layer, decoder forward propagation network, visual encoder, text encoder, visual multi-head self-attention layer, visual forward propagation network, text multi-head self-attention layer, and text forward propagation network, etc., can all adopt applicable network models or network modules in the existing technology, or adopt network models or network modules in the technology and construct them using conventional technical means.

[0111] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments described above, and any obvious improvements, substitutions, or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A method for efficient fine-tuning of VLP model parameters based on knowledge coordination, characterized in that, Based on the given image-text pairs and downstream tasks, multiple network modules are selected from the pre-trained VLP model to construct the VLP base model; a network adapter is constructed to coordinate knowledge with each network module in the VLP base model. The corresponding network adapters are inserted into the network modules in the VLP base model to form the VLP extended model. When training and optimizing the model parameters of the VLP extended model, the parameters of the original network modules in the VLP base model are not adjusted, only the parameters of the network adapters and the subsequent regularization layers are adjusted. The network adapter includes an input fully connected layer, an activation function layer, a routing policy layer, and multiple parallel output fully connected layers, each connected to the routing policy layer, connected in sequence. Let the knowledge coordination function for knowledge coordination between the network adapter and the network modules in the VLP basic model be: , The expression is as follows: ； in, , m = 1, 2, 3, ..., N; In the formula: m is the network adapter serial number; N is the number of network adapters; The weights of the downsampling layer are shared by all network adapters; Let be the weights of the upsampling layer for the m-th network adapter; This involves knowledge of the internal workings of the network module into which the m-th network adapter is inserted. It is the ReLU activation function; Input the dimension for the m-th network adapter; Let m be the dimension of the intermediate hidden layer of the m-th network adapter; set up To ensure a lightweight design of the model; the downsampling layer of the network adapter uses parameter sharing to effectively reduce parameters; The VLP basic model includes a single-modal encoder, a cross-modal encoder, and / or a cross-modal decoder.

2. The efficient fine-tuning method for VLP model parameters based on knowledge coordination according to claim 1, characterized in that, A single-modal encoder includes a single-modal multi-head attention layer and a single-modal forward propagation network; a cross-modal encoder includes a cross-modal multi-head self-attention layer and a cross-modal forward propagation network; a cross-modal decoder includes a decoder multi-head self-attention layer and a decoder forward propagation network.

3. The efficient fine-tuning method for VLP model parameters based on knowledge coordination according to claim 2, characterized in that, A network adapter is inserted after both the unimodal multi-head self-attention layer and the unimodal forward propagation network. This network adapter is called a unimodal network adapter. Let the input of the unimodal network adapter be... , Input features The range of the set to which it belongs; For the hidden layer dimension of a single-mode network adapter; The output of the single-mode network adapter is: ,in, This is a routing planning strategy based on single-modal knowledge; This is a feature within a single mode; This is based on pre-defined knowledge information from a single modality.

4. The efficient fine-tuning method for VLP model parameters based on knowledge coordination according to claim 2, characterized in that, A single-modal encoder includes a visual encoder and a text encoder; the visual encoder includes a visual multi-head self-attention layer and a visual forward propagation network; the text encoder includes a text multi-head self-attention layer and a text forward propagation network.

5. The efficient fine-tuning method for VLP model parameters based on knowledge coordination according to claim 2, characterized in that, A network adapter is inserted after both the cross-modal multi-head self-attention layer and the cross-modal forward propagation network. The network adapter inserted after the cross-modal multi-head self-attention layer and the cross-modal forward propagation network is called a cross-modal network adapter. Let the input of the cross-modal network adapter be... , = + + , For cross-modal internal features, For cross-modal intra-visual features, Features of the corresponding text within a cross-modal context. Features that correspond to multiple modes within a cross-modal context; , Input features The range of the set to which it belongs; For the hidden layer dimension of the cross-modal network adapter; The output of the cross-modal network adapter is: ,in, For a route planning strategy based on modal knowledge, For pre-set visual knowledge information, For pre-defined knowledge information from the text, For pre-set knowledge information from multiple modalities, Assign weights to the corresponding visual, textual, and multimodal features.

6. The efficient fine-tuning method for VLP model parameters based on knowledge coordination according to claim 5, characterized in that, make ,in Assign weights to the corresponding visual features; Assign weights to the corresponding text features; Assign weights to the corresponding multimodal features; , , The calculation formula is as follows: set up for Features after transformation by the cross-modal network adapter; i =1,2,3; ， i =1,2,3； ， i =1,2,3； In the formula: For the input dimension of the cross-modal network adapter; The temperature coefficient of the softmax function; The bias of the corresponding visual features in the cross-modal network adapter routing planning strategy; The bias of the corresponding text features in the cross-modal network adapter routing planning strategy; This refers to the bias of the corresponding multimodal features in the routing planning strategy for cross-modal network adapters.

7. A knowledge-coordinated VLP model parameter fine-tuning system, comprising a memory and a processor, characterized in that, The memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the steps of the knowledge coordination-based VLP model parameter fine-tuning method as described in any one of claims 1 to 6.

Images