Visual adapter based on complex linear projection optimization and adaptation method thereof

Abstract

The application relates to a visual adapter based on complex linear projection optimization and an adaptation method thereof, wherein the visual adapter comprises a multi-branch low-rank projection module used for constructing up and down projection matrices through linear combination of multiple low-rank matrices; a complex sharing strategy module used for realizing that the up and down projection matrices of the same branch share a core matrix and all branches share multiple shared sub-matrices based on preset core sharing and branch sharing mechanisms; a parameter space orthogonal optimization module used for constraining the orthogonality of the shared sub-matrices through a preset orthogonal loss function so that a parameter update direction is aligned with an ideal gradient direction; and an initialization module used for singular value decomposition of an initial projection matrix to initialize the shared sub-matrices. Thus, the problems that in the prior art, due to excessively high full-parameter fine-tuning cost, a delta-tuning method is inefficient in visual task adaptation, a low-rank adapter has convergence difficulty, and it is difficult to balance parameter efficiency and adaptation performance and the like are solved.

Description

Technical Field

[0001] This application relates to the fields of computer vision, deep learning and visual basic model deployment technology, and in particular to a visual adapter based on complex linear projection optimization and its adaptation method. Background Technology

[0002] Currently, visual foundational models, through large-scale data pre-training, possess extremely strong feature extraction and generalization capabilities, and are widely used in various fields such as image recognition and object detection. However, the feature distribution during the pre-training stage often differs significantly from the feature distribution of downstream deployment tasks, leading to a decline in model performance. How to efficiently adapt large models to downstream tasks, enabling them to quickly adapt to new domains while maintaining strong generalization capabilities, is a key technical challenge for achieving practical model deployment.

[0003] Among the related technologies, full parameter fine-tuning, delta-tuning methods (such as cue learning, adapter adjustment, etc.) and low-rank adapters are mainly used to adapt the pre-trained model to the characteristics of downstream tasks by introducing trainable parameters or structural adjustments.

[0004] However, in related technologies, the full fine-tuning method requires adjusting all parameters of the model, resulting in excessively high computational and storage costs, making it difficult to meet the lightweight requirements of large-scale deployment. The delta-tuning method has insufficient adaptation efficiency and limited performance in visual tasks, and the low-rank adapter has gradient entanglement problems, making it difficult to converge during model training. This makes it difficult to balance parameter efficiency and adaptation performance, and there is an urgent need for improvement. Summary of the Invention

[0005] This application provides a visual adapter and its adaptation method based on complex linear projection optimization, in order to solve the problems in related technologies, such as the high cost of full parameter fine-tuning, the insufficient adaptation efficiency of delta-tuning methods in visual tasks, and the convergence difficulty of low-rank adapters, which make it difficult to balance parameter efficiency and adaptation performance.

[0006] The first aspect of this application provides a visual adapter based on complex linear projection optimization, comprising: a multi-branch low-rank projection module for constructing upper and lower projection matrices by linearly combining multiple low-rank matrices, wherein each low-rank matrix is ​​composed of the product of three sub-matrices, and the upper and lower projection matrices are obtained by summing the products, and the rank of the low-rank matrix is ​​less than the input and output dimensions of the upper and lower projection matrices; a complex sharing strategy module for realizing that the upper and lower projection matrices of the same branch share a kernel matrix based on a preset kernel sharing and branch sharing mechanism, and all branches share multiple shared sub-matrices, wherein the kernel matrix is ​​the trainable parameter corresponding to the branch; a parameter space orthogonal optimization module for constraining the orthogonality of the shared sub-matrices by a preset orthogonal loss function so that the parameter update direction is aligned with the ideal gradient direction; and an initialization module for performing singular value decomposition on the initial projection matrix to initialize the shared sub-matrices.

[0007] Through the above-mentioned technical means, the embodiments of this application can design a multi-branch low-rank adapter architecture, combining a complex sharing strategy of kernel sharing and branch sharing. At the same time, based on matrix theory, the gradient entanglement problem of low-rank composite matrices is analyzed, an orthogonal loss function is introduced, and the parameter initialization strategy is optimized to improve the model convergence efficiency. Thus, excellent adaptation effect is achieved with a small number of new parameters. While ensuring superior performance and fast convergence, the deployment cost of the visual basic model is reduced, providing a brand-new solution for the efficient deployment of the visual basic model.

[0008] Optionally, in one embodiment of this application, the expression for the upper and lower projection matrix is: , , in, Let be the upward projection matrix. The upper projection matrix is ​​a low-rank matrix that retains the size of the first dimension feature. For the first The kernel matrix of the branch, The upper projection matrix is ​​a low-rank matrix that retains the second dimension of the feature size. Let be the lower projection matrix. The lower projection matrix is ​​a low-rank matrix that retains the size of the first dimension feature. The lower projection matrix is ​​a low-rank matrix that retains the second dimension of the feature size. This is the matrix transpose operation.

[0009] Through the above technical means, the embodiments of this application can be based on a dual sharing mechanism, where the upper and lower projections of the same branch share the kernel matrix to ensure the continuity of feature transformation. All branches share two general sub-matrices, and only the kernel matrix is ​​a branch-specific parameter, which takes into account both parameter efficiency and generalization ability.

[0010] A second aspect of this application provides a method for adapting a visual adapter, comprising the following steps: inserting the visual adapter into a visual base model to initialize the parameters of the visual adapter, and performing gradient descent training using a weighted total loss function to optimize the parameters of the visual adapter and obtain a multi-branch low-rank structure; converting the multi-branch low-rank structure into a standard linear layer to generate adaptation results based on the visual task.

[0011] Through the above-mentioned technical means, the embodiments of this application can initialize the model and parameters, use a weighted total loss function for gradient descent training, optimize the adapter parameters, convert the trained multi-branch low-rank structure into a standard linear layer, and complete inference in combination with the base model. This significantly reduces the cost of fine-tuning and deployment without sacrificing performance, improves deployment convenience, and is applicable to various visual tasks such as object detection, semantic segmentation, image classification, and remote sensing rotating object detection. It can be widely used in scenarios that require efficient deployment of visual base models, such as autonomous driving, medical image analysis, geographic information systems, and urban planning.

[0012] Optionally, in one embodiment of this application, the parameter initialization of the visual adapter includes: generating an initial projection matrix; performing singular value decomposition on the initial projection matrix to obtain a left singular vector matrix, a singular value matrix, and a right singular vector matrix; initializing the lower projection matrix and the upper projection matrix respectively, and freezing the backbone parameters of the basic model.

[0013] Through the above-mentioned technical means, the embodiments of this application initialize the adapter matrix by singular value decomposition, so that the shared submatrices maintain orthogonality in the early stage of training, laying the foundation for subsequent optimization and further accelerating the convergence speed.

[0014] Optionally, in one embodiment of this application, the weighted total loss function is: , in, For the total loss, For mission losses, Let I be the Frobenius norm, and I be the identity matrix. For the loss trade-off coefficient, This is a low-rank matrix that retains the size of the first eigenvalue in the projection matrix. The projection matrix is ​​a low-rank matrix that retains the second-dimensional feature size. and This is the transpose of the matrix.

[0015] Through the above technical means, the embodiments of this application can solve the gradient entanglement problem of low-rank composite matrices by constraining the orthogonality of shared submatrices based on orthogonal loss functions, and combine the initialization strategy to guide the parameter update direction to align with the ideal gradient, generate optimized adapter parameters, and improve the model convergence efficiency.

[0016] Optionally, in one embodiment of this application, generating the adaptation result based on the visual task includes: replacing the multi-branch low-rank projection layer in the visual adapter according to the standard linear layer; obtaining the visual task input image of the visual task; and generating the adaptation result based on the visual task input image using the visual base model and the adapter.

[0017] Through the above-mentioned technical means, the embodiments of this application can replace the complex multi-branch low-rank projection structure with a standard linear layer, which simplifies the design of the visual adapter, simplifies the inference calculation process while maintaining parameter efficiency, reduces the computational overhead and storage requirements in the deployment stage, and improves the convenience and compatibility of model deployment, providing a brand-new solution for the efficient deployment of visual basic models.

[0018] A third aspect of this application provides an adaptation device for a visual adapter, comprising: an optimization module for inserting the visual adapter into a visual base model to initialize the parameters of the visual adapter and perform gradient descent training using a weighted total loss function to optimize the parameters of the visual adapter and obtain a multi-branch low-rank structure; and an adaptation module for converting the multi-branch low-rank structure into a standard linear layer to generate an adaptation result based on the visual task.

[0019] Through the above-mentioned technical means, the embodiments of this application can initialize the model and parameters, use a weighted total loss function for gradient descent training, optimize the adapter parameters, convert the trained multi-branch low-rank structure into a standard linear layer, and complete inference in combination with the base model. This significantly reduces the cost of fine-tuning and deployment without sacrificing performance, improves deployment convenience, and is applicable to various visual tasks such as object detection, semantic segmentation, image classification, and remote sensing rotating object detection. It can be widely used in scenarios that require efficient deployment of visual base models, such as autonomous driving, medical image analysis, geographic information systems, and urban planning.

[0020] Optionally, in one embodiment of this application, the optimization module includes: a generation unit for generating an initial projection matrix; a decomposition unit for performing singular value decomposition on the initial projection matrix to obtain a left singular vector matrix, a singular value matrix, and a right singular vector matrix; and an initialization unit for initializing the lower projection matrix and the upper projection matrix respectively, and freezing the backbone parameters of the basic model.

[0021] Through the above-mentioned technical means, the embodiments of this application initialize the adapter matrix by singular value decomposition, so that the shared submatrices maintain orthogonality in the early stage of training, laying the foundation for subsequent optimization and further accelerating the convergence speed.

[0022] Optionally, in one embodiment of this application, the weighted total loss function is: , in, For the total loss, For mission losses, Let I be the Frobenius norm, and I be the identity matrix. For the loss trade-off coefficient, This is a low-rank matrix that retains the size of the first eigenvalue in the projection matrix. The projection matrix is ​​a low-rank matrix that retains the second-dimensional feature size. and This is the transpose of the matrix.

[0023] Through the above technical means, the embodiments of this application can solve the gradient entanglement problem of low-rank composite matrices by constraining the orthogonality of shared submatrices based on orthogonal loss functions, and combine the initialization strategy to guide the parameter update direction to align with the ideal gradient, generate optimized adapter parameters, and improve the model convergence efficiency.

[0024] Optionally, in one embodiment of this application, the adaptation module includes: a replacement unit, configured to replace the multi-branch low-rank projection layer in the visual adapter according to the standard linear layer; an acquisition unit, configured to acquire the visual task input image of the visual task; and a result generation unit, configured to generate the adaptation result based on the visual task input image using the visual basic model and the adapter.

[0025] Through the above-mentioned technical means, the embodiments of this application can replace the complex multi-branch low-rank projection structure with a standard linear layer, which simplifies the design of the visual adapter, simplifies the inference calculation process while maintaining parameter efficiency, reduces the computational overhead and storage requirements in the deployment stage, and improves the convenience and compatibility of model deployment, providing a brand-new solution for the efficient deployment of visual basic models.

[0026] A fourth aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the adaptation method of the visual adapter as described in the above embodiments.

[0027] A fifth aspect of this application provides a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described adaptation method for a visual adapter.

[0028] A sixth aspect of this application provides a computer program product that stores a computer program that, when executed by a processor, implements the above-described adaptation method for a visual adapter.

[0029] This application's embodiments can design a multi-branch low-rank adapter architecture, combining a complex sharing strategy of kernel sharing and branch sharing. Simultaneously, based on matrix theory analysis of the gradient entanglement problem of low-rank composite matrices, an orthogonal loss function is introduced and parameter initialization strategies are optimized to improve model convergence efficiency. This achieves excellent adaptation results with only a few new parameters, reducing the deployment cost of the visual base model while ensuring superior performance and fast convergence, providing a novel solution for the efficient deployment of visual base models. Therefore, it solves the problems in related technologies, such as the high cost of full parameter fine-tuning, insufficient adaptation efficiency of delta-tuning methods in visual tasks, and convergence difficulties in low-rank adapters, which make it difficult to balance parameter efficiency and adaptation performance.

[0030] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0031] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the structure of a visual adapter based on complex linear projection optimization according to an embodiment of this application; Figure 2 This is a flowchart of a vision adapter adaptation method provided according to an embodiment of this application; Figure 3 This is a flowchart of a vision adapter adaptation method according to an embodiment of this application; Figure 4 This is a schematic diagram of a multi-branch low-rank adapter structure and its sharing mechanism according to an embodiment of this application; Figure 5 This is a schematic diagram showing the position of CoLin inserted into the Swing Transformer according to an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an adapter device for a vision adapter according to an embodiment of this application; Figure 7This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application.

[0032] Figure label: 10-Visual adapter based on complex linear projection optimization; 110-Multi-branch low-rank projection module, 120-Complex sharing strategy module, 130-Parameter space orthogonal optimization module, 140-Initialization module; 20-Adaptation device for visual adapter; 100-Optimization module, 200-Adaptation module; 701-Memory, 702-Processor, 703-Communication interface. Detailed Implementation

[0033] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0034] The following description, with reference to the accompanying drawings, illustrates a visual adapter and its adaptation method based on complex linear projection optimization, according to embodiments of this application. Addressing the issues raised in the background section regarding related technologies, such as the high cost of full-parameter fine-tuning, insufficient adaptation efficiency of delta-tuning methods in visual tasks, and convergence difficulties in low-rank adapters, this application provides a visual adapter adaptation method. This method designs a multi-branch low-rank adapter architecture, combining a complex sharing strategy of kernel sharing and branch sharing. Simultaneously, it analyzes the gradient entanglement problem of low-rank composite matrices based on matrix theory, introduces an orthogonal loss function, and optimizes the parameter initialization strategy to improve model convergence efficiency. This achieves excellent adaptation results with only a few additional parameters, reducing the deployment cost of the basic visual model while ensuring superior performance and fast convergence, providing a novel solution for the efficient deployment of basic visual models. Therefore, this method solves the problems in related technologies, such as the high cost of full-parameter fine-tuning, insufficient adaptation efficiency of delta-tuning methods in visual tasks, and convergence difficulties in low-rank adapters, which make it difficult to balance parameter efficiency and adaptation performance.

[0035] Specifically, Figure 1 This is a flowchart illustrating a visual adapter adaptation method provided in an embodiment of this application.

[0036] like Figure 1 As shown, the visual adapter 10 based on complex linear projection optimization includes: a multi-branch low-rank projection module 110, a complex sharing strategy module 120, a parameter space orthogonal optimization module 130, and an initialization module 140.

[0037] The multi-branch low-rank projection module 110 is used to construct the upper and lower projection matrices by linearly combining multiple low-rank matrices. Each low-rank matrix is ​​composed of the product of three submatrices, which are summed to obtain the upper and lower projection matrices. The rank of the low-rank matrix is ​​less than the input and output dimensions of the upper and lower projection matrices.

[0038] It is understood that in the embodiments of this application, a low-rank matrix can be understood as a matrix whose rank is much smaller than the number of rows or columns of the matrix itself; the upper and lower projection matrices can be understood as the sum of the low-rank matrices of each branch.

[0039] In actual implementation, the embodiments of this application can construct upper and lower projection matrices by linear combination of multiple low-rank matrices. Each low-rank matrix is ​​composed of the product of three sub-matrices. The final projection matrix is ​​obtained by summing all branch low-rank matrices. The rank of the low-rank matrix is ​​much smaller than the input and output dimensions of the projection matrix.

[0040] Among them, the number of parameters in the multi-branch low-rank projection module 110 is compressed by more than 98% compared with the standard linear layer, and the newly added parameters account for about 1% of the total parameters of the visual basic model, which greatly reduces the computation and storage costs.

[0041] The embodiments of this application can draw on the concept of hybrid experts, replacing the traditional single-block projection matrix with a linear combination of multi-branch low-rank matrices. This reduces parameter redundancy, expands the solution space through branch diversity, compensates for errors using integration effects, and does not affect inference efficiency. The parameter efficiency is extremely high, and the computation and storage costs are significantly reduced.

[0042] Optionally, in one embodiment of this application, the expression for the upper and lower projection matrices is: , , in, Let be the upward projection matrix. This is a low-rank matrix that retains the size of the first eigenvalue in the upward projection matrix. For the first The kernel matrix of each branch, This is a low-rank matrix that retains the second eigenvalue in the upward projection matrix. Let be the lower projection matrix. This is a low-rank matrix that retains the size of the first eigenvalue in the downward projection matrix. This is a low-rank matrix that retains the second eigenvalue in the lower projection matrix. This is the matrix transpose operation.

[0043] For example, in this embodiment, the Swin Transformer (Swin-B / Swin-L) can be used as the base model. The CoLin adapter is inserted after the skip connections of each SwinBlock. Specifically, the number of branches is set to 4, the rank of the low-rank matrix is ​​28, and the projection dimension is 100. The expression for the low-rank matrix of each branch is as follows: The upper and lower projection matrices are sums of the low-rank matrices of each branch, expressed as: , .

[0044] Furthermore, a 3×3 deep convolutional layer is added before the GeLU activation function to enhance local feature extraction.

[0045] The embodiments of this application can be based on a dual-sharing mechanism, where the kernel matrix is ​​shared between the upper and lower projections of the same branch to ensure the continuity of feature transformation. All branches share two general sub-matrices, while only the kernel matrix is ​​a branch-specific parameter, thus balancing parameter efficiency and generalization ability.

[0046] The complex sharing strategy module 120 is used to realize the sharing of the kernel matrix between the upper and lower projection matrices of the same branch based on the preset kernel sharing and branch sharing mechanism, and the sharing of multiple shared sub-matrices among all branches. The kernel matrix is ​​the trainable parameter corresponding to the branch.

[0047] It is understood that the preset kernel sharing and branch sharing mechanism in the embodiments of this application can be understood as the upper and lower projection matrices of the same branch sharing the kernel matrix, and all branches sharing two sub-matrices. The preset kernel sharing and branch sharing mechanism can be set by those skilled in the art according to the actual situation, and no specific restrictions are made here.

[0048] In actual implementation, the embodiments of this application can use a kernel sharing and branch sharing mechanism, where the upper and lower projection matrices of the same branch share the kernel matrix, all branches share two sub-matrices, and only the kernel matrix is ​​a branch-specific trainable parameter.

[0049] The embodiments of this application can design a dual sharing mechanism, in which the upper and lower projections of the same branch share the kernel matrix to ensure the continuity of feature transformation, and all branches share two general sub-matrices, with only the kernel matrix being a branch-specific parameter, thus balancing parameter efficiency and generalization ability.

[0050] The parameter space orthogonal optimization module 130 is used to constrain the orthogonality of the shared submatrix by using a preset orthogonal loss function, so that the parameter update direction is aligned with the ideal gradient direction.

[0051] It is understood that the preset orthogonal loss function in the embodiments of this application can be understood as a loss function used to measure the orthogonality deviation of the shared submatrix. The shared submatrix can be constrained to remain orthogonal during training through loss constraints. The preset orthogonal loss function can be set by those skilled in the art according to the actual situation, and no specific restrictions are made here. The ideal gradient direction can be understood as the parameter should be updated along the steepest descent path under the condition of no interference.

[0052] In actual implementation, the embodiments of this application can constrain the orthogonality of the shared submatrix through the orthogonal loss function to improve the efficiency of parameter update; the total loss function of the orthogonal optimization module is the weighted sum of the task loss and the orthogonal loss, and the orthogonal loss weight can be adjusted according to the task requirements.

[0053] The embodiments of this application can address the gradient entanglement problem of low-rank composite matrices by introducing an orthogonal loss function to constrain the orthogonality of shared submatrices, thereby aligning the parameter update direction with the ideal gradient direction and significantly improving convergence efficiency.

[0054] Initialization module 140 is used to perform singular value decomposition on the initial projection matrix to initialize the shared submatrix.

[0055] It is understood that singular value decomposition in this embodiment can be understood as a mathematical method to decompose any matrix into the product of three specific matrices. It can decompose the initial projection matrix into a left singular vector matrix, a singular value matrix, and a right singular vector matrix. The left singular vector matrix reflects the input feature direction of the initial projection matrix, the right singular vector matrix reflects the output feature direction, and the singular value matrix reflects the importance of each feature. Through singular value decomposition, the core feature information of the initial projection matrix can be fully explored, providing a reliable basis for the initialization of the shared submatrix.

[0056] In actual implementation, the embodiments of this application can perform singular value decomposition on the initial projection matrix, initialize each sub-matrix based on the decomposition results, and ensure orthogonality in the early stage of training.

[0057] For example, in the embodiments of this application, the initial projection matrix can be generated by uniform initialization using Kaiming, and then the matrix required for initialization can be obtained by singular value decomposition.

[0058] The embodiments of this application can initialize the adapter matrix through singular value decomposition, so that the shared submatrices maintain orthogonality in the early stage of training, laying the foundation for subsequent optimization and further accelerating the convergence speed.

[0059] Specifically, Figure 2 This is a flowchart illustrating a visual adapter adaptation method provided in an embodiment of this application.

[0060] like Figure 2As shown, the adaptation method of this visual adapter includes the following steps: In step S201, the visual adapter is inserted into the visual base model to initialize the parameters of the visual adapter, and gradient descent training is performed using a weighted total loss function to optimize the parameters of the visual adapter and obtain a multi-branch low-rank structure.

[0061] It is understood that the hyperparameters of the visual adapter in this application embodiment are adjustable, with the default configuration being 100 projection dimensions, 28 kernel size, and 4 branch counts, balancing parameter efficiency and performance.

[0062] In practical implementation, this embodiment of the application can insert the adapter into the SwinBlock of the visual base model, freeze the backbone parameters, and retain only the adapter parameters for training. Singular value decomposition is performed on the initial projection matrix to initialize the shared sub-matrices, thus completing the initialization of each matrix of the adapter. Gradient descent training is performed using a weighted total loss function to optimize the adapter parameters.

[0063] Furthermore, the embodiments of this application are adapted to datasets such as Pascal VOC, ADE20K, MS COCO, and DOTA, and adopt task frameworks such as RetinaNet, UpperNet, and Cascade Mask RCNN accordingly; the loss is calculated, the input image is processed by the base model to extract features, and after passing through the adapter, the fused features are output. The adapter parameters are updated by backpropagation based on the total loss, and training stops when a specified number of rounds are reached.

[0064] In this embodiment, parameter initialization can be completed through the initialization module. A weighted total loss function is adopted, which takes into account the adaptation performance, convergence speed and generalization ability. Gradient descent training is used to specifically optimize the trainable parameters of the adapter, further alleviate gradient entanglement and improve adaptation performance. The backbone parameters of the base model are frozen, which not only retains the pre-trained generalization ability of the base model, but also reduces the number of training parameters and improves parameter efficiency.

[0065] Optionally, in one embodiment of this application, the visual adapter is initialized with parameters including: generating an initial projection matrix; performing singular value decomposition on the initial projection matrix to obtain a left singular vector matrix, a singular value matrix, and a right singular vector matrix; initializing the lower projection matrix and the upper projection matrix respectively, and freezing the backbone parameters of the basic model.

[0066] It is understood that in the embodiments of this application, the left singular vector matrix, the singular value matrix, and the right singular vector matrix can be understood as the core decomposition components of the initial projection matrix, corresponding to the input feature direction, feature importance, and output feature direction, respectively; freezing the backbone parameters of the basic model can be used to protect the pre-training generalization ability of the basic model, avoid the backbone parameters from being destroyed during training, and reduce the number of training parameters.

[0067] In actual implementation, the embodiments of this application can use Kaiming uniform initialization to generate the initial projection matrix. ;right Perform singular value decomposition to obtain the left singular vector matrix U, the singular value matrix S, and the right singular vector matrix V; initialize the lower projection matrix: , , Initialization of the projection matrix: , Freeze the backbone parameters of the basic model, and retain only the low-rank matrix and deep convolutional layer parameters for training.

[0068] In this embodiment, the adapter matrix is ​​initialized by singular value decomposition, so that the shared submatrices maintain orthogonality in the early stage of training, laying the foundation for subsequent optimization and further accelerating the convergence speed.

[0069] Optionally, in one embodiment of this application, the weighted total loss function is: , in, For the total loss, For mission losses, Let I be the Frobenius norm, and I be the identity matrix. For the loss trade-off coefficient, This is a low-rank matrix that retains the size of the first eigenvalue in the projection matrix. This is a low-rank matrix that retains the second eigenvalue of the projection matrix. and It is the transpose of two matrices.

[0070] In practical implementation, this embodiment of the application can use a weighted total loss function for gradient descent training to optimize adapter parameters. The total loss function is a weighted sum of the task loss and the orthogonal loss, and the weights of the orthogonal loss can be adjusted according to task requirements. By constraining the orthogonality of the shared submatrices through the orthogonal loss function, the efficiency of parameter update is improved.

[0071] The embodiments of this application can solve the gradient entanglement problem of low-rank composite matrices by constraining the orthogonality of shared submatrices based on orthogonal loss functions, and combine the initialization strategy to guide the parameter update direction to align with the ideal gradient, generate optimized adapter parameters, and improve the model convergence efficiency.

[0072] In step S202, the multi-branch low-rank structure is converted into a standard linear layer to generate adaptation results based on the vision task.

[0073] It is understood that the standard linear layer in the embodiments of this application can be understood as a feature transformation layer commonly used in the deployment of visual basic models. It has a simple structure and strong compatibility. Visual tasks may include object detection, semantic segmentation, image classification and remote sensing rotating object detection, and are adapted to corresponding mainstream datasets.

[0074] In practical implementation, the embodiments of this application can convert the trained multi-branch low-rank structure into a standard linear layer, and combine it with the base model to complete inference. The following formula is used to convert all the above low-rank matrices into standard linear matrices: , .

[0075] Replace the multi-branch low-rank projection layer in the adapter with these matrices.

[0076] For example, embodiments of this application can embed the optimized adapter into the skip connection path of the visual base model, enhance local features through deep convolution, and complete feature recovery and fusion through upper and lower projection matrices to output exclusive features adapted to downstream tasks for processing by the task head. A structural transformation strategy is adopted to convert multi-branch low-rank structures into standard linear layers to reduce inference overhead, integrating and validating the adaptation results for various visual tasks such as object detection and segmentation, and outputting high-precision task results that surpass traditional methods.

[0077] The embodiments of this application can convert multi-branch low-rank structures into standard linear layers without requiring large-scale modifications to the deployment framework of existing visual basic models, thereby reducing the actual deployment difficulty and cost of the model. The standard linear layer has a simple structure and fast computation speed, which can improve the output efficiency of adaptation results, enabling the model to generate adaptation results stably and efficiently.

[0078] Optionally, in one embodiment of this application, generating an adaptation result based on a visual task includes: replacing the multi-branch low-rank projection layer in the visual adapter with the standard linear layer; obtaining the visual task input image of the visual task; and generating the adaptation result based on the visual task input image using the visual base model and the adapter.

[0079] It is understood that the multi-branch low-rank projection layer in this application embodiment can be understood as being composed of a linear combination of multiple low-rank matrices, which has the characteristics of high parameter efficiency and strong adaptation potential, but has a complex structure and cumbersome inference calculation process; the adaptation result can be understood as the final result that meets the requirements of the visual task, such as image classification labels, object detection boxes, semantic segmentation masks, etc.

[0080] In actual implementation, the embodiments of this application can adopt a structure transformation strategy to convert multi-branch low-rank structures into standard linear layers to reduce inference overhead.

[0081] For example, embodiments of this application can convert all the above-mentioned low-rank matrices into standard linear matrices, as shown in the following formula: , ; in, Let be the upward projection matrix. This is a low-rank matrix that retains the size of the first eigenvalue in the upward projection matrix. For the first The kernel matrix of each branch, This is a low-rank matrix that retains the second eigenvalue in the upward projection matrix. Let be the lower projection matrix. This is a low-rank matrix that retains the size of the first eigenvalue in the downward projection matrix. This is a low-rank matrix that retains the second eigenvalue in the lower projection matrix. This is a matrix transpose operation; these matrices are used to replace the multi-branch low-rank projection layer in the adapter.

[0082] Furthermore, embodiments of this application can acquire the visual task input image of a visual task, generate the result using the visual base model and adapter, extract features from the image through the base model, adapt it through the converted adapter, and output the result from the task head. In complex scenarios, a multi-scale fusion strategy can be adopted to improve robustness.

[0083] The embodiments of this application can replace the complex multi-branch low-rank projection structure with a standard linear layer, which simplifies the design of the visual adapter. While maintaining parameter efficiency, it simplifies the inference calculation process, reduces the computational overhead and storage requirements in the deployment phase, and improves the convenience and compatibility of model deployment, providing a brand-new solution for the efficient deployment of visual basic models.

[0084] Specifically, it can be combined with Figures 3 to 4 As shown, the working principle of the visual adapter adaptation method in this application is explained in detail with a specific embodiment.

[0085] like Figure 3 As shown, embodiments of this application may include the following steps: Step S301: Construction of multi-branch low-rank structure and adaptation of basic features: Obtain the input image of the visual task, extract global features through the visual base model (Swin Transformer), input the features into the CoLin adapter, construct the projection layer through the multi-branch low-rank matrix, complete the initial compression and transformation of basic features, and prepare for subsequent adaptation.

[0086] Step S302: Orthogonal optimization and parameter update mechanism: Based on the orthogonal loss function constraint of the shared submatrix, the gradient entanglement problem of low-rank composite matrix is ​​solved. Combined with the singular value decomposition initialization strategy, the parameter update direction is guided to align with the ideal gradient, and the optimized adapter parameters are generated to improve the model convergence efficiency.

[0087] Step S303: Feature fusion and adaptation feature output: The optimized adapter is embedded into the skip connection path of the visual base model, local features are enhanced by deep convolution, and feature recovery and fusion are completed by the upper and lower projection matrices to output exclusive features adapted to downstream tasks for processing by the task head.

[0088] Step S304: Multi-task adaptation and result output: Using a structural transformation strategy, the multi-branch low-rank structure is transformed into a standard linear layer to reduce inference overhead. The adaptation results of various visual tasks such as object detection and segmentation are integrated and verified, and the high-precision task results that surpass traditional methods are output.

[0089] like Figure 4 As shown, Figure 4 The specific structure of the multi-branch low-rank projection layer and the visual adapter is shown. In the multi-branch low-rank projection layer, the parameters of the two low-rank matrices in multiple branches are shared, and the kernel matrix parameters in the upper and lower projection matrices are also shared. The matrix parameters of the linear projection layer are calculated based on the low-rank matrices. The visual adapter consists of two multi-branch low-rank projections, a 3x3 depthwise separable convolutional layer, and a non-linear activation layer.

[0090] like Figure 5 As shown, Figure 5 This demonstrates where the visual adapter is inserted in the Swing Transformer. The visual adapter is inserted after two jump connections in each Swing block.

[0091] The visual adapter adaptation method proposed in this application can design a multi-branch low-rank adapter architecture, combining a complex sharing strategy of kernel sharing and branch sharing. Simultaneously, based on matrix theory analysis of the gradient entanglement problem of low-rank composite matrices, an orthogonal loss function is introduced and the parameter initialization strategy is optimized to improve model convergence efficiency. This achieves excellent adaptation results with only a few new parameters, reducing the deployment cost of the visual base model while ensuring superior performance and fast convergence, providing a novel solution for the efficient deployment of visual base models. This solves the problem in related technologies where the high cost of full parameter fine-tuning, insufficient adaptation efficiency of delta-tuning methods in visual tasks, and convergence difficulties in low-rank adapters make it difficult to balance parameter efficiency and adaptation performance.

[0092] Next, the adapter device for the visual adapter according to the embodiments of this application is described with reference to the accompanying drawings.

[0093] Figure 6 This is a schematic diagram of the structure of the adaptation device of the visual adapter according to an embodiment of this application.

[0094] like Figure 6 As shown, the adaptation device 20 of the visual adapter includes an optimization module 100 and an adaptation module 200.

[0095] Among them, the optimization module 100 is used to insert the visual adapter into the visual base model to initialize the parameters of the visual adapter, and to perform gradient descent training using a weighted total loss function to optimize the parameters of the visual adapter and obtain a multi-branch low-rank structure. The adaptation module 200 is used to convert multi-branch low-rank structures into standard linear layers to generate adaptation results based on vision tasks.

[0096] Optionally, in one embodiment of this application, the optimization module 100 includes: a generation unit, a decomposition unit, and an initialization unit.

[0097] The generating unit is used to generate the initial projection matrix.

[0098] The decomposition unit is used to perform singular value decomposition on the initial projection matrix to obtain the left singular vector matrix, the singular value matrix, and the right singular vector matrix.

[0099] The initialization unit is used to initialize the lower projection matrix and the upper projection matrix respectively, and to freeze the backbone parameters of the basic model.

[0100] Optionally, in one embodiment of this application, the weighted total loss function is: , in, For the total loss, For mission losses, Let I be the Frobenius norm, and I be the identity matrix. For the loss trade-off coefficient, This is a low-rank matrix that retains the size of the first eigenvalue in the projection matrix. This is a low-rank matrix that retains the second eigenvalue of the projection matrix. and It is the transpose of two matrices.

[0101] Optionally, in one embodiment of this application, the adaptation module 200 includes: a replacement unit, an acquisition unit, and a result generation unit.

[0102] The replacement unit is used to replace the multi-branch low-rank projection layer in the visual adapter according to the standard linear layer.

[0103] The acquisition unit is used to acquire the visual task input image of the visual task.

[0104] The result generation unit is used to generate the adaptation result based on the visual task input image, using the visual basic model and adapter.

[0105] It should be noted that the foregoing explanation of the adaptation method embodiment for the visual adapter also applies to the adaptation device of the visual adapter in this embodiment, and will not be repeated here.

[0106] The adaptation device for the visual adapter proposed in this application can design a multi-branch low-rank adapter architecture, combining a complex sharing strategy of kernel sharing and branch sharing. Simultaneously, based on matrix theory analysis of the gradient entanglement problem of low-rank composite matrices, an orthogonal loss function is introduced and the parameter initialization strategy is optimized to improve model convergence efficiency. This achieves excellent adaptation results with only a few new parameters, reducing the deployment cost of the visual base model while ensuring superior performance and fast convergence, providing a novel solution for the efficient deployment of visual base models. This solves the problem in related technologies where the cost of full parameter fine-tuning is too high, the delta-tuning method has insufficient adaptation efficiency in visual tasks, and low-rank adapters suffer from convergence difficulties, making it difficult to balance parameter efficiency and adaptation performance.

[0107] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0108] When the processor 702 executes the program, it implements the adaptation method of the visual adapter provided in the above embodiments.

[0109] Furthermore, electronic devices also include: Communication interface 703 is used for communication between memory 701 and processor 702.

[0110] The memory 701 is used to store computer programs that can run on the processor 702.

[0111] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0112] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0113] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0114] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0115] This application also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described adaptation method for the visual adapter.

[0116] This application also provides a computer program product on which a computer program is stored, which, when executed by a processor, implements the above-described adaptation method for the visual adapter.

[0117] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0118] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0119] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0120] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0121] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0122] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0123] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0124] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A visual adapter based on complex linear projection optimization, characterized in that, include: The multi-branch low-rank projection module is used to construct upper and lower projection matrices by linear combination of multiple low-rank matrices. Each low-rank matrix is ​​composed of the product of three submatrices, which are summed to obtain the upper and lower projection matrices. The rank of the low-rank matrix is ​​less than the input and output dimensions of the upper and lower projection matrices. The complex sharing strategy module is used to realize the sharing of the kernel matrix between the upper and lower projection matrices of the same branch based on the preset kernel sharing and branch sharing mechanism, and the sharing of multiple shared sub-matrices among all branches, wherein the kernel matrix is ​​the trainable parameter corresponding to the branch; The parameter space orthogonal optimization module is used to constrain the orthogonality of the shared submatrices by using a preset orthogonal loss function, so that the parameter update direction is aligned with the ideal gradient direction; An initialization module is used to perform singular value decomposition on the initial projection matrix to initialize the shared submatrix.

2. The visual adapter based on complex linear projection optimization according to claim 1, characterized in that, The expression for the upper and lower projection matrices is: ， ， in, Let be the upward projection matrix. The upper projection matrix is ​​a low-rank matrix that retains the size of the first dimension feature. For the first The kernel matrix of each branch, The upper projection matrix is ​​a low-rank matrix that retains the second dimension of the feature size. Let be the lower projection matrix. The lower projection matrix is ​​a low-rank matrix that retains the size of the first dimension feature. The lower projection matrix is ​​a low-rank matrix that retains the second dimension of the feature size. This is the matrix transpose operation.

3. A method for adapting a visual adapter, characterized in that, Applied to the visual adapter based on complex linear projection optimization as described in any one of claims 1-2, wherein the method includes the following steps: The visual adapter is inserted into the visual base model to initialize the parameters of the visual adapter, and gradient descent training is performed using a weighted total loss function to optimize the parameters of the visual adapter and obtain a multi-branch low-rank structure. The multi-branch low-rank structure is converted into a standard linear layer to generate adaptation results based on the vision task.

4. The method according to claim 3, characterized in that, The parameter initialization of the visual adapter includes: Generate the initial projection matrix; The initial projection matrix is ​​subjected to singular value decomposition to obtain the left singular vector matrix, the singular value matrix, and the right singular vector matrix; Initialize the lower projection matrix and the upper projection matrix respectively, and freeze the backbone parameters of the basic model.

5. The method according to claim 3, characterized in that, The weighted total loss function is: ， in, For the total loss, For mission losses, Let I be the Frobenius norm, and I be the identity matrix. For the loss trade-off coefficient, This is a low-rank matrix that retains the size of the first eigenvalue in the projection matrix. The projection matrix is ​​a low-rank matrix that retains the second-dimensional feature size. and This is the transpose of the matrix.

6. The method according to claim 3, characterized in that, The generation of adaptation results based on the visual task includes: Replace the multi-branch low-rank projection layer in the visual adapter with the standard linear layer; Obtain the visual task input image for the visual task; Based on the input image of the visual task, the adaptation result is generated using the visual basic model and adapter.

7. An adapter for a vision adapter, characterized in that, Applied to the visual adapter based on complex linear projection optimization as described in any one of claims 1-2, wherein the device comprises: An optimization module is used to insert the visual adapter into the visual base model to initialize the parameters of the visual adapter and perform gradient descent training using a weighted total loss function to optimize the parameters of the visual adapter and obtain a multi-branch low-rank structure. An adaptation module is used to convert the multi-branch low-rank structure into a standard linear layer to generate adaptation results based on the visual task.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the adaptation method of the visual adapter as described in any one of claims 3-6.

9. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the adaptation method of the visual adapter as described in any one of claims 3-6.

10. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the adaptation method of the visual adapter as described in any one of claims 3-6.

Images