YOLO model parameter efficient fine tuning method based on low-rank self-adaption

Abstract

The invention discloses a YOLO model parameter efficient fine tuning method based on low-rank self-adaption. Aiming at the problems that a traditional full-parameter fine tuning method is high in calculation cost and large in storage pressure and an existing efficient parameter fine tuning method is insufficient in adaptation in a visual task, a low-rank adaptive layer is introduced into a convolutional layer of a YOLO model, a weight update quantity is decomposed into a low-rank matrix product, and a feature fusion process is optimized in combination with a lightweight adapter. In specific implementation, customized improvement is performed on a core module of the YOLO model, including multi-scale feature fusion optimization, attention weight dynamic adjustment and the like, and meanwhile, pre-training parameters are frozen and only a low-rank matrix is updated during training. The parameter efficiency and the detection performance are effectively balanced under the condition that only a small number of parameters are finely adjusted, and the method is suitable for edge device deployment and real-time detection scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to a low-rank adaptive-based YOLO model parameter efficient fine-tuning method, belonging to the target detection and model lightweight technology in the field of computer vision. BACKGROUND

[0002] With the rapid development of computer vision technology, target detection is increasingly widely used in intelligent security, autonomous driving, industrial quality inspection and other fields. The complexity of its application scenarios also brings multiple challenges. The target detection task not only needs to deal with multi-class target recognition, complex background interference and small target detection precision problems, but also faces the dilemma of parameter efficiency and detection performance, and the bottleneck of multi-scale feature fusion and spatial attention mechanism adaptation.

[0003] Although advanced target detection models represented by the YOLO series have made breakthroughs in precision and speed, the full parameter fine-tuning (FPFT) method they rely on needs to update tens of millions of parameters, resulting in a dramatic increase in computing cost and storage pressure. This high resource consumption not only causes low training efficiency, but also makes traditional models face serious constraints in edge device deployment and real-time scene applications. Two-stage models (such as Faster R-CNN) and single-stage models (such as YOLO) are difficult to balance the relationship between computing resource consumption, storage demand and detection performance, especially in dynamic and complex scenarios, revealing the lack of adaptability.

[0004] In order to solve the above problems, parameter efficient fine-tuning (PEFT) technology has emerged. The PEFT method significantly reduces the fine-tuning computing cost and storage demand by introducing a small number of trainable parameters in the pre-trained model, while maintaining the performance of the model. However, existing PEFT methods are mainly designed for natural language processing tasks, and their application in computer vision tasks, especially the adaptation of convolutional neural networks and YOLO series models, lacks optimization schemes for the characteristics of convolutional networks, making it difficult to effectively adapt to the needs of target detection tasks. SUMMARY

[0005] In view of the shortcomings of the prior art, the present application proposes a low-rank adaptive-based YOLO model parameter efficient fine-tuning method, which fuses the parameter efficient fine-tuning framework and the low-rank matrix optimization theory, and introduces a lightweight low-rank adaptive Adaptation layer in the convolutional layer of the YOLO model. The pre-trained YOLO target detection model can be efficiently fine-tuned to adapt to various downstream vision tasks without full parameter fine-tuning of the entire YOLO target detection model.

[0006] To achieve the above technical purposes, the present application provides the following technical solutions:

[0007] I. A low-rank adaptive-based YOLO model parameter efficient fine-tuning method, comprising the following steps:

[0008] Step 1: Embed a low-rank adaptive layer in the convolutional layers of the backbone network, neck network and head network of the YOLO model, wherein the low-rank adaptive layer generates a trainable weight increment matrix △W by performing low-rank matrix decomposition on a pre-trained weight matrix W of the YOLO model, wherein the pre-trained weight matrix C out is the number of convolutional layer output channels, C in is the number of convolutional layer input channels, and k is the size of the convolutional kernel, wherein the number of convolutional layer output channels C out , the number of convolutional layer input channels C in , and the size of the convolutional kernel k are all positive integers, and the trainable weight increment matrix △W = B·A, wherein B is a low-rank matrix and A is a low-rank matrix and r is a rank parameter, wherein the rank parameter r is a positive integer, and the rank parameter r is much smaller than the number of convolutional layer output channels C out and the number of convolutional layer input channels C in , and the trainable weight increment matrix The backbone network of the YOLO model includes a C3K2 module for multi-branch feature fusion, an SPPF module for multi-scale feature extraction, and a C2PSA module for pixel-level spatial self-attention mechanism; the low-rank adaptive layer constitutes a parameter efficient fine-tuning module, which is integrated in the convolutional layers of the YOLO model, the C3K2 module for multi-branch feature fusion, the SPPF module for multi-scale feature extraction, and the C2PSA module for pixel-level spatial self-attention mechanism, and the parameter efficient fine-tuning module is used to dynamically regulate the trainable weight increment matrix △W.

[0009] Step 2: Approximate the trainable weight increment matrix △W by a low-rank decomposition method, wherein the method includes one of LoRA method, DoRA method or PiSSA method; wherein the LoRA method reconstructs the weight increment matrix by low-rank matrix multiplication, the DoRA method decouples the pre-trained weight matrix W into an independently controllable amplitude vector and a low-rank adaptive direction matrix by amplitude-direction decomposition, and the PiSSA method retains the principal component matrix of the pre-trained weight matrix W based on singular value decomposition.

[0010] Step 3: Freeze the pre-trained weight matrix W, only update the low-rank matrix B and the low-rank matrix A, balance the parameter efficiency and detection performance by adjusting the rank parameter r, reduce the trainable parameter amount to less than 1% of the trainable parameter amount of full parameter fine-tuning based on the pre-trained weight matrix W;

[0011] Step 4: Perform low-rank adaptive optimization on the modules contained in the backbone network of the YOLO model, in the C3K2 module for multi-branch feature fusion, split the input features into skip connection path features and depth separable convolution processing path features through Split operation, and optimize multi-branch feature fusion by using low-rank adaptive convolution method; in the SPPF module for multi-scale feature extraction, construct different scale features by using multi-level maximum pooling method, and embed low-rank adaptive convolution in the branch convolution layer of the maximum pooling method to enhance cross-scale feature interaction; in the C2PSA module for pixel-level spatial self-attention mechanism, generate query matrix, key matrix and value matrix by using low-rank adaptive convolution method, and construct self-attention mechanism which is easy to dynamically adjust;

[0012] Step 5: Train the YOLO model by using a comprehensive loss function, the comprehensive loss function includes classification loss, bounding box regression loss and target confidence loss, optimize the low-rank matrix B and the low-rank matrix A by minimizing the comprehensive loss function, and output an efficient fine-tuning model suitable for target detection.

[0013] The convolution method of the low-rank adaptive layer in step 1 satisfies:

[0014] The pre-trained weight matrix W remains frozen, the pre-trained weight matrix W generates the trainable weight increment matrix △W by low-rank matrix decomposition method, the shape of the trainable weight increment matrix △W is consistent with the shape of the pre-trained weight matrix W, the trainable weight increment matrix △W is superimposed on the pre-trained weight matrix W in the form of residual, forming the adjusted weight matrix W'=W+△W, and the adjusted weight matrix W' is used for convolution calculation on the input convolution layer feature X, that is:

[0015] Conv adaptation (X)=Conv(X,W′);

[0016] Where, Conv adaptation (·) represents the convolution operation of the low-rank adaptive layer, Conv adaptation (X) represents the convolution operation of the low-rank adaptive layer on the input convolution layer feature X, and Conv(X,W') represents the convolution operation of the adjusted weight W' on the input convolution layer feature X.

[0017] The specific implementation of the C3K2 module of the backbone network for multi-branch feature fusion in step 1 is as follows:

[0018] The input feature X1 of the C3K2 module of the backbone network for multi-branch feature fusion is segmented, and the formula is as follows:

[0019] X c3_skip , proc =Split(Conv adaptation (X1));

[0020] wherein Split(·) represents that the input feature X1 of the C3K2 module of the backbone network for multi-branch feature fusion is segmented into a skip connection path feature X c3_skip and a depth separable convolution processing path feature X proc .

[0021] The processing formula of the input C3KBlock module feature X2 in the C3K2 module of the backbone network for multi-branch feature fusion is as follows:

[0022] C3KBlock(X2)=Concat[Bottleneck 2 (Conv adaptation (X2))+Conv adaptation (X2)];

[0023] wherein C3KBlock(X2) represents that the input feature X2 of the C3KBlock module is subjected to multi-branch feature fusion, Bottleneck 2 (·) represents two secondary cascaded low-rank adaptive convolution and nonlinear activation operations, and Concat(·) represents a feature concatenation operation along the channel dimension;

[0024] The output feature map X out after the fused feature is generated by low-rank adaptive convolution on the concatenated feature, and the feature fusion formula of the C3K2 module of the backbone network for multi-branch feature fusion is as follows:

[0025] X out =Conv adaptation (Concat[X c3_skip ,C3KBlock N (X proc )]);

[0026] wherein C3KBlock N (X proc ) represents the depth separable convolution processing path feature X procIterate the C3KBlock module N times, where N is a positive integer.

[0027] The specific implementation of the SPPF module for multi-scale feature extraction in step 1 of the backbone network is as follows:

[0028] In each pooling branch of the SPPF module for multi-scale feature extraction in the backbone network, a low-rank adaptive convolution is embedded, different scale features are captured through a max-pooling operation, and the multi-scale features are spliced and fused through a low-rank adaptive convolution. The improved multi-scale feature fusion formula is:

[0029] X out =Conv adaptation (Concat[X3,MaxPool1(X3),MaxPool2(X3),MaxPool3(X3)]);

[0030] Where MaxPool s (·) is a max-pooling operation with a step size of 1 for s consecutive times, s is a positive integer, and MaxPool s (X3) represents a max-pooling operation with a step size of 1 for s consecutive times on the input feature X3 of the SPPF module for multi-scale feature extraction in the backbone network.

[0031] The specific implementation of the C2PSA module for pixel-level spatial self-attention mechanism in step 1 of the backbone network is as follows:

[0032] The feature X4 input to the C2PSA module for pixel-level spatial self-attention mechanism in the backbone network is divided into a skip connection path feature X c2_skip and a self-attention processing path feature X att by a Split operation, with the formula being:

[0033] X c2_skip ,X att =Split(Conv adaptation (X4));

[0034] In the self-attention processing path feature, the channel dimension of the feature tensor output by the low-rank adaptive convolution is divided into three parts, and the spatial dimension of the feature tensor output by the low-rank adaptive convolution in each part is adjusted to generate a query matrix Q, a key matrix K, and a value matrix V. The specific formula is:

[0035] Q,K,V=ReshapeAndSplit(Conv adaptation (X att ));

[0036] wherein ReshapeAndSplit(·) is an operation of feature dimension segmentation and dimension adjustment;

[0037] The self-attention Attention module of the C2PSA module of the backbone network for the pixel-level spatial self-attention mechanism adopts a Softmax function to calculate self-attention weights, and obtains fused features through residual connection and fusion with the output of the value matrix V, and the specific formula is:

[0038]

[0039] wherein Attention(·) is a self-attention module, which includes multiple self-attention heads to capture the feature correlation of different subspaces, and Attention(X att ) represents that the pixel-level spatial self-attention mechanism of the multiple self-attention heads is used to model the self-attention processing path feature X att , d k is the dimension of the self-attention head;

[0040] In the fusion PSABlock module of the self-attention feature of the C2PSA module of the backbone network for the pixel-level spatial self-attention mechanism, the residual connection is performed between the output of the self-attention Attention module and the original feature input into the fusion PSABlock module of the self-attention feature, so as to retain the structure information of the original feature, and the feature after residual fusion is further processed by low-rank adaptive convolution, and the formula is:

[0041] PSABlock(X att )=Conv adaptation (X att +Attention(X att ));

[0042] wherein PSABlock(·) is a fusion module of self-attention features, and PSABlock(X att ) represents that the self-attention processing path feature X att is fused in the form of residual connection to obtain the self-attention feature;

[0043] The feature processed by the self-attention mechanism is spliced with the skip connection path feature X c2_skip , and the final output feature is obtained through low-rank adaptive convolution, that is:

[0044] X' out =Conv adaptation (Concat[PSABlock N (X att )+X c2_skip ]);

[0045] wherein, PSABlock N (X att ) denotes the progressive feature optimization of the self-attention processing path feature X att by applying the fusion PSA Block module of the self-attention feature through N iterations.

[0046] The low-rank decomposition method in step 2 is specifically:

[0047] LoRA method: in the training process, the low-rank matrix B is initialized as a full zero matrix, and the low-rank matrix A is randomly initialized as Gaussian, to ensure that the pre-training weight matrix W remains unchanged in the initial stage, and only the low-rank matrix B and the low-rank matrix A are updated, and the trainable weight increment matrix AW = B·A;

[0048] DoRA method: the pre-training weight matrix W is decomposed into an independently controllable amplitude vector m and a low-rank adaptive direction matrix E, the independently controllable amplitude vector m ∈ R 1×i , the low-rank adaptive direction matrix E ∈ R o×i , i is the input channel number of the DoRA method, o is the output channel number of the DoRA method, and only the low-rank adaptive direction matrix E is optimized, and the specific formula is:

[0049]

[0050] wherein, AE is an incremental direction matrix, the incremental direction matrix AE is parameterized as the product of the low-rank matrix B and the low-rank matrix A, and ||·||F c denotes the vector norm of the matrix calculated column by column;

[0051] PiSSA method: the pre-training weight matrix W is decomposed by singular value decomposition, and the pre-training weight matrix W is expressed as W = U·S·Q T , wherein, U and Q represent singular vectors with orthogonal columns, S is a diagonal matrix containing singular values, by setting the rank parameter r, the left singular vector and the right singular vector corresponding to the diagonal matrix S are reserved, to initialize the low-rank matrix B and the low-rank matrix A, the low-rank matrix the low-rank matrix the singular vectors corresponding to the remaining singular values constitute a residual matrix, and the residual matrix is frozen.

[0052] II. A target detection system

[0053] The target detection system comprises a pre-trained YOLO target detection model and a module of the low-rank adaptive parameter efficient fine-tuning method.

[0054] Three, an electronic device

[0055] The electronic device comprises a processor and a memory, and the memory stores program code, which, when executed by the processor, causes the processor to perform the steps of the method.

[0056] The beneficial effects of the present application are:

[0057] The present application integrates a low-rank adaptive strategy in the YOLO target detection framework, reconstructs the parameter update path by low-rank matrix decomposition, realizes efficient fine-tuning with only a small amount of new parameters and a lightweight low-rank adaptive Adaptation layer, and significantly reduces the consumption of computing resources compared with the traditional full-parameter fine-tuning method. For the YOLO core module, the multi-scale feature fusion and self-attention mechanism are customized and optimized to improve the target detection accuracy and feature capture ability in complex scenes. Based on the residual structure, the pre-trained parameters are frozen, only the low-rank adaptive Adaptation layer is fine-tuned, the downstream task is quickly adapted and the training efficiency is improved, and the single model multi-task parameter switching is supported, the maintenance cost of industrial deployment is reduced, and the real-time detection performance and robustness are considered. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 It is a schematic diagram of the overall architecture of the YOLOv11 model in the embodiment of the present application.

[0059] Figure 2 It is a schematic diagram of adding a low-rank adaptive Adaptation layer structure to the convolutional layer of the YOLOv11 model in the embodiment of the present application.

[0060] Figure 3 It is a schematic diagram of the three parameter efficient fine-tuning methods of the YOLOv11 model in the embodiment of the present application.

[0061] Figure 4 It is a schematic diagram of embedding a low-rank adaptive Adaptation layer structure into the C3K2 module of the YOLOv11 model in the embodiment of the present application.

[0062] Figure 5 It is a schematic diagram of embedding a low-rank adaptive Adaptation layer structure into the SPPF module of the YOLOv11 model in the embodiment of the present application.

[0063] Figure 6A low-rank adaptive Adaptation layer structure embedded in a C2PSA module of a YOLOv11 model in an embodiment of the present application is shown in a schematic diagram. DETAILED DESCRIPTION

[0064] In order for those skilled in the art to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0065] The present application is a low-rank adaptive YOLO model parameter efficient fine-tuning method, which comprises the following steps:

[0066] Step 1: Based on the pre-trained YOLO target detection model, taking the YOLOv11 model as an example, the core architecture is composed of three modules of backbone network, neck network and head network. As shown in Figure 1 , three types of key modules are integrated in the backbone network, including C3K2 module for multi-branch feature fusion, SPPF module for multi-scale feature extraction, and C2PSA module for pixel-level spatial self-attention mechanism. In all convolutional layers of the backbone network, neck network and head network of the YOLOv11 model, a low-rank adaptive Adaptation layer is embedded, and the low-rank adaptive Adaptation layer is lightened by low-rank matrix decomposition technology to the pre-training weight matrix.

[0067] Step 1.1: As shown in Figure 2 , the specific implementation of adding the Adaptation layer with low-rank adaptive characteristics to the convolutional layer is as follows:

[0068] Considering that the pre-training weight matrix of the convolutional layer of the YOLOv11 model is a four-dimensional tensor , where C out is the number of output channels of the convolutional layer, C in is the number of input channels of the convolutional layer, and k is the size of the convolution kernel. out , the number of output channels C in of the convolutional layer, the number of input channels C in of the convolutional layer, and the size of the convolution kernel k are all positive integers. Two low-rank matrices are constructed, which are low-rank matrix B and low-rank matrix A, where the low-rank matrix and the low-rank matrix r is a rank parameter, and the rank parameter r is a positive integer, and the rank parameter r is much smaller than the number of output channels Cout C in . By matrix multiplication, a trainable weight increment matrix is generated, the trainable weight increment matrix

[0069] The trainable weight increment matrix The pre-trained weight matrix W is added to the trainable weight increment matrix

[0070] W' = W + AW = W + Reshape(B·A, [C out × C in × k × k])

[0071] where Reshape(·) represents reshaping the matrix from a two-dimensional matrix to a four-dimensional tensor.

[0072] The adjusted weight matrix W' is used to perform convolution calculation on the input convolution layer feature X, that is:

[0073] Conv adaptation (X) = Conv(X, W')

[0074] where Conv adaptation (·) represents the convolution operation of the low-rank adaptive layer, Conv adaptation (X) represents applying the convolution operation of the low-rank adaptive layer to the input convolution layer feature X, and Conv(X, W') represents performing convolution operation on the input convolution layer feature X using the adjusted weight W'.

[0075] Step 1.2: The specific implementation of the C3K2 module of the backbone network for multi-branch feature fusion in step 1 is:

[0076] X c3_skip ,X proc = Split(Conv adaptation (X1));

[0077] where Split(·) represents splitting the input feature X1 of the C3K2 module of the backbone network for multi-branch feature fusion into the skip connection path feature X c3_skip and the depth separable convolution processing path feature X proc .

[0078] The formula for processing the input C3KBlock module feature X2 in the C3K2 module of the backbone network for multi-branch feature fusion is:

[0079] C3KBlock(X2) = Concat[Bottleneck 2(Conv adaptation (X2))+Conv adaptation (X2)];

[0080] wherein C3KBlock(X2) represents multi-branch feature fusion on the input C3KBlock module feature X2, Bottleneck 2 (·) represents twice low-rank adaptive convolution and nonlinear activation operation, and Concat(·) represents feature splicing operation along the channel dimension;

[0081] The output feature map X out after the fusion of the fused features is generated by fusing the spliced features through the low-rank adaptive convolution, and the feature fusion formula of the C3K2 module of the backbone network used for multi-branch feature fusion is:

[0082] X out =Conv adaptation (Concat[X c3_skip ,C3KBlock N (X proc )]);

[0083] wherein C3KBlock N (X proc ) represents the depth separable convolution processing path feature X proc , and N is a positive integer.

[0084] Step 1.3: The specific implementation of the SPPF module of the backbone network used for multi-scale feature extraction in step 1 is:

[0085] In each pooling branch of the SPPF module of the backbone network used for multi-scale feature extraction, a low-rank adaptive convolution is embedded, different scale features are captured through a maximum pooling operation, and the multi-scale features are spliced and fused through a low-rank adaptive convolution, and the improved multi-scale feature fusion formula is:

[0086] X out =Conv adaptation (Concat[X3,MaxPool1(X3),MaxPool2(X3),MaxPool3(X3)]);

[0087] wherein MaxPool s (·) is a maximum pooling with a step of 1 for s consecutive times, s is a positive integer, and MaxPool s (X3) represents a maximum pooling operation with a step of 1 for s consecutive times on the input feature X3 of the SPPF module of the backbone network used for multi-scale feature extraction.

[0088] Step 1.4: The implementation of the C2PSA module of the backbone network for pixel-level spatial self-attention mechanism in step 1 is as follows:

[0089] The features X4 of the C2PSA module of the input backbone network for pixel-level spatial self-attention mechanism are divided into the skip connection path features X c2_skip and the self-attention processing path features X att by the Split operation, and the formula is as follows:

[0090] X c2_skip ,X att =Split(Conv adaptation (X4));

[0091] In the self-attention processing path features, the channel dimension of the feature tensor output by the low-rank adaptive convolution is divided into three parts, and the spatial dimension of the feature tensor output by the low-rank adaptive convolution in each part is adjusted to generate the query matrix Q, the key matrix K, and the value matrix V, and the specific formula is as follows:

[0092] Q,K,V=ReshapeAndSplit(Conv adaptation (X att ));

[0093] where ReshapeAndSplit(·) is the operation of feature dimension segmentation and dimension adjustment;

[0094] The self-attention Attention module of the C2PSA module of the backbone network for pixel-level spatial self-attention mechanism adopts the Softmax function to calculate the self-attention weight, and the fusion features are obtained by fusing the output of the value matrix V through the residual connection, and the specific formula is as follows:

[0095]

[0096] where Attention(·) is the self-attention module, which includes multiple self-attention heads to capture the feature correlation of different subspaces, and Attention(X att ) represents the pixel-level spatial self-attention modeling of the self-attention processing path features X att by the self-attention mechanism with multiple self-attention heads, and d k is the dimension of the self-attention head;

[0097] The fusion of the self-attention features of the C2PSA module for pixel-level spatial self-attention mechanism is performed in the PSABlock module. The output of the self-attention Attention module is connected in residual manner with the original features input to the PSABlock module, so as to retain the structural information of the original features. The features after residual fusion are further processed by low-rank adaptive convolution, and the formula is as follows:

[0098] PSABlock(X att )=Conv adaptation (X att +Attention(X att ));

[0099] wherein, PSABlock(·) is a fusion module of self-attention features, and PSABlock(X att ) represents that the self-attention feature fusion is performed on the self-attention processing path feature X att in the form of residual connection;

[0100] The features processed by the self-attention mechanism are spliced with the skip connection path feature X c2_skip , and the final output feature is obtained by low-rank adaptive convolution, that is:

[0101] X' out =Conv adaptation (Concat[PSABlock N (X att )+X c2_skip ]);

[0102] wherein, PSABlock N (X att ) represents that the self-attention feature fusion PSABlock module is applied to the self-attention processing path feature X att by N times of iteration to perform progressive feature optimization.

[0103] Step 2: The trainable weight increment matrix △W is approximated by a low-rank decomposition method.

[0104] Step 2.1: The structure of the LoRA method is shown in Figure 3 (a). The trainable weight increment matrix △W is reconstructed by low-rank decomposition of the pre-trained weight matrix W. It is assumed that W ∈ R λ×γ , and △W is composed of a low-rank matrix B and a low-rank matrix A, that is, △W = B·A, wherein the low-rank matrix B ∈ R λ×r , the low-rank matrix A ∈ R r×γ , and the trainable weight increment matrix △W ∈ R λ×γ. In the training process, only the low-rank matrix B and the low-rank matrix A are updated, γ is the input channel number of the LoRA method, and λ is the output channel number of the LoRA method. The low-rank matrix B is initialized as a full zero matrix, and the low-rank matrix A is initialized randomly using a Gaussian distribution, so as to ensure that the pre-training weight matrix W remains in a frozen state in the initial stage, and only the low-rank matrix B and the low-rank matrix A are updated to realize fine-tuning of parameters.

[0105] Step 2.2: The structure of the DoRA method is as shown in Figure 3 (b), the pre-training weight matrix W is decomposed into an independently controllable amplitude vector m and a low-rank adaptive direction matrix E, the independently controllable amplitude vector m ∈ R 1×i , and the low-rank adaptive direction matrix E ∈ R o×i , i is the input channel number of the DoRA method, o is the output channel number of the DoRA method, and only the low-rank adaptive direction matrix E is optimized, and the specific formula is:

[0106]

[0107] , wherein △E is an incremental direction matrix, the incremental direction matrix △E is parameterized as the product of the low-rank matrix B and the low-rank matrix A, and ||·|| F c represents a vector norm of a matrix calculated column by column. The low-rank matrix B is initialized as a full zero matrix, and the low-rank matrix A is initialized randomly using a Gaussian distribution, only the low-rank matrix B and the low-rank matrix A are updated in the training process, and the pre-training weight W and the amplitude vector m remain frozen.

[0108] Step 2.3: The structure of the PiSSA method is as shown in Figure 3 (c), the pre-training weight W is decomposed by singular value decomposition, and the pre-training weight matrix W is expressed as W = U·S·Q T , wherein U and Q represent singular vectors with orthogonal columns, S is a diagonal matrix containing singular values, by setting a rank parameter r, the first r largest singular values and the left singular vectors and right singular vectors corresponding to the diagonal matrix S are reserved, and the first r largest singular values and the singular vectors corresponding to the principal component matrix W pri are selected as res . W pri = A·B is composed of the low-rank matrix A and the low-rank matrix B, and W pri = A·B, the low-rank matrix low-rank matrix , and the residual matrix

[0109] The low-rank matrix B and the low-rank matrix A are initialized using the first r principal singular values and singular vectors, and the residual matrix W resinitialized by the remaining singular values and singular vectors and kept frozen throughout the fine-tuning process. This initialization ensures that the PiSSA method is consistent with the pre-trained weights W in the initial stage, thus preserving the initial performance of the model.

[0110] Step 3: According to the technical differences of the low-rank parameterization method, adjust the rank parameter r to balance the efficiency and detection performance:

[0111] According to the initialization strategy in step 2, different parameter efficient fine-tuning methods are ensured to maintain the frozen state of the pre-trained weight matrix W in the initial stage, and only through trainable parameters to achieve efficient adjustment. In LoRA, DoRA and PiSSA methods, adjusting the rank parameter r directly affects the parameter quantity, expression ability and task performance. All three control the size of trainable parameters through the rank parameter r. When r is much smaller than the minimum of the input channel number and the output channel number, the parameter quantity can be compressed to less than 1% of the full parameter fine-tuning.

[0112] Step 4: Low-rank adaptive optimization is performed on the modules contained in the backbone network of the YOLOv11 model, and the specific principles and effects of the improved key modules and convolutional layers are:

[0113] The C3K2 module contains multiple convolutional layers, bottleneck structures, splitting, splicing and addition operations. Through the splitting operation, the features input into the C3K2 module are divided into skip connection path features and depth separable convolution processing path features. As shown in Figure 4 The low-rank adaptive Adaptation layer is added to the convolutional layer in the structure of step 1.2, and the low-rank adaptive convolution method is used to optimize the multi-branch feature fusion method, allowing the model to fine-tune the features related to the new task without significantly increasing the number of parameters. This makes the C3K2 module not only maintain the original function, but also reduce the computational complexity and memory occupation, more flexibly adapt to the new task demand, improve the efficiency and accuracy of feature extraction, and enhance the generalization ability and training efficiency of the model.

[0114] The SPPF module is used to process different scale features, containing convolutional layers, multiple maximum pooling layers and splicing operations, and uses a multi-level maximum pooling method to construct different scale features. As shown in Figure 5 The low-rank adaptive Adaptation layer is added to the convolutional layer in each pooling branch, so that the SPPF module can optimize with fewer parameters when fusing multi-scale features. This not only retains its original multi-scale feature capture ability, but also improves the generalization ability of the model on new data sets, while reducing the computational burden.

[0115] The C2PSA module generates a query matrix, a key matrix and a value matrix using a low-rank adaptive convolution method, constructs a self-attention mechanism that is easy to dynamically adjust, enhances the spatial relationship in the feature map, and weights each pixel in the feature map to improve the precision of the feature representation. As shown in FIG. 13, a low-rank adaptive Adaptation layer is introduced in the convolution layer of the C2PSA module, and the model can more accurately adjust the attention mechanism to pay more attention to the features related to the new task and better adapt to the requirements of the new task. Figure 6

[0116] Step 5: During the training process, since the pre-training weight matrix W remains in a frozen state, only the low-rank matrix A and the low-rank matrix B are updated. Through continuous iterative training, the values of the low-rank matrix A and the low-rank matrix B are adjusted, so that the trainable weight increment matrix ΔW can better approximate the original weight update, thereby achieving efficient adjustment of the YOLOv11 pre-training model without significantly changing the structure of the pre-training model. During the training process, the comprehensive loss L cls , the bounding box regression loss L reg and the target confidence loss L conf are continuously minimized to form a comprehensive loss L total , and the comprehensive loss L total = L cls + L reg + L conf , and the parameters of the low-rank matrix A and the low-rank matrix B are continuously optimized, so that the adjusted model has better performance on the new task.

[0117] Step 5.1: The training process of the low-rank adaptive-based YOLO model parameter efficient fine-tuning method includes the following steps:

[0118] Step 5.1.1: Load the pre-trained YOLOv11 target detection model, and determine the C3K2 module, the SPPF module and the C2PSA module that need to be improved, as well as the convolution layers of the neck network and the head network.

[0119] Step 5.1.2: Add a low-rank adaptive Adaptation layer in the convolution layers of the above-mentioned C3K2 module, SPPF module and C2PSA module that need to be improved, construct a low-rank matrix A and a low-rank matrix B, and initialize the low-rank matrix B and the low-rank matrix A, and set the rank parameter r = 4.

[0120] Step 5.1.3: Set the training parameters, including selecting a suitable loss function, an optimizer, an initial learning rate setting, etc.

[0121] ​Step 5.1.4: During the training process, the pre-trained weights W are frozen, and only the low-rank matrix A and the low-rank matrix B are updated. By continuously iterating the training, the values of the low-rank matrix A and the low-rank matrix B are adjusted to minimize the loss function, achieving efficient adjustment of the pre-trained model.

[0122] Step 5.2: The target detection process, first input the input image into the adjusted YOLOv11 target detection model. The model extracts features from the image through the improved backbone network key module and convolutional layer, and adjusts the features using the low-rank matrix in the low-rank adaptive Adaptation layer. After processing by the neck network and head network parts, the target detection result is output, including the class and position information of the target.

[0123] The following will refer to the accompanying Figure 1 The low-rank adaptive parameter efficient fine-tuning embodiment based on the YOLO target detection model proposed by the present application is described. The target detection in this embodiment includes two parts: the low-rank adaptive Adaptation layer and the improved YOLO target detection model. Specifically, the low-rank adaptive parameter efficient fine-tuning method based on the YOLO model proposed by the present application is further described. The experimental steps and experimental preparation of this example specifically include:

[0124] Step 5.2.1: Perform target detection experiments on the COCO (Common Objects in Context) dataset. This dataset contains over 330,000 images, each with multiple object annotations, covering 80 different object categories such as people, animals, vehicles, furniture, etc., suitable for training and evaluation of target detection tasks. Considering that the model training will load the pre-trained weights based on the COCO2017 training set, training is performed using the COCO2014 training set, and 30% (24832 images) of the data are randomly selected as experimental data, while the COCO2017 validation set (4952 images) and test set (3000 images) are used for validation. This step aims to evaluate the impact of different rank parameters r on model performance, and compare the differences in parameter quantity, detection accuracy, etc. between the improved model and the original YOLO model.

[0125] Step 5.2.2: For the improved YOLOv11 model, the input image needs to be preprocessed, including scaling, normalization, etc. to ensure that the image data meets the model input requirements. At the same time, set the training hyperparameters, such as batch size 16, training rounds 50, to provide stable conditions for model training.

[0126] Step 5.2.3: The pre-processed image is input into the improved YOLOv11 model. During the training process, the model continuously adjusts the parameters of the low-rank matrix A and the low-rank matrix B, so that the model can better adapt to the characteristics of the new data set.

[0127] Step 5.2.4: During the training process, the loss value, detection accuracy, and other indicators of the model are continuously recorded, and the training effect of the model is evaluated by analyzing these indicators. After the training is completed, the model is comprehensively evaluated using the test set, and indicators such as the average precision mean (mAP) of the model are calculated. To accurately evaluate the performance of the model, the embodiment uses evaluation indicators such as precision P, recall R, mean average precision (mAP), model parameter size S (unit: MB), etc.

[0128] The effectiveness of the method described in this embodiment is verified by the following performance analysis:

[0129] 1. Parameter analysis experiment

[0130] As shown in Table 1, by comparing the number of parameters of the improved model and the original YOLOv11 model, it is found that the trainable parameters of the improved model are significantly reduced. For example, when the rank parameter r = 4, the trainable parameters of the YOLOv11 model using the PiSSA method are only 0.30 MB, which is 1.56% of the original YOLOv11 full parameter fine-tuning (FPFT) of 19.18 MB, even 46.15% of the LoRA method of 0.65 MB, and 41.10% of the DoRA method of 0.73 MB, significantly reducing the storage and computing cost of the model.

[0131] Table 1 Experimental results of YOLOv11 under different low-rank adaptive parameter efficient fine-tuning methods

[0132]

[0133] 2. Detection accuracy experiment

[0134] The experimental results on the COCO data set show that the improved model has a significant improvement in detection accuracy. From the comparison of different detection models, the present invention is significantly better than the traditional full parameter fine-tuning YOLO model in various indicators. Specifically, the precision of the model reaches 71.35%, which is 4.18% higher than the traditional full parameter fine-tuning; the mAP @0.5:0.95 reaches 49.90%, which is 7.03% higher than the traditional full parameter fine-tuning. These results show that the YOLOv11 model provided by the present invention has higher detection accuracy and stronger generalization ability in complex scene target detection tasks.

[0135] Table 2 Experimental results of different models under the COCO dataset

[0136]

[0137] From the comparison of different detection models, the present application also performs well in model size. Although the low-rank adaptive parameter fine-tuning technology is introduced, the model size is only 19.5MB, which is almost equivalent to the 19.2MB model parameter amount of the benchmark, less than the 24.7MB of YOLOv8m and the 23.9MB of YOLOv5m, which shows that the improved model significantly reduces the computational resource demand of the model while maintaining high detection accuracy, especially suitable for resource-constrained edge computing devices and real-time detection scenarios.

[0138] 3. Experiment on the influence of different rank parameters on model performance

[0139] Table 3 shows the performance comparison of the parameter efficient fine-tuning methods (LoRA, DoRA, PiSSA) of the present application under different rank parameter r settings. The experimental results show that different rank parameters have a significant impact on model performance, and low rank (such as r = 4) is more suitable for parameter efficient fine-tuning tasks, which can reduce the amount of calculation while maintaining high detection performance.

[0140] The PiSSA method performs best under low rank, and when the rank parameter is 4, the precision of the PiSSA method is 71.35%, the recall rate is 61.62%, the mAP @0.5 is 66.69%, and the mAP @0.5:0.95 is 49.90%. But as the rank parameter increases, its precision, recall rate and mAP value all decrease slightly, but it is still better than the LoRA method and the DoRA method, highlighting its balance between parameter efficiency and performance. For example, when the rank parameter is 16, the precision of the PiSSA method is 69.77%, the recall rate is 60.70%, the mAP @0.5 is 65.61%, and the mAP @0.5:0.95 is 49.27%.

[0141] The LoRA method has slightly improved in some indicators under high rank, such as when the rank parameter is 16, its mAP @0.5:0.95 reaches 50.01%, which is suitable for different resource constraint scenarios. The performance of the DoRA method is relatively low, and it further decreases as the rank parameter increases, indicating that it is sensitive to high rank and is limited by the stability of the decomposition optimization strategy. When the rank parameter increases from 4 to 16, the mAP @0.5 of the DoRA method decreases from 63.02% to 61.56%, and the mAP @0.5:0.95 decreases from 46.56% to 45.13%.

[0142] Table 3 Experimental results of high-efficiency fine-tuning method of YOLOv11 under different ranks

[0143]

[0144] The application is described in detail above through specific examples. These examples are only exemplary and cannot be used to define the scope of the application. The protection scope of the application is defined by the content of the claims. Whether the technical solutions proposed by the application are directly used, or the technical solutions with similar functions and the same technical effects are designed by the technical personnel in the field under the inspiration of the technical solutions of the application without deviating from the essence and protection scope of the application, or reasonable equivalent changes and improvements of the protection scope of the application are made, etc., they should all be included in the patent protection scope of the application.

Claims

1. A low-rank adaptive-based YOLO model parameter efficient fine-tuning method, characterized in that, The method comprises the following steps: Step 1: embedding low-rank adaptive layers in convolutional layers of a backbone network, a neck network and a head network of a YOLO model, the low-rank adaptive layers generating a trainable weight increment matrix △W by performing low-rank matrix decomposition on a pre-training weight matrix W of the YOLO model, wherein the pre-training weight matrix W is a matrix of a convolutional layer of the YOLO model, and the low-rank adaptive layers are generated by performing low-rank matrix decomposition on the pre-training weight matrix W of the YOLO model C out is a number of output channels of the convolutional layer, C in is a number of input channels of the convolutional layer, and k is a size of a convolution kernel, the number of output channels C out of the convolutional layer, the number of input channels C in of the convolutional layer, and the size k of the convolution kernel are all positive integers, the trainable weight increment matrix △W = B·A, wherein B is a low-rank matrix and A is a low-rank matrix and r is a rank parameter, the rank parameter r is a positive integer, and the rank parameter r is much smaller than the number of output channels C out of the convolutional layer and the number of input channels C in of the convolutional layer, the trainable weight increment matrix The backbone network of the YOLO model comprises a C3K2 module for multi-branch feature fusion, an SPPF module for multi-scale feature extraction, and a C2PSA module for pixel-level spatial self-attention mechanism; the low-rank adaptive layers constitute a parameter efficient fine-tuning module, the parameter efficient fine-tuning module is integrated in the convolutional layers of the YOLO model, the C3K2 module for multi-branch feature fusion, the SPPF module for multi-scale feature extraction, and the C2PSA module for pixel-level spatial self-attention mechanism, and the parameter efficient fine-tuning module is used for dynamically regulating the trainable weight increment matrix △W. Step 2: approximating the trainable weight increment matrix △W by a low-rank decomposition method, the method including one of LoRA method, DoRA method or PiSSA method; wherein the LoRA method reconstructs the weight increment matrix by low-rank matrix product, the DoRA method decouples the pre-training weight matrix W into independently controllable amplitude vector and low-rank adaptive direction matrix by amplitude-direction decomposition, and the PiSSA method retains the principal component matrix of the pre-training weight matrix W based on singular value decomposition; Step 3: freezing the pre-training weight matrix W, only updating the low-rank matrix B and the low-rank matrix A, and balancing parameter efficiency and detection performance by adjusting the rank parameter r, so that the trainable parameter quantity is reduced to less than 1% of the trainable parameter quantity of full-parameter fine-tuning based on the pre-training weight matrix W; Step 4: performing low-rank adaptive optimization on the modules contained in the backbone network of the YOLO model, in the C3K2 module for multi-branch feature fusion, the features input into the C3K2 module for multi-branch feature fusion are divided into skip connection path features and depth separable convolution processing path features by Split operation, and low-rank adaptive convolution method is used to optimize multi-branch feature fusion; in the SPPF module for multi-scale feature extraction, different scale features are constructed by using multi-level maximum pooling method, and low-rank adaptive convolution is embedded in the branch convolution layer of the maximum pooling method to enhance cross-scale feature interaction; in the C2PSA module for pixel-level spatial self-attention mechanism, low-rank adaptive convolution method is used to generate query matrix, key matrix and value matrix to construct self-attention mechanism which is easy to dynamically adjust; Step 5: training the YOLO model by using a comprehensive loss function, the comprehensive loss function including classification loss, bounding box regression loss and target confidence loss, the low-rank matrix B and the low-rank matrix A are optimized by minimizing the comprehensive loss function, and an efficient fine-tuning model suitable for target detection is output.

2. The low-rank adaptive-based YOLO model parameter efficient fine-tuning method according to claim 1, characterized in that, The convolution method of the low-rank adaptive layer in step 1 satisfies: The pre-training weight matrix W remains frozen, the pre-training weight matrix W generates the trainable weight increment matrix △W by low-rank matrix decomposition method, the shape of the trainable weight increment matrix △W is consistent with the shape of the pre-training weight matrix W, the trainable weight increment matrix △W is superimposed on the pre-training weight matrix W in residual form to form an adjusted weight matrix W'=W+△W, and the adjusted weight matrix W' is used for convolution calculation on the features X input into the convolution layer, that is: Conv adaptation (X) = Conv(X, W'); where Conv adaptation (·) denotes a convolution operation of a low-rank adaptive layer, Conv adaptation (X) denotes applying a convolution operation of a low-rank adaptive layer to the input feature X of the convolution layer, and Conv(X, W') denotes performing a convolution operation on the input feature X of the convolution layer with the adjusted weight W'.

3. The low-rank adaptive-based YOLO model parameter efficient fine-tuning method according to claim 1, characterized in that, The specific implementation of the C3K2 module for multi-branch feature fusion of the backbone network in step 1 is: The features X1 input into the C3K2 module for multi-branch feature fusion of the backbone network are segmented, and the formula is: X c3_skip ,X proc = Split( Conv adaptation (X1)); wherein Split(·) represents splitting the feature X1 input to the C3K2 module for multi-branch feature fusion of the backbone network into a skip connection path feature X c3_skip and a depth separable convolution processing path feature X proc ; The formula for processing the features X2 input into the C3KBlock module in the C3K2 module for multi-branch feature fusion of the backbone network is: C3KBlock(X2) = Concat[Bottleneck 2 (Conv adaptation (X2))+ Conv adaptation (X2)]; wherein C3KBlock(X2) represents multi-branch feature fusion on the input feature X2 of the C3KBlock module, Bottleneck 2 (·) represents twice cascaded low-rank adaptive convolution and nonlinear activation operation, and Concat(·) represents feature concatenation operation along the channel dimension; The fused feature after the low-rank adaptive convolution is fused with the spliced feature to generate an output feature map X out The backbone network is used for a C3K2 module feature fusion formula for multi-branch feature fusion. X out = Conv adaptation (Concat[X c3_skip ,C3KBlock N (X proc )]) ; wherein C3KBlock N (X proc ) denotes the feature X processed by the depthwise separable convolution processing path proc The iteration processing of the C3KBlock module is performed N times, where N is a positive integer.

4. The low-rank adaptive-based YOLO model parameter efficient fine-tuning method according to claim 1, characterized in that, The specific implementation of the SPPF module of the backbone network for multi-scale feature extraction in step 1 is as follows: In each pooling branch of the SPPF module of the backbone network for multi-scale feature extraction, low-rank adaptive convolution is embedded, different scale features are captured through the max-pooling operation, and the multi-scale features are spliced and fused through low-rank adaptive convolution. The improved multi-scale feature fusion formula is as follows: X out = Conv adaptation (Concat[X3, MaxPool1(X3), MaxPool2(X3), MaxPool3(X3)]); where MaxPool s (·) is a max pooling with a stride of 1 for s consecutive times, s is a positive integer, MaxPool s (X3) represents a max pooling operation with a stride of 1 for s consecutive times on the feature X3 input to the SPPF module of the backbone network for multi-scale feature extraction.

5. The low-rank adaptive-based YOLO model parameter efficient fine-tuning method according to claim 1, characterized in that, The specific implementation of the C2PSA module of the backbone network for pixel-level spatial self-attention mechanism in step 1 is as follows: The input of the backbone network is used for the feature X4 of the C2PSA module of the pixel-level spatial self-attention mechanism, which is divided into a skip connection path feature X c2_skip and a self-attention processing path feature X att , and the formula is: X c2_skip ,X att = Split( Conv adaptation (X4)); In the self-attention processing path feature, the channel dimension of the feature tensor output by the low-rank adaptive convolution is divided into three parts, and the spatial dimension of the feature tensor output by the low-rank adaptive convolution in each part is adjusted to generate the query matrix Q, the key matrix K, and the value matrix V. The specific formula is as follows: Q, K, V = ReshapeAndSplit(Conv adaptation (X att )); where ReshapeAndSplit(·) is the operation of feature dimension segmentation and dimension adjustment; The self-attention Attention module of the C2PSA module of the backbone network for pixel-level spatial self-attention mechanism adopts a Softmax function to calculate the self-attention weight, and the fusion feature is obtained by fusing the output of the value matrix V through a residual connection. The specific formula is as follows: wherein Attention(·) is a self-attention module, which contains multiple self-attention heads to capture the feature correlation of different subspaces, Attention(X att ) represents pixel-level spatial self-attention modeling of the self-attention processing path feature X att by using a self-attention mechanism with multiple self-attention heads, d k is the dimension of the self-attention head. In the fusion PSABlock module of the self-attention feature of the C2PSA module of the backbone network for pixel-level spatial self-attention mechanism, the residual connection is performed between the output of the self-attention Attention module and the original feature input into the fusion PSABlock module, so as to retain the structure information of the original feature. The residual fused feature is further processed through low-rank adaptive convolution. The formula is as follows: PSA Block (X att ) = Conv adaptation (X att + Attention (X att )) ; wherein PSABlock(·) is a self-attention feature fusion module, and PSABlock(X att ) represents that the self-attention feature fusion is performed on the self-attention processing path feature X att in the form of a residual connection. concatenate the features processed by the self-attention mechanism with the features of the skip connection path X c2_skip concatenate the features processed by the self-attention mechanism with the features of the skip connection path X concatenate the features processed by the self-attention mechanism with the features of the skip connection path X X' out = Conv adaptation (Concat[PSABlock N (X att )+X c2_skip ]) ; wherein PSABlock N (X att ) denotes the progressive feature optimization of the self-attention processing path feature X att by applying the fusion PSA Block module of the self-attention feature through N iterations.

6. The low-rank adaptive-based YOLO model parameter efficient fine-tuning method according to claim 1, characterized in that, The specific implementation of the low-rank decomposition method in step 2 is as follows: LoRA method: In the training process, the low-rank matrix B is initialized as a zero matrix, and the low-rank matrix A is randomly initialized as a Gaussian matrix, so that the pre-training weight matrix W remains unchanged in the initial stage, and only the low-rank matrix B and the low-rank matrix A are updated. The trainable weight increment matrix ΔW = B·A; DoRA method: decomposing the pre-trained weight matrix W into an independently controllable magnitude vector m and a low-rank adaptive direction matrix E, the independently controllable magnitude vector m e R 1×i , the low-rank adaptive direction matrix E e R o×i , i is the input channel number of the DoRA method, o is the output channel number of the DoRA method, only the low-rank adaptive direction matrix E is optimized, and the specific formula is: where ΔE is an incremental direction matrix, which is parameterized as the product of the low-rank matrix B and the low-rank matrix A, ||·||F c denotes the vector norm of a column-wise computed matrix; PiSSA method: decomposing the pre-trained weight matrix W by singular value decomposition, the pre-trained weight matrix W is expressed as W = U · S · Q T wherein U and Q represent singular vectors with orthogonal columns, S is a diagonal matrix containing singular values, by setting the rank parameter r, the first r largest singular values and the left and right singular vectors corresponding to the diagonal matrix S are retained to initialize the low-rank matrix B and the low-rank matrix A, the low-rank matrix the low-rank matrix The singular vectors corresponding to the remaining singular values constitute a residual matrix, and the residual matrix is frozen.

7. A target detection system characterized by, The module of the low-rank adaptive parameter efficient fine-tuning method of claim 1 to 6 is integrated with the convolution layer, C3K2 module, SPPF module, or C2PSA module of the YOLO model through a plug-in interface, and supports dynamic task parameter switching.

8. An electronic device comprising a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method of any one of claims 1 to 6.

Images