Large model fine-tuning acceleration method and device based on convolution

Abstract

The present disclosure relates to the field of artificial intelligence, and provides a convolution-based large model fine-tuning acceleration method and device, the method comprising: obtaining a pre-trained large model and fine-tuning training data; introducing a convolution module into the self-attention module of the large model to obtain an initial fine-tuning model; under the condition of fixing the parameters of the large model, using the fine-tuning training data to train the initial fine-tuning model to obtain a fine-tuning training model; based on the fine-tuning training model, a fine-tuned large model is obtained. The present disclosure can solve the problems of high training memory usage, slow training speed, and pre-training knowledge forgetting that may be encountered in fine-tuning training of large models, realize the accelerated learning of the model on the fine-tuning training data, and optimize the performance of the model in the task or field to which the fine-tuning training data belongs.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a large model fine-tuning acceleration method and device based on convolution. Background Art

[0002] Large Model (LM) refers to machine learning models with large-scale parameters and complex computational structures. These models are usually built from deep neural networks and have billions or even hundreds of billions of parameters. Large models have better model expression and prediction performance and can handle more complex tasks and data. Large models are widely used in various fields, including in natural language processing (NLP), computer vision (CV), visual question answering (VQA) and other technologies.

[0003] Pre-trained large models are usually pre-trained based on public data or general scenario data. They perform poorly on specific tasks or in certain vertical fields, and usually require the collection of domain knowledge or application data for further fine-tuning and training.

[0004] However, due to the huge number of parameters of large models, which can range from billions to hundreds of billions, directly fine-tuning large models may encounter problems such as high training memory usage, slow training speed, and forgetting of pre-trained knowledge. Therefore, it is necessary to explore efficient fine-tuning methods for large models. Summary of the invention

[0005] The present disclosure provides a convolution-based large model fine-tuning acceleration method and device to at least solve or alleviate the problems of high training memory usage, slow training speed, and forgotten pre-training knowledge that may be encountered in fine-tuning training of large models in related technologies. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a convolution-based large model fine-tuning acceleration method is provided, the method comprising: obtaining a pre-trained large model and fine-tuning training data, wherein the large model comprises a self-attention module, the fine-tuning training data comprises training samples and annotation information for the training samples, the training samples comprise image data, and the large model is a visual large model or a multimodal large model; introducing a convolution module into the self-attention module to obtain an initial fine-tuning model, wherein the number of channels of the convolution module is the same as the number of attention heads of the self-attention module, and the number of convolution groups of the convolution module is the same as the number of attention heads; under the condition of fixing the parameters of the large model, using the fine-tuning training data to train the initial fine-tuning model to obtain a fine-tuning training model; based on the fine-tuning training model, obtaining a fine-tuned large model.

[0007] Optionally, the convolution module convolves the value vector from the self-attention module and superimposes the output convolution result onto the output result of the self-attention module, and the fine-tuned large model includes a static attention module corresponding to the trained convolution module.

[0008] Optionally, the convolution module includes an expansion unit, a head-to-head convolution unit, a batch normalization unit and a folding unit, wherein the expansion unit receives the value vector output from the self-attention module, and transforms the sequence length in the value vector into a size after scaling the height and width of the image data; the head-to-head convolution unit receives the image data output from the expansion unit, and performs depth-separable convolution on the image data to obtain head-to-head convolution features; the batch normalization unit receives the head-to-head convolution features output from the head-to-head convolution unit, normalizes the head-to-head convolution features to obtain normalized features; the folding unit receives the normalized features output from the batch normalization unit, transforms the normalized features into sequence features, and superimposes the sequence features on the calculation results of the self-attention module.

[0009] Optionally, the fine-tuned large model is obtained by: fusing the parameters of the trained convolution module into the self-attention module, and removing the trained convolution module from the fine-tuned training model to obtain the fine-tuned large model; or, using the fine-tuned training model as the fine-tuned large model.

[0010] Optionally, the parameters of the trained convolution module include bias parameters and weight parameters, wherein the parameters of the trained convolution module are fused into the trained self-attention module in the following manner: the weight parameters are fused with the self-attention map in the self-attention module by superposition; the bias parameters are fused into the linear projection layer in the self-attention module.

[0011] Optionally, the weight parameters are fused with the self-attention map in the following manner: flattening the weight parameters from the convolution sliding window of the convolution module to obtain the flattened weight parameters; extracting the parameters in the self-attention module from the flattened weight parameters to obtain a static self-attention module; superimposing the output of the static self-attention module on the output result of the self-attention map, wherein the bias parameters are fused into the linear projection layer in the following manner: fusing the bias parameters with the original weight parameters of the linear projection layer to obtain a fusion result; superimposing the fusion result with the original bias parameters of the linear projection layer, and using the superimposed bias parameters as the bias parameters of the fused linear projection layer.

[0012] According to a second aspect of the present disclosure, a convolution-based large model fine-tuning acceleration device is provided, the device comprising: an acquisition unit, configured to acquire a pre-trained large model and fine-tuning training data, wherein the large model comprises a self-attention module, the fine-tuning training data comprises training samples and annotation information for the training samples, the training samples comprise image data, and the large model is a visual large model or a multimodal large model; a first determination unit, configured to introduce a convolution module into the self-attention module to obtain an initial fine-tuning model, wherein the number of channels of the convolution module is the same as the number of attention heads of the self-attention module, and the number of convolution groups of the convolution module is the same as the number of attention heads; a training unit, configured to train the initial fine-tuning model using the fine-tuning training data under the condition of fixing the parameters of the large model to obtain a fine-tuning training model; a second determination unit, configured to obtain a fine-tuned large model based on the fine-tuning training model.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein when the instructions executable by the processor are executed by the processor, the processor is prompted to execute the convolution-based large model fine-tuning acceleration method according to the present disclosure.

[0014] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the convolution-based large model fine-tuning acceleration method described in the present disclosure.

[0015] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, which, when executed by at least one processor, implement the convolution-based large model fine-tuning acceleration method according to the present disclosure.

[0016] The technical solution provided by the present disclosure brings at least the following beneficial effects:

[0017] According to the fine-tuning scheme of the large model disclosed in the present invention, a convolution module can be introduced into the self-attention module of the pre-trained large model, wherein the number of channels of the convolution module is the same as the number of attention heads of the self-attention module, and the number of convolution groups of the convolution module is the same as the number of attention heads. In this way, fine-tuning training can be performed based on the initial fine-tuning model after the introduction of the convolution module under the condition of fixing the original parameters of the large model, so that the convolution module is adjusted through training without changing the original parameters in the pre-trained large model to obtain a fine-tuning training model to accelerate the fine-tuning of the large model, avoid the problems of high training memory usage, slow training speed, and forgetting of pre-training knowledge caused by adjusting a large number of original parameters in the large model, realize accelerated learning of the model on the fine-tuning training data, and optimize the performance of the model in the task or field to which the fine-tuning training data belongs.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0020] Figure 1 It is a schematic flowchart of a convolution-based large model fine-tuning acceleration method according to an exemplary embodiment of the present disclosure.

[0021] Figure 2 is a schematic block diagram of an initial fine-tuning model in a convolution-based large model fine-tuning acceleration method according to an exemplary embodiment of the present disclosure.

[0022] Figure 3 It is a schematic flowchart of the steps of fusing weight parameters with self-attention maps in a convolution-based large model fine-tuning acceleration method according to an exemplary embodiment of the present disclosure.

[0023] Figure 4 It is a schematic flowchart of the steps of fusing bias parameters into a linear projection layer in a convolution-based large model fine-tuning acceleration method according to an exemplary embodiment of the present disclosure.

[0024] Figure 5 Detailed description is a schematic block diagram of a fine-tuned large model in a convolution-based large model fine-tuning acceleration method according to an exemplary embodiment of the present disclosure.

[0025] Figure 6 It is a schematic block diagram of a convolution-based large model fine-tuning acceleration device according to an exemplary embodiment of the present disclosure.

[0026] Figure 7 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the present application will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0029] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three types of parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three types of parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0030] As mentioned earlier, due to the large number of parameters of large models, when fine-tuning the model for specific tasks or processing fields, problems such as high training memory usage, slow training speed, and forgetting of pre-training knowledge may be encountered.

[0031] For example, for a pre-trained large model based on a Tranformer network, it can be pre-trained based on public data or general scenario data. In actual applications, it may be necessary to optimize the current application scenario based on such a pre-trained large model so that the model can better adapt to the current application scenario. For example, for a large model involving image processing, image data in a certain field or scenario may not be specifically specified during pre-training. In actual applications, the model is more required to focus on processing images involving certain content. For example, in the industrial field, it can identify whether the position of production equipment in the image is correct; in the transportation field, it can identify whether vehicles on the road are driving illegally, etc. Therefore, for a pre-trained general large model, it can be further trained using fine-tuning training data to obtain a model that is more suitable for the scenario to which the fine-tuning training data belongs.

[0032] However, compared with image data, fine-tuning training for pure language / text is easier to implement. Therefore, in the related technology, there are some fine-tuning schemes for large natural language models. For large models that process image data, such as large visual models or large multimodal models, due to the larger number of parameters, fine-tuning the model will generate greater computing memory usage and slower computing speed, so it is difficult to achieve efficient fine-tuning training.

[0033] In view of the above problems, the exemplary embodiments of the present disclosure propose a convolution-based large model fine-tuning acceleration method, a convolution-based large model fine-tuning acceleration device, an electronic device, a computer-readable storage medium and a computer program product, which can solve or at least alleviate the above problems.

[0034] In a first aspect of an exemplary embodiment of the present disclosure, a convolution-based large model fine-tuning acceleration method is provided.

[0035] The convolution-based large model fine-tuning acceleration method according to the exemplary embodiment of the present disclosure can be applied to model training or optimization scenarios. For example, fine-tuning training software can be loaded on the user terminal, and the user can input model fine-tuning instructions on the user terminal. The user terminal can obtain a fine-tuned large model by executing the convolution-based large model fine-tuning acceleration method according to the exemplary embodiment of the present disclosure.

[0036] Specifically, the user terminal can obtain a pre-trained large model and fine-tuning training data, which can be provided by the user or obtained from a data source specified by the user. Here, the large model includes a self-attention module, the fine-tuning training data includes training samples and annotation information for the training samples, the training samples include image data, and the large model is a visual large model or a multimodal large model.

[0037] The user terminal can introduce a convolution module into the self-attention module to obtain an initial fine-tuning model, wherein the number of channels of the convolution module is the same as the number of attention heads of the self-attention module, and the number of convolution groups of the convolution module is the same as the number of attention heads.

[0038] The user terminal can also use the fine-tuning training data to train the initial fine-tuning model under the condition of fixing the parameters of the large model to obtain a fine-tuning training model, and can obtain a fine-tuned large model based on the fine-tuning training model.

[0039] The above-mentioned user terminal can be, for example, a tablet computer, a laptop computer, a digital assistant, etc. However, the implementation scenario of the above method is only an example scenario. The convolution-based large model fine-tuning acceleration method according to the exemplary embodiment of the present disclosure can also be applied to other application scenarios. For example, the user can request a fine-tuning model from the server through the network on the user terminal (for example, a mobile phone, a desktop computer, a tablet computer, etc.). The server can complete the fine-tuning by executing the convolution-based large model fine-tuning acceleration method according to the exemplary embodiment of the present disclosure. Here, the server can be an independent server, a server cluster, a cloud computing platform, or a virtualization center.

[0040] According to the convolution-based large model fine-tuning acceleration method of the exemplary embodiment of the present disclosure, the convolution module can be adjusted by training to obtain a fine-tuning training model to achieve fine-tuning of the large model without changing the original parameters in the pre-trained large model. This avoids the problems of high memory usage, slow training speed, and forgetting of pre-training knowledge caused by adjusting a large number of original parameters in the large model when facing a large model for processing image data. This enables the model to learn the fine-tuning training data containing image data in the scene of interest, and optimizes the performance of the model in the image processing tasks or fields to which the fine-tuning training data belongs.

[0041] The following will refer to Figures 1 to 5 An example of a convolution-based large model fine-tuning acceleration method according to an embodiment of the present disclosure is described.

[0042] like Figure 1 As shown, the fine-tuning method of the large model may include the following steps:

[0043] In step S110, a pre-trained large model and fine-tuning training data may be obtained.

[0044] Here, the large model may include a self-attention module, for example, the large model may be a model based on a Transformer network. The self-attention module may have a multi-head self-attention mechanism.

[0045] Specifically, the big model can be a visual big model for processing visual data or a multimodal big model for processing multimodal data. For example, it can be but is not limited to: an image recognition model for identifying objects in an image; a visual question-answering model for answering questions about an image, etc.

[0046] Figure 2 An example structure of a large model is shown in Figure 2 As shown, the large model may include the first normalization layer (LayerNorm1), the self-attention module, the second normalization layer (LayerNorm2) and the feed-forward neural network (FFN). The self-attention module may include the SoftMax layer, the self-attention layer (Attentionmap), the multiplication layer (Matmul) and the linear projection layer (Proj). The layer normalization layer, feed-forward neural network, SoftMax layer, self-attention layer, multiplication layer and linear projection layer here may be existing structures and will not be described here. In addition, Figure 2 The structure of the large model shown is only an example, and it may also have other structures as long as it includes a self-attention module.

[0047] The fine-tuning training data includes training samples and labeling information of the training samples, and the training samples include image data.

[0048] The fine-tuning training data may include, for example, unimodal image data or multimodal data including image data related to the target scene to which the user expects to apply the large model. For example, the training sample may be unimodal image data or multimodal data including image data, wherein the multimodal data may include image data and question data for the image data. For unimodal image data, the annotation information may be an annotation of the content in the image data (e.g., location, type, etc.); for multimodal data, the annotation information may be an answer to the question data for the image data.

[0049] Whether it is unimodal image data or image data in multimodal data, it can be of any type, form and content. For example, it can be image data related to the scene where the fine-tuned large model is to be applied. For example, in the industrial field, it can be images related to production equipment; in the transportation field, it can be images related to roads and vehicles.

[0050] As an example, fine-tuning training data can be obtained by collecting image data of the target scene (or both image data and question data for the image data) and manually annotating the collected data. Here, before manual annotation, the collected data can also be cleaned and / or enhanced.

[0051] As an example, the collected image data can be unified into the same size, for example, can be uniformly scaled to multiples of 16 to facilitate subsequent processing such as a Transformer network, and the scaled width and height of each image can be stored in the fine-tuning training dataset.

[0052] In step S120, a convolution module may be introduced into the self-attention module to obtain an initial fine-tuning model.

[0053] Here, the number of channels of the convolution module can be the same as the number of attention heads of the self-attention module, and the number of convolution groups of the convolution module can be the same as the number of attention heads. As an example, the convolution module can be constructed based on depthwise separable convolution, wherein the number of channels and the number of convolution groups can be adjusted on the basis of depthwise convolution so that they are respectively the same as the number of attention heads of the self-attention module. Here, by adjusting the number of channels and the number of convolution groups, the number of parameters of the convolution module is smaller than that of ordinary depthwise separable convolution, which can better adapt to the multi-head self-attention mechanism of large models, and has low memory usage and fast training speed during fine-tuning training.

[0054] In this step, the structure of the model can be fine-tuned by introducing a convolution operation in the self-attention module of the large model to obtain an initial fine-tuned model. In subsequent fine-tuning training, the model with fine-tuned structure can be trained.

[0055] As an example, a convolution module can be added to the output of the value vector V, query vector Q, or key vector K of the self-attention module to convolve the corresponding vectors and superimpose the convolution result on the output result of the self-attention module.

[0056] According to an embodiment of the present disclosure, the convolution module can convolve the value vector from the self-attention module, and superimpose the output convolution result on the output result of the self-attention module. By adding the convolution module to the output of the value vector of the self-attention module, after fine-tuning training, the convolution module can be converted into or merged into a static attention module. For example, the fine-tuned large model can include a static attention module corresponding to the trained convolution module. The calculation process of the static attention mechanism is simple and the calculation efficiency is high, which makes the calculation efficiency of the fine-tuned trained large model higher and reduces the inference time of the model.

[0057] In the example after the convolution module is added to the output of the value vector, such as Figure 2As shown, the convolution module may include, for example, an expansion unit (UnFold), a head-wise convolution unit (Head-Wise Conv), a batch normalization unit (BatchNorm) and a folding unit (Fold), wherein the expansion unit may be an operation before performing convolution so that the input value vector can adapt to the operation of the head-wise convolution unit, and the folding unit may be an operation after performing convolution so that the output of the head-wise convolution unit can be superimposed on the output of the self-attention module.

[0058] The expansion unit may receive the value vector output from the self-attention module and transform the sequence length in the value vector into a size scaled by the height and width of the image data.

[0059] Specifically, the value vector V of the self-attention module can be unfolded, for example, the value vector V can be permuted and reshaped. As an example, the value vector V can have the original shape The sequence data can be transformed into a sequence data with the shape Class image data, where represents the batch size for training, represents the number of heads of the self-attention module, represents the length of the tag sequence, Indicates the number of channels of the model, and It can be the size after scaling the height and width of the original image. Indicates rounding down, for example, 1 / 16 of the original height and width respectively. Here, the image-like data may refer to the size of the sequence length converted to the height and width of the original image after scaling. For example, the above N may be equal to and As an example, for the value vector V , the expansion operation of the expansion unit can be expressed by the following formula (1):

[0060] (1)

[0061] in, represents the output features of the unfolded unit, represents a value vector, represents the dimension permutation operation, where Indicates that the original dimension position Transformed to , that is, the third column (column number is "2") in the value vector is transformed to the last column of the matrix. However, the dimension permutation operation is not limited to this. As long as the sequence length can be transformed to the last column of the matrix, it can be adapted to the convolution operation in the subsequent head-to-head convolution unit. represents the shape-changing operation, which transforms the sequence length N into two dimensions, H and W, as described above.

[0062] The separate convolution unit can receive the image data output from the expansion unit, and perform depth-separable convolution on the image data to obtain separate convolution features.

[0063] Specifically, the expanded image (such as the above ) to perform convolution operations, for example, depthwise separable convolution can be performed, where the head-separated convolution unit can be a variant of depthwise separable convolution, that is, it is obtained by adjusting the number of channels and the number of convolution groups of the depthwise separable convolution, for example, making the number of channels of the convolution layer in the head-separated convolution unit the same as the number of heads of the multi-head self-attention mechanism of the self-attention module, and the number of convolution groups (groups) is equal to the number of heads of the multi-head self-attention mechanism of the self-attention module, so that lightweight convolution can be achieved. In addition, the stride (Stride) of the head-separated convolution unit can be, for example, 1, and the padding (padding) can be, for example, 1. As an example, the convolution operation of the head-separated convolution unit can be expressed by the following formula (2):

[0064] (2)

[0065] in, represents the output features of the head-wise convolutional unit, Represents the above variant convolution operation based on depthwise separable convolution.

[0066] The batch normalization unit can receive the head-convolution features output from the head-convolution unit, normalize the head-convolution features, and obtain normalized features. Here, the number of channels of the batch normalization unit can be the same as the number of heads of the multi-head self-attention.

[0067] As an example, the normalization operation of the batch normalization unit can be expressed by the following equation (3):

[0068] (3)

[0069] in, represents the normalized features output by the normalization unit, Represents a batch normalization operation.

[0070] The folding unit can receive the normalized features output from the batch normalization unit, transform the normalized features into sequence features, and superimpose the sequence features into the calculation results of the self-attention module.

[0071] Specifically, the folding unit can perform the inverse operation of the unfolding operation, mapping the image data back to sequence data, and superimposing the sequence data into the output of the self-attention module.

[0072] In this example, the folding unit can receive the normalized features output from the normalization unit, transform the normalized features into sequence features, and superimpose the sequence features onto the calculation results of the self-attention module. As an example, the folding operation of the folding unit can be expressed by the following formula (4):

[0073] (4)

[0074] in, Represents the sequence features of the folding unit output or the output of the convolution module, Represents the output result of the multi-head self-attention calculated by the self-attention module.

[0075] Through the above modules, the convolution module and the self-attention module can be connected, so that the convolution module can be introduced into the large model. However, the implementation of the convolution module is not limited to the above examples. For example, other structures can be added or some units can be replaced.

[0076] Return to reference Figure 1 In step S120, each self-attention module in the original large model can be replaced by the above-mentioned convolution module, and the parameters therein (such as convolution parameters and batch normalization parameters) are randomly initialized to obtain an initialized fine-tuning network model, that is, an initial fine-tuning model.

[0077] In step S130, the initial fine-tuning model may be trained using the fine-tuning training data under the condition of fixing the parameters of the large model to obtain a fine-tuning training model.

[0078] In this step, fine-tuning training can be performed based on the initial fine-tuning model constructed or modified as described above. During the training process, the parameters of the original large model are fixed, and the head convolution unit and batch normalization unit of the lightweight convolution module added to the original self-attention module are trained, thereby reducing the memory usage during fine-tuning of the large model, speeding up the model training speed, and achieving efficient fine-tuning training for the scenario or field to which the fine-tuning training data belongs.

[0079] Here, the training samples in the fine-tuning training data can be input into the initial fine-tuning model to obtain the prediction results, and the prediction results of the model can be compared with the labeled information to determine the prediction loss. The parameters of the model (specifically, the parameters in the convolution module) are adjusted by optimizing the prediction loss, and the prediction results of the model are continuously optimized to achieve fine-tuning training of the model.

[0080] As an example, the Adam optimizer can be used to perform fine-tuning training on the initial fine-tuning model. However, the training method of fine-tuning training is not limited to this, and the process of fine-tuning training can be similar to the existing model training process, for example, it can be similar to the pre-training process, so it will not be repeated here.

[0081] In step S140, a fine-tuned large model can be obtained based on the fine-tuned training model.

[0082] In one example, the fine-tuned training model can be used as a fine-tuned large model, thereby achieving efficient fine-tuning of the large model.

[0083] In another example, the structure of the fine-tuning training model can be simplified to achieve efficient fine-tuning of the large model while reducing the inference or prediction overhead of the fine-tuned model due to fine-tuning.

[0084] Specifically, in the fine-tuning training model, the parameters of the trained convolution module can be fused into the self-attention module, and the trained convolution module can be removed from the fine-tuning training model to obtain a large fine-tuned model. Through this parameter fusion, the impact of the convolution module introduced in the fine-tuning stage on the operation process of the fine-tuned model can be minimized, thereby improving the operation speed and efficiency of the model.

[0085] Here, the parameter fusion of the trained convolution module and the self-attention module can be achieved through mathematical operations, such as merging the parameter matrices of the convolution module and the self-attention module, extracting common factors from the parameters of the convolution module and the self-attention module, etc.

[0086] As an example, according to an embodiment of the present disclosure, an example of fusing the bias parameters and weight parameters of a convolution module is given.

[0087] Specifically, the parameters of the trained convolution module may include bias parameters and weight parameters. Here, the weight parameter may be the coefficient of the convolution kernel used to extract features in the convolution operation; the bias parameter may be a constant added to each output feature map after the convolution operation, which is used to control the range of the output value of the feature map.

[0088] by Figure 2Taking the structure shown in FIG. 1 as an example, when the convolution module includes a head-separated convolution unit and a batch normalization unit, the parameters of the batch normalization unit can be merged into the head-separated convolution unit to obtain the weight parameters and bias parameters as shown in the following equations (5) and (6), respectively:

[0089] (5)

[0090] (6)

[0091] in, represents the weight parameters of the convolution module, represents the bias parameters of the convolution module, represents the weight parameter of the head-wise convolution unit, represents the bias parameter of the head-wise convolution unit, represents the mean of the batch normalization unit, represents the variance of the batch normalization unit, represents the hyperparameters of the batch normalization unit, The role of is to prevent the denominator in the above formula from being zero. and Represents the linear transformation parameters of the batch normalization unit.

[0092] In this example, the parameters of the trained convolution module can be fused into the trained self-attention module in the following ways: the weight parameters are fused with the self-attention map in the self-attention module by superposition; the bias parameters are fused into the linear projection layer in the self-attention module.

[0093] Here, by fusing the weight parameters in an overlapping manner and fusing the bias parameters into the linear projection layer, the convolution module and the self-attention module can be fused, and only a small amount of addition calculation is added, which greatly reduces the inference or prediction overhead caused by fine-tuning.

[0094] Specifically, on the one hand, the weight parameters of the convolution module can be fused. Here, fusion is performed by superposition, which can make the model's operation in the reasoning or prediction process simpler and faster, and avoid complex operations caused by multiplication and other methods.

[0095] As an example, Figure 3 As shown, the weight parameters can be fused with the self-attention map in the following way:

[0096] In step S310, the weight parameters may be flattened from the convolution sliding window of the convolution module to obtain the flattened weight parameters.

[0097] In step S320, the parameters in the self-attention module can be extracted from the flattened weight parameters to obtain a static self-attention module.

[0098] In step S330, the output of the static self-attention module can be superimposed on the output result of the self-attention map.

[0099] Specifically, in order to merge the operations of the convolution module into addition (or superposition) operations and reduce the additional computing modules and amount of computing required for model fine-tuning, it can be considered that the sliding window calculation of the convolution is converted into a sparse matrix element-by-element multiplication, and then the multiplication distribution rate is used to advance the calculation to the value vector V. In fact, the convolution calculation can be considered as the convolution of the convolution kernel W and the value vector V, which can be expressed as in the form of Figure 2 As shown, the calculation formula during fine-tuning training is ,here, Represents the output result of the multi-head self-attention calculated by the self-attention module, For example, it can include the operations of both batch normalization unit and head-wise convolution unit (or assume that batch normalization unit has been integrated into head-wise convolution unit). The convolution parameters in are flattened from the sliding window, and the new weight parameters can be obtained , that is, the above calculation formula Transformed to , and then use the multiplication distribution rate to get the fusion result of weight parameters and self-attention map .

[0100] Here, the convolution module can be equivalent to a static self-attention module, and the output of the static self-attention module can be superimposed on the output result of the self-attention map. Figure 5 Shown with Figure 2 The fine-tuned training model based on lightweight convolution is equivalent to the fine-tuned training model shown in the figure, where the weight parameters of the convolution module can be equivalent to the static attention module (Static Attention).

[0101] Taking the weight parameter in the above example For example, the initialized static attention module can be expressed as The all-zero value of the shape, for example, can be expressed by the following formula (7):

[0102] (7)

[0103] in, represents floor division, , , , ,% means taking the remainder, express The index of the first dimension (index), Integers in the interval, and Respectively The indices of the 2nd and 3rd dimensions, , , K Indicates the convolution kernel size of the head-based convolution unit.

[0104] On the other hand, the bias parameters of the convolution module can be fused. As an example, Figure 4 As shown, the bias parameters can be fused into the linear projection layer in the following way:

[0105] In step S410, the bias parameter may be fused with the original weight parameter of the linear projection layer to obtain a fusion result.

[0106] For example, the bias parameter can be multiplied with the original weight parameter of the linear projection layer, as Figure 5 As shown, with the bias parameters in the above example For example, you can Weight parameters fused to the linear projection layer Proj The fusion result is .

[0107] In step S420, the fusion result may be superimposed with the original bias parameters of the linear projection layer, and the superimposed bias parameters are used as the bias parameters of the fused linear projection layer.

[0108] Specifically, taking the above example, the fusion result can be and the bias parameter in the linear projection layer Proj Superposition, thus, the bias parameter of the fused linear projection layer It can be expressed by the following formula (8):

[0109] (8)

[0110] In the above manner, by merging parameters and constructing static self-attention according to the shape of the self-attention map, the convolution module can be equivalent to a static attention module, so that the static attention module can be superimposed and multiplied with the value vector, further reducing the inference or prediction overhead caused by fine-tuning.

[0111] Here, it should be noted that the calculation methods shown in the above formulas are only examples, and their forms are not limited to the above formulas. For example, they can be modified, weighted, etc. according to actual needs, or other fusion methods can be used, as long as the parameters of the convolution module can be fused into the self-attention module.

[0112] According to the large model fine-tuning acceleration method based on convolution in the embodiment of the present disclosure, a large model efficient fine-tuning method based on head-wise convolution can be proposed, which can fine-tune the visual data in the visual large model or multimodal large model, and accelerate the fine-tuning training process by introducing the convolution module. In this method, by fixing the parameters of the original large model, a lightweight fine-tuning unit based on head-wise convolution is added to the original self-attention module, and only the head-wise convolution unit and the batch normalization unit are trained, thereby reducing the memory usage during large model fine-tuning, speeding up the model training speed, and realizing efficient fine-tuning training in vertical fields.

[0113] In addition, according to the convolution-based large model fine-tuning acceleration method of the embodiment of the present disclosure, in the forward reasoning stage, the trained convolution module can be accumulated with the original standard self-attention map and multiplied with the value vector, and the bias of the convolution module can be fused with the bias of the subsequent linear projection layer. Here, the fine-tuned convolution module can be merged into static attention parameters, accumulated on the self-attention map, and then multiplied with the original standard value vector. The bias term can be fused into the bias parameter in the linear projection layer, thereby eliminating the complex calculations of multiple modules introduced during fine-tuning training. In the final fine-tuned model, only the addition calculation of static attention and original standard attention is introduced, reducing the reasoning or prediction overhead caused by fine-tuning.

[0114] In a second aspect of the exemplary embodiments of the present disclosure, Figure 6 As shown, a convolution-based large model fine-tuning acceleration device is provided, and the fine-tuning device includes an acquisition unit 610, a first determination unit 620, a training unit 630 and a second determination unit 640.

[0115] The acquisition unit 610 is configured to acquire a pre-trained large model and fine-tuning training data, wherein the large model includes a self-attention module, the fine-tuning training data includes training samples and annotation information for the training samples, the training samples include image data, and the large model is a visual large model or a multimodal large model.

[0116] The first determination unit 620 is configured to introduce a convolution module into the self-attention module to obtain an initial fine-tuning model, wherein the number of channels of the convolution module is the same as the number of attention heads of the self-attention module, and the number of convolution groups of the convolution module is the same as the number of attention heads.

[0117] The training unit 630 is configured to train the initial fine-tuning model using the fine-tuning training data under the condition of fixing the parameters of the large model to obtain a fine-tuning training model.

[0118] The second determining unit 640 is configured to obtain a fine-tuned large model based on the fine-tuned training model.

[0119] As an example, the convolution module convolves the value vector from the self-attention module and superimposes the output convolution result onto the output result of the self-attention module. The fine-tuned large model includes a static attention module corresponding to the trained convolution module.

[0120] As an example, the convolution module includes an expansion unit, a head-separated convolution unit, a batch normalization unit and a folding unit, wherein the expansion unit receives the value vector output from the self-attention module, and transforms the sequence length in the value vector into a size after scaling the height and width of the image data; the head-separated convolution unit receives the image data output from the expansion unit, and performs depth-separable convolution on the image data to obtain head-separated convolution features; the batch normalization unit receives the head-separated convolution features output from the head-separated convolution unit, normalizes the head-separated convolution features, and obtains normalized features; the folding unit receives the normalized features output from the batch normalization unit, transforms the normalized features into sequence features, and superimposes the sequence features onto the calculation results of the self-attention module.

[0121] As an example, a fine-tuned large model is obtained by fusing the parameters of the trained convolution module into the self-attention module, and removing the trained convolution module from the fine-tuned training model to obtain the fine-tuned large model; or, using the fine-tuned training model as the fine-tuned large model.

[0122] As an example, the parameters of the trained convolution module include bias parameters and weight parameters, wherein the parameters of the trained convolution module are fused into the trained self-attention module in the following manner: the weight parameters are fused with the self-attention map in the self-attention module by superposition; and the bias parameters are fused into the linear projection layer in the self-attention module.

[0123] As an example, the weight parameters are fused with the self-attention map in the following manner: the weight parameters are flattened from the convolution sliding window of the convolution module to obtain the flattened weight parameters; the parameters in the self-attention module are extracted from the flattened weight parameters to obtain a static self-attention module; the output of the static self-attention module is superimposed on the output result of the self-attention map, wherein the bias parameters are fused into the linear projection layer in the following manner: the bias parameters are fused with the original weight parameters of the linear projection layer to obtain a fusion result; the fusion result is superimposed with the original bias parameters of the linear projection layer, and the superimposed bias parameters are used as the bias parameters of the fused linear projection layer.

[0124] Regarding the device in the above-mentioned embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method. Each unit in the convolution-based large model fine-tuning acceleration device can execute the corresponding steps in the method according to the convolution-based large model fine-tuning acceleration method in the method embodiment of the first aspect above, which will not be elaborated here.

[0125] Figure 7 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Figure 7 As shown, the electronic device 10 includes a processor 101 and a memory 102 for storing processor executable instructions. Here, when the processor executable instructions are executed by the processor, the processor is prompted to execute the convolution-based large model fine-tuning acceleration method as described in the above exemplary embodiment.

[0126] As an example, the electronic device 10 is not necessarily a single device, but may be any collection of devices or circuits that can execute the above instructions (or instruction sets) individually or in combination. The electronic device 10 may also be part of an integrated control system or system manager, or may be configured as a server that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0127] In the electronic device 10, the processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor 101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0128] The processor 101 may execute instructions or codes stored in the memory 102, wherein the memory 102 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0129] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 102 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 101 can read files stored in the memory 102.

[0130] In addition, the electronic device 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 10 may be connected to each other via a bus and / or a network.

[0131] In an exemplary embodiment, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the convolution-based large model fine-tuning acceleration method as described in the above exemplary embodiment. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: read-only memory (ROM), random access memory (RAM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0132] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, which includes computer executable instructions. When the computer executable instructions are executed by at least one processor, the convolution-based large model fine-tuning acceleration method according to the exemplary embodiment of the present disclosure is implemented.

[0133] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0134] In addition, it should be noted that although several examples of each step are described above with reference to specific drawings, it should be understood that the embodiments of the present disclosure are not limited to the combinations given in the examples, and the steps appearing in different drawings may be combined, and the order of execution of the steps may be changed, which is not exhaustive here.

[0135] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A convolution-based large model fine-tuning acceleration method, characterized in that: The method comprises: Obtain a pre-trained large model and fine-tuning training data, wherein the large model includes a self-attention module, the fine-tuning training data includes training samples and annotation information of the training samples, the training samples include image data, and the large model is a visual large model or a multimodal large model; Introducing a convolution module into the self-attention module to obtain an initial fine-tuning model, wherein the number of channels of the convolution module is the same as the number of attention heads of the self-attention module, and the number of convolution groups of the convolution module is the same as the number of attention heads; Under the condition of fixing the parameters of the large model, using the fine-tuning training data, training the initial fine-tuning model to obtain a fine-tuning training model; Based on the fine-tuned training model, a fine-tuned large model is obtained. The convolution module includes an expansion unit, a head convolution unit, a batch normalization unit and a folding unit. The expansion unit receives the value vector from the self-attention module and performs a dimension permutation operation and a shape change operation on the value vector, wherein the dimension permutation operation transforms the sequence length in the value vector to the last column of the matrix, and the shape change operation transforms the sequence length to a size after scaling the height and width of the image data; The head-separated convolution unit receives the image data output from the expansion unit, and performs depth-separable convolution on the image data to obtain head-separated convolution features, wherein the number of channels of the convolution layer in the head-separated convolution unit is the same as the number of attention heads of the multi-head self-attention mechanism of the self-attention module, and the number of convolution groups of the head-separated convolution unit is the same as the number of attention heads of the multi-head self-attention mechanism of the self-attention module; The batch normalization unit receives the head-separated convolution features output from the head-separated convolution unit, normalizes the head-separated convolution features, and obtains normalized features, wherein the number of channels of the batch normalization unit is the same as the number of attention heads of the multi-head self-attention mechanism of the self-attention module; The folding unit receives the normalized features output from the batch normalization unit, transforms the normalized features into sequence features, and superimposes the sequence features onto the calculation results of the self-attention module.

2. The method according to claim 1, characterized in that The fine-tuned large model includes a static attention module corresponding to the trained convolution module.

3. The method according to claim 1, characterized in that The fine-tuned large model is obtained by: Fusion the parameters of the trained convolution module into the self-attention module, and remove the trained convolution module from the fine-tuned training model to obtain the fine-tuned large model; or, The fine-tuned training model is used as the fine-tuned large model.

4. The method according to claim 3, characterized in that The parameters of the trained convolution module include bias parameters and weight parameters, wherein the parameters of the trained convolution module are fused into the trained self-attention module in the following manner: The weight parameter is fused with the self-attention map in the self-attention module by superposition; The bias parameters are fused into the linear projection layer in the self-attention module.

5. The method according to claim 4, characterized in that The weight parameters are fused with the self-attention map in the following way: Flattening the weight parameter from the convolution sliding window of the convolution module to obtain a flattened weight parameter; Extracting parameters in the self-attention module from the flattened weight parameters to obtain a static self-attention module; The output of the static self-attention module is superimposed on the output result of the self-attention map, The bias parameters are integrated into the linear projection layer in the following manner: Fusing the bias parameter with the original weight parameter of the linear projection layer to obtain a fusion result; The fusion result is superimposed with the original bias parameter of the linear projection layer, and the superimposed bias parameter is used as the bias parameter of the fused linear projection layer.

6. A large model fine-tuning acceleration device based on convolution, characterized in that: The fine-tuning acceleration device comprises: an acquisition unit configured to acquire a pre-trained large model and fine-tuning training data, wherein the large model includes a self-attention module, the fine-tuning training data includes training samples and annotation information of the training samples, the training samples include image data, and the large model is a visual large model or a multimodal large model; A first determining unit is configured to introduce a convolution module into the self-attention module to obtain an initial fine-tuning model, wherein the number of channels of the convolution module is the same as the number of attention heads of the self-attention module, and the number of convolution groups of the convolution module is the same as the number of attention heads; A training unit is configured to train the initial fine-tuning model using the fine-tuning training data under the condition that the parameters of the large model are fixed, so as to obtain a fine-tuning training model; The second determining unit is configured to obtain a fine-tuned large model based on the fine-tuned training model, The convolution module includes an expansion unit, a head convolution unit, a batch normalization unit and a folding unit. The expansion unit receives the value vector from the self-attention module and performs a dimension permutation operation and a shape change operation on the value vector, wherein the dimension permutation operation transforms the sequence length in the value vector to the last column of the matrix, and the shape change operation transforms the sequence length to a size after scaling the height and width of the image data; The head-separated convolution unit receives the image data output from the expansion unit, and performs depth-separable convolution on the image data to obtain head-separated convolution features, wherein the number of channels of the convolution layer in the head-separated convolution unit is the same as the number of attention heads of the multi-head self-attention mechanism of the self-attention module, and the number of convolution groups of the head-separated convolution unit is the same as the number of attention heads of the multi-head self-attention mechanism of the self-attention module; The batch normalization unit receives the head-separated convolution features output from the head-separated convolution unit, normalizes the head-separated convolution features, and obtains normalized features, wherein the number of channels of the batch normalization unit is the same as the number of attention heads of the multi-head self-attention mechanism of the self-attention module; The folding unit receives the normalized features output from the batch normalization unit, transforms the normalized features into sequence features, and superimposes the sequence features onto the calculation results of the self-attention module.

7. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor executable instructions, Wherein, when the processor executable instructions are executed by the processor, the processor is prompted to execute the convolution-based large model fine-tuning acceleration method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the convolution-based large model fine-tuning acceleration method according to any one of claims 1 to 5.

9. A computer program product comprising computer executable instructions, characterized in that: When the computer executable instructions are executed by at least one processor, the convolution-based large model fine-tuning acceleration method according to any one of claims 1 to 5 is implemented.

Images