Image classification method and system based on fine-tuning of GNF3 linear mapping function

Abstract

The disclosure provides an image classification method and system based on GNF3Linear mapping function fine-tuning, relating to the technical field of large language models, comprising: obtaining image data to be classified; inputting the image data to be classified into an image classification model to output an image classification result; wherein the backbone network of the image classification model is a ViT-base model, in the linear mapping layer of the ViT-base model, the original Linear function is replaced by a GNF3Linear function, the GNF3Linear function keeps linear combination of input image features and weight matrix, and then realizes the calculation process of mapping from input space to output space through adjustment of the bias term, only changes the memory size of the intermediate variable stored in each component in the activation memory in the derivation process, and stores the segment label, mean and standard deviation of each component according to the set memory.

Description

Technical Field

[0001] This disclosure relates to the field of large language model technology, specifically to an image classification method and system based on fine-tuning of the GNF3Linear mapping function. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Since its introduction in 2017, the Transformer model has achieved tremendous success in the field of Natural Language Processing (NLP), demonstrating outstanding performance in tasks such as machine translation, text generation, and speech recognition. Its self-attention-based architecture effectively captures long-range dependencies in text, unlike traditional sequence models (such as RNNs and LSTMs) that rely on sequential processing, allowing for parallel computation and significantly improving training efficiency. In recent years, the application scope of the Transformer model has continued to expand, extending beyond NLP to computer vision (CV). Significant performance improvements have been achieved by segmenting images into patches and feeding them as sequences into the Transformer model.

[0004] Large Language Models (LLMs), as a key application of Transformer models, have seen rapid development in recent years. Ultra-large-scale pre-trained language models, such as the GPT series and BERT, have achieved remarkable results in tasks like language generation, question answering systems, and text classification. For example, the emergence of ultra-large-scale models like GPT-3 has enabled the wider application of natural language processing techniques in content creation, intelligent customer service, and other fields. Furthermore, large language models have demonstrated potential in multimodal processing, such as image and text fusion, further expanding their application scope.

[0005] While Transformer models have demonstrated powerful performance in image classification tasks, their training process suffers from significant memory consumption. Transformer models typically have a massive number of image input parameters, sometimes reaching billions or even tens of billions. These parameters need to be stored in memory during training, consuming a substantial amount of memory. Furthermore, during backpropagation, various intermediate variables from the forward propagation process need to be saved to calculate gradients; the memory occupied by these intermediate variables, known as activation memory, further exacerbates memory consumption. For large-scale Transformer models, techniques such as distributed training and mixed-precision training are often used to alleviate memory pressure. However, even with these measures, insufficient memory remains a major bottleneck limiting further scaling of the model and improvements in training efficiency, severely impacting the efficiency of image classification tasks. Summary of the Invention

[0006] To address the aforementioned issues, this disclosure proposes an image classification method and system based on fine-tuning of the GNF3Linear mapping function. In the image classification process, the linear mapping layer of the classification model is improved by replacing the original Linear function with the GNF3Linear function. This only requires storing an average of 3-bit intermediate variables for each sub-variable in the active memory, thereby improving classification efficiency and saving GPU computing memory.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions:

[0008] Image classification methods based on fine-tuning of the GNF3Linear mapping function include:

[0009] Obtain the image data to be classified;

[0010] The image data to be classified is input into the image classification model, and the image classification result is output.

[0011] The backbone network of the image classification model is the ViT-base model. In the linear mapping layer of the ViT-base model, the GNF3Linear function is used to replace the original Linear function. The GNF3Linear function maintains the linear combination of the input image features and the weight matrix, and then realizes the calculation process of mapping from the input space to the output space through the adjustment of the bias term. Only the memory size of the intermediate variables stored in each component in the activation memory is changed during the differentiation process. For each component, the segment label, mean and standard deviation are stored in the set memory.

[0012] According to some embodiments, the present disclosure adopts the following technical solutions:

[0013] Image classification systems based on fine-tuning of the GNF3Linear mapping function include:

[0014] The data acquisition module is used to acquire the image data to be classified.

[0015] The classification module is used to input the image data to be classified into the image classification model and output the image classification result.

[0016] The backbone network of the image classification model is the ViT-base model. In the linear mapping layer of the ViT-base model, the GNF3Linear function is used to replace the original Linear function. The GNF3Linear function maintains the linear combination of the input image features and the weight matrix, and then realizes the calculation process of mapping from the input space to the output space through the adjustment of the bias term. Only the memory size of the intermediate variables stored in each component in the activation memory is changed during the differentiation process. For each component, the segment label, mean and standard deviation are stored in the set memory.

[0017] According to some embodiments, the present disclosure adopts the following technical solutions:

[0018] A computer program product includes a computer program that, when executed by a processor, implements the image classification method based on GNF3Linear mapping function fine-tuning.

[0019] According to some embodiments, the present disclosure adopts the following technical solutions:

[0020] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the image classification method based on GNF3Linear mapping function fine-tuning.

[0021] According to some embodiments, the present disclosure adopts the following technical solutions:

[0022] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the image classification method based on GNF3Linear mapping function fine-tuning.

[0023] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0024] This disclosure presents an image classification method based on fine-tuning of the GNF3Linear mapping function. In the ViT-base model, the linear mapping layer contributes a large proportion of the activation memory. Since the derivative of its input variables is needed during backpropagation, the input variables (usually 16-bit or 32-bit data format) must be stored in the activation memory during the forward pass. This disclosure replaces the original Linear function with the GNF3Linear function. The proposed GNF3Linear function significantly reduces its activation memory usage, requiring only an average of 3-bit intermediate variables to be stored in the activation memory for each sub-variable.

[0025] The image classification method based on fine-tuning of the GNF3Linear mapping function disclosed herein maintains the forward process consistent with that of general linear mapping, but only requires 3 bits for each component to store the segment label, mean, and standard deviation during the differentiation process. This is 3 / 16 or 3 / 32 of the original active memory (16 bits or 32 bits), which greatly reduces its active memory usage, reduces memory consumption, and alleviates memory pressure. Attached Figure Description

[0026] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0027] Figure 1 This is a flowchart of an image classification method based on fine-tuning of the GNF3Linear mapping function according to an embodiment of this disclosure. Detailed Implementation

[0028] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] Example 1

[0032] One embodiment of this disclosure provides an image classification method based on fine-tuning of the GNF3Linear mapping function, the specific steps of which include:

[0033] Step 1: Obtain the image data to be classified;

[0034] Step 2: Input the image data to be classified into the image classification model and output the image classification result;

[0035] The backbone network of the image classification model is the ViT-base model. In the linear mapping layer of the ViT-base model, the GNF3Linear function is used to replace the original Linear function. The GNF3Linear function maintains the linear combination of the input image features and the weight matrix, and then realizes the calculation process of mapping from the input space to the output space through the adjustment of the bias term. Only the memory size of the intermediate variables stored in each component in the activation memory is changed during the differentiation process. For each component, the segment label, mean and standard deviation are stored in the set memory.

[0036] As one embodiment, the specific implementation process of an image classification method based on GNF3Linear mapping function fine-tuning disclosed herein is as follows:

[0037] Step 1: Obtain image data to be classified; obtain image data from public datasets.

[0038] Step 2: Input the image data to be classified into the image classification model and output the image classification result;

[0039] Specifically, the backbone network of the image classification model is the ViT-base model, which is pre-trained on ImageNet-22k and fine-tuned on the CIFAR10 / 100 and FGVC datasets. GELU, FP16Linear, and LayerNorm are default modules in ViT. This disclosure uses a single 2080Ti GPU for processing. The core of the ViT-base model is to split the image into sequential image patches and model global relationships through a Transformer Encoder structure.

[0040] As one example, the specific process of traditional ViT-base model for image classification includes: first, the image is patched. The image size input to the ViT-base model is 224×224×3, the patch size is 16×16, and there are a total of (224 / 16)^2 = 196 image patches. The dimensions of each patch after flattening are 16×16×3 = 768, and finally a 196×768 sequence is output.

[0041] Then, linear projection (Patch Embedding) is performed, mapping each Patch to a hidden dimension through a learnable linear layer (nn.Linear).

[0042] Furthermore, in the Transformer architecture, the linear mapping layer contributes a significant proportion of the activation memory. Since the derivative of its input variables is needed during backpropagation, the input variables (usually 16-bit or 32-bit data format) must be stored in the activation memory during the forward pass. In neural network architectures, the linear function (linear mapping) is one of the most fundamental and crucial components. Essentially, it maps image features from the input space to the output space through a linear combination of input data and a weight matrix, adjusted by bias terms. Mathematically, this can be expressed as:

[0043]

[0044] in, For the input vector, It is a weight matrix. It is a bias vector. This is the output result. This simple linear transformation provides the necessary foundation for building complex nonlinear models later.

[0045] In the Transformer model, the queries, keys, and values ​​in the Attention module are all generated by linear mappings, and the FFN module also contains multiple Linear functions. During backpropagation (BP) training, these Linear components need to store the input vectors in the activation memory because of the derivative of the weight matrix. Linear functions typically use 32-bit or 16-bit storage formats, which puts considerable memory pressure on the Transformer model training.

[0046] To address the above issues, this disclosure improves the linear mapping layer of the ViT-base model by using the GNF3Linear function to replace the original Linear function. The GNF3Linear function can replace the original Linear function, significantly reducing its activation memory usage. It only needs to store an average of 3-bit intermediate variables for each sub-variable in the activation memory.

[0047] Specifically, this disclosure uses Approximation Backpropagation (Approx. BP) theory to give an approximate derivative of the Linear function. More specifically, it unifies the Linear function using... This means, that is:

[0048]

[0049] Define the approximate function class as follows:

[0050]

[0051] here, It is a quantization operation. Typically, It can be represented as a low-bit data type instead of the original 16-bit data type. Gradient pass-through estimation is used. ,have:

[0052]

[0053] According to Approx-BP, it only needs to be stored in GPU memory. To calculate about The derivative of the value is used to reduce the amount of active memory.

[0054] Furthermore, since the GNF3Linear function replaces the original Linear function, the above operations are quantized in GNF3Linear. Follow these steps:

[0055] (1) 32 consecutive input feature values ​​in memory Calculate the mean within each group. and standard deviation Then normalize to get ;

[0056] (2) Assumption The components in the model are independent and identically distributed, and follow a generalized Gaussian distribution. And the variance is 1 (from this we can obtain) And use moment estimation to obtain parameter estimates. and according to The value matches the corresponding parameter in the "GNF3Linear Parameter Value Table" and parameters The parameter values ​​for GNF3Linear are shown in Table 1.

[0057] The specific method for moment estimation is as follows: first calculate the empirical statistic. Then compare it with different parameters in the generalized normal distribution. Compare the corresponding theoretical values ​​of this statistic (i.e., the "RHS" column in Table 1 below), and select the "RHS" value that is closest to the theoretical value. As an estimate .

[0058] Table 1 GNF3Linear Parameter Values ​​Table

[0059]

[0060] (3) To Calculate the classification label for each component. ;

[0061] (4) Calculation Each component The original eigenvalue Q(z) is replaced with this new eigenvalue Q(z).

[0062] Steps (1) and (3) are completed in the forward process, and step (4) is completed in the reverse process.

[0063] GNF3Linear maintains the same forward process as a general linear mapping, but during the differentiation process, it only needs 3 bits to store the segment label, mean, and standard deviation for each component, which is 3 / 16 or 3 / 32 of the original active memory (16 bits or 32 bits).

[0064] As one embodiment, the GNF3Linear function of this disclosure is applied to a large language model of the Transformer architecture, and its performance is evaluated as follows:

[0065] (a) This disclosure discloses a method for fine-tuning ViT on image classification tasks.

[0066] Benchmark Introduction: The ViT-base model, pre-trained on ImageNet-22k, was used as the backbone network and fine-tuned on the CIFAR10 / 100 and FGVC datasets. GELU, FP16 Linear, and LayerNorm are default modules in ViT. A single 2080Ti GPU was used.

[0067] Fine-tuning method: The method of this disclosure is implemented on full-tuning and LoRA. For LoRA, all linear layers are fine-tuned.

[0068] Comparison method: The method disclosed herein is compared with AAL. AAL is an auxiliary activation method, which can be regarded as a special 1-bit quantization method of linear mapping.

[0069] Results: Table 2 compares the training performance, activation memory usage, and training throughput of ViT-base fine-tuned using Full-Tuning and LoRA, respectively. It can be seen that using GNF3Linear in LoRA fine-tuning reduces peak GPU memory usage by approximately 1.3 GiB, accounting for about 22% of the peak GPU memory usage of ordinary LoRA fine-tuning. Similarly, GNF3Linear reduces peak GPU memory usage by approximately 26% in Full-Tuning. Furthermore, GNF3Linear shows almost no decrease in training throughput and training performance. In contrast, AAL significantly reduces training throughput.

[0070] Table 2 shows the main results of fine-tuning ViT-base using LoRA on the CIFAR10 / 100 and FGVC datasets.

[0071]

[0072] (ii) Fine-tuning LLaMA for language comprehension tasks

[0073] Benchmark Introduction: This disclosure also allows fine-tuning of LLaMA-7B and LLaMA-13B on Alpaca and evaluating the fine-tuned models on 5-shot MMLU. When fine-tuning LLaMA-7B, a single A6000 GPU is used. When fine-tuning LLaMA-13B, this disclosure uses model parallelism provided by the Transformers library, training with 2×A6000 GPUs, and the reported peak GPU memory usage is the sum of the peak memory usage of the two GPUs.

[0074] Fine-tuning method: The method of this disclosure is applied to QLoRA fine-tuning to fine-tune LLaMA-7B and LLaMA-13B. QLoRA uses NF4 data type to store pre-trained weights and Bfloat16 to store parameters in LoRA. In QLoRA, the weights of all linear layers are fine-tuned via LoRA.

[0075] Results: The results of fine-tuning LLaMA-7B and LLaMA-13B are summarized in Table 3. GNF3Linear significantly reduced peak memory usage when fine-tuning LLaMA using QLoRA, reducing it by approximately 6.5 GiB for LLaMA-7B and by approximately 10.0 GiB for LLaMA-13B, resulting in significant GPU memory savings. These figures represent approximately 16% of the peak GPU memory usage of the baseline methods. Furthermore, GNF3Linear improved training throughput by 12% and 19% respectively when fine-tuning LLaMA-7B and LLaMA-13B using QLoRA, while achieving MMLU accuracy comparable to the baseline methods.

[0076] Table 3 shows the main results of fine-tuning LLaMA-7B and LLaMA-13B using QLoRA on Alpaca.

[0077]

[0078] (III) Fine-tuning RoBERTa for language comprehension tasks

[0079] Benchmark Introduction: This disclosure fine-tunes a pre-trained RoBERTa-base on five GLUE tasks: CoLA, SST-2, MRPC, STS-B, and RTE. Training was performed using a single RTX 3090 GPU.

[0080] Fine-tuning method: The method disclosed herein is applied to fine-tuning the pre-trained RoBERTa-base using LoRA. The data type used in this experiment is FP16.

[0081] Results: The results of fine-tuning RoBERTa-base are summarized in Table 4. Fine-tuning RoBERTa-base using the method of this disclosure achieves slightly higher accuracy than the baseline, reduces training throughput by 15%, and reduces peak GPU memory usage by 24%.

[0082] Table 4 shows the main results of fine-tuning RoBERTa-base using LoRA on GLUE.

[0083]

[0084] Example 2

[0085] One embodiment of this disclosure provides an image classification system based on fine-tuning of the GNF3Linear mapping function, comprising:

[0086] The data acquisition module is used to acquire the image data to be classified.

[0087] The classification module is used to input the image data to be classified into the image classification model and output the image classification result.

[0088] The backbone network of the image classification model is the ViT-base model. In the linear mapping layer of the ViT-base model, the GNF3Linear function is used to replace the original Linear function. The GNF3Linear function maintains the linear combination of the input image features and the weight matrix, and then realizes the calculation process of mapping from the input space to the output space through the adjustment of the bias term. Only the memory size of the intermediate variables stored in each component in the activation memory is changed during the differentiation process. For each component, the segment label, mean and standard deviation are stored in the set memory.

[0089] Example 3

[0090] One embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the image classification method based on GNF3Linear mapping function fine-tuning.

[0091] Example 4

[0092] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions. When these computer instructions are executed by a processor, they implement the image classification method based on GNF3Linear mapping function fine-tuning.

[0093] Example 5

[0094] One embodiment of this disclosure provides an electronic device, including a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the image classification method based on GNF3Linear mapping function fine-tuning.

[0095] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. An image classification method based on fine-tuning of the GNF3Linear mapping function, characterized in that, include: Obtain the image data to be classified; The image data to be classified is input into the image classification model, and the image classification result is output. The backbone network of the image classification model is the ViT-base model. In the linear mapping layer of the ViT-base model, the GNF3Linear function is used to replace the original Linear function. The GNF3Linear function maintains the linear combination of the input image features and the weight matrix, and then realizes the calculation process of mapping from the input space to the output space through the adjustment of the bias term. Only the memory size of the intermediate variables stored in each component in the activation memory is changed during the differentiation process. For each component, the segment label, mean and standard deviation are stored in the set memory. Using approximate backpropagation theory, an approximate derivative of the Linear function is given, and the Linear function is unified using... This means, that is: Define the approximate function class as follows: in, It is a quantization operation. It is represented as a low-bit data type instead of the original 16-bit data type. It is a weight matrix. It is the bias vector; The GNF3Linear function is used to replace the original Linear function, and quantization operations are performed within the GNF3Linear function. The process includes: grouping every 32 consecutive memory input components into a set, calculating the mean and standard deviation within each group, and then normalizing them; Assuming the normalized components are independent and identically distributed, following a generalized Gaussian distribution with a variance of 1, and using moment estimation to obtain parameter estimates, we match the estimated parameter values ​​with the corresponding parameters from the GNF3Linear parameter value table. For each normalized category, we calculate the classification label, and based on the parameters obtained from value matching, we calculate the parameters for each component. This calculation process is implemented in reverse; The specific method of moment estimation is as follows: first calculate the empirical statistic, then compare it with the theoretical values ​​of the statistics corresponding to different parameters in the generalized normal distribution, and select the parameter corresponding to the RHS with the closest value as the estimated value.

2. An image classification system based on fine-tuning of the GNF3Linear mapping function, characterized in that, Specifically, the image classification method based on GNF3Linear mapping function fine-tuning as described in claim 1 includes: The data acquisition module is used to acquire the image data to be classified. The classification module is used to input the image data to be classified into the image classification model and output the image classification result. The backbone network of the image classification model is the ViT-base model. In the linear mapping layer of the ViT-base model, the GNF3Linear function is used to replace the original Linear function. The GNF3Linear function maintains the linear combination of the input image features and the weight matrix, and then realizes the calculation process of mapping from the input space to the output space through the adjustment of the bias term. Only the memory size of the intermediate variables stored in each component in the activation memory is changed during the differentiation process. For each component, the segment label, mean and standard deviation are stored in the set memory.

3. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image classification method based on GNF3Linear mapping function fine-tuning as described in claim 1.

4. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the image classification method based on GNF3Linear mapping function fine-tuning as described in claim 1.

5. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the image classification method based on GNF3Linear mapping function fine-tuning as described in claim 1.

Images