Infrared image task adaptation method based on prompt learning and adapter
By freezing the backbone network parameters, adding visual cue vectors, and inserting adapter modules, the high cost and easy forgetting of pre-trained infrared base models in downstream task adaptation are solved, enabling efficient deployment and adaptation on edge devices.
Patent Information
- Application Number
- CN202610531718.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-21
- Publication Date
- 2026-06-26
AI Technical Summary
Existing technologies suffer from high computational costs, strong data dependence, susceptibility to catastrophic forgetting, and difficulty in deploying pre-trained infrared basic models for downstream task adaptation. Traditional methods also suffer from performance loss or insufficient flexibility.
We employ a cue-based learning and adapter approach, freezing the backbone network parameters, adding visual cue vectors and inserting a lightweight adapter module, configuring a lightweight task head, and fine-tuning it with a small amount of labeled data. We then perform pruning and quantization compression to adapt it to edge devices.
It reduces computational and data requirements, avoids catastrophic amnesia, adapts to a variety of downstream tasks, and significantly lowers the threshold for the application and deployment of infrared AI models.
Smart Images
Figure CN122287782A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of transfer learning and edge computing technology, specifically relating to an adaptive method for infrared image tasks based on cue learning and adapters. Background Technology
[0002] With the continuous development of artificial intelligence vision basic model technology, the infrared basic model obtained by pre-training based on a huge infrared dataset has a powerful and general representation capability for infrared objects and scenes, providing core technical support for various downstream tasks of infrared images such as industrial equipment fault classification, military target recognition, and medical thermal image analysis.
[0003] However, adapting such pre-trained infrared base models to specific downstream tasks still faces many severe challenges: traditional full-parameter fine-tuning methods require updating all parameters of the model, which not only brings extremely high computational costs but also imposes stringent requirements on the scale of task-labeled data, relying on a large number of labeled samples to ensure the adaptation effect; more importantly, during the full-parameter fine-tuning process, the model is prone to quickly forgetting the general knowledge learned in the pre-training stage, i.e., the phenomenon of "catastrophic forgetting", which leads to a decline in the overall generalization ability of the model; in addition, full-parameter fine-tuning generates an independent model copy for each downstream task, which greatly increases the storage cost of the model, makes deployment burden heavy, and makes it difficult to achieve multi-task support on edge devices with limited computing power and storage resources, which seriously limits the scope of application of pre-trained infrared base models.
[0004] To address these issues, some lightweight fine-tuning techniques have emerged in the industry, but they still have significant drawbacks: for example, while linear probing reduces the cost of parameter updates, it sacrifices model performance and has limited adaptability; while methods that only fine-tune the bias term further reduce the amount of parameter adjustments, they lack flexibility, making it difficult to fully adapt to the personalized needs of different downstream tasks and fully unleash the potential of pre-trained models.
[0005] In view of this, the present invention is hereby proposed. Summary of the Invention
[0006] To address the aforementioned technical problems in the prior art, this invention provides an infrared image task adaptation method based on cue learning and adapters, which solves the problems of high cost of full parameter fine-tuning, strong data dependence, and susceptibility to catastrophic forgetting in the downstream adaptation of pre-trained infrared base models.
[0007] To achieve the above objectives, the technical solution of the present invention is as follows: Infrared image task adaptation methods based on cue learning and adapters include: S1. Obtain the pre-trained infrared base model and freeze the backbone network parameters of the pre-trained infrared base model. S2. For the target downstream task, modify the parameters of the model after freezing the backbone network parameters. The parameter modification includes: adding a visual cue vector before the input embedding layer of the infrared image encoder; inserting an adapter module into some or all of the Transformer layers of the encoder; and configuring a task output header for the target downstream task. S3. Obtain the labeled dataset of the target downstream task, and use the visual cue vector, the parameters of the adapter module and the parameters of the task output head as training objects to fine-tune the modified model; S4. Compress and transform the fine-tuned model, and deploy the processed model to edge computing devices to perform inference tasks.
[0008] Furthermore, the visual cue vector is a set of optimized parameters with the same dimension as the infrared image patch embedding vector, specifically expressed as:
[0009] in, To indicate the number of vectors, The feature dimension; the visual cue vector and the input image patch embedding vector The sequences are concatenated along the sequence dimension to form the input sequence of the encoder, specifically expressed as:
[0010] in, To concatenate the visual cue vector P with the image patch embedding vector X along the sequence dimension, [ [] represents the concatenation operation along the sequence dimension. The number of embedding vectors for the input image patch.
[0011] Furthermore, the adapter module is inserted after the feedforward network and before the residual connection of the Transformer layer. The adapter module is a bottleneck layer structure, and its forward propagation formula is:
[0012] in, Output characteristics for the adapter module, For the input characteristics of the adapter module, and For the projection matrix, It is a non-linear activation function.
[0013] Furthermore, the nonlinear activation function is the GELU function, and the approximate expression of the GELU function is:
[0014] in, The output value of the GELU activation function. As input features, It is the hyperbolic tangent function. Pi is the mathematical constant of a circle.
[0015] Furthermore, the total loss function fine-tuned in step S3 Loss due to mission Sparse regularization loss for visual cue vectors The total loss function is expressed as follows:
[0016] in, For hyperparameters, For the visual cue vector P Norm regularization.
[0017] Furthermore, when the target downstream task is a classification task, the task loss... This represents the cross-entropy loss.
[0018] Furthermore, the compression and conversion process described in step S4 specifically includes: The fine-tuned model is then subjected to structured pruning, removing low-contribution cue vectors and adapter connections; The pruned model weights are quantized, and the floating-point parameters are converted into a low-bit-width fixed-point representation. The quantized model is converted into an inference engine format supported by the edge computing device.
[0019] Furthermore, the quantization is post-training static quantization. First, a small amount of unlabeled data is passed through the model to record the distribution of activation values in each layer. Then, the FP32 weights W and activation values are quantized to INT8. The quantization formula is:
[0020] in, The scaling factor determined from the calibration data. This is a numerical truncation function.
[0021] Furthermore, the salience of each cue vector is evaluated by calculating its L2 norm or activation contribution, and the least important cue vectors are removed. A prompt vector, are positive integers and .
[0022] The beneficial effects of this invention are as follows: To address the challenges of high cost, strong data dependence, susceptibility to catastrophic forgetting, and difficulty in edge deployment during downstream adaptation of pre-trained infrared baseline models, an efficient adaptive method is proposed. Its core principles are: freezing the model's backbone parameters to preserve general knowledge; adding learnable visual cue vectors to the input side; inserting a lightweight adapter module into the Transformer layer and configuring a lightweight task header; fine-tuning newly added parameters using only a small number of labeled samples; and converting the model to a format supported by edge devices after pruning and quantization compression. This invention eliminates the need to update a large number of backbone parameters, significantly reducing computational and data requirements, avoiding catastrophic forgetting, adapting to various downstream infrared tasks, and significantly lowering the barrier to entry for infrared AI model application and deployment. Attached Figure Description
[0023] Figure 1 A flowchart of an infrared image task adaptive method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the model structure of the injectable learnable cue vector and adapter module provided in an embodiment of the present invention. Detailed Implementation
[0024] The technical solution of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.
[0025] It should be noted that, unless otherwise specifically stated, the relative arrangement and numerical expressions of the components and steps described in these embodiments should not be construed as limiting the scope of the invention.
[0026] The following description of exemplary embodiments is merely illustrative and is not intended to limit the invention or its application or use in any way. Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail herein, but where applicable, such techniques, methods, and apparatus should be considered part of this specification.
[0027] Example See Figure 1 , Figure 1 This is a flowchart of the infrared image task adaptation method based on cue learning and adapter proposed in this invention. Specific steps may include: S1. Model preparation and parameter freezing: Obtain the pre-trained infrared base model and freeze the backbone network parameters of the pre-trained infrared base model. Obtain a pre-trained infrared multimodal base model (e.g.) Model ), directly freeze all parameters of its backbone network; by For example, its input image size is 224×224, the block size is 16×16, and the number of image blocks is... Model hidden layer dimension After freezing, the backbone parameters are no longer updated to ensure that the general representation ability of infrared images learned during the pre-training phase is not destroyed; the complete set of parameters of this pre-trained model is denoted as... To preserve its general knowledge and prevent forgetting during subsequent fine-tuning, all trunk parameters are frozen before adaptation.
[0028] S2, Efficient Parameter Injection Module: Constructs a noise prediction model with a U-Net structure, including 4 downsampling layers and 4 upsampling layers; single-channel input / output design adapts to infrared image characteristics, and uses an innovative attention mechanism configuration. Downsampling path: The AttnDownBlock2D attention module is introduced in layer 3; Upsampling path: The second layer uses the AttnUpBlock2D attention module.
[0029] See Figure 2 For the target downstream tasks, the backbone network parameters are modified. The parameter modifications include: S21. Add visual cue vectors: Add visual cue vectors before the input embedding layer of the infrared image encoder; the visual cue vectors are a set of optimized parameters with the same dimension as the infrared image patch embedding vectors, and the specific expression is as follows:
[0030] in, To indicate the number of vectors, For feature dimensions; this embodiment sets the number of prompts. Dimension For the input image, after processing by the model's original block embedding layer, a standard block embedding sequence is obtained. ; Visual cue vectors and input image patch embedding vectors Will and The sequences are concatenated along the sequence dimension to form the input sequence of the encoder. The specific expression is:
[0031] in, For splicing operations, The number of embedding vectors for the input image patch; the cue vectors serve as optimizable "task prefixes" to guide the model to focus on features relevant to ship identification.
[0032] S22. Inserting Adapter Modules: Adapter modules are inserted into some or all of the Transformer layers of the encoder; an adapter module is inserted after the feedforward network (FFN) and before layer normalization and residual connections in each Transformer layer of the ViT encoder. Let the characteristics of the input adapter of a certain layer be... The adapter module adopts a bottleneck layer structure, and its forward propagation formula is:
[0033] in, Output characteristics for the adapter module, For the input characteristics of the adapter module, and For the projection matrix, It is a non-linear activation function.
[0034] In this embodiment, the nonlinear activation function is the GELU function, whose approximate expression is:
[0035] in, The output value of the GELU activation function. As input features, It is the hyperbolic tangent function. Pi is the mathematical constant of a circle.
[0036] Set bottleneck dimensions ,satisfy The design meets the lightweight design requirements. This design introduces only [amount] per Transformer layer. One parameter, for a 12-layer layer The model and adapter module have a total of only about 590,000 parameters, achieving extremely low parameter increment.
[0037] S23. Configure lightweight task header: Configure task output header for the target downstream task; For downstream tasks (such as ship classification), add a linear classification header after the [CLS] token feature at the end of the model:
[0038] in, , , To predict probabilities, only a small number of parameters are added to complete the mapping from features to task output.
[0039] S3. Sparse supervised fine-tuning: Obtain the labeled dataset of the target downstream task, and use the visual cue vector, the parameters of the adapter module and the parameters of the task output head as training objects to fine-tune the modified model. Specifically, fine-tuning is performed using a small-scale labeled dataset from the target domain; this embodiment uses a marine infrared ship dataset containing approximately 1,200 labeled images, and the total loss function is fine-tuned. Loss due to mission Sparse regularization loss for visual cue vectors The total loss function is expressed as follows:
[0040] in, For hyperparameters, For the visual cue vector P Norm regularization; task loss Using cross-entropy loss, let the number of samples in a batch be... The real label is The model predicts the distribution as follows: ,but:
[0041] Sparse regularization loss Apply L1 norm constraints to the cue vector:
[0042] Wherein, regularization coefficient The AdamW optimizer was used, with an initial learning rate of ; The weight decay is 0.01. During training, the data is processed in batches (batch_size=8). A periodic verification mechanism is set (verification is performed every 50 steps), and a total of 50 training cycles are performed.
[0043] S4. Edge Deployment: The fine-tuned model is compressed and transformed, and then deployed to edge computing devices to perform inference tasks; specifically including: S41. Structured Pruning: Perform structured pruning on the fine-tuned model, removing low-contribution cue vectors and adapter connections; evaluate cue vectors. The significance of each vector in the dataset (e.g., calculating its L2 norm or activation contribution) is assessed, and the least significant vectors are removed. vectors (such as) ,and Correspondingly, the projection rows associated with the pruned cue vectors in the adapter can be removed, further compressing the model parameter size; S42, Weight Quantization: Quantize the weights of the pruned model, converting floating-point parameters into low-bit-width fixed-point representations; This embodiment employs a post-training static quantization method: First, approximately 100 unlabeled marine infrared images are used as calibration data, and the distribution of activation values in each layer is recorded using the model; then, the model weights and activation values with FP32 precision are quantized to INT8 precision, using the following quantization formula:
[0044] in, The scaling factor determined from the calibration data. `round` is the numerical truncation function, and `round` is the rounding function; this step can reduce the model size by about 75% while significantly improving inference speed.
[0045] S43. Conversion and Deployment: Convert the quantized model into an inference engine format supported by the edge computing device; export the quantized model in ONNX format; perform further graph optimization and kernel fusion using the inference engine (such as TensorRT) of the target edge device (such as NVIDIA Jetson AGX Orin), and finally generate and deploy an efficient inference engine file.
[0046] Through the above steps, this invention uses minimal task-labeled data and trainable parameters to efficiently adapt a general infrared pre-trained model to a specific task and achieves lightweight deployment for edge devices, verifying the effectiveness of this method in reducing application costs and barriers to entry.
[0047] The above specific embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. An infrared image task adaptive method based on cue learning and adapter, characterized in that, include: S1. Obtain the pre-trained infrared base model and freeze the backbone network parameters of the pre-trained infrared base model. S2. For the target downstream task, modify the parameters of the model after freezing the backbone network parameters. The parameter modification includes: adding a visual cue vector before the input embedding layer of the infrared image encoder; inserting an adapter module into some or all of the Transformer layers of the encoder; and configuring a task output header for the target downstream task. S3. Obtain the labeled dataset of the target downstream task, and use the visual cue vector, the parameters of the adapter module and the parameters of the task output head as training objects to fine-tune the modified model; S4. Compress and transform the fine-tuned model, and deploy the processed model to edge computing devices to perform inference tasks.
2. The infrared image task adaptive method based on cue learning and adapter according to claim 1, characterized in that, The visual cue vector is a set of optimized parameters with the same dimension as the infrared image patch embedding vector, and its specific expression is as follows: in, To indicate the number of vectors, The feature dimension; the visual cue vector and the input image patch embedding vector The sequences are concatenated along the sequence dimension to form the input sequence of the encoder, specifically expressed as: in, To concatenate the visual cue vector P with the image patch embedding vector X along the sequence dimension, [ [] represents the concatenation operation along the sequence dimension. The number of embedding vectors for the input image patch.
3. The infrared image task adaptive method based on cue learning and adapter according to claim 1, characterized in that, The adapter module is inserted after the feedforward network and before the residual connection of the Transformer layer. The adapter module is a bottleneck layer structure, and its forward propagation formula is: in, Output characteristics for the adapter module, For the input characteristics of the adapter module, and For the projection matrix, It is a non-linear activation function.
4. The infrared image task adaptive method based on cue learning and adapter according to claim 3, characterized in that, The nonlinear activation function is the GELU function, and the approximate expression of the GELU function is: in, The output value of the GELU activation function. As input features, It is the hyperbolic tangent function. Pi is the mathematical constant of a circle.
5. The infrared image task adaptive method based on cue learning and adapter according to claim 1, characterized in that, The total loss function fine-tuned in step S3 Loss due to mission Sparse regularization loss for visual cue vectors The total loss function is expressed as follows: in, For hyperparameters, For the visual cue vector P Norm regularization.
6. The infrared image task adaptive method based on cue learning and adapter according to claim 5, characterized in that, When the target downstream task is a classification task, the task loss This represents the cross-entropy loss.
7. The infrared image task adaptive method based on cue learning and adapter according to claim 1, characterized in that, The compression and conversion process described in step S4 specifically includes: The fine-tuned model is then subjected to structured pruning, removing low-contribution cue vectors and adapter connections; The pruned model weights are quantized, and the floating-point parameters are converted into a low-bit-width fixed-point representation. The quantized model is converted into an inference engine format supported by the edge computing device.
8. The infrared image task adaptive method based on cue learning and adapter according to claim 7, characterized in that, The quantization is post-training static quantization. First, a small amount of unlabeled data is passed through the model to record the distribution of activation values in each layer. Then, the FP32 weights W and activation values are quantized to INT8. The quantization formula is: in, The scaling factor determined from the calibration data. This is a numerical truncation function.
9. The infrared image task adaptive method based on cue learning and adapter according to claim 7, characterized in that, The saliency of each visual cue vector is evaluated by calculating its L2 norm or activation contribution, and the least significant cue vector is removed. A prompt vector, are positive integers and .