Model fusion method based on knowledge distillation and continuous learning

Abstract

The invention provides a model fusion method based on knowledge distillation and continuous learning, and belongs to the technical field of target detection, and the method comprises the following steps: S1, constructing a model architecture, and obtaining a fusion model architecture with a plurality of task heads; s2, initializing the head part of the fusion model architecture; s3, the knowledge of the second model CLIP is transferred to the first model ZoomNeXt, and finally a fusion model ZoomNeXt-CLIP is obtained; according to the fusion method, the ZoomNeXt and the CLIP with the powerful language and visual semantic understanding ability are integrated into one model, obvious forgetting caused by a traditional weight average fusion method is overcome, and finally the performance obviously superior to that of an existing model is achieved in multiple benchmark tests such as zero sample semantic segmentation.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, specifically relating to a fusion method based on knowledge distillation and continuous learning models. Background Technology

[0002] Disguised object detection is a novel and interesting visual detection task, but it is also full of challenges. The limitations of most current methods are mainly reflected in the high similarity between the disguised object and the background, the diversity of the object in scale and appearance, the ambiguity, and the problem of severe occlusion, or the poor performance under the influence of salient objects on the COD task.

[0003] With the development and maturation of the field of deep learning, deep learning-based COD methods have gradually become mainstream. Compared with traditional manual methods, deep learning methods can use feature extraction networks to extract features with higher semantic value from images.

[0004] Current advanced COD methods include diffusion model-based methods, weakly supervised and semi-supervised methods, zero-shot and small-shot methods, multimodal data fusion methods, and general fully supervised methods. The basic model presented in this paper, ZoomNeXt, is a general fully supervised missile method. It improves the detection performance of camouflaged targets by simulating human zooming in and out behavior, employing a collaborative pyramid structure, and fusing multi-scale information. Summary of the Invention

[0005] To improve ZoomNeXt's zero-shot capability and semantic understanding ability, and to overcome the problem of significant forgetting caused by traditional weighted average fusion methods, this invention uses a novel multi-task distillation method to combine ZoomNeXt with CLIP, which has strong language and visual semantic understanding capabilities, into a single model, thus overcoming the significant forgetting caused by traditional weighted average fusion methods.

[0006] This invention provides a method for fusing models based on knowledge distillation and continuous learning, comprising the following steps:

[0007] S1. Construct the model architecture to obtain a fusion model architecture with multiple task headers;

[0008] S2. Initialize the header of the fusion model;

[0009] S3. Transfer the knowledge of the second model CLIP to the first model ZoomNeXt to obtain the fusion model ZoomNeXt-CLIP.

[0010] Furthermore, the fusion model architecture described in step S1 includes a first model ZoomNeXt and a second model CLIP;

[0011] The first model, ZoomNeXt, includes an image encoder, EncZoomNeXt, and a mask decoder, MaskDecZoomNeXt.

[0012] The second model CLIP includes an image encoder EncCLIP and a text encoder TextEncCLIP.

[0013] Furthermore, step S1 specifically includes the following steps:

[0014] S11. Construct a shared backbone network EncZoomNeXt-CLIP;

[0015] S12. Configure two lightweight task headers for the shared backbone network: a first header HeadZoomNeXt for executing the original task of the first model and a second header HeadCLIP for executing the original task of the second model.

[0016] Furthermore, in step S12, the first head HeadZoomNeXt is initialized by the mask decoder MaskDecZoomNeXt of the first model ZoomNeXt;

[0017] The second head, HeadCLIP, is initialized by the weights of the last layer of the first model, ZoomNeXt.

[0018] Furthermore, step S2 specifically includes the following steps:

[0019] S21. Freeze the shared backbone network EncZoomNeXt-CLIP and the first headZoomNeXt;

[0020] S22. Train only the second head CLIP and use the cosine distillation loss function to minimize the difference between the output of the student model (ZoomNeXt-CLIP) and the teacher model (CLIP).

[0021] Furthermore, the cosine distillation loss function described in step S22 satisfies the following formula: In formula (1), x is a sample from the CLIP dataset, Enc ZoomNeXt-CLIP (x) is the feature map of the shared backbone pair x, Enc CLIP (x) is the image embedding vector of x by the teacher CLIP's image encoder, φ Pooling It is a space pooling operator.

[0022] Furthermore, step S3 specifically includes the following steps:

[0023] S31. Prepare a mixed dataset for data training:

[0024] S32. Construct and calculate the multi-task loss function, i.e., the total loss function;

[0025] S33. Execute the training loop to obtain the unified fusion model ZoomNeXt-CLIP after fusion.

[0026] Furthermore, in step S32, the total loss function satisfies: Let cosine be the distillation loss function. Let be the spatial mask distillation loss function.

[0027] Furthermore, in the training loop described in step S33, the learning rate of the second head header is set to 4 × 10⁻⁶. -5 Furthermore, the learning rates of the second model ZoomNeXt and the second head HeadZoomNeXt are both an order of magnitude smaller than those of HeadCLIP.

[0028] Compared to traditional methods, this invention first employs a lower learning rate to update the shared backbone network EncZoomNeXt-CLIP and the first headZoomNeXt portion, in order to reduce the forgetting of the original ZoomNeXt capabilities.

[0029] By using a multi-task learning approach, CLIP's semantic understanding and ZoomNeXt's spatial understanding capabilities are effectively integrated into a unified model, ultimately achieving significantly better performance than existing models in multiple benchmark tests, such as zero-shot semantic segmentation.

[0030] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. Attached Figure Description

[0031] Figure 1 This is a diagram of the lightweight CLIP header structure.

[0032] Figure 2 This is a flowchart of the training process for the fusion model architecture.

[0033] Figure 3 This is a flowchart of the fusion model architecture inference process. Detailed Implementation

[0034] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the specific implementation methods, structural features and effects of the present invention are described in detail below with reference to the accompanying drawings and embodiments.

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] In the description of this invention, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "aligned", "overlapping", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0037] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature; in the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0038] Example 1

[0039] This embodiment provides, for example Figures 1-3 The method shown is a fusion of knowledge distillation and continuous learning models, including the following steps:

[0040] S1. Construct a model architecture to obtain a fusion model architecture with multiple task heads; Step S1 constructs the foundation of this invention: a multi-head fusion model architecture, which is the prerequisite for realizing model fusion. The fusion model is what this invention ultimately needs to complete, a fusion model that can perform multiple tasks by fusing the first model ZoomNeXt and the second model CLIP: ZoomNeXt-CLIP, which is finally obtained from the fusion model architecture through training and distillation.

[0041] ZoomNeXt is a unified collaborative pyramid network for camouflaged object detection. It is a framework that unifies the processing of image and video COD tasks through feature selection and aggregation, and can reliably capture camouflaged objects in complex scenes.

[0042] CLIP is a deep learning model proposed by OpenAI, designed to process text and images simultaneously. It uses self-supervised contrastive learning to pre-train on unlabeled image and text data, enabling the model to understand the semantic relationships between images and text.

[0043] ZoomNeXt performs well in processing camouflaged targets due to its ability to extract multi-scale features, but it is insufficient in understanding complex semantic information in images and lacks generalization ability. In order to further improve ZoomNeXt's shortcomings in semantic understanding and improve the model's generalization ability, this invention uses ZoomNeXt as the first model: the basic model, and CLIP, which has strong language and visual semantic understanding capabilities, as the second model: the auxiliary model. Both are integrated into one model. Based on shared feature representation, CLIP's semantic capabilities are carried by an independent semantic head, while the segmentation head of ZoomNeXt and its existing parameters remain unchanged, thereby further overcoming the obvious forgetting caused by the traditional weighted average fusion method.

[0044] S2. Initialize the header of the fusion model architecture;

[0045] Step S2 aims to ensure that the knowledge of the second model CLIP can be effectively transferred to the new model, while also avoiding interference with the original capabilities of the first model ZoomNeXt during the fusion process. That is, when introducing a new task (the task of CLIP), the existing capabilities (the capabilities of ZoomNeXt) are protected from being destroyed. This is a key step in "continuous learning".

[0046] S3. Using multi-task distillation technology, the knowledge of the CLIP model is effectively transferred to the ZoomNeXt model, and finally a unified fusion model ZoomNeXt-CLIP is obtained.

[0047] In the multi-task distillation stage, i.e. step S3, the distillation loss of the second model CLIP and the first model ZoomNeXt is jointly optimized to transfer the global semantic understanding capability of CLIP to the ZoomNeXt backbone while minimizing the forgetting of the original masquerading segmentation capability of ZoomNeXt. Finally, the fusion of the fusion model architecture is truly completed, and a unified fusion model ZoomNeXt-CLIP is obtained.

[0048] In the multi-task distillation process, in order to avoid the loss of the original capabilities of the ZoomNeXt model, this invention uses a low learning rate, that is, a low step size for each update, to update the EncZoomNeXt-CLIP and HeadZoomNeXt parts, and introduces a data replay mechanism for repeated training and learning to ensure that the learning of new knowledge does not interfere with the preservation of the original knowledge.

[0049] Furthermore, the fusion model architecture in step S1 includes the first model ZoomNeXt and the second model CLIP;

[0050] The first model, ZoomNeXt, includes an image encoder, EncZoomNeXt, and a mask decoder, MaskDecZoomNeXt.

[0051] The second model CLIP includes an image encoder EncCLIP and a text encoder TextEncCLIP;

[0052] The first model, ZoomNeXt, and the second model, CLIP, are the main architectural components of the fusion model architecture. Specifically, the image encoders EncZoomNeXt and CLIP are used to construct the shared backbone network EncZoomNeXt-CLI of the fusion model. The mask decoder MaskDecZoomNeXt of the first model is used to construct the lightweight head.

[0053] Furthermore, step S1 specifically includes the following steps:

[0054] S11. Construct a shared backbone network EncZoomNeXt-CLIP; the construction of the shared backbone network EncZoomNeXt-CLIP is based on the image encoder EncZoomNeXt of the first model ZoomNeXt and the image encoder EncCLIP of the second model CLIP.

[0055] The backbone is initialized with the weights of the first model ZoomNeXt's image encoder EncZoomNeXt. This backbone undertakes the unified feature extraction function and subsequently provides consistent intermediate representations for both the semantic head CLIP and the segmentation head ZoomNeXt. Based on this, subsequent training will gradually inject the semantic understanding capabilities of the open vocabulary into the same backbone through semantic alignment and distillation with the CLIP image encoder, so that it has the synergistic advantages of spatial representation and semantic representation.

[0056] The second model CLIP's image encoder, EncCLIP, adopts a visual Transformer structure and outputs image-level semantic embeddings. It has open semantic representation capabilities. A dimension adaptation layer is added on the output side of the shared backbone network to align the hidden dimensions of EncCLIP, and the image-level semantic embeddings generated by EncCLIP are used as the alignment target.

[0057] The shared backbone network EncZoomNeXt-CLIP is one of the core innovations of this invention. It is not simply a matter of connecting two models in parallel or in series, but rather aims to create a single, powerful feature extractor. The shared backbone network initializes the image encoder EncZoomNeXt of the first model ZoomNeXt, which means that it has powerful spatial detail and multi-scale feature extraction capabilities from the beginning to handle camouflaged targets. The goal is to subsequently "inject" the semantic understanding capabilities of CLIP into this unified shared backbone network through distillation learning.

[0058] S12. Configure two lightweight task headers for the shared backbone network: a first header HeadZoomNeXt for executing the original task of the first model and a second header HeadCLIP for executing the original task of the second model.

[0059] The first head, HeadZoomNeXt, is the segmentation head, responsible for decoding features into pixel-level segmentation masks and performing the original task of the first model, ZoomNeXt: camouflaged target segmentation. The second head, HeadCLIP, is the semantic head, responsible for converting features into image-level embedding vectors and aligning them with the output of the CLIP text encoder in the same semantic space, performing CLIP's original tasks: image-text matching and zero-shot classification.

[0060] The architecture of the fusion model is as follows Figure 2 As shown, the goal of step S1 is to merge the image encoders of the two models into a shared backbone network EncZoomNeXt-CLIP. Step S2 configures lightweight heads HeadZoomNeXt and HeadCLIP for the shared backbone network EncZoomNeXt-CLIP, respectively, to form a fusion model architecture. Each head of the fusion model architecture is a lightweight neural network layer (such as several convolutional layers or Transformer layers), which maps the general features extracted by the shared backbone network to the output space of a specific task. The shared backbone network combines features from ZoomNeXt and CLIP through a multi-head collaboration mechanism to form a unified network, ensuring that the advantages of the two are complementary. This "shared backbone + dedicated head" design is the key to the model's ability to perform multiple tasks efficiently, which avoids the huge overhead of maintaining a complete model for each task.

[0061] Furthermore, in step S12, the first head HeadZoomNeXt is initialized by the mask decoder MaskDecZoomNeXt of the first model ZoomNeXt. The first head HeadZoomNeXt is originally intended to perform the same segmentation task. It is initialized by the mask decoder MaskDecZoomNeXt of the first model ZoomNeXt. The initial parameters of the mask decoder MaskDecZoomNeXt are already close to the optimal, which can accelerate convergence and improve the final performance.

[0062] The second head, HeadCLIP, is initialized by the weights of the last layer of the first model, ZoomNeXt. It can directly inherit the stable spatial representations learned by the shared backbone and segmentation head in the target domain, thereby accelerating early convergence and reducing training oscillations. It is more preferred in scenarios with small sample sizes or where it is necessary to quickly achieve usable performance. The last layer features of the first model, ZoomNeXt, include high-level semantic information. Starting from this, the fusion model is more conducive to learning the semantic mapping of CLIP.

[0063] Furthermore, step S2 specifically includes the following steps:

[0064] S21. Freeze the shared backbone network EncZoomNeXt-CLIP and freeze the first head HeadZoomNeXt. Only allow the parameters of the second head HeadCLIP to be updated. By preserving the spatial representation obtained by initializing ZoomNeXt, we can avoid disturbing the original segmentation ability in the early stage of introducing semantic capabilities. This is the first line of defense against forgetting.

[0065] S22. Train only the second head CLIP and use the cosine distillation loss function to minimize the difference between the output of the student model (ZoomNeXt-CLIP) and the teacher model (CLIP);

[0066] Under the frozen settings in step S21, only the second head CLIP is trained. With image-level semantic embedding as the alignment target, the cosine distillation loss function is used to minimize the cosine distillation loss between the image embedding of HeadCLIP and the image embedding of the teacher CLIP image encoder, that is, to minimize the difference between the output of the student model (ZoomNeXt-CLIP) and the teacher model (CLIP).

[0067] Step S22 mainly involves preheating the head for distillation, which is a preliminary step for the subsequent overall distillation. With the shared backbone network and the first head (HeadZoomNeXt) frozen, only the second head (HeadCLIP) is trained. To better support semantic alignment capabilities, HeadCLIP adopts a lightweight structure with three Transformer layers. The structure diagram of the CLIP head is shown below. Figure 1 As shown, it aims at image-level semantic embedding. After the patch token is encoded, it is max-pooled to form an image-level embedding. Then, it minimizes the cosine distance between the image embedding of HeadCLIP and the image embedding of Teacher CLIP through linear projection (after L2 normalization of the two vectors, the negative of their inner product is added by 1, which reflects the magnitude of the difference in the direction of the two vectors. The smaller the value, the more consistent the semantics). In this way, the cosine distance between the image embedding of HeadCLIP and the image embedding of Teacher CLIP is minimized. Here, image-level semantic embedding refers to the fixed-length vector obtained by aggregating the patch-level features output by HeadCLIP in the spatial dimension and performing linear projection. It is a global representation of the entire image in the semantic embedding space.

[0068] In the CLIP pre-training dataset Knowledge distillation is applied, using a cosine distillation loss function to measure the output difference between the student model (i.e., the merged model: ZoomNeXt-CLIP) and the teacher model (CLIP). The goal is to train the second head (HeadCLIP) through the cosine distillation loss function so that its output is aligned with the output of the image encoder of the second model, CLIP. The goal of training only the second head (HeadCLIP) is to teach it how to "understand" the features of the current output of the shared backbone network (these features are currently more in line with the style of ZoomNeXt) and map them into the semantic space of CLIP.

[0069] In order to further avoid forgetting the original capabilities of ZoomNeXt, this invention employs multi-task distillation based on memory replay in subsequent steps. This can facilitate the effective transfer of knowledge from the auxiliary model to the basic model while preserving the original functionality of the basic model.

[0070] Furthermore, the cosine distillation loss function in step S22 satisfies the following formula: In the diagram, x represents a sample from the CLIP dataset, and Enc... ZoomNeXt-CLIP (x) is the feature map of the shared backbone pair x, Enc CLIP (x) is the image embedding vector of x by the teacher CLIP's image encoder. It is the expectation of the data distribution, φ PoolingIt is a spatial pooling operator. The spatial pooling operator is used to aggregate the patch-level features of the second head CLIP into an image-level vector. Patch-level features are obtained from the second head CLIP, and a normalized image-level embedding is generated. By minimizing the angle between the image-level embedding of the student-side head CLIP and the image embedding of the teacher CLIP, the current (ZoomNeXt-style) features of the shared backbone network are mapped to the semantic space of the CLIP, injecting zero-sample semantic capabilities in a low-interference manner.

[0071] This stage uses the Adam optimizer, with 8×10 -4 The low learning rate training for 20 rounds, using 20 training rounds, can accelerate the semantic alignment of HeadCLIP, shorten the convergence time, and avoid disturbing the existing spatial representation, all while sharing the backbone and freezing the segmentation head.

[0072] Furthermore, step S3 specifically includes the following steps:

[0073] S31. Prepare a mixed dataset for data training: The hybrid dataset is the pre-training dataset for the second model CLIP. The training dataset of the first model, ZoomNeXt. A portion of data was extracted from each of the two groups and mixed together to provide a data foundation for subsequent training and learning steps.

[0074] As a knowledge source, it is used to drive the shared backbone network to learn the semantic features of CLIP. As a "memory anchor," it serves to remind the model not to forget how to complete the segmentation task, which is key to overcoming catastrophic forgetting;

[0075] S32. Construct and calculate the multi-task loss function, i.e., the total loss function; in subsequent training cycles, for each sample in the training batch, calculate different loss components according to their data sources, and combine them into the total loss according to their weights.

[0076] The total loss function is The total loss function is in mixed data The dataset trained on the first model The dataset of the second model Mixed datasets: The total loss function is jointly optimized using a multi-task learning approach. During training, each batch contains some data from... Samples and some from The samples, respectively for and Make a contribution;

[0077] This forces the image features output by the fusion model to align with the features of the original CLIP model in the semantic space, thereby transferring the semantic knowledge of CLIP.

[0078] By having the fusion model mimic the output of its "predecessor" (i.e., the first model ZoomNeXt), we ensure that its segmentation performance does not degrade.

[0079] S33. Execute the training loop to obtain the unified fusion model ZoomNeXt-CLIP after fusion;

[0080] This step involves 40,000 training cycles, each cycle consisting of two independent forward and backward propagation processes: CLIP distillation and ZoomNeXt distillation are performed independently, and the gradients of the CLIP and ZoomNeXt parts are calculated and accumulated independently. After each forward and backward propagation process is completed, the parameters are updated.

[0081] Furthermore, in step S32, the total loss function describing cross-modal consistency satisfies: Let cosine be the distillation loss function. The spatial mask distillation loss function;

[0082] Cosine distillation loss function The following is obtained in advance in step S22:

[0083] Spatial mask distillation loss function Then it is:

[0084]

[0085] (2), in formula (2), It is a combined distillation loss used to simultaneously align probabilities and shape boundaries, where x is the original image, and Enc is the shape boundary. ZoomNeXt-CLIP (x) is the feature map of the shared backbone pair x. ZoomNeXt It is a soft mask label output by the student-side segmentation head, which is the same size as or downsampled from the image. It is the expectation of data distribution. By having the mask prediction of the student segmentation head closely match the soft label of the teacher's ZoomNeXt in terms of both pixel probability and contour consistency, it consolidates the original spatial details and boundary expressions while introducing CLIP semantic capabilities. This is equivalent to providing "memory anchors" for spatial behavior for the shared backbone, thereby suppressing the forgetting of the original ZoomNeXt capabilities. z is the segmentation mask score generated by the frozen ZoomNeXt, z = MaskDec ZoomNeXt(Enc ZoomNeXt (x));

[0086] The total loss function is the core of the multi-task distillation stage. It enables the model to simultaneously learn the semantic understanding of CLIP and the segmentation ability of ZoomNeXt, and balances the two tasks through the weight λ to prevent catastrophic forgetting and achieve efficient knowledge fusion. This allows us to use the standard gradient descent algorithm to optimize the two tasks simultaneously without the need for a complex alternating training strategy. During the training loop, the balance coefficient λ = 10 can achieve a relative balance between learning the ability of CLIP and mitigating the forgetting of the ability of ZoomNeXt.

[0087] Furthermore, in step S33 of the training loop, the learning rate of the second head header is set to 4 × 10. -5， Furthermore, the learning rates of the second model ZoomNeXt and the second head HeadZoomNeXt are both an order of magnitude smaller than those of HeadCLIP.

[0088] At this stage, the learning rate of HeadCLIP is reduced to 4×10. -5 In order to reduce forgetting, the learning rate of ZoomNeXt and HeadZoomNeXt is an order of magnitude smaller than that of HeadCLIP at this stage. Since HeadZoomNeXt and ZoomNeXt are the main carriers of the original knowledge, drastic updates will lead to catastrophic forgetting, so a low learning rate strategy is adopted.

[0089] The parameters of the shared backbone encoding network and the first head are updated using a learning rate lower than that used in the head initialization phase, and a memory replay mechanism is introduced to update the first model dataset. Some of the data was used for replay training to forget the problem;

[0090] At the same time, a data replay mechanism is introduced during the loop process, that is, 5%-10% of the original data is used for continuous learning during the training process to ensure that the learning of new knowledge does not interfere with the retention of the original knowledge, and further prevent catastrophic forgetting problems.

[0091] To avoid the forgetting problem of traditional models, this invention employs a novel training strategy that merges ZoomNeXt and CLIP into a single multi-task model with minimal forgetting. During the fusion process, the new model (i.e., the fused model ZoomNeXt-CLIP) is allowed to review a small amount of the original training data (5%-10% of the original training data) during training, which is used as "memory replay" in continuous learning. This avoids forgetting the knowledge of the original pre-trained model. Multi-task distillation is performed only on a subset of the original training data, and knowledge forgetting is prevented through phased loss constraints (head initialization + multi-task distillation).

[0092] The ZoomNeXt model performs well in processing camouflaged targets due to its ability to extract multi-scale features, but it is insufficient in understanding complex semantic information in images and lacks generalization ability. In order to further improve ZoomNeXt's shortcomings in semantic understanding and improve the model's generalization ability, this patent proposes a fusion method based on knowledge distillation, which can effectively combine CLIP, which has strong language and visual semantic understanding capabilities, with ZoomNeXt into a single model, overcoming the obvious forgetting caused by the traditional weighted average fusion method.

[0093] Unlike direct weighted averaging, this method adopts a "shared backbone + dedicated head" structure: new functions are mounted with independent heads and work in parallel with the original segmentation heads on the same backbone, without linearly mixing the existing weights; the design of parameter isolation and feature reuse reduces the perturbation to the existing representation from a structural perspective, thus significantly alleviating the forgetting that is easily caused by weighted averaging fusion. At the same time, CLIP is used as the teacher model and the fusion model ZoomNeXt-CLIP is used as the student model, thereby realizing the transfer of the zero-shot capability of the teacher model to the student model in subsequent steps.

[0094] We use CLIP as the teacher model and ZoomNeXt-CLIP as the student model to transfer the zero-shot capability of the teacher model to the student model. At the same time, we propose a method based on memory replay and multi-person distillation to further overcome catastrophic forgetting when new knowledge is introduced, and finally fuse the two models with minimal forgetting.

[0095] Compared to traditional multi-task training from scratch, our method requires significantly less computation and only a fraction of the pre-training dataset initially used to train a single model. By applying this method to ZoomNeXt and CLIP, we obtain a unified model, ZoomNeXt-CLIP, that combines the capabilities of both. Compared to deploying ZoomNeXt and CLIP independently, our merged model, ZoomNeXt-CLIP, reduces the storage and computational costs of inference, making it well-suited for edge device applications and improving performance for camouflaged target segmentation.

[0096] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for fusing models based on knowledge distillation and continuous learning, characterized in that, Includes the following steps: S1. Construct the model architecture to obtain a fusion model architecture with multiple task headers; S2. Initialize the header of the fusion model architecture; S3. Transfer the knowledge of the second model CLIP to the first model ZoomNeXt to obtain the fusion model ZoomNeXt-CLIP.

2. The fusion method of a model based on knowledge distillation and continuous learning as described in claim 1, characterized in that: The fusion model architecture described in step S1 includes a first model ZoomNeXt and a second model CLIP; The first model, ZoomNeXt, includes an image encoder, EncZoomNeXt, and a mask decoder, MaskDecZoomNeXt. The second model CLIP includes an image encoder EncCLIP and a text encoder TextEncCLIP.

3. The fusion method of a model based on knowledge distillation and continuous learning as described in claim 2, characterized in that, Step S1 specifically includes the following steps: S11. Construct a shared backbone network EncZoomNeXt-CLIP; S12. Configure two lightweight task headers for the shared backbone network: a first header HeadZoomNeXt for executing the original task of the first model and a second header HeadCLIP for executing the original task of the second model.

4. The fusion method of a model based on knowledge distillation and continuous learning as described in claim 3, characterized in that: In step S12, the first head HeadZoomNeXt is initialized by the mask decoder MaskDecZoomNeXt of the first model ZoomNeXt; The second head, HeadCLIP, is initialized by the weights of the last layer of the first model, ZoomNeXt.

5. The fusion method of a model based on knowledge distillation and continuous learning as described in claim 1, characterized in that, Step S2 specifically includes the following steps: S21. Freeze the shared backbone network EncZoomNeXt-CLIP and the first headZoomNeXt; S22. Train only the second head CLIP and use the cosine distillation loss function to minimize the difference between the output of the student model (ZoomNeXt-CLIP) and the teacher model (CLIP).

6. The fusion method of a model based on knowledge distillation and continuous learning as described in claim 5, characterized in that, The cosine distillation loss function described in step S22 satisfies the following formula: In formula (1), x is a sample from the CLIP dataset, Enc ZoomNeXt-CLIP (x) is the feature map of the shared backbone pair x, Enc CLIP (x) is the image embedding vector of x by the teacher CLIP's image encoder, φ Pooling It is a space pooling operator.

7. The fusion method of a model based on knowledge distillation and continuous learning as described in claim 1, characterized in that, Step S3 specifically includes the following steps: S31. Prepare a mixed dataset for data training: S32. Construct and calculate the multi-task loss function, i.e., the total loss function; S33. Execute the training loop to obtain the unified fusion model ZoomNeXt-CLIP after fusion.

8. The fusion method of a model based on knowledge distillation and continuous learning as described in claim 7, characterized in that, In step S32, the total loss function satisfies: Let cosine be the distillation loss function. Let be the spatial mask distillation loss function.

9. The fusion method of a model based on knowledge distillation and continuous learning as described in claim 7, characterized in that: In the training loop described in step S33, the learning rate of the second head header is set to 4 × 10. -5 Furthermore, the learning rates of the second model ZoomNeXt and the second head HeadZoomNeXt are both an order of magnitude smaller than those of HeadCLIP.

Images