Cross-task knowledge distillation method, device and medium from image classification to semantic segmentation
By re-encoding the pre-trained large model and performing cross-task knowledge distillation, the application problem of large models on devices with limited computing resources is solved, and the performance of the semantic segmentation model is improved.
Patent Information
- Application Number
- CN202310499648.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-05
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2043-05-05
AI Technical Summary
In existing technologies, large models are difficult to apply on hardware devices with limited computing resources, and existing knowledge distillation methods fail to effectively solve the problem of model compression in cross-task scenarios.
By re-encoding the pre-trained large model, inserting a projection layer to align the output dimension of the student model, and using cross-entropy, inter-class distance and KD loss functions to train the backbone network of the student model, cross-task knowledge distillation is achieved.
It reduces the computational cost of training and improves the performance of semantic segmentation models, enabling their effective application on devices with limited computing resources.
Smart Images
Figure CN116681887B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of data processing and model compression technology, and in particular to a cross-task knowledge distillation method, device and medium for image classification to semantic segmentation. Background Art
[0002] With the development of deep learning, large models based on numerous parameters and massive amounts of data have been proposed repeatedly. These models, when fully fine-tuned, have demonstrated impressive performance advantages. This has been accompanied by a gradual increase in computing power requirements. The enormous computational overhead has made it difficult for ordinary users to apply these large models to specific downstream tasks.
[0003] To reduce the computational cost of training large models, model recoding techniques first adjust the model's structure and then train only a subset of its parameters. This effectively reduces the overhead of fine-tuning large models for downstream tasks without compromising final performance. However, while recoding techniques reduce model training costs, they cannot compress large models or reduce the cost of model inference. Therefore, pre-trained large models remain difficult to implement on hardware devices with limited storage and computing resources.
[0004] Knowledge distillation, on the other hand, is a model compression technique that primarily transfers structured or unstructured knowledge to enable a smaller model (the student model) to approach the performance of a larger model (the teacher model). The key issue lies in designing a suitable knowledge transfer mechanism. However, existing knowledge distillation methods assume that the teacher and student models are trained on the same task and lack specialized design for cross-task scenarios. Summary of the Invention
[0005] In order to at least solve one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a cross-task knowledge distillation method, device and medium from image classification to semantic segmentation.
[0006] The technical solution adopted in the present invention is:
[0007] A cross-task knowledge distillation method from image classification to semantic segmentation includes the following steps:
[0008] Re-encode the structure of the pre-trained large model;
[0009] Fine-tune some parameters of the pre-trained large model;
[0010] Copy the decoder parameters of the trained large model to the decoder of the student model;
[0011] Fix the decoder of the student model and use the preset loss function to train the backbone network of the student model;
[0012] Reduce the learning rate and use the preset loss function to continue training all student models to obtain the final semantic segmentation model for semantic segmentation.
[0013] Furthermore, the re-encoding of the structure of the pre-trained large model includes:
[0014] According to the position of the feature layer {L1,…,L i ,…,L P}, in the L i A fully connected layer, an activation layer, and a layer normalization layer are inserted between the layer and the decoder to construct a projection layer, and the output dimension of the projection layer is aligned with the output dimension of the student model; where P represents a total of P feature maps input to the decoder;
[0015] The decoder of the teacher model reuses the decoder structure of the student model.
[0016] Furthermore, the fine-tuning of some parameters of the pre-trained large model includes:
[0017] The projection layer and decoder in the teacher model are trained using the cross-entropy loss function and the AdamW optimizer.
[0018] Furthermore, copying the parameters of the decoder in the trained large model to the decoder of the student model includes:
[0019] Since the projection layer is used to align the output feature shape of the large model with the output feature shape of the student model when re-encoding the pre-trained large model, the decoder parameters in the large model are copied to the decoder of the student model after the large model is trained.
[0020] Furthermore, the decoder of the fixed student model uses a preset loss function to train the backbone network of the student model, including:
[0021] The student model decoder is fixed and copied, and a batch of training samples is randomly selected. The backbone network parameters W of the student model are optimized by the AdamW algorithm. b Optimize;
[0022] Among them, the loss functions used include the cross entropy loss function L ce , inter-class distance loss function L icrkd And KD loss function L kd .
[0023] Furthermore, the cross entropy loss function L ce The expression is:
[0024]
[0025] The inter-class distance loss function L icrkd The expression is:
[0026]
[0027] The KD loss function L kd The expression is:
[0028]
[0029] The final loss function expression is:
[0030] L(W b )=L ce (W b )+λL kd (W b )+βL icrkd (W b )
[0031] Where n represents the total number of sample batches, H, W are the width and height of the model prediction, h, w are the width and height of the feature, respectively; y i,j Indicates the true category of the j-th pixel in the i-th sample, predict i,j Represents the output of the model at the jth pixel in the i-th sample; C represents the total number of categories in the i-th sample, sim i,c1,c2 It represents the inter-class distance between category c1 and category c2 extracted from the teacher or student model, the superscript S represents the student model, and the superscript T represents the teacher model; λ and β represent the weights of the corresponding loss function.
[0032] Furthermore, the inter-class distance sim i,c1,c2 Calculated in the following way:
[0033] Scale the true label of the image to the matrix sim i,c Same width and height;
[0034] According to the category of the scaled true label, the intermediate layer features of the decoder of the teacher model and / or student model are extracted according to the category to obtain features [features i,0 ,…,features i,c ,…,features i,C ];
[0035] Calculate sim based on the extracted features i,c1,c2 :
[0036]
[0037] Furthermore, the learning rate is reduced, and a preset loss function is used to continue training all student models to obtain the final semantic segmentation model, including:
[0038] The decoder of the student model is no longer fixed, and the learning rate of the decoder is set to a preset value;
[0039] Use the preset loss function to train the parameters W of the backbone network and decoder b and W d , until convergence.
[0040] Another technical solution adopted in the present invention is:
[0041] A cross-task knowledge distillation device from image classification to semantic segmentation, including:
[0042] at least one processor;
[0043] at least one memory for storing at least one program;
[0044] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.
[0045] Another technical solution adopted in the present invention is:
[0046] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to perform the method described above when executed by the processor.
[0047] The beneficial effect of the present invention is that the present invention solves the problem that it is difficult to directly apply large models to downstream tasks when computing resources are limited, and the performance of the semantic segmentation model can be improved with a smaller computing cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.
[0049] Figure 1 is a schematic diagram of a cross-task knowledge distillation method from image classification to semantic segmentation in an embodiment of the present invention;
[0050] Figure 2 1 is a schematic diagram of calculating the distance between classes in an embodiment of the present invention;
[0051] Figure 3This is a flowchart of the steps of a cross-task knowledge distillation method from image classification to semantic segmentation in an embodiment of the present invention. DETAILED DESCRIPTION
[0052] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
[0053] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.
[0054] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.
[0055] Furthermore, in the description of this invention, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.
[0056] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.
[0057] In order to efficiently transfer the performance of large models to downstream tasks, such as Figure 1 and Figure 3 As shown, this embodiment provides a cross-task knowledge distillation method from image classification to semantic segmentation, including the following steps:
[0058] S1. Re-encode the structure of the pre-trained large model.
[0059] In this embodiment, the position {L1, ..., L i ,…,L P} (where P indicates that a total of P feature maps are input to the decoder), in the Lth i A projection layer consisting of a fully connected layer, an activation layer, and a layer normalization layer is inserted between the layer and the decoder. The output dimension of the projection layer is aligned with the output dimension of the student model. The decoder of the teacher model reuses the decoder structure of the student model.
[0060] S2. Fine-tune some parameters of the pre-trained large model.
[0061] In this embodiment, the projection layer and decoder in the teacher model are trained using the cross entropy loss function and the AdamW optimizer. It is worth noting that since only the decoder and projection layers are trained, the computational cost of training is greatly reduced compared to fully fine-tuning the pre-trained large model.
[0062] S3. Copy the parameters of the decoder in the trained large model to the decoder of the student model.
[0063] Since the projection layer is used to align the output feature shape of the large model with the output feature shape of the student model when re-encoding the pre-trained large model, after the training is completed, the decoder parameters in the large model are copied to the decoder of the student model.
[0064] S4. Fix the decoder of the student model and use the preset loss function to train the backbone network of the student model.
[0065] In this embodiment, the decoder of the copied student model is fixed, a batch of training samples is randomly selected, and the backbone network parameters W of the student model are optimized by the AdamW algorithm. b We minimize the cross entropy loss function L ce , inter-class distance loss function L icrkd , KD loss function L kd The number of optimization iterations is manually selected, usually 5000 batches.
[0066] Cross entropy loss function L ce The expression is:
[0067]
[0068] Inter-class distance loss function L icrkd The expression is:
[0069]
[0070] KD loss function L kd The expression is:
[0071]
[0072] Where n represents the total number of sample batches, H and W are the width and height of the model prediction, and h and w are the width and height of the feature. i,c1,c2 Represents the inter-class distance between category c1 and category c2 extracted from the teacher or student model, see Figure 2 , the specific calculation steps are as follows:
[0073] S41. Scale the true label of the image to sim i,c Same width and height
[0074] S42, according to the category of the real label after scaling, extract the intermediate layer features of the teacher (student) model decoder according to the category, and obtain [features i,0 ,…,features i,c ,…,features i,C ].
[0075] S43, calculation sim i,c1,c2 .
[0076]
[0077] The final loss function is defined as:
[0078] L(W b )=L ce (W b )+λL kd (W b )+βL icrkd (W b )
[0079] Among them, λ and β represent the weights of the corresponding loss function.
[0080] S5. Reduce the learning rate and use the preset loss function to continue training all student models to obtain the final semantic segmentation model for semantic segmentation.
[0081] The decoder of the student model is no longer fixed and its learning rate is set to one tenth of the backbone network. The parameters W of the backbone network and decoder are trained using the above loss function. b and W d , until convergence.
[0082] L(W b ,W d )=L ce (Wb ,W d )+λL kd (W b ,W d )+βL icrkd (W b ,W d )
[0083] In this embodiment, the pre-trained large model refers to any pre-trained large model, and the training set is the semantic segmentation training set used by the student model.
[0084] In summary, compared with the prior art, the present invention has at least the following beneficial effects:
[0085] (1) This paper adopts model recoding technology to migrate pre-trained large models across tasks, reducing the computational cost of training.
[0086] (2) The present invention takes the student model into consideration during the re-encoding of the large model, and transfers the parameters as a kind of knowledge.
[0087] (3) Currently, most pre-trained large models are based on image classification tasks. Therefore, the inter-class distance loss function is used to improve the category identification ability of the semantic segmentation model.
[0088] (4) This paper proposes a framework for applying pre-trained large models to semantic segmentation tasks.
[0089] This embodiment also provides a cross-task knowledge distillation device from image classification to semantic segmentation, including:
[0090] at least one processor;
[0091] at least one memory for storing at least one program;
[0092] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 3 The method shown.
[0093] A cross-task knowledge distillation device from image classification to semantic segmentation in this embodiment can execute a cross-task knowledge distillation method from image classification to semantic segmentation provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0094] The present application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 3 The method shown.
[0095] This embodiment also provides a storage medium storing instructions or programs that can execute a cross-task knowledge distillation method from image classification to semantic segmentation provided by an embodiment of the method of the present invention. When the instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.
[0096] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.
[0097] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0098] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0099] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0100] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.
[0101] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0102] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
[0103] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.
[0104] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A cross-task knowledge distillation method from image classification to semantic segmentation, characterized by: The following steps are involved: Re-encode the structure of the pre-trained large model; Fine-tune some parameters of the pre-trained large model; Copy the decoder parameters of the trained large model to the decoder of the student model; Fix the decoder of the student model and use the preset loss function to train the backbone network of the student model; the loss function used includes the cross entropy loss function , inter-class distance loss function and KD loss function ; Wherein, the inter-class distance loss function The expression is: Where, represents the total number of sample batches, Indicates the The total number of categories in the samples, Indicates the inter-class distance between category c1 and category c2 extracted from the teacher or student model, with the superscript Indicates student model, superscript represents the teacher model; Reduce the learning rate and use the preset loss function to continue training all student models to obtain the final semantic segmentation model for semantic segmentation.
2. The cross-task knowledge distillation method from image classification to semantic segmentation according to claim 1, characterized in that: The re-encoding of the structure of the pre-trained large model includes: According to the position of the feature layer input to the decoder by the pre-trained large model { }, in the A fully connected layer, an activation layer, and a layer normalization layer are inserted between the layer and the decoder to construct a projection layer, and the output dimension of the projection layer is aligned with the output dimension of the student model; Indicates the total input to the decoder feature maps; The decoder of the teacher model reuses the decoder structure of the student model.
3. The cross-task knowledge distillation method from image classification to semantic segmentation according to claim 2, characterized in that: The fine-tuning of some parameters of the pre-trained large model includes: The projection layer and decoder in the teacher model are trained using the cross-entropy loss function and the AdamW optimizer.
4. The cross-task knowledge distillation method from image classification to semantic segmentation according to claim 2, characterized in that: The process of copying the parameters of the decoder in the trained large model to the decoder of the student model includes: Since the projection layer is used to align the output feature shape of the large model with the output feature shape of the student model when re-encoding the pre-trained large model, the decoder parameters in the large model are copied to the decoder of the student model after the large model is trained.
5. The cross-task knowledge distillation method from image classification to semantic segmentation according to claim 1, characterized in that: The decoder of the fixed student model uses a preset loss function to train the backbone network of the student model, including: Fixed the copied student model decoder, randomly selected batch training samples, and optimized the backbone network parameters of the student model through the AdamW optimization algorithm. Optimize.
6. The cross-task knowledge distillation method from image classification to semantic segmentation according to claim 1, characterized in that: The cross entropy loss function The expression is: The KD loss function The expression is: The final loss function expression is: in, are the width and height predicted by the model, are the width and height of the feature respectively; Indicates the In the sample The true category of pixels, Indicates that the model In the sample The output of pixels; 、 Represents the weight of the corresponding loss function.
7. The cross-task knowledge distillation method from image classification to semantic segmentation according to claim 1, characterized in that: The inter-class distance Calculated in the following way: Scale the true label of the image to a matrix Same width and height; According to the category of the scaled true label, the intermediate layer features of the decoder of the teacher model and / or student model are extracted according to the category to obtain the features ; Calculate based on the extracted features : 。 8. The cross-task knowledge distillation method from image classification to semantic segmentation according to claim 5, characterized in that: The learning rate is reduced, and the preset loss function is used to continue training all student models to obtain the final semantic segmentation model, including: The decoder of the student model is no longer fixed, and the learning rate of the decoder is set to a preset value; Use the preset loss function to train the parameters of the backbone network and decoder and , until convergence.
9. A cross-task knowledge distillation device from image classification to semantic segmentation, characterized by: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 8.
10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 8 when executed by the processor.