Feature alignment system and training method for large-small model suitable for object detection

Abstract

A large-small model feature alignment system and training method suitable for target detection solve the problem of feature alignment between large-small models. The system comprises a spatial alignment submodule that adjusts the spatial size of the feature map of a teacher model to be consistent with the spatial size of a student model, and outputs a first image feature map that is the same as the teacher model in channel number and matches the student model in spatial size; a channel alignment submodule that performs channel compression and semantic remapping on the first image feature map to generate a second feature map that is the same as the student model in channel number; and the two submodules are connected in series in the order of spatial alignment first and channel alignment later, so that spatial structure adjustment and channel semantic mapping are independently performed in different stages. The system is deployed between large-small models, and uses a progressive layer decoupling distillation training method to gradually release the trainable layers of the network and introduce distillation constraints, effectively alleviating gradient conflicts and training shocks, and improving the stability and convergence of the overall training process.

Description

Technical Field

[0001] This invention relates to a technique for achieving refined feature alignment between large and small models, applicable to object detection. Background Technology

[0002] With the rapid development of deep learning technology, large-scale neural network models, represented by convolutional neural networks and visual Transformers, have made significant progress in fields such as computer vision, intelligent perception, and pattern recognition. Large models typically have deeper network structures, wider channel dimensions, and higher feature representation capabilities, exhibiting superior performance in tasks such as multi-scale object recognition and small object detection. However, such models often rely on substantial computational resources and storage overhead, making them difficult to deploy directly on edge devices, embedded platforms, or real-time application scenarios with limited computing power.

[0003] To balance model performance and deployment efficiency, researchers have proposed techniques such as model compression, knowledge distillation, and multi-model collaborative learning. Among these, knowledge distillation, based on a teacher model (large model) and a student model (small model), guides the small model to learn the intermediate image features or output distribution of the large model, and is widely considered an effective way to improve the performance of lightweight models. However, in practical applications, large and small models often differ significantly in network structure, feature resolution, and channel dimension, making it difficult to directly align their feature spaces, thus limiting the efficiency and stability of knowledge transfer.

[0004] In existing technologies, some methods perform coarse-grained matching of image features through simple upsampling, downsampling, or linear mapping, failing to simultaneously address the issues of spatial resolution differences and channel semantic inconsistencies. Other methods rely on fixed-structure alignment modules or additional highly complex networks, leading to unstable training processes, increased parameter overhead, and even the introduction of new performance bottlenecks. Therefore, how to achieve fine-grained feature alignment between large and small models while ensuring computational efficiency and structural flexibility remains a pressing scientific problem to be solved in the field of deep model compression and collaborative learning.

[0005] Therefore, there is an urgent need for a new feature alignment mechanism and training method to achieve efficient, flexible and stable feature alignment at both spatial resolution and channel dimension levels, thereby providing reliable support for knowledge transfer between large and small models. Summary of the Invention

[0006] To address the problems of existing feature alignment methods, which struggle to simultaneously consider spatial resolution differences and channel semantic inconsistencies, and which generally suffer from structural rigidity or excessive computational overhead, this invention proposes a "feature alignment system for heterogeneous models" and a "progressive layer decoupling distillation training method," providing an effective solution for efficient and accurate feature mapping between large and small models.

[0007] In terms of structural design, the feature alignment system proposed in this invention can establish an efficient and accurate feature mapping relationship between the teacher model and the student model. At the spatial level, the system introduces an adaptive spatial alignment mechanism, dynamically adjusting the feature sampling and reconstruction methods according to the scale differences of feature maps from different models, achieving unified alignment of multi-scale spatial information. At the channel level, it employs a learnable channel alignment strategy to map and reorganize the semantic distribution of feature channels in heterogeneous models, enabling the student model to selectively inherit the discriminative semantic information of the teacher model. Through decoupled modeling and collaborative optimization of spatial alignment and channel alignment, the system improves training stability and structural flexibility while reducing computational complexity and parameter redundancy.

[0008] Regarding the training strategy, this invention proposes a progressive training method that matches the feature alignment system to achieve stable and efficient feature alignment and knowledge transfer between large and small models. This method employs a phased, gradually unfreezing training strategy: initially, the backbone parameters of the teacher and student models are frozen, and only the feature alignment system and task output layer are trained, establishing a stable alignment benchmark through feature distillation loss; subsequently, the student model target layer is gradually unfrozen in order from high to low layers, introducing multi-level constraints of intermediate feature distillation and predictive distillation to ensure a smooth and stable feature transfer process; after all target layers are unfrozen, the student model and feature alignment system are jointly optimized, and the weights of distillation loss and task loss are dynamically adjusted to balance knowledge inheritance and task performance improvement.

[0009] In a first aspect, the present invention proposes a feature alignment system for large-small models suitable for object detection, comprising a feature alignment module, which includes: a spatial alignment submodule configured to adjust the spatial size of the image feature map of the teacher model to be consistent with the spatial size of the image feature map of the student model, and output a first image feature map with the same number of channels as the teacher model and a spatial size matching the student model; and a channel alignment submodule configured to perform channel compression and semantic remapping on the first image feature map to generate a second image feature map with the same number of channels as the student model; the spatial alignment submodule and the channel alignment submodule are connected in series in the order of spatial alignment first and channel alignment later, so that spatial structure adjustment and channel semantic mapping are executed independently at different stages.

[0010] In some examples, the spatial alignment submodule selects a downsampling path or an upsampling path for scaling based on a resolution comparison between the image feature maps of the teacher model and the student model, and performs structural consistency reshaping on the first image feature map after scaling by a spatial structure preservation layer. The structural consistency reshaping includes intra-channel spatial structure modeling and inter-channel structural consistency planning.

[0011] In some examples, when the resolution of the image feature map of the teacher model is higher than that of the image feature map of the student model, the downsampling path extracts local aggregation information and contextual information through parallel strided convolution and dilated convolution, respectively, and adds the image feature map output by the strided convolution to the image feature map output by the dilated convolution element by element; when the resolution of the image feature map of the teacher model is lower than that of the image feature map of the student model, the upsampling path performs bilinear interpolation on the image feature map of the teacher model and then uses learnable convolution for local reconstruction.

[0012] In some examples, the channel alignment submodule includes: a channel compression layer configured to map the number of channels of the first image feature map to the number of channels of the image feature map of the student model through a 1×1 convolution to obtain compressed features; a channel semantic remapping layer configured to perform global average pooling on the compressed features and then generate channel semantic descriptions through a multilayer perceptron; and a dynamic channel weighting layer configured to map the channel semantic descriptions to weight coefficients using an activation function and multiply the weight coefficients by the compressed features channel by channel to obtain the second image feature map.

[0013] In some examples, the KL divergence between the channel response distributions of the teacher model's image feature map and the student model's image feature map after channel compression weighting is calculated using the channel alignment loss function to constrain channel semantic alignment.

[0014] Secondly, this invention proposes a large-small model progressive layer decoupling distillation training method, which deploys the feature alignment system between the teacher model and the student model. The method includes: an initial stage, in which target intermediate layers of the student model are selected, the parameters of the teacher model are kept frozen, the parameters of the student model backbone network are kept frozen, and only the parameters of the feature alignment module and the parameters of the task output layer of the student model are trained according to the task loss of the student model itself, and a feature alignment benchmark is established through feature distillation loss; a progressive unfreezing stage, under the feature alignment benchmark, the parameters of the selected target intermediate layers are gradually unfrozen in order from high to low layers, and after each unfreezing, the parameters of the currently unfrozen target intermediate layer, the parameters of the feature alignment module, and the parameters of the task output layer are updated, and a multi-level distillation loss is applied; a joint optimization stage, after the parameters of all target intermediate layers are unfrozen, all parameters of the student model and the parameters of the feature alignment module are jointly trained, and the weights of the task loss, the feature distillation loss, and the multi-level distillation loss are dynamically adjusted, so that the weights of the feature distillation loss and the multi-level distillation loss gradually decrease, and the weight of the task loss gradually increases.

[0015] In some examples, the target intermediate layer of the student model is determined based on the semantic level of features, the matching relationship between the feature map scale and the receptive field, and the contribution to the task loss. Among them, the middle and high-level feature layers of the student model are used as candidates, and the semantic level correspondence is confirmed by comparing the downsampling ratio of the corresponding layers of the teacher model and the student model. Then, redundant layers whose contribution to the task loss is lower than the preset condition are eliminated.

[0016] In some examples, during the progressive unfreezing phase, the multi-level distillation loss includes a hierarchical distillation loss for the intermediate feature layer and a predictive distillation loss for the task output layer; the hierarchical distillation loss is used to constrain the image features of the target intermediate layer of the currently unfrozen model to be consistent with the image features of the corresponding layer of the teacher model, and the predictive distillation loss is used to constrain the task prediction output of the student model to be consistent with the task prediction output of the teacher model.

[0017] In some examples, during the joint optimization phase, the weights of the task loss and distillation loss are dynamically adjusted as follows:

[0018]

[0019] In the formula, As a weight for distillation losses, the distillation losses include the characteristic distillation losses and the multi-level distillation losses. As the initial distillation weight, For task loss weight, This represents the current number of training steps. This represents the total number of training steps.

[0020] Thirdly, the present invention proposes a computer system including a processor and a memory, wherein the memory stores program instructions, and the processor executes the program instructions to implement the progressive layer decoupled distillation training method.

[0021] The technical solution proposed in this invention has the following advantages: (1) The proposed feature alignment system adopts a spatial and channel decoupling design, which can adapt to different network structures, different level features and various teacher-student combinations, and has good versatility and scalability. (2) By introducing lightweight convolution, 1×1 linear mapping and small-scale MLP structure, and combining it with a progressive unfreezing training strategy, the parameter scale and computational cost can be effectively controlled while ensuring the alignment effect. (3) Fine alignment is performed in both spatial resolution and channel semantic dimensions to guide the student model to accurately learn the key structure and high-level semantic information of the teacher model, which significantly improves the feature mapping accuracy. (4) A progressive layer decoupling distillation training method is adopted to gradually release the trainable layers of the network and introduce distillation constraints, which effectively alleviates gradient conflict and training oscillation, and improves the stability and convergence of the overall training process. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a feature alignment module for a large-small model according to an embodiment of the present invention.

[0023] Figure 2 This is a flowchart of a progressive training method for a large-small model according to an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram comparing the application effects of a feature alignment module according to an embodiment of the present invention under different backbone networks. Detailed Implementation

[0025] This invention focuses on the fields of deep learning and computer vision, aiming to achieve refined feature alignment between large and small models while ensuring computational efficiency and structural flexibility. It is mainly applicable to object detection tasks.

[0026] To address the challenges of feature alignment, training instability, and excessive computational overhead caused by differences in spatial resolution, channel semantics, and network structure between heterogeneous large and small models, this invention proposes: a feature alignment system for heterogeneous models; and a progressive layer decoupling distillation training method.

[0027] Example 1: Feature alignment system for heterogeneous models (large-small models).

[0028] like Figure 1As shown, in this embodiment, the feature alignment system is deployed between the corresponding intermediate layers of the teacher model and the student model to achieve fine-grained alignment of cross-model features without changing the original backbone network structure. The system includes a hierarchical feature alignment module (hereinafter referred to as the feature alignment module). The feature alignment module adopts a hierarchical decoupled design, comprising a spatial alignment submodule and a channel alignment submodule, which operate on different dimensions of the feature mapping respectively, and are combined through a unified interface, making them functionally independent but cooperative during training. This decoupled design avoids the problems of severe parameter coupling and unstable training in the traditional "spatial-channel hybrid mapping" structure, effectively reducing the structural complexity of the alignment module.

[0029] 1.1 Decoupling Design of Hierarchical Feature Alignment Module To address the simultaneous spatial resolution and channel semantic differences between teacher and student models, this invention proposes a hierarchical feature alignment module, employing a decoupled architecture design that prioritizes spatial alignment over channel alignment. This design achieves stable and efficient feature mapping by splitting the feature alignment task into two independent but sequentially collaborative sub-processes.

[0030] In deep vision models, feature tensors are typically represented as:

[0031] in For batch size, For the number of channels, The feature map space size, For the real number field. The main difference between the teacher model and the student model lies in the difference in spatial dimension ( or ) and channel dimension differences ( ).

[0032] If an attempt is made to perform both spatial and channel transformations simultaneously using a single module, the network would need to learn both spatial structure adjustment and semantic channel reorganization objectives in the same computational process, which would significantly increase the optimization difficulty. Therefore, this invention decouples the feature alignment problem into two independent stages through task decoupling.

[0033] (1) Decoupling architecture goal Spatial alignment and channel alignment are essentially two different types of feature transformation tasks.

[0034] The goal of spatial alignment is to adjust the spatial size of the feature map while keeping the semantic structure relationship unchanged. Its focus is on the target boundary, geometry and local structural relationship.

[0035] Channel alignment focuses on adjusting the semantic distribution of channels while maintaining the stability of the spatial structure. Its core task is to reorganize the semantic expressions between different channels so that the student model can reconstruct the semantic information of the teacher model with a smaller number of channels.

[0036] If two types of transformations are coupled in the same module, such as using a single convolutional structure to adjust the spatial size and the number of channels at the same time, the convolutional kernel needs to learn the following two types of functions simultaneously: how to perform upsampling or downsampling in the spatial dimension; and how to perform semantic filtering and recombination in the channel dimension.

[0037] Because these two types of objectives are not aligned in their optimization directions, the model parameters are simultaneously affected by the gradients of spatial and semantic errors during training, leading to mixed gradient signals. The final results may manifest as follows: spatial transformation disrupts channel semantic priority; channel compression loses key structural information; and a single module needs to fit two objective functions simultaneously, resulting in slower training convergence.

[0038] Therefore, separating spatial alignment from channel alignment allows each module to focus on a single optimization objective, thereby reducing learning difficulty and improving alignment accuracy.

[0039] (2) Detailed design of decoupled architecture The decoupling in this invention refers not only to the modular decomposition at the structural level, but also to the functional decoupling, parameter decoupling, and computational graph decoupling.

[0040] First, at the functional level, the spatial alignment submodule is only responsible for adjusting the spatial size of the feature map so that the teacher features are consistent with the student features in the spatial dimension, without changing the number of channels; the channel alignment submodule, on the other hand, maps and reorganizes the channel semantics while keeping the spatial size unchanged.

[0041] Secondly, at the parameter level, the two sub-modules have independent parameter sets. The parameters of the spatial alignment sub-module are mainly used to learn spatial structure transformations, while the parameters of the channel alignment sub-module are specifically used to learn channel semantic relationships. This parameter independence avoids parameter competition between different tasks, thereby improving training stability.

[0042] Furthermore, at the computational graph level, the two sub-modules are connected in series in a fixed order within the network structure, forming a clear data flow path. The output of the spatial alignment sub-module directly serves as the input of the channel alignment sub-module, resulting in a clear stage division in the feature transformation process.

[0043] This multi-level decoupling method allows spatial structure adjustment and semantic channel mapping to be completed in the most suitable feature space, thereby improving the overall feature alignment effect.

[0044] (3) Module data flow and interface relationship in decoupled architecture In the specific implementation process, the teacher model is in the first... Layer output image features for:

[0045] Where B represents the batch size. The teacher model is represented in the first... l The number of channels in the layer and These represent the teacher model in the first... l The height and width of the layer feature map.

[0046] The spatial alignment submodule first adjusts the resolution of the image features to ensure their spatial dimensions match those of the corresponding layer in the student model. After spatial alignment, the output features are... for:

[0047] in, and These represent the height and width of the feature map for the corresponding layer in the student model, respectively. It can be seen that only the spatial dimensions are adjusted in this stage, while the number of channels... It remains unchanged.

[0048] The image features are then used as input to the channel alignment submodule. During the channel alignment stage, the channel features of the teacher model are projected into the channel space of the student model through channel compression and semantic remapping, ultimately resulting in:

[0049] in, This represents the number of channels in the corresponding layer of the student model. This output feature... Both the spatial dimensions and channel dimensions are consistent with the student model features, thus enabling them to directly participate in subsequent feature distillation or loss calculation processes.

[0050] Through this explicit data interface design, a stable data transmission relationship is formed between the spatial alignment submodule and the channel alignment submodule, enabling the entire hierarchical feature alignment module to be embedded as a unified structure in different network architectures.

[0051] (4) Design Mechanism of "Space First, Passage Later" In the module design, this invention adopts a sequence of spatial alignment first, followed by channel alignment. This sequence is not arbitrarily chosen, but rather designed based on the stability of feature representation.

[0052] If channel alignment is performed first, the high-dimensional channel features of the teacher model will be compressed to a lower dimension. Inevitably, some fine-grained semantic information will be lost in this process. If spatial interpolation or resampling is then performed on the compressed low-dimensional features, these operations will be performed on features that have already lost semantic details, which may amplify noise or erroneous responses.

[0053] For example, when some channels in a low-dimensional feature contain mixed semantics, spatial interpolation will spread this mixed semantics to a larger area, causing the spatial structure to become blurred or distorted. This not only reduces the accuracy of spatial alignment but may also introduce large gradient fluctuations during training.

[0054] Conversely, if spatial alignment is performed first, spatial scaling can be completed within the original high-dimensional semantic space of the teacher model. Since the features still retain complete channel semantic representation, spatial resampling can more accurately maintain the geometric stability of the target bounding box and the relative positional relationships between key feature points. Subsequent channel compression and semantic mapping can then complete semantic reorganization on the basis of the already stable spatial structure, thereby reducing the impact of information loss.

[0055] Therefore, the "space first, channel second" processing order can effectively avoid the mutual interference between spatial structure reconstruction and semantic compression, making the feature alignment process more stable and improving the convergence of the training process.

[0056] Therefore, this embodiment proposes a large-small model hierarchical feature alignment module for object detection. This module includes a cascaded spatial alignment submodule and a channel alignment submodule. The spatial alignment submodule first adjusts the spatial size of the teacher model's image feature map to match that of the student model, outputting a first image feature map that maintains the teacher's channel count. Subsequently, the channel alignment submodule performs channel compression and semantic remapping on this first image feature map, generating a second image feature map with the same number of channels as the student model. This module strictly follows the processing order of spatial alignment first, then channel alignment, ensuring that spatial structure adjustment and channel semantic mapping are performed independently at different stages.

[0057] 1.2 Spatial Alignment Submodule (Spatial Resolution Alignment Submodule) The spatial resolution alignment submodule automatically selects the corresponding scaling path based on the spatial size relationship between the teacher model image feature map and the student model image feature map, in order to adjust the spatial size of the teacher image feature map to match that of the student model image feature map. Figure 1Specifically, the spatial alignment submodule compares the resolutions of the image feature maps of the teacher model and the student model, selects a downsampling or upsampling path for scaling, and then performs structural consistency reshaping on the first image feature map through a spatial structure preservation layer after scaling. This structural consistency reshaping includes intra-channel spatial structure modeling and inter-channel structural consistency planning. When the resolution of the teacher model's image feature map is higher than that of the student model's image feature map, the downsampling path extracts local aggregation information and contextual information through parallel strided convolutions and dilated convolutions, respectively, and adds the image feature map output by the strided convolution to the image feature map output by the dilated convolution element-wise. When the resolution of the teacher model's image feature map is lower than that of the student model's image feature map, the upsampling path performs bilinear interpolation on the teacher model's image feature map and then uses learnable convolutions for local reconstruction.

[0058] Let the teacher model be in the first place. Layer output image features for:

[0059] Student model corresponding layer output image features for:

[0060] when or At that time, there is an issue of inconsistent spatial resolution. This module first automatically determines this through size comparison: or

[0061] If the condition is met, the downsampling path is executed; otherwise, the upsampling path is executed.

[0062] Output of the spatial alignment submodule for:

[0063] That is, the output image feature map is consistent with the student model feature in terms of spatial size, while the number of channels remains unchanged.

[0064] (1) Spatial compression of teacher features in high-resolution images ( ) When the feature resolution of the teacher model image is higher than that of the student model image ( When performing scale matching, spatial compression of teacher image features is required. In this case, a learnable spatial compression strategy is employed, using local information aggregation to achieve effective downsampling of high-resolution features.

[0065] Specifically, the image feature map is first downsampled using a convolution with stride. For example, the kernel size is... The step size is set according to the resolution ratio of the image feature map of the corresponding layer in the student model, and the padding method is "same". This operation can reduce the spatial resolution while performing weighted aggregation of local neighborhood features.

[0066] However, simple stride convolution shrinks the receptive field of each feature point when the resolution is reduced, resulting in the loss of some contextual information. To alleviate this problem, this invention introduces an expanded convolution path during the downsampling process. By setting an interval sampling method inside the convolution kernel, the effective receptive field is expanded, allowing feature points to cover a larger range of contextual regions.

[0067] Structurally, the strided convolutional path and the dilated convolutional path are executed in parallel. They extract local aggregation information and large-scale contextual information, respectively. Then, the outputs of the two paths are added element by element to form a compressed feature representation that takes into account both local details and global structural information.

[0068] (2) Spatial reconstruction of features in low-resolution teacher images ) When the feature resolution of the teacher model image is lower than that of the student model image ( When reconstructing images, it is necessary to use spatial reconstruction strategies to restore the fine-grained structure of image features.

[0069] In this case, bilinear interpolation is first used to spatially scale the image feature map to obtain a model that matches the student image features. Figure 1 The required space dimensions. Then use... Convolution reconstructs the interpolated image features locally to supplement details that may have been lost during the interpolation process.

[0070] Compared with traditional deconvolution methods, this combination of "interpolation + convolution" can effectively avoid the chessboard artifact problem commonly encountered in deconvolution, while maintaining low computational complexity.

[0071] Furthermore, this convolution operator has learnable parameters during training, thus enabling it to adaptively adjust the spatial structure reconstruction method according to task requirements, thereby preserving key structural information as much as possible while amplifying the feature map.

[0072] (3) Spatial structure preservation layer To avoid damaging the geometry of the object being detected during scale transformation, this invention introduces a Spatial Structure Preservation Layer after the scale transformation path.

[0073] It should be noted that this spatial structure preservation layer is a universal component. Regardless of whether an upsampling or downsampling path is used, structural correction must be performed through this layer after spatial scale adjustment. Therefore, this layer is connected in series with the aforementioned scale transformation path, and its position is after all scale transformation operations.

[0074] The design goal of the spatial structure preservation layer is to further reorganize the feature map to ensure structural consistency after spatial size matching, so that the adjusted teacher image features can more closely resemble the expression of student image features in terms of spatial distribution pattern.

[0075] In its specific implementation, the spatial structure preservation layer achieves feature regularization through two stages: The first stage (intra-channel spatial structure modeling): Local neighborhood feature aggregation is performed within each channel, enabling the network to perceive target boundaries, texture changes, and local geometric structures, and to repair spatial relationship disturbances caused by scale transformation.

[0076] The second stage (inter-channel structural consistency planning) involves linearly combining identical spatial locations across different channels, for example, through weighted 1×1 convolutions. Specifically, for each spatial location (h, w), a shared C×C weight matrix is ​​used to weight and sum the responses from different channels, generating new channel features. This operation does not change the channel dimensions; it only enhances the spatial consistency of the target contour and local structure through collaborative adjustment of cross-channel responses.

[0077] This phased processing approach has two important functions: First, by prioritizing the modeling of spatial neighborhood relationships, the continuity of the target contour, edges, and local structure can be maintained, thereby avoiding structural distortion or information breakage during scale transformation. Second, through subsequent collaborative planning of inter-channel responses (channel information fusion), the semantic expression between different channels can be reorganized, enabling structural information to form a consistent response pattern in the multi-channel feature space.

[0078] During training, the spatial structure preservation layer also constrains the response distribution of features before and after alignment at corresponding spatial locations, so that the scale-adjusted teacher image features can gradually approach the response pattern of student features in the spatial dimension, thereby achieving a more stable and refined spatial mapping relationship between features of different resolutions.

[0079] 1.3 Channel Alignment Submodule To address the differences between the teacher model and the student model in terms of the number of channels and the semantic distribution of channels, this invention proposes a learnable channel alignment submodule to enable channel-level knowledge transfer between the teacher model and the student model.

[0080] In deep neural networks, each channel typically corresponds to a specific semantic response pattern, such as texture structure, edge information, or local features of a target. However, because teacher models usually have deeper or wider network structures, their number of channels and semantic expressive power are often significantly higher than those of student models. Therefore, multiple channels in a teacher model may jointly express a complex semantic meaning, while a student model needs fewer channels to achieve the same semantic expression. This leads to a mismatch between the two models in terms of both the number of channels and the semantic distribution of channels.

[0081] Direct feature distillation or simple channel pruning can easily lead to two problems: semantic information with discriminative capabilities in the teacher model is randomly discarded during compression; and channels with different semantics are mixed during mapping, causing semantic aliasing, which reduces the distillation effect.

[0082] To address the aforementioned issues, this invention constructs a learnable channel mapping mechanism, enabling teacher model channel features to form stable and consistent semantic representations within the student model channel space. This channel alignment submodule works collaboratively through three steps: channel compression, semantic remapping, and dynamic channel weighting, to achieve effective transfer of teacher semantic information to the student model.

[0083] (1) Channel compression layer In practical applications, the number of channels in the teacher model Typically much larger than the number of channels in the student model. To establish the mapping relationship between the two, it is first necessary to project the high-dimensional channel features of the teacher model onto the corresponding channel space of the student model.

[0084] To address this, the present invention introduces a channel compression layer to perform dimensional mapping on the teacher model image features. This layer... Convolution achieves a linear combination of channel dimensions, making the input features:

[0085] Mapped to spatial dimensional information:

[0086] or intermediate dimension .

[0087] From a mathematical perspective, this process is equivalent to applying a linear transformation matrix to the channel dimension, generating new feature representations through weighted combinations of different channels. Compared to directly mapping spatial features using fully connected layers, this convolutional implementation only transforms the channel dimension without destroying the original spatial structure, thus significantly reducing parameter size and computational complexity while maintaining the integrity of spatial information.

[0088] More importantly, this mapping process does not simply delete redundant channels, but learns the combination relationships between different channels, enabling each channel of the student model to integrate relevant semantic information from multiple teacher channels, thereby preserving the semantic expressive power of the teacher model as much as possible at a lower channel dimension.

[0089] (2) Channel semantic remapping layer Although channel compression layers can perform dimension mapping, this process is essentially still a linear combination. If only linear compression is relied upon, channels with different semantics may be mixed with equal weight, making it difficult for student models to distinguish which channels contain more important discriminative information.

[0090] To further model the semantic relationships between channels, this invention introduces a channel semantic remapping mechanism on the compressed features, and dynamically evaluates the importance of channels by analyzing the overall response strength of each channel.

[0091] First, global average pooling is performed on the compressed feature map to extract spatial dimensional information. Compression into channel-level statistical description:

[0092] In the formula, This is a channel semantic description vector.

[0093] This operation transforms the response intensity of each channel across the entire spatial range into a scalar, thus forming a channel-level semantic representation. This vector reflects the overall response of different channels to the current input sample.

[0094] Subsequently, a nonlinear mapping is performed on the channel description vector using a multilayer perceptron (MLP):

[0095] in This represents the channel compression ratio. This represents the number of channels. This mapping process learns the semantic dependencies between different channels, enabling the model to identify which channels have complementary or redundant relationships.

[0096] Through the above steps, the model can establish the association structure between channel semantics in the low-dimensional channel space, thereby providing a basis for subsequent channel importance allocation.

[0097] It should be noted that the semantic remapping operation is performed after channel compression. If semantic modeling is performed directly in the high-dimensional channel space of the teacher model, the parameter size of the MLP will increase significantly with the number of channels, thereby increasing the computational cost. Therefore, this invention effectively controls the computational complexity while ensuring semantic expressiveness by compressing first and then modeling.

[0098] (3) Dynamic channel weighting mechanism After obtaining the channel semantic description, the present invention further realizes the selective transfer of teacher semantics to the student model through a dynamic channel weighting mechanism.

[0099] Specifically, the channel semantic description vector is mapped to an activation function. Weighting coefficients between :

[0100] In the formula, This is a channel semantic description vector; For activation functions, the Sigmoid function is typically used. For multilayer perceptrons, the semantic description vector of the channels is... Perform a nonlinear transformation and output a vector with the same dimension as the number of channels.

[0101] This weight is then applied to the channel image features:

[0102] in, This indicates a channel-by-channel multiplication operation. The input feature map (i.e., the image features after channel compression). This is the weighted feature map (i.e., the output of the dynamic weighting mechanism).

[0103] The core idea of ​​this process is not to choose "whether to use a certain channel", but to assign different importance weights to each channel, so that the network can dynamically adjust the channel contribution based on the input samples and task requirements.

[0104] In this way, the student model can preferentially inherit the channel features with higher response intensity and richer semantic information from the teacher model, thereby assigning lower weights to redundant or low-contribution channels and avoiding the loss of key semantic information during compression.

[0105] Furthermore, the weight allocation is dynamic: different input samples will generate different channel weight distributions, so the model can adaptively adjust the knowledge transfer strategy according to the specific scenario. Compared with fixed weight or static channel pruning methods, this mechanism can more finely control the process of semantic information transmission.

[0106] (4) Channel alignment loss After completing channel mapping and dynamic weighting, in order to further constrain the feature representation of the student model, this invention introduces channel-level alignment loss to constrain the channel response distribution between the teacher model and the student model.

[0107] Specifically, the channel response distributions of the teacher model and the student model are normalized to probability distributions, and the KL divergence (Kullback–Leibler Divergence) is used as a metric to minimize the distribution difference between the two.

[0108] This loss function can statistically constrain the overall distribution of the channel features of the student model, allowing it to gradually approach the semantic distribution structure of the teacher model while maintaining its expressive power.

[0109] In this way, the student model can be effectively prevented from learning only strong response patterns of a few local channels while ignoring other channel features with discriminative significance in the teacher model, thereby alleviating the semantic shift problem caused by channel mismatch.

[0110] Example 2: Progressive layer decoupling distillation training method.

[0111] To address the challenge of achieving stable and refined feature alignment between large and small models while maintaining computational efficiency and structural flexibility, this invention further proposes a progressive training method that matches the hierarchical feature alignment module. This method achieves efficient and stable cross-model knowledge transfer through staged training, layer-by-layer unfreezing, and multi-level distillation constraints.

[0112] like Figure 2 As shown, the method includes: an initial stage, selecting target intermediate layers of the student model, keeping the teacher model parameters frozen, keeping the student model backbone network parameters frozen, training only the parameters of the feature alignment module and the parameters of the student model's task output layer based on the student model's own task loss, and establishing a feature alignment benchmark through feature distillation loss; a progressive unfreezing stage, under the feature alignment benchmark, gradually unfreezing the parameters of the selected target intermediate layers in order from high to low layers, updating the parameters of the currently unfrozen target intermediate layer, the parameters of the feature alignment module, and the parameters of the task output layer after each unfreezing, and applying multi-level distillation loss; a joint optimization stage, after all the parameters of the target intermediate layers are unfrozen, jointly training all parameters of the student model and the parameters of the feature alignment module, and dynamically adjusting the weights of the task loss, the feature distillation loss, and the multi-level distillation loss, so that the weights of the feature distillation loss and the multi-level distillation loss gradually decrease, and the weight of the task loss gradually increases.

[0113] Step 1, Initialization and target layer (also known as: target intermediate layer) filtering steps.

[0114] This invention selects a set of the most representative hidden layers from the backbone network of the student model as the target intermediate layers.

[0115] In the initial training phase, the teacher model, student model, and the feature alignment module are constructed first. Specifically: the teacher model parameters are kept frozen; in the student model, except for the classification head or task output layer, the parameters of the remaining backbone network are frozen; and the parameters of the hierarchical feature alignment module (including the spatial alignment submodule and the channel alignment submodule) are set to be trainable.

[0116] After model initialization, target layer selection is required for the student model to determine the set of feature layers involved in the subsequent progressive unfreezing phase. The target layer refers to the intermediate feature extraction layer in the student model selected for feature alignment with the teacher model. This process is designed to avoid updating a large number of network parameters simultaneously in the early stages of training, thereby reducing the risk of gradient instability and enabling the feature alignment process to establish stable mapping relationships layer by layer.

[0117] (1) Target layer screening principles The selection of the target layer is based on three objective criteria: the semantic level of the features, the matching relationship between the feature scale and the receptive field, and the information contribution to the task.

[0118] (1.1) Semantic hierarchy of features In deep visual networks, features at different levels have different levels of semantic abstraction: low-level feature layers mainly encode local visual information such as edges and textures; mid-level feature layers begin to express the local structure and combination patterns of the target; and high-level feature layers mainly contain semantic information related to the target category and global context information.

[0119] In object detection tasks, mid-to-high-level features typically play a major semantic representation role and are therefore prioritized as candidate layers for distillation and feature alignment. In the network structure, the backbone network is often composed of stacked Block units with identical structures, and image data passes through the "low-mid-high" layers sequentially. In this invention, mid-to-high-level feature layers are defined as feature layers whose network depth is located in the latter half of the overall structure and whose receptive field can cover the main structural regions of the target. For example, in common convolutional or Transformer visual backbone networks, mid-to-high levels typically correspond to the output feature layers of the last 2 to 4 stages.

[0120] (1.2) Relationship between feature scale and receptive field The semantic hierarchical correspondence between the teacher model and the student model is mainly determined by the feature map scale and the receptive field size.

[0121] Let the teacher model be... The layer feature map size is:

[0122] In the formula, For the teacher model Image feature tensor of the layer; For the real number field; B This refers to the batch size, which is the number of samples input in one training session. For the teacher model The number of channels in the layer feature map; , For the teacher model The height and width of the layer feature map.

[0123] Student Model No. The layer feature map size is:

[0124] In the formula, For the student model The image feature tensor of the layer, For the student model The number of channels in the layer feature map; , For the student model The height and width of the layer feature map.

[0125] If the two feature maps satisfy the following relationship in terms of spatial scale:

[0126] If two layers have the same order of magnitude of downsampling ratio (e.g., 1 / 8, 1 / 16, 1 / 32), they are considered to have similar receptive field coverage and can therefore be regarded as semantically corresponding layers. This represents the original height of the input image. The formula means: Teacher Model... i Layer feature map height With input image height The ratio is approximately equal to the student model's first... j Layer feature map height With input image height The ratio of the two downsampling ratios relative to the original input indicates a correspondence between the semantic level and receptive field of the two layers in their respective networks. This scale matching relationship ensures that the distilled feature layers are comparable in spatial semantic representation, thereby avoiding misalignment between different semantic levels.

[0127] (1.3) Contribution of task information To further filter the layers that have a greater impact on task performance, this invention evaluates the degree of influence of different feature layers on the gradient contribution or feature response intensity of the detection head output.

[0128] Specifically, it can be measured by the following metrics: the average gradient magnitude of the features of this layer during the backpropagation of the task loss; the frequency of use of the features of this layer in the detection head or the feature fusion weight; and the average response intensity of the target region in the feature map.

[0129] Feature layers with higher gradient contribution or response intensity indicate that they play a more important role in target recognition and localization tasks, and therefore should be included in the target layer set first.

[0130] (2) Target layer screening process Based on the above principles, this invention adopts the following screening process: First, determine the set of mid-to-high-level feature layers of the student model (e.g., the output layers of the last few stages); second, further screen the set based on the contribution of task information, eliminating redundant layers with low contribution to the final task output and retaining the core feature layers; then, based on the matching relationship between feature map scale and receptive field, find feature layers in the teacher model that correspond to the core layers of the remaining student model at the semantic level; finally, establish a set of semantically corresponding and highly contributing teacher-student feature layer pairings, which are used as the target layer set for subsequent feature distillation and progressive unfreezing training.

[0131] Through the above screening, a set of stable feature layer pairs with semantic correspondence can be obtained, providing a reliable foundation for subsequent feature alignment and knowledge transfer.

[0132] Step 2: Stable training phase led by the feature alignment module After the target layer selection is completed, the first training phase begins. The core objective of this phase is to establish a stable teacher-student feature alignment benchmark without changing the backbone network parameters of the student model. To achieve this objective, this phase employs a hierarchical feature alignment module-led training strategy.

[0133] In this phase: the teacher model parameters remain frozen; the student model backbone network parameters remain frozen; only the parameters of the hierarchical feature alignment module and the student model task output layer parameters are updated. By freezing the student backbone network, it is ensured that the student feature representation remains stable in the early stages of training, thus avoiding drastic changes in network parameters before feature alignment is established. Meanwhile, the feature alignment module, as the only structure responsible for cross-model mapping, can focus on learning the spatial structure mapping relationship and channel semantic mapping relationship between the teacher model and the student model.

[0134] (1) Characteristic distillation loss In this stage, feature distillation loss is introduced to constrain the consistency between teacher and student features. The distillation loss includes a feature reconstruction loss based on KL divergence and a distribution alignment loss based on channel statistics.

[0135] KL divergence-based feature reconstruction loss minimizes the difference between teacher and student features in the probability distribution space, making the aligned teacher features close to student features in terms of spatial structure and overall response distribution, thereby ensuring that spatial structure information can be correctly transmitted.

[0136] Distribution alignment loss based on channel statistics aligns the statistical characteristics (mean) of teacher features and student features in the channel dimension, enabling the student model to gradually learn the overall distribution pattern of the channel semantics of the teacher model.

[0137] (2) Formation mechanism of stable alignment reference The two types of losses mentioned above are complementary in this stage: the feature reconstruction loss mainly constrains the spatial structure and the overall response distribution; while the statistical distribution alignment loss mainly constrains the channel semantic expression.

[0138] Since the parameters of the student backbone network remain frozen, the gradient generated by the distillation loss primarily acts on the feature alignment module, enabling it to gradually learn the cross-structural mapping relationship between the teacher and student models. As training progresses, the feature alignment module will gradually establish stable spatial mapping functions and channel semantic mapping functions, thus forming a reliable feature transformation interface that allows teacher features to be mapped into the student feature space.

[0139] This stage is equivalent to establishing a stable feature alignment benchmark for subsequent training: that is, while keeping the student network structure unchanged, the mapping learning across the model feature space is completed first. Once this benchmark is established, when the student model parameters are gradually unfrozen, the network can update the parameters based on the already stable feature alignment relationship, thereby significantly reducing the risk of gradient conflicts and feature space collapse.

[0140] Step 3, the progressive feature transfer training phase of gradually unfreezing.

[0141] After establishing a stable alignment benchmark, the progressive feature transfer training phase begins. This phase employs a gradual unfreezing strategy, proceeding from high-level to low-level layers, to release the target layers selected in step 1 in stages, thereby achieving a smooth transition in the expressive power of the student model.

[0142] Specifically, in each round of thawing, only one set of target layers is thawed, while the remaining layers remain frozen. Simultaneously, the following parameters are updated based on gradient backpropagation: the student model parameters of the currently thawed layer, the parameters of the hierarchical feature alignment module, and the parameters of the student model task output layer.

[0143] (1) Gradual thawing sequence from high-rise to low-rise Compared to training by unfreezing the entire network all at once, gradual unfreezing can effectively avoid the following problems: gradient conflicts caused by simultaneous updates of large-scale parameters; training oscillations caused by abrupt changes in feature distribution; and feature space collapse that occurs before a stable mapping relationship is established.

[0144] The core design mechanism of the "from high to low" unfreezing order is that high-level features are closer to the task semantics and have less disturbance to the overall network, and can achieve stable optimization on the established feature alignment benchmark; while low-level features have a greater impact on the overall distribution. If they are unfrozen too early, they are prone to gradient conflicts and feature space drift. Therefore, they need to be released gradually after the high-level semantics are stable, so as to ensure the continuity of the feature transfer process and the stability of training.

[0145] (2) Coordination of parameter updates The three components—the target layer, the alignment module, and the task output layer—are updated collaboratively within a unified optimization framework. Specifically, the target layer is responsible for gradually absorbing the semantic information from the teacher model, the alignment module continuously corrects the spatial and channel mapping relationship, and the task output layer provides stable supervision signals. This forms a dynamic and collaborative optimization mechanism that ensures the continuity and stability of the feature space during the transfer process.

[0146] (3) Multilevel distillation loss function To ensure that the network does not lose the spatial and semantic alignment knowledge learned by the "hierarchical feature alignment module" in the "stabilized training" stage of step 2 during the "gradual unfreezing" training process, this stage introduces multi-level distillation loss. Taking the target layer selected in step 1 as the constraint object, it strengthens the multi-scale constraints of the intermediate and output layers of the network, thereby achieving a smooth transition from "module alignment" to "full network alignment".

[0147] The multi-level distillation loss includes the hierarchical distillation loss of the intermediate feature layer and the predictive distillation loss of the task output layer.

[0148] The hierarchical distillation loss of the intermediate feature layer constrains the image features of the target layer that is currently being thawed layer by layer, ensuring that they are consistent with the corresponding layer of the teacher model and preventing local semantic shifts.

[0149] The predictive distillation loss of the task output layer imposes a global constraint on the output space, ensuring that the task performance does not degrade during the gradual unfreezing process.

[0150] The multi-level distillation loss function in this stage forms a progressive relationship with the feature distillation loss in the first stage: the feature distillation loss in the first stage is mainly used to establish a stable feature alignment benchmark between the teacher model and the student model under the condition of freezing the student backbone network, focusing on the consistency constraints of the overall feature distribution and spatial structure; the multi-level distillation loss introduced in the second stage further applies hierarchical feature constraints and output layer constraints on top of this alignment benchmark, achieving fine-grained alignment from local to global. The two form a progressive relationship from "coarse-grained alignment" to "fine-grained constraints" during the optimization process, jointly ensuring the stability and effectiveness of feature transfer.

[0151] Step 4, Joint Optimization and Weight Adjustment Stage After all target layers have been unfrozen, the training process enters the joint optimization phase. The core mechanism of this phase is as follows: after the gradual unfreezing and alignment benchmark establishment in the first three phases, through full-parameter joint optimization and dynamic adjustment of distillation-task loss weights, knowledge transfer is achieved from "teacher knowledge-driven" to "task performance optimization-driven". This ensures that the student model fully retains the high-level semantic and structural knowledge of the teacher model while being fully adapted to the specific target detection task, thereby achieving an independent and efficient optimal performance state.

[0152] During this phase, all parameters of the student model and the parameters of the hierarchical feature alignment module are used in the training.

[0153] Simultaneously, the weights of the total loss function, which includes distillation loss (feature distillation loss and multi-level distillation loss) and the student model's own task loss, are dynamically adjusted, gradually decreasing the weight of distillation loss and gradually increasing the weight of task loss. This adjustment is based on the fact that stable knowledge transfer has been achieved in the earlier stages, and this stage requires gradually weakening the dependence on teacher features to prevent the student model from overfitting the teacher distribution and losing its targeted generalization ability for specific tasks. At the same time, strengthening the task loss further improves detection accuracy.

[0154] Ultimately, the student model achieves near-teacher-level performance while maintaining a low parameter scale and computational overhead, and possesses higher inference efficiency and deployment flexibility, making it suitable for edge devices and real-time application scenarios.

[0155] The total loss function is expressed as: ,in: For distillation loss weighting, Distillation losses (characteristic distillation losses and multi-level distillation losses); For task loss weight, This refers to task loss (such as classification loss or regression loss in object detection).

[0156] A linear decay strategy is adopted to gradually reduce the distillation weight from an initial high value to a lower level, while the task loss weight is correspondingly increased to become dominant, as shown in the following formula:

[0157]

[0158] in: The initial distillation loss weight (i.e., the distillation loss weight value at t=0) is a preset hyperparameter, which is usually set to a high value (such as 0.9 or 1.0) at the beginning of training in order to strengthen the transfer of teacher knowledge to student model in the early stage; This represents the current number of training steps. This represents the total number of training steps.

[0159] This adjustment mechanism can enhance the transfer of teacher knowledge to student models in the early stages of training and prevent student models from becoming overly dependent on teacher distribution in the later stages of training, thereby improving their adaptability and generalization performance for specific tasks.

[0160] Figure 3 This diagram illustrates a comparison of the application performance of the feature alignment module of this invention under different backbone networks in object detection tasks. It verifies the adaptability of the feature alignment module to multiple mainstream visual backbones and its impact on object detection performance. Three sets of control experiments are set up in the diagram, using ViT-T, DINOv2, and DINOv3 as backbone networks respectively. Each experiment includes two columns of detection results: the left column shows the original detection results without the feature alignment module of this invention, and the right column shows the detection results after incorporating the feature alignment module. The detected targets are marked with red rectangles in the diagram; the position, size, and confidence level of the rectangles visually reflect the model's ability to locate and recognize targets.

[0161] The comparison shows that after introducing the feature alignment module of this invention, the target detection performance of each backbone network is comparable to that of the original model, and there is no degradation in detection performance. The detection boxes are accurately located and have clear boundaries, and the identification of different targets in the scene is stable, with no obvious missed detections or confidence fluctuations. The above results demonstrate that the feature alignment module of this invention has good adaptability and robustness to different network structures, and can achieve cross-model and cross-scale feature mapping while maintaining target detection accuracy, verifying its versatility and effectiveness in target detection scenarios.

[0162] Example 3: A computer system. The computer system includes a processor and memory. The memory and processor are interconnected via a bus system and / or other forms of connection. The processor may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP). The memory may include volatile memory, such as random access memory (RAM). The memory may also include non-volatile memory. volatile memory), such as read-only memory (read... ROM (Mobile Memory), flash memory, hard disk drive (HDD), or solid state drive (SSD). The memory stores executable program code, which the processor executes to call the hierarchical feature alignment module described in Example 1 and the progressive layer decoupling distillation training method described in Example 2, thereby achieving knowledge transfer from a large model to a small model and completing the model training for the object detection task.

[0163] Example 4: A computer-readable storage medium. For example, the computer-readable storage medium may be a read-only memory (ROM). Read-only memory (ROM), random access memory (RAM), and compact disc read-only memory (CD-ROM). Only memory, CD ROM, magnetic tape, floppy disk, and optical data storage devices, etc. This computer-readable storage medium is used to store computer-readable instructions, which, when executed by a computer, can implement the progressive layer decoupling distillation training method as described in Example 2.

Claims

1. A large-to-small model progressive layer decoupling distillation training method, characterized in that, A feature alignment system for large-small models suitable for object detection is deployed between a teacher model and a student model. The feature alignment system includes a feature alignment module, which comprises: a spatial alignment submodule configured to adjust the spatial size of the image feature map of the teacher model to match that of the image feature map of the student model, outputting a first image feature map with the same number of channels as the teacher model and a spatial size matching that of the student model; and a channel alignment submodule configured to perform channel compression and semantic remapping on the first image feature map, generating a second image feature map with the same number of channels as the student model. The spatial alignment submodule and the channel alignment submodule are connected in series, with spatial alignment preceding channel alignment, allowing spatial structure adjustment and channel semantic mapping to be executed independently at different stages. The large-small model progressive layer decoupling distillation training method includes: In the initial stage, the target intermediate layer of the student model is selected, the parameters of the teacher model are kept frozen, the parameters of the student model backbone network are kept frozen, and the parameters of the feature alignment module and the parameters of the student model's task output layer are trained only according to the task loss of the student model itself, and the feature alignment benchmark is established through feature distillation loss. In the progressive unfreezing phase, under the feature alignment benchmark, the parameters of the selected target intermediate layer are gradually unfrozen in order from high to low layers. After each unfreezing, the parameters of the currently unfrozen target intermediate layer, the parameters of the feature alignment module, and the parameters of the task output layer are updated, and multi-level distillation loss is applied. In the joint optimization phase, after all the parameters of the target intermediate layer are unfrozen, the parameters of all student model parameters are jointly trained with the parameters of the feature alignment module, and the weights of the task loss, the feature distillation loss and the multi-level distillation loss are dynamically adjusted so that the weights of the feature distillation loss and the multi-level distillation loss gradually decrease and the weight of the task loss gradually increases.

2. The large-small model progressive layer decoupling distillation training method according to claim 1, characterized in that, The spatial alignment submodule selects a downsampling path or an upsampling path for scaling based on a resolution comparison between the image feature maps of the teacher model and the student model. After scaling, it performs structural consistency reshaping on the first image feature map through a spatial structure preservation layer. The structural consistency reshaping includes intra-channel spatial structure modeling and inter-channel structural consistency planning.

3. The large-small model progressive layer decoupling distillation training method according to claim 2, characterized in that, When the resolution of the image feature map of the teacher model is higher than that of the image feature map of the student model, the downsampling path extracts local aggregation information and context information respectively through parallel strided convolution and dilated convolution, and adds the image feature map output by the strided convolution to the image feature map output by the dilated convolution element by element; When the resolution of the image feature map of the teacher model is lower than that of the image feature map of the student model, the upsampling path performs bilinear interpolation on the image feature map of the teacher model and then uses learnable convolution for local reconstruction.

4. The large-small model progressive layer decoupling distillation training method according to claim 1, characterized in that, The channel alignment submodule includes: A channel compression layer is configured to map the number of channels of the first image feature map to the number of channels of the student model's image feature map through a 1×1 convolution to obtain compressed features; A channel semantic remapping layer is configured to perform global average pooling on the compressed features and then generate a channel semantic description through a multilayer perceptron. A dynamic channel weighting layer is configured to map the channel semantic description to weight coefficients using an activation function, and then multiply the weight coefficients by the compressed features channel by channel to obtain the second image feature map.

5. The large-small model progressive layer decoupling distillation training method according to claim 4, characterized in that, The KL divergence between the channel response distributions of the teacher model's image feature map and the student model's image feature map after channel compression weighting is calculated using the channel alignment loss function to constrain channel semantic alignment.

6. The large-small model progressive layer decoupling distillation training method according to claim 1, characterized in that, The target intermediate layer for screening the student model is determined based on the semantic level of features, the matching relationship between the feature map scale and the receptive field, and the contribution to the task loss. Among them, the mid-to-high-level feature layers of the student model are used as candidates, and the semantic level correspondence is confirmed by comparing the downsampling ratio of the corresponding layers of the teacher model and the student model. Then, redundant layers whose contribution to the task loss is lower than the preset condition are eliminated.

7. The large-small model progressive layer decoupling distillation training method according to claim 1, characterized in that, During the progressive thawing phase, the multi-level distillation loss includes the hierarchical distillation loss of the intermediate feature layer and the predictive distillation loss of the task output layer. The hierarchical distillation loss is used to constrain the image features of the target intermediate layer of the currently thawing model to be consistent with the image features of the corresponding layer of the teacher model, and the predictive distillation loss is used to constrain the task prediction output of the student model to be consistent with the task prediction output of the teacher model.

8. The large-small model progressive layer decoupling distillation training method according to claim 1, characterized in that, During the joint optimization phase, the weights of the task loss and distillation loss are dynamically adjusted as follows: In the formula, As a weight for distillation losses, the distillation losses include the characteristic distillation losses and the multi-level distillation losses. As the initial distillation weight, For task loss weight, This represents the current number of training steps. This represents the total number of training steps.

9. A computer system, characterized in that, It includes a processor and a memory, the memory storing program instructions, and the processor executing the program instructions to implement the large-small model progressive layer decoupling distillation training method according to any one of claims 1-8.

Images