A model compression method and a computer readable storage medium

Abstract

The application provides a model compression method and a computer readable storage medium. The method comprises the following steps: obtaining a sample image and label information, processing the sample image by using a teacher network to obtain a teacher intermediate layer feature map and a teacher prediction result; processing the sample image by using a student network to obtain a student intermediate layer feature map and a student prediction result; inputting the label information and the teacher intermediate layer feature map into a knowledge conversion network to obtain a target feature map and a background feature map, extracting key information from the target feature map and the background feature map respectively to obtain a target key feature map and a background key feature map; iteratively training the knowledge conversion network and the student network according to a total loss function; repeating the steps of extracting the intermediate layer feature, the step of extracting the key feature and the step of knowledge migration until an iteration stopping condition is reached, taking the student network as a compressed model to complete the knowledge migration from the teacher network to the student network and the compression of the network model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a model compression method and a computer-readable storage medium. Background Technology

[0002] Semantic-intensive deep vision detection solutions, such as semantic segmentation, offer advantages like high accuracy and reliability, making them the preferred technology for industrial image defect detection. However, the deep neural networks used in semantic-intensive tasks are complex, memory-intensive, and computationally intensive, placing high demands on hardware and hindering deployment on CPUs, mobile devices, or edge devices. This not only increases costs for manufacturers but also limits the further development and adoption of deep learning models. Meanwhile, the market demand for miniaturized and lightweight models is growing rapidly. Research on lightweight models mainly includes:

[0003] (1) Lightweight Model Structure Design: From the initial design stage, the goal is to optimize the network structure by designing a network structure with fewer parameters and lower computational cost. Examples of lightweight models include SqueezeNet (a simplified, lightweight convolutional neural network structure that heavily utilizes 1x1 convolutions), MobileNet (a lightweight deep neural network structure that employs depthwise separable convolutions), and the latest GhostNet (a lightweight deep neural network structure). The main disadvantages of lightweight networks are: longer training times or difficulty in fitting the model; and lower general applicability of the specialized network structure design. For example, MobileNet is primarily designed for mobile devices, making it less suitable for GPU deployment.

[0004] (2) Model quantization: This refers to converting the weight parameters of a trained network model from high-precision floating-point storage (operation) to low-precision floating-point (or integer) storage (operation), such as binarization, linear quantization, logarithmic quantization, etc. Weight quantization can effectively compress parameters, improve the inference speed of the network model, and reduce memory usage. However, quantization is an approximate calculation method, which is often accompanied by a relatively serious decrease in accuracy.

[0005] (3) Model pruning: This involves removing unnecessary neurons or structures from the network model. Based on the granularity of pruning, it can be roughly divided into four levels: fine-grained pruning (pruning neurons); vector pruning (pruning the internal structure of convolutional kernels); kernel pruning (removing a specific convolutional kernel); and filter pruning (pruning the entire convolutional kernel group). Model pruning can reduce network structure, decrease computational load, and improve inference speed. To avoid reducing model accuracy, iterative training is generally required, resulting in a long training time. Furthermore, the pruning method significantly impacts model accuracy, and there is a lack of universally applicable, highly accurate, fully automatic pruning strategies.

[0006] (4) Knowledge distillation: This refers to the transfer of knowledge from a trained, high-accuracy teacher (large) model to a student (small) model. Essentially, it involves guiding the more lightweight student (small) model to "learn and acquire" knowledge from the higher-performing, more complex teacher model. The effectiveness of distillation largely depends on the algorithm used. Student models typically have lower latency and fewer parameters, making distillation a relatively effective model compression method. In practice, to ensure accuracy, student models generally use classic networks (smaller than the teacher model), for example, ResNet-101 for the teacher network and ResNet-18 for the student network. Therefore, there is still significant room for compression optimization in the student model used for distillation.

[0007] For semantically intensive tasks, such as semantic segmentation, network models are often complex, computationally intensive, and have high latency to ensure detection performance, making model compression a more urgent need. Among existing compression methods, knowledge distillation performs well, but it still suffers from issues such as feature redundancy, inaccurate knowledge transformation, poor distillation results, and excessively large distilled models.

[0008] Therefore, it is necessary to improve upon the shortcomings of existing technologies. Summary of the Invention

[0009] The main technical problem solved by this invention is to provide a model compression method and a computer-readable storage medium, which improves the distillation effect and enhances the fitting and generalization ability of the compressed model through efficient knowledge conversion.

[0010] According to a first aspect, one embodiment provides a model compression method. The method includes:

[0011] The steps for extracting intermediate layer features are as follows: First, obtain sample images and corresponding annotation information. Then, process the sample images using a pre-trained teacher network to obtain n teacher intermediate layer feature maps at different resolutions and teacher prediction results. Next, process the sample images using a student network to be compressed to obtain n student intermediate layer feature maps at different resolutions and student prediction results. Wherein, n ≥ 3, and n is an integer.

[0012] The steps for extracting key features are as follows: The annotation information and the n teacher intermediate layer feature maps of different resolutions are input into a knowledge conversion network to obtain a target feature map and a background feature map. Key information is then extracted from both the target feature map and the background feature map to obtain a target key feature map corresponding to the target feature map and a background key feature map corresponding to the background feature map. The target feature map represents the target region in the teacher intermediate layer feature map, the background feature map represents the background region in the teacher intermediate layer feature map, and the annotation information is used to annotate the target region and the background region.

[0013] The steps of knowledge transfer are: iteratively training the knowledge transfer network R and the student network according to the total loss function;

[0014] The total loss function includes a feature map distillation loss function and a prediction result distillation loss function; the feature map distillation loss function is determined based on the differences between the target key feature map, the background key feature map, and the student intermediate layer feature map, and the prediction result distillation loss function is determined based on the differences between the teacher's prediction result and the student's prediction result.

[0015] Repeat the steps of extracting intermediate layer features, extracting key features, and knowledge transfer until a preset iteration stopping condition is reached. The student network that has reached the preset iteration stopping condition is then used as the compressed model.

[0016] In one embodiment, the knowledge transfer step further includes:

[0017] Pruning sub-step: When the number of iterations reaches a preset pruning threshold, the same pruning operation is applied to the knowledge transfer network and the student network;

[0018] Wherein, after the pruning operation, the channels in the knowledge transformation network that output the target key feature map and the background key feature map are matched with the channels in the student network that output the student intermediate layer feature map;

[0019] in,

[0020] The preset iteration stopping condition is: the student network converges and all pruning operations have been completed.

[0021] In one embodiment, the corresponding annotation information and the n teacher intermediate layer feature maps of different resolutions are input into the knowledge conversion network to obtain the target feature map and the background feature map;

[0022] include:

[0023] The target mask and background mask are obtained based on the corresponding annotation information. The resolution of the target mask and background mask are adjusted to obtain the target mask and background mask corresponding to the resolution of the teacher intermediate layer feature map. Based on the target mask and background mask corresponding to the resolution of the teacher intermediate layer feature map, the target feature map and background feature map of the corresponding resolution are obtained.

[0024] The teacher intermediate layer feature map is the feature map output by n consecutive intermediate layers of the teacher network, and the number of these n intermediate layers is not higher than a preset layer number threshold.

[0025] The number of teacher intermediate layer feature maps is the same as the number of student intermediate layer feature maps, and the resolution of each student intermediate layer feature map is the same as the resolution of its corresponding teacher intermediate layer feature map.

[0026] In one embodiment, the knowledge transformation network R includes an adaptive transformation module, which includes multiple adaptive transformation units. Each adaptive transformation unit includes an attention layer and a convolutional layer corresponding to the attention layer. The convolutional layer is used to reduce the dimensionality of the feature channels so that the number of feature channels in the feature map output by the convolutional layer matches the number of feature channels in the corresponding student intermediate layer feature map. The number of adaptive transformation units is equal to 2n, and the data processing flow inside each adaptive transformation unit is the same.

[0027] in,

[0028] The knowledge transfer network R extracts key information from the target feature map and the background feature map respectively, to obtain a target key feature map corresponding to the target feature map and a background key feature map corresponding to the background feature map, including:

[0029] The target feature map at the corresponding resolution is used as the input feature map of the adaptive transformation unit corresponding to the target feature map at the corresponding resolution, and is processed sequentially through the attention layer and the convolutional layer to output the target key feature map at the corresponding resolution; the background feature map at the corresponding resolution is used as the input feature map of the adaptive transformation unit corresponding to the background feature map at the corresponding resolution, and is processed sequentially through the attention layer and the convolutional layer to output the background key feature map at the corresponding resolution.

[0030] In one embodiment, the attention layer includes a channel sublayer and a spatial sublayer;

[0031] The step of taking the target feature map of the corresponding resolution as the input feature map of the adaptive transformation unit corresponding to the target feature map of the corresponding resolution, and processing it sequentially through the attention layer and the convolutional layer to output the target key feature map of the corresponding resolution includes:

[0032] The target feature map at the corresponding resolution is used as the input feature map of the adaptive transformation unit corresponding to the target feature map at the corresponding resolution. The input feature map is input into the channel sub-layer to obtain the channel attention map. The input feature map is input into the spatial sub-layer to obtain the spatial attention map. The channel attention map and the spatial attention map are added together to obtain the attention output feature map. The attention output feature map is input into the convolutional layer to perform a convolution operation to obtain the target key feature map at the corresponding resolution.

[0033] The process of taking the background feature map of the corresponding resolution as the input feature map of the adaptive transformation unit corresponding to the background feature map of the corresponding resolution, and processing it sequentially through the attention layer and the convolutional layer to output the background key feature map of the corresponding resolution includes:

[0034] The background feature map at the corresponding resolution is used as the input feature map of the adaptive transformation unit corresponding to the background feature map at the corresponding resolution. The input feature map is input into the channel sub-layer to obtain the channel attention map. The input feature map is input into the spatial sub-layer to obtain the spatial attention map. The channel attention map and the spatial attention map are added together to obtain the attention output feature map. The attention output feature map is input into the convolutional layer to perform a convolution operation to obtain the background key feature map at the corresponding resolution.

[0035] In one embodiment, inputting the input feature map into the channel sub-layer to obtain the channel attention map includes:

[0036] The input feature map is subjected to a first convolution process to obtain a feature map after the first convolution process, and the input feature map is subjected to a second convolution process to obtain a feature map after the second convolution process;

[0037] A first data reshaping operation is performed on the feature map after the first convolutional processing to obtain a feature map after the first data reshaping operation; a first softmax operation is performed on the feature map after the first data reshaping operation to obtain a feature map after the first softmax operation.

[0038] A second data reshaping operation is performed on the feature map after the second convolution process to obtain a feature map after the second data reshaping operation.

[0039] A first multiplication operation is performed on the feature map after the first Softmax operation and the feature map after the second data reshaping operation to obtain the feature map after the first multiplication operation.

[0040] A third data reshaping operation is performed on the feature map after the first multiplication operation to obtain the feature map after the third data reshaping operation.

[0041] A third convolution process is performed on the feature map after the third data reshaping operation to obtain a feature map after the third convolution process; a first sigmoid operation is performed on the feature map after the third convolution process to obtain a feature map after the first sigmoid operation.

[0042] The feature map after the first Sigmoid operation is multiplied by the input feature map to obtain the feature map after the second multiplication operation;

[0043] The feature map after the second multiplication operation is used as the output of the channel sub-layer.

[0044] In one embodiment, inputting the input feature map into the spatial sub-layer to obtain a spatial attention map includes:

[0045] The input feature map is subjected to a fourth convolution process to obtain a feature map after the fourth convolution process, and the input feature map is subjected to a fifth convolution process to obtain a feature map after the fifth convolution process;

[0046] A global average pooling operation is performed on the feature map after the fourth convolution to obtain the feature map after the global average pooling operation.

[0047] A fourth data reshaping operation is performed on the feature map after the global average pooling operation to obtain a feature map after the fourth data reshaping operation.

[0048] The fifth data reshaping operation is performed on the feature map after the fifth convolution to obtain the feature map after the fifth data reshaping operation;

[0049] A third multiplication operation is performed on the feature map after the fourth data reshaping operation and the feature map after the fifth data reshaping operation to obtain the feature map after the third multiplication operation;

[0050] A second Softmax operation is performed on the feature map after the third multiplication operation to obtain a feature map after the second Softmax operation.

[0051] A sixth data reshaping operation is performed on the feature map after the second Softmax operation to obtain the feature map after the sixth data reshaping operation.

[0052] Perform a second Sigmoid operation on the feature map after the sixth data reshaping operation to obtain the feature map after the second Sigmoid operation;

[0053] The feature map after the second Sigmoid operation is multiplied by the input feature map to obtain the feature map after the fourth multiplication operation;

[0054] The feature map after the fourth multiplication operation is used as the output of the spatial sub-layer.

[0055] In one embodiment, the pruning operation includes:

[0056] If the pruned object contains convolutional layers, the L2 norm of each convolutional kernel in each channel of the pruned object is calculated, and the L2 norm of the convolutional kernel is used as the importance index of the convolutional kernel. The importance indices of all the convolutional kernels in the channel are sorted from largest to smallest or smallest to largest. Every preset interval of training iterations, the N1 convolutional kernels with the smallest importance index in the current channel are pruned based on a preset pruning rate. Wherein, N1 is obtained based on the preset pruning rate and the number of convolutional kernels in the current channel.

[0057] If the pruned object contains a batch normalization layer or other normalization layers, then the weights of the channels in each of the current batch normalization layers or other normalization layers in the pruned object are sorted from largest to smallest or smallest to largest, and the N2 channels with the smallest weights in the batch normalization layer or the normalization layer are pruned based on the preset pruning rate every preset interval training number of iterations.

[0058] Wherein, N2 is obtained based on the preset pruning rate and the number of channels corresponding to the currently pruned batch normalization layer or the current other normalization layers;

[0059] The pruned object is the part of the student network S other than the channel that outputs the student prediction result, or the convolutional layer in the adaptive transformation unit corresponding to the attention layer; during the iterative training process, the network structure and weight parameters of the teacher network remain fixed.

[0060] In one embodiment, the total loss function is expressed as: L = αL f +βL r ,

[0061] Wherein, α and β are preset weighting coefficients, and L f The feature map distillation loss function is L. r The distillation loss function is used for the predicted results.

[0062] In one embodiment, the feature map distillation loss function L f The expression is:

[0063]

[0064] Wherein, the s i p represents the i-th layer of the student intermediate layer feature map output by the student network S. i The q represents the target key feature map of the i-th layer. i This represents the background key feature map of the i-th layer, the ω represents a preset coefficient; n is the total number of layers in the target key feature map, the total number of layers in the background key feature map, or the total number of layers in the student intermediate layer feature map;

[0065] The expression for the MSE is:

[0066]

[0067] Wherein, h represents s i The flattened one-dimensional vector, the s i The flattening of a one-dimensional vector refers to the s i A one-dimensional vector formed by arranging the elements in order, where k represents p. i or the q mentioned i The flattened one-dimensional vector, where u represents the position index of each element in h or k, z represents the total number of elements in h or k, and h u The element at position u represents h, and k represents... u This represents the element at position u where k is located.

[0068] In one embodiment, the distillation loss function L of the prediction result r The expression is:

[0069] L r =MSE(P s ,P t );

[0070] Wherein, P s This represents the student's prediction result, where P is... t This indicates the teacher's prediction result;

[0071] The expression for the MSE is:

[0072]

[0073] Wherein, the teacher prediction result includes a predicted score graph, the student prediction result also includes a predicted score graph, w represents the flattened one-dimensional vector of the predicted score graph in the student prediction result, v represents the flattened one-dimensional vector of the predicted score graph in the teacher prediction result, j represents the index of an element of w or v, m represents the total number of elements of w or v, and w j v represents the element located at position j in w. j This represents the element located at position j in v.

[0074] According to a second aspect, one embodiment provides a computer-readable storage medium. The computer-readable storage medium includes a program. The program is executable by a processor to perform the methods described in any of the embodiments herein.

[0075] The beneficial effects of this application are:

[0076] This model compression method includes: acquiring sample images and corresponding annotation information; processing the sample images using a pre-trained teacher network to obtain n teacher intermediate layer feature maps and teacher prediction results at different resolutions; processing the sample images using the student network to be compressed to obtain n student intermediate layer feature maps and student prediction results at different resolutions; inputting the annotation information and the n teacher intermediate layer feature maps at different resolutions into a knowledge transfer network to obtain target feature maps and background feature maps, and extracting key information from the target feature maps and background feature maps respectively to obtain target key feature maps corresponding to the target feature maps and background key feature maps corresponding to the background feature maps; iteratively training the knowledge transfer network R and the student network according to the total loss function; repeating the steps of extracting intermediate layer features, extracting key features, and knowledge transfer until a preset iteration stopping condition is reached, and using the student network that reaches the preset iteration stopping condition as the compressed model. This method improves the distillation effect and enhances the fitting and generalization ability of the compressed model through the efficient knowledge transfer mechanism of the knowledge transfer network. Attached Figure Description

[0077] Figure 1 This is a flowchart illustrating a model compression method according to one embodiment;

[0078] Figure 2 This is a schematic diagram of the teacher network, knowledge transfer network, and student network in one embodiment.

[0079] Figure 3 This is a schematic diagram of an adaptive conversion module according to one embodiment;

[0080] Figure 4 This is a schematic diagram of the network structure and data processing of the attention layer in one embodiment.

[0081] Figure 5 This is a flowchart illustrating another embodiment of the model compression method. Detailed Implementation

[0082] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0083] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0084] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0085] As mentioned earlier, network models for semantically intensive tasks are characterized by a large number of parameters and high computational cost, while knowledge distillation is a good compression technique. Currently, feature distillation is commonly used to achieve knowledge transfer from the teacher model to the student model by directly matching the transformation relationships between intermediate layer features, thereby compressing the network models used for semantically intensive tasks. However, using feature distillation to compress network models has two major problems: First, due to the large size of the teacher network, the intermediate layer features it extracts are redundant, resulting in a lack of accuracy in knowledge transfer. Forcing the student model to imitate the same feature output as the teacher network makes it impossible for the student network to distinguish between important features and general features (or redundant features), leading to a lack of generalization in the fitting effect. Second, the student model after distillation has a large number of parameters, resulting in high computational cost and making it unsuitable for deployment on CPUs or mobile devices, leaving considerable room for compression optimization.

[0086] To address the first problem with feature distillation, this application proposes a model compression method. The first technical objective of this model compression method is that the knowledge conversion network of this application guides the teacher network to perform accurate feature distillation on the student network through an efficient knowledge conversion module (i.e., the adaptive conversion module below). The training (compression) time is short, and the obtained compressed model (i.e., the distilled student model) has strong fitting and generalization capabilities.

[0087] For details, please refer to Figure 2 In its overall design, this compression method adopts a TS (Teacher-Student) architecture, consisting of three parts: the teacher network T, the knowledge transfer network R, and the student network S. The teacher network T is a pre-trained task model with the required accuracy. Its network structure and weight parameters remain fixed throughout training without updates. Its main function is to output intermediate layer feature maps and task results (e.g., for semantic segmentation tasks, the result can be a pixel-by-pixel multi-class score map). The student network is the network model to be compressed. The knowledge transfer network R comprises two functions: first, it decouples the foreground and background of the multi-layer intermediate feature maps output by the teacher network based on the target mask, obtaining decoupled foreground and background feature maps; second, it adaptively transforms the decoupled foreground and background feature maps to obtain adaptively transformed target and background feature maps. During training, important / critical information learned from the adaptively transformed target feature maps (i.e., the target key feature map below) and background feature maps (i.e., the background key feature map below) is selectively and accurately transmitted to the student network. This model compression method designs a specific loss function for training the feature distillation process of the entire network model. The efficient and accurate knowledge transfer process of the knowledge transfer network R and the design of the corresponding loss function are key to the successful compression of the entire network model.

[0088] The technical solution of this application will be described in detail below with reference to the embodiments.

[0089] Please refer to Figure 1 The model compression method includes:

[0090] Step S100 for extracting intermediate layer features: Obtain sample images and corresponding annotation information; process the sample images using the pre-trained teacher network to obtain n teacher intermediate layer feature maps and teacher prediction results at different resolutions; process the sample images using the student network to be compressed to obtain n student intermediate layer feature maps and student prediction results at different resolutions; where n≥3;

[0091] Step S200 for extracting key features: Input the annotation information and n teacher intermediate layer feature maps of different resolutions into the knowledge conversion network to obtain target feature maps and background feature maps, and extract key information from the target feature maps and background feature maps respectively to obtain target key feature maps corresponding to the target feature maps and background key feature maps corresponding to the background feature maps; wherein, the target feature map is used to represent the target region in the teacher intermediate layer feature maps, the background feature map is used to represent the background region in the teacher intermediate layer feature maps, and the annotation information is used to annotate the target region and the background region;

[0092] Step S300 of knowledge transfer: Iteratively train the knowledge transfer network and the student network according to the total loss function;

[0093] The total loss function includes the feature map distillation loss function and the prediction result distillation loss function. The feature map distillation loss function is determined based on the differences between the target key feature map, the background key feature map and the student intermediate layer feature map, while the prediction result distillation loss function is determined based on the differences between the teacher's prediction result and the student's prediction result.

[0094] Step S400: Repeat steps S100 (extracting intermediate layer features), S200 (extracting key features), and S300 (knowledge transfer) until the preset iteration stopping condition is reached. The student network that has reached the preset iteration stopping condition is used as the compressed model.

[0095] In step S100 above, those skilled in the art can select the above sample image and corresponding annotation information according to actual needs. For example, the sample image can be an image obtained by taking pictures of industrial products.

[0096] It should be noted that the "acquiring sample images and corresponding annotation information" in step S100 above is prior art in this field, and therefore will not be described in detail here.

[0097] In step S100 above, please refer to... Figure 2The teacher network T is a pre-trained task model with the required accuracy. Throughout the training process (e.g., distillation), the network structure and weight parameters of the teacher network T remain fixed and are not updated. The weight parameters of the knowledge transfer network R and the student network S change and are updated normally during training iterations. The backbone network in the teacher network T used for feature extraction is generally much larger than the backbone network in the student network S. For example, the teacher network can use the feature extraction layer of ResNet-101 as its backbone, while the student network can use the feature extraction layer of ResNet-18. ResNet-101 is a deep convolutional neural network, and ResNet-18 is an 18-layer convolutional neural network. A key feature of both ResNet-101 and ResNet-18 is the use of residual blocks to address the vanishing gradient problem in deep network training.

[0098] It should be noted that for semantically intensive tasks (such as semantic segmentation), the teacher network T typically uses a large feature extractor as the backbone, resulting in high computational cost, high memory usage, slow inference speed, and significant feature redundancy. Therefore, distilling a smaller network is an efficient solution for network compression. Please refer to [reference needed]. Figure 2 The teacher network T is primarily responsible for: accepting a sample image x as input and outputting n teacher intermediate layer feature maps of different resolutions (e.g., when n equals 3, corresponding to t1, t2, and t3 in the image) and the teacher prediction results (e.g., P). t If the target task is semantic segmentation, the teacher's prediction result P t This is a pixel-by-pixel multi-class score map.

[0099] In some embodiments, the student network may adopt the same network structure as the teacher network, and a compressed student model may be obtained through subsequent steps S200, S300 and S400.

[0100] In some embodiments, to avoid the training process being too long, a student network smaller than the teacher network is generally selected for training (such as knowledge distillation training), that is, the size of the student network to be compressed is smaller than that of the teacher network.

[0101] It should be noted that the student network S is smaller in model size compared to the teacher network T, but its functionality remains consistent with that of the teacher network T. Please refer to [reference needed]. Figure 2 During the training (distillation) process, the sample image x is used as the input to the student network S. The student network S outputs n intermediate layer feature maps of different resolutions (such as s1, s2, s3) and the corresponding student prediction results P.s The student intermediate layer feature maps (e.g., s1, s2, s3) are compared with the corresponding target key feature maps (e.g., p1, p2, p3) and the corresponding background key feature maps (e.g., q1, q2, q3) to calculate the feature distillation loss function (e.g., d1, d2, d3 and d4, d5, d6, collectively referred to as feature distillation loss). The corresponding teacher prediction result P is then used. t And student prediction results P s Calculate the distillation loss function (result distillation loss) for the predicted results mentioned below. After the above training is complete, the student network S serves as the compressed model.

[0102] In some embodiments, the teacher intermediate layer feature map can be the feature maps output by n discontinuous intermediate layers of the teacher network. It should be noted that if the teacher intermediate layer feature map is the feature map output by n discontinuous intermediate layers of the teacher network, the performance of the final compressed model may not be optimal.

[0103] In some embodiments, to improve the performance of the final compressed model, the teacher intermediate layer feature map is the feature map output by n consecutive intermediate layers of the teacher network, and the number of these n intermediate layers does not exceed a preset layer threshold. Those skilled in the art can determine the preset layer threshold according to the actual scenario requirements. For example, when the preset layer threshold is 5 and n equals 3, the n consecutive intermediate layers of the teacher network can be the 1st, 2nd, and 3rd layers closest to the network model input, or the 2nd, 3rd, and 4th layers closest to the network model input, or the 3rd, 4th, and 5th layers closest to the network model input.

[0104] In some embodiments, the number of teacher intermediate layer feature maps is the same as the number of student intermediate layer feature maps. The resolution of each student intermediate layer feature map is the same as the resolution of its corresponding teacher intermediate layer feature map. In some embodiments, the number of teacher intermediate layer feature maps may also be different from the number of student intermediate layer feature maps.

[0105] It should be noted that there are no particular restrictions on the resolution of the n teacher intermediate layer feature maps. However, in practice, those skilled in the art typically adopt the following approach: the resolution of the teacher intermediate layer feature map output by the next intermediate layer is half the resolution of the teacher intermediate layer feature map output by the previous intermediate layer.

[0106] It should be noted that in some embodiments, the prediction result (such as the teacher's prediction result and the student's prediction result) in step S100 above may refer to the pixel-by-pixel classification result of the semantic segmentation task. However, in other embodiments, the prediction result may also be other prediction content. For semantically intensive tasks such as image reconstruction, the prediction result is not the classification task result. Those skilled in the art can determine the specific content of the "prediction result" according to the actual application scenario requirements. Furthermore, the semantically intensive task targeted by this application does not specifically refer to semantic segmentation.

[0107] It should be noted that the specific processes by which the teacher network processes the sample image to obtain the teacher intermediate layer feature map and the teacher prediction result, and the student network processes the sample image to obtain the student intermediate layer feature map and the student prediction result in step S100 above, are existing technologies / common knowledge in this field, and therefore will not be elaborated here.

[0108] In step S200 above, the target feature map can represent the information contained in the target region of the teacher's intermediate layer feature map, and the background feature map can represent the information contained in the background region of the teacher's intermediate layer feature map. Specifically, the target key feature map is used to represent the key information in the target region that affects the teacher's prediction result, and the background key feature map is used to represent the key information in the background region that affects the teacher's prediction result.

[0109] In step S200 above, the corresponding annotation information and n teacher intermediate layer feature maps of different resolutions are input into the knowledge conversion network to obtain the target feature map and the background feature map, including:

[0110] Step S210: Obtain the target mask and background mask according to the corresponding annotation information, adjust the resolution of the target mask and background mask respectively to obtain the target mask and background mask corresponding to the resolution of the teacher intermediate layer feature map, and obtain the target feature map and background feature map of the corresponding resolution according to the target mask and background mask corresponding to the resolution of the teacher intermediate layer feature map.

[0111] In some embodiments, in step S210 above, after obtaining the corresponding annotation information, the target area and background area in the sample image are also determined.

[0112] It should be noted that the process of obtaining the target mask and background mask based on the corresponding annotation information in step S210 above is existing technology / common knowledge in this field. For example, the pixels representing the target region in the above sample image can be marked as 1, while the remaining pixels can be marked as 0, thereby obtaining the target mask corresponding to the above sample image. Similarly, the pixels representing the background region in the above sample image can be marked as 1, while the remaining pixels can be marked as 0, thereby obtaining the background mask corresponding to the above sample image. Since the resolutions of the above n teacher intermediate layer feature maps are different, and the resolutions of the student intermediate layer feature maps are the same as the resolutions of the corresponding teacher intermediate layer feature maps, and in order to facilitate the efficient learning of key information such as the "target key feature map corresponding to the target feature map" and the "background key feature map corresponding to the target feature map" mentioned below by the student network S, it is necessary to adjust the resolutions of the obtained target mask and background mask respectively to obtain the target mask and background mask corresponding to the resolution of the teacher intermediate layer feature map. Then, the target feature map, background feature map, target key feature map and background key feature map of the corresponding resolution are obtained by using the corresponding target mask and background mask.

[0113] In some embodiments, please refer to Figure 2 During the training (distillation) phase, x is the input sample image, y1 is the binarized background mask, and y2 is the binarized target mask. After obtaining the target mask y2 and background mask y1 based on the corresponding annotation information, multiple target resolution resizing operations (such as...) are performed on the target mask y2 and background mask y1 respectively. Figure 2 The process involves resizing the target mask y2 and background mask y1 to the resolution of the original n teacher intermediate layer feature maps, resulting in target and background masks corresponding to the resolution of the teacher intermediate layer feature maps (e.g., t1, t2, or t3). Then, element-wise multiplication (e.g., ...) is performed between the target and background masks at their respective resolutions and the teacher intermediate layer feature maps at their corresponding resolutions. Figure 2 The target feature map and background feature map of the corresponding resolution are obtained by using Mul in the Mul.

[0114] Please refer to Figure 2Structurally, the knowledge transfer network R mainly includes an adaptive transfer module CN. Functionally, it mainly includes two aspects: one is to decouple the output features of the teacher network T; the other is to efficiently and accurately transfer the decoupled feature maps to achieve knowledge transfer from the teacher network T to the student network S. Since the target region and background region contain different information in intensive knowledge distillation tasks, simply having the student network S imitate the teacher intermediate layer feature map output by the teacher network T will cause the student network S to ignore the importance perception and judgment of different regions (such as the target region and background region mentioned above). For semantically intensive tasks, spatial semantics is particularly important. Therefore, using a binarized target mask y2 and a binarized background mask y1 to decouple the teacher intermediate layer feature map output by the teacher network T into the aforementioned target feature map and background feature map is beneficial for the student network S to learn and distinguish the importance of features (such as the "target key feature map corresponding to the target feature map" and the "background key feature map corresponding to the target feature map" mentioned below), thereby improving distillation efficiency and accuracy. The adaptive transformation module CN is the core and key structure of the knowledge transformation network R. The adaptive transformation module CN can efficiently transform the feature information in the target feature map and the background feature map from both channel and spatial aspects to obtain the target key feature map (such as p1, p2, p3) and the background key feature map (such as q1, q2, q3) corresponding to the target feature map, thus accurately guiding the distillation process.

[0115] The aforementioned knowledge transformation network includes an adaptive transformation module, which comprises multiple adaptive transformation units. Each adaptive transformation unit includes an attention layer and a convolutional layer corresponding to the attention layer. The convolutional layer is used to reduce the dimensionality of the feature channels so that the number of feature channels in the feature map output by the convolutional layer matches the number of feature channels in the corresponding student intermediate layer feature map. The number of adaptive transformation units is equal to 2n, and the data processing flow inside each adaptive transformation unit is the same.

[0116] The knowledge transfer network extracts key information from both the target feature map and the background feature map to obtain a target key feature map corresponding to the target feature map and a background key feature map corresponding to the background feature map, including:

[0117] The target feature map at the corresponding resolution is used as the input feature map of the adaptive transformation unit corresponding to the target feature map at the corresponding resolution. The input feature map is processed by the attention layer and the convolutional layer in sequence to output the target key feature map at the corresponding resolution. The background feature map at the corresponding resolution is used as the input feature map of the adaptive transformation unit corresponding to the background feature map at the corresponding resolution. The input feature map is processed by the attention layer and the convolutional layer in sequence to output the background key feature map at the corresponding resolution.

[0118] It should be noted that the reason for "the number of adaptive transformation units equals 2n" is that the adaptive transformation units of the knowledge transformation network R need to be adapted to the number of layers of the teacher intermediate layer feature map output by the teacher network T. For example, if the number of layers of the teacher intermediate layer feature map output by the teacher network T is 3 (i.e., n equals 3), then the adaptive transformation module CN needs 6 (i.e., 3 × 2 = 6) adaptive transformation units, that is, the number of attention layers in the adaptive transformation module CN is 6; if the number of layers of the teacher intermediate layer feature map output by the teacher network is 4, then the adaptive transformation module CN needs 8 (i.e., 4 × 2 = 8) adaptive transformation units, that is, the number of attention layers in the adaptive transformation module CN is 8.

[0119] In some embodiments, please refer to Figure 3 , Figure 3 The diagram illustrates the network structure of the adaptive transformation module CN when the teacher network T outputs a teacher intermediate layer feature map with 3 layers. In this case, the adaptive transformation module CN mainly consists of 6 attention layers (e.g., A1, A2, A3, A4, A5, A6) and 6 convolutional layers (1×1 Conv) with 1×1 kernels corresponding to the attention layers. The inputs f1, f2, and f3 to the adaptive transformation module CN are the target feature maps at the corresponding resolutions after feature decoupling, and b1, b2, and b3 are the background feature maps at the corresponding resolutions after feature decoupling. The inputs f1, f2, f3 and b1, b2, b3 are processed by the attention layers A1, A2, A3, A4, A5, A6 and the corresponding 1×1 Conv, respectively, outputting target key feature maps p1, p2, p3 corresponding to the target feature maps and background key feature maps q1, q2, q3 corresponding to the target feature maps.

[0120] It should be noted that the use of the 3-layer teacher intermediate layer feature map of the teacher network in this application is only as an example and does not mean that the number of layers of the teacher intermediate layer feature map output by the teacher network T must be 3.

[0121] In the aforementioned adaptive transformation module CN, the attention layers (such as A1, A2, A3, A4, A5, and A6) introduce an attention mechanism from both channel and spatial perspectives. This allows the network to automatically learn and selectively focus on important and critical information in the input feature maps, thereby improving distillation performance and enhancing the model's generalization ability.

[0122] Please refer to Figure 4 , Figure 4This diagram illustrates the network structure and general data processing flow of a single attention layer. It should be noted that the structure and processing flow of multiple attention layers (such as A1, A2, A3, A4, A5, and A6) in the adaptive transformation module CN are consistent, differing only in the input and output channels. The specific settings for these input and output channels are common knowledge in the field and will not be elaborated upon here. This diagram uses a single attention layer as an example: an attention layer consists of a channel sublayer (CA) and a spatial sublayer (SA). For semantically intensive tasks, the spatiality and semantics of the features extracted by the network model are crucial. This application specifically designs the channel sublayer (CA) and spatial sublayer (SA) to extract the most important feature information from the input feature map (such as target feature maps f1, f2, f3 and background feature maps b1, b2, b3 at corresponding resolutions) from both channel and spatial perspectives. The channel sublayer (CA) and spatial sublayer (SA) maintain the high resolution of the input feature map throughout the process, more accurately preserving spatial information and capturing contextual information, achieving effective long-distance modeling, and ultimately efficiently transforming the most important feature information from the input feature map, thus improving the performance of the student network.

[0123] The above methods use the target feature maps of corresponding resolutions as input feature maps for the adaptive transformation units corresponding to those resolutions, which are then processed sequentially through attention layers and convolutional layers to output the target key feature maps of corresponding resolutions, including:

[0124] The target feature map at the corresponding resolution is used as the input feature map of the adaptive transformation unit corresponding to the target feature map at the corresponding resolution. The input feature map is input into the channel sub-layer to obtain the channel attention map. The input feature map is input into the spatial sub-layer to obtain the spatial attention map. The channel attention map and the spatial attention map are added together to obtain the attention output feature map. The attention output feature map is input into the convolutional layer for convolution operation to obtain the target key feature map at the corresponding resolution.

[0125] The background feature maps at corresponding resolutions are used as input feature maps to the adaptive transformation units corresponding to those resolutions. These are then processed sequentially through attention layers and convolutional layers to output key background feature maps at the corresponding resolutions, including:

[0126] The background feature maps of the corresponding resolution are used as input feature maps for the adaptive transformation units corresponding to the background feature maps of the corresponding resolution. The input feature maps are input into the channel sub-layer to obtain the channel attention map, and the input feature maps are input into the spatial sub-layer to obtain the spatial attention map. The channel attention map and the spatial attention map are added together to obtain the attention output feature map. The attention output feature map is input into the convolutional layer for convolution operation to obtain the background key feature map of the corresponding resolution.

[0127] The above-mentioned input feature map into the channel sublayer CA to obtain the channel attention map includes:

[0128] Perform a first convolution process on the input feature map to obtain a feature map after the first convolution process, and perform a second convolution process on the input feature map to obtain a feature map after the second convolution process;

[0129] A first data reshaping operation is performed on the feature map after the first convolution to obtain a feature map after the first data reshaping operation; a first softmax operation is performed on the feature map after the first data reshaping operation to obtain a feature map after the first softmax operation.

[0130] A second data reshaping operation is performed on the feature map after the second convolution to obtain the feature map after the second data reshaping operation.

[0131] The first multiplication operation is performed on the feature map after the first Softmax operation and the feature map after the second data reshaping operation to obtain the feature map after the first multiplication operation.

[0132] A third data reshaping operation is performed on the feature map after the first multiplication operation to obtain the feature map after the third data reshaping operation.

[0133] A third convolution process is performed on the feature map after the third data reshaping operation to obtain a feature map after the third convolution process; a first sigmoid operation is performed on the feature map after the third convolution process to obtain a feature map after the first sigmoid operation.

[0134] The feature map after the first Sigmoid operation is multiplied by the input feature map to obtain the feature map after the second multiplication operation;

[0135] The feature map after the second multiplication operation is used as the output of the channel sublayer CA.

[0136] In some embodiments, please refer to Figure 4 The channel sublayer CA is used to process the input feature map (such as...) Figure 4 The input [N, C, H, W] on the left side of the map is subjected to a first convolution to obtain a feature map after the first convolution, so that the number of channels in the feature map after the first convolution is reduced to 1. For the input feature map (e.g., ... Figure 4The input [N,C,H,W] on the left side of the input map is subjected to a second convolution to obtain a feature map after the second convolution, so that the number of channels of the feature map after the second convolution is reduced to half of the number of channels corresponding to the input of the adaptive transformation unit. The data format of the input feature map is [N,C,H,W], where N, C, H, and W represent the number of features, the number of channels, the height, and the width of the input feature map, respectively, and C is greater than 1. The data format of the feature map after the first convolution is [N,1,H,W], and the data format of the feature map after the second convolution is [N,C / 2,H,W].

[0137] A first data reshaping operation is performed on the feature map after the first convolution to obtain a feature map after the first data reshaping operation, so that the height and width are merged; wherein, the data format of the feature map after the first data reshaping operation is [N,H×W,1];

[0138] The first softmax operation is performed on the feature map after the first data reshaping operation to obtain the feature map after the first softmax operation. The channel dimension can be placed on the last dimension before performing the first softmax operation, and the execution dimension of the first softmax operation is [H×W]. In the softmax operation (such as the first softmax operation mentioned above and the second softmax operation described later), softmax is a commonly used activation function, mainly used to solve classification problems. It can transform a k-dimensional vector into a k-dimensional probability distribution.

[0139] A second data reshaping operation is performed on the feature map after the second convolution to obtain a feature map after the second data reshaping operation, so that the height and width [H×W] are merged; wherein, the data format of the feature map after the second data reshaping operation is [N,C / 2,H×W], wherein the above C / 2 can be rounded and the rounded value is used as the value of the above C / 2.

[0140] The feature map after the first Softmax operation and the feature map after the second data reshaping operation are subjected to a first multiplication operation to obtain the feature map after the first multiplication operation. The data format of the feature map after the first multiplication operation is [N, C / 2, 1]. The multiplication operation (such as the first multiplication operation here, the subsequent second multiplication operation, the third multiplication operation, etc.) is a common operation in this field, used to perform element-wise multiplication of two tensors.

[0141] The feature map after the first multiplication operation is subjected to a third data reshaping operation to obtain the feature map after the third data reshaping operation. The third data reshaping operation is mainly to expand the last dimension. The data format of the feature map after the third data reshaping operation is [N,C / 2,1,1].

[0142] A third convolution is performed on the feature map after the third data reshaping operation to obtain a third convolution-processed feature map (so that the number of channels in the third convolution-processed feature map is the same as the number of channels in the input feature map, that is, restoring its number of channels to the input feature map). Figure 4 The number of channels in the input on the left side of the image; where the data format of the feature map after the third convolution is [N,C,1,1];

[0143] The feature map after the third convolution is processed by performing a first Sigmoid operation to obtain the feature map after the first Sigmoid operation. The Sigmoid in the Sigmoid operation (such as the first Sigmoid operation, the second Sigmoid operation, etc.) is the Sigmoid function. The Sigmoid function is a commonly used non-linear activation function that can map any real number to the interval [-1,1] or [0,1]. It is widely used in artificial neural networks, logistic regression, and deep learning models.

[0144] The feature map after the first Sigmoid operation is compared with the input feature map (i.e.) Figure 4 The second multiplication operation is performed on the input to obtain the feature map after the second multiplication operation (e.g., ...). Figure 4 The data format of the feature map after the second multiplication operation is [N,C,H,W]. The feature map after the second multiplication operation is the corresponding channel attention map mentioned above.

[0145] The above-mentioned inputting of the input feature map into the spatial sublayer SA to obtain the spatial attention map includes:

[0146] The input feature map is subjected to a fourth convolution process to obtain a feature map after the fourth convolution process, and the input feature map is subjected to a fifth convolution process to obtain a feature map after the fifth convolution process;

[0147] The feature map after the fourth convolution is processed by global average pooling to obtain the feature map after global average pooling.

[0148] A fourth data reshaping operation is performed on the feature map after global average pooling to obtain the feature map after the fourth data reshaping operation.

[0149] The feature map after the fifth convolution is reshaped by performing the fifth data reshaping operation to obtain the feature map after the fifth data reshaping operation.

[0150] The third multiplication operation is performed on the feature map after the fourth data reshaping operation and the feature map after the fifth data reshaping operation to obtain the feature map after the third multiplication operation.

[0151] The second softmax operation is performed on the feature map after the third multiplication operation to obtain the feature map after the second softmax operation.

[0152] The sixth data reshaping operation is performed on the feature map after the second softmax operation to obtain the feature map after the sixth data reshaping operation.

[0153] The second Sigmoid operation is performed on the feature map after the sixth data reshaping operation to obtain the feature map after the second Sigmoid operation.

[0154] The feature map after the second Sigmoid operation is multiplied by the input feature map to obtain the feature map after the fourth multiplication operation;

[0155] The feature map after the fourth multiplication operation is used as the output of the spatial sub-layer.

[0156] In some embodiments, please refer to Figure 4 The spatial sublayer SA is applied to the input feature map (e.g.) Figure 4 The input [N, C, H, W] on the right side of the input map is subjected to a fourth convolution to obtain a feature map after the fourth convolution, thereby reducing the number of input channels of the adaptive transformation unit to half of the original number. The input feature map (e.g., ...) is then processed. Figure 4 The input on the right side of the input (i.e., [N,C,H,W]) undergoes a fifth convolution to obtain the feature map after the fifth convolution, thereby reducing the number of input channels of the adaptive transformation unit to half of the original number.

[0157] The data format of the feature map after the fourth convolution is [N,C / 2,H,W].

[0158] The data format of the feature map after the fifth convolution is ([N,C / 2,H,W]).

[0159] A global average pooling operation is performed on the feature map after the fourth convolution to obtain a feature map with global average pooling, thereby reducing the resolution to 1×1. The data format of the feature map after global average pooling is [N, C / 2, 1, 1]. Global average pooling is a pooling operation in this field that aggregates information from the entire feature map. It does not set a pooling window size but directly performs average pooling on the entire feature map; by averaging the feature values ​​of each channel, it generates aggregated feature values ​​corresponding to each channel; the final aggregated feature vector can be regarded as a global information representation of the entire feature map.

[0160] A fourth data reshaping operation is performed on the feature map after global average pooling to obtain a feature map after the fourth data reshaping operation, which merges the height and width to 1 and places the channel dimension on the last dimension.

[0161] The data format of the feature map after the fourth data reshaping operation is [N,1,C / 2];

[0162] The feature map after the fifth convolution is processed by the fifth data reshaping operation to obtain the feature map after the fifth data reshaping operation, which merges the height and width [H×W]; wherein, the data format of the feature map after the fifth data reshaping operation is [N,C / 2,H×W];

[0163] The feature map after the fourth and fifth data reshaping operations is subjected to a third multiplication operation to obtain the feature map after the third multiplication operation. The data format of the feature map after the third multiplication operation is ([N,1,H×W]);

[0164] The feature map after the third multiplication operation is then subjected to a second softmax operation to obtain a feature map after the second softmax operation. The execution dimension of the second softmax operation is [H×W].

[0165] A sixth data reshaping operation is performed on the feature map after the second softmax operation to obtain a feature map after the sixth data reshaping operation, so as to restore the aforementioned height and width [H,W]; wherein, the data format of the feature map after the sixth data reshaping operation is [N,1,H,W];

[0166] The second Sigmoid operation is performed on the feature map after the sixth data reshaping operation to obtain the feature map after the second Sigmoid operation.

[0167] The feature map after the second Sigmoid operation is compared with the input feature map (e.g.) Figure 4 The fourth multiplication operation is performed on the input on the right side of the middle to obtain the feature map after the fourth multiplication operation (e.g., the input on the right side of the middle). Figure 4 (outs on the right side of the middle);

[0168] Among them, the feature map after the fourth multiplication operation (i.e. Figure 4 The data format of outs on the right is [N,C,H,W].

[0169] The feature map resulting from the fourth multiplication operation is the corresponding spatial attention map. Next, the channel attention map `outc` and the spatial attention map `outs` corresponding to the target feature map are added together to obtain the corresponding attention output feature map. This attention output feature map is then input into a convolutional layer for convolution to obtain the target key feature map at the corresponding resolution. Similarly, the channel attention map `outc` and the spatial attention map `outs` corresponding to the background feature map are added together to obtain the attention output feature map. This attention output feature map is then input into a convolutional layer for convolution to obtain the background key feature map at the corresponding resolution. It can be understood that for an attention layer, its input is... Figure 4 The input in the middle.

[0170] It should be noted that the various convolutional operations (such as the first convolutional operation), data reshaping operations (such as the first data reshaping operation), multiplication operations (such as the first multiplication operation), softmax operations (such as the first softmax operation), sigmoid operations (such as the first sigmoid operation), and addition operations described above are merely examples and are all common operations in the art. For example, the first convolutional operation can use a 1x1 convolutional layer. Those skilled in the art can adapt the above operations to the actual scenario, and no specific limitations are imposed on the above operations here.

[0171] It should be noted that the attention output feature map obtained from each attention layer will be used as the convolutional layer corresponding to the attention layer in the adaptive transformation module CN (e.g., ...). Figure 3 The input of the 1×1 Conv in the system is used for channel transformation, thereby achieving matching with the feature channels of the student network S. It is important to note that the channel sub-layer CA, the spatial sub-layer SA, and the convolutional layers in the adaptive transformation module CN (excluding the attention layer) are all independent of each other and do not affect each other; the input of the channel sub-layer CA and the spatial sub-layer SA is the same.

[0172] It should be noted that the adaptive transformation module CN includes multiple adaptive transformation units, and each adaptive transformation unit includes an attention layer (such as...). Figure 3 (A1, A2, A3, A4, A5, A6) and a convolutional layer corresponding to the attention layer (e.g., ... Figure 3 The 6 convolutional layers (1×1 Conv) in the middle. The convolutional layers corresponding to the attention layers (e.g.) Figure 3 The 6 convolutional layers (1×1 Conv) in the diagram do not include the 1×1 Conv convolutional layers inside the attention layers (A1, A2, A3, A4, A5, A6 in the diagram). The role of the convolutional layers corresponding to the attention layers is to reduce the dimensionality of the feature channels, extract information, and ensure channel matching with the student network S. Compared to other methods that reduce the number of channels, this method uses convolutional layers corresponding to the attention layers (such as... Figure 3 The purpose of the 6 convolutional layers (1×1 Conv) in the network is to reduce the redundancy of the target and background key feature maps output by the teacher network, and to extract key information through linear combination of features, thereby improving the network's expressive power and achieving richer and more accurate feature distillation. In other embodiments, other methods to reduce the number of channels can also be used.

[0173] In some embodiments, addressing the second problem of feature distillation, the model compression method proposed in this application can simultaneously perform knowledge distillation on the model to be compressed, prune the model, and conduct iterative training. This improves the model compression rate, reduces the number of model parameters, and significantly shortens inference time. The resulting compressed model can then be deployed on GPUs, CPUs, mobile devices, and other common platforms, broadening the application scenarios of deep learning technologies and improving user efficiency. In other words, the second technical objective of the model compression method proposed in this application is to compress the model to be compressed into a smaller, more compact, and more efficient model (i.e., the compressed model in step S400) for semantically intensive tasks, while maintaining or even improving accuracy. This compressed model can be deployed and used on multiple terminals such as GPUs, CPUs, and mobile devices to achieve real-time inference.

[0174] In some embodiments, during training, the knowledge conversion network R maintains consistent pruning operations with the student network S to ensure channel matching during feature distillation. The student network S, as the model to be distilled and compressed (i.e., the compressed model mentioned above), undergoes corresponding pruning of its network structure during training iterations; that is, it is continuously pruned during distillation to further compress the compressed model. Upon completion of training, the student network S, as the compressed model, can be deployed and used on multiple terminals such as GPUs, CPUs, and mobile devices. During training, the network structures and weight parameters of the knowledge conversion network R and the student network S change and update normally. Specifically, during training, the adaptive conversion module CN of the knowledge conversion network R performs a pruning operation with a pruning rate of g (in this application, g = 0.5 in the example). This pruning operation targets the convolutional layer in the adaptive conversion module CN corresponding to the attention layer (see reference...). Figure 3 The six convolutional layers (1×1 Conv) in the student network S correspond to the attention layers and are consistent with the pruning operations of the student network S while maintaining structural adaptation; no pruning operations are performed on the attention layers in the adaptive transformation module CN.

[0175] In some embodiments, please refer to Figure 5 This application provides another model compression method. This other model compression method is essentially the same as the aforementioned model compression method except for the knowledge transfer step S300. The knowledge transfer step S300 of this other model compression method includes, in addition to the content related to the knowledge transfer step S300 in the aforementioned model compression method, also includes:

[0176] Pruning sub-steps: When the number of iterations reaches the preset pruning threshold, the same pruning operation is performed on the knowledge transfer network R and the student network S.

[0177] In this process, after the pruning operation, the channels of the target key feature map and the background key feature map output in the knowledge transfer network R are matched with the channels of the student intermediate layer feature map output in the student network S.

[0178] In this embodiment, the preset iteration stopping condition in step S400 is: the student network converges and all pruning operations have been completed.

[0179] In the above-described pruning sub-steps, those skilled in the art can determine the preset pruning threshold based on the actual application scenario. For example, the preset pruning threshold can be 100 epochs (i.e., the number of iterations of training). It should be noted that the specific process for determining the convergence of the student network is common knowledge in the field, and therefore will not be elaborated here.

[0180] When applying the same pruning operation to both the knowledge transformation network R and the student network S, the pruning operations for student network S and knowledge transformation network R must be compatible because of the channel matching relationship between them. For example, please refer to... Figure 2 After feature decoupling, the target feature map or background feature map obtained have 256, 512, and 1024 channels, respectively. The student intermediate layer feature maps (such as s1, s2, and s3) output by the student network have 64, 128, and 256 channels, respectively. The input and output channels of the multi-layer convolutional layers corresponding to the attention layers in the adaptive transformation module CN of the knowledge transformation network R should be 256->64, 512->128, and 1024->256, respectively. If pruning is performed during training, assuming half of the feature channels are pruned (i.e., the pruning rate is 0.5), then the number of channels in the student intermediate layer feature maps output by the student network will be 32, 64, and 128, respectively. Correspondingly, the input and output channels of the multi-layer convolutional layers corresponding to the attention layers in the adaptive transformation module CN of the knowledge transformation network R should be 256->32, 512->64, and 1024->128, respectively.

[0181] It should be noted that in the student network S, other network layers associated with pruned channels undergo the same pruning operation, while output channels related to the task output results do not undergo pruning during the entire training process. For example, for semantic segmentation tasks, taking the UNet model (a commonly used image segmentation model) as an example, when the encoder network used to extract features from the first few layers is pruned, its corresponding decoder network also needs to undergo the same pruning operation; otherwise, the number of network channels will not correspond, and training will be impossible. For example, convolutional layer A is the last convolutional layer of the encoder, with its input and output channels numbering [256, 512], respectively. Following convolutional layer A is the first convolutional layer of the decoder (such as convolutional layer B), with its input and output channels numbering [512, 256], respectively. When convolutional layer A prunes half of its channels, i.e., the number of input and output channels of convolutional layer A becomes [128, 256], then convolutional layer B also needs to prune half of its channels, i.e., the number of input and output channels of convolutional layer B needs to be [256, 128]. Assuming that after convolutional layer B... The network layers, including convolutional layers C, all require the same pruning operation. For the last convolutional layer performing the semantic segmentation task, the output prediction channel cannot be pruned. For example, the last convolutional layer performing the segmentation task originally had [64, 5] input and output channels, where the number 5 refers to the number of classification categories in the segmentation task. When half of the channels are pruned, the number of input channels (i.e., the aforementioned number 64) is pruned to 32, while the number of output channels (i.e., the aforementioned number 5) remains unchanged.

[0182] The above pruning operations include:

[0183] If the object to be pruned contains convolutional layers, the L2 norm of each convolutional kernel in each channel of the object to be pruned is calculated, and the L2 norm of the convolutional kernel is used as the importance index of the convolutional kernel. The importance indices of all convolutional kernels in the channel are sorted from largest to smallest or smallest to largest. Every preset interval of training iterations, the N1 convolutional kernels with the smallest importance index in the current channel are pruned based on a preset pruning rate. Wherein, N1 is obtained based on the preset pruning rate and the number of convolutional kernels in the current channel.

[0184] If the object to be pruned contains a batch normalization layer or other normalization layers, the weights of the channels in the current batch normalization layers or other normalization layers in the object to be pruned will be sorted from largest to smallest or smallest to largest. The N2 channels with the smallest weights in the batch normalization layers or normalization layers will be pruned based on the preset pruning rate after a preset interval of training iterations.

[0185] N2 is obtained based on the preset pruning rate and the number of channels corresponding to the current pruned batch normalization layer or other current normalization layers.

[0186] The pruned objects are the parts of the student network S other than the channels that output the student's prediction results, or the convolutional layers in the adaptive transformation unit corresponding to the attention layer.

[0187] The L2 norm mentioned above is a measure of a vector, also known as the Euclidean norm or modulus. Typically, the L2 norm is calculated by summing the squares of the elements of a vector and then taking the square root.

[0188] Those skilled in the art can determine the preset pruning rate and preset interval training number according to actual needs.

[0189] N2 here is a value obtained by multiplying the number of channels corresponding to the batch normalization layer or other normalization layer by the preset pruning rate.

[0190] In some embodiments, pruning is not performed if the number of training iterations does not reach a preset pruning threshold (e.g., the preset pruning threshold is the first 100 epochs); when the number of training iterations exceeds the preset pruning threshold, a progressive iterative pruning operation is performed, that is, for each channel's convolutional kernel and each layer's normalization layer, a portion (e.g., 5%) of the channels corresponding to the preset pruning rate is pruned every preset interval of training iterations (e.g., 10 epochs). Iteration is performed during training, that is, feature distillation and pruning operations are performed simultaneously, until all pruning operations for the pruned object have been completed and the model (i.e., the student network S) converges.

[0191] It should be noted that in some embodiments, if the student network S is pruned during knowledge distillation training, the knowledge transfer network R must also maintain the same pruning operation as the student network S to adapt to the network structure of the student network S and ensure channel matching in the distillation process. This improves the distillation accuracy while enabling knowledge transfer from the teacher network T to the student network S and model compression.

[0192] This application designs a corresponding total loss function for the feature distillation process of the aforementioned network model to train the student network, etc. Once training (distillation and pruning operations) is complete, the student network S obtained in step S400 can be deployed and used on multiple terminals such as GPUs, CPUs, and mobile devices. The teacher network T and the knowledge transfer network R do not participate in subsequent deployment and inference. It is important to note that the efficient and accurate knowledge transfer of the knowledge transfer network R, combined with the aforementioned pruning operations, and the design of the loss function are key to the successful compression of the entire network model.

[0193] The expression for the total loss function is: L = αL f +βL r Where α and β are preset weighting coefficients, and Lf L is the feature map distillation loss function. r The distillation loss function is used to predict the results.

[0194] Those skilled in the art can select the aforementioned preset weighting coefficients according to the needs of actual application scenarios. For example, in some embodiments, α = 0.7 and β = 0.3.

[0195] The above feature map distillation loss function L f The expression is:

[0196]

[0197] Among them, s i p represents the intermediate feature map of the i-th student layer output by the student network S. i Let q represent the key feature map of the target in the i-th layer. i This represents the key feature map of the i-th layer background. ω represents a preset coefficient, and n is the total number of layers in the target key feature map, the background key feature map, or the student intermediate layer feature map.

[0198] In some embodiments, n equals 3. The aforementioned preset coefficient... ω = 0.2. Those skilled in the art can determine the specific values ​​of n and the preset coefficients according to the actual needs of the scenario; the statement "n equals 3" is merely an example.

[0199] It should be noted that, since background features (such as the background key feature map mentioned above) are relatively easy to learn, while target features (such as the target key feature map mentioned above) are more difficult to fit and learn, different weight coefficients (i.e., the preset coefficients mentioned above) are set. And ω), so that the student network S can better grasp the "transferred knowledge" and achieve precise distillation.

[0200] The Mean Square Error (MSE) mentioned above calculates the mean square error between the two inputs.

[0201] The expression for this MSE is:

[0202]

[0203] Where h represents s i The flattened one-dimensional vector, s i The flattening of a one-dimensional vector refers to s i A one-dimensional vector formed by arranging the elements of p in order, where k represents p. i or q iLet u be a flattened one-dimensional vector, where u represents the index of each element in h or k, z represents the total number of elements in h or k, and h is the index of the vector. u The element at position u is represented by h, k u This represents the element at position u with index k.

[0204] The above prediction results show the distillation loss function L. r The expression is:

[0205] L r =MSE(P s ,P t );

[0206] Among them, P s P represents the student's prediction result. t This indicates the teacher's prediction results.

[0207] The expression for MSE is:

[0208]

[0209] The teacher's prediction result includes a predicted score graph, and the student's prediction result also includes a predicted score graph. `w` represents the flattened one-dimensional vector of the predicted score graph in the student's prediction result, `v` represents the flattened one-dimensional vector of the predicted score graph in the teacher's prediction result, `j` represents the index of an element in `w` or `v`, and `m` represents the total number of elements in `w` or `v`. j v represents the element at position j in w. j This represents the element at position j in v.

[0210] It should be noted that this application uses a semantic segmentation task as an example, but for other semantically intensive tasks, the specific settings of the distillation loss function for the prediction result are consistent with the principle here.

[0211] It should be noted that in the corresponding loss function examples in this application, the predicted score map in the teacher prediction results or student prediction results is used. This predicted score map is not the final classification result. The use of "predicted score map" is just to better represent the various prediction representations of semantically intensive tasks.

[0212] Please refer to Figure 2 Feature map distillation loss function L f The input to the feature distillation loss is the aforementioned student intermediate layer feature maps (e.g., s1, s2, s3), the aforementioned target key feature maps (e.g., p1, p2, p3), and the aforementioned background key feature maps (e.g., q1, q2, q3). The prediction result distillation loss function L... r The input to (i.e., result distill loss) is the teacher prediction result mentioned above (e.g., P).t ) and the above student prediction results (such as P) s Feature map distillation loss function L f And the predicted distillation loss function L r Together, they control the training of the entire network model.

[0213] In some embodiments, it can be seen that the model compression method for semantically intensive tasks proposed in this application has a short model compression (training) time and the obtained compressed model has strong fitting ability and generalization ability. In addition, the knowledge conversion network in this application guides the teacher network to accurately distill the features of the student network through an efficient knowledge conversion module (i.e., the adaptive conversion module mentioned above), resulting in a short training (compression) time and a compressed model with strong fitting and generalization ability.

[0214] In some embodiments, it can be seen that, compared with similar methods currently available, the model compression method for semantically intensive tasks proposed in this application can compress network models (such as the student network mentioned above) in an end-to-end manner. Its advantages are: based on an efficient adaptive knowledge conversion mechanism, it can achieve accurate distillation, accelerate the model training (compression process) time, and enhance the fitting and generalization ability of the compressed model.

[0215] In some embodiments, the advantages of the model compression method for semantically intensive tasks proposed in this application also include: pruning during model distillation and iterative training to improve model compression ratio, reduce model parameter count, and significantly shorten inference time; furthermore, this method is applicable to a wide range of terminals, and the compressed model (i.e., the compressed model mentioned above) can be deployed on terminals such as GPUs, CPUs, and mobile devices to meet real-time operation requirements, reducing usage costs for production enterprises and achieving cost reduction and efficiency improvement. In other words, this model compression method can compress network models (such as the student network mentioned above) while maintaining or even improving model accuracy, enabling universal deployment on GPUs, CPUs, and mobile devices, broadening the application scenarios of deep learning technologies, and improving user efficiency. In addition, this model compression method can improve compression ratio, reduce computational load, and increase inference speed; by combining pruning operations during the distillation process, unnecessary structures in the student network are more effectively removed, significantly reducing the number of model parameters and shortening inference time.

[0216] The above is a description of a model compression method. Some embodiments of this application also disclose a computer-readable storage medium, including a program that can be executed by a processor to implement the model compression method as described in any embodiment of this application.

[0217] This document describes various exemplary embodiments with reference to them. However, those skilled in the art will recognize that changes and modifications can be made to the exemplary embodiments without departing from the scope of this document. For example, various operational steps and components for performing operational steps can be implemented in different ways depending on the specific application or considering any number of cost functions associated with the operation of the system (e.g., one or more steps can be deleted, modified, or combined with other steps).

[0218] In the above embodiments, implementation can be achieved, in whole or in part, by software, hardware, firmware, or any combination thereof. Furthermore, as those skilled in the art will understand, the principles herein can be reflected in a computer program product on a computer-readable storage medium pre-loaded with computer-readable program code. Any tangible, non-transitory computer-readable storage medium may be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CD-ROMs, DVDs, Blu-ray discs, etc.), flash memory, and / or the like. These computer program instructions can be loaded onto a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to form a machine, such that instructions executing on the computer or other programmable data processing apparatus can generate means for implementing a specified function. These computer program instructions can also be stored in a computer-readable storage medium that can instruct the computer or other programmable data processing apparatus to operate in a particular manner, such that instructions stored in the computer-readable storage medium can form an article of manufacture including means for implementing the specified function. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to perform a series of operational steps on the computer or other programmable apparatus to produce a computer-implemented process, such that instructions executing on the computer or other programmable apparatus can provide steps for implementing the specified function.

[0219] While the principles herein have been illustrated in various embodiments, numerous modifications to the structure, arrangement, proportions, elements, materials, and components, particularly suited to specific environmental and operational requirements, may be used without departing from the principles and scope of this disclosure. These modifications and other alterations or alterations will be included within the scope of this document.

[0220] The foregoing specific descriptions have been described with reference to various embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of this disclosure. Therefore, considerations for this disclosure are to be illustrative rather than restrictive, and all such modifications are to be included within its scope. Similarly, advantages, other advantages, and solutions to problems with respect to various embodiments have been described above. However, benefits, advantages, solutions to problems, and any elements that produce these, or make them more explicit, should not be construed as critical, essential, or necessary. The term “comprising” and any other variations thereof as used herein are non-exclusive inclusion, meaning that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed or not part of the process, method, system, article, or apparatus. Furthermore, the term “coupled” and any other variations thereof as used herein refer to physical connections, electrical connections, magnetic connections, optical connections, communication connections, functional connections, and / or any other connections.

[0221] Those skilled in the art will recognize that many changes can be made to the details of the above embodiments without departing from the basic principles of the invention. Therefore, the scope of the invention should be determined only by the claims.

Claims

1. A model compression method, characterized in that, include: The steps for extracting intermediate layer features are as follows: First, obtain sample images and corresponding annotation information. Then, process the sample images using a pre-trained teacher network to obtain n teacher intermediate layer feature maps at different resolutions and teacher prediction results. Next, process the sample images using a student network to be compressed to obtain n student intermediate layer feature maps at different resolutions and student prediction results. Wherein, n ≥ 3, and n is an integer. The steps for extracting key features are as follows: The annotation information and the n teacher intermediate layer feature maps of different resolutions are input into a knowledge conversion network to obtain a target feature map and a background feature map. Key information is then extracted from both the target feature map and the background feature map to obtain a target key feature map corresponding to the target feature map and a background key feature map corresponding to the background feature map. The target feature map represents the target region in the teacher intermediate layer feature map, the background feature map represents the background region in the teacher intermediate layer feature map, and the annotation information is used to annotate the target region and the background region. The steps of knowledge transfer are: Applying the total loss function to the knowledge transfer network. Iterative training is performed on the student network; The total loss function includes a feature map distillation loss function and a prediction result distillation loss function; the feature map distillation loss function is determined based on the differences between the target key feature map, the background key feature map, and the student intermediate layer feature map, and the prediction result distillation loss function is determined based on the differences between the teacher's prediction result and the student's prediction result. The knowledge transfer step further includes a pruning sub-step: when the number of iterations reaches a preset pruning threshold, the same pruning operation is applied to both the knowledge transfer network and the student network; the pruning operation includes: If the pruned object contains convolutional layers, the L2 norm of each convolutional kernel in each channel of the pruned object is calculated, and the L2 norm of the convolutional kernel is used as the importance index of the convolutional kernel. The importance indices of all the convolutional kernels in the channel are sorted from largest to smallest or smallest to largest. Every preset interval of training iterations, the N1 convolutional kernels with the smallest importance index in the current channel are pruned based on a preset pruning rate. Wherein, N1 is obtained based on the preset pruning rate and the number of convolutional kernels in the current channel. If the pruned object contains a batch normalization layer or other normalization layers, then the weights of the channels in each of the current batch normalization layers or other normalization layers in the pruned object are sorted from largest to smallest or smallest to largest. Every preset interval of training iterations, the N2 channels with the smallest weights in the batch normalization layers or normalization layers are pruned based on the preset pruning rate; wherein, N2 is obtained based on the preset pruning rate and the number of channels corresponding to the currently pruned batch normalization layer or other normalization layers. Repeat the steps of extracting intermediate layer features, extracting key features, and knowledge transfer until a preset iteration stopping condition is reached. The student network that has reached the preset iteration stopping condition is then used as the compressed model.

2. The model compression method as described in claim 1, characterized in that, After the pruning operation, the channels of the target key feature map and the background key feature map output in the knowledge transformation network are matched with the channels of the student intermediate layer feature map output in the student network; in, The preset iteration stopping condition is: the student network converges and all pruning operations have been completed.

3. The model compression method as described in claim 1 or 2, characterized in that, The step of inputting the corresponding annotation information and the n teacher intermediate layer feature maps of different resolutions into the knowledge transformation network to obtain the target feature map and the background feature map includes: The target mask and background mask are obtained based on the corresponding annotation information. The resolution of the target mask and background mask are adjusted to obtain the target mask and background mask corresponding to the resolution of the teacher intermediate layer feature map. Based on the target mask and background mask corresponding to the resolution of the teacher intermediate layer feature map, the target feature map and background feature map of the corresponding resolution are obtained. The teacher intermediate layer feature map is the feature map output by n consecutive intermediate layers of the teacher network, and the number of these n intermediate layers is not higher than a preset layer number threshold. The number of teacher intermediate layer feature maps is the same as the number of student intermediate layer feature maps, and the resolution of each student intermediate layer feature map is the same as the resolution of its corresponding teacher intermediate layer feature map.

4. The model compression method as described in claim 3, characterized in that, The knowledge conversion network The system includes an adaptive transformation module, which comprises multiple adaptive transformation units. Each adaptive transformation unit includes an attention layer and a convolutional layer corresponding to the attention layer. The convolutional layer is used to reduce the dimensionality of the feature channels so that the number of feature channels in the feature map output by the convolutional layer matches the number of feature channels in the corresponding student intermediate layer feature map. The number of adaptive transformation units is equal to 2n, and the data processing flow inside each adaptive transformation unit is the same. in, The knowledge conversion network Key information is extracted from the target feature map and the background feature map respectively to obtain a target key feature map corresponding to the target feature map and a background key feature map corresponding to the background feature map, including: The target feature map at the corresponding resolution is used as the input feature map of the adaptive transformation unit corresponding to the target feature map at the corresponding resolution, and is processed sequentially through the attention layer and the convolutional layer to output the target key feature map at the corresponding resolution; the background feature map at the corresponding resolution is used as the input feature map of the adaptive transformation unit corresponding to the background feature map at the corresponding resolution, and is processed sequentially through the attention layer and the convolutional layer to output the background key feature map at the corresponding resolution.

5. The model compression method as described in claim 4, characterized in that, The attention layer includes a channel sublayer and a spatial sublayer; The step of taking the target feature map of the corresponding resolution as the input feature map of the adaptive transformation unit corresponding to the target feature map of the corresponding resolution, and processing it sequentially through the attention layer and the convolutional layer to output the target key feature map of the corresponding resolution includes: The target feature map at the corresponding resolution is used as the input feature map of the adaptive transformation unit corresponding to the target feature map at the corresponding resolution. The input feature map is input into the channel sub-layer to obtain the channel attention map. The input feature map is input into the spatial sub-layer to obtain the spatial attention map. The channel attention map and the spatial attention map are added together to obtain the attention output feature map. The attention output feature map is input into the convolutional layer to perform a convolution operation to obtain the target key feature map at the corresponding resolution. The process of taking the background feature map of the corresponding resolution as the input feature map of the adaptive transformation unit corresponding to the background feature map of the corresponding resolution, and processing it sequentially through the attention layer and the convolutional layer to output the background key feature map of the corresponding resolution includes: The background feature map at the corresponding resolution is used as the input feature map of the adaptive transformation unit corresponding to the background feature map at the corresponding resolution. The input feature map is input into the channel sub-layer to obtain the channel attention map. The input feature map is input into the spatial sub-layer to obtain the spatial attention map. The channel attention map and the spatial attention map are added together to obtain the attention output feature map. The attention output feature map is input into the convolutional layer to perform a convolution operation to obtain the background key feature map at the corresponding resolution.

6. The model compression method as described in claim 5, characterized in that, The step of inputting the input feature map into the channel sub-layer to obtain the channel attention map includes: The input feature map is subjected to a first convolution process to obtain a feature map after the first convolution process, and the input feature map is subjected to a second convolution process to obtain a feature map after the second convolution process; A first data reshaping operation is performed on the feature map after the first convolutional processing to obtain a feature map after the first data reshaping operation; a first softmax operation is performed on the feature map after the first data reshaping operation to obtain a feature map after the first softmax operation. A second data reshaping operation is performed on the feature map after the second convolution process to obtain a feature map after the second data reshaping operation. A first multiplication operation is performed on the feature map after the first Softmax operation and the feature map after the second data reshaping operation to obtain the feature map after the first multiplication operation. A third data reshaping operation is performed on the feature map after the first multiplication operation to obtain the feature map after the third data reshaping operation. A third convolution process is performed on the feature map after the third data reshaping operation to obtain a feature map after the third convolution process; a first sigmoid operation is performed on the feature map after the third convolution process to obtain a feature map after the first sigmoid operation. The feature map after the first Sigmoid operation is multiplied by the input feature map to obtain the feature map after the second multiplication operation; The feature map after the second multiplication operation is used as the output of the channel sub-layer.

7. The model compression method as described in claim 5, characterized in that, The step of inputting the input feature map into the spatial sub-layer to obtain the spatial attention map includes: The input feature map is subjected to a fourth convolution process to obtain a feature map after the fourth convolution process, and the input feature map is subjected to a fifth convolution process to obtain a feature map after the fifth convolution process; A global average pooling operation is performed on the feature map after the fourth convolution to obtain the feature map after the global average pooling operation. A fourth data reshaping operation is performed on the feature map after the global average pooling operation to obtain a feature map after the fourth data reshaping operation. The fifth data reshaping operation is performed on the feature map after the fifth convolution to obtain the feature map after the fifth data reshaping operation; A third multiplication operation is performed on the feature map after the fourth data reshaping operation and the feature map after the fifth data reshaping operation to obtain the feature map after the third multiplication operation; A second Softmax operation is performed on the feature map after the third multiplication operation to obtain a feature map after the second Softmax operation. A sixth data reshaping operation is performed on the feature map after the second Softmax operation to obtain the feature map after the sixth data reshaping operation. Perform a second Sigmoid operation on the feature map after the sixth data reshaping operation to obtain the feature map after the second Sigmoid operation; The feature map after the second Sigmoid operation is multiplied by the input feature map to obtain the feature map after the fourth multiplication operation; The feature map after the fourth multiplication operation is used as the output of the spatial sub-layer.

8. The model compression method as described in claim 4, characterized in that, The pruning operation also includes: The pruned object is the part of the student network other than the channel that outputs the student prediction result, or the convolutional layer in the adaptive transformation unit that corresponds to the attention layer; During the iterative training process, the network structure and weight parameters of the teacher network remain fixed.

9. The model compression method as described in claim 1, characterized in that, The expression for the total loss function is: , Among them, the and These are preset weighting coefficients, the The feature map distillation loss function is... The distillation loss function is used for the predicted results.

10. The model compression method as described in claim 9, characterized in that, The feature map distillation loss function The expression is: ； Among them, the The student network The output of the i-th layer of the student intermediate layer feature map, the Indicates the first The target key feature map described in the layer, the This represents the background key feature map of the i-th layer, the and This represents a preset coefficient; n is the total number of layers in the target key feature map, the background key feature map, or the student intermediate layer feature map. Among them, the The expression is: ； Among them, the Indicates the The flattened one-dimensional vector, the The flattening of a one-dimensional vector refers to the The elements are arranged in order to form a one-dimensional vector, the Indicates the or the aforementioned The flattened one-dimensional vector, the Indicates the or The position index of each element in the text, the Indicates the or The total number of elements in the middle, the Indicates the In the The element of position, the Indicates the In the The element at a given position.

11. The model compression method as described in claim 9, characterized in that, The prediction result distillation loss function The expression is: ； Wherein, P s This represents the student's prediction result, where P is... t This indicates the teacher's prediction result; The The expression is: ， The teacher's prediction results include a predicted score graph, and the student's prediction results also include a predicted score graph. The flattened one-dimensional vector represents the predicted score graph in the student prediction results. The flattened one-dimensional vector represents the predicted score graph in the teacher's prediction results. Indicates the Or the above The index of the element, the Indicates the Or the above The total number of elements, the Indicates the The middle is located in the The element of position, the Indicates the The middle is located in the The element at a given position.

12. A computer-readable storage medium, characterized in that, Includes a program that can be executed by a processor to implement the model compression method as described in any one of claims 1 to 11.

Images