Target detection knowledge distillation method based on hierarchical perception category prototype alignment

Abstract

The application relates to the technical field of image processing, in particular to a target detection knowledge distillation method based on hierarchical perception category prototype alignment. The method comprises the following steps: inputting a training image batch into a teacher network and a student network respectively; based on the scale of a real label boundary box, the real label boundary box is mapped to a corresponding feature pyramid level, and a corresponding target region feature is intercepted based on the position information of the real label boundary box; a teacher global anchor point prototype is generated, and a student local dynamic prototype is directly taken as a to-be-aligned feature; a teacher global context prototype and a student global context prototype are extracted; a guided feature distillation constraint is determined, and a distillation loss function is constructed in the same feature level. The method can solve the technical problem that the detection precision and generalization ability of a lightweight student network are insufficient due to the neglect of the symbiotic relationship between a target and a macro background, the semantic confusion of multi-scale features and the instability of knowledge anchors in the existing target detection knowledge distillation method.

Description

Technical Field

[0001] The embodiments of this application relate to the field of image processing technology, and in particular to a knowledge distillation method for target detection based on hierarchical perception category prototype alignment. Background Technology

[0002] With the breakthroughs achieved by deep neural networks in object detection tasks, high-precision detection models have been widely applied in fields such as autonomous driving, remote sensing monitoring, and intelligent surveillance. However, high-performance detection networks are usually accompanied by huge computational overhead and a large number of parameters, making them difficult to deploy in real time on edge embedded devices with limited computing power. Knowledge distillation, as a mainstream model lightweighting technique, effectively improves object detection accuracy by transferring the rich knowledge contained in large teacher networks to lightweight student networks, while maintaining the advantages of lightweight models and high inference speed.

[0003] In knowledge distillation research for object detection, effectively transferring the core knowledge of the teacher network—including high-order semantics, discriminative ability, and topological structure—in the high-dimensional feature space to the student network is crucial for improving distillation results. However, in complex scenarios (e.g., urban areas with numerous features or remote sensing images), existing techniques still suffer from the following shortcomings: First, current object detection distillation methods typically ignore the symbiotic semantic relationships between individual targets and their macroscopic background in real-world complex scenes. Isolating and aligning foreground targets and directly removing background regions, or relying on cumbersome feature clustering algorithms to forcibly extract the background, leads to the severe fragmentation and loss of key spatial interaction relationships and environmental priors during transfer, resulting in insufficient scene understanding and generalization ability for complex and variable scenarios. Second, traditional feature map-based distillation methods often ignore the scale decoupling characteristics of feature pyramids in detection tasks. Forcibly aggregating target features at different scales, or performing coarse-grained alignment at a single level, leads to severe semantic confusion and collapse, reducing the model's detection accuracy for extreme-sized targets (extremely small or extremely large targets). Furthermore, in category-based prototype-based knowledge distillation, the prototypes extracted by the teacher network should serve as "standard knowledge anchors" for student network learning. However, most existing methods directly use local batch features with strong randomness and significant individual differences as benchmark anchors for knowledge transfer, resulting in drastic fluctuations and fragmentation of the extracted knowledge anchors. This makes it difficult for student networks to capture stable and essential feature topologies, leading to a significant decrease in the quality of high-order semantic knowledge transfer. Summary of the Invention

[0004] In view of this, embodiments of this application propose a knowledge distillation method for object detection based on hierarchical perception category prototype alignment, aiming to solve the technical problems of insufficient detection accuracy and generalization ability of lightweight student networks in complex scenarios due to the neglect of the symbiotic relationship between the target and the macro background, the semantic confusion of multi-scale features, and the instability of knowledge anchors.

[0005] To achieve the above objectives, embodiments of this application propose a knowledge distillation method for object detection based on hierarchical awareness category prototype alignment, the method comprising the following steps: Obtain batches of training images and their corresponding ground truth bounding boxes; the ground truth bounding boxes contain information on scale and location. The training images are input into the teacher network and the student network to be trained in batches to extract multi-scale feature maps of each level of the feature pyramid in the teacher network and the student network. Based on the scale of the real labeled bounding boxes, they are mapped to the corresponding feature pyramid level. On the mapped level, the corresponding target region features are extracted based on the position information of the real labeled bounding boxes, and aggregated by category to obtain the local dynamic prototype of the teacher and the local dynamic prototype of the student. On the teacher network side, the teacher's local dynamic prototype is used to perform momentum updates on the preset global multi-scale prototype set to generate the teacher's global anchor prototype. On the student network side, the student's local dynamic prototype is directly used as the feature to be aligned. Based on the teacher global anchor prototype, a prototype response graph is constructed, and the prototype response graph is used to perform guided feature distillation constraints on the feature graphs of corresponding levels of the teacher network and student network. Within the same feature level, a distillation loss function is constructed based on the teacher's global anchor prototype, the student's local dynamic prototype, the teacher's global context prototype, the student's global context prototype, and the guided feature distillation constraint. This function is then combined with the basic task loss of object detection for backpropagation to update the parameters of the student network, resulting in the trained student network.

[0006] This application proposes a knowledge distillation method for target detection based on hierarchical perception category prototype alignment. The method involves acquiring batches of training images and their corresponding ground truth bounding boxes. These batches are input into a teacher network and a student network to be trained, respectively, to extract multi-scale feature maps at each level of the feature pyramid in both networks. Based on the scale of the ground truth bounding boxes, they are mapped to the corresponding feature pyramid levels. At the mapped levels, the corresponding target region features are extracted based on the positional information of the ground truth bounding boxes and aggregated by category to obtain teacher and student local dynamic prototypes. In the teacher network, the teacher's local dynamic prototypes are used to perform momentum updates on a preset global multi-scale prototype set to generate a global anchor prototype. In the student network, the student's local dynamic prototypes are directly used as features to be aligned. Global pooling is performed on the feature maps at each level of both the teacher and student networks to extract the teacher's global context prototype and the student's global context prototype, which incorporates information from the entire image environment. The following prototype is used; within the same feature level, a distillation loss function is constructed based on the teacher's global anchor prototype, the student's local dynamic prototype, the teacher's global context prototype, and the student's global context prototype. This function is then combined with the basic task loss of object detection for backpropagation to update the parameters of the student network, resulting in the trained student network. Since different levels of the feature pyramid contain target-specific scale semantic information, and the macroscopic background of real complex scenes contains rich symbiotic environment prior information, fully mining the target-specific semantic features at different scale levels and the contextual environment prior features of the entire image, as well as effectively integrating globally stable teacher knowledge anchors across batches, is beneficial to improving the target detection accuracy and generalization robustness of lightweight detectors in complex and variable scenes. Therefore, this scheme can solve the technical problem that existing target detection knowledge distillation methods in complex scenes suffer from insufficient detection accuracy and generalization ability of lightweight student networks due to ignoring the symbiotic relationship between the target and the macroscopic background, semantic confusion of multi-scale features, and instability of knowledge anchors.

[0007] Optionally, based on the scale of the ground truth bounding box, it is mapped to the corresponding feature pyramid level. Then, at the mapped level, the corresponding target region features are extracted based on the positional information of the ground truth bounding box, and aggregated by category to obtain the teacher's local dynamic prototype and the student's local dynamic prototype. This includes: using a region of interest alignment operation, extracting the corresponding target region feature map at the corresponding feature pyramid level based on the ground truth bounding box; performing global average pooling on the target region feature map in spatial dimension to convert it into a one-dimensional feature vector; and averaging all one-dimensional feature vectors belonging to the same level and category within the current batch to generate the corresponding category's local dynamic prototype at the corresponding level.

[0008] Optionally, the mean of all one-dimensional feature vectors belonging to the same level and category within the current batch is taken to generate a local dynamic prototype of the corresponding category at the corresponding level. The calculation formula is as follows: ; in, Indicates hierarchy, Indicates category, This indicates that the current batch belongs to the hierarchy. And the category is The total number of targets; For the first The categories are The goal is at the level The feature map of the target region obtained by performing region of interest alignment operation; This indicates global average pooling.

[0009] Optionally, on the teacher network side, the momentum of a pre-defined global multi-scale prototype set is updated using the teacher's local dynamic prototype to generate the teacher's global anchor prototype, including: Pre-set and maintain cross-batch feature storage space on the teacher's network terminal to store feature centers of each category under each level; Using the local dynamic prototypes of teachers extracted in the current batch, the corresponding items in the global multi-scale prototype set are updated with momentum. The update formula is as follows: ; in, The feature pyramid hierarchy is as follows And the category is The prototype of the teacher's global anchor point. As a local dynamic prototype for teachers, The preset momentum coefficient; Among them, the teacher global anchor prototype is used as a stable supervisory anchor during the student network training process, and does not participate in gradient backpropagation during the update process.

[0010] Optionally, global pooling is performed on the feature maps of each level of the teacher network and the student network respectively to extract the global context prototypes of the teacher and students that fuse the environmental information of the entire image. This includes: obtaining the multi-scale feature maps output by the teacher network and the student network at each level of the feature pyramid without distinguishing between foreground objects and background regions; performing global average pooling across spatial dimensions on the multi-scale feature maps; and reducing the pooled feature maps to one-dimensional vectors, which are used as the global context prototypes of the teacher and students at that level, respectively. The extraction formula is as follows: ; in, This represents the extracted global context prototype; Representing the feature pyramid hierarchy Upper spatial coordinates are The feature activation value; and These represent the width and height of the multi-scale feature map at this level, respectively.

[0011] Optionally, the semantic similarity between the teacher's global anchor prototype and each spatial location in the teacher's network feature map is calculated to generate a prototype response map. The calculation formula is: ; in, Indicates mapping to the first A set of target categories at each feature level; The teacher network is in the first Spatial location in each feature level Feature activation values ​​at; The prototype response temperature coefficient; Represents the cosine similarity function; Based on the truly labeled bounding box in the first... Spatial projection at each feature level generates a foreground region mask. The foreground region mask is then fused with the prototype response map to obtain a distillation weight map guided by the prototype response map. Distillation weighting diagram The calculation formula is: ; in, These are the basic response weighting coefficients, used to preserve the basic distillation constraints at low-response locations within the foreground region; Calculate the difference in feature orientations between the teacher network and the student network at each spatial location within the corresponding feature level. Differences in characteristic directions The calculation formula is: ; By applying weighted directional consistency constraints to the feature maps of the teacher and student networks at corresponding feature levels using distillation weight maps, a guided feature distillation constraint guided by the prototype response map is obtained. The calculation formula is: ; in, Indicates the first Distillation weights for each feature level To prevent constants with a denominator of zero.

[0012] Optionally, within the same feature level, a distillation loss function is constructed based on the teacher's global anchor prototype, the student's local dynamic prototype, the teacher's global context prototype, the student's global context prototype, and guided feature distillation constraints. This function is then combined with the basic task loss of object detection for backpropagation to update the parameters of the student network, resulting in the trained student network. This includes: within each level, constructing an intra-class prototype alignment loss function based on the cosine similarity between the student's local dynamic prototype and the corresponding teacher's global anchor prototype, by minimizing the directional difference between the two; within the same level, calculating the relative distance matrix between different categories of student prototypes and the corresponding teacher prototypes... The relative distance matrix between category anchor prototypes is calculated, and an inter-class topological relationship alignment loss function is constructed by aligning the two distance matrices. The distance relationship between category prototypes and global context prototypes in the student and teacher sides is calculated separately, and the absolute difference of this relationship between teachers and students is measured to construct an environment relationship difference loss function. The intra-class prototype alignment loss function, inter-class topological relationship alignment loss function, environment relationship difference loss function, guided feature distillation constraint, and the student network's own basic detection task loss function are weighted and summed to obtain the total loss function. Backpropagation is performed based on the total loss function to update the parameters of the student network, resulting in the trained student network.

[0013] Optionally, the intra-class prototype alignment loss function The calculation formula is as follows: ; in, Represents the cosine similarity function; The formula for calculating the inter-class topological alignment loss function is as follows: ; ; ; in, and Representing levels Next category With category Distance of the characteristic spatial relationship between them; This represents the smoothed L1 loss function; The formula for calculating the environmental relationship difference loss function is as follows: ; ; ; in, and These are the global context prototypes for students and teachers, respectively. This represents the sum of the distances between the category prototype and the global context prototype in the student's context. This represents the distance relationship between the category prototype and the global context prototype in the teacher's end. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0015] Figure 1 This is a flowchart of a knowledge distillation method for target detection based on hierarchical perception category prototype alignment provided in one embodiment of this application; Figure 2 This is a schematic diagram of a lightweight object detection knowledge distillation method based on hierarchical perception category prototype alignment provided in one embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0017] With the breakthroughs achieved by deep neural networks in object detection tasks, high-precision detection models have been widely applied in fields such as autonomous driving, remote sensing monitoring, and intelligent surveillance. However, high-performance detection networks are usually accompanied by huge computational overhead and a large number of parameters, making them difficult to deploy in real time on edge embedded devices with limited computing power. Knowledge distillation, as a mainstream model lightweighting technique, effectively improves object detection accuracy by transferring the rich knowledge contained in large teacher networks to lightweight student networks, while maintaining the advantages of lightweight models and high inference speed.

[0018] In knowledge distillation research for object detection, effectively transferring the core knowledge of the teacher network—including its high-order semantics, discriminative ability, and topological structure in a high-dimensional feature space—to the student network is crucial for improving distillation results. However, in complex scenarios (e.g., urban areas with numerous features or remote sensing imagery), existing technologies still suffer from the following limitations: First, existing object detection distillation methods typically ignore the symbiotic semantic relationships between individual targets and their macroscopic background in real-world complex scenes. Isolating and aligning foreground targets and directly removing background regions, or relying on cumbersome feature clustering algorithms to forcibly extract the background, leads to the severe fragmentation and loss of crucial spatial interaction relationships and environmental priors during transfer learning. This results in insufficient scene understanding and generalization ability for complex and ever-changing scenarios.

[0019] Secondly, traditional feature map-based distillation methods often neglect the scale decoupling characteristics of feature pyramids in detection tasks. Forcibly aggregating target features at different scales or performing coarse-grained alignment at a single level can lead to severe semantic confusion and collapse. At the same time, existing methods often impose indiscriminate constraints on feature map regions, making it difficult to highlight key semantic response locations related to the target. This can easily introduce redundant interference from background and weakly related regions, reducing the model's detection accuracy for extreme-sized targets (extremely small or extremely large targets).

[0020] Furthermore, in category-based prototype-based knowledge distillation, the prototypes extracted by the teacher network should serve as "standard knowledge anchors" for student network learning. However, most existing methods directly use local batch features with strong randomness and significant individual differences as benchmark anchors for knowledge transfer, resulting in drastic fluctuations and fragmentation of the extracted knowledge anchors. This makes it difficult for student networks to capture stable and essential feature topologies, leading to a significant decrease in the quality of high-order semantic knowledge transfer.

[0021] The difficulty in solving the above problems and shortcomings lies in the fact that knowledge distillation models for object detection generally only use single-level or mixed feature maps for coarse-grained alignment, and there is no suitable mechanism for accurately decoupling and aligning multi-scale semantic features. At the same time, most current knowledge distillation models rely on local batch data to extract fluctuating knowledge anchors and process foreground targets in isolation. How to further effectively construct globally stable knowledge anchors across batches and integrate the macro-environmental features of the whole map is worthy of further research.

[0022] The significance of solving the above problems and defects lies in the fact that different levels of the feature pyramid contain target-specific scale semantic information, the macro background of real and complex scenes contains rich symbiotic environment prior information, fully explore the target-specific semantic features at different scale levels and the contextual environment prior features of the whole image, and effectively integrate cross-batch globally stable teacher knowledge anchors, which is conducive to improving the target detection accuracy and generalization robustness of lightweight detectors in complex and variable scenes.

[0023] In view of this, embodiments of this application propose a knowledge distillation method for object detection based on hierarchical perception category prototype alignment, aiming to solve the technical problems of insufficient detection accuracy and generalization ability of lightweight student networks in complex scenarios due to the neglect of the symbiotic relationship between the target and the macro background, the semantic confusion of multi-scale features, and the instability of knowledge anchors.

[0024] One embodiment of this application proposes a knowledge distillation method for object detection based on hierarchical awareness and category prototype alignment, applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments will use a server as an example for description. The implementation details of the knowledge distillation method for object detection based on hierarchical awareness and category prototype alignment proposed in this embodiment will be described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0025] This embodiment proposes a lightweight object detection knowledge distillation method based on hierarchical perception category prototype alignment. Its core lies in efficiently transferring the rich semantic knowledge and scene understanding capabilities contained in a large teacher network to a lightweight student network through hierarchical perception prototype extraction, asymmetric prototype update mechanism, and global context alignment. This significantly improves the object detection accuracy in complex scenes without increasing the inference overhead of the student network.

[0026] The specific process of the object detection knowledge distillation method based on hierarchical perception and category prototype alignment proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 101: Obtain the training image batch and the corresponding ground truth bounding boxes.

[0027] The information for the actual bounding box includes scale and location information.

[0028] For example, a batch of training images can be obtained from an object detection dataset. Each training image is accompanied by real annotation information, including the category label of each object of interest in the image and its corresponding bounding box. The bounding box information may include its scale (such as width and height) and location information (such as the coordinates of the center point or the coordinates of the top left and bottom right corners).

[0029] Step 102: Input the training images in batches into the teacher network and the student network to be trained, respectively, to extract multi-scale feature maps of each level of the feature pyramid in the teacher network and the student network.

[0030] For example, training images can be batched and simultaneously input into both the teacher network and the student network to be trained. The teacher network is pre-trained, high-performance but has many parameters, while the student network is lightweight but has lower accuracy. Multi-scale feature maps are extracted from the feature pyramids (FPNs) of both networks at each level. These feature maps at different levels naturally correspond to target semantic information at different scales. Higher-level feature maps contain richer semantic information and are suitable for detecting large targets, while lower-level feature maps retain finer spatial details and are suitable for detecting small targets.

[0031] Step 103: Based on the scale of the real labeled bounding box, map it to the corresponding feature pyramid level, and on the mapped level, extract the corresponding target region features based on the position information of the real labeled bounding box, and aggregate them by category to obtain the teacher local dynamic prototype and the student local dynamic prototype.

[0032] In one possible embodiment, step 103 includes: using the region of interest alignment operation, extracting the corresponding target region feature map at the corresponding feature pyramid level based on the ground truth bounding box; performing global average pooling on the target region feature map in spatial dimension to convert it into a one-dimensional feature vector; and taking the average of all one-dimensional feature vectors belonging to the same level and the same category in the current batch to generate a local dynamic prototype of the corresponding category at the corresponding level.

[0033] For example, such as Figure 2 As shown, firstly, each ground truth bounding box can be mapped to the corresponding level in the feature pyramid best suited for detecting targets at that scale, based on its scale. For example, smaller targets are mapped to lower (higher resolution) levels, while larger targets are mapped to higher (stronger semantics) levels. Next, at the mapped feature levels, a region of interest alignment operation is used to extract the corresponding target region features from the feature maps of the teacher and student networks, respectively, based on the precise location information of the bounding boxes.

[0034] Next, global average pooling is performed on the spatial dimension of each extracted target region feature map, compressing it into a one-dimensional feature vector. Finally, aggregation is performed by category and level: for the current training batch, all features belonging to the same feature level are aggregated. and the same category The average value of the target feature vectors is used to obtain the feature prototype of this type of target at this level. This prototype is dynamically calculated from the current batch of data, and is therefore called the local dynamic prototype of the teacher and the local dynamic prototype of the student.

[0035] In one possible embodiment, the mean value is taken of all one-dimensional feature vectors belonging to the same level and category within the current batch to generate a local dynamic prototype of the corresponding category at the corresponding level. The calculation formula is as follows: ; in, Indicates hierarchy, Indicates category, This indicates that the current batch belongs to the hierarchy. And the category is The total number of targets; For the first The categories are The goal is at the level The feature map of the target region obtained by performing region of interest alignment operation; This indicates global average pooling.

[0036] Step 104: On the teacher network side, the teacher's local dynamic prototype is used to perform momentum update on the preset global multi-scale prototype set to generate the teacher's global anchor prototype. On the student network side, the student's local dynamic prototype is directly used as the feature to be aligned.

[0037] For example, to address the problem of excessive fluctuations in knowledge anchors in existing methods, an asymmetric prototype update mechanism can be introduced. For instance... Figure 2 As shown, on the teacher network side, a learnable global multi-scale prototype set is pre-set and maintained across training batches, which is for each level. Each category below A stable feature center, called the teacher's global anchor prototype, is stored. In each training iteration, the corresponding items in the global prototype set are smoothly updated using an exponential moving average strategy, leveraging the teacher's local dynamic prototype.

[0038] In one possible embodiment, step 104 includes: presetting and maintaining a cross-batch feature storage space on the teacher network terminal to store feature centers of each category under each level; Using the local dynamic prototypes of teachers extracted in the current batch, the corresponding items in the global multi-scale prototype set are updated with momentum. The update formula is as follows: ; in, The feature pyramid hierarchy is as follows And the category is The prototype of the teacher's global anchor point. As a local dynamic prototype for teachers, The preset momentum coefficient; Among them, the teacher global anchor prototype is used as a stable supervisory anchor during the student network training process, and does not participate in gradient backpropagation during the update process.

[0039] For example, This is the teacher global anchor prototype that needs updating. This is a local dynamic prototype of the teacher obtained from the current batch of calculations. This is the momentum coefficient (for example, it can be set to 0.99). This update process does not participate in gradient backpropagation, thus ensuring... It remains stable during training, serving as a reliable knowledge anchor for students' online learning.

[0040] On the student network side, the student's local dynamic prototype is directly used as the feature to be aligned, without similar momentum updates. This asymmetric design keeps the teacher prototype stable while the student prototype changes dynamically, guiding the features of the student network to converge toward the stable feature space established by the teacher network.

[0041] Step 105: Perform global pooling operation on each level feature map of the teacher network and student network respectively, and extract the global context prototype of the teacher and the global context prototype of the student by fusing the environmental information of the whole graph.

[0042] Understandably, this is to compensate for the shortcomings of traditional methods that ignore the relationship between the target and its context. For example... Figure 2 As shown, global average pooling is performed on the complete multi-scale feature maps (regardless of foreground and background) output by the teacher and student networks at each layer. This operation aggregates all semantic information of the entire image at that layer, resulting in a one-dimensional feature vector. This vector contains the macroscopic scene semantics and environmental distribution priors of the image, and is called the global context prototype.

[0043] In one possible embodiment, step 105 includes: acquiring multi-scale feature maps output by the teacher network and student network at each level of the feature pyramid without distinguishing between foreground targets and background regions. Perform global average pooling across spatial dimensions on multi-scale feature maps; The pooled feature map is reduced to a one-dimensional vector, which serves as the teacher's global context prototype and the student's global context prototype at this level, respectively. The extraction formula is as follows: ; in, This represents the extracted global context prototype; Representing the feature pyramid hierarchy Upper spatial coordinates are The feature activation value; and These represent the width and height of the multi-scale feature map at this level, respectively.

[0044] Step 106: Based on the teacher global anchor prototype, construct the prototype response map, and use the prototype response map to perform guided feature distillation constraints on the feature maps of corresponding levels of the teacher network and student network.

[0045] Record No. At each feature pyramid level, the feature map output by the teacher network is as follows: The feature map output by the student network is ; Calculate the semantic similarity between the teacher's global anchor prototype and each spatial location in the teacher's network feature map, and generate a prototype response map. The calculation formula is: ; in, Indicates mapping to the first A set of target categories at each feature level; The teacher network is in the first Spatial location in each feature level Feature activation values ​​at; The prototype response temperature coefficient; Represents the cosine similarity function; Based on the truly labeled bounding box in the first... Spatial projection at each feature level generates a foreground region mask. The foreground region mask is then fused with the prototype response map to obtain a distillation weight map guided by the prototype response map. Distillation weighting diagram The calculation formula is: ; in, These are the basic response weighting coefficients, used to preserve the basic distillation constraints at low-response locations within the foreground region; Calculate the difference in feature orientations between the teacher network and the student network at each spatial location within the corresponding feature level. Differences in characteristic directions The calculation formula is: ; By applying weighted directional consistency constraints to the feature maps of the teacher and student networks at corresponding feature levels using distillation weight maps, a guided feature distillation constraint guided by the prototype response map is obtained. The calculation formula is: ; in, Indicates the first Distillation weights for each feature level To prevent constants with a denominator of zero.

[0046] Step 107: Within the same feature level, construct a distillation loss function based on the teacher global anchor prototype, student local dynamic prototype, teacher global context prototype, student global context prototype, and guided feature distillation constraints. Then, perform backpropagation in conjunction with the basic task loss of object detection to update the parameters of the student network and obtain the trained student network.

[0047] In one possible embodiment, step 107 includes: within each level, constructing an intra-class prototype alignment loss function based on the cosine similarity between the student's local dynamic prototype and the corresponding teacher's global anchor prototype, and minimizing their directional differences; within the same level, calculating the relative distance matrix between different categories of prototypes on the student side and the relative distance matrix between corresponding categories of anchor prototypes on the teacher side, and constructing an inter-class topological relationship alignment loss function by aligning the two distance matrices; calculating the sum of distance relationships between category prototypes and global context prototypes on both the student and teacher sides, and measuring the absolute difference in this relationship between teachers and students, and constructing an environmental relationship difference loss function; weighting and summing the intra-class prototype alignment loss function, the inter-class topological relationship alignment loss function, the environmental relationship difference loss function, the guided feature distillation loss function, and the student network's own basic detection task loss function to obtain a total loss function; performing backpropagation based on the total loss function to update the parameters of the student network, and obtaining the trained student network.

[0048] For example, at the same feature level Internally, by comprehensively utilizing the four types of prototypes generated above—teacher global anchor prototype, student local dynamic prototype, teacher global context prototype, and student global context prototype—a multi-layered distillation loss function is constructed to comprehensively supervise the learning of the student network. The total distillation loss function consists of four parts: intra-class prototype alignment loss function, inter-class topological relationship alignment loss function, environmental relationship difference loss function, and guided feature distillation loss function. Then, it is jointly optimized with the student network's own basic detection task losses (such as classification loss and bounding box regression loss).

[0049] For example, the intra-class prototype alignment loss function The student network is forced to learn the essential representation of each category in the feature space. The cosine similarity between the student's local dynamic prototype and the corresponding teacher's global anchor prototype is calculated, and alignment is achieved by minimizing the directional difference between the two.

[0050] Intraclass prototype alignment loss function The calculation formula is as follows: ; in, Represents the cosine similarity function; For example, the inter-class topological relationship alignment loss function The goal is to enable the student network to learn the overall structure of the teacher network's feature space. Cosine similarity is calculated pairwise between different class prototypes within the same level at both the student and teacher ends, forming a relation matrix. By minimizing the difference between the two relation matrices, the student network mimics the relative distance relationships between classes in the teacher network.

[0051] The formula for calculating the inter-class topological alignment loss function is as follows: ; ; ; in, and Representing levels Next category With category Distance of the characteristic spatial relationship between them; This represents the smoothed L1 loss function; For example, the environment-relationship difference loss function enables the student network to correctly associate the learning objective with the global context. The cosine similarity distance between each category prototype and its corresponding level's global context prototype is calculated separately for both the teacher and student ends. By aligning these two distances, the student network understands the prior occurrence of the objective in a specific context.

[0052] The formula for calculating the environmental relationship difference loss function is as follows: ; ; ; in, and These are the global context prototypes for students and teachers, respectively. This represents the sum of the distances between the category prototype and the global context prototype in the student's context. This represents the distance relationship between the category prototype and the global context prototype in the teacher's end.

[0053] Final total loss function The weighted sum of the losses from the above components: ; in, , , , , These are the weight coefficients for each loss term. The total loss function is minimized using the backpropagation algorithm. The parameters of the student network are updated, while the parameters of the teacher network remain constant during this process.

[0054] For example, in one implementation example, stochastic gradient descent can be used to train the student network. For instance, on the COCO dataset, ResNet-50 is used as the teacher network and MobileNetV2 as the student network, with a total of 12 training epochs, a batch size of 2, and an initial learning rate of 0.0025. By jointly optimizing the above loss, the network can converge effectively, and after saving the optimal parameters, a lightweight object detection model reinforced by knowledge distillation is obtained.

[0055] This application proposes a knowledge distillation method for target detection based on hierarchical perception category prototype alignment. The method involves acquiring batches of training images and their corresponding ground truth bounding boxes. These batches are input into a teacher network and a student network to be trained, respectively, to extract multi-scale feature maps at each level of the feature pyramid in both networks. Based on the scale of the ground truth bounding boxes, they are mapped to the corresponding feature pyramid levels. At the mapped levels, the corresponding target region features are extracted based on the positional information of the ground truth bounding boxes and aggregated by category to obtain teacher and student local dynamic prototypes. In the teacher network, the teacher's local dynamic prototypes are used to perform momentum updates on a preset global multi-scale prototype set to generate a global anchor prototype. In the student network, the student's local dynamic prototypes are directly used as features to be aligned. Global pooling is performed on the feature maps at each level of both the teacher and student networks to extract the teacher's global context prototype and the student's global context prototype, which incorporates information from the entire image environment. The following prototype is used; within the same feature level, a distillation loss function is constructed based on the teacher's global anchor prototype, the student's local dynamic prototype, the teacher's global context prototype, and the student's global context prototype. This function is then combined with the basic task loss of object detection for backpropagation to update the parameters of the student network, resulting in the trained student network. Since different levels of the feature pyramid contain target-specific scale semantic information, and the macroscopic background of real complex scenes contains rich symbiotic environment prior information, fully mining the target-specific semantic features at different scale levels and the contextual environment prior features of the entire image, as well as effectively integrating globally stable teacher knowledge anchors across batches, is beneficial to improving the target detection accuracy and generalization robustness of lightweight detectors in complex and variable scenes. Therefore, this scheme can solve the technical problem that existing target detection knowledge distillation methods in complex scenes suffer from insufficient detection accuracy and generalization ability of lightweight student networks due to ignoring the symbiotic relationship between the target and the macroscopic background, semantic confusion of multi-scale features, and instability of knowledge anchors.

[0056] The technical effects of this application will be explained in detail below with reference to simulation experiments.

[0057] In the simulation experiments, the software platform used in this application is: Linux 22.04 operating system, Python 3.8, and PyTorch 2.4. The Dior remote sensing image dataset used in the simulation experiments of this application includes an image size of [missing information]. As shown in Table 1, this dataset contains 20 categories, and also shows the division between training images and instances, and between test images and instances.

[0058] Table 1. Classification of the Dior dataset and the division of training images and instances, and test images and instances.

[0059] Experimental Content and Result Analysis: To verify the good detection accuracy of this application, four evaluation indicators were used: mean average accuracy (mAP), mean average accuracy for small targets (mAP_s), mean average accuracy for medium targets (mAP_m), and mean average accuracy for large targets (mAP_l). The mean average accuracy (mAP) represents the average accuracy across multiple categories. The mean average accuracy for small targets (mAP_s) represents the average accuracy for small targets. The mean average accuracy for medium targets (mAP_m) represents the average accuracy for medium targets. The mean average accuracy for large targets (mAP_l) represents the average accuracy for large targets.

[0060] In the simulation experiments of this application, the teacher model used ResNet-50 and the student model used ResNet-18. The total number of training epochs was set to 12, the initial learning rate was 0.0025, and the batch size was 2. The Dior dataset was tested using this application and existing comparison algorithms, and the results are shown in Table 2.

[0061] Table 2. Results of the detection of the Dior dataset by this application and existing comparison algorithms.

[0062] As can be seen from Table 2, compared with other algorithms, the mAP, mAP_s, mAP_m and mAP_l of this application reached 52.6%, 11.3%, 33.5% and 61.0% respectively. Among them, mAP was improved by 3.2% compared with the student model, which proves that this application can better improve the performance of the student model.

[0063] Combining all the above technical solutions, the advantages and positive effects of this application are as follows: The lightweight object detection knowledge distillation method based on hierarchical perception category prototype alignment provided in this application fully mines the exclusive scale semantic information of different levels in the feature pyramid by utilizing hierarchical perception feature extraction and scale mapping mechanism, effectively avoiding the semantic collapse problem in multi-scale feature transfer; at the same time, it uses an asymmetric prototype update strategy to provide stable knowledge anchors with global consistency across batches for the distillation process, realizing efficient and lossless transfer of high-order semantic topology structures; in addition, by introducing a low-computational-cost global context alignment mechanism, it supplements the prior semantic information of the relationship between the target and the macro environment, which is conducive to significantly improving the detection accuracy and generalization robustness of the lightweight detector in complex and variable scenarios while maintaining the advantages of lightweight model.

[0064] Simulation experiments show that the proposed lightweight object detection knowledge distillation method based on hierarchical perception category prototype alignment, by introducing hierarchical perception mapping, asymmetric prototype update mechanism and global context alignment strategy, can fully explore the potential exclusive semantic features of multi-scale targets and the symbiotic relationship between targets and the macro environment, effectively improving the detection accuracy of lightweight object detection networks in complex scenes.

[0065] The steps described above are for clarity only. In implementation, they can be combined into one step, or some steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0066] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A knowledge distillation method for target detection based on hierarchical perception and category prototype alignment, characterized in that, The method includes: Obtain batches of training images and their corresponding ground truth bounding boxes; the ground truth bounding boxes contain information on scale and location. The training images are input into the teacher network and the student network to be trained in batches to extract multi-scale feature maps of each level of the feature pyramid in the teacher network and the student network. Based on the scale of the real labeled bounding boxes, they are mapped to the corresponding feature pyramid level. On the mapped level, the corresponding target region features are extracted based on the position information of the real labeled bounding boxes, and aggregated by category to obtain the local dynamic prototype of the teacher and the local dynamic prototype of the student. On the teacher network side, the teacher's local dynamic prototype is used to perform momentum updates on the preset global multi-scale prototype set to generate the teacher's global anchor prototype. On the student network side, the student's local dynamic prototype is directly used as the feature to be aligned. Global pooling is performed on the feature maps of each level of the teacher network and the student network respectively, and the global context prototypes of the teacher and the student are extracted by fusing the environmental information of the whole graph. Based on the teacher global anchor prototype, a prototype response graph is constructed, and the prototype response graph is used to perform guided feature distillation constraints on the feature graphs of corresponding levels of the teacher network and student network. Within the same feature level, a distillation loss function is constructed based on the teacher's global anchor prototype, the student's local dynamic prototype, the teacher's global context prototype, the student's global context prototype, and the guided feature distillation constraint. This function is then combined with the basic task loss of object detection for backpropagation to update the parameters of the student network, resulting in the trained student network.

2. The method according to claim 1, characterized in that, The scale based on the ground truth bounding boxes is mapped to the corresponding feature pyramid level. At the mapped level, the corresponding target region features are extracted based on the positional information of the ground truth bounding boxes, and then aggregated by category to obtain the teacher's local dynamic prototype and the student's local dynamic prototype, including: By using the region of interest alignment operation, the corresponding target region feature map is extracted at the corresponding feature pyramid level based on the ground-labeled bounding box. Perform global average pooling on the spatial dimension of the target region feature map to convert it into a one-dimensional feature vector; Take the average of all one-dimensional feature vectors belonging to the same level and category within the current batch to generate a local dynamic prototype of the corresponding category at the corresponding level.

3. The method according to claim 2, characterized in that, The process involves averaging all one-dimensional feature vectors belonging to the same level and category within the current batch to generate a local dynamic prototype for the corresponding category at the corresponding level. The calculation formula is as follows: ； in, Indicates hierarchy, Indicates category, This indicates that the current batch belongs to the hierarchy. And the category is The total number of targets; For the first The categories are The goal is at the level The feature map of the target region obtained by performing region of interest alignment operation; This indicates global average pooling.

4. The method according to claim 3, characterized in that, On the teacher network side, the momentum of a pre-set global multi-scale prototype set is updated using the teacher's local dynamic prototype to generate the teacher's global anchor point prototype, including: Pre-set and maintain cross-batch feature storage space on the teacher's network terminal to store feature centers of each category under each level; Using the local dynamic prototypes of teachers extracted in the current batch, the momentum of the corresponding items in the global multi-scale prototype set is updated to generate the global anchor point prototype of teachers.

5. The method according to claim 4, characterized in that, The momentum update formula is: ； in, The feature pyramid hierarchy is as follows And the category is The prototype of the teacher's global anchor point. As a local dynamic prototype for teachers, The preset momentum coefficient; Among them, the teacher global anchor prototype is used as a stable supervisory anchor during the student network training process, and does not participate in gradient backpropagation during the update process.

6. The method according to claim 5, characterized in that, The step of performing global pooling operations on the feature maps of each level of the teacher network and the student network respectively, and extracting the global context prototypes of the teacher and students by fusing the environmental information of the entire graph, includes: Without distinguishing between foreground targets and background regions, multi-scale feature maps output by the teacher network and student network at each level of the feature pyramid are obtained. Perform global average pooling across spatial dimensions on multi-scale feature maps; The pooled feature map is reduced to a one-dimensional vector, which serves as the teacher's global context prototype and the student's global context prototype at this level, respectively. The extraction formula is as follows: ； in, This represents the extracted global context prototype; Representing the feature pyramid hierarchy Upper spatial coordinates are The feature activation value; and These represent the width and height of the multi-scale feature map at this level, respectively.

7. The method according to claim 6, characterized in that, Record No. At each feature pyramid level, the feature map output by the teacher network is as follows: The feature map output by the student network is ; The process involves constructing a prototype response map based on the teacher's global anchor prototype, and using this prototype response map to perform guided feature distillation constraints on the feature maps of corresponding layers in the teacher and student networks. This includes: Calculate the semantic similarity between the teacher's global anchor prototype and each spatial location in the teacher's network feature map, and generate a prototype response map. The calculation formula is: ； in, Indicates mapping to the first A set of target categories at each feature level; The teacher network is in the first Spatial location in each feature level Feature activation values ​​at; The prototype response temperature coefficient; Represents the cosine similarity function; Based on the truly labeled bounding box in the first... Spatial projection at each feature level generates a foreground region mask. The foreground region mask is then fused with the prototype response map to obtain a distillation weight map guided by the prototype response map. Distillation weighting diagram The calculation formula is: ； in, These are the basic response weighting coefficients, used to preserve the basic distillation constraints at low-response locations within the foreground region; Calculate the difference in feature orientations between the teacher network and the student network at each spatial location within the corresponding feature level. Differences in characteristic directions The calculation formula is: ； By applying weighted directional consistency constraints to the feature maps of the teacher and student networks at corresponding feature levels using distillation weight maps, a guided feature distillation constraint guided by the prototype response map is obtained. The calculation formula is: ； in, Indicates the first Distillation weights for each feature level To prevent constants with a denominator of zero.

8. The method according to claim 7, characterized in that, Within the same feature level, a distillation loss function is constructed based on the teacher's global anchor prototype, the student's local dynamic prototype, the teacher's global context prototype, the student's global context prototype, and guided feature distillation constraints. This function is then combined with the basic task loss for object detection for backpropagation to update the parameters of the student network, resulting in the trained student network, including: Within each level, the cosine similarity between the student's local dynamic prototype and the corresponding teacher's global anchor prototype is used to construct an intra-class prototype alignment loss function by minimizing the directional difference between the two. Within the same level, the relative distance matrix between different categories of prototypes on the student side and the relative distance matrix between corresponding category anchor prototypes on the teacher side are calculated respectively. By aligning the two distance matrices, an inter-class topological relationship alignment loss function is constructed. Calculate the distance relationship between the category prototype and the global context prototype in the student and teacher sides respectively, measure the absolute difference of this relationship between teachers and students, and construct the environmental relationship difference loss function; The total loss function is obtained by weighted summing of the intra-class prototype alignment loss function, the inter-class topological relationship alignment loss function, the environmental relationship difference loss function, the guided feature distillation constraint, and the student network's own basic detection task loss function. Backpropagation is performed based on the total loss function to update the parameters of the student network, resulting in the trained student network.

9. The method according to claim 8, characterized in that, Intraclass prototype alignment loss function The calculation formula is as follows: ； The formula for calculating the inter-class topological alignment loss function is as follows: ； ； ； in, and Representing levels Next category With category Distance of the characteristic spatial relationship between them; This represents the smoothed L1 loss function; The formula for calculating the environmental relationship difference loss function is as follows: ； ； ； in, and These are the global context prototypes for students and teachers, respectively. This represents the sum of the distances between the category prototype and the global context prototype in the student's context. This represents the distance relationship between the category prototype in the teacher's side and the global context prototype.

10. The method according to claim 9, characterized in that, Total loss function The calculation formula is as follows: ； in, , , , , These are the weighting coefficients for each loss term.

Images