A neural network model distillation method and device, and a storage medium

Abstract

The present disclosure relates to a neural network model distillation method, device and storage medium. The neural network model distillation method comprises: determining a target image, and obtaining an object condition feature of the target image, the object condition feature being artificially input and representing feature information of a target object; performing feature detection on the target image based on a teacher model to obtain a teacher model feature map, and performing feature detection on the target image based on an initial student model to obtain a student model feature map; determining a correlation between an object feature of each pixel position in the teacher model feature map and the object condition feature, and determining an attention distribution of the teacher model based on the correlation; and training the initial student model based on the attention distribution of the teacher model and a feature difference degree between the teacher model feature map and the student model feature map to obtain a target student model. The present disclosure can obtain a student model with high target detection accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of image recognition technology, and in particular to a neural network model distillation method, apparatus and storage medium. Background Technology

[0002] Knowledge distillation is a method to improve network performance by using a pre-trained large model (teacher model) to teach a model to be trained (student model), thereby improving model performance. In related technologies, knowledge distillation is also widely used in image recognition to improve the sampling and detection results for images.

[0003] In related technologies, the easily identifiable regions learned by the model network may not necessarily match human observation and cognition, which limits the detection performance of the model. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a neural network model distillation method, apparatus and storage medium.

[0005] According to a first aspect of the present disclosure, a neural network model distillation method is provided, comprising:

[0006] A target image is identified, and the object condition features of the target image are obtained. The object condition features are manually input feature information that characterizes the target object. Feature detection is performed on the target image based on a teacher model to obtain a teacher model feature map, and feature detection is performed on the target image based on an initial student model to obtain a student model feature map. The correlation between the object features at each pixel position in the teacher model feature map and the object condition features is determined, and the attention distribution of the teacher model is determined based on the correlation. Based on the attention distribution of the teacher model and the feature difference between the teacher model feature map and the student model feature map, the initial student model is trained to obtain the target student model.

[0007] In one embodiment, determining the correlation between object features at each pixel position in the teacher model feature map and the object condition features includes: decomposing the object features at each pixel position in the teacher model feature map into multiple first sub-feature spaces; performing feature weighting on the multiple first sub-feature spaces based on different weights for each pixel position in the teacher model feature map to obtain feature-weighted object features; determining the correlation between the feature-weighted object features and the object condition features, and obtaining the first sub-feature space correlation for each pixel position based on the correlation between the feature-weighted object features and the object condition features; and normalizing the first sub-feature space correlation for each pixel position in the teacher model feature map to obtain the correlation between the object features at each pixel position and the object condition features.

[0008] In one embodiment, training the initial student model to obtain a target student model based on the attention distribution of the teacher model and the feature difference between the feature map of the teacher model and the feature map of the student model includes: optimizing the attention distribution of the teacher model; and training the initial student model to obtain the target student model based on the optimized attention distribution and the feature difference between the feature map of the teacher model and the feature map of the student model.

[0009] In one embodiment, optimizing the attention distribution of the teacher model includes: decomposing the object features at each pixel position in the feature map of the teacher model into multiple second sub-feature spaces; weighting the multiple second sub-feature spaces based on the attention distribution of the teacher model; controlling the teacher model to perform target detection based on the weighted features; and adjusting the attention distribution of the teacher model based on the target detection results and object condition features, so that the teacher model obtains a target detection result consistent with the object condition features.

[0010] In one embodiment, the first sub-feature space and the second sub-feature space use different decomposition methods; wherein, the decomposition methods include key-value decomposition and content decomposition.

[0011] In one embodiment, obtaining a target detection result consistent with the object's conditional features includes obtaining a target detection result consistent with one or a combination of the category features, position features, and scale features corresponding to the object's conditional features.

[0012] In one embodiment, determining the attention distribution of the teacher model based on the correlation includes: assigning weights to each pixel position in the teacher model feature map based on the correlation between the object features and the object condition features at each pixel position in the teacher model feature map, and obtaining the attention distribution of the teacher model according to the assigned weights; wherein, for each pixel position in the teacher model feature map, the pixel position with greater correlation has a greater weight.

[0013] In one embodiment, feature detection is performed on the target image based on the teacher model to obtain a teacher model feature map, and feature detection is performed on the target image based on the initial student model to obtain a student model feature map, including: performing multi-scale feature detection on the target image using the teacher model to obtain a teacher model feature map including multi-scale features; and performing multi-scale feature detection on the target image using the student model to obtain a student model feature map including multi-scale features.

[0014] According to a second aspect of the present disclosure, a neural network model distillation apparatus is provided, comprising:

[0015] A determining unit is used to determine a target image and to determine the correlation between object features at each pixel position in the teacher model feature map and the object conditional features, and to determine the attention distribution of the teacher model based on the correlation; an acquiring unit is used to acquire the object conditional features of the target image, wherein the object conditional features are manually input and characterize the feature information of the target object; a detection unit is used to perform feature detection on the target image based on the teacher model to obtain a teacher model feature map, and to perform feature detection on the target image based on an initial student model to obtain a student model feature map; a processing unit is used to train the initial student model based on the attention distribution of the teacher model and the feature difference between the teacher model feature map and the student model feature map to obtain a target student model.

[0016] In one embodiment, the determining unit determines the correlation between the object features at each pixel position in the teacher model feature map and the object condition features in the following manner: The object features at each pixel position in the teacher model feature map are decomposed into multiple first sub-feature spaces; for each pixel position in the teacher model feature map, the multiple first sub-feature spaces are weighted according to different weights to obtain the weighted object features; the correlation between the weighted object features and the object condition features is determined, and based on the correlation between the weighted object features and the object condition features, the first sub-feature space correlation of the multiple first sub-feature spaces for each pixel position is obtained; for each pixel position in the teacher model feature map, the first sub-feature space correlation of the multiple first sub-feature spaces is normalized to obtain the correlation between the object features at each corresponding pixel position and the object condition features.

[0017] In one embodiment, the processing unit trains the initial student model to obtain a target student model based on the attention distribution of the teacher model and the feature difference between the teacher model feature map and the student model feature map in the following manner: optimizing the attention distribution of the teacher model; and training the initial student model to obtain the target student model based on the optimized attention distribution and the feature difference between the teacher model feature map and the student model feature map.

[0018] In one embodiment, the processing unit optimizes the attention distribution of the teacher model as follows: decomposes the object features at each pixel position in the feature map of the teacher model into multiple second sub-feature spaces; performs feature weighting on the multiple second sub-feature spaces based on the attention distribution of the teacher model; controls the teacher model to perform target detection based on the weighted features, and adjusts the attention distribution of the teacher model based on the target detection results and object condition features, so that the teacher model obtains a target detection result consistent with the object condition features.

[0019] In one implementation, the first sub-feature space and the second sub-feature space use different decomposition methods; wherein, the decomposition methods include key-value decomposition and content decomposition.

[0020] In one embodiment, the processing unit obtains a target detection result consistent with the object's conditional features in the following manner: obtaining a target detection result consistent with one or a combination of the category features, position features, and scale features corresponding to the object's conditional features.

[0021] In one embodiment, the determining unit determines the attention distribution of the teacher model based on the correlation as follows: based on the correlation between the object features and the object condition features at each pixel position in the feature map of the teacher model, weights are assigned to each pixel position in the feature map of the teacher model, and the attention distribution of the teacher model is obtained according to the assigned weights; wherein, the first weight corresponding to the first correlation is greater than the second weight corresponding to the second correlation, and the first correlation is greater than the second correlation.

[0022] In one embodiment, the detection unit performs feature detection on the target image based on a teacher model to obtain a teacher model feature map, and performs feature detection on the target image based on an initial student model to obtain a student model feature map: controlling the teacher model to perform multi-scale feature detection on the target image to obtain a teacher model feature map including multi-scale features; and controlling the student model to perform multi-scale feature detection on the target image to obtain a student model feature map including multi-scale features.

[0023] According to a third aspect of the present disclosure, a neural network model distillation apparatus is provided, comprising:

[0024] Processor; memory used to store processor-executable instructions;

[0025] The processor is configured to execute the neural network model distillation method described in the first aspect or any embodiment of the first aspect.

[0026] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the neural network model distillation method described in the first aspect or any embodiment of the first aspect.

[0027] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: determining a target image and obtaining object condition features of the target image. These object condition features are manually input feature information characterizing the target object. Therefore, based on the correlation between object features and object condition features at each pixel position in the teacher model feature map, the attention distribution of the teacher model is determined to be obtained through manually input object condition features. Thus, using this attention distribution for target detection can yield target detection results that are more closely aligned with human observation perspectives, thereby improving the target detection accuracy of the student model to a certain extent.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0030] Figure 1 This is a flowchart illustrating a neural network model distillation method according to an exemplary embodiment.

[0031] Figure 2 This is a flowchart illustrating another neural network model distillation method according to an exemplary embodiment.

[0032] Figure 3 This is a flowchart illustrating a method for determining the correlation between object features and object condition features at each pixel position in a teacher model feature map, according to an exemplary embodiment.

[0033] Figure 4 This is a flowchart illustrating yet another neural network model distillation method according to an exemplary embodiment.

[0034] Figure 5 This is a flowchart illustrating a method for training an initial student model according to an exemplary embodiment.

[0035] Figure 6 This is a flowchart illustrating a method for optimizing the attention distribution of a teacher model according to an exemplary embodiment.

[0036] Figure 7 This is a flowchart illustrating another method for optimizing the attention distribution of a teacher model, according to an exemplary embodiment.

[0037] Figure 8 This is a schematic diagram illustrating a process for training an initial student model according to an exemplary embodiment.

[0038] Figure 9 This is a block diagram of a neural network model distillation apparatus according to an exemplary embodiment.

[0039] Figure 10 This is a schematic diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0041] In the accompanying drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, of the embodiments of this disclosure. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0042] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0043] With the development of machine learning technology, the use of models for image processing or recognition has become increasingly common. Generally speaking, larger, deeper, and more complex models have better fitting effects and better predictive capabilities, but at the same time, they are less computationally efficient, time-consuming, and have a large number of parameters, making them unsuitable for deployment at the application layer, such as mobile devices and chip-based devices. Simpler models, while having weaker fitting capabilities, are more computationally efficient and have fewer parameters, making them easier to deploy. Knowledge distillation, as an important model compression technique, can transfer knowledge from a complex model (teacher model) to a simpler model (student model), allowing the student model's fitting ability to approach or even surpass that of the teacher model, thus achieving similar predictive results with less time and space complexity.

[0044] In related technologies, knowledge distillation systems are mostly designed for classification problems and cannot be well applied to object detection and instance segmentation. Furthermore, these knowledge distillation systems rely on the network's own predictions, which means that the easily identifiable regions learned by the network may not match human observation and cognition, thus affecting the model's object detection results to some extent.

[0045] This disclosure provides a neural network model distillation method. By manually inputting object conditional features to characterize the target object's features, the attention distribution obtained by the teacher model is made more closely resemble human observation. Using this attention distribution, and the feature difference between the teacher model's feature map and the student model's feature map, the initial student model is trained, improving the training effect of the student model and thus enhancing its object detection capability.

[0046] Figure 1 This is a flowchart illustrating a neural network model distillation method according to an exemplary embodiment, such as... Figure 1 As shown, it includes the following steps.

[0047] In step S11, the target image is determined and the object condition features of the target image are obtained.

[0048] Among these, object conditional features can be understood as manually inputted feature information that characterizes the target object. Furthermore, it can be understood that the target object is an object existing within the target image.

[0049] In step S12, feature detection is performed on the target image based on the teacher model to obtain the teacher model feature map, and feature detection is performed on the target image based on the initial student model to obtain the student model feature map.

[0050] In this embodiment of the disclosure, the initial student model can be understood as the student model to be trained. In one example, the teacher model feature map includes object features obtained by the teacher model performing feature detection at each pixel position of the target image, and the student model feature map includes object features obtained by the initial student model performing feature detection at each pixel position of the target image.

[0051] In step S13, the correlation between object features and object condition features at each pixel position in the feature map of the teacher model is determined, and the attention distribution of the teacher model is determined based on the correlation.

[0052] For example, the correlation between object features and object conditional features can be understood as the degree of similarity between the feature sequences corresponding to the object features and the feature sequences corresponding to the object conditional features. The attention distribution of the teacher model can be understood as the detection weights assigned to each pixel position by the teacher model during object detection.

[0053] In step S14, the initial student model is trained based on the attention distribution of the teacher model and the feature difference between the teacher model feature map and the student model feature map to obtain the target student model.

[0054] The target student model can be understood as the student model that has been trained.

[0055] In this embodiment, prior features (i.e., object condition features) are manually input to provide the teacher model with an index direction for determining the attention distribution. The attention distribution determined by the teacher model in this way is more closely aligned with human observation perspective. Furthermore, training the initial student model with this attention distribution can improve the recognition accuracy of the student model.

[0056] In one example, training an initial student model to obtain a target student model can be achieved by determining the feature difference between the teacher model's feature map and the student model's feature map at each pixel location. Further, the attention distribution of the teacher model is used as a weight assigned when calculating the distillation loss between the teacher model and the initial student model, thus calculating the distillation loss between them. For example, calculating the distillation loss between the teacher model and the initial student model can be done by calculating the mean squared error between their feature maps. Introducing this distillation loss into the initial student model as an auxiliary parameter for object detection can make the object detection results of the initial student model similar to or the same as those of the teacher model. That is, the trained target student model is obtained.

[0057] In one example, a Feature Pyramid Network (FPN) can be introduced into both the teacher and student models. Based on this, the teacher model can perform multi-scale feature detection on the target image to obtain a teacher model feature map containing multi-scale features, and the student model can perform multi-scale feature detection on the target image to obtain a student model feature map containing multi-scale features.

[0058] Figure 2 This is a flowchart illustrating another neural network model distillation method according to an exemplary embodiment, such as... Figure 2 As shown, the implementation process of steps S21, S23, and S24 in this embodiment of the present disclosure is the same as... Figure 1 The execution methods of steps S11, S13 and S14 shown are similar and will not be repeated here.

[0059] In step S22, the target image is subjected to multi-scale feature detection by the teacher model to obtain a teacher model feature map including multi-scale features, and the target image is subjected to multi-scale feature detection by the student model to obtain a student model feature map including multi-scale features.

[0060] The neural network model distillation method provided in this disclosure can perform feature detection on target images at multiple different scales (resolutions) using a feature pyramid network. Specifically, for large-scale (low-resolution) images, it can more accurately identify the relevant features of small objects in the target image; for small-scale (high-resolution) images, it can more accurately identify the relevant features of large objects in the target image. This method can meet the feature detection requirements for objects at different scales in target images, thereby improving the target detection accuracy of teacher and / or student models.

[0061] Typically, for the same image, there are various feature regions that contribute to object detection, such as foreground regions, target object edge regions, overlapping targets, and the relationships between targets.

[0062] The object detection distillation method provided in this disclosure can decompose the teacher model feature map, and then map the decomposed teacher model feature map to multiple sub-feature spaces through a multi-head attention network. Furthermore, different weights are assigned to each sub-feature space, enabling the teacher model to identify features in different regions that are helpful for object detection.

[0063] For ease of description, this disclosure describes the process of calculating the correlation between object features and object condition features at each pixel position of the target image, and the sub-feature space obtained by decomposing the teacher model feature map is called the first sub-feature space.

[0064] Figure 3 This is a flowchart illustrating a method for determining the correlation between object features and object condition features at each pixel position in a teacher model feature map, according to an exemplary embodiment. Figure 3 As shown, it includes the following steps.

[0065] In step S31, the object features at each pixel position in the teacher model feature map are decomposed into multiple first sub-feature spaces.

[0066] In step S32, for each pixel position in the teacher model feature map, the features of multiple first sub-feature spaces are weighted according to different weights to obtain the object features after feature weighting.

[0067] In step S33, the correlation between the weighted object features and the object condition features is determined, and based on the correlation between the weighted object features and the object condition features, the first sub-feature space correlation of multiple first sub-feature spaces for each pixel position is obtained.

[0068] In step S34, for each pixel position in the teacher model feature map, the correlation between the first sub-feature spaces of multiple first sub-feature spaces is normalized to obtain the correlation between the object features and the object condition features corresponding to each pixel position.

[0069] For example, suppose the teacher model feature map has W rows of pixels, H columns of pixels, and C feature dimensions corresponding to each pixel position. When decomposing the teacher model feature map into multiple first sub-feature spaces and weighting these sub-feature spaces with different weights, the teacher model feature map can be decomposed into N first sub-feature spaces (M*N=C) with W rows of pixels, H columns of pixels, and M feature dimensions corresponding to each pixel position. The number of sub-feature spaces N can be any value; that is, the teacher model feature map can be decomposed into any number of first sub-feature spaces. Furthermore, different weights are applied to each of the N first sub-feature spaces. For example, for first sub-feature space A, the foreground region of the image can be assigned a higher weight than other regions. For first sub-feature space B (different from first sub-feature space A), other regions of the image besides the foreground region can be assigned a higher weight than the foreground region, thus setting different weights for different first sub-feature spaces. Of course, various methods can be used to weight features for different first sub-feature spaces, and this disclosure does not impose any specific limitations on this.

[0070] The neural network model distillation method provided in this disclosure assigns different weights to different first sub-feature spaces, enabling the teacher model to use different regions in the target image as the main recognition regions. Furthermore, by normalizing the attention distributions obtained for different sub-feature spaces, the teacher model can identify object features that contribute to target detection in each region of the target image, thereby improving the recognition accuracy of the teacher model.

[0071] In one example, the correlation between pixel positions in the teacher model feature map can be used to assign weights to the correlation between pixel positions in the teacher model feature map, and the attention distribution of the teacher model can be determined accordingly.

[0072] Figure 4 This is a flowchart illustrating another neural network model distillation method according to an exemplary embodiment, such as... Figure 4 As shown, the implementation process of steps S41, S42, and S44 in this embodiment of the present disclosure is the same as... Figure 1 The execution methods of steps S11, S12 and S14 shown are similar and will not be repeated here.

[0073] In step S43, the correlation between object features and object condition features at each pixel position in the teacher model feature map is determined. Based on the magnitude of the correlation between object features and object condition features at each pixel position in the teacher model feature map, weights are assigned to each pixel position in the teacher model feature map. Based on the assigned weights, the attention distribution of the teacher model is determined.

[0074] Specifically, for each pixel position in the teacher model feature map, the pixel position with greater relevance receives a larger weight. For example, when assigning weights to the relevance of each pixel position in the teacher model feature map based on the magnitude of the relevance, pixels with higher relevance values ​​can be assigned higher weights, while pixels with lower relevance values ​​can be assigned lower weights.

[0075] In one example, if the two different correlations between pixel positions are called the first correlation and the second correlation, and the first correlation is greater than the second correlation, then the first correlation can be assigned a corresponding first weight, and the second correlation can be assigned a corresponding second weight, and the first weight corresponding to the first correlation is greater than the second weight corresponding to the second correlation.

[0076] In one embodiment, when assigning weights to the correlation of each pixel position in the teacher model feature map based on the correlation between each pixel position, the sum of the weights assigned to each pixel position in the teacher model feature map is 1.

[0077] In one example, the attention distribution of the teacher model can be optimized, and the initial student model can be trained based on the optimized attention distribution to obtain the target student model, thereby further improving the recognition accuracy of the target student model.

[0078] Figure 5 This is a flowchart illustrating a method for training an initial student model according to an exemplary embodiment, such as... Figure 5 As shown, it includes the following steps.

[0079] In step S51, the attention distribution of the teacher model is optimized.

[0080] In step S52, the initial student model is trained based on the optimized attention distribution and the feature difference between the teacher model feature map and the student model feature map to obtain the target student model.

[0081] The neural network model distillation method provided in this disclosure can optimize the attention distribution of the teacher model. Using the optimized attention distribution to train the initial student model yields the target student model, which can further improve the recognition accuracy of the target student model.

[0082] In one example, the teacher model can be controlled to perform object detection on the target image. Then, based on the object detection results of the teacher model and the object condition features, the attention distribution of the teacher model can be adjusted to optimize the attention distribution.

[0083] For ease of description, this disclosure will decompose the teacher model feature map into a sub-feature space during the process of optimizing the attention distribution of the teacher model, which will be referred to as the second sub-feature space.

[0084] Figure 6 This is a flowchart illustrating a method for optimizing the attention distribution of a teacher model according to an exemplary embodiment, such as... Figure 6 As shown, it includes the following steps.

[0085] In step S61, the object features at each pixel position in the teacher model feature map are decomposed into multiple second sub-feature spaces.

[0086] In step S62, based on the attention distribution of the teacher model, feature weighting is performed on multiple second sub-feature spaces.

[0087] In step S63, the teacher model is controlled to perform target detection based on the weighted features, and the attention distribution of the teacher model is adjusted based on the target detection results and object condition features so that the teacher model obtains target detection results consistent with the object condition features.

[0088] In one example, target detection based on weighted features can be achieved by aggregating multiple second sub-feature spaces into a feature vector that represents the features of objects in the target image. The teacher model can then perform target detection on the target image based on the aggregated feature vector to obtain the target detection result.

[0089] In one implementation, a multilayer perceptron (MLP) can be used to perform target detection on the weighted and aggregated feature vectors to obtain the target detection result.

[0090] In this embodiment of the disclosure, the first sub-feature space and the second sub-feature space can be obtained by decomposing the teacher model feature map using different decomposition methods. For example, the decomposition methods used may include key-value decomposition and content decomposition.

[0091] In one embodiment, the teacher model feature map can be decomposed using key-value decomposition to obtain multiple first sub-feature spaces, and the teacher model feature map can be decomposed using content decomposition to obtain multiple second sub-feature spaces. Since different decomposition methods can yield different features for object detection for the same teacher model feature map, this method can enrich the feature samples and thereby improve the recognition accuracy of the teacher model. Of course, this disclosure can also perform feature decomposition on the teacher model feature map using other methods, and this disclosure does not specifically limit the decomposition method used.

[0092] In this embodiment of the disclosure, the object condition features manually input into the teacher model may include one or a combination of the target object's category features, location features, and scale features. In one example, obtaining a target detection result consistent with the object condition features may mean obtaining a target detection result consistent with one or a combination of the category features, location features, and scale features corresponding to the object condition features.

[0093] Figure 7 This is a flowchart illustrating another method for optimizing the attention distribution of a teacher model according to an exemplary embodiment. The implementation process of steps S71 and S72 in this embodiment is similar to... Figure 6 The execution methods of steps S61 and S62 shown are similar and will not be repeated here.

[0094] In step S73, the teacher model is controlled to perform target detection based on the weighted features, and the attention distribution of the teacher model is adjusted based on the target detection results and object condition features, so that the teacher model obtains a target detection result that is consistent with one or a combination of the category features, position features, and scale features corresponding to the object condition features.

[0095] In this embodiment of the disclosure, the positional features of the target object can be encoded and input into the teacher model via position embedding. The category features and / or scale features of the target object can be encoded using one-hot vectors and input into the teacher model.

[0096] In one embodiment, the input of the target object's positional features can be the coordinates of the center point of the target object in the target image. Alternatively, it can be the coordinates of any point within the area where the target object is located.

[0097] In another implementation, the category features of the target object can be determined based on human observation. Furthermore, a feature sequence matching the target object's category can be found in a pre-defined category dataset of the teacher model, and this feature sequence can be used as the target object's category feature and input into the teacher model.

[0098] In another implementation, the scale features of the input target object can be a predefined object scale range for the teacher model. For example, target objects with a scale smaller than 64 bits can be defined as small objects, objects with a scale between 64 and 256 bits can be defined as medium objects, and objects with a scale larger than 256 bits can be defined as large objects. Further, the scale of the target object is determined based on human observations, and the feature sequence representing the scale range is input into the teacher model.

[0099] In this embodiment, by adjusting the relevant parameters of the teacher model, the teacher model can output target detection results consistent with the conditional features of the object. In one embodiment, a corresponding loss function can be set for the teacher model, which can adaptively adjust the attention distribution.

[0100] The neural network model distillation method provided in this disclosure, by adjusting the attention distribution of the teacher model through a loss function, can be understood as assigning an auxiliary task to the teacher model. For example, this can be understood as assigning the following auxiliary task to the teacher model.

[0101] Auxiliary Task 1: Determine whether a target object exists in the target image.

[0102] Auxiliary Task 2: Determine whether the location area marked by the teacher's model matches the location of the target object in the target image.

[0103] In one implementation, the teacher model can be controlled to perform object detection and determine whether there is a target object in the target image that matches the type features input to the teacher model. For example, if the object exists, the teacher model's output can be 1; if it does not exist, the output can be 0. Of course, the teacher model's output can also be a floating-point number between 0 and 1. When the teacher model's output is a floating-point number, an output value approaching 1 can be considered as the presence of an object, and an output value approaching 0 can be considered as the absence of an object. For example, if the output does not match the result represented by the category features, the teacher model's attention distribution is adjusted in the direction of decreasing loss function output value until the teacher model's object detection result is consistent with the category feature representation.

[0104] In another implementation, if the presence of a target object in the target image is confirmed, it can be determined whether the location region of the target object labeled by the teacher model in the target image matches the location region represented by the positional features and scale features in the target image. For example, the image position represented by the positional features of the target object is determined as the coordinates of a random point within the target image's location region, and the teacher model is controlled to label the location region of the target object in the target image accordingly. Further, the distances between the positional coordinates represented by the positional features and the coordinates of each edge vertex of the location region labeled by the teacher model are determined, and the resulting multiple different distance values ​​are normalized and converted into floating-point numbers between 0 and 1. Further, output values ​​approaching 1 are considered to match the labeled location regions in the target image, while output values ​​approaching 0 are considered to be inconsistent. For example, if the output result does not match the results represented by the positional features and scale features, the attention distribution of the teacher model is adjusted in the direction of decreasing output value of the teacher model's loss function until the target detection result of the teacher model is consistent with the representation of the positional features and scale features.

[0105] In this embodiment, the teacher model can be controlled to perform object detection based on a previously obtained attention distribution. Furthermore, the object conditional features input to the teacher model are used as the target detection result the teacher model expects to obtain, enabling the teacher model to adaptively adjust its attention distribution, thereby allowing the teacher model to obtain target detection results consistent with the object conditional features. This method can improve the target detection accuracy of the teacher model, and training the initial student model with the adjusted attention distribution can improve the target student model's target detection accuracy.

[0106] Figure 8 This is a schematic diagram illustrating a process for training an initial student model according to an exemplary embodiment, such as... Figure 8 As shown, for the same target image, object conditional feature 1 (relevant features representing "people" in the target image) and object conditional feature 2 (relevant features representing "horses" in the target image) can be input into the teacher model. For example, feature detection can be performed on the target image to obtain a teacher model feature map. Further, the teacher model feature map can be decomposed into multiple first sub-feature spaces using key-value decomposition, and into multiple second sub-feature spaces using content decomposition. In this case, for example, object conditional feature 1 (or object conditional feature 2) can be used as an index feature, and the correlation between the weighted first sub-feature space and object conditional feature 1 can be calculated to determine the teacher model's attention distribution for object conditional feature 1. Furthermore, given the determined attention distribution of the teacher model, the teacher model can be controlled to perform target detection by weighting the multiple second sub-feature spaces with the obtained attention distribution. Further, the representations of the object conditional features (object conditional feature 1 and / or object conditional feature 2) input into the teacher model are used as the target output of the teacher model to optimize the teacher model's attention distribution. The initial student model is then trained with the optimized attention distribution to obtain the trained target student model. For example, after the initial student model training is complete, only the feature extraction network and the final detection head of the target student model can be retained for object detection-related testing and deployment. Furthermore, the target student model trained in this disclosure shows a significant improvement in performance on commonly used datasets for object detection distillation.

[0107] The neural network model distillation method provided in this disclosure divides the teacher model feature map obtained from detecting target images by the teacher model into multiple first sub-feature spaces using key-value decomposition. Different weights are assigned to different first sub-feature spaces to enable the teacher model to identify various types of features helpful for target detection. Furthermore, object conditional features representing the target object can be input as auxiliary parameters into the teacher model. Since these object conditional features are manually calibrated and input, the attention distribution of the teacher model obtained through these features is more closely aligned with human observation perspectives, thereby improving the target detection accuracy of the teacher model. In addition, since the teacher model feature map can be a multi-scale feature map obtained through a feature pyramid network, superior target detection results can be obtained for both large-scale and small-scale objects. Based on this, the neural network model distillation method provided in this disclosure introduces a special loss function to adjust the attention distribution obtained from the teacher model, achieving optimization of the attention distribution and significantly improving the target detection accuracy of the teacher model. Furthermore, training the initial student model with the optimized attention distribution yields a target student model with high target detection accuracy. Using the target student model trained by this method for target detection can yield better target detection results, thus meeting the user's target detection needs.

[0108] Based on the same concept, embodiments of this disclosure also provide a neural network model distillation apparatus.

[0109] It is understood that the neural network model distillation apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0110] Figure 9 This is a block diagram of a neural network model distillation apparatus 100 according to an exemplary embodiment. (Refer to...) Figure 9 The device 100 includes a determining unit 101, an acquiring unit 102, a detecting unit 103, and a processing unit 104.

[0111] The determination unit 101 is used to determine the target image and the correlation between object features and object condition features at each pixel position in the teacher model feature map, and to determine the attention distribution of the teacher model based on the correlation. The acquisition unit 102 is used to acquire the object condition features of the target image, which are manually input feature information representing the target object. The detection unit 103 is used to perform feature detection on the target image based on the teacher model to obtain the teacher model feature map, and to perform feature detection on the target image based on the initial student model to obtain the student model feature map. The processing unit 104 is used to train the initial student model based on the attention distribution of the teacher model and the feature difference between the teacher model feature map and the student model feature map to obtain the target student model.

[0112] In one embodiment, the determining unit 101 determines the correlation between object features and object condition features at each pixel position in the teacher model feature map as follows: The object features at each pixel position in the teacher model feature map are decomposed into multiple first sub-feature spaces. For each pixel position in the teacher model feature map, the multiple first sub-feature spaces are weighted according to different weights to obtain the feature-weighted object features. The correlation between the feature-weighted object features and the object condition features is determined, and the correlation between the feature-weighted object features and the object condition features is used to obtain the first sub-feature space correlation for each pixel position. For each pixel position in the teacher model feature map, the first sub-feature space correlation of the multiple first sub-feature spaces is normalized to obtain the correlation between the object features and object condition features corresponding to each pixel position.

[0113] In one embodiment, the processing unit 104 trains an initial student model to obtain a target student model based on the attention distribution of the teacher model and the feature difference between the teacher model feature map and the student model feature map, in the following manner: The attention distribution of the teacher model is optimized. Based on the optimized attention distribution and the feature difference between the teacher model feature map and the student model feature map, the initial student model is trained to obtain the target student model.

[0114] In one implementation, the processing unit 104 optimizes the attention distribution of the teacher model as follows: The object features at each pixel position in the teacher model's feature map are decomposed into multiple second sub-feature spaces. Based on the teacher model's attention distribution, the multiple second sub-feature spaces are weighted. The teacher model is controlled to perform object detection based on the weighted features, and the attention distribution of the teacher model is adjusted based on the object detection results and object condition features, so that the teacher model obtains object detection results consistent with the object condition features.

[0115] In one implementation, the first sub-feature space and the second sub-feature space use different decomposition methods. These decomposition methods include key-value decomposition and content decomposition.

[0116] In one embodiment, the processing unit 104 obtains a target detection result consistent with the object condition features in the following manner: obtaining a target detection result consistent with one or a combination of the category features, position features, and scale features corresponding to the object condition features.

[0117] In one implementation, the determining unit 101 determines the attention distribution of the teacher model based on correlation as follows: Based on the correlation between object features and object condition features at each pixel position in the teacher model feature map, weights are assigned to each pixel position in the teacher model feature map, and the attention distribution of the teacher model is obtained according to the assigned weights. Specifically, the first weight corresponding to the first correlation is greater than the second weight corresponding to the second correlation, and the first correlation is greater than the second correlation.

[0118] In one embodiment, the detection unit 103 performs feature detection on the target image based on the teacher model to obtain a teacher model feature map, and performs feature detection on the target image based on the initial student model to obtain a student model feature map: The teacher model is controlled to perform multi-scale feature detection on the target image to obtain a teacher model feature map including multi-scale features. The student model is also controlled to perform multi-scale feature detection on the target image to obtain a student model feature map including multi-scale features.

[0119] like Figure 10 As shown, one embodiment of this disclosure provides an electronic device 200. The electronic device 200 includes a memory 201, a processor 202, and an input / output (I / O) interface 203. The memory 201 is used to store instructions. The processor 202 is used to execute the neural network model distillation method of this disclosure embodiment by calling the instructions stored in the memory 201. The processor 202 is connected to both the memory 201 and the I / O interface 203, for example, via a bus system and / or other forms of connection mechanisms (not shown). The memory 201 can be used to store programs and data, including the program for the neural network model distillation method involved in the embodiments of this disclosure. The processor 202 executes various functional applications and data processing of the electronic device 200 by running the program stored in the memory 201.

[0120] In this embodiment of the disclosure, the processor 202 may be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 202 may be one or a combination of several of the following: a central processing unit (CPU) or other processing units with data processing capabilities and / or instruction execution capabilities.

[0121] The memory 201 in this embodiment may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD).

[0122] In this embodiment of the disclosure, the I / O interface 203 can be used to receive input instructions (such as numeric or character information, and to generate key signal inputs related to user settings and function control of the electronic device 200), and can also output various information (such as images or sounds) to the outside. In this embodiment of the disclosure, the I / O interface 203 may include one or more of the following: a physical keyboard, function keys (such as volume control keys, power buttons, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel.

[0123] In some embodiments, this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, perform any of the methods described above.

[0124] Although the operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the operations shown to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0125] The methods and apparatus disclosed herein can be implemented using standard programming techniques, utilizing rule-based logic or other logic to implement various method steps. It should also be noted that the terms "apparatus" and "module" as used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0126] Any step, operation, or procedure described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product comprising a computer-readable medium containing computer program code, which is executable by a computer processor to perform any or all of the described steps, operations, or procedures.

[0127] The foregoing description of embodiments of this disclosure has been provided for purposes of illustration and description. The foregoing description is not exhaustive and is not intended to limit this disclosure to the exact form disclosed; various modifications and variations may be made in accordance with the foregoing teachings, or may be derived from practice of this disclosure. These embodiments were chosen and described to illustrate the principles of this disclosure and its practical application, enabling those skilled in the art to utilize this disclosure in various implementations and modifications suitable for the particular purpose conceived.

[0128] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0129] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0130] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0131] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0132] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0133] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following scope of claims.

[0134] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A neural network model distillation method, characterized in that... The method includes: A target image is determined, and the object condition features of the target image are obtained. The object condition features are manually input feature information that characterizes the target object. Feature detection is performed on the target image based on the teacher model to obtain the teacher model feature map, and feature detection is performed on the target image based on the initial student model to obtain the student model feature map; The correlation between object features at each pixel location in the feature map of the teacher model and the object condition features is determined, and the attention distribution of the teacher model is determined based on the correlation. Based on the attention distribution of the teacher model and the feature difference between the feature map of the teacher model and the feature map of the student model, the initial student model is trained to obtain the target student model. Determining the correlation between object features at each pixel position in the teacher model feature map and the object condition features includes: The object features at each pixel position in the feature map of the teacher model are decomposed into multiple first sub-feature spaces; For each pixel position in the feature map of the teacher model, the multiple first sub-feature spaces are weighted according to different weights to obtain the object features after feature weighting; Based on the correlation between the weighted object features and the object condition features, the first sub-feature space correlation of multiple first sub-feature spaces for each pixel position is obtained; For each pixel position in the feature map of the teacher model, the correlation between the first sub-feature spaces of the multiple first sub-feature spaces is normalized to obtain the correlation between the object features and the object condition features corresponding to each pixel position.

2. The neural network model distillation method according to claim 1, characterized in that... Based on the attention distribution of the teacher model and the feature difference between the teacher model feature map and the student model feature map, the initial student model is trained to obtain the target student model, including: The attention distribution of the teacher model is optimized; Based on the optimized attention distribution and the feature difference between the teacher model feature map and the student model feature map, the initial student model is trained to obtain the target student model.

3. The neural network model distillation method according to claim 2, characterized in that... Optimizing the attention distribution of the teacher model includes: The object features at each pixel position in the feature map of the teacher model are decomposed into multiple second sub-feature spaces; Based on the attention distribution of the teacher model, feature weighting is applied to the multiple second sub-feature spaces; The teacher model is controlled to perform target detection based on weighted features, and the attention distribution of the teacher model is adjusted based on the target detection results and object condition features, so that the teacher model obtains target detection results consistent with the object condition features.

4. The neural network model distillation method according to claim 3, characterized in that... The first sub-feature space and the second sub-feature space use different decomposition methods; The decomposition methods include key-value decomposition and content decomposition.

5. The neural network model distillation method according to claim 3, characterized in that... Obtaining target detection results consistent with the conditional features of the object includes: Obtain a target detection result that is consistent with one or a combination of the category features, position features, and scale features corresponding to the object's conditional features.

6. The neural network model distillation method according to claim 1, 4, or 5, characterized in that... Based on the correlation, the attention distribution of the teacher model is determined, including: Based on the correlation between the object features and the object condition features at each pixel position in the teacher model feature map, weights are assigned to each pixel position in the teacher model feature map, and the attention distribution of the teacher model is obtained according to the assigned weights. Specifically, for each pixel position in the feature map of the teacher model, the pixel position with greater relevance has a greater weight.

7. The neural network model distillation method according to claim 1, characterized in that... The target image is subjected to feature detection based on the teacher model to obtain a teacher model feature map, and the target image is subjected to feature detection based on the initial student model to obtain a student model feature map, including: The teacher model is used to perform multi-scale feature detection on the target image, resulting in a teacher model feature map that includes multi-scale features; and The student model is used to perform multi-scale feature detection on the target image to obtain a student model feature map that includes multi-scale features.

8. A neural network model distillation apparatus, characterized in that... The device includes: A determining unit is used to determine the target image, and to determine the correlation between object features and object condition features at each pixel position in the feature map of the teacher model, and to determine the attention distribution of the teacher model based on the correlation; The acquisition unit is used to acquire the object condition features of the target image, wherein the object condition features are manually input feature information that characterizes the target object; The detection unit is used to perform feature detection on the target image based on the teacher model to obtain a teacher model feature map, and to perform feature detection on the target image based on the initial student model to obtain a student model feature map; The processing unit is used to train the initial student model based on the attention distribution of the teacher model and the feature difference between the feature map of the teacher model and the feature map of the student model to obtain the target student model. The determining unit is used to decompose the object features at each pixel position in the teacher model feature map into multiple first sub-feature spaces; for each pixel position in the teacher model feature map, the multiple first sub-feature spaces are weighted according to different weights to obtain the weighted object features; based on the correlation between the weighted object features and the object condition features, the correlation between the first sub-feature spaces of the multiple first sub-feature spaces for each pixel position is obtained; for each pixel position in the teacher model feature map, the correlation between the first sub-feature spaces of the multiple first sub-feature spaces is normalized to obtain the correlation between the object features at each pixel position and the object condition features.

9. An electronic device, characterized in that... ,include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the neural network model distillation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that... The storage medium stores instructions that, when executed by a processor, enable the processor to perform the neural network model distillation method according to any one of claims 1 to 7.

Images