Image feature recognition method, device, equipment and medium

Abstract

The embodiments of the present application disclose a method, device, equipment and medium for identifying image features. The selected teacher network model and student network model are used to extract features of sample images respectively to obtain teacher features and student features; a significance threshold is determined based on the ratio corresponding to the teacher features, the student features and the number of iterations; and the teacher significance information and the student significance information are determined according to the significance threshold. The student network model is trained using the teacher features, the student features, the teacher significance information and the student significance information to obtain a trained student network model. The student network model is used to analyze the image to be processed to identify the features contained in the image to be processed. The significance threshold is used to generate significance information, which improves the depth of image feature mining. The significance information is used to guide the student network model to imitate the significance information of the teacher network model, which effectively improves the detection accuracy of the student network model.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to an image feature recognition method, device, equipment and computer-readable storage medium. Background Art

[0002] Algorithms based on deep learning have achieved success in many fields such as image processing, speech recognition, and natural language processing. With the development of deep learning technology in academia, various artificial intelligence products based on deep learning technology have also been implemented in the industry. Nowadays, devices with intelligent functions are widely used in various practical scenarios, such as autonomous driving, smart speakers, automatic translation, etc.

[0003] In the industry, in order to obtain high-performance neural network models, engineers usually use neural networks with depths of dozens or even hundreds of layers as the backbone network to design neural network models. However, it is very challenging to deploy these high-precision neural network models to resource-constrained devices to perform real-time application tasks. In actual applications, in order to complete specified tasks in real time, such as target detection and scene classification, application servers need to be equipped with high-performance GPUs (General Processing Units) to complete model reasoning tasks. When the server is not equipped with a high-performance GPU card, these applications will be restricted.

[0004] In order to avoid the above limitations, the industry urgently needs to develop and deploy high-precision neural network models with low computational complexity. How to reduce the computational complexity of the model while maintaining the same accuracy or with little accuracy degradation, that is, model lightweighting, is a widely concerned technology. For deep convolutional neural network models, typical model lightweighting methods include: low-bit quantization, model pruning, low rank factorization, knowledge distillation, etc.

[0005] In deep learning, knowledge distillation is the process of transferring knowledge from a large model, the teacher model, to a smaller model, the student model. In order to achieve knowledge transfer, knowledge distillation technology uses the output of the student network to imitate the output of the teacher network. Because the teacher network has stronger expressive power, the output of the teacher network can guide the training of the student network. When the output of the student network gradually approaches the output of the teacher network, the performance of the student network also gradually approaches the performance of the teacher network. The student network has a low computational cost during forward reasoning, and they can be deployed on devices with weaker computing power. Although the current model training method considers the feature expression of the intermediate layer of the teacher network and the student network, and converts the feature expression into an attention map, it does not deeply explore the hidden information of the attention map, resulting in limited accuracy of the student network model.

[0006] It can be seen that how to improve the depth of image feature mining is a problem that technical personnel in this field need to solve. Summary of the invention

[0007] The purpose of the embodiments of the present application is to provide a method, device, equipment and computer-readable storage medium for recognizing image features, which can enhance the depth of image feature mining.

[0008] In order to solve the above technical problems, the present application provides an image feature recognition method, including:

[0009] The selected teacher network model and student network model are used to extract features of the sample images respectively to obtain teacher features and student features;

[0010] Determining a significance threshold based on a ratio of the teacher characteristic, the student characteristic, and the number of iterations;

[0011] Determining teacher significance information corresponding to the teacher characteristics and student significance information corresponding to the student characteristics according to the significance threshold;

[0012] Training the student network model using the teacher characteristics, the student characteristics, the teacher significance information, and the student significance information;

[0013] The trained student network model is used to analyze the image to be processed to identify the features contained in the image to be processed.

[0014] Optionally, the extracting features of the sample image using the selected teacher network model and student network model to obtain teacher features and student features includes:

[0015] When a recognition task is obtained, a teacher network model and a student network model matching the recognition task are selected;

[0016] The teacher network model and the student network model are used to extract features of the sample image respectively to obtain teacher features and student features.

[0017] Optionally, determining a significance threshold based on a ratio of the teacher feature, the student feature, and the number of iterations includes:

[0018] Counting the first data distribution of the teacher's characteristics and the second data distribution of the student's characteristics;

[0019] Determine a first significance threshold corresponding to the teacher network by using a ratio of the first data distribution to the current number of iterations;

[0020] A second significance threshold corresponding to the student network is determined by using a ratio of the second data distribution to the current number of iterations.

[0021] Optionally, determining the teacher significance information corresponding to the teacher feature and the student significance information corresponding to the student feature according to the significance threshold comprises:

[0022] Adjusting the feature values ​​of the teacher features that are less than the first significance threshold to zero, and using the adjusted teacher features as teacher significance information;

[0023] The feature values ​​of the student features that are less than the second significance threshold are adjusted to zero, and the adjusted student features are used as student significance information.

[0024] Optionally, the training of the student network model using the teacher characteristics, the student characteristics, the teacher significance information, and the student significance information includes:

[0025] Constructing a loss function of a student network model according to the teacher characteristics, the student characteristics, the teacher significance information and the student significance information; wherein the teacher significance information and the student significance information correspond to different weights at different levels of the model;

[0026] The student network model is trained using the student characteristics, the student significance information and the loss function.

[0027] Optionally, the loss function of constructing the student network model according to the teacher characteristics, the student characteristics, the teacher significance information and the student significance information includes:

[0028] According to the teacher characteristics and the student characteristics, determining an initial loss function corresponding to the student network and a distillation loss function corresponding to the characteristics;

[0029] Determining a distillation loss function corresponding to the saliency information according to the teacher saliency information and the student saliency information;

[0030] Based on the initial loss function, the distillation loss function corresponding to the feature, and the distillation loss function corresponding to the saliency information, a loss function of the student network model is constructed.

[0031] Optionally, when the task corresponding to the image to be processed is a face recognition task, the analyzing the image to be processed by using the trained student network model to identify the features contained in the image to be processed includes:

[0032] The trained student network model is used to analyze the image to be processed to identify the facial features contained in the image to be processed.

[0033] Optionally, when the task corresponding to the image to be processed is a digital classification task, the step of analyzing the image to be processed by using the trained student network model to identify features contained in the image to be processed includes:

[0034] The trained student network model is used to analyze the image to be processed to extract digital features contained in the image to be processed.

[0035] The embodiment of the present application also provides an image feature recognition device, including an extraction unit, a threshold determination unit, an information determination unit, a training unit, and a recognition unit;

[0036] The extraction unit is used to extract features of the sample image using the selected teacher network model and student network model to obtain teacher features and student features;

[0037] The threshold determination unit is used to determine a significance threshold based on the ratio of the teacher feature, the student feature and the number of iterations;

[0038] The information determination unit is used to determine the teacher significance information corresponding to the teacher characteristics and the student significance information corresponding to the student characteristics according to the significance threshold;

[0039] The training unit is used to train the student network model by using the teacher characteristics, the student characteristics, the teacher significance information, and the student significance information;

[0040] The recognition unit is used to analyze the image to be processed by using the trained student network model to identify the features contained in the image to be processed.

[0041] Optionally, the extracting unit includes selecting a subunit and obtaining a subunit;

[0042] The selection subunit is used to select a teacher network model and a student network model that match the recognition task when a recognition task is obtained;

[0043] The obtaining subunit is used to use the teacher network model and the student network model to extract features of the sample image respectively to obtain teacher features and student features.

[0044] Optionally, the threshold determination unit includes a statistical subunit, a first determination subunit and a second determination subunit;

[0045] The statistical subunit is used to count the first data distribution of the teacher characteristics and the second data distribution of the student characteristics;

[0046] The first determination subunit is used to determine a first significance threshold corresponding to the teacher network by using a ratio of the first data distribution to a current number of iterations;

[0047] The second determination subunit is used to determine a second significance threshold corresponding to the student network by using a ratio of the second data distribution to the current number of iterations.

[0048] Optionally, the information determination unit includes a first adjustment subunit and a second adjustment subunit;

[0049] The first adjustment subunit is used to adjust the feature values ​​of the teacher features that are less than the first significance threshold to zero, and use the adjusted teacher features as teacher significance information;

[0050] The second adjustment subunit is used to adjust the feature values ​​of the student features that are less than the second significance threshold to zero, and use the adjusted student features as student significance information.

[0051] Optionally, the training unit includes a construction subunit and a completion subunit;

[0052] The construction subunit is used to construct a loss function of the student network model according to the teacher characteristics, the student characteristics, the teacher significance information and the student significance information; wherein the teacher significance information and the student significance information correspond to different weights at different levels of the model;

[0053] The completion subunit is used to complete the training of the student network model using the student characteristics, the student significance information and the loss function.

[0054] Optionally, the construction subunit is used to determine the initial loss function corresponding to the student network and the distillation loss function corresponding to the features according to the teacher characteristics and the student characteristics; determine the distillation loss function corresponding to the significance information according to the teacher significance information and the student significance information; and construct the loss function of the student network model based on the initial loss function, the distillation loss function corresponding to the features, and the distillation loss function corresponding to the significance information.

[0055] Optionally, when the task corresponding to the image to be processed is a face recognition task, the recognition unit is used to analyze the image to be processed using a trained student network model to identify facial features contained in the image to be processed.

[0056] Optionally, when the task corresponding to the image to be processed is a digital classification task, the recognition unit is used to analyze the image to be processed using a trained student network model to extract digital features contained in the image to be processed.

[0057] The present application also provides an electronic device, including:

[0058] Memory for storing computer programs;

[0059] A processor is used to execute the computer program to implement the steps of the above-mentioned image feature recognition method.

[0060] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image feature recognition method are implemented.

[0061] It can be seen from the above technical scheme that the selected teacher network model and student network model are used to extract features of the sample image respectively to obtain teacher features and student features; the significance threshold is determined based on the ratio corresponding to the teacher features, student features and the number of iterations; the significance threshold can be used to deeply explore the hidden information between the teacher features and the student features, so the teacher significance information corresponding to the teacher features and the student significance information corresponding to the student features can be determined according to the significance threshold. The student network model is trained using the teacher features, the student features, the teacher significance information and the student significance information to obtain a student network model with higher detection accuracy. The trained student network model is used to analyze the image to be processed to identify the features contained in the image to be processed. In this technical scheme, the significance threshold is used to generate the significance information of the attention map, which improves the depth of image feature mining. On the basis of the teacher features and the student features, the significance information of the teacher network and the significance information of the student network are used to determine the supervision loss, and the student network model is guided to imitate the significance information of the teacher network model, which effectively improves the detection accuracy of the student network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 A flowchart of an image feature recognition method provided in an embodiment of the present application;

[0064] Figure 2 A schematic diagram of the structure of an image feature recognition device provided in an embodiment of the present application;

[0065] Figure 3 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0067] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0068] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0069] Next, an image feature recognition method provided by an embodiment of the present application is described in detail. Figure 1 A flowchart of an image feature recognition method provided in an embodiment of the present application, the method comprising:

[0070] S101: Using the selected teacher network model and student network model to extract features of the sample image respectively, to obtain teacher features and student features.

[0071] In practical applications, there are various types of recognition tasks. For different types of recognition tasks, corresponding teacher network models and student network models can be set.

[0072] Therefore, in an embodiment of the present application, when a recognition task is obtained, a teacher network model and a student network model that match the recognition task can be selected; the teacher network model and the student network model are used to extract features of the sample image respectively to obtain teacher features and student features.

[0073] S102: Determine a significance threshold based on the ratio of teacher characteristics, student characteristics and the number of iterations.

[0074] In order to facilitate the analysis and mining of the image, in a specific implementation, the teacher features and the student features can be presented in the form of an attention map. In order to deeply mine the hidden information of the attention map, in an embodiment of the present application, a significance threshold can be set.

[0075] Considering that in practical applications, as the number of model training iterations increases, the teacher network model and the student network model learn more and more elements from each other, the value of the ratio can be gradually increased based on the increase in the number of iterations. For example, the initial value of the ratio can be set to 20%, and the ratio can be increased to 40%, 60%, 80%, and 100% at the 20th, 40th, 60th, and 80th epochs, respectively.

[0076] The teacher network model and the student network model have their own corresponding significance thresholds. For the sake of distinction, the significance threshold corresponding to the teacher network model can be called the first significance threshold, and the significance threshold corresponding to the student network model can be called the second significance threshold.

[0077] In a specific implementation, the first data distribution of teacher characteristics and the second data distribution of student characteristics can be statistically calculated; the first significance threshold corresponding to the teacher network is determined by using the ratio of the first data distribution to the current number of iterations; and the second significance threshold corresponding to the student network is determined by using the ratio of the second data distribution to the current number of iterations.

[0078] For example, for the first significance threshold The calculation method may include: first determine a ratio based on the number of training iterations During training, this ratio increases with the number of iterations. Assuming the total number of training epochs is 100, the initial value of the ratio can be set to =20% , at the 20th, 40th, 60th, and 80th epochs, Increase to 40%, 60%, 80%, and 100% respectively. Then, statistics Finally, based on the above data distribution histogram and ratio Decide , ensure that it is greater than or equal to The ratio of the elements is not less than the determined ratio The first significance threshold The calculation method and The calculation method is the same and will not be repeated here.

[0079] S103: Determine the teacher significance information corresponding to the teacher characteristics and the student significance information corresponding to the student characteristics according to the significance threshold.

[0080] The purpose of setting the significance threshold is to screen out features that help the training process converge to the global optimal value when the student network model imitates the teacher network model, thereby reducing the negative impact of non-significant information on the training process.

[0081] Therefore, in a specific implementation, the feature value of the teacher feature that is less than the first significance threshold can be adjusted to zero, and the adjusted teacher feature can be used as the teacher significance information; the feature value of the student feature that is less than the second significance threshold can be adjusted to zero, and the adjusted student feature can be used as the student significance information. The significance information can be presented in the form of a significance map.

[0082] For example, the attention map is , the adaptive threshold is , attention map The corresponding saliency map is , saliency map The corresponding mask image is . Any element in is defined as , The corresponding elements in are , The corresponding elements in are ,So , .

[0083] Using the above method, we can get the teacher network threshold value Saliency map of attention map and mask map , the student network thresholded Saliency map of attention map and mask map . Further, define , The first Attention Map and student network Attention Map The common significant regions of represents pixel-by-pixel multiplication. During training, Based on the feature map The data distribution and the number of training iterations are determined dynamically; Based on the feature map The data distribution and the number of training iterations are dynamically determined, where =1, 2, 3, 4, 5.

[0084] S104: Train the student network model using teacher characteristics, student characteristics, teacher saliency information, and student saliency information.

[0085] In the traditional method, the student network model is trained using teacher features and student features. In the embodiment of the present application, in order to improve the accuracy of the student network model training, the student network model can be trained using teacher features, student features, teacher saliency information and student saliency information.

[0086] In the specific implementation, the loss function of the student network model can be constructed according to the teacher characteristics, the student characteristics, the teacher significance information and the student significance information; and the student network model can be trained using the student characteristics, the student significance information and the loss function.

[0087] Considering the practical application, the low-level feature map of the neural network mainly expresses the low-level features of the image, such as corner points, straight lines, etc., while the high-level feature map mainly expresses high-level features, such as high-level features such as faces and components. High-level features are more important to the final model performance, so different weight values ​​can be used for the low-level supervision loss and the high-level supervision loss to strengthen the student network model to learn the high-level features of the teacher network model. That is, in an embodiment of the present application, different weights can be set for the teacher's significant information and the student's significant information at different levels of the model. The higher the level, the greater the weight.

[0088] Since the teacher's significant information and the student's significant information are extracted in this application, the loss function of the constructed student network model can include three parts, including determining the initial loss function corresponding to the student network and the distillation loss function corresponding to the features according to the teacher's characteristics and the student's characteristics; determining the distillation loss function corresponding to the significant information according to the teacher's significant information and the student's significant information. Based on the initial loss function, the distillation loss function corresponding to the features and the distillation loss function corresponding to the significant information, the loss function of the student network model can be constructed.

[0089] Taking the face recognition task as an example, in the specific implementation, the initial loss function of the teacher network can be defined as , the initial loss function of the student network is , is the softmax loss of the teacher network output, is the softmax loss of the student network output. The type of loss function in the embodiment of the present application is not limited to the softmax loss, and other types of loss functions may also be used, such as ArcFace loss or CosFace loss.

[0090] Define the output features of the FC1 layer of the teacher network And the output features of the FC1 layer of the student network The distillation loss function between ,

[0091] ,

[0092] in, express The number of elements in , Represents the 2-norm of a vector.

[0093] The distillation loss function between the saliency information is defined as ,in =1, 2, 3, 4, 5.

[0094] ,

[0095] in, Represents the matrix Norm.

[0096] Combining the above function definitions, we can get the loss function of the teacher network as follows: ; The loss function of the student network is ,

[0097] in, =0.10.

[0098] In the above formula, Indicates the weights of significant information at different levels of the model. In practical applications, you can set =0.02， =0.04， =0.06， =0.08， =0.10 .

[0099] After determining the loss function, the training of the student network model can refer to the conventional model training method. The training of the network model requires setting some basic parameters. In the specific implementation, the data enhancement strategy can use a 50% probability to horizontally flip the input image. In addition, by subtracting 127.5 from each pixel value of the original image and then dividing it by 127.5, all image values ​​​​can be normalized to [-1, +1]. In this embodiment of the present application, training can be performed on a server with 8 GPUs, and the batch size is 512. This embodiment of the application uses SGD(m) to optimize the weight parameters. The initial learning rate is set to 0.1, the momentum is set to 0.9, and the weight decay is , the total number of training epochs is set to 120, and the learning rate is decreased by 1 / 10 at the 30th and 60th epochs.

[0100] S105: Analyze the image to be processed using the trained student network model to identify features contained in the image to be processed.

[0101] The image feature recognition method provided in the embodiment of the present application can be applied to various types of image tasks. Common image tasks include face recognition tasks, digital classification tasks, etc.

[0102] When the task corresponding to the image to be processed is a face recognition task, the trained student network model can be used to analyze the image to be processed to identify the facial features contained in the image to be processed.

[0103] When the task corresponding to the image to be processed is a digital classification task, the trained student network model can be used to analyze the image to be processed to extract the digital features contained in the image to be processed.

[0104] Taking the face recognition task as an example, the sample image is the face training set. In order to train a high-precision face recognition model, the face recognition training set needs to have sufficient depth and breadth. Among them, depth refers to the number of face images of each person (ID) (the more face images, the better, and these images should come from a variety of scenes, including factors such as changes in lighting, changes in face posture, and changes in age); breadth refers to the number of people included in the training set (the more people, the better, and different races, different genders, different skin colors, etc. should be included). Generally, when the face training set contains more training data, the model trained using the method proposed in the embodiment of the present application has higher accuracy.

[0105] The facial images in the training data set used in the embodiment of the present application are grouped according to individual identity (ID), that is, the facial images of one person are grouped into the same group. Then each group (each person) is given a category label. Assuming that the data set contains C people, a simple method of assigning category labels is from 1 to C; other category label assignment methods can also be used, for example: 0 to C-1. The embodiment of the present application does not limit the way labels are assigned. In order to train a high-precision face recognition system, it is recommended that the training set contain at least 10,000 facial images of people, of which each person has no less than 10 facial images.

[0106] Generally speaking, the input image of the face recognition algorithm based on deep learning is a three-channel (R, G, B three channels) color image of a fixed size. The face images in the training set of the embodiment of the present application are also three-channel (R, G, B three channels) color images of a fixed size. The size of each face image is 112*112*3 pixels, where the first "112" represents the width of the image, the second "112" represents the height of the image, and "3" represents 3 channels. The face image is a normalized face image preprocessed by the face alignment algorithm. When the size of the input image is not 112*112*3 pixels, the embodiment of the present application can be simply expanded to adapt to changes in the size of the input image. When the input image is a single-channel grayscale image or other forms of image, the embodiment of the present application can be simply expanded to adapt to changes in the input image.

[0107] In an embodiment of the present application, the teacher network model can be referred to as the teacher network, and the student network model can be referred to as the student network. In order to construct an effective training framework based on knowledge distillation technology, the computational complexity of the teacher network should be greater than that of the student network, and the teacher network and the student network should have the same or similar basic structure. Under the guidance of this basic principle, the embodiment of the present application adopts ResNet50 as the teacher network and ResNet18 as the student network. Both ResNet50 and ResNet18 use residual structure as the basic structural unit, and the specific implementation methods are slightly different. Although the example provided in the embodiment of the present application uses ResNet50 as the teacher network and ResNet18 as the student network, the embodiment of the present application is not limited to the specific teacher network structure and student network structure.

[0108] The embodiment of the present application redesigns the network structure of ResNet50 and ResNet18 so that the redesigned ResNet50 and ResNet18 can be applied to face recognition tasks.

[0109] The redesigned ResNet50 in the embodiment of the present application includes the first 5 groups of convolutions conv1, conv2_x, conv3_x, conv4_x, conv5_x from ResNet50 and the newly added BN1 (Batch Normalization, batch normalization layer), dropout, FC1 (Fully-connected layer, the first fully connected layer), BN2, FC2 (Fully-connected layer, the second fully connected layer) and loss function layer. The input of the FC1 layer is the output feature map of the conv5_x layer, which is sequentially output through the BN1 layer and the dropout layer, and the output of the FC1 layer is a 512-dimensional feature. The input of the FC2 layer is the output feature of the FC1 layer through the BN2 layer. The output dimension of the FC2 layer is the same as the number of categories in the training set, and one dimension corresponds to one category. The original conv1 is a 7*7 convolution layer with a convolution step size of 2. In fact, conv1 is a 7*7*3 convolution, where 3 means that the number of input channels of conv1 is 3 channels. In the embodiment of the present application, the conv1 layer is a 3*3 convolution layer, and the convolution step size is 1. The backbone network of the teacher network defined in the embodiment of the present application includes: the first 5 groups of convolutions conv1, conv2_x, conv3_x, conv4_x, conv5_x of ResNet50 and the newly added BN1 (Batch Normalization, batch normalization layer), dropout, FC1 (Fully-connected layer, the first fully connected layer).

[0110] The ResNet18 used in the embodiment of the present application includes the first 5 groups of convolutions conv1, conv2_x, conv3_x, conv4_x, conv5_x from ResNet18 and the newly added BN (Batch Normalization, batch normalization layer), dropout, FC1 (Fully-connected layer, the first fully connected layer), BN, FC2 (Fully-connectedlayer, the second fully connected layer) and loss function layer. The input of the FC1 layer is the output feature map of the conv5_x layer, which passes through the output of the BN1 layer and the dropout layer in sequence, and the output of the FC1 layer is a 512-dimensional feature. The input of the FC2 layer is the output feature of the FC1 layer through the BN2 layer. The output dimension of the FC2 layer is the same as the number of categories in the training set, and one dimension corresponds to one category. The original conv1 is a 7*7 convolution layer with a convolution step of 2. In fact, conv1 is a 7*7*3 convolution, where 3 means that the number of input channels of conv1 is 3 channels. In the embodiment of the present application, the conv1 layer is a 3*3 convolution layer, and the convolution step size is 1. The backbone network of the student network defined in the embodiment of the present application includes: the first 5 groups of convolutions conv1, conv2_x, conv3_x, conv4_x, conv5_x of ResNet18 and the newly added BN1 (Batch Normalization, batch normalization layer), dropout, FC1 (Fully-connected layer, the first fully connected layer).

[0111] After configuring the teacher network and the student network, the feature map can be generated using the teacher network and the student network.

[0112] Generally speaking, the feature map of the teacher network has stronger expression ability. Letting the feature map of the student network imitate the feature map of the teacher network can further improve the performance of the student network. In order to achieve the imitation of the teacher network by the student network, the embodiment of the present application can select the same number of feature maps from the output feature map of the teacher network and the output feature map of the student network, respectively, and pair the extracted feature maps (a feature map of the teacher network and a feature map of the student network are combined into a pair). The embodiment of the present application pairs the output feature maps of the first 5 groups of convolutions (conv1, conv2_x, conv3_x, conv4_x and conv5_x) of ResNet50 and the output feature maps of the first 5 groups of convolutions (conv1, conv2_x, conv3_x, conv4_x and conv5_x) of ResNet18. The output feature maps of the first 5 groups of convolutions of the teacher network ResNet50 are defined as ,in, =1,2,3,4,5, Represents the sequence number of each group, Represents the height of the feature map, represents the width of the feature map, represents the number of channels of the feature map, Represents the Teachers Network.

[0113] Specifically, The size is 112*112*64, The size is 56*56*256, The size is 28*28*512, The size is 14*14*1024, The size is 7*7*2048.

[0114] Define the output feature maps of the first five groups of convolutions of the student network ResNet18 as ,in =1, 2, 3, 4, 5; Represents the sequence number of each group. Represents the height of the feature map, represents the width of the feature map, represents the number of channels of the feature map, Represents the student network.

[0115] Specifically, The size is 112*112*64, The size is 56*56*128, The size is 28*28*256, The size is 14*14*512, The size is 7*7*1024.

[0116] The above feature maps are combined into 5 feature map pairs ,in, =1, 2, 3, 4, 5; Represents the sequence number of each group. In this configuration, The width and are equal in width, Height and are equal in height, The number of channels and The number of channels is not equal.

[0117] In this embodiment, The width and are equal in width, The height and are equal in height, where =1, 2, 3, 4, 5; Represents the sequence number of each group. When the teacher network and the student network adopt other network structures, The width and The widths may be unequal. The height and The heights of the feature maps may also be unequal. In this case, image processing methods (e.g., image scaling) or additional convolution operations can be used to make the feature maps The width and height are equal.

[0118] Based on the feature map, an attention map can be generated. To the corresponding attention map The mapping is .

[0119] Mapping An optional implementation of is as follows: feature map The value of any point in , feature map The corresponding attention map The corresponding points in .

[0120] in, , , .

[0121] Mapping Another optional implementation of is as follows: feature map The value of any point in , feature map The corresponding attention map The corresponding points in .

[0122] in, , , .

[0123] Then, the teacher network No. Feature Map The corresponding attention map is , student network No. Feature Map The corresponding attention map is .

[0124] Feature map pair The generated attention map pair ,in =1, 2, 3, 4, 5; Represents the sequence number of each group. The width and are equal in width, The height and The heights of the two channels are equal, and the number of output channels is the same (all are one channel).

[0125] In order to extract saliency information from the attention maps of the teacher network and the student network and ensure that the training process of the neural network can converge to the global optimal value, an embodiment of the present application sets an adaptive threshold to calculate the saliency map of the attention map.

[0126] Specifically, during the network training process, the embodiment of the present application adaptively adjusts the threshold according to the data distribution of the attention map and the number of training iterations, and uses the adaptive threshold to calculate the saliency map of the attention map. The process of determining the saliency information according to the saliency threshold has been introduced in S103 and will not be repeated here. The calculation method of the loss function has been introduced in the above and will not be repeated here.

[0127] Take the digital classification task as an example. The digital classification task is used to extract the numbers contained in the image. The sample image is a data set containing numbers. The difference between the implementation of the digital classification task and the implementation of the face recognition task lies in: the preparation of sample images, the network configuration of the student network and the teacher network, the loss function calculation method and the training strategy of the neural network. Next, these four aspects will be introduced in turn.

[0128] Sample image preparation:

[0129] For digital classification tasks, the data set is an image and a category label. For example, for the digital classification problem from 0 to 9, the input set contains color images or grayscale images of the numbers 0 to 9. The size of each image is 32x32 pixels, and the label of each image is the corresponding number. The training samples are divided into 10 subsets, each subset corresponds to a number, and each subset contains at least one image. In order to train a high-precision model, each subset should contain enough samples, such as thousands of samples or tens of thousands of samples. In addition, the training samples should be diverse, for example: including background changes, font changes, lighting changes, etc.

[0130] Network configuration for student network and teacher network:

[0131] The embodiment of the present application uses ResNet50 as the teacher network and ResNet18 as the student network.

[0132] The ResNet50 in the embodiment of the present application includes the first 5 groups of convolutions conv1, conv2_x, conv3_x, conv4_x, conv5_x and average pool layers from ResNet50, FC (Fully-connected layer) and loss function layer (the embodiment of the present application adopts the softmax loss function.

[0133] In fact, the loss function used for face recognition can also be used for digital classification tasks. The input of the FC layer is the output feature of the average pool layer. The dimension of the output feature of the FC layer is the same as the number of categories in the training set (for digital classification tasks from 0 to 9, the number of categories is 10), and one dimension corresponds to one category. The original conv1 is a 7*7 convolution layer with a convolution step of 2; in this embodiment of the application, the conv1 layer is a 3*3 convolution layer with a convolution step of 1. In addition, the embodiment of the application removes the 3*3 max pool layer of conv2_x. In this embodiment of the application, the backbone network of the teacher network includes: the first 5 groups of convolutions conv1, conv2_x, conv3_x, conv4_x, and conv5_x of ResNet50.

[0134] The ResNet18 used in the embodiment of the present application includes the first 5 groups of convolutions conv1, conv2_x, conv3_x, conv4_x, conv5_x and average pool layer, FC (Fully-connected layer) and loss function layer (the embodiment of the present application uses softmax loss function) from ResNet18. The input of the FC layer is the output feature of the average pool layer. The dimension of the output feature of the FC layer is the same as the number of categories in the training set, and one dimension corresponds to one category. The original conv1 is a 7*7 convolution layer with a convolution step of 2; in the embodiment of the present application, the conv1 layer is a 3*3 convolution layer with a convolution step of 1. In addition, the 3x3 max pool layer of conv2_x is removed in the embodiment of the present application. In this embodiment, the backbone network of the student network includes: the first 5 groups of convolutions conv1, conv2_x, conv3_x, conv4_x, conv5_x of ResNet18.

[0135] Loss function calculation method:

[0136] The loss function for the teacher network is defined as , the loss function of the student network is , is the softmax loss of the teacher network output, is the softmax loss of the student network output.

[0137] The distillation loss between saliency maps is defined as in =1, 2, 3, 4, 5.

[0138] ,

[0139] in, Represents the matrix Norm.

[0140] Combining the above function definitions, we can get the loss function of the teacher network as follows: ; The loss function of the student network is ,

[0141] In the above formula, Indicates the weights of significant information at different levels of the model. In practical applications, you can set =0.02， =0.04， =0.06， =0.08， =0.10 .

[0142] Neural network training strategy:

[0143] In the embodiment of the present application, SGD(m) can be used to optimize the weight parameters, the initial learning rate is set to 0.1, the momentum is set to 0.9, and the weight decay is , the total number of training epochs is set to 120, and the learning rate is reduced by 1 / 10 at the 30th and 60th epochs. By subtracting 127.5 from each pixel value of the original image and then dividing it by 127.5, all image values ​​are normalized to [-1, +1]. The embodiment of the present application can be trained on a server with 1 GPUs, and the batch size is 32.

[0144] The embodiment of the present application takes into account the salient features of the attention map of the intermediate layer feature map of the teacher network and the student network, and uses the salient features of the teacher network and the student network to calculate the supervision loss of the intermediate layer, which promotes the intermediate layer output of the student network to better imitate the intermediate layer output of the teacher network; the traditional technology does not mine the salient features of the attention map. The present application uses a dynamic threshold to extract the salient features of the attention map. And different weights are given to the intermediate layer supervision losses of different depths to ensure that the loss function from the deep layer of the network has a greater impact on the overall loss, while the loss function from the shallow layer of the network has a smaller impact on the overall loss.

[0145] It can be seen from the above technical scheme that the selected teacher network model and student network model are used to extract features of the sample image respectively to obtain teacher features and student features; the significance threshold is determined based on the ratio corresponding to the teacher features, student features and the number of iterations; the significance threshold can be used to deeply explore the hidden information between the teacher features and the student features, so the teacher significance information corresponding to the teacher features and the student significance information corresponding to the student features can be determined according to the significance threshold. The student network model is trained using the teacher features, the student features, the teacher significance information and the student significance information to obtain a student network model with higher detection accuracy. The trained student network model is used to analyze the image to be processed to identify the features contained in the image to be processed. In this technical scheme, the significance threshold is used to generate the significance information of the attention map, which improves the depth of image feature mining. On the basis of the teacher features and the student features, the significance information of the teacher network and the significance information of the student network are used to determine the supervision loss, and the student network model is guided to imitate the significance information of the teacher network model, which effectively improves the detection accuracy of the student network model.

[0146] Figure 2 A schematic diagram of the structure of an image feature recognition device provided in an embodiment of the present application, comprising an extraction unit 21, a threshold determination unit 22, an information determination unit 23, a training unit 24 and a recognition unit 25;

[0147] An extraction unit 21 is used to extract features of the sample image using the selected teacher network model and student network model to obtain teacher features and student features;

[0148] A threshold determination unit 22, configured to determine a significance threshold based on a ratio of a teacher feature, a student feature, and a number of iterations;

[0149] An information determination unit 23, used to determine the teacher significance information corresponding to the teacher characteristics and the student significance information corresponding to the student characteristics according to the significance threshold;

[0150] A training unit 24, used to train the student network model using teacher characteristics, student characteristics, teacher saliency information, and student saliency information;

[0151] The recognition unit 25 is used to analyze the image to be processed by using the trained student network model to recognize the features contained in the image to be processed.

[0152] Optionally, extracting the unit includes selecting a subunit and obtaining the subunit;

[0153] A selection subunit is used to select a teacher network model and a student network model that match the recognition task when a recognition task is obtained;

[0154] A subunit is obtained, which is used to extract features of the sample image using the teacher network model and the student network model to obtain teacher features and student features.

[0155] Optionally, the threshold determination unit includes a statistical subunit, a first determination subunit and a second determination subunit;

[0156] A statistical subunit, used for statistically analyzing a first data distribution of teacher characteristics and a second data distribution of student characteristics;

[0157] A first determination subunit is used to determine a first significance threshold corresponding to the teacher network by using a ratio of the first data distribution to the current number of iterations;

[0158] The second determination subunit is used to determine a second significance threshold corresponding to the student network by using the ratio of the second data distribution to the current number of iterations.

[0159] Optionally, the information determination unit includes a first adjustment subunit and a second adjustment subunit;

[0160] A first adjustment subunit is used to adjust the characteristic values ​​of the teacher characteristics that are less than the first significance threshold to zero, and use the adjusted teacher characteristics as teacher significance information;

[0161] The second adjustment subunit is used to adjust the feature values ​​of the student features that are less than the second significance threshold to zero, and use the adjusted student features as student significance information.

[0162] Optionally, the training unit includes a construction subunit and a completion subunit;

[0163] Constructing a subunit for constructing a loss function of a student network model according to teacher characteristics, student characteristics, teacher saliency information, and student saliency information; wherein the teacher saliency information and the student saliency information correspond to different weights at different levels of the model;

[0164] The completion subunit is used to complete the training of the student network model using student features, student saliency information and loss function.

[0165] Optionally, the construction subunit is used to determine the initial loss function corresponding to the student network and the distillation loss function corresponding to the features based on the teacher characteristics and the student characteristics; determine the distillation loss function corresponding to the significance information based on the teacher significance information and the student significance information; and construct the loss function of the student network model based on the initial loss function, the distillation loss function corresponding to the features, and the distillation loss function corresponding to the significance information.

[0166] Optionally, when the task corresponding to the image to be processed is a face recognition task, the recognition unit is used to analyze the image to be processed using a trained student network model to identify facial features contained in the image to be processed.

[0167] Optionally, when the task corresponding to the image to be processed is a digital classification task, the recognition unit is used to analyze the image to be processed using a trained student network model to extract digital features contained in the image to be processed.

[0168] Figure 2 The description of the features in the corresponding embodiments can be found in Figure 1 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0169] It can be seen from the above technical scheme that the selected teacher network model and student network model are used to extract features of the sample image respectively to obtain teacher features and student features; the significance threshold is determined based on the ratio corresponding to the teacher features, student features and the number of iterations; the significance threshold can be used to deeply explore the hidden information between the teacher features and the student features, so the teacher significance information corresponding to the teacher features and the student significance information corresponding to the student features can be determined according to the significance threshold. The student network model is trained using the teacher features, the student features, the teacher significance information and the student significance information to obtain a student network model with higher detection accuracy. The trained student network model is used to analyze the image to be processed to identify the features contained in the image to be processed. In this technical scheme, the significance threshold is used to generate the significance information of the attention map, which improves the depth of image feature mining. On the basis of the teacher features and the student features, the significance information of the teacher network and the significance information of the student network are used to determine the supervision loss, and the student network model is guided to imitate the significance information of the teacher network model, which effectively improves the detection accuracy of the student network model.

[0170] Figure 3 A structural diagram of an electronic device provided in an embodiment of the present application, such as Figure 3 As shown, the electronic device includes: a memory 30 for storing a computer program;

[0171] The processor 31 is used to implement the steps of the image feature recognition method in the above embodiment when executing the computer program.

[0172] The electronic device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer.

[0173] Among them, the processor 31 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 31 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 31 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 31 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 31 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0174] The memory 30 may include one or more computer-readable storage media, which may be non-transitory. The memory 30 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 30 is at least used to store the following computer program 301, wherein, after the computer program is loaded and executed by the processor 31, the relevant steps of the image feature recognition method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 30 may also include an operating system 302 and data 303, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 302 may include Windows, Unix, Linux, etc. The data 303 may include but is not limited to teacher characteristics, student characteristics, ratios corresponding to the number of iterations, etc.

[0175] In some embodiments, the electronic device may further include a display screen 32 , an input / output interface 33 , a communication interface 34 , a power source 35 , and a communication bus 36 .

[0176] Those skilled in the art will understand that Figure 3 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure.

[0177] It is understandable that if the image feature recognition method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk and other media that can store program code.

[0178] Based on this, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image feature recognition method are implemented.

[0179] The functions of each functional module of the computer-readable storage medium described in the embodiment of the present application can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0180] The above is a detailed introduction to an image feature recognition method, device, equipment and computer-readable storage medium provided in the embodiments of the present application. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0181] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0182] The above is a detailed introduction to the image feature recognition method, device, equipment and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the embodiments of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for image feature recognition, characterized in that: include: The selected teacher network model and student network model are used to extract features of the sample images respectively to obtain teacher features and student features; Determining a significance threshold based on a ratio of the teacher characteristic, the student characteristic, and the number of iterations; Determining teacher significance information corresponding to the teacher characteristics and student significance information corresponding to the student characteristics according to the significance threshold; Training the student network model using the teacher characteristics, the student characteristics, the teacher significance information, and the student significance information; Analyzing the image to be processed by using the trained student network model to identify the features contained in the image to be processed; Wherein, determining the significance threshold based on the ratio of the teacher characteristic, the student characteristic and the number of iterations comprises: Counting the first data distribution of the teacher's characteristics and the second data distribution of the student's characteristics; Determine a first significance threshold corresponding to the teacher network model by using a ratio of the first data distribution to the current number of iterations; Determine a second significance threshold corresponding to the student network model by using a ratio of the second data distribution to the current number of iterations; Determining the teacher significance information corresponding to the teacher characteristics and the student significance information corresponding to the student characteristics according to the significance threshold comprises: Adjusting the feature values ​​of the teacher features that are less than the first significance threshold to zero, and using the adjusted teacher features as teacher significance information; Adjusting the feature values ​​of the student features that are less than the second significance threshold to zero, and using the adjusted student features as student significance information; The training of the student network model by using the teacher characteristics, the student characteristics, the teacher significance information, and the student significance information includes: Constructing a loss function of a student network model according to the teacher characteristics, the student characteristics, the teacher significance information and the student significance information; wherein the teacher significance information and the student significance information correspond to different weights at different levels of the model; Training the student network model using the student characteristics, the student significance information and the loss function; The loss function of the student network model is: ； ； ； ； in, represents the loss function of the student network model, represents the initial loss function of the student network model, represents the weight coefficient, Represents the output features of the FC1 layer of the teacher network model And the output features of the FC1 layer of the student network model The distillation loss function between Indicates the weights corresponding to the significant information at different levels of the model, represents the distillation loss function between saliency information, represents the threshold value of the teacher network model Saliency map of attention map, represents the threshold value of the student network model Saliency map of attention map, represents the width of the attention map, Represents the height of the attention map; express The number of elements in , represents the 2-norm of a vector, Represents a vector norm; represents pixel-by-pixel multiplication, represents the first Attention Map and the student network model Attention Map The common significant regions of represents the threshold value of the teacher network model The mask map of the attention map, represents the threshold value of the student network model Mask map of the attention map.

2. The image feature recognition method according to claim 1, characterized in that: The method of using the selected teacher network model and student network model to extract features of the sample image respectively, and obtaining the teacher features and the student features includes: When a recognition task is obtained, a teacher network model and a student network model matching the recognition task are selected; The teacher network model and the student network model are used to extract features of the sample image respectively to obtain teacher features and student features.

3. The image feature recognition method according to claim 1 or 2, characterized in that: In the case where the task corresponding to the image to be processed is a face recognition task, the use of the trained student network model to analyze the image to be processed to identify the features contained in the image to be processed includes: The trained student network model is used to analyze the image to be processed to identify the facial features contained in the image to be processed.

4. The image feature recognition method according to claim 1 or 2, characterized in that: In the case where the task corresponding to the image to be processed is a digital classification task, the use of the trained student network model to analyze the image to be processed to identify the features contained in the image to be processed includes: The trained student network model is used to analyze the image to be processed to extract digital features contained in the image to be processed.

5. An image feature recognition device, characterized in that: It includes an extraction unit, a threshold determination unit, an information determination unit, a training unit and a recognition unit; The extraction unit is used to extract features of the sample image using the selected teacher network model and student network model to obtain teacher features and student features; The threshold determination unit is used to determine a significance threshold based on the ratio of the teacher feature, the student feature and the number of iterations; The information determination unit is used to determine the teacher significance information corresponding to the teacher characteristics and the student significance information corresponding to the student characteristics according to the significance threshold; The training unit is used to train the student network model by using the teacher characteristics, the student characteristics, the teacher significance information, and the student significance information; The recognition unit is used to analyze the image to be processed by using the trained student network model to identify the features contained in the image to be processed; Wherein, determining the significance threshold based on the ratio of the teacher characteristic, the student characteristic and the number of iterations comprises: Counting the first data distribution of the teacher's characteristics and the second data distribution of the student's characteristics; Determine a first significance threshold corresponding to the teacher network model by using a ratio of the first data distribution to the current number of iterations; Determine a second significance threshold corresponding to the student network model by using a ratio of the second data distribution to the current number of iterations; Determining the teacher significance information corresponding to the teacher characteristics and the student significance information corresponding to the student characteristics according to the significance threshold comprises: Adjusting the feature values ​​of the teacher features that are less than the first significance threshold to zero, and using the adjusted teacher features as teacher significance information; Adjusting the feature values ​​of the student features that are less than the second significance threshold to zero, and using the adjusted student features as student significance information; The training of the student network model by using the teacher characteristics, the student characteristics, the teacher significance information, and the student significance information comprises: Constructing a loss function of a student network model according to the teacher characteristics, the student characteristics, the teacher significance information and the student significance information; wherein the teacher significance information and the student significance information correspond to different weights at different levels of the model; Training the student network model using the student characteristics, the student significance information and the loss function; The loss function of the student network model is: ； ； ； ； in, represents the loss function of the student network model, represents the initial loss function of the student network model, represents the weight coefficient, Represents the output features of the FC1 layer of the teacher network model And the output features of the FC1 layer of the student network model The distillation loss function between Indicates the weights corresponding to the significant information at different levels of the model, represents the distillation loss function between saliency information, represents the threshold value of the teacher network model Saliency map of attention map, represents the threshold value of the student network model Saliency map of attention map, represents the width of the attention map, Represents the height of the attention map; express The number of elements in , represents the 2-norm of a vector, Represents a vector norm; represents pixel-by-pixel multiplication, represents the first Attention Map and the student network model Attention Map The common significant regions of represents the threshold value of the teacher network model The mask map of the attention map, represents the threshold value of the student network model Mask map of the attention map.

6. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the image feature recognition method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image feature recognition method according to any one of claims 1 to 4 are implemented.

Images