Training method and training device of neural network for object recognition

Abstract

The present invention relates to a training method and apparatus for a neural network for object recognition. A training method comprises: inputting a training image set containing an object to be recognized into the neural network to extract student features for each image sample, the training image set comprising a set of normal image samples and a set of variation image samples; dividing the image samples in the training image set into simple samples and difficult samples based on the extracted student features; for each of the normal image samples and the variation image samples: performing a transfer transfer based on the division on the student features of the image sample by a corresponding transfer transfer adapter to obtain a transferred student feature; calculating a distillation loss of the transferred student feature of the image sample relative to a teacher feature extracted from another corresponding image sample; classifying the image sample based on the student feature by a corresponding classifier; and calculating a classification loss for the image sample; calculating a total loss associated with the training image set based on the distillation loss and the classification loss calculated for all image samples; and updating parameters of the neural network according to the calculated total loss.

Description

Technical Field

[0001] The present invention relates to a neural network for object recognition, and more particularly to a training method and a training device for a neural network model. Background Art

[0002] In recent years, object detection / recognition / matching / tracking in static images or a series of moving images (such as videos) has been widely and importantly applied in the fields of image processing, computer vision, and pattern recognition. Face / object recognition is one of the most important computer vision tasks, whose goal is to identify or verify a specific person / object based on an input photo / video. In recent years, neural network models for face / object recognition, especially convolutional neural network (CNN) models, have made breakthrough progress in significantly improving performance. Given a training dataset, the CNN training process uses a general CNN architecture as a feature extractor to extract features from the training images. Then, various loss functions are used to calculate loss data for supervised training of the CNN model. Therefore, once the CNN architecture is selected, the performance of the object recognition model is driven by the loss function and the training dataset.

[0003] However, existing public training datasets mostly consist of high-quality normal images, and CNNs trained on such datasets generally have good recognition capabilities for normal images. However, when input images exhibit a variety of variations in object pose, ethnicity, resolution, lighting, and other factors, the recognition performance of object recognition models degrades significantly. This is because object recognition models fail to learn robust feature representations from training datasets with limited variation. Therefore, the main challenge for current object recognition models is maintaining high recognition performance in unconstrained scenarios where input images may exhibit a wide range of variations.

[0004] The conventional solution is to use image data synthesis or augmentation methods to generate specially modified images and add them to the training dataset for object recognition model training. However, due to differences in feature domains, directly training on synthesized or augmented images may make it difficult for the network to converge. Even if convergence occurs, the resulting model's recognition performance on synthesized or augmented images may improve while its recognition performance on normal images decreases, i.e., the so-called "negative transfer" occurs.

[0005] For example, in "Improving Face Recognition from Hard Samples via Distribution Distillation Loss," Yuge Huang et al. proposed a distillation loss function based on similarity distribution to narrow the performance gap between easy and hard samples. This method first constructs two similarity distributions using advanced classifiers such as Arcface: a teacher distribution for easy samples and a student distribution for hard samples. A distribution-driven loss function is then used to constrain the student distribution to approximate the teacher distribution, minimizing the overlap between positive and negative pairs in the student distribution. However, since easy and hard samples are divided offline, their similarity distributions actually have a fixed difference. When there is a significant difference between easy and hard samples, their actual feature domains also differ significantly. Using only a similarity distribution-driven loss function to push the student distribution of hard samples closer to the teacher distribution of easy samples can make it difficult for the network to converge, preventing the final trained model from being obtained, and may also lead to negative knowledge transfer. This method assumes that the teacher distribution of easy samples is optimal in terms of recognition ability and is used to supervise the learning of difficult samples. However, since easy samples and difficult samples actually act on the same classification loss function at the same time, difficult samples will produce a large gradient that affects the training model to shift towards its feature domain, resulting in the final training model cannot guarantee the optimality on both easy and difficult samples. Summary of the Invention

[0006] This paper proposes a novel neural network training method based on transitive knowledge distillation and bilateral classification. By introducing a transitive domain as a transition bridge, different feature domains are simultaneously brought closer together from two directions, thus avoiding large inter-domain differences and narrowing the performance gap between normal and altered images. Furthermore, by employing a curriculum-based learning strategy for bilateral classification, the learning focus is gradually shifted from normal images to altered images during the learning process, reducing the negative impact of altered images and raising the upper limit of representation learning.

[0007] According to one aspect of the present invention, a training method for a neural network for object recognition is provided, comprising: inputting a training image set containing an object to be recognized into the neural network to extract student features for each image sample, the training image set comprising a group of normal image samples and a group of variation image samples; dividing the image samples in the training image set into simple samples and difficult samples based on the extracted student features; for each of the normal image samples and the variation image samples: performing a transfer transfer based on the division on the student features of the image sample by a corresponding transfer transfer adapter to obtain a transferred student feature; calculating the distillation loss of the transferred student feature of the image sample relative to the teacher feature extracted from another corresponding image sample; classifying the image sample based on the student feature by a corresponding classifier; and calculating the classification loss of the image sample, calculating the total loss associated with the training image set based on the distillation loss and the classification loss calculated for all image samples; and updating the parameters of the neural network according to the calculated total loss.

[0008] According to one aspect of the present invention, there is also provided a training device for a neural network for object recognition, comprising a unit for executing each step of the above training method.

[0009] According to one aspect of the present invention, an iterative training method for training a neural network for multiple changing scenarios using the above training method is also provided.

[0010] According to one aspect of the present invention, a neural network trained using the above training method is also provided.

[0011] According to one aspect of the present invention, a non-transitory computer-readable storage medium storing executable instructions is also provided. When the executable instructions are executed, they can be used to implement the above training method. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present invention may be better understood by referring to the detailed description given below in conjunction with the accompanying drawings, wherein the same or similar reference numerals are used throughout the drawings to represent the same or similar elements. All drawings, together with the following detailed description, are incorporated into and form a part of this specification and are used to further illustrate embodiments of the present invention and to explain the principles and advantages of the present invention. Among them:

[0013] Figure 1 is a schematic diagram showing how to fuse face images of different variations using transitive transfer;

[0014] Figure 2 is a schematic diagram summarizing the neural network training method according to the present invention;

[0015] Figure 3 The pseudo code of dynamic sample partitioning according to the present invention is shown;

[0016] Figure 4 shows how the threshold used to define easy examples changes during training;

[0017] Figure 5 is a schematic diagram showing conventional direct migration and transfer migration according to the present invention;

[0018] Figure 6 1 is a schematic diagram showing classification learning of normal images and changed images;

[0019] Figure 7 is a schematic diagram of an iterative training method according to the present invention;

[0020] Figure 8 is a block diagram of a configuration of a training device according to the present invention;

[0021] Figure 9 is a flow chart of a training method according to the present invention;

[0022] Figure 10 is a flow chart of a training process according to a first example of the present invention;

[0023] Figure 11 is a flow chart of a training process according to a second example of the present invention;

[0024] Figure 12 is a flowchart of a training process according to a third example of the present invention;

[0025] Figure 13 is a flowchart of a training process according to a fourth example of the present invention;

[0026] Figure 14 A block diagram illustrating an exemplary hardware configuration of a computer system capable of implementing an embodiment of the present invention.

[0027] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. DETAILED DESCRIPTION

[0028] Various exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all implementations of the embodiments are described in this specification. However, it should be noted that many implementation-specific settings can be made according to specific needs when implementing the embodiments of the present invention.

[0029] In addition, in order to avoid obscuring the present invention with unnecessary details, only the processing steps and / or device structures closely related to the technical solution of the present invention are shown in the accompanying drawings. The following description of the exemplary embodiments is merely illustrative and is not intended to limit the present invention and its applications.

[0030]

Overview

[0031] Before describing each embodiment in detail, the gist of the present invention will be described. The present invention uses a training image set including normal image samples and variation image samples containing objects to be recognized to train an object recognition model to achieve excellent recognition performance in unconstrained scenarios.

[0032] In the context of the present invention, an "image sample" in a training image set may refer to any image, such as a color image, a grayscale image, or the like. The type and format of the image sample are not particularly limited, as long as such an image can be processed so as to detect whether the image contains an object. Furthermore, the image may be the original image or a processed version thereof, such as a version of the image that has undergone preliminary filtering or pre-processing before performing the operations of the present invention on the image.

[0033] Furthermore, an image sample containing an object to be identified means that the image sample contains an object image of the object. This object image may also sometimes be referred to as an object region within the image sample. Object recognition also refers to identifying the object region within the image sample. The object to be identified may be a person or a part thereof (such as a face, hand, or body), another animal or plant, or any other object desired to be detected. A typical human face is used below as an example of an object to be identified, but the present invention is not limited thereto.

[0034] A typical example of an object recognition model is a neural network, such as a convolutional neural network (CNN). The following description will primarily use CNN as an example, but it should be understood that the present invention can be similarly applied to other neural networks, such as generative adversarial networks (GANs), recent neural networks (RNNs), and even to other deep learning models, as long as the model can extract features of the object to be recognized.

[0035] As used herein, a "normal image" refers to a high-quality image for object recognition, such as those in traditional public datasets, which generally includes rich distinguishing information. In contrast, a "varied image" refers to an image that includes one or more changes compared to a normal image, such as changes in the object's posture, race, resolution, lighting, etc. The varied image can be an image captured by an imaging device or an image obtained through specific image processing. In the present invention, samples of varied images are mainly generated for one or more specific varied scenes by augmentation technology, but the varied images are not limited thereto.

[0036] In order to obtain a CNN with better generalization for different changed images, the changed image samples are input into the model together with the normal image samples. In the scenario of face recognition, the deep recognition model maps the face to the feature space to extract the corresponding feature representation. The feature representation of an object can be in the form of a vector, for example, which can be called the "feature vector" of the object. For example, CNN will select the pixel texture information, position coordinates, etc. of the representative part of the face as features to construct the feature vector of the image. By designing the loss function to achieve a small intra-class distance and a large inter-class distance, object recognition / detection / tracking can be performed based on the obtained feature vector. It should be understood that the form of feature representation may vary depending on the object recognition model used and is not particularly limited.

[0037] However, there may be significant differences between the feature domains of normal images and changed images, and a key requirement for learning knowledge from images with different changes is that the feature domains of normal images (normal domain) and changed images (changed domain) must be related. Figure 1 As shown in [1], due to the significant difference in feature domains between normal images and images with different variations, the normal and variation domains do not have enough common factors to bridge the gap, making it difficult to find direct correlations. Therefore, forced knowledge transfer is not feasible. Directly minimizing the distance of the same object will inevitably lead to negative transfer, as knowledge from the normal domain is difficult to propagate to the target variation domain.

[0038] In view of this, the present invention proposes the concept of transferring knowledge distillation. Figure 1 As shown in

[15] , since direct transfer is not possible, an adapter is introduced as a bridge between the normal domain and the target variation domain. This adapter converts the extracted features into intermediate features and uses transitive transfer learning (TTL) to transfer knowledge, thereby establishing an association between the normal domain and the variation domain, which have fewer common factors. As a result, the mutual influence between the variation images can be controlled during training, and the feature representations of images with different variations can be fused. Subsequently, knowledge distillation learning can be performed through the transfer domain generated by the adapter. By defining and reducing the distillation loss between the transfer domain and the target variation domain, the feature representation of the specific variation and the feature representation of the normal image are fused into the same feature representation.

[0039] Considering that it is unreasonable to use adapters with the same structure to convert training samples of different qualities, the present invention divides image samples into simple samples and difficult samples through a dynamic sample division strategy, and enables the adapter to perform different degrees of transfer migration on simple samples and difficult samples.

[0040] Furthermore, when training on both normal and altered image samples simultaneously, if the model assigns equal credibility to both, the learned pattern will be the center of all training samples. However, since the large variations in altered image samples are unrealistic and difficult to converge, they tend to provide a greater gradient influence, pushing the mixed pattern closer to the altered image samples, resulting in suboptimal classification of normal image samples.

[0041] To address this issue, the present invention implements dual-branch classification for both normal and altered image samples. Specifically, corresponding classifiers are used to classify each, ensuring that the classification training for normal and altered image samples does not affect each other. Based on a curriculum learning strategy, the impact of the classification loss on the altered image sample branch gradually increases over the training process, allowing the model's learning to gradually transition from normal images to altered images. This mitigates the negative impact of the difficulty in optimizing altered images and raises the upper limit of representation learning.

[0042] [Neural Network Training Method]

[0043] The neural network training method according to the present invention is described in more detail below.

[0044] As an example of a neural network, the convolutional neural network (CNN) is a widely used deep learning model that incorporates convolutional computations and a deep structure, with strong nonlinear fitting capabilities. Leveraging the nonlinear fitting capabilities of a CNN, for example, when an image sample containing an object region is fed into the CNN, information about the object hidden within the image can be extracted. CNNs are also self-learning, allowing the parameters of the neural network to be determined through training with real-world data, eliminating the need for complex manual parameter design.

[0045] In the context of this invention, an epoch of training refers to training on all training data sets once. Depending on actual needs, the training data set can be divided into several batches according to the batch size. Training on a batch of training data is called an iteration. Each iteration updates the parameters of the neural network.

[0046] The training data set used for training may include a large number of image samples including objects to be recognized (such as faces), such as tens of thousands, hundreds of thousands, or millions of images. The training data set according to the present invention includes a set of normal image samples (x N ,L N ) and a set of change image samples (x A ,L AEach changed image sample has a corresponding normal image sample, that is, this set of changed image samples corresponds to a subset of this set of normal image samples. In the context of the present invention, the normal image samples and the changed image samples "correspond" to each other in that they contain the same object to be recognized (e.g., person identity).

[0047] Normal image samples can be, for example, high-quality images from public datasets (e.g., CAISA-WebFace). Changed image samples can be, for example, low-quality images obtained through image augmentation techniques, and will be referred to as "augmented images" in the following embodiments. Image augmentation techniques perform a series of random changes on normal image samples to generate similar yet distinct training samples, thereby expanding the size of the training dataset and reducing the model's reliance on certain attributes, thereby improving the model's generalization ability. Typical image augmentations include flipping, such as left-right or up-down flipping; cropping, which positions objects of interest in different locations, thereby reducing the model's reliance on the object's location; color adjustment, such as changing the brightness, contrast, saturation, and hue of image samples to reduce the model's sensitivity to color; and resolution adjustment, such as generating low-quality images of varying resolutions. Depending on actual needs, augmentation can be performed for only one changing scenario to obtain an augmented image containing one change, or for two or more changing scenarios to obtain an augmented image containing multiple changes.

[0048] The image samples in the training data set can be pre-processed. Pre-processing operations may include, for example, object detection, object alignment, normalization, and the like. In particular, object detection may, for example, refer to detecting a face from an image containing a face and obtaining an image mainly containing the face to be recognized, and object alignment may refer to aligning object images in different postures in the image to the same or appropriate posture, thereby performing object detection / recognition / tracking based on the aligned object images. Face recognition is a common object recognition operation, and for face recognition training image sets, pre-processing including, for example, face detection, face alignment, and the like may be performed. Since the gradient descent algorithm is used for learning, the image sample data is normalized to a standard distribution with a mean of 0 and a variance of 1, thereby accelerating the convergence of the neural network. It should be noted that the pre-processing operation may also include other types of pre-processing operations known in the art, which will not be described in detail here.

[0049] Figure 2 Schematic diagram showing the neural network training method according to the present invention. Figure 2 Some symbol definitions in:

[0050] (x N ,LN ): normal image sample, where x N Represents normal image data, L N Is a marker indicating that the sample is a normal image;

[0051] (x A ,L A ): Augmented image samples, where x A represents the augmented image data, L A Is a marker indicating that the sample is an augmented image;

[0052] M S : The neural network that needs to be trained acts as the student in transfer learning, so it is also called the student network below;

[0053] The neural network trained for normal images serves as a teacher to provide guidance in transfer learning, so it is also referred to as the normal image teacher network below;

[0054] The teacher network trained on the augmented images serves as the teacher for providing guidance in transfer learning, so it is also referred to as the augmented image teacher network below.

[0055] When the normal image sample is input into the student network M S When , the student features are extracted by embedding the image into d-dimensional Euclidean space, where d is the dimension of the embedded features;

[0056] When the augmented image sample is input into the student network M S The student features extracted when , the feature dimension is d;

[0057] When the normal image sample is input into the normal image teacher network The teacher features extracted when , the feature dimension is d;

[0058] When the augmented image sample is input into the variation image teacher network The teacher features extracted when , the feature dimension is d;

[0059] Student Characteristics The transferred student features obtained after being transformed by the corresponding transfer transfer adapter (normal → augmented adapter) have a feature dimension of d;

[0060] Student Characteristics The transferred student features obtained after being transformed by the corresponding transfer transfer adapter (augmented → normal adapter) have a feature dimension of d;

[0061] The weights of the last classification layer (mostly the fully connected layer) in a deep convolutional neural network, where C represents the number of categories in the training dataset.

[0062] Refer to the following Figure 2 Describe each part of the training method in detail.

[0063] Feature extraction

[0064] exist Figure 2 In the training method shown, first, the training data set is input into the student network M S Although in Figure 2 Normal image sample (x N ,L N ) and augmented image samples (x A ,L A ) are shown as being fed into separate CNNs, but this is for intuitive illustration purposes only; they are actually fed into the same CNN, i.e., the student network M S . Student Network M S Various convolutional neural network architectures known in the art, such as VGG16, ResNet, SENet, etc., can be used without particular limitation.

[0065] Generally speaking, student network M S It can include convolutional layers, activation functions, pooling layers, etc.

[0066] In the convolutional layer, a filter (i.e., convolution kernel) with self-learnable parameters is convolved with the data matrix to extract hidden features in the input data. Considering that the size of the convolution kernel is often much smaller than the data matrix, the convolution kernel moves on the data matrix to traverse the data matrix, and the distance moved is called the stride. In addition, to match the movement of the convolution kernel, the data matrix may be edge-extended (i.e., edge-padded). Convolution kernels with different parameters are used to extract different features from the data matrix, and the corresponding convolution output is called a feature channel. In order to extract richer features, the number of feature channels gradually increases as the number of network layers deepens.

[0067] The output of a convolutional layer often passes through an activation function, such as ReLU or Sigmoid, before being input to the next layer. Activation functions are typically nonlinear, and therefore can introduce nonlinear fitting capabilities to CNNs. Deep learning achieves high performance precisely because high nonlinearity is achieved through repeated nonlinear transformations across multiple layers. If there were no activation functions to account for nonlinear changes and the network consisted solely of linear transformations, then regardless of the number of layers, there would be only the equivalent of a single linear transformation, and the multiple layers would be useless. Clearly, as the number of layers increases, deep learning exhibits stronger nonlinearity and higher performance.

[0068] The pooling layer downsamples the input matrix to reduce the amount of data and computation required by the neural network. Pooling operations include maximum pooling and average pooling. Feature vectors obtained by different pooling layers can be combined into a single feature vector to facilitate output prediction by subsequent network structures.

[0069] Student Network M S With initial parameters to facilitate the start of the neural network. Student network M S Extract corresponding features from each input image sample, such as from a normal image sample (x N ,L N ) to extract student features From the augmented image sample (x A ,L A ) Extract student features Therefore, the student network M S It can be regarded as a feature extractor composed of the backbone network of the convolutional neural network.

[0070] Dynamic sample partitioning

[0071] Before transferring the extracted student features to learn the transfer function, the image samples can be divided into ( Figure 2 (not shown) in order to adapt the corresponding transfer features for object images of different qualities.

[0072] In view of the fact that the off-line segmentation of image samples in the prior art will result in fixed differences in their feature domains, the present invention proposes a dynamic sample segmentation strategy based on the extracted features. S Easy samples and hard samples are defined by the distances (such as but not limited to L1-norm distance) from the features extracted from the training samples to the centers of different feature domains.

[0073] Figure 3 The pseudo code of the dynamic sample partitioning algorithm according to the present invention is shown. Figure 3 , for each epoch of training, perform the following steps:

[0074] 1. Calculate the dynamic parameter α, which can be calculated as a function of the training algebra, such as Where e is the current training generation, E is the maximum training generation, and the value of the parameter α increases with the training process;

[0075] 2. Assume that the number of classes involved in the training dataset is C (each image sample can be accompanied by a label indicating the class to which it actually belongs). For each class in the training dataset, calculate the center f of the extracted student feature domain. c , including the center of the feature domain corresponding to the normal image sample (normal feature center), the center of the feature domain corresponding to the augmented image sample (augmented feature center), and the center of the mixed feature domain corresponding to all image samples (mixed feature center). At this stage, the normal image sample and the augmented image sample can be identified by the corresponding flag L N 、L A to distinguish;

[0076] 3. For each image sample, its student feature f can be calculated to each feature center f c The intra-domain distance D = ||ff c ||, the distance calculated here can be the L1-norm distance;

[0077] 4. For each feature domain, calculate its maximum intra-domain difference D d =D max -D m)n , that is, the difference between the maximum intra-domain distance and the minimum intra-domain distance in each feature domain;

[0078] 5. For each feature domain, calculate the threshold γ used to define a simple sample easy =(∈+λα)D d +D min There are three parameters involved here, where threshold ∈ is the initial threshold, the dynamic parameter α is the parameter related to the training generation calculated as above, and the hyperparameter λ is the parameter set before training and does not change during the training process;

[0079] 6. Subsequently, the samples can be divided based on the calculated threshold. Specifically, for a normal image sample, if the distance between its feature and the center of the normal feature is less than the corresponding threshold The distance between its feature and the center of the mixed feature is less than the corresponding threshold The image sample is a simple sample, otherwise it is a difficult sample; and for the augmented image sample, if the distance between its feature and the augmented feature center is less than the corresponding threshold The distance between its feature and the center of the mixed feature is less than the corresponding threshold Then the image sample is a simple sample, otherwise it is a difficult sample.

[0080] At the beginning of training, a small initial threshold ∈ can be set to define the initial simple samples. Figure 4 The threshold γ used to define a simple sample changes during the training process. In the early stage of training, since the student network M is being trained S The image samples with different variations are not stable enough, and most of the image samples are defined as difficult samples. Figure 4 As shown in the S To enhance the ability to distinguish difficult samples, the dynamic parameter α dynamically increases the threshold γ that defines simple samples, so that more and more image samples are divided into simple samples, thereby gradually learning the distinguishing information of different feature domains and integrating them into a compact intra-class feature representation.

[0081] Transfer knowledge distillation

[0082] Continue to refer to Figure 2 , by the student network M S The extracted student features can be fed into different adapters to transform to the transfer domain. Specifically, the student features extracted from the normal image samples are is input to the normal→augmented adapter, and the student features extracted from the augmented image samples is input to the augmented→normal adapter.

[0083] Based on the concept of transfer transfer learning, the present invention utilizes a transfer transfer adapter to convert features extracted by the student network into intermediate features (also known as transferred student features). This adapter acts as a bridge between the normal domain and the target variation domain, amplifying the hidden information of specific variations. Because the adaptation directions between the two domains differ, a normal→augmented adapter and an augmented→normal adapter are used to preserve and restore knowledge, respectively. This transfer transfer effectively expands the potential discriminative factor information.

[0084] The normal→augmented adapter and the augmented→normal adapter can be implemented as neural networks with the same structure, such as convolutional neural networks, but with different parameters and used separately. As an example, both transfer adapters can be the backbone of a convolutional neural network with multiple convolutional layers (activation functions) and pooling layers, each of which ensures that the feature dimension remains unchanged. They have initial parameters and their parameters are continuously updated during training.

[0085] Each transfer transfer adapter can adapt corresponding transfer transfer features for image samples of different qualities. For example, for normal image samples and augmented image samples containing rich recognition information, deep transfer transfer features can be used as output to fully extract cross-region information; while for augmented images that lack recognition capabilities, such as low-resolution images, shallow transfer transfer features can be used to extract sufficient information to ensure fast convergence.

[0086] By defining the quality of image samples through the dynamic sample partitioning strategy described above, the transfer adaptation from shallow to deep layers can be naturally applied to image samples of different qualities. Therefore, the output of the transfer adapter according to the present invention can be defined as:

[0087]

[0088] Among them, G N→A Normal → Amplifier adapter, G A→N is the augmented→normal adapter. When normal / augmented image samples are defined as simple samples, ξ N / A = 1, then select deep migration conversion feature as is the output; when normal / augmented image samples are defined as difficult samples, ξ N / A = 0, then select shallow migration conversion features as output.

[0089] Figure 5 is a schematic diagram illustrating conventional direct migration and transfer migration according to the present invention. Figure 5 (a) shows a direct reduction in feature domain differences. However, due to the large domain difference between the normal image and the augmented image, it is difficult to find enough common factors to narrow the gap between the two. Directly minimizing the distance of the same object will inevitably lead to negative transfer, because the knowledge of the normal domain is difficult to propagate to the target variation domain. Figure 5 (b) Specific transfer features are selected to adapt to object images of different qualities so that information with common factors can be associated.

[0090] On the other hand, in order to balance the knowledge transfer between images with different changes, two teacher models can be introduced for knowledge distillation. Figure 2 As shown in , the neural network model can be pre-trained and fixed on a normal image set (i.e., normal image teacher network), pre-trained and fixed neural network model on the augmented image set (i.e., augmented image teacher network) to serve as the teacher network in the subsequent training of the student network. Here, the teacher network and It may be, for example, a convolutional neural network, but the present invention is not limited thereto.

[0091] Subsequently, it is input into the student network M S Normal image samples in the training image set are also input into the normal image teacher network To extract teacher features Entered into the student network M S The augmented image samples in the training image set are also input into the augmented image teacher network To extract teacher features It should be understood that due to the teacher network and It has been pre-trained, so it can be used to input the training data set into the student network M S At the same time, the corresponding image samples are input into the teacher network, or in the student network M S The teacher features are extracted and stored before training in order to facilitate S used during training.

[0092] like Figure 2 As shown in the lower part of Lack of recognition ability, so the normal image teacher network can be used To guide the augmented image restoration and differentiation ability in the student network; at the same time, Figure 2 As shown in the upper part of To guide the normal image in the student network to preserve the target distinguishing information. In this way, the student network M S Learn feature representations across variations from both sides.

[0093] Subsequently, knowledge distillation can be performed on the transferred features from the adapter (transfer domain) and the teacher features provided by the teacher network (target domain). In the distillation to the target variation domain, the transferred student features of the normal image samples can be Teacher features relative to the corresponding augmented image samples The distillation loss is calculated as the L1-norm distance between them Accordingly, in the distillation to the target normal domain, the transfer student features of the augmented image samples can be transferred Teacher features relative to the corresponding normal image samples The distillation loss is calculated as the L1-norm distance between them Therefore, the overall distillation loss function is defined as follows:

[0094]

[0095] Among them, G N→A and GA→N Denote normal→augmented adapter and augmented→normal adapter respectively. By reducing the distillation loss between the transfer domain and the target domain, the feature representation of the specific variation can be fused with the normal feature representation in the same feature representation.

[0096] With the help of the transfer effect of the transfer transfer adapter and the constraint effect of the distillation function, knowledge distillation is performed to different target domains respectively, reducing the domain gap between normal images and augmented images from both sides, promoting positive transfer while reducing negative transfer, and ensuring that the features of differently changed images are integrated into the target representation.

[0097] Two-branch classification

[0098] When training the student network using normal image samples and changed image samples, if it is assumed that all image samples finally pass through the same classifier (for example, a fully connected layer), then the final trained model will be at the center of all image samples. However, since image samples with large changes are often unrealistic and difficult to converge, they will continue to have large gradients during the training process, pushing the trained model closer to the augmented image, such as Figure 6 As shown in (a), Figure 6 is a schematic diagram showing the classification learning of normal image samples and augmented image samples. Therefore, it is difficult to achieve the optimal classification of normal image samples, resulting in a lower upper limit for representation learning.

[0099] To solve this problem, the present invention proposes a dual-branch classification strategy based on curriculum learning.

[0100] First, based on the idea of curriculum learning, we focus on learning the model of normal images in the early stages of training. As the training process gradually increases the credibility of the augmented images, the network model transitions to learning difficult samples. Specifically, we use the parameter β to represent the credibility of the augmented images. In the early stages of training, we assign a small value to the parameter β so that normal image samples can be well learned without being negatively affected by the augmented image samples. Then, as the training process progresses, we gradually increase the guiding role of the augmented image samples so that the changed images can approach the normal images, such as Figure 6 (b) is shown in Figure 2. The classification loss function at this time can be defined as:

[0101]

[0102] Among them, e is the current training generation, e max is the maximum training generation, W represents the classifier, L N 、L A Labels representing image samples.

[0103] However, when only using the curriculum learning strategy, as the parameter β increases, the learned model may continue to be biased towards the gradient of the augmented image, such as Figure 6 Therefore, in order to ensure that the learned model is always close to the feature domain of the normal image and does not deviate from the gradient of the augmented image due to the increase of parameter β, as shown in (c). Figure 6 As shown in (d), the present invention adopts dual-branch classification so that the classification branch training of normal images and change images do not affect each other.

[0104] Specifically, if Figure 2 As shown in , independent classifiers are used to classify normal image samples and augmented image samples. As a branch, classifier W N Based on the student features extracted from normal image samples To classify normal image samples to obtain the probability distribution As another branch, the classifier W A Based on the student features extracted from the augmented image samples To classify the augmented image samples to obtain the probability distribution Classifier W N and W A It can be implemented as a fully connected layer, for example, to integrate the discriminative information in the feature vector. Assuming that the total number of classes in the training set is C, the probability distribution and Each probability in indicates the probability (or confidence) that the image sample belongs to each of the C classes.

[0105] Subsequently, the classification loss function can be used to determine the difference between the obtained classification probability and the true value (e.g., (0, 1, 0, ..., 0), where 1 indicates the true value) by comparing the two, such as cross entropy, as classification loss data. According to the present invention, the classification loss function may include but is not limited to ArcFace loss function, CosFace loss function, SphereFace loss function, etc.

[0106] Therefore, the final classification loss function is calculated as:

[0107]

[0108] in, represents the classifier of the normal image branch, Represents the classifier of the augmented image branch.

[0109] Under the dual guidance of parameter β and dual classifiers, the classification training of the student network can greatly reduce the negative impact of augmented image samples and maintain the characteristics of intra-class compactness and inter-class separation in each feature domain.

[0110] Parameter Update

[0111] The loss data from transferred knowledge distillation and the loss data from classification can be combined for parameter updating.

[0112] According to the present invention, the hyperparameter η is used to balance the total loss function, and the total loss function can be defined as:

[0113]

[0114] Then, based on the calculated total loss data Update the student network M through back propagation method S The specific parameter updating algorithm is well known in the art and will not be described here in detail.

[0115] In addition, it is also possible to base the total loss data on The two transfer adapters (i.e., Normal→Augmented Adapter and Augmented→Normal Adapter) and two classifiers (i.e., Classifier W N and classifier W A ) parameters.

[0116] The process described above can be completed in one generation of training or one iteration. In order to achieve satisfactory performance, the above process can be repeated multiple times using the same training image set until a predefined condition is met, such as the number of repetitions reaches a predetermined maximum number, or the total loss calculated is less than a predefined threshold.

[0117] Iterative training method for neural networks

[0118] In the training method described above, the student network is guided by both a teacher network trained from normal images and a teacher network trained from modified images. This effectively integrates different knowledge to enhance the robustness of the student network model. However, learning from multiple modified images simultaneously is difficult. One approach is to use teacher networks trained from different sets of modified images for simultaneous supervision. However, due to the large differences in feature domains between different modified images, this inevitably leads to reduced performance of the final trained model.

[0119] The present invention proposes an iterative learning strategy, which trains different changes in different iterations, thereby gradually learning multiple changes.

[0120] Figure 7 Schematic diagram of the iterative training method according to the present invention is shown. Figure 7 As shown in , the iterative training method includes:

[0121] 1. In the first iteration, normal image samples (x N ,L N ) and the first variation (e.g. low resolution LR) image sample (x LR ,L LR ) is input into the student network to be trained, and the neural network trained for normal images and the neural network trained for the first type of change image samples are used as the teacher network. Figure 2 The description is a training method to train the student network to obtain the distilled network (MR);

[0122] 2. In the second iteration, the normal image samples (x N ,L N ) and the second variation (e.g. low light LI) image sample (x LI ,L LI ) is input into the distillation network (MR) trained in step 1, and the obtained distillation network (MR) and the neural network trained for the second type of change image sample are used as the teacher network. Figure 2 The described training method is used to continue training the distilled network (MR) to obtain the distilled network (general);

[0123] 3. If there are other changes, repeat step 2 until all changes have been learned.

[0124] Through the above iterative training process, the student network trained in the previous iteration becomes the teacher network Used to supervise the training of new change images that appear in the current iteration to prevent forgetting prior knowledge and fine-tune it as the initial backbone network model of the student network. Another teacher network pre-trained for new changes is used to expand the recognition capability of the student network. Therefore, the updated model can focus on integrating the feature domain of a specific changing image in each iteration while retaining the previously learned knowledge.

[0125]

Experimental verification

[0126] The following experiments will compare the performance of the neural network training method according to the present invention with that of the prior art.

[0127] Experiment 1: Validation on a small training set

[0128] Training set: CASIA-WebFace, including 10,000 person identities and a total of 500,000 images.

[0129] Test set: Scface, including 130 person identities, a total of 4160 static images taken at different camera distances.

[0130] Evaluation criteria: 1:N TPIR (True Positive Recognition Rate, Rank1@106), same as Megaface challenge

[0131] Convolutional neural network architecture: RestNet50

[0132] Comparison of existing technologies: ArcFace, ArcFace+Augmentation, DDL (Distributed Deep Learning)

[0133] Experimental results:

[0134] Table 1 Performance comparison of the training method of the present invention and the prior art

[0135]

[0136] Among them, the d1 column is the performance on the low-quality image subset taken from the long-distance camera, the LFW column is the performance on the medium-quality image subset taken from the medium-distance camera, the CFP-FP column is the performance on the high-quality image subset taken from the close-range camera, and AGEDB represents the average performance.

[0137] As can be seen from Table 1, while existing techniques using image augmentation can improve recognition performance on low-quality image sets, they can lead to a decrease in recognition performance on high-quality image sets. In contrast, the neural network training method according to the present invention can achieve performance improvements on low-quality image sets while simultaneously suppressing negative transfer on high-quality image sets.

[0138] Training device and training method

[0139] The following will describe a training device and a training method for a neural network for object recognition according to the present invention with reference to the accompanying drawings.

[0140] Figure 8 is a block diagram showing a training apparatus 100 for a neural network model for object recognition according to the present invention, and Figure 9 2 is a flow chart illustrating a method 200 for training a neural network model for object recognition according to the present invention. The training apparatus 100 can be used to train a convolutional neural network, for example, to achieve excellent recognition performance for images of varying quality. The training apparatus 100 may include a feature extractor 101, a partitioning unit 102, a transfer transfer adapter 103, a distillation loss calculation unit 104, a classifier 105, a classification loss calculation unit 106, a total loss calculation unit 107, and an update unit 108.

[0141] The feature extractor 101 is configured to input a training image set containing objects to be recognized into the neural network to be trained to extract student features of each image sample (i.e., executing step S201 of method 200). The training image set includes a set of normal image samples and a set of changed image samples.

[0142] The segmentation unit 102 is configured to segment the image samples in the training image set into easy samples and difficult samples based on the student features extracted by the feature extractor 101 (i.e., executing step S202 of method 200). The segmentation unit 102 may perform the segmentation based on the distance between the student features of the image samples and various feature centers (e.g., L1-normalized distance).

[0143] The transfer transfer adapter 103 is configured to transfer the student features of the normal image samples based on the segmentation results of the segmentation unit 102 to obtain transferred student features (i.e., executing step S203 of the method 200). The transfer transfer adapter 103 may include transfer transfer adapters for normal image samples and change image samples respectively.

[0144] The distillation loss calculation unit 104 is configured to calculate the distillation loss of the transferred student features of the normal image samples relative to the teacher features extracted from the corresponding changed image samples, and calculate the distillation loss of the transferred student features of the changed image samples relative to the teacher features extracted from the corresponding normal image samples (i.e., executing step S204 of method 200).

[0145] The classifier 105 is configured to classify the image samples based on the student features extracted by the feature extractor 101 (ie, execute step S205 of the method 200). The classifier 105 may include classifiers for normal image samples and change image samples respectively, thereby achieving dual-branch classification.

[0146] The classification loss calculation unit 106 is configured to calculate the classification loss of the classification of each image sample by the classifier 105 (i.e., perform step S206 of the method 200). The classification loss calculation unit 106 can use, for example, an ArcFace loss function, a CosFace loss function, a SphereFace loss function, etc. to calculate the classification loss.

[0147] The total loss calculation unit 107 is configured to calculate the total loss associated with the training image set based on the distillation loss and classification loss calculated for all image samples (i.e., executing step S207 of method 200). The total loss calculation unit 107 can calculate the sum of the distillation losses of the normal image samples and the corresponding changed image samples as the total distillation loss, calculate the weighted sum of their classification losses as the total classification loss, and calculate the weighted sum of the total distillation loss and the total classification loss as the total loss. In calculating the total classification loss, the total loss calculation unit 107 can increase the weight of the changed image samples as the current training epoch increases, so as to gradually shift the training focus toward the changed image samples.

[0148] The updating unit 108 is configured to update the parameters of the neural network (i.e., execute step S208 of the method 200) according to the total loss calculated by the total loss calculating unit 107. In addition, the updating unit 108 can also update the parameters of the transfer migration adapter 103 and the classifier 105 according to the total loss.

[0149] It should be pointed out that Figure 8 This is merely a schematic structural configuration of the training device. The training device 100 may also include other possible units / components, such as a memory. The memory may store various information generated by the training device (e.g., features of image samples, loss data, parameter values, etc.), programs and data used for the operation of the training device, etc. For example, the memory may include, but is not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. As an example, the memory may also be located outside the training device 100. The training device 100 may be connected to the memory directly or indirectly (e.g., with other components connected in between) to access data. The memory may be a volatile memory and / or a non-volatile memory.

[0150] It should be understood that the above-mentioned units are merely logical modules divided according to the specific functions they implement, and are not intended to limit specific implementation methods. For example, they can be implemented in software, hardware, or a combination of software and hardware. In actual implementation, the above-mentioned units can be implemented as independent physical entities, or can be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). In addition, the above-mentioned units are shown with dotted lines in the accompanying drawings to indicate that these units may not actually exist, and the operations / functions they implement can be implemented by the processing circuit itself.

[0151] It should be understood that, in addition to including multiple units, the training device 100 described above can be implemented in a variety of other forms, such as a general-purpose processor or a dedicated processing circuit, such as an ASIC. For example, the training device 100 can be constructed from circuits (hardware) or a central processing device (such as a central processing unit (CPU)). In addition, the training device 100 can carry a program (software) for operating the circuits (hardware) or the central processing device. This program can be stored in a memory (such as a memory) or an externally connected external storage medium, or downloaded via a network (such as the Internet).

[0152] It should be understood that the training method 200 according to the present invention may also include the various operations described above, which will not be described in detail here. It should be noted that the various steps / operations of the training method 200 according to the present invention can be performed by the various units of the training device 100 described above, or by various forms of processing circuits.

[0153] Various exemplary implementations of the neural network training process according to the present invention are described below.

[0154] [First example]

[0155] The first example involves the training process when the training dataset includes enough normal image samples and variation image samples (possibly with one or more variations). Here, the number of samples "sufficient" means that it can meet the requirements of the neural network model convergence.

[0156] Figure 10 A flowchart of a neural network training process according to a first example is shown, which includes the following steps.

[0157] S1100: Obtain network training data through pre-processing

[0158] In this step, the original image with real object or face labels is input, and then the input original image is converted into training data that meets the requirements of the convolutional neural network model through a series of existing pre-processing operations. This series of pre-processing operations includes face or object detection, face or object alignment, etc.

[0159] S1200: Training the Teacher Network Model

[0160] Optionally, in this step, image sample data with real-world object or face labels that meet the requirements of the convolutional neural network model is input. This image sample data includes normal image samples and transformed image samples. Then, using existing advanced recognition methods such as ArcFace and CosFace, a commonly used convolutional neural network architecture such as ResNet is selected to train the neural network on the normal image training set and the transformed image training set, respectively. Ultimately, a teacher network model with certain recognition performance in both the normal image domain and the transformed image domain is obtained.

[0161] S1300: Extract student features corresponding to image samples from the student backbone network

[0162] In this step, all image samples containing objects or faces that meet the requirements of the convolutional neural network model are input. Then, the student network is used to extract the student features of each image sample using the selected neural network structure with initial parameters. The convolutional neural network structure can be a commonly used network structure such as VGG16, ResNet, SENet, etc.

[0163] S1400: Extract teacher features corresponding to normal / varied image training data

[0164] In this step, all image sample data with objects or faces that meet the requirements of the convolutional neural network are input, and then the teacher network model trained in S1200 is used to extract corresponding teacher features from normal image samples and change image samples respectively.

[0165] S1500: Divide image samples into easy samples and difficult samples

[0166] In this step, the student features extracted in S1300 are input, and all image samples are divided into easy samples and difficult samples according to the dynamic sample division strategy based on feature distance described above.

[0167] S1600: The corresponding transfer migration adapter performs transfer migration based on sample partitioning

[0168] In this step, on the one hand, the student features extracted from the normal image samples are input into the transfer transfer adapter from the normal domain to the variation domain, and are converted into transfer student features according to their easy / difficult sample division; on the other hand, the student features extracted from the variation image samples are input into the transfer transfer adapter from the variation domain to the normal domain, and are converted into transfer student features according to their easy / difficult sample division.

[0169] S1700: Calculate distillation losses

[0170] In this step, the distillation loss is calculated according to the defined distillation loss function, which is defined as a part of the intra-class loss, which consists of two parts: (1) the distance between the transferred student features of the changed image samples and the teacher features of the corresponding normal image samples, such as the L1-norm distance; (2) the distance between the transferred student features of the normal image samples and the teacher features of the corresponding changed image samples, such as the L1-norm distance.

[0171] S1800: Calculate classification loss

[0172] In this step, classification is performed on both branches of the normal and modified image samples. The classification loss for the normal and modified images, i.e., the intra-class classification loss, is calculated based on the defined classification loss function. Based on the curriculum learning strategy, the inter-class classification loss is obtained by calculating the weighted sum of the classification loss for the normal and modified images. Specific classification loss functions include ArcFace, CosFace, and SphereFace.

[0173] S1900: Calculate the total loss and update the parameters based on the total loss

[0174] Based on the distillation loss calculated in S1700 and the classification loss calculated in S1800, a total loss associated with the training image set is calculated. Based on the calculated total loss, the backpropagation algorithm is used to update the parameters of the student backbone network, the parameters of the transfer adapter, and the parameters of the classifier.

[0175] S11000: Determine whether the training process is completed

[0176] In this step, you can determine whether to terminate training based on predefined conditions, such as whether the total loss is less than a predefined threshold or whether the current training epochs / iterations have reached a predefined maximum. If the conditions are met, model training terminates; otherwise, the next training step continues.

[0177] S11100: Output the trained student backbone network model

[0178] In this step, the current parameters of all layers in the student backbone network constitute the trained model, thereby outputting the student backbone network structure and current parameter information.

[0179] Second Example

[0180] The second example involves obtaining a neural network model with good recognition capabilities in a specific variable scene when the training dataset only includes normal image samples. For example, there may be scenes such as outdoor, night, surveillance, and sports scenes, where the object images captured in these scenes may exhibit various angles, postures, lighting, blur, and sizes.

[0181] Figure 11 FIG2 is a flowchart of a neural network training process according to a second example. Compared with the first example, the second example is different in that it includes steps S2101-S2102.

[0182] Step S2101: Select the corresponding image augmentation method for the changing scene

[0183] In this step, the corresponding image augmentation method is selected for the considered changing scene. For example, to enhance the performance of the neural network model in low-light scenes, an image augmentation method for reducing image brightness can be selected to augment the low-light changing image; or to adapt the neural network model to low-resolution scenes, an image augmentation method for reducing image resolution can be selected to augment the low-resolution changing image, and so on.

[0184] In this step, the image augmentation methods that can be selected can include simple image processing algorithms or image processing models trained using deep learning.

[0185] Step S2102: Augmenting the normal image to generate a modified image sample

[0186] In this step, the selected image augmentation method is used to augment the normal image processed in step S2100 to generate modified image samples. It is not necessary to augment the modified image samples corresponding to all normal image samples, as long as the number of modified image samples meets the requirement.

[0187] Thus, the normal image samples and the augmented changed image samples can be input into the backbone student network as network training data. The remaining steps S2200-S21100 are the same as steps S1200-S11100 of the first example and are not repeated here.

[0188] [Third Example]

[0189] The third example involves obtaining a neural network model with good recognition ability in a specific changing scene when the training data set includes sufficient normal image samples and a small number of changing image samples in such a changing scene.

[0190] Figure 12FIG3 is a flowchart of a neural network training process according to a third example. Compared with the first example, the third example is different in that it includes steps S3101-S3102.

[0191] Step S3101: Select the corresponding image augmentation method for the changing scene

[0192] In this step, the corresponding image augmentation method is selected for the considered change scenario. In a third example, a deep learning-based image processing model (e.g., a generative adversarial network) can be used to learn the change scenarios involved in the change images in the training dataset, thereby augmenting more change image samples under the same change scenario.

[0193] Step S3102: Augmenting the normal image to generate a modified image sample

[0194] In this step, the selected image augmentation method is used to augment the normal image processed in S3100 to obtain the changed image samples to expand the training data set.

[0195] Thus, the normal image samples and the augmented changed image samples can be input into the backbone student network as network training data. The remaining steps S3200-S31100 are the same as steps S1200-S11100 of the first example and are not repeated here.

[0196] [Fourth Example]

[0197] The fourth example involves using the iterative training method introduced above to obtain a neural network model with good recognition capabilities in a variety of changing scenarios.

[0198] Figure 13 A flowchart of a neural network training process according to a fourth example is shown, which includes the following steps.

[0199] S4100: Obtaining network training data through pre-processing

[0200] This step is the same as for S1100.

[0201] S4200: Divide the network training data into normal images and various types of changed images, and use the normal images as the initial network training data

[0202] In this step, the input is all the network training data that has passed the pre-processing. We can then use human judgment or certain predefined thresholds to classify these training data into normal image samples and various types of changed image samples, and assign corresponding labels. The normal image samples serve as the initial network training data.

[0203] S4300: Determine whether all changes to be trained have completed training

[0204] In this step, the desired training types are first defined, and then it is determined whether all the training types have been completed. If so, the training is terminated and the trained student backbone network model is output. Otherwise, the process proceeds to step S4400.

[0205] S4400: Select a specific change as the target change for this iterative training, and add the original image of the specific change to the network training data

[0206] In this step, for a variation type that has not been trained, if there are variation image samples of the variation type in the network training data obtained through pre-processing in S4100, these variation images are added to the network training data.

[0207] S4500: Determine whether to augment the current change type

[0208] In this step, the number of image samples containing the current change type in the current network training data is first determined. Based on this number, it is then determined whether augmentation is necessary. If this number is below a given threshold, augmentation is necessary and the process proceeds to S4600. Otherwise, the process proceeds to S4700.

[0209] S4600: Select the data augmentation method corresponding to the specific change and add the augmented change image to the network training data

[0210] This step is the same as S2102 and S3102.

[0211] S4700: Use current network training data to train the student backbone network

[0212] In this step, the input is network training data including normal image samples and changed image samples of the current change type (already in the original training data set or obtained through augmentation), and the student backbone network is trained using the training process in the first example introduced above.

[0213] S4800: Use the trained student backbone network as the initial student backbone network and normal image teacher network model for the next iterative training

[0214] In this step, on the one hand, the trained student backbone network is used as the initial student backbone network for the next iterative training, and on the other hand, the previous normal image teacher network model is replaced with the currently trained student backbone network model.

[0215] S4900: Output the trained student backbone network model

[0216] This step is the same as for S11100.

[0217] Through the fourth example, a new variation scenario can be learned in each iteration while retaining the knowledge learned for other variation scenarios.

[0218] Computer System

[0219] Figure 14 is a block diagram illustrating an exemplary hardware configuration of a computer system 1000 in which embodiments of the present invention may be implemented.

[0220] like Figure 14 As shown, the computer system includes a computer 1110. The computer 1110 may be implemented as described above. Figure 8 The training device 100 is used to perform the above-mentioned Figure 9 The training method 200 is described. The computer 1110 includes a processing unit 1120, a system memory 1130, a non-removable non-volatile memory interface 1140, a removable non-volatile memory interface 1150, a user input interface 1160, a network interface 1170, a video interface 1190, and an output peripheral interface 1195, which are connected via a system bus 1121.

[0221] The system memory 1130 includes a ROM (Read Only Memory) 1131 and a RAM (Random Access Memory) 1132. A BIOS (Basic Input Output System) 1133 resides in the ROM 1131. An operating system 1134, application programs 1135, other program modules 1136, and some program data 1137 reside in the RAM 1132.

[0222] A non-removable non-volatile memory 1141 (such as a hard disk) is connected to the non-removable non-volatile memory interface 1140. The non-removable non-volatile memory 1141 can store, for example, an operating system 1144, application programs 1145, other program modules 1146, and some program data 1147.

[0223] Removable nonvolatile memory such as a floppy disk drive 1151 and a CD-ROM drive 1155 are connected to the removable nonvolatile memory interface 1150. For example, a floppy disk 1152 can be inserted into the floppy disk drive 1151, and a CD (Compact Disc) 1156 can be inserted into the CD-ROM drive 1155.

[0224] Input devices such as a mouse 1161 and a keyboard 1162 are connected to the user input interface 1160 .

[0225] Computer 1110 can be connected to a remote computer 1180 via a network interface 1170. For example, network interface 1170 can be connected to remote computer 1180 via a local area network 1171. Alternatively, network interface 1170 can be connected to a modem (modulator-demodulator) 1172, and modem 1172 can be connected to remote computer 1180 via a wide area network 1173.

[0226] Remote computer 1180 may include memory 1181 , such as a hard disk, that stores remote application programs 1185 .

[0227] Video interface 1190 is connected to monitor 1191 .

[0228] The output peripheral interface 1195 is connected to a printer 1196 and a speaker 1197 .

[0229] Figure 14 The computer system shown is for illustrative purposes only and is not intended to limit the invention, its application, or uses.

[0230] Figure 14 The illustrated computer system may be implemented for any embodiment as a stand-alone computer, or as a processing system in a device, where one or more unnecessary components may be removed or one or more additional components may be added.

[0231] The present invention can be used in many applications. For example, the present invention can be used to monitor, identify, track objects in static images or moving videos captured by a camera, and is particularly advantageous for portable devices equipped with cameras, (camera-based) mobile phones, etc.

[0232] It should be noted that the methods and apparatus described herein may be implemented as software, firmware, hardware, or any combination thereof. Some components may, for example, be implemented as software running on a digital signal processor or microprocessor. Other components may, for example, be implemented as hardware and / or an application-specific integrated circuit.

[0233] In addition, the method and system of the present invention can be implemented in a variety of ways. For example, the method and system of the present invention can be implemented by software, hardware, firmware or any combination thereof. The order of the steps of the method described above is merely illustrative, and unless otherwise specifically stated, the steps of the method of the present invention are not limited to the order specifically described above. In addition, in some embodiments, the present invention can also be embodied as a program recorded in a recording medium, including machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also encompasses a recording medium storing a program for implementing the method according to the present invention.

[0234] Those skilled in the art will appreciate that the boundaries between the above-mentioned operations are merely illustrative. Multiple operations can be combined into a single operation, a single operation can be distributed among additional operations, and operations can be performed at least partially overlapping in time. Moreover, alternative embodiments can include multiple instances of specific operations, and the order of operations can be changed in various other embodiments. However, other modifications, variations, and replacements are also possible. Therefore, this specification and the accompanying drawings should be considered illustrative, not restrictive.

[0235] Although some specific embodiments of the present invention have been described in detail by way of example, it will be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. The various embodiments of the present invention may be combined in any manner without departing from the spirit and scope of the present invention. It will also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

[0236] In addition, embodiments of the present invention may further include the following illustrative examples (EE).

[0237] EE1. A method for training a neural network for object recognition, comprising:

[0238] Inputting a training image set containing an object to be identified into the neural network to extract a student feature of each image sample, the training image set comprising a set of normal image samples and a set of changed image samples;

[0239] Based on the extracted student features, the image samples in the training image set are divided into simple samples and difficult samples;

[0240] For each of the normal image sample and the variation image sample:

[0241] The corresponding transfer transfer adapter performs transfer transfer on the student features of the image sample based on the division to obtain transferred student features;

[0242] Calculate the distillation loss of the transferred student features of the image sample relative to the teacher features extracted from the corresponding other image sample;

[0243] The image sample is classified by a corresponding classifier based on the student features; and

[0244] Calculate the classification loss of the image sample,

[0245] Calculating a total loss associated with the training image set based on the distillation loss and the classification loss calculated for all image samples; and

[0246] Based on the calculated total loss, the parameters of the neural network are updated.

[0247] EE 2. The training method according to EE 1, wherein the set of changed image samples comprises image samples augmented from the set of normal image samples using an augmentation method for one or more changed scenes.

[0248] EE 3. The training method according to EE 1, wherein the set of changed image samples includes original changed image samples and changed image samples expanded from the set of normal image samples by referring to the original changed image samples.

[0249] EE 4. The training method according to EE 1, wherein the dividing further comprises:

[0250] respectively calculating a first feature center of the student features of all image samples, a second feature center of the student features of the group of normal image samples, and a third feature center of the student features of the group of changed image samples;

[0251] According to the distance between each image sample and the first feature center, the second feature center, and the third feature center, the image sample is divided into a simple sample or a difficult sample.

[0252] EE 5. The training method according to EE 4, wherein, for each normal image sample, if the distance between the student feature of the image sample and the center of the first feature is less than a threshold or the distance between the student feature of the image sample and the center of the second feature is less than a threshold, the image sample is classified as a simple sample; otherwise, the image sample is classified as a difficult sample;

[0253] For each change image sample, if the distance between the student feature of the image sample and the first feature center is less than a threshold or the distance between the student feature of the image sample and the three feature centers is less than a threshold, the image sample is classified as a simple sample; otherwise, the image sample is classified as a difficult sample.

[0254] EE 6. The training method according to EE 5, wherein the threshold is determined by an adjustable parameter, and the adjustable parameter is a function of a current training generation and increases as the current training generation increases.

[0255] EE 7. The training method as described in EE 1, wherein the transfer transfer adapter is composed of multiple convolutional layers that keep the feature dimension unchanged.

[0256] EE 8. The training method according to EE 1, wherein the transfer migration based on the division further comprises:

[0257] For image samples classified as simple samples, using the deep output of the transfer transfer adapter as the transfer student feature;

[0258] For image samples classified as hard samples, the shallow output of the transfer adapter is used as the transfer student feature.

[0259] EE 9. The training method according to EE 1, wherein for normal image samples, the teacher features are features extracted using a first teacher neural network trained on a training image set including only normal images;

[0260] For the change image samples, their teacher features are features extracted using a second teacher neural network trained on a training image set that only includes change images.

[0261] EE 10. The training method as described in EE 1, wherein the distillation loss is calculated as: the distance between the transferred student features of the normal image sample and the teacher features of the corresponding changed image sample, or the distance between the transferred student features of the changed image sample and the teacher features of the corresponding normal image sample.

[0262] EE 11. The training method according to EE 1, wherein the classification loss is calculated by one of the following classification loss functions: ArcFace loss function, CosFace loss function, or SphereFace loss function.

[0263] EE 12. The training method according to EE 1, wherein calculating the total loss associated with the training image set further comprises:

[0264] Calculate the sum of the distillation losses of the normal image sample and the corresponding changed image sample as the total distillation loss;

[0265] Calculate the weighted sum of the classification losses of the normal image samples and the corresponding changed image samples as the total classification loss;

[0266] The weighted sum of the total distillation loss and the total classification loss is calculated as the total loss.

[0267] EE 13. The training method as described in EE 12, wherein, when calculating the total classification loss, the weight of the classification loss of the change image sample is a function of the current training generation and increases as the current training generation increases.

[0268] EE 14. The training method according to EE 1, wherein the total loss is used to update the parameters of the neural network through a back-propagation algorithm.

[0269] EE 15. The training method as described in EE 1 further includes using the total loss to update the parameters of the transfer transfer adapter and the parameters of the classifier.

[0270] EE 16. The training method according to EE 1, wherein the steps of the training method are repeated multiple times until at least one of the following conditions is met: the number of repetitions reaches a predefined maximum number, and the calculated total loss is less than a predefined threshold.

[0271] EE 17. In the training method according to EE 1, the object to be recognized includes a face, a person, or an object.

[0272] EE 18. A training device for a neural network for object recognition, comprising:

[0273] a feature extractor configured to input a training image set containing an object to be recognized into the neural network to extract a student feature of each image sample, the training image set comprising a set of normal image samples and a set of changed image samples;

[0274] a dividing unit configured to divide the image samples in the training image set into simple samples and difficult samples based on the extracted student features;

[0275] For each of the normal image sample and the variation image sample:

[0276] a transfer migration adapter configured to perform transfer migration on the student features of the image sample based on the division to obtain a migrated student feature;

[0277] a distillation loss calculation unit configured to calculate a distillation loss of the transferred student feature of the image sample relative to the teacher feature extracted from another corresponding image sample;

[0278] a classifier configured to classify the image sample based on the student feature; and

[0279] A classification loss calculation unit is configured to calculate the classification loss of the image sample,

[0280] a total loss calculation unit configured to calculate a total loss associated with the training image set based on the distillation loss and the classification loss calculated for all image samples; and

[0281] An updating unit is configured to update parameters of the neural network according to the calculated total loss.

[0282] EE 19. A method for iterative training of a neural network for object recognition, comprising:

[0283] A first training step is to train the neural network using the training method according to claim 1, using an image set including a set of normal image samples and a set of changed image samples for a first changed scene as a training image set; and

[0284] In a second training step, an image set including the set of normal image samples and a set of changed image samples for a second changed scene is used as a training image set, and the neural network that has undergone the first training step is trained using the training method according to claim 1.

[0285] EE 20. A neural network trained according to the training method described in EE 1 or 19.

[0286] EE 21. A device comprising

[0287] processor; and

[0288] A memory having executable instructions stored thereon, which, when executed by the processor, causes the processor to perform the training method as described in any one of EEs 1-17 and 19.

[0289] EE 22. A non-transitory computer-readable storage medium storing executable instructions, wherein the executable instructions implement the training method according to any one of EEs 1-17 and 19 when executed.

Claims

1. A method for training a neural network for object recognition, characterized in that: include: Inputting a training image set containing an object to be identified into the neural network to extract a student feature of each image sample, the training image set comprising a set of normal image samples and a set of changed image samples; Based on the extracted student features, the image samples in the training image set are divided into simple samples and difficult samples; For each of the normal image sample and the variation image sample: The corresponding transfer transfer adapter performs transfer transfer on the student features of the image sample based on the division to obtain transferred student features; Calculate the distance between the transferred student feature of the image sample and the teacher feature extracted from the corresponding image sample as the distillation loss of the image sample; The image sample is classified by a corresponding classifier based on the student features; and Calculate the classification loss of the image sample, Calculating a total loss associated with the training image set based on the distillation loss and the classification loss calculated for all image samples; as well as Based on the calculated total loss, the parameters of the neural network are updated, The transfer migration based on the division further includes: For image samples classified as simple samples, using the deep output of the transfer transfer adapter as the transfer student feature; For image samples classified as hard samples, the shallow output of the transfer adapter is used as the transfer student feature.

2. The training method according to claim 1, wherein: The set of changed image samples includes image samples augmented from the set of normal image samples using an augmentation method for one or more changed scenes.

3. The training method according to claim 1, wherein: The set of change image samples includes original change image samples and change image samples expanded from the set of normal image samples by referring to the original change image samples.

4. The training method according to claim 1, wherein: The division further includes: respectively calculating a first feature center of the student features of all image samples, a second feature center of the student features of the group of normal image samples, and a third feature center of the student features of the group of changed image samples; According to the distance between each image sample and the first feature center, the second feature center, and the third feature center, the image sample is divided into a simple sample or a difficult sample.

5. The training method according to claim 4, wherein: For each normal image sample, if the distance between the student feature of the image sample and the center of the first feature is less than a threshold or the distance between the student feature of the image sample and the center of the second feature is less than a threshold, the image sample is classified as a simple sample; otherwise, the image sample is classified as a difficult sample; For each change image sample, if the distance between the student feature of the image sample and the first feature center is less than a threshold or the distance between the student feature of the image sample and the three feature centers is less than a threshold, the image sample is classified as a simple sample; otherwise, the image sample is classified as a difficult sample.

6. The training method according to claim 5, wherein: The threshold is determined by an adjustable parameter, and the adjustable parameter is a function of the current training generation and increases as the current training generation increases.

7. The training method according to claim 1, wherein: The transfer adapter consists of multiple convolutional layers that keep the feature dimension unchanged.

8. The training method according to claim 1, wherein: For normal image samples, their teacher features are features extracted using the first teacher neural network trained on a training image set that only includes normal images; For the change image samples, their teacher features are features extracted using a second teacher neural network trained on a training image set that only includes change images.

9. The training method according to claim 1, wherein: The distillation loss is calculated as the distance between the transferred student features of the normal image samples and the teacher features of the corresponding changed image samples, or the distance between the transferred student features of the changed image samples and the teacher features of the corresponding normal image samples.

10. The training method according to claim 1, wherein: The classification loss is calculated by one of the following classification loss functions: ArcFace loss function, CosFace loss function, SphereFace loss function.

11. The training method according to claim 1, wherein: Calculating the total loss associated with the training image set further includes: Calculate the sum of the distillation losses of the normal image sample and the corresponding changed image sample as the total distillation loss; Calculate the weighted sum of the classification losses of the normal image samples and the corresponding changed image samples as the total classification loss; The weighted sum of the total distillation loss and the total classification loss is calculated as the total loss.

12. The training method according to claim 11, wherein: When calculating the total classification loss, the weight of the classification loss of the change image sample is a function of the current training generation and increases as the current training generation increases.

13. The training method according to claim 1, wherein: The total loss is used to update the parameters of the neural network through a back-propagation algorithm.

14. The training method of claim 1, further comprising updating parameters of the transfer transfer adapter and parameters of the classifier using the total loss.

15. The training method of claim 1, wherein the steps of the training method are repeated multiple times until at least one of the following conditions is met: the number of repetitions reaches a predefined maximum number, and the calculated total loss is less than a predefined threshold. The training method according to claim 1 , wherein the object to be recognized comprises a face, a person, or an object.

17. A training device for a neural network for object recognition, characterized in that: include: a feature extractor configured to input a training image set containing an object to be recognized into the neural network to extract a student feature of each image sample, the training image set comprising a set of normal image samples and a set of changed image samples; a dividing unit configured to divide the image samples in the training image set into simple samples and difficult samples based on the extracted student features; For each of the normal image sample and the variation image sample: a transfer migration adapter configured to perform transfer migration on the student features of the image sample based on the division to obtain a migrated student feature; a distillation loss calculation unit configured to calculate a distance between the transferred student feature of the image sample and the teacher feature extracted from the corresponding another image sample as the distillation loss of the image sample; a classifier configured to classify the image sample based on the student feature; and A classification loss calculation unit is configured to calculate the classification loss of the image sample, a total loss calculation unit configured to calculate a total loss associated with the training image set based on the distillation loss and the classification loss calculated for all image samples; as well as An updating unit, configured to update the parameters of the neural network according to the calculated total loss, The transfer migration based on the division further includes: For image samples classified as simple samples, using the deep output of the transfer transfer adapter as the transfer student feature; For image samples classified as hard samples, the shallow output of the transfer adapter is used as the transfer student feature.

18. A method for iterative training of a neural network for object recognition, characterized in that: include: A first training step, using an image set including a set of normal image samples and a set of changed image samples for a first changed scene as a training image set, and using the training method according to claim 1 to train the neural network; as well as In a second training step, an image set including the set of normal image samples and a set of changed image samples for a second changed scene is used as a training image set, and the neural network that has undergone the first training step is trained using the training method according to claim 1.

19. A neural network trained according to the training method of claim 1 or 18.

20. A device comprising processor; and A memory having executable instructions stored thereon, which, when executed by the processor, cause the processor to perform the training method according to any one of claims 1 to 16 and 18.

21. A non-transitory computer-readable storage medium storing executable instructions, the executable instructions implementing the training method of any one of claims 1-16 and 18 when executed.

Images