Deep network model construction method and system based on difficult sample mining

Abstract

The present invention discloses a method and system for constructing a deep network model based on difficult sample mining. The deep network model is constructed and training samples are input to train the deep network model. The feature vector of the training sample is extracted during each iterative training process, the angular boundary margin between the feature vector and its class center weight vector is calculated, and whether the feature vector is a difficult sample within the class is judged according to the angular boundary margin, and the feature vector is marked as a difficult sample according to the judgment result. The angular boundary margin loss value and the difficult sample loss value within the class output by the deep network model are calculated based on the marked feature vector, and the network parameters of the deep network model are optimized by combining the angular boundary margin loss value and the difficult sample loss value within the class. The present invention takes into account the influence of difficult samples within the class on the model accuracy, and supervises the training process based on the angular boundary margin loss and the difficult sample loss within the class, which can improve the compactness of the distribution of features within the class, thereby improving the discrimination of the model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision recognition, and in particular to a method and system for constructing a deep network model based on difficult sample mining. Background Art

[0002] In recent years, computer vision has developed rapidly due to the development of deep neural network technology, including typical tasks such as image classification, image retrieval, object detection, face recognition, pedestrian re-identification, human posture estimation, and action recognition.

[0003] The development of deep learning-based visual tasks mainly depends on the development of loss functions, model network structures, and training data sets. Researchers are more committed to building new loss functions to improve the discriminative power of models. The center loss function has been proven to effectively enhance the compactness of facial feature distribution and improve model performance in face recognition. In recent years, the Softmax loss method based on corner margin margin has been widely used in the field of computer vision, especially in face recognition, pedestrian re-identification, image retrieval and other tasks, achieving advanced levels.

[0004] As an important means to improve training speed and model generalization ability, difficult sample mining technology is widely used in the field of deep metric learning. Difficult sample mining technology is usually related to the loss function. Different loss functions may use different difficult sample mining methods to deal with the problems faced by the loss function. Tripletloss and contrastive loss are widely used in the field of deep metric learning. Constructing triplets or sample pairs through difficult sample mining can speed up the training process and improve the fitting ability of the model. However, the widely used Softmax loss method based on corner margin margin usually ignores the study of difficult samples because the model training is easier and the recognition accuracy is high. In fact, the method based on corner margin margin Softmax loss still has some limitations, which limits the recognition ability of the model to a certain extent. Summary of the invention

[0005] The present invention provides a method and system for constructing a deep network model based on difficult sample mining, which are used to solve the technical problems of low recognition accuracy and poor robustness of existing computer vision methods in practical applications.

[0006] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0007] A method for constructing a deep network model based on difficult sample mining, comprising:

[0008] Construct a deep network model and input training samples to train the deep network model:

[0009] During each training iteration:

[0010] Extract the feature vector of the training sample, calculate the angular margin between the feature vector and its class center weight vector, and determine whether the feature vector is a difficult sample within the class based on the angular margin, and mark the feature vector as a difficult sample based on the determination result; wherein the class center weight vector is the ground truth weight of the class to which the sample belongs

[0011] Based on the labeled feature vector, the corner boundary margin loss value and the intra-class difficult sample loss value output by the deep network model are calculated, and the network parameters of the deep network model are optimized by combining the corner boundary margin loss value and the intra-class difficult sample loss value.

[0012] Preferably, calculating the angular margin between the feature vector and its class center weight vector, and judging whether the feature vector is a difficult sample within the class according to the angular margin, comprises the following steps:

[0013] Calculate the angle between the feature vector and its non-ground truth class center weight vector, select the minimum inter-class angle min(inter) from the angle, and calculate the cosine value cos(min(inter)) of the minimum inter-class angle;

[0014] The feature vectors whose angular boundary margin cosine value between the feature vector and its class center weight vector is less than the cosine value cos(min(inter)) are selected as in-class difficult samples.

[0015] Preferably, the angle between the feature vector and its non-ground truth class center weight vector is calculated by the following formula:

[0016]

[0017] In the formula, x i is the eigenvector of the i-th class, W j is the j-th non-ground truth class center weight vector.

[0018] Preferably, the feature vector is marked as a difficult sample according to the judgment result, which is achieved by the following formula:

[0019]

[0020] Where h(x) represents the labeling function. When the feature vector is a difficult sample within the class, it is labeled as 1, and a non-difficult sample is labeled as 0. i represents the ground truth value of the i-th class; is a margin function for calculating the cosine distance of the feature vector to its class center weight vector after adding the margin; It is the angle between the eigenvector of the i-th class and its class center weight vector.

[0021] Preferably, the corner margin loss value and the intra-class difficult sample loss value are combined to achieve the following loss function:

[0022]

[0023] Where N is the number of training batches, n is the total number of categories, is a A monotonically decreasing function, y i represents the ground truth value of the i-th class feature; It is a margin function, which is used to calculate the cosine distance from the feature vector of the sample in the class to the class center weight vector after adding the margin; is the angle between the feature vector of the sample and the class center weight vector of the i-th class feature, s is a hyperparameter for scaling, λ is a hyperparameter for adjusting the weight of the loss function within the difficult sample class, θ j is the weight W j With the eigenvector x i The angle between.

[0024] Preferably, the Calculated by the following formula:

[0025]

[0026] Preferably, the method further comprises the following steps:

[0027] Use the trained deep network model for visual recognition.

[0028] A computer system comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0029] The present invention has the following beneficial effects:

[0030] 1. The method and system for constructing a deep network model based on difficult sample mining in the present invention, by constructing a deep network model, inputting training samples to train the deep network model: extracting the feature vector of the training sample in each iterative training process, calculating the angular boundary margin between the feature vector and its class center weight vector, and judging whether the feature vector is a difficult sample within the class according to the angular boundary margin, marking the feature vector as a difficult sample according to the judgment result; calculating the angular boundary margin loss value and the difficult sample loss value within the class output by the deep network model based on the marked feature vector, and combining the angular boundary margin loss value and the difficult sample loss value within the class to optimize the network parameters of the deep network model. The present invention takes into account the influence of difficult samples within the class on the accuracy of the model. When training the model, the training process is supervised based on the angular boundary margin loss and the difficult sample loss within the class, which can improve the compactness of the feature distribution within the class, thereby improving the discriminative power of the model, that is, in the training process, the compactness of the feature distribution of difficult samples is emphasized more, so that the feature distribution within the class is more compact as a whole, and the model obtains higher recognition accuracy and stronger robustness.

[0031] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0033] Figure 1 is a flowchart of a method for constructing a deep network model based on difficult sample mining in a preferred embodiment of the present invention;

[0034] Figure 2 It is a flowchart of difficult sample mining in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0035] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0036] Embodiment 1:

[0037] This embodiment discloses a method for constructing a deep network model based on difficult sample mining, including:

[0038] Construct a deep network model and input training samples to train the deep network model:

[0039] During each training iteration:

[0040] Extract the feature vector of the training sample, calculate the angular margin between the feature vector and its class center weight vector, and determine whether the feature vector is a difficult sample within the class (hereinafter referred to as a difficult sample) based on the angular margin, and mark the feature vector as a difficult sample based on the determination result; wherein the class center weight vector is the ground truth weight of the class to which the sample belongs That is, the yth weight of the model i Row vector.

[0041] Based on the labeled feature vector, the corner boundary margin loss value and the intra-class difficult sample loss value output by the deep network model are calculated, and the network parameters of the deep network model are optimized by combining the corner boundary margin loss value and the intra-class difficult sample loss value.

[0042] In addition, in this embodiment, a computer system is also disclosed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0043] In the present invention, the compactness of the feature distribution of difficult samples is emphasized more during the training process, so that the feature distribution within the class is more compact as a whole, and the model obtains higher recognition accuracy and stronger robustness.

[0044] Embodiment 2:

[0045] Embodiment 2 is a preferred embodiment of Embodiment 1. The difference between Embodiment 2 and Embodiment 1 is that the specific steps of the method for building a deep network model based on difficult sample mining are refined:

[0046] In this embodiment, a method for constructing a deep network model based on difficult sample mining is disclosed, comprising the following steps:

[0047] Construct a deep network model and input training samples to train the deep network model:

[0048] During each training iteration:

[0049] Extracting a feature vector of a training sample, calculating an angular margin between the feature vector and its class center weight vector, and judging whether the feature vector is a difficult sample according to the angular margin, and marking the feature vector as a difficult sample according to the judgment result;

[0050] Among them, the positive difficult samples are screened and marked, and the cosine value of the minimum angle between the sample and the non-ground truth class center weight vector is used as the judgment basis, expressed as: cos(min(inter)). If there is a sample whose cosine value of the angular boundary margin from the ground truth class center weight vector is less than cos(min(inter)), then the sample is marked as a difficult sample. Among them, the angle of the sample from the class center weight vector is in the range of [0,π], and its corresponding cosine value is monotonically decreasing. The smaller the angle, the larger the cosine value. The specific formula for judging difficult samples is defined as follows:

[0051]

[0052] Among them, y i represents the ground truth value of the i-th class; It is a margin function, which is used to calculate the cosine distance from the feature vector of the sample in the class to the class center weight vector after adding the margin; It is the angle between the feature vector of the sample and the class center weight vector of class i.

[0053] Based on the labeled feature vector, the corner boundary margin loss value and the intra-class difficult sample loss value output by the deep network model are calculated, and the network parameters of the deep network model are optimized by combining the corner boundary margin loss value and the intra-class difficult sample loss value.

[0054] The calculation of the intra-class difficult sample loss value is realized through the intra-class difficult sample loss function, which is used to optimize the feature distribution of the intra-class positive and difficult samples and improve the intra-class compactness. Its formula is defined as follows:

[0055]

[0056] in, is a A monotonically decreasing function is defined as follows:

[0057]

[0058] Furthermore, by combining the corner margin loss value with the intra-class difficult sample loss value, the network parameters of the deep network model are optimized through the following loss function:

[0059]

[0060] Among them, s is a hyperparameter for scaling, λ is a hyperparameter for adjusting the weight of the loss function within the hard sample class, and θ j is the weight W j With feature x i The angle between.

[0061] Specifically, the deep network model in this embodiment can be applied to various visual recognition tasks, including but not limited to image classification, image retrieval, target detection, face recognition, pedestrian re-identification, human posture estimation, action recognition and other typical tasks.

[0062] Specifically, the deep network model in this embodiment can adopt any one or a combination of AlexNet, VGGNet, Inception, ResNet, DenseNet and MobileNet. The specific model framework can be selected or constructed according to the application scenario and requirements combined with experience, and is not limited here.

[0063] Embodiment three:

[0064] Embodiment 3 is a preferred embodiment of Embodiment 2. The difference between Embodiment 3 and Embodiment 1 is that the application of the deep network model construction method based on difficult sample mining in the field of face recognition is specifically introduced:

[0065] In this embodiment, if Figure 1 As shown, a method for constructing a deep network model based on difficult sample mining is disclosed, wherein the deep network model is used for face recognition, and comprises the following steps:

[0066] S1. Image input. The face training image uses a size of 112*112 pixels. MTCNN is used for face detection and alignment.

[0067] S2, feature extraction, feature extraction of face images in the current iteration process. In order to obtain stronger feature representation capabilities, the feature dimension is selected as 512, and ResNet100 is used as the network structure of the model;

[0068] S3, intra-class difficult sample mining, complete the screening and marking of difficult samples in the current iteration process according to the difficult sample selection formula (3);

[0069] Among them, Figure 2 As shown in Figure 1, the intra-class difficult sample mining specifically includes the following steps:

[0070] 101) Calculate sample feature X i and the non-ground truth class center weight vector W j The angle between them is defined as:

[0071]

[0072] 102) Calculate the cosine value of the minimum angle between the sample feature and the non-ground truth class center weight vector, expressed as cos(min(inter)). Where min(inter) is the minimum inter-class angle, that is, the sample feature X iand the non-ground truth class center weight vector W j The minimum value of the angle between them.

[0073] 103) Screen difficult samples whose cosine value of the corner margin between the sample features and the class center weight vector is less than cos(min(inter)). In this embodiment, ArcFace is used to directly add the corner margin, and the difficult sample selection function is equivalent to:

[0074]

[0075] Among them, m is the margin hyperparameter, which is set to 0.5 during training.

[0076] 104) Mark the hard samples in the current iteration.

[0077] S4, based on the joint supervision training of corner margin loss and intra-class difficult sample loss, this embodiment adopts Arcface, a typical representative of corner margin loss, and combines it with the intra-class difficult sample loss to jointly supervise the training of the deep network model. The joint loss function is defined as follows:

[0078]

[0079] Among them, m is the boundary margin hyperparameter, which is set to 0.5 during training, λ is set to 10, and s is set to 64.

[0080] In summary, the method and system for constructing a deep network model based on difficult sample mining in the present invention constructs a deep network model and inputs training samples to train the deep network model: extract the feature vector of the training sample in each iterative training process, calculate the angular boundary margin between the feature vector and its class center weight vector, and judge whether the feature vector is a difficult sample based on the angular boundary margin, and mark the feature vector as a difficult sample based on the judgment result; calculate the angular boundary margin loss value and the intra-class difficult sample loss value output by the deep network model based on the marked feature vector, and optimize the network parameters of the deep network model by combining the angular boundary margin loss value and the intra-class difficult sample loss value. The present invention takes into account the influence of intra-class difficult samples on the model accuracy. When training the model, the training process is supervised based on the angular boundary margin loss and the intra-class difficult sample loss, which can improve the compactness of the intra-class feature distribution, thereby improving the discriminative power of the model, that is, in the training process, more emphasis is placed on the compactness of the feature distribution of difficult samples, so that the intra-class feature distribution is more compact as a whole, and the model obtains higher recognition accuracy and stronger robustness.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for constructing a deep network model based on difficult sample mining, characterized in that: include: Construct a deep network model and input training samples to train the deep network model: During each training iteration: Extract the feature vector of the training sample, calculate the angular margin between the feature vector and its class center weight vector, and determine whether the feature vector is a difficult sample within the class based on the angular margin, and mark the feature vector as a difficult sample based on the determination result; wherein the class center weight vector is the ground truth weight of the class to which the sample belongs The corner margin loss value and the intra-class difficult sample loss value output by the deep network model are calculated based on the labeled feature vector, and the network parameters of the deep network model are optimized by combining the corner margin loss value and the intra-class difficult sample loss value. Calculating the angular margin between the feature vector and its class center weight vector, and judging whether the feature vector is a difficult sample within the class according to the angular margin, comprises the following steps: Calculate the angle between the feature vector and its non-ground truth class center weight vector, select the minimum inter-class angle min(inter) from the angle, and calculate the cosine value cos(min(inter)) of the minimum inter-class angle; Select the feature vector whose angular margin cosine value between the feature vector and its class center weight vector is less than the cosine value cos(min(inter)) as the difficult sample within the class; The angle between the feature vector and its non-ground truth class center weight vector is calculated using the following formula: Where xi is the feature vector of the i-th class, and Wj is the center weight vector of the j-th non-ground truth class; The feature vector is marked as a difficult sample according to the judgment result, which is achieved by the following formula: Where h(x) represents the labeling function. When the feature vector is a difficult sample within the class, it is labeled as 1, and a non-difficult sample is labeled as 0. yi represents the ground truth value of the i-th class. is a margin function for calculating the cosine distance of the feature vector to its class center weight vector after adding the margin; is the angle between the eigenvector of the i-th class and its class center weight vector; Combining the corner margin loss value with the intra-class difficult sample loss value, this is achieved through the following loss function: Where N is the number of training batches, n is the total number of categories, is a In the monotonically decreasing function, yi represents the ground truth value of the i-th class feature; It is a margin function, which is used to calculate the cosine distance from the feature vector of the sample in the class to the class center weight vector after adding the margin; is the angle between the feature vector of the sample and the class center weight vector of the i-th class feature, s is a hyperparameter for scaling, λ is a hyperparameter for adjusting the weight of the loss function within the difficult sample class, and θj is the angle between the weight Wj and the feature vector xi; Said Calculated by the following formula: The following steps are also included: Use the trained deep network model for visual recognition.

2. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to claim 1 are implemented.

Images