Training method and device for object recognition model
By dynamically controlling the optimization amplitude and using weight functions of the convolutional neural network model, the noise problem of training data set is solved, the performance and generalization ability of the object recognition model are improved, and overfitting is reduced.
Patent Information
- Application Number
- CN201911082558.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-11-07
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-11-07
AI Technical Summary
In the prior art, neural network models for face/object recognition are difficult to effectively process noise-containing data sets during training, resulting in poor model performance and overfitting problems.
By dynamically controlling the optimization/update amplitude of the convolutional neural network model, using the loss function of the same direction monotonic change in a specific value interval, adaptive matching training process, avoid overfitting the noise sample, and improve the generalization of the model ability to transform.
Even in the case of training data sets containing noise, high-performance training models can be obtained, which improves the accuracy of object recognition and the generalization ability of the model, and reduces overfitting.
Smart Images

Figure CN112784953B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to object recognition, and more particularly to neural network models for object recognition. Background Art
[0002] In recent years, object detection / recognition / matching / tracking in static images or a series of moving images (such as videos) has been widely and importantly applied in the fields of image processing, computer vision and pattern recognition, and plays an important role in them. The object can be a human body part such as face, hand, body, etc., other organisms or plants, or any other object that you want to detect. Face / object recognition is one of the most important computer vision tasks, and its goal is to identify or verify a specific person / object based on the input photo / video.
[0003] In recent years, neural network models for face recognition, especially deep convolutional neural network (CNNs) models, have made breakthrough progress in significantly improving performance. Given a training dataset, the CNN training process uses a general CNN architecture as a feature extractor to extract features from the training images, and then calculates the loss data for supervised training of the CNN model by using various designed loss functions. So when the CNN architecture is selected, the performance of the face recognition model is driven by the loss function and the training dataset. Currently, the Softmax loss function and its variants (boundary-based Softmax loss function) are commonly used supervisory functions in face / object recognition.
[0004] However, it should be pointed out that training datasets are often not ideal datasets. On the one hand, they cannot fully describe the real world. On the other hand, existing training datasets still contain noise samples even after cleaning. For such training datasets, the existing Softmax loss function and its variants cannot achieve ideal results and cannot effectively improve the performance of the training model.
[0005] Therefore, improved techniques are needed to improve the training of object recognition models.
[0006] Unless otherwise stated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Likewise, unless otherwise stated, problems identified with respect to one or more approaches should not be assumed to have been identified in any prior art based on this section. Summary of the invention
[0007] One object of the present disclosure is to improve the training optimization of recognition models for object recognition. Another object of the present disclosure is to improve object recognition in images / videos.
[0008] The present disclosure proposes an improved training of a convolutional neural network model for object recognition, wherein the optimization / update amplitude for the convolutional neural network model, also known as the convergence gradient descent speed, is dynamically controlled during the training process so that it can adaptively match the progress of the training process, thereby obtaining a high-performance training model even for a noisy training data set.
[0009] The present disclosure also proposes to use the model obtained through the above training to perform object recognition, so as to further obtain improved object recognition results.
[0010] In one aspect, an optimization device for a neural network model for object recognition is provided, comprising: a loss determination unit, configured to determine loss data for features extracted from a training image set using the neural network model and a loss function with a weight function, and an update unit, configured to perform an update operation on parameters of the neural network model based on the loss data and the update function, wherein the update function is derived from a loss function with a weight function of the neural network model, and the weight function and the loss function change monotonically in the same direction in a specific value range.
[0011] On the other hand, a training method for a neural network model for object recognition is provided, comprising: a loss determination step for determining loss data for features extracted from a training image set using the neural network model and a loss function with a weight function, and an update step for performing an update operation on parameters of the neural network model based on the loss data and the update function, wherein the update function is derived from a loss function with a weight function of the neural network model, and the weight function and the loss function change monotonically in the same direction in a specific value range.
[0012] In yet another aspect, a method is provided comprising at least one processor and at least one storage device having instructions stored thereon, which when executed by the at least one processor may cause the at least one processor to perform the method as described herein.
[0013] In still another aspect, a storage medium storing instructions is provided, which when executed by a processor may cause the method as described herein to be performed.
[0014] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention. In the drawings, like reference numerals indicate like items.
[0016] Figure 1 A schematic diagram of the prior art of face recognition / authentication using a convolutional neural network model is shown.
[0017] Figure 2A A convolutional neural network model training flow chart according to the prior art is shown.
[0018] Figure 2B A schematic diagram showing the training results of a convolutional neural network model according to the prior art.
[0019] Figure 3A The mapping of image feature vectors on the hypersphere manifold is shown.
[0020] Figure 3B A schematic diagram showing the training results of an image feature vector when trained by a convolutional neural network model.
[0021] Figure 3C A schematic diagram showing the training results of a convolutional neural network model according to the present disclosure is shown.
[0022] Figure 4A A block diagram of a convolutional neural network model training device according to the present disclosure is shown.
[0023] Figure 4B A flowchart of a convolutional neural network model training method according to the present disclosure is shown.
[0024] Figure 5A Plots of the intra-class weight function and the inter-class weight function are shown.
[0025] Figure 5B The final adjustment curves of intra-class gradients and inter-class gradients are shown.
[0026] Figure 5C Indicates that the optimized gradient is along the tangent direction.
[0027] Figure 5D Plots of the intra-class gradient rescaling function and the inter-class gradient rescaling function as a function of the parameters are shown.
[0028] Figure 5E The final adjustment curves of intra-class gradient and inter-class gradient with respect to the parameters are shown.
[0029] Fig. 5F The adjustment curves of intra-class gradient and inter-class gradient of the prior art are shown.
[0030] Figure 6 A basic conceptual flow chart of convolutional neural network model training according to the present disclosure is shown.
[0031] Figure 7A flowchart of convolutional neural network model training according to the first embodiment of the present disclosure is shown.
[0032] Figure 8 A flowchart of convolutional neural network model training according to the second embodiment of the present disclosure is shown.
[0033] Fig. 9 A flowchart of adjusting weight function parameters in a convolutional network model according to the third embodiment of the present disclosure is shown.
[0034] Fig.10 A flowchart of adjusting weight function parameters in a convolutional network model according to the fourth embodiment of the present disclosure is shown.
[0035] Fig.11 A flowchart of online training of a convolutional neural network model according to the fifth embodiment of the present disclosure is shown.
[0036] Fig.12 Schematic diagram showing that an input image can be used as a suitable training sample for an object in the training dataset
[0037] Fig.13 A schematic diagram showing that an input image can serve as a suitable training sample for a new object in a training dataset.
[0038] Fig.14 A block diagram showing an exemplary hardware configuration of a computer system capable of implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0039] Exemplary possible embodiments related to model training optimization for object recognition are described herein. In the following description, for the purpose of illustration, many specific details are set forth in order to provide a thorough understanding of the present invention. However, it is apparent that the present invention can be practiced without these specific details. In other cases, well-known structures and devices are not described in detail to avoid unnecessary obstruction, obscuring or obscuring the present invention.
[0040] In the context of the present disclosure, an image may refer to any of a variety of images, such as a color image, a grayscale image, etc. It should be noted that in the context of the present specification, the type of image is not specifically limited, as long as such an image can be processed so that it can be detected whether the image contains an object. In addition, the image can be an original image or a processed version of the image, such as a version of the image that has been subjected to preliminary filtering or preprocessing before performing the operations of the present application on the image.
[0041] In the context of this specification, an image containing an object refers to an image containing an object image of the object. The object image may also be sometimes referred to as an object region in an image. Object recognition also refers to recognition of an image in an object region in an image.
[0042] In this article, the object can be a human body part, such as face, hand, body, etc., other organisms or plants, or any other object that you want to detect. As an example, the features of the object, especially the representative features, can be represented in the form of a vector, which can be called the "feature vector" of the object. For example, in the case of detecting a face, the pixel texture information, position coordinates, etc. of the representative part of the face are selected as features to form the feature vector of the image. Thus, based on the obtained feature vector, object recognition / detection / tracking can be performed. It should be noted that the feature vector may vary depending on the model used in object recognition and is not particularly limited.
[0043] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it need not be discussed again for subsequent drawings.
[0044] In the present disclosure, the terms "first", "second", etc. are used merely to distinguish elements or steps, but are not intended to indicate time sequence, priority or importance.
[0045] Figure 1 A basic operational concept diagram of face recognition / authentication using a deep face model in the prior art is shown, which mainly includes a training phase and an application phase of the deep face model, and the deep face model may be, for example, a deep convolutional neural network model.
[0046] In the training phase, the face image training set is first input into the deep face model to obtain the feature vector of the face image, and the existing loss function, such as the Softmax loss function and its variants, is used to obtain the classification probability P from the feature vector. 1 , P 2 , P 3 ,…,P c (where c indicates the number of categories in the training set, for example, there are c categories of face IDs), the classification probability indicates the probability that the image belongs to each of the c categories, and then the obtained classification probability is compared with the true value 0, 1, 0, ..., 0 (where 1 indicates the true value) to determine the difference between the two, such as cross entropy, as loss data, and feedback is performed based on the difference to update the deep face model, and the aforementioned operation is continued with the updated face model until specific conditions are met, thereby obtaining a trained deep face model.
[0047] During the testing phase, the face image to be identified or authenticated can be input into the trained deep face model to extract features for identification or authentication. Specifically, in the actual application system, there can be two specific applications: face / object recognition and face / object verification. The input of face / object recognition is generally a single face / object image, and a trained convolutional neural network is used to identify whether the face / object in the current image is the object to be identified; the input of face / object verification is generally a face / object image pair, and a trained convolutional neural network is used to extract the feature pairs of the input image pair, and finally the similarity of the feature pairs is used to determine whether the input image pair is the same object.
[0048] Figure 1 An exemplary face authentication operation is shown in FIG. In operation, two face images to be authenticated are input into the trained deep face model to authenticate whether the two face images are face images of the same person. Specifically, the deep face model can obtain feature vectors for the two face images respectively to form a feature vector pair, and then determine the similarity between the two feature vectors, for example, the similarity can be determined by a cosine function. When the similarity is not less than a specific threshold, the two face images can be considered to be face images of the same person, and when the similarity is less than a specific threshold, the two face images can be considered not to be face images of the same person.
[0049] From the above description, it can be seen that the performance of the deep face model directly affects the accuracy of object recognition, and various methods are used in the prior art to train the deep face model, such as the deep convolutional neural network model, to obtain a more complete deep convolutional neural network model. Figure 2A To describe the training process of the deep convolutional neural network model in the prior art.
[0050] First, a training data set is input, which may include a large number of object images, such as face images, such as tens of thousands, hundreds of thousands, or millions of object images.
[0051] Then, the images in the input training data set may be preprocessed, and the preprocessing operations may include, for example, object detection, object alignment, normalization, and the like. In particular, object detection may, for example, refer to detecting a face from an image containing a face and obtaining an image mainly containing a face to be recognized, and object alignment may refer to aligning object images in different postures in the image to the same or appropriate posture, thereby performing object detection / recognition / tracking based on the aligned object images. Face recognition is a common object recognition operation, and for a face recognition training image set, preprocessing including, for example, face detection, face alignment, and the like may be performed. It should be noted that the preprocessing operation may also include other types of preprocessing operations known in the art, which will not be described in detail here.
[0052] Then, the preprocessed training set images are input into the deep convolutional neural network model for feature extraction. The convolutional neural network model can adopt various structures and parameters known in the art, etc., which will not be described in detail here.
[0053] The loss is then calculated using a loss function, especially the Softmax loss function and its variants. The Softmax loss function and its variants (boundary-based Softmax loss function) are commonly used supervisory signals in face / object recognition. These loss functions encourage separation between features, with the goal of minimizing the intra-class distance while maximizing the inter-class distance in an ideal case. The general form of the Softmax loss function is as follows:
[0054]
[0055] where x i ∈ d is the embedding feature of the i-th training image, y i is x i The category label, C represents the number of categories in the training data set, W = {W 1 ,W 2 ,…,W C}∈ d×C represents the weight of the last fully connected layer in DCNN, W j ∈ d is the weight vector of the jth column of the last fully connected layer in DCNN, b j ∈R C In the prior art, the softmax-based loss function removes the bias term and converts it to W j T x i =scosθ j . L indicates probability.
[0056] Then, the parameters of the convolutional neural network are updated through back propagation based on the calculated loss data.
[0057] However, existing technical methods all assume that given an ideal training dataset, the intra-class distance is strictly minimized while the inter-class distance is maximized. Therefore, the loss functions adopted are all aimed at strictly minimizing the intra-class distance and maximizing the inter-class distance. In this way, the convergence gradient scale adopted when updating / optimizing during the model training process is fixed, which may lead to overfitting due to defects in the current training dataset, such as interference from noise samples.
[0058] Specifically, during the training process, the existing training method is to first learn the features of clean samples so that the model can effectively identify most of the clean samples; then continuously optimize the noise samples along the gradient direction. However, regardless of the current distance between the training samples, the optimized convergence gradient scale is fixed. In this way, the noisy samples may also move in the wrong direction at a constant speed. Therefore, in the later stages of training, the noise samples will be incorrectly projected to the feature space area of other objects, causing the model to overfit. Figure 2B As shown in the figure, the noise samples will be included in the W2 class samples corresponding to ID2 as the training set is processed, and cannot be effectively separated from the clean samples. This results in the training effect not being optimal, affecting the trained model, and may even lead to recognition errors for the face image of ID2.
[0059] In addition, the loss function used in the model training process of the prior art performs a relatively complex transformation on the extracted features, for example, from the domain of feature vectors to the probability domain. Such a transformation will inevitably bring about certain conversion errors, which may lead to reduced accuracy and increased computational overhead.
[0060] Moreover, the loss functions used in the model training process of the prior art, such as the Softmax loss function and its variants, all mix the intra-class distance and the out-of-class distance together to calculate the probability. This mixing of the intra-class distance and the out-of-class distance is not convenient for providing targeted analysis and optimization, which may result in inaccurate convergence of the model training and inability to obtain a further optimized model.
[0061] The present disclosure is proposed in view of the above-mentioned situation in the prior art. In the model optimization method in the present disclosure, the model update / optimization amplitude during the model training process, especially the convergence gradient descent speed, can be dynamically controlled. In particular, the convergence gradient descent speed can adaptively match the progress of the training process, and in particular, it can dynamically change as the training process progresses, especially when the convergence is close to the optimal training result, the slower the convergence or even stops.
[0062] This can prevent overfitting of noise samples to ensure that model training can effectively adapt to the training data set, so that a high-performance training model can be obtained even for a noisy training data set. In other words, even for a training data set that may contain noisy images, the noisy images can still be effectively separated from the clean images, and overfitting can be suppressed as much as possible, so that the model training can be further optimized to obtain an improved recognition model, and thus obtain better face recognition results.
[0063] The following will exemplarily explain the specific parameters involved in the training of the deep convolutional neural network model according to the present disclosure in conjunction with the accompanying drawings, especially the image feature vector, in-class loss and out-of-class loss, etc.
[0064] In the implementation of the present disclosure, the deep embedding features of all training samples are mapped to a hyperspherical manifold, where x i ∈ d represents the embedded features of the i-th training image, y i is x i The category label, W yi ∈ d For category y i The target center feature. yi is x i and the target center feature W yi The angle between j is the target center feature of other categories, θ j is x i and the target center features W of other categories j The angle between intra (θ yi ) is the scale of the intra-class gradient, ν inter (θ i ) is the scale of the gradient between classes. The longer the arrow, the larger the gradient. Figure 3A As shown. The optimized gradient direction always moves along the tangent of the hypersphere, where the movement direction of the intra-class gradient indicates that it is intended to reduce the intra-class angle, while the movement direction of the inter-class gradient indicates that it is intended to increase the inter-class angle. Based on such a mapping process, the intra-class angle and the inter-class angle can be used as the intra-class distance and the inter-class distance for adjustment targets respectively.
[0065] According to the implementation of the present disclosure, an improved weight function is proposed for dynamically controlling the model update / optimization amplitude during the training process, that is, the convergence gradient descent speed.
[0066] In order to constrain the optimization amplitude, the design idea of the weight function disclosed in the present invention is to design an effective mechanism to limit the gradient amplitude, that is, to be able to flexibly control the convergence speed of the gradient suitable for a noisy training data set. In other words, through the use of the weight function, a variable amplitude is used to control the training convergence during the training process of the convolutional neural network model, and when approaching the optimal training result, the convergence speed will become slower and slower or even not be executed, so that it will not forcibly fix the convergence like the prior art, but the convergence can be appropriately stopped or slowed down, avoiding overfitting of noisy samples, ensuring that the model can effectively adapt to the training data set, thereby improving the performance of the training model from the generalization aspect.
[0067] Figure 3BA schematic diagram of the training results of the convolutional neural network model according to the present disclosure is shown, in which the noise characteristics are basically effectively separated from the category characteristics without causing overfitting. Specifically, during the training process, the convergence is strong in the early stage of the iteration. As the iteration process proceeds, the gradient convergence ability becomes smaller and smaller in the middle stage of the training, and finally in the late stage of the training, the gradient convergence almost stops, so that the noise characteristics basically do not affect the trained model.
[0068] Figure 3C The basic situation after classification is shown, where ID1, ID2, and ID3 indicate three categories respectively, where the features of the training images of the same category are clustered together as much as possible, that is, the intra-category angle is as small as possible, and the features of the training images of different categories are separated as much as possible, that is, the inter-category angle is as large as possible.
[0069] The following will describe an embodiment of the object recognition model training of the present disclosure with reference to the accompanying drawings.
[0070] Figure 4A A block diagram of an optimization device for a neural network model for object recognition according to the present disclosure is shown. The device 400 includes a loss determination unit 401, configured to determine loss data for features extracted from a training image set using the neural network model, and an update unit 402, configured to perform an update operation of parameters of the neural network model based on the loss data and an update function. The update function is obtained based on the loss function and the corresponding weight function of the neural network model, and the weight function and the loss function change monotonically in the same direction in a specific value interval.
[0071] In an embodiment of the present disclosure, the neural network model may be a deep neural network model, and the acquired image features are deeply embedded features of the image.
[0072] The weight function of the present disclosure will be described below in conjunction with the accompanying drawings.
[0073] In this disclosure, Figure 3A In the case where the deep embedding features of the training samples shown in are mapped to the hypersphere manifold, the weight function can constrain the optimized gradient direction of the training samples and the target center to always move along the tangent speed of the hypersphere.
[0074] According to an embodiment of the present disclosure, the weight function and the loss function may both be functions of an angle, wherein the angle is the angle between the extracted features mapped onto the hypersphere manifold and a specific weight vector in the fully connected layer of the neural network model. In particular, the specific weight vector may be the feature center of a certain category of objects in the training atlas. For example, for the features extracted from the training image, the specific weight vector may be the target feature center of the category to which the training image belongs and the target feature center of other categories, and accordingly, the angle may include at least one of an intra-class angle and an inter-class angle.
[0075] Therefore, the angle between the feature vectors can be directly optimized as the loss function target, without converting the feature vector into cross entropy and using cross entropy loss as the loss function as in the prior art, so as to ensure that the loss function target is consistent with the prediction process target. Specifically, the target of the loss function can be the angle between specific object vectors, and in the prediction stage, such as the aforementioned object verification stage, the angle between the two extracted object feature vectors is used to determine whether they are the same object. In this case, the target of the prediction process is also the angle, so the target of the loss function can be known to the target of the prediction process. In this way, the operation of determining the loss data and providing feedback based on it can be simplified, the intermediate conversion processing can be reduced, the computational overhead can be reduced, and the degradation of computational accuracy can be avoided.
[0076] According to an embodiment of the present disclosure, the weight function corresponds to the loss function. According to an embodiment of the present disclosure, when the loss function includes at least one sub-function, the weight function may correspond to at least one of the at least one sub-function included in the loss function. As an example, the weight function may be a weight function corresponding to one of the at least one sub-functions included in the loss function. As another example, the weight function may include more than one sub-weight functions corresponding to more than one sub-functions included in the loss function, where the more than one sub-weight functions are the same as the number of the more than one sub-functions.
[0077] According to an embodiment of the present disclosure, monotonic change in the same direction means that the weight function and the loss function change in the same direction as the value changes within a specific value interval, for example, they increase or decrease in the same direction as the value increases. According to an embodiment of the present disclosure, the specific value interval can be a specific angle value interval, in particular, an optimized angle interval corresponding to the intra-class angle or the inter-class angle. Preferably, in the case of the hypersphere manifold mapping as described above, the intra-class angle and the inter-class angle can be optimized within [0, π / 2], so the specific angle value interval is [0, π / 2], and preferably, the weight function and the loss function can change monotonically in the same direction in the specific angle value interval, in particular, can change monotonically and smoothly in the same direction.
[0078] According to an embodiment of the present disclosure, the weight function can be various types of functions, as long as it can change monotonically in the same direction as the loss function in the specific angle value range, and has cutoff points near the two end points of the value range, especially the slope of the curve is basically zero near the end points of the value range.
[0079] According to an embodiment of the present disclosure, the weight function may be a Sigmoid function or a similar function, and its expression may be:
[0080]
[0081] Among them, S is the initial scale parameter that controls the gradient of the Sigmoid curve; n and m are the parameters that control the slope and horizontal intercept of the Sigmoid curve respectively. These parameters actually control the flexible interval to suppress the movement speed of the gradient. Therefore, the scalar function as an angle can be obtained from the weight function to re-adjust the optimization target, that is, the angle. Figure 5A A graph of a possible weight function is shown, which is either monotonically increasing or monotonically decreasing between 0 and π / 2, and maintains a substantially constant value near the endpoints close to 0 and π / 2, wherein the left graph may refer to a weight function for intra-class loss, and the right graph may refer to a weight function for inter-class loss, the horizontal axis indicates an angle value range, such as approximately 0 to 1.5, and the vertical axis indicates a scale range, such as approximately 0 to 70, which may also be similar to Figures 5D to 5E The values of are similar, but it should be noted that the values are only exemplary and will be described in detail below.
[0082] According to one implementation, since the original gradient magnitude of the loss is related to the sine function of the angle, the final gradient magnitude is proportional to the combination (e.g., product) of the weight function and the sine function. Therefore, when the weight function is such a Sigmoid function or a similar function, the converged gradient magnitude during the update process can be determined accordingly, such as Figure 5B As shown, the left curve graph may refer to the convergence gradient amplitude of the intra-class loss, and the right curve graph may refer to the convergence gradient amplitude of the inter-class loss. The horizontal axis indicates the angle value range, and the vertical axis indicates the scale range. The values are only exemplary, for example, Figures 5D to 5E The values of are similar.
[0083] According to an embodiment of the present disclosure, the parameters of the weight function can be set / selected based on the average performance of the training data and the validation data. According to an embodiment of the present disclosure, the parameters of the weight function, including the slope parameter and the intercept parameter, such as at least one of the parameters n, m, etc. in the above-mentioned Sigmoid function or similar functions, can also be adjusted.
[0084] According to an embodiment of the present disclosure, after a round of training (possibly after a specific number of iterations) is completed, it can be determined whether to further adjust the parameter based on specific conditions. As an example, the specific condition may be related to the training result or the number of adjustments. As an example, when a predetermined number of adjustments is reached, parameter adjustment may no longer be performed. As another example, the comparison between the current training result and the previous training result can be used for selection. The training result can be, for example, loss data determined by a model determined by this training. If the current training result is inferior to the previous training result, the parameter is not adjusted, and if the current training result is better than the previous training result, it can be attempted to continue adjusting the parameter in the previous parameter adjustment method until the predetermined number of adjustments is reached or the training result is no longer improved.
[0085] According to an embodiment of the present disclosure, for a parameter of a weight function, two initial parameter values can be set, and then each parameter value is used to perform iterative loss data determination operations and update operations. After each training is completed, a parameter value that leads to a better training result (for example, the loss data caused by the trained model) is selected, and the other two parameter values around this parameter value are set as the parameters of the weight function used in the next training operation. This is repeated until a predetermined number of adjustments is reached or the training result is no longer optimized. As an example, for a parameter n of the weight function, the initial value can be set to 1 and 1.2, and after a round of iterations, it is found that the parameter effect of n=1 is better, then the value of n can be further set to 0.9, 1.1, and then the subsequent iterations and adjustments are repeated until a predetermined number of adjustments is reached or the training result is no longer optimized.
[0086] According to an embodiment, for multiple parameters in the weight function, multiple ways can be used to adjust. As an example, the parameters can be adjusted one by one, that is, after one parameter is adjusted, another parameter is adjusted. Each parameter can be adjusted as described above, and other parameters can maintain fixed values during its adjustment process. In particular, for the above-mentioned Sigmoid function or similar function, the slope parameter can be adjusted first, and then the intercept parameter can be adjusted. As another example, multiple parameters can be adjusted simultaneously, for example, each parameter can be adjusted in the same way as the previous adjustment method, so as to use a new set of parameter values for subsequent training.
[0087] For example, two initial sets of values, i.e., a first set of values and a second set of values, may be set for the hyperparameters, and each set of initial values may be used to perform model training to obtain the performance of the corresponding validation data set. The better performance is selected by comparing the performance of the validation data set obtained, and two sets of improved parameter values are set near the initial parameter values corresponding to the better performance, and the model is trained again using the two sets of improved parameter values, and the process is iterated in sequence until the most appropriate hyperparameter position is determined.
[0088] The loss function according to the present disclosure will be described below.
[0089] According to an embodiment of the present disclosure, the loss function used to calculate the loss data is not particularly limited. As an example, a general loss function can be used to calculate the loss data, and the loss function can be, for example, the original loss function of the neural network model, which can be related to the angle.
[0090] According to an embodiment of the present disclosure, the loss function is a function that changes substantially monotonically within a specific value interval. As an example, when the specific value interval is a specific angle value interval [0, π / 2], the loss function may be a cosine function of the angle, and accordingly, the weight function will also change monotonically in the same direction as the loss function within the specific value interval.
[0091] According to another implementation of the present disclosure, a new loss function determined based on the weight function according to the present disclosure is proposed, whereby the loss function is used to obtain loss data in model training for object recognition, and the model is updated / optimized based on the loss data and the weight function, so that the update / optimization amplitude of the model can be further adaptively controlled, and the update / optimization of the model can be further improved. According to one embodiment, the loss function used to calculate the loss data may be a combination of the loss function and the weight function of the neural network model. According to one embodiment, the loss function used to calculate the loss data may be the product of the loss function and the weight function of the neural network model. In particular, the loss function of the neural network model here refers to the original loss function that is not weighted by the weight function, and the loss function used to calculate the loss data may refer to a weighted function obtained by weighting the original loss function by the weight function.
[0092] According to an embodiment of the present disclosure, the loss data to be considered may include both intra-class loss and inter-class loss. Thus, the loss function used for model training may include two sub-functions: an intra-class loss function and an inter-class loss function. In the case of the aforementioned hypersphere manifold mapping, these two sub-functions may both be related to angles, namely, an intra-class angle loss function and an inter-class angle loss function. Thus, the intra-class loss and the out-of-class loss may be analyzed and optimized respectively, so that the intra-class gradient term and the inter-class gradient term may be decoupled, which helps to analyze and optimize the intra-class loss term and the inter-class loss term respectively.
[0093] According to the implementation of the present disclosure, the loss function of the present disclosure may include an intra-class loss function and an inter-class loss function, and at least one of the intra-class loss function and the inter-class loss function may have a weight function corresponding thereto, so that the weight function may be used to update / optimize the model in the model training for object recognition. For example, for the intra-class loss or the inter-class loss, the model may be updated / optimized by using the intra-class update function or the inter-class update function determined based on the weight function, thereby improving the control of the model update / optimization to a certain extent.
[0094] According to the implementation of the present disclosure, the loss function of the present disclosure may include an intra-class loss function and an inter-class loss function, and at least one of the intra-class loss function and the inter-class loss function is determined based on a weight function corresponding thereto, so that the at least one of the intra-class loss function and the inter-class loss function is a weighted function weighted by the corresponding weight function. Preferably, both the intra-class loss function and the inter-class loss function included in the loss function may be weighted functions weighted by the corresponding weight function.
[0095] According to an embodiment of the present disclosure, the loss function includes an intra-class angle loss function, wherein the intra-class angle is the angle between the extracted features mapped onto the hypersphere manifold and the weight vector representing the true value object in the fully connected layer of the neural network model, and wherein the update function is determined based on the intra-class angle loss function and the weight function of the intra-class angle.
[0096] According to an embodiment of the present disclosure, the intra-class angle loss function is mainly intended to optimize the intra-class angle, especially to moderately reduce the intra-class angle, so the intra-class angle loss function should decrease as the intra-class angle decreases. In other words, the intra-class angle loss function should be a monotonically increasing function over a specific value interval. Accordingly, the weight function of the intra-class angle is a non-negative monotonically increasing function over a specific value interval, preferably a smooth monotonically increasing function.
[0097] As an example, the intra-class angle value interval is [0, π / 2], and the intra-class angle loss function can be a cosine function of the intra-class angle, especially a cosine function of the intra-class angle with a negative value. And the weight function of the intra-class angle has a horizontal cutoff point near 0.
[0098] As an example, the intra-class angle loss function can be − For x i / ‖x i ‖and The intra-class angular distance between .
[0099] As another example, the intra-class angle loss function can be determined based on the weight function in is the gradient rescaling function for intra-class angles, which corresponds to the weight function of the present disclosure, The block gradient operator is used to weight the intra-class cosine angle distance loss during training. Its constant value is calculated for weighting at each training iteration, and its contribution is not considered in the subsequent gradient calculations.
[0100] According to an embodiment of the present disclosure, the loss function further includes an inter-class angle loss function, and wherein the inter-class angle is the angle between the extracted features mapped onto the hypersphere manifold and other weight vectors in the fully connected layer of the neural network model, and wherein the update function is determined based on the inter-class angle loss function and the weight function of the inter-class angle.
[0101] According to an embodiment of the present disclosure, the inter-class angle loss function is mainly intended to optimize the inter-class angle, especially to moderately increase the inter-class angle, so the inter-class angle loss function should decrease as the inter-class angle increases. In other words, the intra-class angle loss function should be a monotonically decreasing function over a specific value interval. Accordingly, the weight function of the intra-class angle is a non-negative monotonically decreasing function over a specific value interval, preferably a smooth monotonically decreasing function.
[0102] As an example, the inter-class angle value interval is [0, π / 2], and the inter-class angle loss function may be a cosine function of the inter-class angle, in particular, a cosine function of the inter-class angle. And the inter-class angle weight function has a horizontal cutoff point near π / 2.
[0103] As an example, the inter-class angle loss function can be in j≠y i ,θ j (j≠y i ) is x i / ‖x i ‖ and W j / ||W j Here, C is the number of categories in the training image set.
[0104] As another example, the inter-class angle loss function can be determined based on the weight function where r inter (θ j ) is the gradient rescaling function for the inter-class angle, which corresponds to the weight function of the present disclosure, [r inter (θ j )] b The block gradient operator is used to weight the inter-class cosine angle distance loss during training. Its constant value is calculated for weighting at each training iteration, and its contribution is not considered in subsequent gradient calculations.
[0105] The operation of the update function and the update unit according to the present disclosure will be described below.
[0106] According to an embodiment of the present disclosure, the update function may be determined based on a loss function and a weight function. According to one embodiment, the update function may be based on a partial derivative of the loss function and a weight function. Preferably, the update unit is further configured to multiply the partial derivative of the loss function by the weight function to determine an update gradient for updating the neural network model. It should be noted that, as an example, the loss function described here may refer to an initial loss function in a neural network model, such as a loss function that is not weighted by a weight function.
[0107] According to an embodiment of the present disclosure, in the case where the loss function includes at least one sub-loss function, the update function may be determined based on at least one of the at least one sub-loss function, such as its partial derivative, and a weight function corresponding thereto. As an example, the update function may be determined based on a sub-loss function of the at least one sub-loss function, such as its partial derivative, and a weight function corresponding to the sub-loss function. As another example, the update function may be determined based on more than one sub-loss function, such as its partial derivative, and weight functions corresponding to the more than one sub-loss function, respectively.
[0108] According to an embodiment of the present disclosure, the updating unit is further configured to update the parameters of the neural network model using a back propagation method and the determined update gradient. After the neural network model is updated, the updating unit will operate using the updated neural network model.
[0109] According to an embodiment of the present disclosure, when the loss data determined after updating the neural network model is greater than a threshold value and the number of iterative operations performed by the loss determination unit and the update unit has not reached a predetermined number of iterations, the update unit will perform the next iterative update operation until the determined loss function is less than or equal to the threshold value, or the number of iterative operations has reached a predetermined number of iterations. As an example, the update unit may include a determination unit to determine whether the loss data is greater than a threshold value, and / or determine whether the number of iterative operations has reached a predetermined number of times, and a processing unit to perform an update operation according to the determination result.
[0110] An exemplary implementation in which the loss function according to the present disclosure includes both an intra-class loss function and an inter-class loss function and their corresponding weight functions will be described below.
[0111] In order to constrain the degree of optimization, the loss function of the present disclosure is conceived to provide an effective mechanism to limit the gradient amplitude, which can appropriately constrain the reduction of the intra-class angle and the increase of the inter-class angle during the training process. i The new Sigmoid-constrained loss function (SFace) on the hypersphere manifold consists of intra-class loss and inter-class loss, namely:
[0112] In particular, the intra-class loss and the inter-class loss L inter (θ j ) is defined as:
[0113]
[0114]
[0115] in, For x i / ‖x i ‖and The intra-class angular distance between j (j≠y i ) is x i / ‖x i ‖ and W j / ||W j || between classes. Among them, cos(θ j )=W j T x i / ||W j ||‖x i ‖,j≠y i , […] bIndicates the gradient scalar calculated by the weight function, which weights the intra-class cosine angular distance loss and the inter-class cosine angular distance loss during training, and its constant value is calculated at each iteration for weighting.
[0116] During the forward propagation process, the current loss is calculated based on the new loss function SFace, and the formula is as follows:
[0117]
[0118] In the back propagation process used for updating, according to the principle of the back propagation algorithm, the partial derivative function used for parameter updating will also be weighted by the block gradient operator. The specific formula is as follows:
[0119]
[0120]
[0121]
[0122] in,
[0123]
[0124]
[0125]
[0126]
[0127] Among them, the above equations (5)-(7) may correspond to the update function according to the present disclosure.
[0128] The above equations (8)-(11) can be directly obtained through mathematical derivation operations. The derivation of equation (8) is described in detail below, and it should be understood that other equations can be obtained through similar mathematical derivation calculations.
[0129] for in 1≤k≤d, the derivation is as follows:
[0130] first:
[0131]
[0132] Its partial derivative is:
[0133]
[0134] Therefore, it can be concluded that:
[0135]
[0136] Because Figure 5C As shown, Therefore, the optimized gradient direction is always along the tangent direction of the hypersphere. Since the gradient has no component in the radial direction, ‖x i ‖, and ||W j || remains almost unchanged during training, so we further design and [r inter (θ j )] b for and θ j The scalar functions of rescale the optimization objectives respectively.
[0137] like Figure 3A As shown in Figure 2, there are actually two factors that re-adjust the gradient, namely, the speed of moving the training sample and the target center. Therefore, we can set and [r inter (θ j )] b is the gradient rescaling function, which is determined based on the weight function. Since the gradient magnitudes of the original intra-class loss and inter-class loss are respectively and sinθ j Proportional to, the final gradient amplitude is respectively and v inter (θ j )=r inter (θ j )sinθ j Proportional.
[0138] It is well known that at the beginning of training, the initial intra-class and inter-class angular distances and θ j Approximately As training progresses, the intra-class loss function gradually decreases the intra-class angle At the same time, the inter-class loss function prevents the inter-class angle θ j Therefore, according to the function of the gradient amplitude control disclosed in the present invention, and ν inter (θ j ) can satisfy the following properties:
[0139] (1) Function Should be in the range is a non-negative single increasing function, ensuring that x i and The moving speed gradually decreases during the process of approaching each other.
[0140] (2) Function ν inter (θj ) should be in the interval is a non-negative single decreasing function, ensuring that x i and W j If they are close to each other, the weight will increase rapidly.
[0141] (3) Considering the presence of noise in the training data, the function A flexible cutoff point should be designed near the intra-class angle of 0 to limit the convergence rate of the intra-class loss; function ν inter (θ j ) should be in the inter-class perspective A flexible cutoff point is designed near to control the convergence speed of inter-class loss. In this way, the optimization objectives within and between classes are moderately adjusted instead of strictly maximizing or minimizing the objectives.
[0142] In order to flexibly control the movement speed of the gradient to adapt to noisy training data, a Sigmoid-based weight function is proposed. and r inter (θ j ), the specific formula is as follows:
[0143]
[0144]
[0145] Among them, S is the initial scale parameter that controls the gradient of the two Sigmoid curves; a and b are the parameters that control The slope and horizontal intercept parameters of the Sigmoid curve, c and d are the parameters for controlling [ν inter (θ j )] b The parameters of the slope and horizontal intercept of the Sigmoid curve actually control the flexible interval to suppress the movement speed of the gradient. The weight function equation of the Sigmoid curve and r inter (θ j ) changes with its parameters as Figure 5D As shown. Furthermore, the theoretical amplitudes of the intra-class gradient rescaling function and the inter-class gradient rescaling function are and v inter (θ j )=r inter (θ j )sinθ j , and through such a function, the appropriate adjustment curves of intra-class gradient and inter-class gradient can be obtained, such as Figure 5E As shown, Figure 5E The final adjustment curves of intra-class gradient and inter-class gradient according to the method of the present disclosure are shown.
[0146] The intra-class loss and the out-of-class loss are controlled using the weight function according to the present disclosure, thereby controlling the gradient convergence speed to be suitable for different training sets. Preferably, the weight function of the intra-class angle should decrease smoothly and monotonically as the intra-class angle decreases, and the gradient amplitude can be made more suitable for the training data set by adjusting the hyperparameters of the weight function. The weight function of the inter-class angle should decrease smoothly and monotonically as the inter-class angle increases, and the gradient amplitude can be made more suitable for the training data set by adjusting the hyperparameters of the weight function.
[0147] The following describes the difference between the model training method according to the present disclosure and the existing softmax-based training method.
[0148] In addition to the original Softmax function mentioned above, in order to further improve the accuracy, the existing technology also introduces the idea of large boundaries to In this way, the softmax-based loss function can be defined as:
[0149]
[0150] Among them, in the NSoftmax method In the CosFace method In the ArcFace method Theoretically, it can be seen that Will decrease with the loss function, θ j It will increase with the optimization of the loss function. During the back propagation process, its partial derivative formula is as follows:
[0151]
[0152]
[0153] Among them, in the NSoftmax method and the CosFace method In the ArcFace method
[0154] It should be noted that the above partial derivative formulas (18) and (19) are derived only for comparison with the technical solution of the present disclosure, and the derivation process is as follows. However, in the current prior art implementation, this formula transformation is not necessarily performed.
[0155] The derivation of formula (18) is as follows:
[0156] First, according to the chain rule, we can get:
[0157]
[0158] in:
[0159]
[0160]
[0161] So we can get:
[0162]
[0163] The derivation of formula (19) is as follows:
[0164]
[0165] Furthermore, the softmax-based loss function is equivalent to the following formula:
[0166]
[0167] in
[0168] It should be pointed out that the above formula (25) is derived only for comparison with the technical solution of the present disclosure, but in the current prior art implementation, the loss function is not necessarily derived as formula (25).
[0169] Moreover, because the parameters of the deep neural network are only updated during the back propagation process during the entire training process, and the back propagation functions of formula (17) and formula (25) are the same, that is,
[0170]
[0171]
[0172] Therefore, in the model training stage, formula (17) and formula (25) are equivalent.
[0173] As can be seen from the loss function rewritten above, the Softmax-based loss function can be considered as a metric learning method with a specific optimization speed limit on the sphere. According to the experimental analysis of existing methods, most of the θ j During the actual training process, There are slight changes around this, so we assume This leads to the following conclusions:
[0174]
[0175]
[0176] In order to make a more intuitive comparison, we adjust the intra-class gradient function of NSoftmax, CosFace and ArcFace methods and the inter-class gradient adjustment function v inter (θ j )=r inter (θ j )sinθ j The corresponding curve is shown in Fig. 5F Where (1) is the intra-class gradient of the NSoftmax method and the inter-class gradient v inter (θ j ), (2) is the intra-class gradient of the CosFace method and the inter-class gradient v inter (θ j ), (3) is the intra-class gradient of the ArcFace method and the inter-class gradient v inter (θ j ) curve. In the inter-class gradient adjustment function curve is set to But in practice, this assumption does not always hold true, because θ j Actually Fluctuations nearby and But it is gradually decreasing.
[0177] Depend on Figure 5E and 5F It is clear from the comparison between the two that the softmax-based loss function cannot actually accurately control the optimization process within and between classes. In particular, the gradient curves corresponding to the loss functions in the prior art are basically a set of curves of the same shape, that is, the gradient amplitude changes follow a similar regular change, so that the gradient amplitude changes are basically fixed during the model training / optimization process, and overfitting cannot be effectively avoided. However, the loss function according to the present invention can accurately control the optimization process, in particular, by using parameters to accurately determine the gradient amplitude change law of the gradient curve to adapt to different training data sets, effectively reducing or even avoiding overfitting.
[0178] According to the present disclosure, the device may further include an image feature acquisition unit configured to acquire image features from a training image set using the neural network model. The acquisition of image features may be performed in a manner known in the art and will not be described in detail here. Of course, the image feature unit may be located outside the device according to the present disclosure.
[0179] It should be pointed out that Figure 4AThis is only a schematic structural configuration of the training device, and the training device may also include other possible units / components (e.g., memory, etc.). The memory may store various information generated by the training device (e.g., training set image features, loss data, function parameter values, etc.), programs and data for the operation of the training device, etc. For example, the memory may include, but is not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. As an example, the memory may also be located outside the training device. The training device may be directly or indirectly (e.g., other components may be connected in between) connected to the memory to access data. The memory may be a volatile memory and / or a non-volatile memory.
[0180] It should be noted that the above-mentioned units are only logical modules divided according to the specific functions implemented by them, and are not used to limit the specific implementation methods. For example, they can be implemented in software, hardware, or a combination of software and hardware. In actual implementation, the above-mentioned units can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). In addition, the above-mentioned units are shown with dotted lines in the accompanying drawings to indicate that these units may not actually exist, and the operations / functions implemented by them can be implemented by the processing circuit itself.
[0181] It should be noted that, in addition to including a plurality of units, the above-mentioned training device can be implemented in a variety of other forms, such as a general-purpose processor or a dedicated processing circuit, such as an ASIC. For example, the training device can be constructed by a circuit (hardware) or a central processing device (such as a central processing unit (CPU)). In addition, the training device can carry a program (software) for making the circuit (hardware) or the central processing device work. The program can be stored in a memory (such as arranged in the memory) or in an external storage medium connected from the outside, and downloaded via a network (such as the Internet).
[0182] According to an embodiment of the present disclosure, a training method for a neural network model for object recognition is proposed, such as Figure 4B As shown, the method 500 includes a loss determination step 502 for determining loss data for features extracted from a training image set using the neural network model, and an update step 504 for performing an update operation on parameters of the neural network model based on the loss data and an update function, wherein the update function is obtained based on the loss function and a corresponding weight function of the neural network model, and the weight function and the loss function change monotonically in the same direction in a specific value range.
[0183] According to the present disclosure, the method may further include an image feature acquisition step for acquiring image features from a training image set using the neural network model. The acquisition of image features may be performed in a manner known in the art and will not be described in detail herein. Of course, the image feature acquisition step may not be included in the method according to the present disclosure.
[0184] It should be noted that the method according to the present disclosure may also include the various operations described above, which will not be described in detail here. It should be noted that the various steps / operations of the method according to the present disclosure may be performed by the above-mentioned units, or by various forms of processing circuits.
[0185] The following will refer to Figure 6 To describe the model training operation according to the present disclosure. Figure 6 The basic flow of the model training operation according to the present disclosure is shown.
[0186] First, a training data set is input, which may include a large number of object images, such as face images, such as tens of thousands, hundreds of thousands, or millions of object images.
[0187] Then, the images in the input training data set may be preprocessed, and the preprocessing operations may include, for example, object detection, object alignment, etc. Taking face recognition as an example, the preprocessing may include face detection, such as detecting a face from an image containing a face and obtaining an image mainly containing the face to be recognized, and face alignment belongs to the normalization operation for face recognition. The main purpose of face alignment is to eliminate undesirable intra-class variations by aligning the image to some standardized shapes or structures. It should be noted that the preprocessing operation may also include other types of preprocessing operations known in the art, which will not be described in detail here.
[0188] Then, the preprocessed training set images are input into the convolutional neural network model for feature extraction. The convolutional neural network model can adopt various structures known in the art, which will not be described in detail here.
[0189] Then, the loss is calculated by a loss function. The loss function may be a function known in the art, or a loss function based on a weight function proposed in the present disclosure.
[0190] Then, the parameters of the convolutional neural network are updated by back propagation according to the calculated loss data. It should be noted that the update function defined in the present disclosure is used in the back propagation to update the parameters of the convolutional neural network model. The update function is defined as described above and will not be described in detail here.
[0191] Existing unconstrained learning methods strictly drag noise samples to the wrong identification, thereby causing the noise samples to overfit. The model training according to the present disclosure alleviates this problem to a certain extent because it optimizes the noise samples in a gentle way.
[0192] According to an embodiment of the present disclosure, by using an improved weight function for dynamically controlling the model update / optimization amplitude during the training process, i.e., the gradient descent speed, the training of models for object recognition, such as convolutional neural network models, can be further optimized compared to the prior art, thereby obtaining a more optimized object recognition model, thereby further improving the accuracy of object recognition / authentication.
[0193] In addition, in the embodiments of the present disclosure, the intra-class angle and the inter-class angle are directly optimized as the loss function objectives, rather than using the cross entropy loss as the loss function, to ensure consistency with the prediction process objectives, thereby simplifying the intermediate processes in the training process, reducing computational overhead and improving optimization accuracy.
[0194] Moreover, in the embodiments of the present disclosure, the loss function considers the intra-class loss and the inter-class loss respectively, so that the intra-class gradient term and the inter-class gradient term are decoupled, which helps to analyze the loss of the intra-class gradient term and the inter-class gradient term respectively and guide the optimization of the intra-class gradient term and the inter-class gradient term. In particular, appropriate weight functions are used for the intra-class loss and the inter-class loss respectively to control the convergence speed of the gradient, prevent overfitting of noisy training samples, and thus obtain an optimized training model even for a noisy training set.
[0195] The following will compare the effects of the model training method disclosed in the present invention with that in the prior art through experiments.
[0196] Experiment 1: Validation on a small training set
[0197] Training set: CASIA-WebFace, including 10,000 person identities, a total of 500,000 images.
[0198] Test set: YTF, LFW, CFP-FP, AGEDB-30, CPLFW, CALFW
[0199] Evaluation criteria: 1: N TPIR (True Positive Recognition Rate, Rank1@10 6 ), same as Megafacechallenge
[0200] Convolutional Neural Network Architecture: RestNet50
[0201] Comparison of existing technologies: Softmax, NSoftmax, SphereFace, CosFace, ArcFace, D-Softmax
[0202] The experimental results are shown in Table 1 below, where SFace is based on the technical solution disclosed in the present invention.
[0203] Table 1: Comparison of the training results of the present invention and the prior art
[0204]
[0205] Experiment 2: Validation on a large-scale training set
[0206] Training set: MS1MV2, including 85,000 person identities, a total of 5,800,000 images.
[0207] Evaluation set: LFW, YTF, CPLFW, CALFW, IJB-C
[0208] Evaluation criteria: 1: N TPIR (True Positive Identification Rate, Rank1@10 6 ) and TPR / FPR
[0209] Convolutional Neural Network Architecture: RestNet100
[0210] Comparison with existing technologies: ArcFace
[0211] The experimental results are shown in Tables 2 and 3 below, where SFace is based on the technical solution disclosed in the present invention.
[0212] Table 2: Comparison of the training results of the present invention and the prior art
[0213]
[0214] Table 3: Comparison of the training results of the present invention and the prior art
[0215]
[0216] Experimental results show that the model training scheme according to the present disclosure has better performance than the prior art.
[0217] The following will describe in detail an exemplary implementation according to the present disclosure with reference to the accompanying drawings. It should be noted that the following description is mainly intended to clearly explain the training operation process according to the present disclosure, and some steps or operations therein are not necessary, for example, the pre-processing step and the feature extraction step are not necessary, and the operation according to the present disclosure can be performed directly based on the received features.
[0218] Figure 7The process of training a convolutional neural network model using the joint loss function proposed in the present disclosure according to the first embodiment of the present disclosure is shown. The model training process according to the first embodiment of the present disclosure includes the following steps.
[0219] S7100: Obtaining network training data through pre-processing
[0220] In this step, the input is the original image with real object or face labels, and then the input original image is converted into training data that meets the requirements of the convolutional neural network through a series of existing pre-processing operations. This series of pre-processing operations includes face or object detection, face or object alignment, image augmentation, image normalization, etc.
[0221] S7200: Extract features using current convolutional neural networks
[0222] In this step, the input is image data with objects or faces that meet the requirements of the convolutional neural network, and then the image features are extracted using the selected convolutional neural network structure and the current corresponding parameters. The structure of the convolutional neural network can be a commonly used network structure, such as VGG16, ResNet, SENet, etc.
[0223] S7300: Calculate the current joint loss based on the proposed weighted intra-class loss function and weighted inter-class loss function
[0224] In this step, the input is the extracted image features and the last fully connected layer of the convolutional neural network, and then the current intra-class loss and inter-class loss are calculated respectively according to the proposed joint weighted loss function. For the specific definition of the loss function, please refer to the introduction of equations (2) to (4) above.
[0225] S7400: Determine whether to end the training process
[0226] In this step, it can be determined whether to end the training by certain pre-set conditions. The pre-set conditions may include a loss threshold condition, an iteration number condition, a gradient descent speed condition, etc. If at least one condition is met, the training can be ended and the process goes to S7600. If none of the pre-set conditions are met, the process goes to S7500.
[0227] As an example, judgment can be made by setting a threshold. In this case, the input is the loss data calculated in the previous step, including intra-class loss data and inter-class loss data. Judgment can be made by comparing the loss data with the set threshold, such as whether the current loss is greater than a given threshold. If the current loss is less than or equal to the given threshold, the training ends.
[0228] According to one implementation, the set threshold may be a threshold set for the intra-class loss data and the inter-class loss data, respectively, and the training is terminated as long as any one of the intra-class loss data and the inter-class loss data is less than or equal to the corresponding threshold. According to another implementation, the set threshold may be an overall loss threshold, the overall loss value of the intra-class loss data and the inter-class loss data is compared with the overall loss threshold, and the training is terminated when it is less than the overall loss threshold. The overall loss value may be various combinations of the intra-class loss data and the inter-class loss data, such as summation, weighted summation, etc.
[0229] As another example, the judgment can be made by setting a predetermined number of training iterations, such as whether the current number of training iterations reaches the predetermined number of training iterations. In this case, the input is a count of the training iteration operations that have been performed, and the training is terminated when the number of training iterations has reached the predetermined number of training iterations.
[0230] Otherwise, when none of the above predetermined conditions are met, the next training iteration process is continued. For example, if the loss data is greater than a predetermined threshold and the number of iterations is less than a predetermined number of training iterations, the next training iteration process is continued.
[0231] S7500: Update convolutional neural network parameters
[0232] In this step, the input is the joint loss calculated in S7300, and the parameters of the convolutional neural network model are updated using the weight function according to the present disclosure.
[0233] Specifically, according to the weight function-based update function disclosed in the present invention, such as the partial derivative function (Formula (5) to Formula (11)) mentioned above, the gradient of the current loss to the output layer of the convolutional neural network is calculated, and then the back propagation algorithm is used to update the convolutional neural network parameters of S7200, and the updated neural network is transmitted back to S7200.
[0234] S7600: Output the trained CNN model
[0235] In this step, we use the current parameters of all layers in the CNN model structure as the trained model, thereby obtaining an optimized neural network model.
[0236] In this embodiment, the proposed intra-class loss function and inter-class loss function are used to collaboratively control the gradient descent speed, which can find a good balance between the contributions of intra-class loss and inter-class loss, thereby training a model with better generalization.
[0237] Figure 8The process of training a convolutional neural network model using the joint loss function proposed in the present disclosure according to the second embodiment of the present disclosure is shown. In this embodiment, the joint loss function proposed in the present disclosure is used to train a piecewise convolutional neural network model. The model training process according to the second embodiment of the present disclosure includes the following steps.
[0238] S8100: Obtaining network training data through pre-processing
[0239] The operation of this step is the same as or similar to that of S7100 and will not be described in detail here.
[0240] S8200: Extract features using current convolutional neural networks
[0241] The operation of this step is the same as or similar to that of S7200 and will not be described in detail here.
[0242] S8300: Calculate the intra-class loss as the current loss according to the proposed weighted intra-class loss function.
[0243] The input is the extracted image features and the last fully connected layer of the convolutional neural network, and then the current intra-class loss is calculated according to the weighted intra-class loss function disclosed in the present invention. The weighted intra-class loss function can be defined as formula (2) above, which will not be described in detail here.
[0244] S8400: Determine whether it is the early training process
[0245] In this step, whether it is preliminary training can be determined by certain pre-set conditions, and the pre-set conditions may include loss threshold conditions, iteration number conditions, gradient descent speed conditions, etc. If at least one of the above conditions is met, it can be determined that the preliminary training can be ended, and the process goes to S8600. If none of the pre-set conditions is met, it is determined that the preliminary training needs to be continued, and the process goes to S8500.
[0246] As an example, if the current gradient descent rate is less than or equal to a given threshold, the current intra-class loss is less than a given threshold, or the current number of training iterations has reached any one of the given numbers of previous training iterations, it can be considered that the previous training can be ended and transferred to S8600 for the later training operation, in which the weighted inter-class loss function will be used for training.
[0247] As an example, when the above conditions may not be met, that is, the gradient descent rate is greater than a given threshold, the current intra-class loss is greater than a given threshold, and the current number of training iterations has not yet reached a given number of previous training iterations, it is considered that the previous training needs to continue, and it will go to S8500.
[0248] S8500: Update the parameters of the convolutional neural network using the back propagation algorithm based on the calculated intra-class loss and intra-class weight function
[0249] In this step, the input is the intra-class loss calculated in S8300. It is necessary to first calculate the gradient of the current intra-class loss to the output layer of the convolutional neural network based on the re-derived partial derivative function, and then use the back-propagation algorithm to update the parameters of the convolutional neural network model, and then send the updated neural network model parameters back to S8200. The derived partial derivative formula is as follows:
[0250]
[0251]
[0252]
[0253]
[0254] S8600: After the parameters of the training model have been optimized for the intra-class loss, the parameters of the training model are optimized for the inter-class loss.
[0255] As an example, the proposed weighted intra-class loss function and weighted inter-class loss function can be used to calculate the current joint loss. Specifically, the input is the extracted image features and the last fully connected layer of the convolutional neural network, and then the current intra-class loss and inter-class loss are calculated respectively according to the proposed joint weighted loss function to obtain the joint loss. For the specific loss function definition, please refer to the introduction of equations (2) to (4) above.
[0256] Alternatively, as an example, the weighted inter-class loss function proposed in the present disclosure may be used to calculate the inter-class loss as described above, and then the sum of the calculated inter-class loss and the intra-class loss at the end of the aforementioned previous training is taken as the current joint loss.
[0257] S8700: Determine whether to end the training process
[0258] In this step, it can be determined whether to end the training by certain pre-set conditions. The pre-set conditions may include a loss threshold condition, an iteration number condition, a gradient descent speed condition, etc. If at least one condition is met, the training can be ended and the process goes to S8900. If none of the pre-set conditions are met, the process goes to S8800.
[0259] The specific operations of this step may be the same as or similar to the operations of the aforementioned S7400, and will not be described in detail here.
[0260] S8800: Update convolutional neural network parameters
[0261] In this step, the input is the joint loss calculated in S8600. According to the partial derivative function derived in the previous article (Formula (5) to Formula (11)), the gradient of the current inter-class loss to the output layer of the convolutional neural network is first calculated, and then the back propagation algorithm is used to update the convolutional neural network model parameters. The updated convolutional neural network model parameters are then sent back to S8200 for the next iterative training.
[0262] As an example, preferably, in this case, steps S8300-S8500 in the next iteration process can be directly omitted, and the step S8200 can be directly transferred to step S8600, so that the training process can be simplified. For example, an indicator can be added to the data transmission after the end of the preliminary training to indicate the end of the preliminary training, and then if such an indicator is recognized during the iterative training process, it means that the preliminary training process can be skipped.
[0263] As an example, step S8200 may further include an indicator detection step, which is used to detect whether there is an indicator indicating the end of the previous training. As an example, after determining that the previous training is over in step S8400, an indicator indicating the end of the previous training may be fed back to step S8200, so that in the feedback update operation of the later training, the previous training process is directly skipped when the indicator is detected. As another example, after determining that the previous training is over in step S8400, an indicator indicating the end of the previous training is added to the data stream when switching to the later training, and the indicator is fed back to step S8200 in the feedback update operation of the later training, and step S8200 may skip the previous training process when the indicator is detected.
[0264] S8900: Output the trained CNN model.
[0265] This step is the same or similar to the operation of the aforementioned step S7600 and will not be described in detail here.
[0266] Compared with Example 1, Example 2 simplifies the parameter adjustment process of the weight function and accelerates the training of the model. The optimal intra-class loss weight function for the current data set is first found, and the intra-class loss is used to constrain the model training process, and the joint loss is quickly iterated to a certain extent; then the optimal inter-class weight function for the current data set and the joint loss is found, and the model training process is finely constrained using the intra-class loss and the inter-class loss at the same time, so as to obtain the final training model faster.
[0267] It should be pointed out that the process shown in the above flowchart mainly corresponds to the execution when the parameters of the intra-class weight function and the inter-class weight function remain unchanged. As mentioned above, the parameters of the intra-class weight function and the inter-class weight function can be further adjusted to further optimize the design of the weight function.
[0268] A third embodiment according to the present disclosure will be described below with reference to the drawings. Fig. 9 The diagram shows a process of adjusting the weight function parameters according to the third embodiment of the present disclosure.
[0269] S9100: Obtaining network training data through pre-processing
[0270] The operation of this step is the same as or similar to that of S7100 and will not be described in detail here.
[0271] S9200: Training a convolutional neural network model
[0272] This step may adopt the operation according to any one of the first and second embodiments to perform convolutional neural network model training to obtain an optimized convolutional neural network model according to the present disclosure.
[0273] S9300: Determine whether to adjust weight function parameters
[0274] In this step, it can be determined whether parameter adjustment is required by certain pre-set conditions, and the pre-set conditions may include adjustment times conditions, convolutional neural network performance conditions, etc. If at least one of the above conditions is met, it can be determined that the adjustment operation can be ended, and the process goes to S9500. If none of the pre-set conditions is met, it is determined that the adjustment operation needs to be continued, and the process goes to S9400.
[0275] As an example, when the number of parameter adjustments performed has reached a predetermined number of adjustments, or the performance of the current convolution application network model is inferior to the performance of the previous convolution neural network model, it is considered that no further parameter adjustment operation is required, that is, the adjustment operation is terminated. Otherwise, if the number of parameter adjustments has not reached the predetermined number of adjustments and the performance of the current convolution application network model is better than the performance of the previous convolution neural network model, it is determined that further adjustment is required.
[0276] S9400: Set new weight function parameters
[0277] In this step, you can try to continue adjusting the parameters in a specific parameter adjustment manner until a predetermined number of adjustments is reached or the training result is no longer optimized.
[0278] As an example, a specific parameter adjustment method can be to adjust the parameters according to a certain rule, such as increasing or decreasing the parameters with a specific step size or following a specific function. As another example, the parameter adjustment can be performed following a previous adjustment method. As an example, the parameter adjustment of the weight function can be performed as described above.
[0279] S9500: Output the parameters of the adjusted weight function
[0280] In this step, we output the parameters of the adjusted weight function, thereby obtaining a more optimized weight function, thereby improving the performance of subsequent convolutional neural network model training.
[0281] A fourth embodiment according to the present disclosure will be described below with reference to the accompanying drawings. Fig.10 The diagram shows an adjustment process of weight function parameters according to the fourth embodiment of the present disclosure.
[0282] S10100: Obtaining network training data through pre-processing
[0283] The operation of this step is the same as or similar to that of S7100 and will not be described in detail here.
[0284] S10200: Training a convolutional neural network model
[0285] This step may adopt the operation according to any one of the first and second embodiments to perform convolutional neural network model training to obtain an optimized convolutional neural network model according to the present disclosure.
[0286] It should be noted that two initial values are set for the parameters of the weight function, so that in this step, the convolutional neural network models determined are two convolutional neural network models corresponding one-to-one to the two parameter values.
[0287] S10300: Compare the performance of the two convolutional neural network models to select a convolutional neural network model with better performance.
[0288] S10400: Determine whether to adjust weight function parameters
[0289] In this step, for the convolutional neural network model with better performance selected in S10300, it can be determined whether parameter adjustment is required by certain pre-set conditions, and the pre-set conditions may include adjustment times conditions, convolutional neural network performance conditions, etc. If at least one of the above conditions is met, it can be determined that the adjustment operation can be ended, and go to S10500. If none of the pre-set conditions are met, it is determined that the adjustment operation needs to be continued, and go to S10600. The operation of this step is the same or similar to the previous step S9300, and will not be described in detail here.
[0290] S10500: Set new weight function parameters
[0291] In this step, the parameters can be adjusted in a specific parameter adjustment manner until a predetermined number of adjustments is reached or the training result is no longer optimized. The operation of this step is the same or similar to the previous step S9400 and will not be described in detail here.
[0292] S10600: Output the adjusted parameters of the weight function
[0293] In this step, we output the parameters of the adjusted weight function, thereby obtaining a more optimized weight function, thereby improving the performance of subsequent convolutional neural network model training.
[0294] It should be noted that in the above embodiments, the adjustment of one parameter is mainly introduced, while the adjustment of two or more possible parameters in the weight function may be performed in various ways, such as the various implementation ways described above.
[0295] According to the implementation of the present disclosure, when the loss function includes both the intra-class loss function and the inter-class loss function, it is necessary to adjust the parameters of the weight function for the intra-class loss and the weight function for the inter-class loss. As an implementation, the parameters of the weight function for the intra-class loss can be adjusted first, and then the parameters of the function for the inter-class loss can be adjusted. As another implementation, the parameters of the weight function for the intra-class loss and the weight function for the inter-class loss can also be adjusted at the same time. The specific adjustment process of the parameters of each function can be implemented in a variety of ways.
[0296] As an example, after setting the initial parameters of the intra-class and inter-class weight functions, the aforementioned convolutional neural network model training is then performed. After a predetermined number of iterations are performed or the loss meets the threshold requirement, the parameters of the intra-class weight function are further adjusted until parameters that do not lead to further optimization of the loss data determined by the convolutional neural network model are found. It should be noted that in this case, the inter-class weight function can always maintain the initial parameters. Then, based on the optimized intra-class weight function, the inter-class weight parameters are adjusted with operations that are basically similar to the above-mentioned preliminary training until parameters that do not lead to further optimization of the loss data determined by the convolutional neural network model are found. In this way, the optimal intra-class weight function and inter-class weight function can be finally determined, and the optimal convolutional neural network model can also be determined.
[0297] As another example, after a round of iterative training process is completed, it can be determined whether parameter adjustment is required, and if optimization is required, the value of the parameter to be adjusted is set, such as the value of both the in-class weight function and the out-of-class weight function. Based on this, a new round of iterative training is performed until the parameter adjustment is completed.
[0298] It should be noted that the training of the convolutional neural network model described in the above embodiment belongs to offline training, that is, the model training is carried out using the selected training data set / image set, and the trained model is directly used for face / object recognition or verification. According to the present disclosure, the convolutional neural network model can also be trained online. The online training process refers to the process of adding at least some of the recognized images to the training image set during the process of using the trained model for face recognition / verification, so that the model can be trained and updated and optimized during the recognition process, so that the obtained model is further improved and can be more suitable for the image set to be recognized, thereby achieving a good recognition effect.
[0299] The fifth embodiment according to the present disclosure will be described in detail below, which relates to online training of a convolutional neural network model. Fig.11 The flowchart of updating the trained convolutional neural network model by online learning using the loss function proposed by us in the application system according to the fifth embodiment of the present disclosure is shown.
[0300] S11100: Pre-processing the output face / object image to be identified / authenticated
[0301] In this step, the input is the original image with real object or face labels, and then the input original image is converted into training data that meets the requirements of the convolutional neural network through a series of existing pre-processing operations. This series of pre-processing includes face or object detection, face or object alignment, image augmentation, image normalization, etc., to meet the requirements of the convolutional neural network model.
[0302] S11200: Extract features using current convolutional neural networks
[0303] This step is basically the same as the feature extraction operation in the aforementioned embodiment, and will not be described in detail here.
[0304] S11300: Face / object recognition or verification based on extracted features
[0305] In this step, the face / object is identified or verified based on the extracted image features. The operation here can be performed in a variety of ways known in the art and will not be described in detail here.
[0306] S11400: Calculate the angle between the two based on the extracted features and the final fully connected layer of the convolution application network model
[0307] In this step, the input is the extracted image features and the weight matrix of the last fully connected layer of the convolutional neural network, and then the angle of the currently extracted image features and each dimension of the weight matrix is calculated according to the defined angle calculation formula. The specific angle calculation formula is defined as follows:
[0308]
[0309] Among them, x is the extracted image feature, W = {W 1 ,W 2 ,…,W C}∈ d×C is the weight matrix of the current fully connected layer, W j is the jth weight vector, representing the target feature center of the jth object of the currently trained CNN model.
[0310] S11500: Determine whether it is a suitable training sample
[0311] In this step, the input is the angle information calculated in the previous step, and some pre-set judgment conditions can be used to determine whether the input image is a suitable training sample. A suitable training sample means that the input image can be judged not to belong to any object in the original training set according to the calculated angle, or belongs to an object in the original training set, but the feature of the input image is at a certain distance from the feature center of the object, indicating that the image is a sample that is difficult to distinguish for the object, that is, a suitable training sample.
[0312] The pre-set condition may refer to whether the angular distance between the feature of the input image and the feature center of the specific object is greater than or equal to a specific threshold. If it is greater than or equal to the specific threshold, the training sample may be considered to be a suitable training sample.
[0313] As an example, if the input image sample is identified as not belonging to any category in the convolutional neural network model, the image sample may belong to a new object category and must be suitable as a training sample. As an example, if the input image sample is identified as belonging to a category of the convolutional neural network model, but the angular distance (angle value) between the feature of the image sample and the feature center of the category is greater than a predetermined threshold, the input image sample is considered suitable as a training sample.
[0314] It should be noted that the above steps S11300 and S11400 can be combined together. Specifically, when performing face / object recognition, if the recognized object does not belong to any object in the original training set, the angle distance calculation is no longer performed, and when the recognized object belongs to an object in the original training set, only the angle distance between the recognized object and the feature center of the object is calculated. This can appropriately simplify the calculation process and reduce the calculation overhead.
[0315] Fig.12 Schematic diagram showing that the input image is a suitable training sample for an object in the training dataset. Fig.12 As shown, x i is the extracted image feature, Wj is the target feature center of the jth object of the current CNN model. If the condition is met, the input image is a suitable training sample for an object, such as Fig.12 x in 1 is a training sample for object 1; otherwise, it is not a suitable training sample, such as Fig.12 x in 2 It is judged that it is not a training sample of object 1.
[0316] Fig.13 A schematic diagram showing that the input image is a suitable training sample for a new object.
[0317] In this step, if it is determined that the image sample is a suitable training sample, go to step S11600, otherwise end directly.
[0318] S11600: Use the newly determined suitable training samples to train the object recognition model.
[0319] In particular, the newly determined suitable training samples can be used as new training sets, and then the model training operation according to the present disclosure can be used to perform model training. As an example, the model training operation according to the first and second embodiments of the present disclosure can be used to perform model training based on the new training set, and the parameters of the weight function used to train the model can also be adjusted according to the third and fourth embodiments of the present disclosure.
[0320] According to one implementation, the training performed in this step may be performed only for the determined suitable training samples. According to another implementation, the training performed in this step may be performed for a combined training set of the determined suitable training samples and the original training set.
[0321] According to one implementation, the training performed in this step can be performed in real time, that is, whenever a new suitable training sample is determined, the model training operation is performed. According to another implementation, the training performed in this step can be performed periodically, for example, after a certain number of new suitable training samples have been accumulated, the model training operation is performed.
[0322] As an example, when the appropriate training sample is a suitable training sample of an object in the original training data set, the model training is performed by the following operations. Specifically, based on the characteristics of the training sample, the current joint loss is calculated according to the weighted intra-class loss function and the weighted inter-class loss function disclosed in the present invention, and the parameters of the convolutional neural network are updated using the back propagation algorithm according to the calculated joint loss and the intra-class and inter-class weight functions. The updated neural network is transmitted back to S11200 for the next recognition / verification process.
[0323] As another example, when the appropriate training sample is a new training sample that does not belong to any object in the original training data set, the model training can be performed by the following operations. Specifically, first, the weight matrix of the last fully connected layer of the CNN is adjusted according to the extracted features. In this step, the input is the features determined to be new objects and the weight matrix W of the current fully connected layer = {W 1 ,W 2 ,…,W C}∈ d×C , we need to expand the weight matrix to W′={W 1 ,W 2 ,…,W C ,W C+1}∈ d×(C+1) , so that W C+1 It can represent the target feature center of the new object. The simplest adjustment method is to directly use the features of the new object as the target feature center of the new object. A more reasonable adjustment method is to find a vector W that is approximately orthogonal to the original weight matrix near the features of the new object. C+1 , added to the original weight matrix as the feature center of the new object. Then, based on the features of the training sample, the current joint loss is calculated according to the weighted intra-class loss function and the weighted inter-class loss function disclosed in the present invention, and the parameters of the convolutional neural network are updated using the back propagation algorithm according to the calculated joint loss and the intra-class and inter-class weight functions. The updated neural network is transmitted back to S11200 for the next recognition / verification process.
[0324] The fifth embodiment can continuously optimize the model by online learning in the actual application process, so that the model has better adaptability to the real application scenario. The recognition ability of the model can be expanded by online learning in the actual application process, so that the model has better flexibility for the real application scenario.
[0325] Fig.14 is a block diagram illustrating an exemplary hardware configuration of a computer system 1000 in which embodiments of the present invention may be implemented.
[0326] like Fig.14 As shown, the computer system includes a computer 1110. The computer 1110 includes a processing unit 1120, a system memory 1130, a non-removable non-volatile memory interface 1140, a removable non-volatile memory interface 1150, a user input interface 1160, a network interface 1170, a video interface 1190, and an output peripheral interface 1195, which are connected via a system bus 1121.
[0327] The system memory 1130 includes a ROM (Read Only Memory) 1131 and a RAM (Random Access Memory) 1132. A BIOS (Basic Input Output System) 1133 resides in the ROM 1131. An operating system 1134, application programs 1135, other program modules 1136, and some program data 1137 reside in the RAM 1132.
[0328] A non-removable non-volatile memory 1141 (such as a hard disk) is connected to the non-removable non-volatile memory interface 1140. The non-removable non-volatile memory 1141 can store, for example, an operating system 1144, application programs 1145, other program modules 1146, and some program data 1147.
[0329] Removable nonvolatile memory such as a floppy disk drive 1151 and a CD-ROM drive 1155 are connected to the removable nonvolatile memory interface 1150. For example, a floppy disk 1152 can be inserted into the floppy disk drive 1151, and a CD (Compact Disc) 1156 can be inserted into the CD-ROM drive 1155.
[0330] Input devices such as a mouse 1161 and a keyboard 1162 are connected to the user input interface 1160 .
[0331] The computer 1110 can be connected to a remote computer 1180 via a network interface 1170. For example, the network interface 1170 can be connected to the remote computer 1180 via a local area network 1171. Alternatively, the network interface 1170 can be connected to a modem (modulator-demodulator) 1172, and the modem 1172 is connected to the remote computer 1180 via a wide area network 1173.
[0332] Remote computer 1180 may include storage 1181 , such as a hard disk, that stores remote application programs 1185 .
[0333] Video interface 1190 is connected to monitor 1191 .
[0334] The output peripheral interface 1195 is connected to a printer 1196 and a speaker 1197 .
[0335] Fig.14 The computer system shown is merely illustrative and is in no way intended to limit the invention, its application, or uses.
[0336] Fig.14 The illustrated computer system may be implemented for any embodiment as a stand-alone computer, or as a processing system in a device, where one or more unnecessary components may be removed or one or more additional components may be added.
[0337] The present invention can be used for many applications. For example, the present invention can be used to monitor, identify, track objects in static images or moving videos captured by a camera, and is particularly advantageous for portable devices equipped with cameras, (camera-based) mobile phones, etc.
[0338] It should be noted that the methods and devices described herein may be implemented as software, firmware, hardware, or any combination thereof. Some components may, for example, be implemented as software running on a digital signal processor or microprocessor. Other components may, for example, be implemented as hardware and / or an application specific integrated circuit.
[0339] In addition, the method and system of the present invention can be implemented in a variety of ways. For example, the method and system of the present invention can be implemented by software, hardware, firmware or any combination thereof. The order of the steps of the method described above is only illustrative, and unless otherwise specifically stated, the steps of the method of the present invention are not limited to the order specifically described above. In addition, in some embodiments, the present invention can also be embodied as a program recorded in a recording medium, including machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also encompasses a recording medium storing a program for implementing the method according to the present invention.
[0340] Those skilled in the art will appreciate that the boundaries between the above operations are merely illustrative. Multiple operations can be combined into a single operation, a single operation can be distributed in additional operations, and operations can be performed at least partially overlapping in time. Moreover, alternative embodiments may include multiple instances of specific operations, and the order of operations may be changed in other various embodiments. However, other modifications, variations, and replacements are equally possible. Therefore, this specification and accompanying drawings should be considered illustrative, not restrictive.
[0341] In addition, embodiments of the present disclosure may further include the following illustrative examples (EE).
[0342] EE 1. An optimization device for a neural network model for object recognition, comprising:
[0343] a loss determination unit configured to determine loss data for features extracted from a training image set using the neural network model and a loss function with a weight function, and
[0344] an updating unit configured to perform an updating operation of parameters of the neural network model based on the loss data and the updating function,
[0345] The update function is derived from a loss function with a weight function based on the neural network model, and the weight function and the loss function change monotonically in the same direction in a specific value range.
[0346] EE2. An apparatus according to EE1, wherein the weight function and the loss function are both functions of an angle, wherein the angle is the angle between the extracted features mapped onto the hypersphere manifold and a specific weight vector in a fully connected layer of a neural network model, and wherein the specific value interval is a specific angle value interval.
[0347] EE3. An apparatus according to EE2, wherein the specific angle value interval is [0, π / 2], and the weight function and the loss function change monotonically and smoothly in the same direction in the specific angle value interval.
[0348] EE4. An apparatus according to EE2, wherein the loss function is a cosine function of the angle.
[0349] EE5. The apparatus according to EE1, wherein the loss function comprises an intra-class angle loss function, wherein the intra-class angle is the angle between the extracted features mapped onto the hypersphere manifold and the weight vector representing the true value object in the fully connected layer of the neural network model, and
[0350] The update function is determined based on the intra-class angle loss function and the intra-class angle weight function.
[0351] EE6. An apparatus according to EE1, wherein the intra-class angle loss function is a negative cosine function of the intra-class angle, and the weight function of the intra-class angle is a non-negative function that increases smoothly and monotonically as the angle increases over a specific value interval.
[0352] EE7. The apparatus according to EE1, wherein the value interval is [0, π / 2], and the weight function of the intra-class angle has a horizontal cutoff point near 0.
[0353] EE8. The apparatus according to EE1, wherein the loss function further comprises an inter-class angle loss function, and wherein the inter-class angle is the angle between the extracted features mapped onto the hypersphere manifold and other weight vectors in the fully connected layer of the neural network model, and
[0354] The update function is determined based on the inter-class angle loss function and the inter-class angle weight function.
[0355] EE9. An apparatus according to EE1, wherein the inter-class angle loss function is the sum of cosine functions of the inter-class angles, and the weight function of the inter-class angles is a non-negative function that smoothly and monotonically decreases as the angle increases over a specific value interval.
[0356] EE10. The apparatus according to EE1, wherein the value interval is [0, π / 2], and the weight function of the intra-class angle has a horizontal cutoff point near π / 2.
[0357] EE11. An apparatus according to EE1, wherein the update function is based on a partial derivative of the loss function and the weight function.
[0358] EE12. An apparatus according to EE1, wherein the update unit is further configured to multiply the partial derivative of the loss function by the weight function to determine an update gradient for updating the neural network model.
[0359] EE13. An apparatus according to EE12, wherein the updating unit is further configured to update the parameters of the neural network model using a back-propagation method and the determined update gradient.
[0360] EE14. An apparatus according to EE1, wherein, after the neural network model is updated, the loss determination unit and the update unit will operate using the updated neural network model.
[0361] EE15. An apparatus according to EE1, wherein the update unit is configured to perform an update using the determined update gradient when the determined loss data is greater than a threshold and the number of iterative operations performed by the loss determination unit and the update unit does not reach a predetermined number of iterations.
[0362] EE16. An apparatus according to EE1, wherein the loss data determination unit further comprises determining the loss data by utilizing a combination of a loss function of the neural network model and the weight function.
[0363] EE17. An apparatus according to EE1, wherein the combination of the loss function and the weight function of the neural network model is the product of the loss function and the weight function of the neural network model.
[0364] EE18. The device according to EE1 further includes an image feature acquisition unit configured to acquire image features from a training image set using the neural network model.
[0365] EE19. An apparatus according to EE1, wherein the neural network model is a deep neural network model, and the acquired image features are deeply embedded features of the image.
[0366] EE20. An apparatus according to EE1, wherein parameters of the weight function can be adjusted according to loss data determined on a training set or a validation set.
[0367] EE21. An apparatus according to EE20, wherein, after setting the first parameter and the second parameter of the weighted weight function to respectively perform iterative loss data determination operations and update operations, two parameters near one of the first parameter and the second parameter that results in better loss data are selected as the first parameter and the second parameter of the weighted function in the next iterative operation.
[0368] EE22. The apparatus according to EE20, wherein the weighting function is a Sigmoid function or a variant function thereof with similar characteristics, and the parameters include a slope parameter and a horizontal intercept parameter.
[0369] EE23. A method for training a neural network model for object recognition, comprising:
[0370] a loss determination step for determining loss data for features extracted from a training image set using the neural network model and a loss function with a weight function, and
[0371] An updating step for performing an updating operation of the parameters of the neural network model based on the loss data and the updating function,
[0372] The update function is derived from a loss function with a weight function based on the neural network model, and the weight function and the loss function change monotonically in the same direction in a specific value range.
[0373] EE 24. A device comprising
[0374] at least one processor; and
[0375] At least one storage device, the at least one storage device storing thereon instructions, which, when executed by the at least one processor, cause the at least one processor to perform the method according to EE 23.
[0376] EE25. A storage medium storing instructions, which, when executed by a processor, enable execution of the method according to EE 23.
[0377] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
[0378] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration and not for limiting the scope of the present disclosure. The various embodiments disclosed herein may be combined arbitrarily without departing from the spirit and scope of the present disclosure. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present disclosure.
Claims
1. An optimization device for a neural network model for object recognition, It is characterized in that include: a loss determination unit configured to determine loss data for features extracted from a training image set using the neural network model and a loss function with a weight function, and an updating unit configured to perform an updating operation of parameters of the neural network model based on the loss data and the updating function, The update function is derived from a loss function with a weight function based on the neural network model, and the weight function and the loss function change monotonically in the same direction in a specific value interval. Wherein, the weight function and the loss function are both functions of an angle, wherein the angle is the angle between the extracted features mapped onto the hypersphere manifold and a specific weight vector in the fully connected layer of the neural network model, and wherein the specific value interval is a specific angle value interval, and wherein the specific weight vector is at least one of the target feature center of the category to which the training image belongs and the target feature center of other categories.
2. The device according to claim 1, in, The specific angle value interval is [0, π / 2], and the weight function and the loss function change monotonically and smoothly in the same direction in the specific angle value interval.
3. The device according to claim 1, in, The loss function is a cosine function of the angle.
4. The device according to claim 1, in, The loss function includes an intra-class angle loss function, wherein the intra-class angle is the angle between the extracted features mapped to the hypersphere manifold and the weight vector representing the true value object in the fully connected layer of the neural network model, and The update function is determined based on the intra-class angle loss function and the intra-class angle weight function.
5. The device according to claim 4, in, The intra-class angle loss function is a negative cosine function of the intra-class angle, and the intra-class angle weight function is a non-negative function that increases smoothly and monotonically as the angle increases in a specific angle value interval.
6. The device according to claim 5, in, The specific angle value interval is [0, π / 2], and the weight function of the intra-class angle has a horizontal cutoff point near 0.
7. The device according to claim 1, in, The loss function further includes an inter-class angle loss function, and wherein the inter-class angle is the angle between the extracted features mapped onto the hypersphere manifold and other weight vectors in the fully connected layer of the neural network model except the weight vector representing the true value object, and The update function is determined based on the inter-class angle loss function and the inter-class angle weight function.
8. The device according to claim 7, in, The inter-class angle loss function is the sum of the cosine functions of the inter-class angles, and the inter-class angle weight function is a non-negative function that smoothly and monotonically decreases as the angle increases in a specific angle value interval.
9. The device according to claim 8, in, The specific angle value interval is [0, π / 2], and the weight function of the inter-class angle has a horizontal cutoff point near π / 2.
10. The device according to claim 1, in, The update function is based on the partial derivative of the loss function and the weight function.
11. The device according to claim 1, in, The updating unit is further configured to multiply the partial derivative of the loss function by the weight function to determine an update gradient for updating the neural network model.
12. The device according to claim 11, in, The updating unit is further configured to update the parameters of the neural network model using a back-propagation method and the determined update gradient.
13. The device according to claim 1, in, After the neural network model is updated, the loss determination unit and the updating unit will operate using the updated neural network model.
14. The device according to claim 1, in, The updating unit is configured to perform updating using the determined update gradient when the determined loss data is greater than a threshold and the number of iterative operations performed by the loss determining unit and the updating unit does not reach a predetermined number of iterations.
15. The device according to claim 1, in, The loss determination unit is further configured to determine the loss data using a combination of a loss function of the neural network model and the weight function.
16. The device according to claim 1, in, The combination of the loss function and the weight function of the neural network model is the product of the loss function and the weight function of the neural network model.
17. The apparatus according to claim 1, further comprising an image feature acquisition unit, wherein the image feature acquisition unit is configured to acquire image features from a training image set using the neural network model.
18. The apparatus according to claim 17, wherein the neural network model is a deep neural network model, and the acquired image features are deeply embedded features of the image.
19. The device according to claim 1, in, The parameters of the weight function can be adjusted according to the loss data determined on the training set or the validation set.
20. The device according to claim 19, in, After setting the first parameter and the second parameter of the weight function to perform iterative loss data determination and update operations respectively, two parameters near one of the first parameter and the second parameter that results in better loss data are selected as the first parameter and the second parameter of the weight function in the next iterative operation.
21. The device according to claim 19, in, The weight function is a Sigmoid function or a modified function thereof with similar characteristics, and the parameters include a slope parameter and a horizontal intercept parameter.
22. A method for training a neural network model for object recognition, It is characterized in that include: a loss determination step for determining loss data for features extracted from a training image set using the neural network model and a loss function with a weight function, and An updating step for performing an updating operation of the parameters of the neural network model based on the loss data and the updating function, The update function is derived from a loss function with a weight function based on the neural network model, and the weight function and the loss function change monotonically in the same direction in a specific value interval. Wherein, the weight function and the loss function are both functions of an angle, wherein the angle is the angle between the extracted features mapped onto the hypersphere manifold and a specific weight vector in the fully connected layer of the neural network model, and wherein the specific value interval is a specific angle value interval, and wherein the specific weight vector is at least one of the target feature center of the category to which the training image belongs and the target feature center of other categories.
23. A device comprising at least one processor; and At least one storage device storing thereon instructions which, when executed by the at least one processor, cause the at least one processor to perform the method of claim 22.
24. A storage medium storing instructions which, when executed by a processor, cause the method of claim 22 to be performed.
