Object recognition model training method and device

Abstract

The present disclosure relates to a training method and apparatus for an object recognition model. A training sample optimization apparatus for a neural network model for object recognition is provided. The apparatus comprises: for each training sample in a training sample library, a volatility determination unit configured to determine the volatility of a model prediction of the training sample relative to a corresponding label identifier of the training sample when training the neural network model; and an optimization unit configured to determine, based on the volatility of the training sample, whether the training sample can be used for training the neural network model in the next training cycle.

Description

Technical Field

[0001] The present disclosure relates to object recognition, and more particularly to the training and application of neural network models for object recognition. Background Art

[0002] In recent years, object detection, recognition, matching, and tracking in static images or a series of moving images (such as videos) have become widely and significantly applied in image processing, computer vision, and pattern recognition, playing a crucial role. Objects can be human body parts such as faces, hands, and bodies, other living things or plants, or any other object that one wishes to detect. Face / object recognition is one of the most important computer vision tasks, aiming to identify or verify a specific person / object based on an input image / video.

[0003] In recent years, neural network models for face recognition, particularly deep convolutional neural networks (CNNs), have achieved breakthroughs in significantly improving performance, greatly advancing the state of the art in face and object recognition. Given a training dataset, the CNN training process utilizes a common CNN architecture as a feature extractor to extract features from the training samples. Loss data is then calculated using various designed loss functions to supervise the training of the CNN model. Currently, the Softmax loss function and its variant (boundary-based Softmax loss function) are commonly used supervisory functions in face and object recognition. In addition to the evolving deep convolutional neural network architectures and effective softmax-based loss functions, large-scale training databases play a crucial role in deep face and object recognition.

[0004] However, current large-scale training databases are often suboptimal, often containing a certain proportion of noisy samples. This significantly impacts model training and recognition performance, resulting in inaccurate object recognition models and suboptimal recognition results. Furthermore, obtaining well-annotated large-scale training databases is expensive and time-consuming, leading to low processing efficiency.

[0005] Therefore, improved techniques are needed to improve the training of object recognition models, especially to optimize the training samples used for object recognition model training, especially to optimize the training samples to improve model training during training using a training database containing a certain proportion of noise samples.

[0006] Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Likewise, unless otherwise indicated, it should not be assumed that problems identified with respect to one or more approaches qualify as prior art based on this section. Summary of the Invention

[0007] One object of the present disclosure is to improve the training of recognition models for object recognition. Another object of the present disclosure is to improve object recognition in images / videos.

[0008] The present disclosure proposes an improved training method for a neural network model for object recognition, wherein the training samples used for training the neural network model are optimized during the training process so that the training samples used can be adaptively adjusted as the training process progresses. In particular, the use of inappropriate training samples for training can be effectively avoided during the training process, thereby obtaining a high-performance training model even for a training data set containing noise.

[0009] The present disclosure also proposes using the model obtained through the above training to perform object recognition, thereby further achieving improved object recognition.

[0010] In one aspect, a training sample optimization apparatus for a neural network model for object recognition is provided, the apparatus comprising: for each training sample in a training sample library, a volatility determination unit configured to determine the volatility of a model prediction of the training sample relative to a corresponding label identification of the training sample when training the neural network model; and an optimization unit configured to determine, based on the volatility of the training sample, whether the training sample can be used for training the neural network model in the next training cycle.

[0011] On the other hand, a neural network model training device for object recognition is provided, which includes a training sample optimization unit configured to optimize the training samples used for neural network model training for a given training sample library using the training sample library optimization device according to an embodiment of the present disclosure; and a training unit configured to use the optimized training samples to train the neural network model.

[0012] In yet another aspect, a method for optimizing training samples of a neural network model for object recognition is provided, the method comprising: for each training sample in a training sample library, a volatility determination step for determining the volatility of the model prediction of the training sample relative to the corresponding label identification of the training sample when training the neural network model; and an optimization step for determining whether the training sample can be used for training the neural network model in the next training cycle based on the volatility of the training sample.

[0013] On the other hand, a neural network model training method for object recognition is provided, which includes a training sample optimization step for optimizing the training samples used for neural network model training for a given training sample library using the training sample optimization method according to an embodiment of the present disclosure; and a training step for training the neural network model using the optimized training samples.

[0014] In yet another aspect, a method is provided comprising at least one processor and at least one storage device having instructions stored thereon, which when executed by the at least one processor may cause the at least one processor to perform the method as described herein.

[0015] In yet another aspect, a storage medium storing instructions is provided, which, when executed by a processor, may cause the method as described herein to be performed.

[0016] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention. In the drawings, like reference numerals designate like items.

[0018] Figure 1 A schematic diagram of the prior art using a neural network model for face recognition / authentication is shown.

[0019] Figure 2A and 2B A flowchart of neural network model training according to the prior art is shown.

[0020] Figure 3 Shown are the similarity differences between the model predictions and their labeled identities for clean and noisy samples during the training cycle.

[0021] Figure 4 A conceptual flow chart of a neural network model training process according to an embodiment of the present disclosure is shown.

[0022] Figure 5 A block diagram of a training sample optimization device for object recognition model training according to an embodiment of the present disclosure is shown.

[0023] Figure 6A The fluctuation of clean and noisy samples in a training cycle is schematically shown.

[0024] Figure 6B An exemplary algorithm flow chart of model training using dynamic sample optimization according to an embodiment of the present disclosure is shown.

[0025] Figure 7 A flowchart of a training sample optimization method for object recognition model training according to an embodiment of the present disclosure is shown.

[0026] Figure 8A block diagram of an object recognition model training device according to an embodiment of the present disclosure is shown.

[0027] Figure 9 A flowchart of an object recognition model training method according to an embodiment of the present disclosure is shown.

[0028] Figure 10 A flowchart of an exemplary neural network model training process according to an embodiment of the present disclosure is shown.

[0029] Figure 11 A flowchart of object recognition model training according to the first embodiment of the present disclosure is shown.

[0030] Figure 12 A flowchart of object recognition model training according to the second embodiment of the present disclosure is shown.

[0031] Figure 13 A flowchart of object recognition model training according to the third embodiment of the present disclosure is shown.

[0032] Figure 14 A flowchart of object recognition model training according to the fourth embodiment of the present disclosure is shown.

[0033] Figure 15 A block diagram illustrating an exemplary hardware configuration of a computer system capable of implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0034] Described herein are exemplary possible embodiments related to model training optimization for object recognition. In the following description, for illustrative purposes, numerous specific details are set forth to provide a thorough understanding of the present invention. However, it is apparent that the present invention can be practiced without these specific details. In other cases, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, obscuring, or obscuring the present invention.

[0035] In the context of the present disclosure, object recognition model training is performed based on training data / sample sets / libraries, which contain a large amount of data / samples used for object recognition model training. A typical example of training data / samples is training images.

[0036] In the context of this disclosure, an image may refer to any of a variety of images, such as a color image, a grayscale image, and the like. It should be noted that, in the context of this specification, the type of image is not specifically limited, as long as such an image can be processed for subsequent processing, such as for model training or for detecting / identifying whether the image contains an object. In particular, the image can be an original image or a processed version of the image, such as a version of the image that has undergone preliminary filtering or preprocessing before performing the operations of this application on the image.

[0037] In the context of this specification, an image containing an object refers to an image containing an object image of the object. This object image may also be referred to as an object region within the image. Object recognition also refers to identifying an image within an object region within an image.

[0038] In this article, the object can be a part of a person's body, such as a face, hand, body, etc., other organisms or plants, or any other object that you want to detect. As an example, the features of the object, especially the representative features, can be represented in the form of a vector, which can be called the "feature vector" of the object. For example, in the case of detecting a face, the pixel texture information, position coordinates, etc. of the representative part of the face are selected as features to form the feature vector of the image. Thus, based on the obtained feature vector, object recognition / detection / tracking can be performed. It should be noted that the feature vector may vary depending on the model used in object recognition and is not particularly limited.

[0039] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it need not be discussed again for subsequent drawings.

[0040] In the present disclosure, the terms "first," "second," etc. are used merely to distinguish between elements or steps, but are not intended to indicate temporal order, priority, or importance.

[0041] Figure 1 A basic operational concept diagram of face recognition / authentication using a deep face model in the prior art is shown, which mainly includes the training stage and the application stage of the deep face model, and the deep face model can be, for example, a deep convolutional neural network model.

[0042] In the training phase, the face image training set is first input into the deep face model to obtain the feature vector of the face image, and the existing loss function, such as the Softmax loss function and its variants, is used to obtain the classification probability P1, P2, P3, ..., P from the feature vector. c(where c indicates the number of categories in the training set, for example, there are c categories of face IDs), the classification probability indicates the probability that the image belongs to each of the c categories, and then the obtained classification probability is compared with the true value 0, 1, 0, ..., 0 (where 1 indicates the true value) to determine the difference between the two, such as cross entropy, as loss data, and feedback is performed based on the difference to update the deep face model, and the aforementioned operation is continued with the updated face model until a specific condition is met, thereby obtaining a trained deep face model.

[0043] During the testing phase, the face image to be identified or authenticated can be input into the trained deep face model to extract features for identification or authentication. Specifically, in actual application systems, there can be two specific applications: face / object recognition and face / object verification. The input of face / object recognition is generally a single face / object image, and a trained convolutional neural network is used to identify whether the face / object in the current image is the object to be identified; the input of face / object verification is generally a face / object image pair, and a trained convolutional neural network is used to extract the feature pairs of the input image pair. Finally, the similarity of the feature pairs is used to determine whether the input image pair is the same object.

[0044] Figure 1 An exemplary face authentication operation is shown in FIG. During the operation, two face images to be authenticated are input into the trained deep face model to authenticate whether the two face images are the face images of the same person. Specifically, the deep face model can obtain feature vectors for each of the two face images to form a feature vector pair, and then determine the similarity between the two feature vectors. For example, the similarity can be determined by a cosine function. When the similarity is not less than a specific threshold, the two face images can be considered to be the face images of the same person, and when the similarity is less than the specific threshold, the two face images can be considered to be different face images of the same person.

[0045] From the above description, it can be seen that the performance of the deep face model directly affects the accuracy of object recognition, and various methods are used in the existing technology to train the deep face model, such as the deep convolutional neural network model, to obtain a more complete deep convolutional neural network model. Figure 2A To describe the training process of a conventional deep convolutional neural network model.

[0046] First, a training dataset is input, which may include a large number of object images, such as face images, such as tens of thousands, hundreds of thousands, or millions of object images.

[0047] Then, the images in the input training data set can be preprocessed, and the preprocessing operations may include, for example, object detection, object alignment, normalization, and the like. In particular, object detection may, for example, refer to detecting a face from an image containing a face and obtaining an image mainly containing the face to be recognized, and object alignment may refer to aligning object images in different postures in the image to the same or appropriate posture, thereby performing object detection / recognition / tracking based on the aligned object images. Face recognition is a common object recognition operation, and for a face recognition training image set, preprocessing including, for example, face detection, face alignment, and the like may be performed. It should be noted that the preprocessing operations may also include other types of preprocessing operations known in the art, which will not be described in detail here.

[0048] Then, the preprocessed training set images are input into a deep convolutional neural network model for feature extraction. The convolutional neural network model can adopt various structures and parameters known in the art, etc., which will not be described in detail here.

[0049] The loss is then calculated using a loss function. Common loss functions are especially the Softmax loss function and its variants (e.g., the boundary-based Softmax loss function), which are commonly used supervisory signals in face / object recognition. These loss functions encourage separation between features, with the goal of ideally minimizing the intra-class distance while maximizing the inter-class distance. The general form of the Softmax loss function is as follows:

[0050]

[0051] Where N represents the number of minimum batches, is the identification feature of the i-th training image, such as the embedding feature, y i is x i The category label, is feature x i Assigned as category y i The predicted probability of , C represents the number of categories in the training data set, represents the weight of the last fully connected layer in DCNN, is the weight vector of the jth column of the last fully connected layer in DCNN, b j ∈R C is the bias term.

[0052] Then, the parameters of the convolutional neural network are updated through back propagation based on the calculated loss data.

[0053] The above operations are performed iteratively until a specific condition is met and the training ends.

[0054] In recent years, softmax-based loss functions such as NSoftmax, CosFace, ArcFace, etc. have also been developed. Although these functions have achieved great success in deep face / object recognition, they are designed for training clean datasets. However, they cannot achieve ideal results for existing face / object recognition training datasets that often contain noisy images, which will significantly reduce the recognition performance of deep face / object training models.

[0055] Training datasets often contain many different types of noise samples, such as label noisy images and low-quality images. Label noisy images may refer to images of non-labeled objects, for example, objects in the image are not included in the dataset labels, or are objects that are included in the dataset labels but are not labeled for training. Label noisy images can seriously affect the training and recognition performance of the model because they indicate incorrect objects. Low-quality images may refer to images that cannot provide sufficient effective features. Due to their low quality, effective information cannot be extracted to identify the identity, which also affects the recognition performance of the training model. For example, low-quality images may include misaligned images, blurred images, images with extreme postures, etc.

[0056] These noisy images, because they fail to provide accurate label identity information, can degrade the recognition performance of deep face / object training models. In particular, existing approaches assume a strict minimization of intra-class distances while maximizing inter-class distances given an ideal training dataset. Consequently, during training, noisy samples are often incorrectly projected onto the feature space of other objects, failing to effectively separate them from clean samples. This can lead to overfitting of the trained model due to the interference of noisy samples.

[0057] Although large-scale training databases can currently be optimized using semi-automatic data cleaning algorithms with manual annotation, noisy samples are still unavoidable. This is primarily due to the different recognition mechanisms of manual annotation and deep face / object recognition models, so manual annotation may not be as reliable as we would like. Consequently, even face / object recognition models trained on cleaned large-scale training databases still fall short of ideal recognition performance.

[0058] Jiankang Deng et al., in their study Sub-center ArcFace: Boosting Face Recognition by Large-scale Noisy Web Faces, proposed a multi-subcenter ArcFace loss function, called the "subcenter ArcFace" method, to improve the stability of trained models on noisy training datasets. This method first designs K subcenters for each class. Training samples only need to be close to any one of these K subcenters, rather than just one. This method encourages a primary subclass to contain the majority of clean samples, while non-primary subclasses contain difficult or noisy samples. Once the model achieves sufficient recognition capability, most clean samples will converge to the primary subclass center, while other difficult and noisy samples will converge to the centers of other subclasses. Next, non-primary subclass centers and high-confidence noisy samples are removed from the original training dataset to generate a new training dataset. Finally, a deep face / object training model is trained using this new training dataset, minimizing the impact of noisy samples on the model. Figure 2B The process of model training using the sub-center ArcFace method in the prior art is shown.

[0059] However, in the Sub-center ArcFace method, two trainings are required to obtain the final face / object recognition model; and since the noise samples are only cleaned once using a fixed angle threshold, the new training dataset may not be an ideal training dataset.

[0060] Therefore, it is necessary to further optimize the training sample set used for model training so as to effectively suppress the influence of noise samples in the training sample set on model training during the training process.

[0061] Through research, the inventors realized that during training, the gradient of a noisy image causes the face model to predict the labeled identity; however, generalizing the training to other face images prevents these images from falling into the feature space of the labeled identity. In other words, during training, noise always has an opposite effect on clean samples. These two opposing effects caused by noisy samples in the training set lead to fluctuations in the training process, providing clues for distinguishing clean samples from noisy ones.

[0062] Figure 3The relationship between the simulated prediction curves and the label identification curves of training samples of different training periods discovered by the inventors through research is shown. As an example, the relationship is indicated by the cosine similarity comparison of the model prediction and its label identification, where (a) shows the cosine similarity comparison of the model prediction curve of the clean sample and the label identification curve, and (b) shows the cosine similarity comparison of the model prediction curve of the noise sample and the label identification curve. It can be seen that with the increase of the training cycle, the similarity difference of the clean sample tends to be stable, with very small volatility, and its model prediction curve almost coincides with the label identification curve. However, the similarity difference of the noise sample is often unstable, with large volatility, and its model prediction curve often does not coincide with the label identification curve. In particular, even for selected "difficult samples" (such as large postures and occlusions), the similarity difference between the corresponding clean samples and the noise samples is still significant.

[0063] In view of this, the present disclosure proposes an improved training data optimization method, which can optimize the training data by utilizing the volatility of training samples in the training database during the training process and perform model training based on the optimized training data, thereby achieving improved model training and improving model recognition capabilities.

[0064] According to one aspect of the present disclosure, during the training process, the volatility of training samples in a training database is used to detect / distinguish clean samples from noisy samples. Based on the detection / distinguishment results, the noisy data is discarded and the clean data is used for model training. It should be noted that the clean and noisy samples are merely exemplary. Clean samples essentially refer to samples with small model training volatility and are suitable for model training, while noisy samples essentially refer to samples with excessive model training volatility and are unsuitable for model training. In this way, utilizing volatility enables more appropriate selection of samples for model training, improving model training and enhancing model recognition capabilities.

[0065] The training data optimization method according to the present disclosure can be performed in various appropriate ways. According to one embodiment, the training data optimization method can be performed in a so-called "static" manner, wherein the training data can be optimized for each training session / round of training during the training process, and the training data used for all training cycles included in a training session / round of training is fixed. According to another embodiment, the training data optimization method can be performed in a so-called "dynamic" manner, wherein the training data can be optimized for each training cycle included in a training session / round of training during the training process, so that the training data used in each training session / round of training is dynamically changing.

[0066] As an example, the training data optimization method can adopt a phased training strategy during the training process, such as a two-round or multi-round training strategy. For example, in a two-round training strategy, the first round of training uses all images in the training database to train a noise detection model; then, the fluctuation scores of all images are calculated, and noisy images are detected based on the calculated fluctuation scores; then, the noisy images are removed from the original training database to obtain a new training database; and then, in a second round of training, the new training database is used to train the final network model. A multi-round training strategy can also be implemented in a similar manner. It should be noted that the database remains unchanged in each training round, and the database used for model training in each training round can be derived from the initial training database or from the training database used in the previous training round. It should be noted that each training round can include a specific number of training cycles, where the specific number can be a predetermined number of training cycles or the number of training cycles required to meet the training end condition.

[0067] As another example, more preferably, the training data optimization method can employ a dynamic training data optimization method. During the training process, the fluctuation value of the training samples is continuously updated based on the model prediction, and inappropriate training data is dynamically discarded in each training cycle. Specifically, in each training cycle during the training process, the fluctuation value of the training samples is determined based on the model prediction, and appropriate samples for model training in the next training cycle are selected from the training database based on the fluctuation value. In this way, training samples can be dynamically updated and optimized rather than directly eliminated, thereby training a more accurate model.

[0068] According to another aspect of the present disclosure, during the training process, the volatility of training samples in the training database is used to mark / determine the confidence of the training samples, and the model training is effectively guided based on the confidence. In particular, the model loss function is weighted according to the confidence, thereby effectively improving the convergence of model training and improving model training.

[0069] According to yet another aspect of the present disclosure, more preferably, the volatility of training samples in the training database is used to detect / distinguish clean samples from noise samples during the training process, thereby discarding noise data based on the detection / distinguishing results and using clean data for model training, and marking / determining the confidence of the clean data based on its volatility for weighting the model loss function, so that the model training can be performed with more appropriate samples and the model training convergence can be effectively improved, thereby improving the model training.

[0070] The present disclosure further proposes improved model training for object recognition, wherein during the training process, a training data optimization method according to an embodiment of the present disclosure is adopted to obtain an optimized training data set, and the optimized training data set is used for model training. This training process will be iteratively executed until the training end condition is met.

[0071] The following will refer to Figure 4 A conceptual flowchart is provided to describe model training according to the present disclosure using the training data optimization method according to the present disclosure.

[0072] First, a noisy training database is input. The training database may include a large number of object images, such as human face images. For example, tens of thousands, hundreds of thousands, or even millions of object images may contain a certain proportion of noisy images. It should be noted that the object images in the training database may also be appropriately preprocessed, as described above.

[0073] The current network model is then trained using the selected loss function based on the current training database, and a volatility score is calculated for all training samples based on the training results. The volatility score is merely exemplary and essentially refers to any suitable value representing the volatility of the training samples during model training. For example, the volatility score can be calculated based on a defined volatility score function. The volatility score function can be any suitable function as long as the function can obtain a value indicating the volatility of the training samples during model training.

[0074] Then, clean samples and noise samples are detected according to the fluctuation score calculated for each sample to update the network database, so that the updated network database only contains clean samples, thereby effectively avoiding the use of noise samples in model training.

[0075] Afterwards, it can be determined whether to terminate the training, for example, by determining whether a training termination condition has been met. If not, the process returns to the model training process, where the updated network database is used for model training, thereby iteratively performing the aforementioned operations. Conversely, if the training termination condition is met, the trained network model is output, thereby terminating the model training.

[0076] In particular, the parameters of the convolutional neural network can usually be updated through back propagation based on the calculated loss data, and the model training can be performed using the updated network database, thereby iteratively performing the aforementioned operations. In particular, back propagation can also include weighting the loss function based on the calculated sample volatility score to update the loss function, so that subsequent model training can be performed using the updated loss function. As an alternative, the loss function weighting operation can also be included in the aforementioned operation of training the current network model using the selected loss function, and the selection of the loss function here can include loss function weighting.

[0077] It should be pointed out that Figure 4 The operation sequence shown in is only exemplary, and the operation sequence of the steps can be adjusted appropriately. As an example, the determination of whether to end the training can be performed before updating the network database, or even before calculating the fluctuation score.

[0078] In addition, it should be noted that Figure 4 The iterative operations in [ ] can be applied to the aforementioned various appropriate methods of training data optimization operations. As an example, iterations may be performed for each model training run. In this case, the fluctuation score calculation may be performed for the current model training run, such as for all training cycles in a training run, the last predetermined number of training cycles, and so on. As another example, iterations may be performed for each training cycle. In this case, the fluctuation score calculation may be performed for the current training cycle.

[0079] In this way, according to the embodiments of the present disclosure, as the training cycle increases, the training data can be dynamically optimized and the model training can be performed based on the optimized training data. The recognition ability of deep features will gradually increase, and the accuracy of noise detection will also improve. The two can complement each other and gradually improve together.

[0080] The following describes embodiments of sample optimization for object recognition model training and corresponding object recognition model training according to the present disclosure with reference to the accompanying drawings.

[0081] Figure 5 A block diagram of a training sample optimization apparatus for a neural network model for object recognition according to the present disclosure is shown. The apparatus 500 includes a volatility determination unit 502 and an optimization unit 504. For each training sample in a training sample library, the volatility determination unit 502 is configured to determine the volatility of the model prediction of the training sample relative to the corresponding label identification of the training sample when training the neural network model; and the optimization unit 504 is configured to determine whether the training sample can be used for subsequent training of the neural network model based on the volatility of the training sample.

[0082] In an embodiment of the present disclosure, the neural network model may be any type of neural network model capable of realizing object recognition, preferably a deep neural network (DNN) model, such as a convolutional neural network (CNN) model, a recurrent neural network (RNN), and the like.

[0083] In the embodiments of the present disclosure, a model prediction for a training sample may refer to the training result / prediction result corresponding to the training sample when training a network model based on a training database. This may be represented, for example, by a feature vector, particularly an embedded feature vector. The corresponding label identifier for the training sample refers to the class label identifier that the training sample should belong to. These may be represented in a suitable form in the art, such as a vector. As an example, volatility indicates the deviation between the feature vector of a training sample and its corresponding class label vector.

[0084] According to the present disclosure, the volatility of training samples may be determined in various appropriate ways.

[0085] According to one embodiment, for training samples in a training sample library that can be used for neural network model training in the current training, the volatility of the training sample is determined based on the mathematical statistics of the difference between the model prediction similarity and the label identification similarity of the training sample. According to one embodiment, the model prediction similarity of the training sample indicates the minimum value of the offset between the predicted feature of the training sample and the weight vector representing all categories in the last fully connected layer of the neural network model, and / or the label identification similarity of the training sample indicates the offset between the predicted feature of the training sample and the weight vector representing the true label category in the last fully connected layer of the neural network model. As an example, the offset essentially refers to the amount of separation between the predicted feature and the weight vector, which can be indicated by various appropriate forms, such as angle / angular distance and various appropriate function values ​​thereof, etc.

[0086] According to one embodiment, the model prediction similarity of the training sample is determined based on the minimum angle between the predicted feature of the training sample and the weight vector representing all categories in the last fully connected layer of the neural network model, and / or the label identification similarity of the training sample is determined based on the angle between the predicted feature of the training sample and the weight vector representing the true label category in the last fully connected layer of the neural network model.

[0087] Figure 6A Schematic diagram showing an example of the volatility of clean and noisy samples in a training cycle, where the volatility is determined based on the angle. yi Indicates the embedded features x of the training samples i With the mark logo W yi The angle between min Indicates the embedded feature x iWith all class labels W j ∈{W1,W2,…,W c Thus, for each of the clean sample and the noise sample, we can calculate the angle between the clean sample and the noise sample based on θ yi and θ min The difference between θ and θ determines the volatility of the sample in the training cycle. Of course, volatility can also be based on θ yi and θ min The volatility is determined based on the difference between the values ​​of a specific function of , such as a cosine function, etc. In particular, the volatility is determined based on the cumulative statistical value of the angle difference.

[0088] According to one embodiment, the volatility of a training sample corresponds to a mathematical statistic of the cumulative value of the volatility of the training sample in the current training and a specific number of previous training cycles. As an example, the volatility of the training sample can be calculated based on the θ yi and θ min The mathematical statistics of the cumulative values ​​of the differences between the training cycles and the training cycles are used to determine the volatility of the samples in the training cycles. As an example, the mathematical statistics may be an average, a median, or any other appropriate statistical value. As an example, the previous specific number of training cycles may include all training cycles before the current training cycle, or may be a specific number of training cycles immediately before the current training cycle, or may even be a specific number of training cycles that meet specific conditions.

[0089] According to an embodiment of the present disclosure, for a training sample in a training sample library that cannot be used for neural network model training in the current training cycle, the volatility of the training sample corresponds to a mathematical statistic of the volatility of a specific number of training samples in the training sample library that can be used for neural network model training. As an example, the statistic can be an average, a median, or any other suitable statistic. As an example, when a training sample is a sample in the training sample library that is not used for the current model training, its similarity score difference is the mean of the similarity score differences of other training samples that can be used for model training in the current training cycle.

[0090] An example of training sample volatility calculation according to the present disclosure is described below, where volatility is indicated by a volatility score.

[0091] As an example, the training sample X i The volatility fraction F(X i ) can be defined as:

[0092]

[0093] Among them, d k (X i ) represents the training sample X in the kth training cycle iThe similarity score difference is , K is the number of training cycles that have been conducted, and K is not greater than the maximum number of training cycles K max That is, for a training cycle, the fluctuation score of the training samples in the training cycle may be the average value of the similarity score differences in each training cycle from the initial training cycle to the training cycle.

[0094] For samples that can be used for model training in the current training cycle (e.g., the kth training cycle), the similarity score difference d k (X i ) is calculated from the similarity cosθ of the kth training cycle min and The difference is calculated, namely:

[0095]

[0096] Among them, θ min =minimum{θ1,θ2,…,θ C}, and cos(θ j )=W j T x i / ||W j ||‖x i ‖. For this training sample, if the training sample has been involved in model training, the fluctuation score of the sample can be

[0097]

[0098] For samples that are not used in the model training of the current training cycle (e.g., the kth training cycle), the similarity score difference d k (X j ) can be calculated as the mean of the similarity score differences of the samples that can be used for model training in this training cycle, that is:

[0099]

[0100] Figure 6B A portion of an algorithm for model training using dynamic training sample optimization according to an embodiment of the present disclosure is exemplarily shown, wherein as the training cycle progresses, the fluctuation scores of each training sample are dynamically determined and training samples that can be used for model training are selected based on the fluctuation scores.

[0101] Therefore, based on the determined volatility of each training sample in the database, it is possible to determine whether the training sample is suitable for subsequent model training, thereby appropriately updating the training sample for subsequent model training according to the volatility of the training sample.

[0102] The updating of training samples can be performed in various appropriate ways. Preferably, as an example, the updating can include selecting training samples suitable for model training from the training sample database, so that the training sample database remains unchanged during the training process, and appropriate training samples are selected for training in each training session, or even in each training cycle. In this way, all training samples in the training sample database can be fully utilized. Even if a training sample is judged to be a low-quality sample in a certain training session or a certain training cycle, as long as the training sample has recognition ability, the training sample may still be used for subsequent model training, thereby obtaining a network model with better generalization. Of course, as another example, the updating can include removing training samples that are not suitable for model training from the training sample database, so that the model training is performed using the updated training sample database after the training samples are removed. Here, as the training progresses, the training sample database may gradually decrease. It should be noted that in this example, all samples in the training sample database are used for model training, so the calculation of the fluctuation score of samples not used for model training described above will no longer be performed.

[0103] According to one embodiment, when the volatility of a training sample is less than a specific threshold, the training sample can be used for subsequent training of the neural network model. In particular, for a training cycle in training, the volatility of the training sample in the current training cycle can be used to determine whether the training sample can be used for model training in the next training cycle.

[0104] As an example, a corresponding discarding mask may be set for a training sample based on a determination result of whether the volatility is less than a specific threshold, so as to use the discarding mask to indicate whether the training sample is suitable for subsequent model training.

[0105] As an example, we can use M(X i ) is the image X i The training data dropout mask is defined as follows:

[0106]

[0107] Among them, F(X i ) represents the training sample X i The fluctuation score can be calculated as described above, for example, thd is a specific threshold. Thus, training samples whose discard mask is 0 are identified as noise samples and excluded from model training, while training samples whose discard mask is 1 are identified as clean samples and can be used in subsequent model training. It should be noted that the above values ​​of 0 and 1 are merely exemplary, and any other appropriate values ​​can be used as long as they can distinguish clean samples from noise samples.

[0108] According to an embodiment of the present disclosure, a specific threshold for detecting / selecting training samples based on volatility may be determined in various appropriate ways.

[0109] According to one embodiment, the specific threshold is a pre-set fixed threshold. For example, the fixed threshold may be set based on user experience, or may be set with reference to previous training results, or may correspond to a specific model training / application scenario.

[0110] According to one embodiment, the specific threshold may be a dynamic threshold, for example, dynamically determined based on the volatility of training samples. In particular, it may be dynamically determined based on the volatility of training samples in current training or previous training.

[0111] According to one embodiment, the dynamic threshold may be determined based on mathematical statistics of volatility in previous training sessions. For example, it may be determined based on mathematical statistics of volatility of clean samples in a predetermined number of previous training sessions, such as the mean, median, or maximum value. The predetermined number of previous training sessions may be all previous training sessions immediately preceding the current training session, a predetermined number of consecutive previous training sessions immediately preceding the current training session, or a predetermined number of previous training sessions preceding the current training session, arranged in ascending order of volatility, etc.

[0112] According to one embodiment, the dynamic threshold may be determined based on the proportion of training samples in the training sample library that can be used for model training and the volatility of each training sample in the training sample library during current training.

[0113] As an example, it can be pre-set that in each training cycle, a specific proportion of training samples in the training sample library can be used for model training. This setting can be reflected in the threshold, which can correspond to a specific number of volatility values ​​arranged from small to large in terms of volatility values. The specific number can correspond to the product of the total number of samples in the sample library and the specific proportion.

[0114] The following describes the loss functions that can be used in model training according to the present disclosure.

[0115] According to embodiments of the present disclosure, the loss function used to calculate the loss data may take any appropriate form. As an example, a general loss function, such as the original loss function of model training, may be used to calculate the loss data. For example, the loss function used in the model training process may adopt a loss function known in the art, such as the Softmax loss function and its variants, such as the various known loss functions described above.

[0116] According to an embodiment of the present disclosure, the loss function may be a function of an angle, and in particular, may be a cosine function of an angle. The angle is the angle between the extracted features of the training sample and a specific weight vector in the fully connected layer of the neural network model. In particular, the specific weight vector may be the feature center of a certain category of objects in the training atlas. According to one embodiment, model training may mainly consider intra-class loss, and accordingly, the loss function may be a function of the intra-class angle, such as a cosine function.

[0117] As an example, the softmax-based loss function removes the bias term and can be converted to W j T x i =scosθ i . Further, we can introduce the idea of ​​large boundaries into Therefore, the loss function based on softmax can be defined as:

[0118]

[0119] Among them, in the NSoftmax method In the CosFace method In the ArcFace method

[0120] According to embodiments of the present disclosure, the loss data to be considered in model training may include both intra-class loss and inter-class loss. Thus, the loss function used for model training may be a function that considers both intra-class loss and inter-class loss. For example, it may include two sub-functions: an intra-class loss function and an inter-class loss function, which are, respectively, an intra-class angle loss function and an inter-class angle loss function.

[0121] The present disclosure can directly optimize the angle between eigenvectors as the loss function target, without requiring conversions that would otherwise degrade performance. This ensures that the loss function target is consistent with the prediction process target. Specifically, the loss function target and the element used for judgment in the prediction process can both be angles. This simplifies the determination of loss data and the feedback based on it, reduces intermediate conversion processing, lowers computational overhead, and avoids degradation of computational accuracy.

[0122] According to one embodiment, the loss function of the training sample when the neural network model is trained using the training sample library can be weighted based on the volatility of the training sample. According to one embodiment, the loss function used in the training can be a combination of the loss function of the neural network model and a weighting function. According to one embodiment, the loss function used to calculate the loss data can be the product of the loss function of the neural network model and the weighting function. In particular, the loss function of the neural network model here refers to the original loss function that is not weighted by the weighting function, and the weighting function is a function determined based on the volatility of the training sample, which can take any appropriate form.

[0123] Preferably, in the embodiment of the present disclosure, the aforementioned image X may be used. i The training data discard mask M(X i ) is used as a weighting function to weight the loss function, so that the influence of the training samples in the training database on the model training can be dynamically adjusted according to the volatility of the training samples. In this way, M(X i ) In addition to indicating whether a training sample participates in model training, the loss function of the training sample can also be weighted accordingly.

[0124] As an example, in the softmax-based loss function, the new loss function obtained by weighting is defined as follows:

[0125]

[0126] Among them, M(X i ) can be determined based on the comparison of volatility with a specific threshold, as described above.

[0127] In the present disclosure, the training data can be further discarded mask M(X i ) is appropriately weighted to obtain a weighted discard mask, so that the influence of training samples on model training during training can be more appropriately adjusted according to the characteristics of the samples.

[0128] According to one embodiment, the drop mask M(X i ) can be further determined / weighted based on the ratio between the volatility of the training sample and the maximum value of the volatility of all training samples.

[0129] As an example, M(X i ) can be:

[0130]

[0131] Among them, F(X i ) is the training sample X iThe fluctuation score, max(F(X i )) is the maximum fluctuation score corresponding to the training sample in the current database, D thr is the sample discard threshold, δ is set to a very small initial value, which can be set arbitrarily.

[0132] Here, M(X i ) essentially reflects the confidence level of a training sample during training. The smaller the volatility of a training sample, the more suitable it is for training. The higher its confidence level for model training, the greater the weighting of the loss function for that training sample. In this way, the confidence level of a training sample can be used to effectively guide network model training.

[0133] According to one embodiment, the drop mask M(X i ) can be further determined based on the number of training cycles until the current training.

[0134] As an example, M(X i ) can be:

[0135]

[0136] Among them, F(X i ) is the training sample X i The fluctuation score, max(F(X i )) is the maximum fluctuation score corresponding to the training sample in the current database, D thr is the sample discard threshold, δ is set to a very small initial value, T thr is a fixed training cycle threshold, and t is the number of current training cycles. Specifically, when the number of training cycles is less than the training cycle threshold T thr When , the weighted discard mask of the clean sample is When the number of training cycles is equal to the training cycle threshold T thr When the weighted discard mask of the clean sample is 1, the number of training cycles is greater than the training cycle threshold T thr When , the weighted discard mask of the clean sample is

[0137] Here M(X i ) also reflects the confidence of the training samples during the training process, and in particular further considers the influence of the training cycle. The confidence of the training samples that can reach a specific stability within a specific training cycle is high, while the confidence of the training samples that exceed the specific training cycle is low.

[0138] It should be noted that in the above weighted discard mask M(X i), a value of 0 indicates that the training sample can be discarded, while a value other than 0 indicates that the training sample can be retained for model training, thus M(X i ) can be used to determine whether a sample is to be retained or discarded while providing weighting. Similarly, the above 0 and 1 are merely exemplary, and any other appropriate value can be used as long as it can distinguish between clean samples and noise samples.

[0139] According to embodiments of the present disclosure, the weighting of the loss function can be performed dynamically as the training process progresses. For example, during each round of training, the volatility score of each sample is calculated, and the loss function of each sample is weighted according to the volatility score. The loss functions of all samples are then summed to determine the total loss. In this way, the loss function can be dynamically updated based on volatility as the model trains, thereby further adaptively controlling the range of model updates / optimizations and improving model updates / optimizations.

[0140] According to an embodiment of the present disclosure, the training data optimization of the present disclosure can be performed as the training process is executed, for example, in an iterative manner. As an example, after model training, it can be determined whether the trained neural network model meets the end condition. If the trained neural network model does not meet the training end condition, subsequent model training and data optimization are performed until the trained neural network model meets the training end condition, and then the iterative training and optimization are stopped.

[0141] According to one embodiment, the training termination condition may be any suitable termination condition commonly used in the field of model training. For example, the specific condition may be related to the training results or the number of adjustments. For example, the termination condition may be at least one of the following: the number of training cycles reaches a predefined maximum number of training cycles; the calculated loss of the neural network model is lower than a predefined threshold; the difference between the current training result and the previous training result is lower than a predefined threshold, etc.

[0142] As an example, parameter adjustment may no longer be performed when a predetermined number of adjustments is reached. As another example, the selection may be made by comparing the current training result with the previous training result. The training result may, for example, be loss data determined by the model determined by the current training. If the current training result is inferior to the previous training result, the parameters are not adjusted. However, if the current training result is superior to the previous training result, the parameters may be adjusted further using the previous parameter adjustment method until the predetermined number of adjustments is reached or the training result no longer improves.

[0143] It should be pointed out that Figure 5 This is only a schematic structural configuration of the training sample optimization device. The training device may also include other possible units / components (for example, memory, etc.).

[0144] According to an embodiment of the present disclosure, the apparatus may further include an image feature acquisition unit configured to acquire image features from the training image set using the neural network model. Image feature acquisition can be performed using methods known in the art and will not be described in detail here. Of course, the image feature unit may be located outside the training sample optimization apparatus according to the present disclosure.

[0145] According to an embodiment of the present disclosure, the memory can store various information generated by the training device (for example, training set image features, loss data, function parameter values, etc.), programs and data used for the operation of the training device, etc. For example, the memory can include but is not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. As an example, the memory can also be located outside the training device. The training device can be directly or indirectly (for example, other components may be connected in the middle) connected to the memory to access data. The memory can be a volatile memory and / or a non-volatile memory.

[0146] It should be noted that the above-mentioned units are merely logical modules divided according to the specific functions they implement, and are not intended to limit specific implementation methods. For example, they can be implemented in software, hardware, or a combination of software and hardware. In actual implementation, the above-mentioned units can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). In addition, the above-mentioned units are shown with dotted lines in the drawings to indicate that these units may not actually exist, and the operations / functions they implement can be implemented by the processing circuit itself.

[0147] It should be noted that, in addition to including multiple units, the above-mentioned training device can be implemented in a variety of other forms, such as a general-purpose processor or a dedicated processing circuit, such as an ASIC. For example, the training device can be constructed by a circuit (hardware) or a central processing device (such as a central processing unit (CPU)). In addition, the training device can carry a program (software) for operating the circuit (hardware) or the central processing device. The program can be stored in a memory (such as, arranged in the memory) or in an externally connected external storage medium, and can be downloaded via a network (such as the Internet).

[0148] According to an embodiment of the present disclosure, a training sample optimization method for a neural network model for object recognition is proposed, such as Figure 7As shown. Method 700 includes a volatility determination step S702 and an optimization step S704. For each training sample in the training sample library, volatility determination step S702 may be used to determine the volatility of the model prediction of the training sample relative to the corresponding label identifier of the training sample when training the neural network model; and optimization step S704 may be used to determine, based on the volatility of the training sample, whether the training sample can be used for training the neural network model in the next training cycle. Through the above operations, an optimized training sample can be obtained for model training.

[0149] According to the present disclosure, the method may further include an image feature acquisition step for acquiring image features from the training image set using the neural network model. Image feature acquisition can be performed using methods known in the art and will not be described in detail here. Of course, the image feature acquisition step may not be included in the method according to the present disclosure.

[0150] It should be noted that the training sample optimization method according to the present disclosure may also include the various operations described above, which will not be described in detail here. It should be noted that the various steps / operations of the training sample optimization method according to the present disclosure may be performed by the aforementioned units or by various forms of processing circuits.

[0151] According to the present disclosure, a neural network model training device 800 for object recognition is proposed, which includes a training sample optimization unit 802 configured to optimize the training samples used for neural network model training for a given training sample library using the training sample optimization device according to an embodiment of the present disclosure; and a training unit 804 configured to train the neural network model using the optimized training samples. It should be noted that, as an example, the training sample optimization unit can be implemented as the training sample optimization device according to the present disclosure, or can be implemented as a unit that can operate using the training sample optimization device according to the present disclosure.

[0152] According to an embodiment of the present disclosure, the training apparatus 800 further includes an end determination unit 806 for determining whether the trained neural network model satisfies a training end condition. If it is determined that the trained neural network model does not satisfy the training end condition, further optimization and training are performed using the optimized training samples by the training sample optimization unit and the training unit. The training end condition may be as described above and will not be described in detail here.

[0153] According to an embodiment of the present disclosure, the training apparatus 800 may further include a weighting unit 808 for weighting the loss function of the training sample based on the volatility of the training sample. In particular, as an example, with reference to the discard mask of each training sample, the weighting unit may only weight the loss function of the training sample used for neural network model training, and the weighting operation may be as described above. For training samples that do not participate in the neural network model training, weighting may not be performed directly, or even if weighting is performed, because the loss function of the sample can be set to 0, the weighted loss function is still 0.

[0154] It should be noted that the aforementioned end determination unit and weighting unit may also be included in the training sample optimization unit, for example, in the training sample optimization device according to the present disclosure, or may be located outside the training device according to the present disclosure, as indicated by dotted lines in the accompanying drawings. According to an embodiment of the present disclosure, the device may also include the image feature acquisition unit described above, which may also be located outside the training device.

[0155] According to the present disclosure, a neural network model training method for object recognition is proposed, such as Figure 9 As shown, the method 900 includes a training step S902 for performing neural network model training using optimized training samples; and a training sample optimization step S904 for optimizing the training samples used for neural network model training using a training sample optimization method according to an embodiment of the present disclosure for a given training sample library. It should be noted that at the beginning of training, the training step can use the initial training sample library to perform neural network model training, and as training progresses, the training step will use optimized training samples for training.

[0156] According to an embodiment of the present disclosure, method 900 may further include a weighting step S906 for weighting the loss function of the training sample based on the volatility of the training sample; and a training end determination step S908 for determining whether the trained neural network model meets the training end condition. If training is not ended, the training step and the training sample optimization step are repeated.

[0157] It should be noted that the above-mentioned end determination step and weighting step can also be included in the training sample optimization step, for example, included in the training sample optimization method according to the present disclosure, or located outside the training method according to the present disclosure, as indicated by dotted lines in the accompanying drawings. According to an embodiment of the present disclosure, the method may also include the image feature acquisition step as described above, of course, this image feature acquisition step can also be located outside the training method.

[0158] It should be pointed out that Figure 9The order of execution of the steps in the method shown in the figure is merely exemplary, and the order of the steps can be adjusted appropriately. For example, the end determination step can be performed before the training sample optimization step or even before the weighting step. In this way, if it is determined that training can be terminated after the current training is executed, there is no need to perform training sample optimization and loss function weighting.

[0159] It should be noted that the model training method according to the present disclosure may also include the various operations described above, which will not be described in detail here. It should be noted that the various steps / operations of the model training method according to the present disclosure can be performed by the above-mentioned units or by various forms of processing circuits.

[0160] The following will refer to Figure 10 To describe an exemplary implementation of a model training operation according to the present disclosure.

[0161] First, a noisy training database is input. The training database may include a large number of object images, such as human face images. For example, tens of thousands, hundreds of thousands, or even millions of object images may contain a certain proportion of noisy images. It should be noted that the object images in the training database may also be appropriately preprocessed, as described above.

[0162] Then, the discard mask of each training sample is initialized. For example, the discard mask of each training sample can be set to 1 to indicate that all training samples can be used for model training.

[0163] Then, the current network model is trained using the determined loss function according to the current training database. The loss function here can be any appropriate function, such as the various loss functions described above, so as to obtain the model prediction results of each sample.

[0164] Then, the similarity between the minimum model prediction and the label identification of each sample in the current training database is calculated and recorded, and the fluctuation score of all training samples is calculated based on this. The calculation operation here can be performed as described above.

[0165] Subsequently, a discarding mask for each training sample can be generated based on the fluctuation score calculated for each sample. For example, the fluctuation score can be compared with a threshold to generate a discarding mask as described above, so that clean samples and noisy samples can be detected / distinguished to update the network database, so that the updated network database only contains clean samples for performing model training.

[0166] Afterwards, it can be determined whether to end the training, for example, whether the training end conditions have been met. If not, the process returns to the model training process. In particular, the parameters of the convolutional neural network can usually be updated through backpropagation based on the calculated loss data, and the model training can be performed using the updated network database, thereby iteratively performing the aforementioned operations. In particular, backpropagation can also include weighting the loss function according to the calculated sample volatility score to update the loss function, so that subsequent model training can be performed using the updated loss function. The loss function update can be defined as described above and will not be described in detail here. Alternatively, the loss function weighting operation can also be included in the aforementioned operation of training the current network model using the determined loss function, and the determination of the loss function here can include the loss function weighting.

[0167] On the contrary, if the training end condition is met, the trained network model is output, thereby ending the model training.

[0168] The following experiment compares the effectiveness of the model training method disclosed in this disclosure with that of the prior art. This experiment aims to verify the effectiveness on a small-scale training set. The experimental settings are as follows:

[0169] Training set: CASIA-WebFace, including 10,000 person identities and a total of 500,000 images.

[0170] Test sets: LFW, CFP-FP, AGEDB-30, CPLFW, CALFW, IJB-C

[0171] Evaluation criteria: 1:N TPIR (True Positive Identification Rate, Rank 1@10 6 ) and TPR / FPR

[0172] Convolutional neural network architecture: RestNet50

[0173] Comparison of existing technologies: ArcFace, Sub-center ArcFace

[0174] The experimental results are shown in Table 1 and Table 2 below. The experimental results show that the solution disclosed in the present invention has better performance than the prior art, and in particular, achieves better model recognition performance.

[0175] Table 1: Comparison of the results of the solution disclosed in this disclosure and the existing technology on a general dataset

[0176]

[0177]

[0178] Table 2: Comparison of the results of the disclosed solution and the existing technology on the IJB-C dataset

[0179]

[0180] In particular, considering that the prediction features and marker features of the model obtained during different training periods are relatively stable for clean samples but unstable for noise samples, they can be used to distinguish between clean and noise samples. The disclosed solution determines the fluctuation scores of defined training samples based on the model's prediction features during each training cycle. It then detects the most suspicious noise samples in that training cycle and excludes them from model training, thereby obtaining an appropriate training database. Furthermore, the fluctuation scores of training samples can be dynamically updated as the training cycles progress, and the detected noise samples can be dynamically discarded during each training cycle accordingly, thereby gradually optimizing the training database.

[0181] Moreover, preferably, in the scheme of the present invention, the optimization of training samples in each training cycle is performed based on the initial training sample library, so that all training samples in the training sample library can be fully utilized. Even if a training sample is judged to be a noise sample in a certain training or a certain training cycle, as long as the training sample has recognition ability, the training sample may still be used as a clean sample for subsequent model training, thereby obtaining a network model with better generalization.

[0182] Furthermore, according to the disclosed solution, a more accurate and higher-performance network model can be trained using the optimized training database. Furthermore, as the training cycle progresses, the optimized training database can be dynamically utilized to train a more accurate network model, allowing the training model and noise detection to mutually enhance each other, gradually improving recognition capabilities as the number of training cycles increases. In particular, as the number of training cycles increases, the recognition capabilities of deep features will gradually increase, while the accuracy of noise detection will also improve. The two can learn from each other and gradually improve together.

[0183] In addition, the disclosed solution can be applied to existing training databases and combined with existing loss functions to improve the recognition performance of the training model. Preferably, the disclosed solution can also use the fluctuation score of the training samples to weight the existing loss function, thereby further optimizing the model training and improving the recognition performance of the training model.

[0184] The following will describe in detail exemplary implementations of the present disclosure with reference to the accompanying drawings. It should be noted that the following description is primarily intended to clearly illustrate the training operation process according to the present disclosure, and some steps or operations therein are not required, such as the pre-processing step and the feature extraction step. The operations according to the present disclosure can be performed directly based on the received features.

[0185] The object recognition model training process according to the first embodiment of the present disclosure will be described below. In the first embodiment, for a large-scale training database containing noisy samples, dynamic training sample optimization with a fixed discard ratio is performed, and the object recognition model is trained based on this. Figure 11 A flowchart of training an object recognition model according to the first embodiment of the present disclosure is shown, wherein the object recognition model is, for example, a convolutional neural network model.

[0186] S1100: Initialize the drop mask and fixed drop ratio of all training samples

[0187] In this step, the input is an initial training database with noise. For example, the database can include any suitable images, including raw images or even pre-processed images. For example, the input raw images are converted into training data that meets the requirements of the convolutional neural network through a series of pre-processing operations. This series of pre-processing operations includes face or object detection, face or object alignment, image augmentation, image normalization, etc.

[0188] For all N training samples in the initial training database, the discard masks corresponding to all training samples are initialized to 1, indicating that all training samples are used to train the convolutional neural network model in the first training cycle, and the given discard ratio is a fixed value between (0, 1).

[0189] S1200: Train the current neural network model using the selected loss function based on the current training database

[0190] In this step, the input is the current network database with the current drop mask set. Specifically, in the first training cycle, the input is the initial training database with the initial drop mask set, while in subsequent training cycles, the input is the network database with the drop mask updated based on the volatility of the training samples. The current convolutional neural network is then trained using the selected convolutional neural network architecture and the selected loss function. The convolutional neural network architecture can be a commonly used network architecture such as VGG16, ResNet, SENet, etc. The loss function can be a proposed softmax-based loss function such as NSoftmax, CosFace, ArcFace, etc.

[0191] S1300: Calculate and record the similarity between the model prediction and the label identification of each sample in the current training database based on the current network model

[0192] In this step, for each sample in the current training database, the corresponding embedding features and the final fully connected layer of the current training convolutional neural network are extracted from the current training convolutional neural network model. Then, the cosine similarity corresponding to the minimum angle between the embedding features extracted from the training sample and the weight vectors representing all categories in the final fully connected layer of the neural network model is used as the similarity of its model prediction; at the same time, the cosine similarity corresponding to the angle between the embedding features extracted from the training sample and the weight vectors representing the true labeled category in the final fully connected layer of the neural network model is used as the similarity of its label identification; finally, the model prediction similarity and label identification similarity of all samples in the current training database are recorded. As an example, these similarities can be recorded in a specific storage device, such as an external memory, an internal cache, etc.

[0193] S1400: Calculate the fluctuation scores of all training samples based on the defined fluctuation score function and the sample similarity of the recorded current training database

[0194] In this step, the input is the model prediction similarity and label identification similarity of all samples in the current training database recorded in step S1300. For training samples in the current training database, such as training samples in the initial training database that can participate in the current model training, the difference score corresponding to the training sample is first calculated based on the recorded model prediction similarity and label identification similarity of the training sample, and then the fluctuation score of the training sample is cumulatively calculated using a defined fluctuation score function, such as the cumulative value, cumulative average, or any other appropriate mathematical statistical value of the fluctuation scores of the training sample in all training cycles from the initial training cycle to the current training cycle; for training samples not in the current training database, such as training samples in the initial training database that do not participate in the current model training, the average of the difference scores corresponding to all these training samples is first calculated based on the recorded model prediction similarity and label identification similarity of all training samples in the current training database, and then the fluctuation score of the training sample is cumulatively calculated using a defined fluctuation score function, such as the cumulative value, cumulative average, or any other appropriate mathematical statistical value of the fluctuation score described above.

[0195] S1500: Update the dropout mask of all training samples according to the calculated fluctuation score and the initialized fixed dropout ratio

[0196] In this step, the input is the fluctuation scores of all training samples calculated in step S1400. First, all training samples are sorted in descending order according to their fluctuation scores. The larger the fluctuation score of a training sample, the more unstable the prediction and true label of the training sample. Then, based on the initialized fixed drop ratio α, the number of discardable training samples M = N * α is calculated. Finally, the drop mask of the first M training samples is set to 0, and the drop mask of the last NM training samples is set to 1. The specific drop mask function calculation formula is defined as follows:

[0197]

[0198] Among them, F(X i ) is the training sample X i The fluctuation score, α is a fixed drop ratio, and N is the number of all training samples.

[0199] It should be noted that this setting of the discard mask based on a fixed discard ratio can also be equivalent to setting the discard mask based on a specific threshold, and accordingly, the specific threshold can correspond to the N*αth fluctuation score when the fluctuation scores are arranged in descending order, so that the discard mask of the training samples with volatility less than the specific threshold is set to 1, and the discard mask of the training samples with volatility greater than or equal to the specific threshold is set to 0.

[0200] S1600: Generate a new network database based on the discard masks of all updated training samples

[0201] In this step, the input is the discarding mask of all training samples calculated in step S1500. For training samples with a discarding mask greater than 0, they are added to the new network database; for training samples with a discarding mask of 0, it indicates that they are detected noise samples and they are not added to the new network database.

[0202] S1700: Determine whether to end the training process

[0203] In this step, it is possible to determine whether to end the training by using certain pre-set thresholds, such as whether the current loss is less than a given threshold, or whether the current number of training iteration cycles reaches a given maximum number of training cycles, and so on. If the conditions are met, the network model training is terminated; otherwise, the next training cycle is continued, that is, the generated new network database is used as the current training database to execute the aforementioned steps S1200-S1600. It should be noted that, although not shown, when iteratively executing the next training cycle, the parameters of the convolutional neural network are often updated by backpropagation based on the calculated loss data, and the updated training database is used to perform model training, that is, the updated training database and the convolutional neural network are used to perform the aforementioned S1200-S1600 operations.

[0204] In addition, it should be pointed out that step S1700 can be located at other positions in the training process, for example, it can be located at any position after step S1200, or even before steps S1300-S1600, so that if the end condition is met, the network model training can be ended directly without performing the fluctuation score calculation, discarding mask setting and model training database update in the current training cycle. On the contrary, if the end condition is not met, the operations corresponding to steps S1300-S1600 are performed again.

[0205] S1800: Output the trained convolutional neural network model

[0206] In this step, we use the current parameters of all layers in the convolutional neural network structure as the trained network model, and we output the network model and the corresponding parameter information.

[0207] The solution of the first embodiment is particularly suitable for situations where a rough estimate of the noise ratio in the large-scale training database is available. Thus, for a large-scale training database containing noisy samples, a fixed proportion of noise samples can be discarded based on the sample volatility to obtain an optimized large-scale training database, which can then be used to train a more stable network model. Preferably, the current training library used in each training cycle can be generated based on the initial training library, thereby maintaining a more diverse training sample base for training, further improving the recognition capability of the resulting training model.

[0208] The following describes the object recognition model training process according to the second embodiment of the present disclosure. In the second embodiment, for a large-scale training database containing noisy samples / low-quality samples, a fixed discard threshold is used to perform dynamic training sample optimization, and the object recognition model is trained based on this. Figure 12A flowchart of performing object recognition model training according to a second embodiment of the present disclosure is shown, wherein the object recognition model is, for example, a convolutional neural network model.

[0209] S2100: Initialize the discard mask and fixed discard threshold of all training samples

[0210] In this step, the input is an original training database containing noise samples, and the original training database can be the same as that in the aforementioned step S1100.

[0211] For all N training samples in the initial training database, the discard masks corresponding to all training samples are initialized to 1, indicating that all training samples are used to train the convolutional neural network model in the first training cycle, and a fixed discard threshold is set. The discard threshold can be set based on experience or based on the statistical value of the discard threshold used in historical training, such as the average value.

[0212] Then steps S2200 to S2400 are executed. The operations of these steps can be performed in the same manner as the aforementioned steps S1200 to S1400. Please refer to the above description of steps S1200 to S1400, which will not be described in detail here.

[0213] S2500: Update the discard mask of all training samples according to the calculated fluctuation score and the set fixed discard threshold

[0214] In this step, the input is the calculated fluctuation scores of all training samples, and the fluctuation scores of all training samples are compared with the initialized fixed discard threshold D thr For comparison, if the fluctuation score of a training sample is less than the threshold, it means that the prediction and the true label of the training sample remain stably consistent, so it is a clean sample, and its discard mask is set to 1; if the fluctuation score of a training sample is greater than or equal to the threshold, it means that the prediction and the true label of the training sample are unstable, so it is judged to be a noise sample, and its discard mask is set to 0. The specific discard mask calculation formula can be as shown in the above formula (6).

[0215] Then steps S2600 to S2800 will be executed. The operations of these steps can be performed in the same manner as the aforementioned steps S1600 to S1800. Please refer to the above description of steps S1600 to S1800, which will not be described in detail here.

[0216] The solution of the second embodiment dynamically optimizes the samples and size of the training database, dynamically discarding noisy samples and samples of low quality, rather than using only a fixed-size training database for model training. Furthermore, preferably, the current training database used in each training cycle can be dynamically generated based on the initial training database. This allows the training database to retain low-quality samples with discriminative capabilities, thereby training a network model with better generalization.

[0217] The following describes the object recognition model training process according to the third embodiment of the present disclosure. In the third embodiment, for a large-scale training database containing noisy samples / low-quality samples, a weighted discard mask is used to perform dynamic training sample optimization, and the object recognition model is trained based on this. Figure 13 A flowchart of performing object recognition model training according to the third embodiment is shown, wherein the object recognition model is, for example, a convolutional neural network model.

[0218] First, step S3100 may be performed as in the aforementioned step S2100. Please refer to the aforementioned description of step S2100, which will not be described in detail here.

[0219] Then, in step S3200: the current neural network model is trained using the selected loss function according to the current training database.

[0220] This step is basically the same as S1200, please refer to the introduction of S1200. It should be noted that in this embodiment, the training samples in the new network database all have their corresponding weight masks, which represent the credibility of the training samples. Therefore, when training the network model, the loss calculated by the training sample is further weighted using its weight mask, that is, the influence of the training sample on the model training is dynamically adjusted according to the weight, thereby dynamically adjusting the influence of the training samples in the training database on the model training. As an example, the weight mask of a training sample can be represented by the weighted discard mask of the training sample.

[0221] In other words, this is equivalent to the loss function selected in the model training being a weighted loss function, that is, the corresponding loss function of the training sample can be a loss function weighted by the weighted dropout mask of the training sample, such as the form of the aforementioned formula (8), where M(X i ) corresponds to the weighted dropout mask, i.e., the weight mask corresponding to the sample.

[0222] Then steps S3300 to S3400 will be executed. The operations of these steps can be performed in the same manner as the aforementioned steps S1300 to S1400. Please refer to the above description of steps S1300 to S1400, which will not be described in detail here.

[0223] S3500: Update the discard mask of all training samples based on the calculated fluctuation score and the initialized fixed discard threshold

[0224] In this step, the input is the fluctuation scores of all training samples calculated in step S3400, and the fluctuation scores of all training samples are compared with the initialized fixed discard threshold D thr For comparison, if the fluctuation score of a training sample is less than a given threshold, it means that the prediction and the true label of the training sample remain stable and consistent, so it is a clean sample, and its discard mask is calculated according to the defined weighted discard mask function; if the fluctuation score of a training sample is greater than or equal to the given threshold, it means that the prediction and the true label of the training sample are unstable, so it is judged to be a noise sample, and its discard mask is set to 0. The specific calculation formula of the weighted discard mask function can be as shown in the above formula (9). As mentioned above, the weighted discard mask obtained in this way can be used as a weight to weight the loss function.

[0225] Then steps S3600 to S3800 will be executed. The operations of these steps can be performed in the same manner as the aforementioned steps S1600 to S1800. Please refer to the above description of steps S1600 to S1800, which will not be described in detail here.

[0226] The third embodiment can dynamically discard noisy samples and samples of excessively low quality, while simultaneously utilizing the credibility of training samples in the current network database to effectively guide neural network training. Specifically, the credibility of training samples is set based on their volatility, and the impact of training samples on model training is dynamically adjusted accordingly, thereby accelerating the convergence of the network model. Furthermore, as previously mentioned, the current training library used in each training cycle can preferably be dynamically generated based on the initial training library. This allows the training database to retain low-quality samples with discriminative capabilities, thereby enabling the training of a network model with better generalization.

[0227] The following describes the object recognition model training process according to the fourth embodiment of the present disclosure. In the fourth embodiment, for a large-scale training database containing noisy samples / low-quality samples, a weighted drop mask is used to perform dynamic training sample optimization, and the object recognition model is trained based on this. In particular, the weighted drop mask here further takes into account the number of training cycles performed, thereby enabling dynamic training sample optimization based on the number of training cycles performed. This optimization can also be referred to as dynamic training sample optimization based on curriculum learning. Figure 14 A flowchart of performing object recognition model training according to a fourth embodiment is shown, wherein the object recognition model is, for example, a convolutional neural network model.

[0228] S4100: Initialize the discard mask of all training samples, fix the discard threshold and fix the training cycle threshold

[0229] In this step, the input is an original training database containing noise samples, and the original training database can be the same as that in the aforementioned step S1100.

[0230] In this step, the input is the original network database containing noise samples. According to the number of training samples N in the original network database, the discarding masks corresponding to all training samples are initialized to 1, indicating that all training samples are used to train the convolutional neural network model in the first round of training cycle. In addition, a fixed discarding threshold can be set as described above, and a fixed training cycle threshold is also set. The fixed training cycle threshold can be set based on experience or based on the statistical value of the number of training cycles in historical training, for example, it is set based on the number of training cycles at the end of training.

[0231] Then steps S4200 to S4400 will be executed. The operations of these steps can be performed in the same manner as the aforementioned steps S3300 to S3400. Please refer to the above description of steps S3300 to S3400, which will not be described in detail here.

[0232] S4500: Update the drop mask of all training samples according to the calculated fluctuation score and the defined weighted drop mask function based on curriculum learning

[0233] In this step, the input is the fluctuation score of all training samples calculated in step S4400. The fluctuation scores of all training samples are compared with the initialized fixed discard threshold. If the fluctuation score of the training sample is less than the threshold, it means that the prediction and the true label of the training sample remain stably consistent, so it is a clean sample, and its discard mask will be further calculated according to the weighted discard mask function based on curriculum learning; if the fluctuation score of the training sample is greater than or equal to the threshold, it means that the prediction and the true label of the training sample are unstable, so it is judged to be a noise sample, and its discard mask is set to 0. The specific calculation formula of the weighted discard mask function based on curriculum learning can be as shown in the above formula (10). As mentioned above, the weighted discard mask obtained in this way can be used as a weight to weight the loss function.

[0234] Here, the weighting of training examples can be further optimized by considering the number of training cycles. In particular, the fewer training cycles a training example undergoes when its volatility score is below a certain threshold, that is, the shorter the training cycle / time for its volatility to stabilize, the more likely the training example is to guide the network for accurate training. Therefore, the weight of this training example can be appropriately increased.

[0235] Then steps S4600 to S4800 will be executed. The operations of these steps can be performed in the same way as the aforementioned steps S3600 to S3800. Please refer to the previous description of steps S3600 to S3800, and they will not be described in detail here.

[0236] The fourth embodiment can dynamically discard noisy samples and samples of excessively low quality, while simultaneously utilizing the credibility of training samples in the current network database to effectively guide neural network training. Specifically, the credibility of training samples is set based on their volatility. In particular, the number of training cycles is considered, and the impact of training samples on model training is dynamically adjusted accordingly, thereby accelerating the convergence of the network model. Furthermore, as previously mentioned, preferably, the current training library used in each training cycle can be dynamically generated based on the initial training library. This allows low-quality samples with discriminative capabilities to be retained in the training database, thereby enabling the training of a network model with better generalization.

[0237] Figure 15 is a block diagram illustrating an exemplary hardware configuration of a computer system 1000 in which embodiments of the present invention may be implemented.

[0238] like Figure 15 As shown, the computer system includes a computer 1110. The computer 1110 includes a processing unit 1120, a system memory 1130, a non-removable non-volatile memory interface 1140, a removable non-volatile memory interface 1150, a user input interface 1160, a network interface 1170, a video interface 1190, and an output peripheral interface 1195, which are connected via a system bus 1121.

[0239] The system memory 1130 includes a ROM (Read Only Memory) 1131 and a RAM (Random Access Memory) 1132. A BIOS (Basic Input Output System) 1133 resides in the ROM 1131. An operating system 1134, application programs 1135, other program modules 1136, and some program data 1137 reside in the RAM 1132.

[0240] A non-removable non-volatile memory 1141 (such as a hard disk) is connected to the non-removable non-volatile memory interface 1140. The non-removable non-volatile memory 1141 can store, for example, an operating system 1144, application programs 1145, other program modules 1146, and some program data 1147.

[0241] Removable nonvolatile memory such as a floppy disk drive 1151 and a CD-ROM drive 1155 are connected to the removable nonvolatile memory interface 1150. For example, a floppy disk 1152 can be inserted into the floppy disk drive 1151, and a CD (Compact Disc) 1156 can be inserted into the CD-ROM drive 1155.

[0242] Input devices such as a mouse 1161 and a keyboard 1162 are connected to the user input interface 1160 .

[0243] Computer 1110 can be connected to a remote computer 1180 via a network interface 1170. For example, network interface 1170 can be connected to remote computer 1180 via a local area network 1171. Alternatively, network interface 1170 can be connected to a modem (modulator-demodulator) 1172, and modem 1172 can be connected to remote computer 1180 via a wide area network 1173.

[0244] Remote computer 1180 may include storage 1181 , such as a hard disk, that stores remote application programs 1185 .

[0245] Video interface 1190 is connected to monitor 1191 .

[0246] The output peripheral interface 1195 is connected to a printer 1196 and a speaker 1197 .

[0247] Figure 15 The computer system shown is merely illustrative and is in no way intended to limit the invention, its application, or uses.

[0248] Figure 15 The illustrated computer system may be implemented for any embodiment as a stand-alone computer, or as a processing system in a device, where one or more unnecessary components may be removed or one or more additional components may be added.

[0249] The present invention can be used in many applications. For example, the present invention can be used to monitor, identify, track objects in static images or moving videos captured by a camera, and is particularly advantageous for portable devices equipped with cameras, (camera-based) mobile phones, etc.

[0250] It should be noted that the methods and apparatus described herein may be implemented as software, firmware, hardware, or any combination thereof. Some components may, for example, be implemented as software running on a digital signal processor or microprocessor. Other components may, for example, be implemented as hardware and / or an application-specific integrated circuit.

[0251] In addition, the method and system of the present invention can be implemented in a variety of ways. For example, the method and system of the present invention can be implemented by software, hardware, firmware or any combination thereof. The order of the steps of the method described above is merely illustrative, and unless otherwise specifically stated, the steps of the method of the present invention are not limited to the order specifically described above. In addition, in some embodiments, the present invention can also be embodied as a program recorded in a recording medium, including machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also encompasses a recording medium storing a program for implementing the method according to the present invention.

[0252] Those skilled in the art will appreciate that the boundaries between the above-mentioned operations are merely illustrative. Multiple operations can be combined into a single operation, a single operation can be distributed among additional operations, and operations can be performed at least partially overlapping in time. Moreover, alternative embodiments can include multiple instances of specific operations, and the order of operations can be changed in various other embodiments. However, other modifications, variations, and replacements are also possible. Therefore, this specification and the accompanying drawings should be considered illustrative, not restrictive.

[0253] In addition, embodiments of the present disclosure may further include the following illustrative examples (EE).

[0254] EE 1. A training sample optimization device for a neural network model for object recognition, characterized by comprising: for each training sample in a training sample library,

[0255] a volatility determination unit configured to determine volatility of a model prediction of the training sample relative to a corresponding label identification of the training sample when training the neural network model;

[0256] The optimization unit is configured to determine whether the training sample can be used for training the neural network model in the next training cycle based on the volatility of the training sample.

[0257] EE 2. The device according to EE 1 is characterized in that, for training samples in the training sample library that can be used for neural network model training in the current training cycle, the volatility of the training sample is determined based on the mathematical statistical value of the difference between the model prediction similarity and the label identification similarity of the training sample.

[0258] EE 3. The device according to EE 1, characterized in that

[0259] The model prediction similarity of the training sample indicates the minimum offset between the predicted features of the training sample and the weight vector representing all categories in the last fully connected layer of the neural network model, and / or

[0260] The label identity similarity of the training sample indicates the offset between the predicted features of the training sample and the weight vector representing the true label category in the last fully connected layer of the neural network model.

[0261] EE 4. The device according to EE 1, characterized in that

[0262] The similarity of the model predictions for a training sample is determined based on the minimum angle between the predicted features of the training sample and the weight vector representing all classes in the last fully connected layer of the neural network model, and / or

[0263] The label identity similarity of the training samples is determined based on the angle between the predicted features of the training samples and the weight vector representing the true label category in the last fully connected layer of the neural network model.

[0264] EE 5. The device according to EE 1, characterized in that

[0265] The volatility of a training sample corresponds to a mathematical statistical value of the accumulated values ​​of the volatility of the training sample in the current training and a specific number of previous training cycles.

[0266] EE 6. The device according to EE 1 is characterized in that, for training samples in the training sample library that cannot be used for neural network model training in the current training cycle, the volatility of the training sample corresponds to the mathematical statistical value of the volatility of a specific number of training samples in the training sample library that can be used for neural network model training.

[0267] EE 7. The device according to EE 1, characterized in that when the volatility of a training sample is less than a specific threshold, the training sample can be used for training the neural network model in the next training cycle.

[0268] EE 8. The device according to EE 1, characterized in that

[0269] The loss function of the training sample when the neural network model is trained using the training sample library is weighted based on the volatility of the training sample.

[0270] EE 9. The device according to EE 8, characterized in that

[0271] The weighting is determined based on the comparison of volatility to a certain threshold.

[0272] EE 10. The device according to EE 8 or 9, characterized in that

[0273] The weighting weight is further determined based on a ratio between the volatility of the training sample and the maximum value of the volatility of all training samples.

[0274] EE 11. The device according to any one of EE 8-10, characterized in that:

[0275] The weighted weights are further determined based on the number of training cycles up to the current training.

[0276] EE 12. The device according to EE 7 or 9, wherein the specific threshold is a preset fixed threshold, and / or

[0277] The specific threshold is a dynamic threshold determined based on the volatility of training samples.

[0278] EE 13. The device according to EE 12, characterized in that

[0279] The dynamic threshold is determined based on mathematical statistics of fluctuations in previous training, and

[0280] The dynamic threshold is determined based on the proportion of training samples in the training sample library that can be used for model training and the volatility of each training sample in the training sample library.

[0281] EE 14. The device according to EE 1, characterized in that the optimization unit is configured to optimize the training sample library when the trained neural network model does not meet the training end condition.

[0282] EE 15. The device according to EE 14, wherein the training end condition is at least one of the following:

[0283] The number of training cycles reaches a predefined maximum number of training cycles; and

[0284] The calculated loss of the neural network model is below a predefined threshold.

[0285] EE 16. A neural network model training device for object recognition, characterized in that:

[0286] A training sample optimization unit, configured to optimize the training sample library for neural network model training using the training sample optimization device according to EE 1 for a given training sample library;

[0287] The training unit is configured to use the optimized training sample library to train the neural network model.

[0288] EE 17. The device according to EE 16, further comprising:

[0289] The end determination unit is configured to determine whether the trained neural network model meets the training end condition,

[0290] If it is determined that the trained neural network model does not meet the training end condition, the training sample optimization unit and the training unit are used to further perform optimization and training using the optimized training sample library.

[0291] EE 18. The apparatus according to EE 16, wherein the training end condition is at least one of the following:

[0292] The number of training cycles reaches a predefined maximum number of training cycles; and

[0293] The calculated loss of the neural network model is below a predefined threshold.

[0294] EE 19. A method for optimizing training samples of a neural network model for object recognition, characterized in that the method comprises: for each training sample in a training sample library,

[0295] a volatility determination step for determining the volatility of the model prediction of the training sample relative to the corresponding label identification of the training sample when training the neural network model;

[0296] The optimization step is used to determine whether the training sample can be used for training the neural network model in the next training cycle based on the volatility of the training sample.

[0297] EE 20. The method according to EE 19, wherein the optimization step further comprises determining that the training sample can be used for training the neural network model in the next training cycle when the volatility of the training sample is less than a specific threshold.

[0298] EE 21. The method according to EE 19, further comprising:

[0299] The loss function of the training sample when the neural network model is trained using the training sample library is weighted based on the volatility of the training sample.

[0300] EE 22. The method according to EE 19, wherein the optimization step further comprises optimizing the training sample library when the trained neural network model does not meet the training end condition.

[0301] EE 23. A neural network model training method for object recognition, characterized in that:

[0302] A training sample optimization step for updating the training sample library used for neural network model training using the training sample optimization method according to EE 19 for a given training sample library;

[0303] The training step is used to train the neural network model using the optimized training sample library.

[0304] EE 24. The method according to EE 23, further comprising:

[0305] The end determination step is used to determine whether the trained neural network model meets the training end conditions.

[0306] If it is determined that the trained neural network model does not meet the training end condition, further optimization and training are performed using the updated training sample library through the training sample updating step and the training step.

[0307] EE 25. A device comprising

[0308] at least one processor; and

[0309] At least one storage device storing thereon instructions which, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of EEs 19 to 24.

[0310] EE 26. A storage medium storing instructions, which, when executed by a processor, enable execution of the method according to any one of EEs 19 to 24.

[0311] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0312] Although some specific embodiments of the present disclosure have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. The various embodiments disclosed herein may be combined in any manner without departing from the spirit and scope of the present disclosure. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present disclosure.

Claims

1. A training sample optimization device for a neural network model for object recognition, characterized in that: include: For each training sample in the training sample library, a volatility determination unit configured to determine volatility of a model prediction of the training sample relative to a corresponding label identification of the training sample when training the neural network model; as well as an optimization unit configured to determine whether the training sample can be used for subsequent training of the neural network model based on the volatility of the training sample, For a training sample in the training sample library that can be used for training the neural network model in the current training cycle, the volatility of the training sample is determined based on a mathematical statistic of the difference between the model prediction similarity and the label identification similarity of the training sample. Among them, the model prediction similarity of the training sample indicates the minimum value of the offset between the predicted features of the training sample and the weight vector representing all categories in the last fully connected layer of the neural network model, and / or the label identification similarity of the training sample indicates the offset between the predicted features of the training sample and the weight vector representing the true label category in the last fully connected layer of the neural network model.

2. The device according to claim 1, wherein The similarity of the model predictions for a training sample is determined based on the minimum angle between the predicted features of the training sample and the weight vector representing all classes in the last fully connected layer of the neural network model, and / or The label identity similarity of the training samples is determined based on the angle between the predicted features of the training samples and the weight vector representing the true label category in the last fully connected layer of the neural network model.

3. The device according to claim 1, characterized in that The volatility of a training sample corresponds to a mathematical statistical value of the accumulated values ​​of the volatility of the training sample in the current training and a specific number of previous training cycles.

4. The device according to claim 1, characterized in that For a training sample in the training sample library that cannot be used for neural network model training in the current training cycle, the volatility of the training sample corresponds to the mathematical statistical value of the volatility of a specific number of training samples in the training sample library that can be used for neural network model training.

5. The device according to claim 1, characterized in that When the volatility of a training sample is less than a specific threshold, the training sample can be used for subsequent training of the neural network model.

6. The device according to claim 1, characterized in that When the neural network model is trained by using the training sample library, the loss function of the training sample is weighted based on the volatility of the training sample.

7. The device according to claim 6, characterized in that The weighting is determined based on the comparison of volatility to a certain threshold.

8. The device according to claim 6, characterized in that The weighting weight is determined based on the ratio of the volatility of the training sample to the maximum value of the volatility of all training samples.

9. The device according to any one of claims 6 to 8, characterized in that The weighted weights are further determined based on the number of training cycles up to the current training.

10. The device according to claim 5 or 7, characterized in that The specific threshold is a preset fixed threshold, and / or The specific threshold is a dynamic threshold determined based on the volatility of training samples.

11. The device according to claim 10, characterized in that The dynamic threshold is determined based on mathematical statistics of fluctuations in previous training, and / or The dynamic threshold is determined based on the proportion of training samples in the training sample library that can be used for model training and the volatility of each training sample in the training sample library.

12. The device according to claim 1, characterized in that The optimization unit is configured to optimize the training sample library when the trained neural network model does not meet the training end condition.

13. The device according to claim 12, characterized in that The training end condition is at least one of the following: The number of training cycles reaches the predefined maximum number of training cycles; and The calculated loss of the neural network model is below a predefined threshold.

14. A training device for a neural network model for object recognition, characterized in that: A training sample optimization unit, configured to optimize the training samples used for neural network model training for a given training sample library by using the training sample optimization device according to any one of claims 1 to 13; as well as The training unit is configured to use the optimized training samples to train the neural network model.

15. The device according to claim 14, characterized in that Further including: The end determination unit is configured to determine whether the trained neural network model meets the training end condition, If it is determined that the trained neural network model does not meet the training end condition, the training sample optimization unit and the training unit are used to further perform optimization and training using the optimized training sample library.

16. The device according to claim 15, characterized in that The training end condition is at least one of the following: The number of training cycles reaches the predefined maximum number of training cycles; and The calculated loss of the neural network model is below a predefined threshold.

17. A method for optimizing training samples of a neural network model for object recognition, characterized in that: The method comprises: for each training sample in the training sample library, a volatility determining step for determining the volatility of the model prediction of the training sample relative to the corresponding labeled identification of the training sample when training the neural network model; and an optimization step for determining whether the training sample can be used for subsequent training of the neural network model based on the volatility of the training sample, For a training sample in the training sample library that can be used for training the neural network model in the current training cycle, the volatility of the training sample is determined based on a mathematical statistic of the difference between the model prediction similarity and the label identification similarity of the training sample. Among them, the model prediction similarity of the training sample indicates the minimum value of the offset between the predicted features of the training sample and the weight vector representing all categories in the last fully connected layer of the neural network model, and / or the label identification similarity of the training sample indicates the offset between the predicted features of the training sample and the weight vector representing the true label category in the last fully connected layer of the neural network model.

18. A method for training a neural network model for object recognition, characterized in that: The method comprises: A training sample optimization step for optimizing the training samples used for neural network model training by using the training sample optimization method according to claim 17 for a given training sample library; The training step is used to train the neural network model using the optimized training samples.

19. A device comprising at least one processor; and At least one storage device storing thereon instructions which, when executed by the at least one processor, cause the at least one processor to perform the method according to claim 17 or 18.

20. A storage medium storing instructions which, when executed by a processor, enable the method according to claim 17 or 18 to be performed.

Images