Classification model training method, object identification method and related device

Abstract

The invention provides a classification model training method, an object recognition method and a related device. The classification model training method comprises the following steps: performing feature extraction on training images in a training image set through a feature extraction module to obtain image features of the training images and uncertainty of the image features; processing the image features based on the uncertainty through the classification module to obtain a classification result of the training image; fusing the classification loss of the training images in the training image set to obtain a first loss of the classification model; and training the classification model based on the first loss. According to the method, the influence of noise and very difficult samples on model training overfitting is reduced, so that the classification efficiency of the trained classification model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a classification model training method, an object recognition method and related devices. BACKGROUND

[0002] Classification models are widely used in computer vision, natural language processing, medical diagnosis, financial risk control, recommendation systems, industrial manufacturing, bioinformatics and other fields, and have various application scenarios. However, the current classification model training method is prone to training overfitting, which leads to relatively low classification efficiency of the trained classification model. SUMMARY

[0003] The present application provides a classification model training method, an object recognition method and related devices, which can reduce the influence of noise and very difficult samples on model training overfitting, thereby improving the classification efficiency of the trained classification model.

[0004] To achieve the above purpose, the present application provides a classification model training method, which comprises:

[0005] The feature extraction module extracts features from the training images in the training image set to obtain image features of the training images and uncertainties of the image features;

[0006] The classification module processes the image features based on the uncertainties to obtain classification results of the training images;

[0007] The classification losses of the training images in the training image set are fused to obtain a first loss of the classification model, the classification loss of the training image is obtained by processing the classification result of the training image based on a scaling factor of the training image, and the scaling factor of the training image is calculated based on the uncertainty of the training image; the range of the uncertainty is divided into a first range, a second range and a third range in order from small to large, and the scaling factor generated by the uncertainty in the second range is higher than the scaling factor generated by the uncertainty in the first range and the third range;

[0008] Based on the first loss, the classification model is trained.

[0009] To achieve the above purpose, the present application provides an object recognition method, which comprises:

[0010] Obtaining a to-be-recognized image;

[0011] Using the classification model trained by the above training method to recognize the object in the to-be-recognized image to obtain a recognition result.

[0012] To achieve the above object, the present application provides an electronic device, which comprises a memory and a processor; the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the above method.

[0013] To achieve the above object, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0014] The method of the present application is as follows: a feature extraction module is used to extract features of training images in a training image set, to obtain image features of the training images and uncertainty of the image features; a classification module is used to process the image features based on the uncertainty, to obtain classification results of the training images; classification losses of the training images in the training image set are fused to obtain a first loss of a classification model, the classification loss of the training image is obtained by processing the classification result of the training image through a scaling factor of the training image, and the scaling factor of the training image is calculated based on the uncertainty of the training image; and the classification model is trained based on the first loss. The uncertainty can exactly represent the difficulty of an image sample. For simple samples, the uncertainty is usually small. For some ambiguous, occluded and large-angle samples, the uncertainty is relatively large, that is, the difficulty of the image sample is usually positively correlated with the uncertainty. By setting the scaling factor to increase first and then decrease with the change of the uncertainty, the scaling factor of the semi-difficult sample is set to be higher than that of the simple sample and the difficult sample, so as to focus on the optimization of the semi-difficult sample, and the influence of noise in the training image set on the training can be effectively reduced. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the following embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 is a schematic diagram of loss changes of various samples;

[0017] Figure 2 is a schematic diagram of the first embodiment of the classification model training method of the present application;

[0018] Figure 3 is a schematic diagram of deterministic embedding representation;

[0019] Figure 4 is a related diagram of the uncertainty module in the classification model training method of the present application, wherein (a) is a schematic diagram of uncertainty embedding representation; and (b) is a schematic diagram of uncertainty learning;

[0020] Figure 5 is a schematic diagram of the relationship between uncertainty and scaling factor in the present application;

[0021] Figure 6 is a schematic diagram of uncertainty module learning in the classification model training method of the present application;

[0022] Figure 7 is a flowchart of the object recognition method of the present application;

[0023] Figure 8 is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0024] Figure 9 is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. DETAILED DESCRIPTION

[0025] In order for those skilled in the art to better understand the technical solutions of the present application, the classification model training method, the object recognition method and the related devices provided by the present application are further described in detail below in combination with the drawings and specific embodiments.

[0026] As shown in the four small graphs in Figure 1 , respectively represent the loss changes of simple samples, semi-difficult samples, difficult samples and noise samples in different iteration processes. It can be seen that at the beginning of training, the model optimizes all samples without distinction, but in the later stage of training, the loss values of difficult samples and noise samples in training dominate, which means that the model focuses on the optimization of these two types of samples. However, this is not what is expected in training, because optimizing the loss of noise samples means that the model is optimizing in a worse direction, and if difficult samples are over-optimized, it is also likely to cause overfitting of the model.

[0027] Therefore, the present application focuses on optimizing semi-difficult samples, reducing the influence of noise in the training set on training, and avoiding over-optimization of difficult samples, thereby improving the training efficiency of the model.

[0028] Specifically, the application provides a classification model training method. The classification model training method performs feature extraction on training images in a training image set through a feature extraction module to obtain image features of the training images and uncertainty of the image features; processes the image features based on the uncertainty through a classification module to obtain classification results of the training images; fuses classification losses of the training images in the training image set to obtain a first loss of the classification model, the classification loss of the training image being obtained by processing the classification result of the training image through a scaling factor of the training image, the scaling factor of the training image being calculated based on the uncertainty of the training image; and trains the classification model based on the first loss. The uncertainty can exactly represent the difficulty of an image sample. For a simple sample, the uncertainty is usually small. For some samples that are blurred, occluded or at a large angle, the uncertainty is relatively large, that is, the difficulty of the image sample is usually positively correlated with the uncertainty. The scaling factor of the semi-difficult sample is set to be higher than that of the simple sample and the difficult sample by setting the scaling factor to increase first and then decrease with the change of the uncertainty, so as to focus on the optimization of the semi-difficult sample, and the influence of noise in the training image set on the training can be effectively reduced.

[0029] Please refer to Figure 2 , Figure 2 is a flowchart of a first embodiment of the classification model training method of the application. Optionally, the classification model of the embodiment comprises a feature extraction module and a classification module. The classification model training method of the embodiment can comprise the following steps.

[0030] S101: performing feature extraction on training images in a training image set through a feature extraction module to obtain image features of the training images and uncertainty of the image features.

[0031] The feature extraction module can be used to perform feature extraction on training images in a training image set to obtain image features of the training images and uncertainty of the image features. In this way, the classification module can be used to process the image features based on the uncertainty to obtain classification results of the training images, and the classification model can be trained based on the classification results.

[0032] The network structure of the feature extraction module is not limited, as long as it has the function of feature extraction. For example, the feature extraction module can be a network structure such as AlexNet or ResNet.

[0033] As shown in Figure 3 , the traditional method maps the input image to a high-dimensional space, and the output feature vector is represented as a specific point in the high-dimensional space. The feature vector is determined, so it is called deterministic embedding. This embedding has certain limitations in some cases, for example, for two good quality pictures, Figure 3As shown in (a) in FIG. 10, the score obtained by the system comparison is 96, and it can be basically determined that the two persons are the same. However, when one or two pictures to be compared have poor quality, as shown in (b) in FIG. 10, the system comparison still gives a high score, and there is great uncertainty in this case, and there may be a high-score false positive. Figure 3 Based on this, as shown in (c) in FIG. 10, the present application sets an uncertainty module in the feature extraction module, to determine the uncertainty of the image feature through the uncertainty module, so that the subsequent image feature can be processed based on the uncertainty through the classification module to obtain the classification result of the training image, and the similarity of the image feature comparison is corrected through the uncertainty, thereby improving the accuracy of the classification. Figure 4

[0034] Optionally, the uncertainty module can estimate the variance of the image feature, to estimate the distribution range of the image feature through the variance, thereby adaptively learning the quality of the image and the degree of containing noise, so that the degree of uncertainty of the image feature can be represented through the variance, and the greater the variance, the greater the uncertainty.

[0035] In an implementation manner, the uncertainty of the image feature can be a vector with the same dimension as the image feature, for example, the image feature is a D-dimensional vector, and an additional uncertainty module is added to output the uncertainty (i.e., variance) of the image feature (embedding), and the uncertainty is also a D-dimensional vector, and each dimension represents the uncertainty of the embedding in the corresponding dimension.

[0036] This implementation manner is relatively inconvenient, because the storage amount is doubled compared to only outputting the image feature, for example, assuming that the image feature is 512-dimensional, then for one picture, an additional 512-dimensional vector is finally needed to store the uncertainty of the image feature; secondly, the calculation amount is also greatly increased, and for a classification processing system (such as an identification system) with a large amount of base library, this method needs to consume huge CPU or GPU computing power, and for some platforms with weak performance, the possibility of implementing this method is low. Based on this, the present application further proposes another implementation manner, which sets the uncertainty of the image feature as a single-dimensional vector, that is, outputs a one-dimensional vector through the uncertainty module to represent the uncertainty of the whole image feature, so as to greatly reduce the storage amount of the uncertainty and the calculation amount of the subsequent classification.

[0037] The uncertainty module can be a Monte Carlo Dropout or a Bayesian neural network, which is not limited here.

[0038] S102: Process the image feature based on the uncertainty through the classification module to obtain the classification result of the training image.

[0039] ​The image features of the training images and the uncertainty of the image features are obtained based on the above steps; the image features can be processed based on the uncertainty by a classification module to obtain a classification result of the training images.

[0040] In an implementation, the uncertainty of the image features is a vector with the same dimension as the image features, and the classification result of the training images can be determined by comparing the similarity of two images during the classification processing.

[0041] Suppose x is an image sample, where x i and x j are two images, z is a representation of the image sample in a mapping space, z i and z j are representations of the two images in the mapping space. Then the probability that the two images are of the same person is:

[0042] p(z i =z j )=∫p(z i |x i )p(z j |x j )δ(z i -z j )dz i dz j ;

[0043] After taking the logarithmic form, the similarity s(x i , x j ) of the two images can be represented by the mutual likelihood score:

[0044]

[0045] where const is a constant related to the feature dimension, represents the lth dimension of the feature of image i, represents the lth dimension of the feature of image j, represents the lth dimension of the uncertainty of the image feature of image i, represents the lth dimension of the uncertainty of the image feature of image j.

[0046] In another implementation, the uncertainty of the image features is a single-dimensional vector, and the similarity s(x i , x j ) of the two images can be:

[0047]

[0048] where σ i represents the uncertainty of the image feature of image i, and σ jan uncertainty of an image feature of the image j.

[0049] Compared with the previous implementation, the amount of calculation of the image similarity in the present implementation is greatly reduced, and the time consumption of the similarity calculation of two images is basically close to that when no uncertainty estimation is performed (e.g., only the cosine similarity calculation is performed), thereby without consuming large CPU or GPU computing power, and the possibility of implementing the scheme is increased.

[0050] In the above implementation, in the classification processing, the similarity of the image feature of the training image with each prototype of the class can be calculated, and the class with the highest similarity can be selected as the class of the training image. The prototype of each class can be the mean vector of all training images of the class in the embedding space.

[0051] Alternatively, in the classification processing, the feature vectors of all training images can be stored, the similarity of the current training image with each of the other training images can be calculated, and the top K samples with the highest similarity can be selected to determine the class by voting.

[0052] The classification result of the training image can be a vector composed of the similarity of the training image with all class prototypes.

[0053] The classification result of the training image can also be a vector composed of the confidence of the training image belonging to each class.

[0054] The classification result of the training image can also be the classification class of the determined training image and the confidence thereof.

[0055] The confidence of the training image belonging to a class can be determined by the similarity of the training image with the prototype or training sample of the class, for example, the confidence can be equal to the similarity of the training image with the prototype or training sample of the class, or the confidence can be obtained by activating the similarity of the training image with the prototype or training sample of the class through an activation function, or the confidence can be obtained by normalizing the similarity of the training image with the prototype or training sample of the class.

[0056] S103: Fuse the classification losses of the training images in the training image set to obtain a first loss of the classification model.

[0057] After the classification result of the training image is determined through the above steps, the classification loss of the training image can be calculated based on the classification result of the training image, and then the classification losses of the multiple training images in the training image set are fused to obtain a first loss of the classification model, so as to subsequently train the classification model based on the first loss of the classification model.

[0058] The classification loss of the training image is obtained by processing the classification result of the training image based on the scaling factor of the training image. Optionally, the single-sample loss of the training image can be calculated from the classification result of the training image, and then the single-sample loss of the training image can be scaled by the scaling factor of the training image to obtain the classification loss of the training image.

[0059] For example, the single-sample loss of the training image can be calculated by a cross-entropy loss function, and specifically, the negative logarithm of the probability of the training image being a correct class can be calculated. Then, the single-sample loss of the training image can be scaled by the scaling factor of the training image to obtain the classification loss of the training image.

[0060] For another example, the single-sample loss of the training image can be calculated by KL divergence. Then, the single-sample loss of the training image can be scaled by the scaling factor of the training image to obtain the classification loss of the training image.

[0061] The scaling factor of the training image is a factor that positively affects the classification loss of the training image. That is, when the scaling factor of the training image increases, the classification loss of the training image increases or remains unchanged; when the scaling factor of the training image decreases, the classification loss of the training image remains unchanged or decreases. Further, the classification loss of the training image can be positively correlated with the scaling factor of the training image.

[0062] The scaling factor of the training image can be determined based on the uncertainty of the training image. The range of the uncertainty can be divided into a first range (smallest), a second range (moderate), and a third range (largest) in order from small to large, wherein the scaling factor generated by the uncertainty in the second range is the largest, higher than the scaling factor generated by the uncertainty in the first range and the third range, that is, the relationship between the uncertainty and the scaling factor is non-monotonic. In view of the fact that the uncertainty can represent the difficulty of a sample, the present application amplifies the scaling factor of the uncertainty in the intermediate range, that is, increases the scaling factor of the semi-difficult sample, so as to focus on optimizing the semi-difficult sample. For example, the range of the uncertainty is 0-1, which can be divided into a first range (0-0.5], a second range (0.5-0.7), and a third range [0.7-1) in order from small to large, wherein the scaling factor corresponding to (0.5-0.7) can be 0.8, the scaling factor corresponding to (0-0.5] can be 0.3, and the scaling factor corresponding to [0.7-1) can be 0.5.

[0063] Further, as the uncertainty increases from small to large, the scaling factor first increases and then decreases, thus forming a unimodal curve, for example, a unimodal curve as shown in FIG. 1. Figure 5

[0064] In an implementation manner, the uncertainty can be substituted into the following formula to obtain the scaling factor of the training image:​ The scaling factor scale is calculated by such a unimodal function.

[0065] Wherein, A is the maximum scaling factor value, which determines the height of the curve.

[0066] μ is the uncertainty corresponding to the peak value of the scaling factor, at which the scaling factor reaches the maximum value.

[0067] σ scale Controls the width of the curve, affecting the rate of scaling factor from growth to decline. A, μ and σ scale above can be set according to actual conditions, and are not limited here.

[0068] In another implementation, the scaling factor can be calculated by substituting the uncertainty into a quadratic function.

[0069] Wherein, the step of fusing the classification loss of the training images in the training image set can be adding the classification loss of multiple training images in the training image set.

[0070] S104: training the classification model based on the first loss.

[0071] After calculating the first loss based on the above steps, the classification model can be trained based on the first loss.

[0072] The classification model can be trained and optimized using gradient descent method or adaptive learning rate algorithm and other optimization methods based on the first loss.

[0073] Optionally, considering that the introduction of uncertainty may cause the loss value to be unstable during training, the training of the classification model can be constrained by the second loss calculated based on the uncertainty. Specifically, the total loss can be calculated by the first loss and the second loss, and then the classification model is trained using the total loss. When training the classification model using the second loss, the second loss of multiple training images in the training image set can be fused, and the classification model is trained using the fused second loss. Wherein, the fusion of the second loss can refer to the addition of the second loss of multiple training images.

[0074] Wherein, the calculation formula of the total loss Loss total may be: Loss total = Loss cls + β * Loss reg ;

[0075] Wherein, Loss cls is the first loss; Loss regThe second loss is a second loss; β is a weighting coefficient of the second loss, which can be set according to actual conditions, and is not limited here.

[0076] The second loss can be monotonically decreasing in the range of the uncertainty, that is, the second loss is negatively correlated with the uncertainty. In this way, when the uncertainty is small, the second loss becomes larger, and the first loss is generally smaller; when the variance is large, the second loss becomes smaller, and the first loss becomes larger, so that the second loss and the first loss have a mutual constraint relationship by increasing the second loss, so as to facilitate the stability of model training.

[0077] The second loss can be a regular loss, that is, the first loss is constrained by adding a regular term. The second loss can be represented as follows:

[0078]

[0079] The change range of the uncertainty σ can be (0, 1), in which case the second loss is monotonically decreasing in the range of the uncertainty, and μ can be calculated by the image features.

[0080] In other embodiments, the second loss can also be calculated by a binary linear function or other formulas, which are not particularly limited here.

[0081] Of course, when training the classification model, the second loss can also be used to constrain the training of the classification model.

[0082] In the above training method, in each iteration process of the training sample, a random ∈ coefficient can be sampled, so that the new feature can be a Gaussian distribution with a mean of μ and a variance of σ 2 This method can better correct the features. However, this method also has certain limitations, which requires the model to be trained from scratch, and cannot fine-tune the existing model, which limits its use to some extent. Based on this, as Figure 6 shown, the uncertainty module outputs the correction amount of the feature mean along with the variance (i.e., the uncertainty), and the correction amount of the mean can be represented by Δμ, so as to eliminate the limitation that the model must be trained from scratch.

[0083] When the classification result of the training image is determined by comparing the similarity of two images, the similarity of the two images can be represented by the following formula:

[0084] When calculating the similarity between two images, the image features of each image can be corrected by adjusting the mean of the features of each image, resulting in corrected image features. The similarity between the two images is then calculated using the corrected image features and the uncertainty of the image features. Specifically, an uncertainty correction term can be added to the difference value to obtain the similarity between the two images. The difference value can be equal to the product of a first value and a preset ratio, where the preset ratio is equal to the ratio of the sum of the squared differences in the image features of the two images to the sum of the squared uncertainties. The first value can be set according to actual conditions, for example, it can be set to -1 / 2 or -1 / 3. In other embodiments, the similarity can also be corrected using a constant term; that is, the difference value can be added to the uncertainty correction term, and then the constant term can be subtracted to obtain the similarity between the two images.

[0085] In addition, when training the classification model, besides the first loss and / or the second loss, a third loss calculated from the similarity between two images can also be used to train the classification model, thereby improving the optimization efficiency of the classification model through the third loss.

[0086] Among them, the third loss Loss mls The calculation formula is as follows:

[0087]

[0088] Here, P represents the set of training images of the same category. The sum of similarities between all pairs of images in each category's training image set can be calculated, and the third loss is derived from the negative value of the sum of similarities.

[0089] In one implementation, the total loss can be calculated using the first loss, the second loss, and the third loss, and then the classification model can be trained using the total loss.

[0090] Of which, total loss total The formula for calculating Loss is: total =Loss cls +α*Loss mls +β*Loss reg ;

[0091] Among them, Loss cls The first loss; Loss reg The second loss; Loss mls The third loss is represented by α and β, which are the weighting coefficients for the third and second losses, respectively. These can be set according to the actual situation and are not restricted here.

[0092] This application also provides an object recognition method, such as Figure 7 As shown, the object recognition method may include the following steps.

[0093] S201: Obtain an image to be recognized.

[0094] S202: Recognize an object in the image to be recognized by using a classification model to obtain a recognition result.

[0095] The classification model can be obtained by the training method of the classification model in the above embodiments.

[0096] The object can be a person or an object, which is not particularly limited herein.

[0097] Please refer to Figure 8 , Figure 8 is a structural schematic diagram of an embodiment of an electronic device of the present application. The electronic device 10 includes a memory 11 and a processor 12 coupled to each other. The memory 11 is configured to store program instructions, and the processor 12 is configured to execute the program instructions to implement the method of any of the above embodiments.

[0098] The logical process of the above method is presented in a computer program. In terms of the computer program, if it is sold or used as a stand-alone software product, it can be stored in a computer storage medium. Therefore, the present application provides a computer readable storage medium. Please refer to Figure 9 , Figure 9 is a structural schematic diagram of an embodiment of a computer readable storage medium of the present application. The computer readable storage medium 20 stores a computer program 21. The computer program 21 is executed by a processor to implement the steps of the above video encoding method.

[0099] The computer readable storage medium 20 can be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or any medium that can store a computer program, or a server that stores the computer program. The server can send the stored computer program to other devices for running, or can run the stored computer program itself. The computer readable storage medium 20 can be a combination of multiple physical entities, such as multiple servers, servers plus memories, or memories plus mobile hard disks, etc.

[0100] The above is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A classification model training method, characterized in that, The classification model comprises a feature extraction module and a classification module, and the method comprises: feature extraction of training images in the training image set by the feature extraction module to obtain image features of the training images and uncertainty of the image features; processing of the image features based on the uncertainty by the classification module to obtain classification results of the training images; fusion of classification losses of the training images in the training image set to obtain a first loss of the classification model, the classification loss of the training image being obtained by processing the classification result of the training image based on a scaling factor of the training image, the scaling factor of the training image being calculated based on the uncertainty of the training image, the range of the uncertainty being divided into a first range, a second range and a third range in the order from small to large, the scaling factor generated by the uncertainty of the second range being higher than the scaling factor generated by the uncertainty of the first range and the third range; training of the classification model based on the first loss.

2. The training method of claim 1, wherein, The method comprises: calculation of a second loss of the classification model based on the uncertainty of the training image, the second loss being monotonically decreasing in the range of the uncertainty; the training of the classification model based on the first loss comprises: training of the classification model based on the first loss and the second loss.

3. The training method of claim 1, wherein, The feature extraction of training images in the training image set by the feature extraction module to obtain image features of the training images and uncertainty of the image features comprises: feature extraction of training images in the training image set by the feature extraction module to obtain image features of the training images, mean offset of the image features and uncertainty of the image features; the processing of the image features based on the uncertainty by the classification module to obtain classification results of the training images comprises: processing of the image features based on the uncertainty and the mean offset by the classification module to obtain classification results of the training images.

4. The training method of claim 3, wherein, The processing of the image features based on the uncertainty and the mean offset by the classification module to obtain classification results of the training images comprises: correction of the image features of each image by the mean correction amount of each image to obtain corrected image features of each image; calculation of the similarity of two images by using the corrected image features and the uncertainty of the image features of the two images; determination of the classification results of the training images based on the similarity data of the training images.

5. The training method of claim 4, wherein, The training of the classification model based on the first loss comprises: calculation of the total similarity of all combinations of two images in the training image set of each category; determination of a third loss based on the negative value of the total similarity; the training of the classification model based on the first loss comprises: training of the classification model based on the first loss and the third loss.

6. The training method of claim 1, wherein, The uncertainty is a single-dimensional vector.

7. The classification model training method according to claim 1, characterized in that, The scaling factor first increases and then decreases with the uncertainty from small to large.

8. An object recognition method characterized by, The method comprises: acquiring a to-be-identified image; using the classification model trained by the training method of any one of claims 1-7 to identify the object in the to-be-identified image to obtain an identification result.

9. An electronic device, comprising: The electronic device comprises a memory and a processor; the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method according to any one of claims 1-8.

Images