Model training method for image recognition, image recognition method, device and medium

Abstract

The present application relates to a model training method, an image recognition method, device and medium for image recognition, belonging to the field of computer technology. The method includes: training a preset first machine learning network based on a training data set to obtain a feature extraction model, which is used to extract image features from an input target image during image recognition to determine a template feature matching the image feature in a preset template feature library; for each of the N sample images corresponding to each category label in the training data set, training a preset second machine learning network based on the N sample images and the feature extraction model to obtain a quality evaluation model; the quality evaluation model is used to determine the quality score of the target image during image recognition, so as to determine the recognition result of the target image in combination with the quality score and the template feature; ensuring the accuracy of the output image recognition result and saving the human resources consumed when annotating the quality score.

Description

Technical Field

[0001] The present application relates to a model training method, an image recognition method, device and medium for image recognition, and belongs to the field of computer technology. Background Art

[0002] Image recognition technology is a technology that uses a computer to analyze an image to obtain effective recognition information therefrom. Image recognition technology is widely used in scenarios such as face recognition, vehicle recognition, license plate recognition, etc.

[0003] Traditional image recognition technology is implemented by training a neural network model. During the model training process, a large number of sample images with objects to be recognized are first collected, and the neural network model is trained based on the sample images and the target labels corresponding to the sample images to obtain an image recognition network. When performing image recognition, the target image is input into the image recognition network to obtain a target recognition result.

[0004] However, the images collected during the image recognition process are greatly affected by the environment. For example, on the one hand, the images will change due to factors such as illumination and background; on the other hand, the objects to be recognized in the images may be freely moving, and at this time, there will be problems such as large changes in illumination or pose of the objects in the images. In some extreme scenarios, blurred images may even be collected due to the movement of the objects to be recognized. However, inputting low-quality images into the image recognition network will reduce the accuracy of image recognition. Summary of the Invention

[0005] The present application provides a model training method, an image recognition method, device and medium for image recognition, which can ensure the accuracy of the output image recognition result, save the human resources consumed when annotating the image quality score, and improve the training efficiency of the quality assessment model. The present application provides the following technical solutions:

[0006] In a first aspect, a model training method for image recognition is provided, and the method includes:

[0007] Obtain a training data set, where the training data set includes multiple sample images and the category label corresponding to each sample image;

[0008] Train a preset first machine learning network based on the training data set to obtain a feature extraction model; the feature extraction model is used to extract features from an input target image during image recognition to obtain image features, so as to determine a template feature matching the image features in a preset template feature library;

[0009] For each of the N sample images corresponding to each category label in the training dataset, the preset second machine learning network is trained based on the N sample images and the feature extraction model to obtain a quality assessment model; the quality assessment model is used to determine the quality score of the target image during the image recognition process, so as to determine the recognition result of the target image in combination with the quality score and the template feature; where N is a positive integer.

[0010] Optionally, the feature extraction model includes a feature extraction layer and a block layer connected to the feature extraction layer, and the block layer is used to divide the feature map output by the feature extraction layer into at least two sub-feature maps according to a preset rule; correspondingly, the quality assessment model outputs the quality score corresponding to each sub-feature map;

[0011] The training of the preset second machine learning network based on the N sample images and the feature extraction model to obtain a quality assessment model includes:

[0012] Each sample image is respectively input into the feature extraction model and the second machine learning network to obtain at least two sub-feature maps corresponding to the sample image and the prediction result of the quality score corresponding to each sub-feature map;

[0013] The sub-feature maps are weighted based on the prediction result of the quality score corresponding to each sub-feature map, and the differently weighted sub-features of the same sample image are concatenated to obtain the weighted fusion feature of the sample image;

[0014] The second machine learning network is trained based on the weighted fusion feature to obtain the quality assessment model.

[0015] Optionally, before each sample image is respectively input into the feature extraction model and the second machine learning network, it further includes:

[0016] A first sample image set, a second sample image set, and a third sample image set are generated based on the N sample images; the third sample image set covers the sample images collected in each image acquisition scenario;

[0017] Correspondingly, the training of the second machine learning network based on the weighted fusion feature to obtain the quality assessment model includes:

[0018] Determining a first distance distribution between the first sample image set and the second sample image set based on the weighted fusion feature;

[0019] Determining a second distance distribution between the first sample image set and the third sample image set based on the weighted fusion feature;

[0020] Training the second machine learning network based on the distance distribution difference between the first distance distribution and the second distance distribution to obtain the quality evaluation model.

[0021] Optionally, at least two classification networks are further connected to the chunking layer, and the classification networks correspond one-to-one with the sub-features Figure 1 In the classification network, the weights corresponding to each classification for the sub-feature map are different;

[0022] The training the second machine learning network based on the distance distribution difference between the first distance distribution and the second distance distribution to obtain the quality evaluation model includes:

[0023] Obtaining the weights of the classification network corresponding to each sub-feature map in each sample image;

[0024] Determining the predicted category of the sample image based on the sub-feature map and the weights;

[0025] Training the second machine learning network based on the classification difference between the predicted category and the category label corresponding to the sample image, as well as the distance distribution difference, to obtain the quality evaluation model.

[0026] Optionally, after generating the first sample image set, the second sample image set, and the third sample image set based on the N sample images, it further includes:

[0027] Generating teacher positive pairs in the teacher distribution based on the sample images in the first sample image set and the sample images in the second sample image set, and generating teacher negative pairs in the teacher distribution based on the sample images of other categories in the training dataset;

[0028] Generating student positive pairs in the student distribution based on the sample images in the first sample image set and the sample images in the third sample image set, and generating student negative pairs in the student distribution based on the sample images of other categories in the training dataset;

[0029] Correspondingly, the determining the first distance distribution between the first sample image set and the second sample image set based on the weighted fusion feature includes:

[0030] Determining the similarity between the weighted fusion features corresponding to the teacher positive pairs and the similarity between the weighted fusion features corresponding to the teacher negative pairs to obtain the first distance distribution;

[0031] Correspondingly, the determining the second distance distribution between the first sample image set and the third sample image set based on the weighted fusion feature includes:

[0032] Determine the similarity between the weighted fusion features corresponding to the positive pairs of the students and the similarity between the weighted fusion features corresponding to the negative pairs of the students, to obtain the second distance distribution.

[0033] Optionally, the generating the first sample image set, the second sample image set, and the third sample image set based on the N sample images includes:

[0034] Divide the N sample images into three sample image sets, where two of the sample image sets are the first sample image set and the second sample image set respectively;

[0035] Perform image expansion on the sample image set other than the two sample image sets according to different image acquisition scenarios to obtain the third sample image set.

[0036] In a second aspect, there is provided an image recognition method, the method including:

[0037] Obtain a target image to be recognized;

[0038] Input the target image into a feature extraction model and a quality evaluation model respectively, so as to output the image features of the target image through the feature extraction model, and output the quality score of the target image through the quality evaluation model; the feature extraction model and the quality evaluation model are trained by the model training method provided in the first aspect;

[0039] Determine a template feature matching the image features in a preset template feature library;

[0040] Determine the recognition result of the target image based on the quality score and the template feature.

[0041] Optionally, the feature extraction model includes a feature extraction layer and a block layer connected to the feature extraction layer, and the block layer is used to divide the feature map output by the feature extraction layer into at least two sub-feature maps according to a preset rule; correspondingly, the quality evaluation model outputs the quality score corresponding to each sub-feature map;

[0042] Correspondingly, the determining the recognition result of the target image based on the quality score and the template feature includes:

[0043] If the similarity between each sub-feature map and the corresponding template feature is greater than a first similarity threshold, and there is at least one sub-feature map whose quality score is greater than a first score threshold, then output the classification result corresponding to the template feature.

[0044] Optionally, the method further includes:

[0045] If the similarity between each sub-feature map and the corresponding template feature is greater than a second similarity threshold and the quality score of each sub-feature map is less than a second score threshold, the classification result corresponding to the template feature is not output.

[0046] In a third aspect, an electronic device is provided. The device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement the model training method for image recognition provided in the first aspect; or to implement the image recognition method provided in the second aspect.

[0047] In a fourth aspect, a computer-readable storage medium is provided. A program is stored in the storage medium, and when the program is executed by a processor, it is used to implement the model training method for image recognition provided in the first aspect; or to implement the image recognition method provided in the second aspect.

[0048] By training a preset first machine learning network based on a training data set, a feature extraction model is obtained. The feature extraction model is used to extract image features from an input target image during image recognition to determine a template feature matching the image features in a preset template feature library; for each of the N sample images corresponding to each category label in the training data set, the preset second machine learning network is trained based on the N sample images and the feature extraction model to obtain a quality evaluation model; the quality evaluation model is used to determine the quality score of the target image during image recognition to determine the recognition result of the target image in combination with the quality score and the template feature; the problem of reducing the accuracy of image recognition when a low-quality image is input into an image recognition network can be solved. Since the quality of the input image can be evaluated based on the quality evaluation model, in the case of low quality, it can be determined that the accuracy of the image recognition result is not high, so that the image recognition result is not output, ensuring the accuracy of the output image recognition result. At the same time, the quality evaluation model does not need to be trained using pre-annotated quality score labels. Therefore, the human resources consumed when annotating the image quality score can be saved, and the training efficiency of the quality evaluation model can be improved.

[0049] In addition, the feature map output by the feature extraction layer is divided into at least two sub-feature maps according to a preset rule through a block layer. Even if the area corresponding to a certain sub-feature map is occluded, image recognition can be performed through other sub-feature maps, which can improve the success rate of image recognition.

[0050] In addition, by training a quality evaluation model based on the distance distribution difference, the trained quality evaluation model can have the ability to evaluate images collected in various image acquisition scenarios, improving the network performance.

[0051] In addition, by determining the distance distribution difference based on the distributed distillation function, not only can the performance gap between simple samples and difficult samples be narrowed, but also the distance between the similarity distribution expectations from negative pairs to positive pairs can be minimized to control the overlap, thereby improving the network performance of the quality assessment model.

[0052] In addition, by jointly training the quality assessment model by combining the distance distribution difference and the classification difference, not only can the performance of simple samples be maintained, but also the performance gap between simple samples and difficult samples can be narrowed, which can further improve the network performance of the quality assessment model.

[0053] In addition, by ensuring that the third sample image set in the student distribution covers each image acquisition scenario, the evaluation performance of the quality assessment model for images in each image acquisition scenario can be guaranteed.

[0054] The above description is only an overview of the technical solution of this application. In order to be able to understand the technical means of this application more clearly and implement it in accordance with the content of the specification, the following will be described in detail with reference to the preferred embodiments of this application and the accompanying drawings. Brief Description of the Drawings

[0055] Figure 1 is a flowchart of a model training method for image recognition provided by an embodiment of this application;

[0056] Figure 2 is a schematic diagram of the training process of a feature extraction model provided by an embodiment of this application;

[0057] Figure 3 is a schematic diagram of the training process of a quality assessment model provided by an embodiment of this application;

[0058] Figure 4 is a flowchart of an image recognition method provided by an embodiment of this application;

[0059] Figure 5 is a block diagram of a model training device for image recognition provided by an embodiment of this application;

[0060] Figure 6 is a block diagram of an image recognition device provided by an embodiment of this application;

[0061] Figure 7 is a block diagram of an electronic device provided by an embodiment of this application. Detailed Description of the Embodiments

[0062] The following will further describe in detail the specific embodiments of this application in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate this application, but are not used to limit the scope of this application.

[0063] In traditional image recognition methods, an image recognition network is usually used to recognize an input target image to obtain a recognition result. However, the target image is greatly affected by the environment, which may cause the image recognition network to output an incorrect recognition result, affecting the accuracy of image recognition.

[0064] Based on the above technical problems, in a possible implementation, a quality assessment model can be pre-trained to obtain the quality score of the target image by inputting the target image into the quality assessment model, so as to determine the quality of the input image in combination with the quality score. In this way, when the quality is low, it can be determined that the accuracy of the image recognition result is not high, and thus the image recognition result is not output, ensuring the accuracy of the output image recognition result.

[0065] However, in the above embodiment, it is necessary to use the sample image and the quality score label corresponding to the sample image to train the quality assessment model; and use the sample image and the class label corresponding to the sample image to train the image recognition network. At this time, a large number of image annotation operations need to be performed on the sample image, and the efficiency of model training is low.

[0066] Based on the above technical problems, this embodiment provides a model training method for image recognition. After training a feature extraction model using the sample image and the class label corresponding to the sample image, a quality assessment model is trained based on the feature extraction model and the sample image, without the need to label the quality score label of the sample image. Therefore, the human resources consumed when annotating the image quality score can be saved, and the training efficiency of the quality assessment model can be improved. At the same time, the trained quality assessment model can also be used to screen out images suitable for the feature extraction model, thereby improving the accuracy of image recognition.

[0067] Next, the model training method for image recognition provided by this application will be described in detail. Optionally, this application takes the model training method for image recognition provided in each embodiment as an example for an electronic device. The electronic device is a terminal or a server. The terminal can be a mobile phone, a computer, a tablet computer, a scanner, an electronic eye, a surveillance camera, etc. The type of the electronic device is not limited in this embodiment.

[0068] Figure 1 FIG. is a flowchart of a model training method for image recognition provided by an embodiment of this application. The method at least includes the following steps:

[0069] Step 101, obtain a training data set, where the training data set includes multiple sample images and the class label corresponding to each sample image.

[0070] In this embodiment, the sample image is an image including the target to be recognized. The sample image can be a single frame image extracted from a video stream, or it can also be a photo taken of the target to be recognized. This embodiment does not limit the type of the sample image. Among them, the target to be recognized includes but is not limited to: face, vehicle, license plate, etc. This embodiment does not limit the type of the target to be recognized.

[0071] The category of the sample image is used to distinguish different targets to be recognized. The category label can be represented by the identifier of the target to be recognized. For example, when the target to be recognized is a face, the category label is the identity ID of the face.

[0072] Each category label in the training dataset corresponds to at least one sample image of the same target to be recognized, and the number of sample images corresponding to different category labels can be the same or different.

[0073] Optionally, the training dataset includes sample images collected in different image acquisition scenarios to improve the ability of the network model to adapt to different image acquisition scenarios. The images obtained by photographing the same target to be recognized in different image acquisition scenarios are different. Different image acquisition scenarios include but are not limited to: different light intensities, different acquisition angles, different shutter speeds, different acquisition distances, and / or different exposures, etc. This embodiment does not limit the implementation manner of the image acquisition scenario.

[0074] Optionally, the category label is obtained by manually annotating the sample image, and / or is read from a pre-annotated public set. This embodiment does not limit the implementation manner of the category label.

[0075] Optionally, the sizes of the sample images in the training dataset are consistent. At this time, after the original image is obtained, object detection is performed on the original image to obtain an object detection frame; the key points of the target to be recognized are located based on the object detection frame; according to the positions of the key points, the original image is normalized to the same size to obtain a sample image. For example, the normalized size is 112×112. In actual implementation, it can also be other sizes. This embodiment does not limit the value of the normalized size.

[0076] Step 102: Train a preset first machine learning network based on the training dataset to obtain a feature extraction model.

[0077] The feature extraction model is used to extract image features from the input target image during the image recognition process to determine the template feature that matches the image features in the preset template feature library. Specifically, the feature extraction model is used to extract the image features of the target to be recognized in the target image.

[0078] Optionally, the first machine learning network is a neural network model, which can be established based on a residual network. In other embodiments, the neural network model can also be established based on other types of network structures. This embodiment does not limit the implementation manner of the first machine learning network.

[0079] In the model training stage, the first machine learning network is connected to a classification network, which is used to predict the probabilities that the feature data output by the first machine learning network belong to each category. The weights corresponding to the feature data in each category in the classification network are different.

[0080] Accordingly, training the preset first machine learning network based on a training data set to obtain a feature extraction model, including:

[0081] Inputting a sample image into the first machine learning network to obtain feature data; inputting the feature data into the classification network to predict the probabilities that the feature data belong to each category; obtaining the weights when the classification network makes probability predictions; determining the predicted category of the sample image based on the weights and the feature data; training the first machine learning network based on the classification difference between the predicted category and the category label corresponding to the sample image to update the model parameters of the first learning model and obtain a feature extraction model. At this time, the feature extraction model is trained by minimizing the classification difference.

[0082] Optionally, the sample images are input into the first machine learning network in the manner of image sampling. Among them, the image sampling method can be realized by randomly selecting sample images.

[0083] Optionally, the classification difference is determined by an additive angular margin loss function (ArcFace loss: Additive Angular Margin Loss, ArcFace loss), and the ArcFace loss function is expressed by the following formula:

[0084]

[0085] where N is the total number of sample images; i is the i-th sample image, 1 ≤ i ≤ N; y i is the category label of the i-th sample image; n is the total number of categories; j is the j-th category; s is a scaling factor, and θ yi is the angular margin between the weight corresponding to y i in the classification network and the feature data; θ j is the angular margin between the weight corresponding to category j in the classification network and the feature data; t is the angular margin.

[0086] Among them, the classification network can be a fully connected network or other types of network structures. This embodiment does not limit the implementation manner of the classification network.

[0087] In one example, since the target to be recognized in the sample image may be partially occluded, at this time, because the target to be recognized in the sample image is incomplete, the traditional feature extraction model may not be able to extract the feature data of the sample image. Based on this, the first machine learning network may specifically include a feature extraction layer and a block layer connected to the feature extraction layer. Among them, the feature extraction layer is used to extract the feature map of the input image; the block layer is used to divide the feature map output by the feature extraction layer into at least two sub-feature maps according to a preset rule; correspondingly, the trained feature extraction model also includes a trained feature extraction layer and a trained block layer. In this way, even if the area corresponding to a certain sub-feature map is occluded, image recognition can be performed through other sub-feature maps.

[0088] Among them, the preset rules include but are not limited to: dividing the feature map into two upper and lower sub-feature maps with the same size; or dividing the feature map into two left and right sub-feature maps with the same size; or dividing the feature map into 4 sub-feature maps with the same size along the horizontal and vertical midlines, etc. This embodiment does not limit the implementation manner of the preset rules.

[0089] Correspondingly, the block layer is also connected to at least two classification networks, and the classification networks correspond one-to-one with the sub-features Figure 1 to predict the category of each sub-feature map. In the above loss function, θ yi can be adaptively modified to the average of the angular intervals between the weights corresponding to y i in the classification network and the feature data of each sub-feature map; θ j is the average of the angular intervals between the weights corresponding to category j in the classification network and the feature data of each sub-feature map; or, θ yi can be adaptively modified to the angular interval between the weights corresponding to y i in the classification network and the feature data of the same sub-feature map; θ j is the angular interval between the weights corresponding to category j in the classification network and the feature data of the same sub-feature map. The average value of the loss function values corresponding to different sub-feature maps is the final loss function value, or the weighted value of the loss function values corresponding to different sub-feature maps is the final loss function value. This embodiment does not limit the calculation method of the loss function value corresponding to the sample image when the sample image corresponds to multiple sub-feature maps.

[0090] In other embodiments, the division layer may not be set in the training stage of the feature extraction model, but the division layer may be set only in the training stage and the image recognition stage of the following quality evaluation model. This embodiment does not limit the training method of the feature extraction model.

[0091] Taking the example where the first machine learning network is based on a residual network and includes a chunking layer, and the classification network is a fully connected network, the training process of the feature extraction model will be described. Refer to Figure 2 During the training process, the model structure includes a first machine learning network 21 to be trained, a fully connected network 22 connected to the first machine learning network 21, a feature fusion layer connected to the fully connected network 22, and a loss function 23 connected to the feature fusion layer.

[0092] Among them, the first machine learning network 21 includes: an input layer 201, a convolutional (Conv) layer 202, and multiple repeated convolutional layers 202, pooling layers 203, and residual units (Resblock) 204, as well as a chunking layer 205. The functions of each network layer are referred to in Table 1 below. During the training process, the sample image passes through the input layer 201, convolutional layer 202, and multiple repeated convolutional layers 202, pooling layer 203, and residual unit 204 in sequence to obtain a feature map. The feature map is chunked by the chunking layer 205 to obtain at least two sub-feature maps; each sub-feature map is input to the corresponding fully connected network 22 for classification to obtain the weights corresponding to different categories for each sub-feature map; after connecting each sub-feature map through the feature fusion layer, the weights and the connected feature map are input to the loss function 23 to obtain a classification loss, which represents the classification difference. Then, based on the classification loss, the model parameters of the first machine learning network 21 are iteratively updated to obtain a feature extraction model.

[0093] Figure 2 Taking the number of sub-feature maps as 2 as an example, correspondingly, the number of fully connected networks 22 is also two. In actual implementation, the number of sub-feature maps and fully connected networks can also be more, and this embodiment does not limit this.

[0094] Table 1:

[0095]

[0096] Step 103, for the N sample images corresponding to each category label in the training dataset, based on the N sample images and the feature extraction model, train a preset second machine learning network to obtain a quality assessment model.

[0097] The quality assessment model is used to determine the quality score of the target image during the image recognition process to combine the quality score and the template feature to determine the recognition result of the target image; where N is a positive integer. During the training process of the quality assessment model, the model parameters of the feature extraction model are fixed.

[0098] In one example, the feature extraction model includes a feature extraction layer and a chunking layer connected to the feature extraction layer. Correspondingly, the quality assessment model is used to output the quality score corresponding to each sub-feature map.

[0099] At this time, based on N sample images and the feature extraction model, the preset second machine learning network is trained to obtain a quality assessment model, including: inputting each sample image into the feature extraction model and the second machine learning network respectively to obtain at least two sub-feature maps corresponding to the sample image and the prediction result of the quality score corresponding to each sub-feature map; weighting the sub-feature maps based on the prediction result of the quality score corresponding to each sub-feature map, and connecting the differently weighted sub-features of the same sample image to obtain the weighted fusion feature of the sample image; training the second machine learning network based on the weighted fusion feature to obtain the quality assessment model.

[0100] Optionally, the second machine learning network is a neural network model, and this neural network model can be established based on a residual network. In other embodiments, the neural network model can also be established based on other types of network structures. This embodiment does not limit the implementation manner of the second machine learning network.

[0101] Among them, training the second machine learning network based on the weighted fusion feature to obtain the quality assessment model includes but is not limited to the following methods:

[0102] The first one: determining a feature distribution distillation loss based on the weighted fusion feature to train the second machine learning network.

[0103] When calculating the feature distribution distillation loss, different sample image sets need to be created in advance to obtain the feature distributions before different sample image sets. At this time, before inputting each sample image into the feature extraction model and the second machine learning network respectively, it further includes: generating a first sample image set, a second sample image set, and a third sample image set based on N sample images; the third sample image set covers the sample images collected in each image acquisition scenario.

[0104] Optionally, generating a first sample image set, a second sample image set, and a third sample image set based on N sample images includes: dividing the N sample images into three sample image sets, where two of the sample image sets are the first sample image set and the second sample image set respectively; expanding the sample image set other than the two sample image sets according to different image acquisition scenarios to obtain the third sample image set. N is an integer greater than or equal to 3.

[0105] Image expansion is used to make the third sample image set cover more image acquisition scenarios. The image expansion methods include, but are not limited to: rotating, blurring, brightness adjusting, scaling, etc. of the sample images in the sample image set. This embodiment does not limit the image expansion methods.

[0106] Correspondingly, training the second machine learning network based on the weighted fusion features to obtain a quality evaluation model, including: determining a first distance distribution between the first sample image set and the second sample image set based on the weighted fusion features; determining a second distance distribution between the first sample image set and the third sample image set based on the weighted fusion features; training the second machine learning network based on the distance distribution difference between the first distance distribution and the second distance distribution to obtain a quality evaluation model.

[0107] The distance distribution (including the first distance distribution and the second distance distribution) is used to characterize the similarity between the weighted fusion features corresponding to different sample image sets. In this embodiment, by making the second distance distribution approximate the first distance distribution, the trained quality evaluation model can have the ability to evaluate the images acquired in each image acquisition scenario, improving the network performance.

[0108] In an example, determining the distance distribution difference based on the Distribution Distillation Loss (DDL) function. At this time, after generating the first sample image set, the second sample image set, and the third sample image set based on N sample images, it further includes: generating teacher positive pairs in the teacher distribution based on the sample images in the first sample image set and the sample images in the second sample image set, and generating teacher negative pairs in the teacher distribution based on the sample images of other categories in the training data set; generating student positive pairs in the student distribution based on the sample images in the first sample image set and the sample images in the third sample image set, and generating student negative pairs in the student distribution based on the sample images of other categories in the training data set.

[0109] Correspondingly, determining the first distance distribution between the first sample image set and the second sample image set based on the weighted fusion features includes: determining the similarity between the weighted fusion features corresponding to the teacher positive pairs and the similarity between the weighted fusion features corresponding to the teacher negative pairs to obtain the first distance distribution.

[0110] Correspondingly, determining the second distance distribution between the first sample image set and the third sample image set based on the weighted fusion features includes: determining the similarity between the weighted fusion features corresponding to the student positive pairs and the similarity between the weighted fusion features corresponding to the student negative pairs to obtain the second distance distribution.

[0111] Accordingly, based on the distance distribution difference between the first distance distribution and the second distance distribution, the second machine learning network is trained to obtain a quality evaluation model, including: determining the positive pair relative entropy loss value based on the similarity corresponding to the teacher positive pair and the similarity corresponding to the student positive pair; determining the negative pair relative entropy loss value based on the similarity corresponding to the teacher negative pair and the similarity corresponding to the student negative pair; determining the positive and negative pair distribution distance based on the distribution expectations of the similarity corresponding to the teacher positive pair and the similarity corresponding to the student positive pair, and the distribution expectations of the similarity corresponding to the teacher negative pair and the similarity corresponding to the student negative pair; training the second machine learning network based on the positive pair relative entropy loss value, the negative pair relative entropy loss value, and the positive and negative pair distribution distance to obtain a quality evaluation model.

[0112] To narrow the performance gap between easy samples and hard samples, the similarity distribution of hard samples (i.e., the student distribution) is constrained to be approximately the similarity distribution of easy samples (i.e., the teacher distribution). The teacher distribution consists of two similarity distributions of positive pairs and negative pairs, denoted as P+ and P- respectively. That is, P+ is the similarity corresponding to the teacher positive pair, and P- is the similarity corresponding to the teacher negative pair. The student distribution also consists of two similarity distributions, denoted as Q+ and Q-. That is, Q+ is the similarity distribution corresponding to the student positive pair, and Q- is the similarity distribution corresponding to the student negative pair. DDL uses the KL divergence L KL to constrain the similarity between the student distribution and the teacher distribution, and its definition is as follows (where λ1 and λ2 are weight parameters):

[0113]

[0114] where s represents the image pair, α1 and α2 are preset weight parameters, D KL (P + ||Q + ) represents the positive pair relative entropy loss value, and D KL (P - ||Q - ) represents the negative pair relative entropy loss value.

[0115] Since when only using the KL divergence, the teacher distribution may choose to be close to the student distribution, which is contrary to the above approximation goal. Based on this, an order loss is also introduced in DDL to minimize the distance between the distribution expectations of the similarity from negative pairs to positive pairs and control the overlap. The order loss L order is represented by the following formula:

[0116]

[0117] where, is the similarity distribution corresponding to the positive pair; is the similarity distribution corresponding to the negative pair; α3 is a preset weight parameter, and E represents the expectation.

[0118] In summary, the DDL loss function L DDL = L KL + L order , which can be specifically expressed by the following formula:

[0119]

[0120] where K is the number of student distributions, and D KL is the KL divergence loss.

[0121] In actual implementation, the loss function for making the second distance distribution approximate the first distance distribution can also be other loss functions, such as minimizing the variance between the second distance distribution and the first distance distribution, etc. This embodiment does not limit the implementation manner of the loss function for making the second distance distribution approximate the first distance distribution.

[0122] Second: Determine the feature distribution distillation loss based on the weighted fusion features, and based on the classification loss of the category corresponding to the weighted fusion features, to train the second machine learning network.

[0123] Among them, the relevant content of the feature distribution distillation loss can be found in the first training method in detail. When calculating the classification loss, the block layer of the feature extraction model is also connected to at least two classification networks. Correspondingly, based on the distance distribution difference between the first distance distribution and the second distance distribution, the second machine learning network is trained to obtain a quality evaluation model, including: obtaining the weights of the classification networks corresponding to each sub-feature map in each sample image; determining the predicted category of the sample image based on the sub-feature map and the weights; training the second machine learning network based on the classification difference between the predicted category and the category label corresponding to the sample image, as well as the distance distribution difference, to obtain a quality evaluation model.

[0124] Optionally, training the second machine learning network based on the classification difference and the distribution difference includes: calculating the sum of the classification loss and the feature distribution distillation loss to obtain a loss sum; iteratively updating the model parameters of the second machine learning network based on the loss sum. Or, calculating the weighted sum before the classification loss and the feature distribution distillation loss to obtain a weighted loss; iteratively updating the model parameters of the second machine learning network based on the weighted loss. This embodiment does not limit the method for determining the loss value based on the classification difference and the distribution difference.

[0125] Among them, the relevant description of the classification difference can be found in the training process of the feature extraction model, and this embodiment will not elaborate here.

[0126] Next, an example will be given to illustrate the training of the quality evaluation model in the second method, and the second machine learning network is based on the residual network. Refer to Figure 3, the model structure during the training process includes the trained feature extraction model 31 and the second machine learning network 32 to be trained. The block layer of the feature extraction model 31 is connected to at least two fully connected networks 34 ( Figure 3 Taking two fully connected networks 34 as an example, correspondingly, the number of sub-feature maps is also two). At this time, the classification network is implemented as a fully connected network, and this fully connected network classifies according to the data output by the feature extraction model 31. The second machine learning network 32 includes an input layer 321, multiple repeated convolutional layers 322, pooling layers 323, and residual units 324, and a fully connected network 325 connected to the last residual unit 324. The fully connected network 34 and the fully connected network 325 are respectively connected to the feature fusion layer 35, and the feature fusion layer 35 is respectively connected to the loss function 36. The functions of each network layer are referred to in Table II below. Among them, the output of the fully connected network 325 is preset to be two. At this time, the output results of the fully connected network 325 include two one-dimensional vectors, which respectively correspond to the prediction results of the quality scores of each sub-feature map.

[0127] During the training process, each sample image is respectively input into the feature extraction model 31 and the second machine learning network 32. The two sub-feature maps output by the block layer of the feature extraction model 31 are respectively input into the corresponding fully connected networks 34 to obtain the weights of the fully connected networks 34 corresponding to each category. The two sub-feature maps output by the block layer and the prediction results of the quality scores output by the second machine learning network 32 are input into the feature fusion layer 35. The sub-feature maps are weighted based on the two sub-feature maps and the prediction results of the quality scores corresponding to each sub-feature map, and the differently weighted sub-features of the same sample image are connected to obtain the weighted fusion feature of the sample image. The weighted fusion feature is respectively input into the classification loss and the feature distribution distillation loss in the loss function 36 to obtain a loss value. Then, based on the loss value, the model parameters of the second machine learning network 32 are iteratively updated to obtain a quality evaluation model.

[0128] Table II:

[0129]

[0130] In other examples, if the feature extraction model does not include a chunking layer, the preset second machine learning network is trained based on N sample images and the feature extraction model to obtain a quality assessment model, including: inputting each sample image into the feature extraction model and the classification network to obtain the feature data output by the feature extraction model and the weights corresponding to each category of the feature data output by the classification network; determining the target quality score based on the similarity between the weight and the feature data; inputting the sample image into the second machine learning network to obtain the prediction result of the quality score of the sample image; determining the quality loss based on the target quality score and the prediction result of the quality score; and iteratively updating the model parameters of the second machine learning network based on the quality loss to obtain the quality assessment model. The training method of the quality assessment model in this embodiment is not limited.

[0131] In summary, the model training method for image recognition provided in this embodiment trains the preset first machine learning network based on the training data set to obtain a feature extraction model, which is used to extract image features from the input target image during the image recognition process to determine the template feature matching the image feature in the preset template feature library; for the N sample images corresponding to each category label in the training data set, the preset second machine learning network is trained based on the N sample images and the feature extraction model to obtain a quality assessment model; the quality assessment model is used to determine the quality score of the target image during the image recognition process to determine the recognition result of the target image by combining the quality score and the template feature; the problem of reducing the accuracy of image recognition when a low-quality image is input into the image recognition network can be solved. Since the quality of the input image can be evaluated based on the quality assessment model, in the case of low quality, it can be determined that the accuracy of the image recognition result is not high, so that the image recognition result is not output, ensuring the accuracy of the output image recognition result. At the same time, the quality assessment model does not need to be trained using pre-annotated quality score labels, so the human resources consumed in annotating the image quality score can be saved, and the training efficiency of the quality assessment model can be improved.

[0132] In addition, the feature map output by the feature extraction layer is divided into at least two sub-feature maps according to a preset rule through the chunking layer. Even if the area corresponding to a certain sub-feature map is occluded, image recognition can be performed through other sub-feature maps, which can improve the success rate of image recognition.

[0133] In addition, by training the quality assessment model based on the distance distribution difference, the trained quality assessment model can have the ability to evaluate the images collected in each image acquisition scenario, improving the network performance.

[0134] In addition, by determining the distance distribution difference based on the distributed distillation function, not only can the performance gap between simple samples and difficult samples be narrowed, but also the distance between the similarity distribution expectations from negative pairs to positive pairs can be minimized to control the overlap, thereby improving the network performance of the quality assessment model.

[0135] In addition, by jointly training the quality assessment model by combining the distance distribution difference and the classification difference, not only can the performance of simple samples be maintained, but also the performance gap between simple samples and difficult samples can be narrowed, which can further improve the network performance of the quality assessment model.

[0136] In addition, by enabling the third sample image set in the student distribution to cover each image acquisition scenario, the evaluation performance of the quality assessment model for images under each image acquisition scenario can be ensured.

[0137] Based on the feature extraction model and the quality assessment model obtained from the above embodiments, the present application further provides an image recognition method. Figure 4 It is a flowchart of an image recognition method provided by an embodiment of the present application. The method at least includes the following steps:

[0138] Step 401, obtain a target image to be recognized.

[0139] In one example, an original image is collected through an image acquisition component, target detection is performed on the original image to obtain the position and size of the target to be recognized; key point positioning is performed on the detected target to be recognized and normalized to a preset size to extract its feature data and quality score.

[0140] Step 402, input the target image into the feature extraction model and the quality assessment model respectively, so as to output the image features of the target image through the feature extraction model and output the quality score of the target image through the quality assessment model.

[0141] Among them, the feature extraction model and the quality assessment model are trained based on the model training method provided in the above embodiments.

[0142] Step 403, determine the template feature that matches the image feature in the preset template feature library.

[0143] Step 404, determine the recognition result of the target image based on the quality score and the template feature.

[0144] In one example, the feature extraction model includes a feature extraction layer and a chunking layer connected to the feature extraction layer. The chunking layer is used to divide the feature map output by the feature extraction layer into at least two sub-feature maps according to a preset rule. Correspondingly, the quality assessment model outputs a quality score corresponding to each sub-feature map. At this time, determining the recognition result of the target image based on the quality score and the template feature includes: if the similarity between each sub-feature map and the corresponding template feature is greater than the first similarity threshold T1 and there is at least one sub-feature map with a quality score greater than the first score threshold T3, then output the classification result corresponding to the template feature; if the similarity between each sub-feature map and the corresponding template feature is greater than the second similarity threshold T2 and the quality score of each sub-feature map is less than the second score threshold T4, then do not output the classification result corresponding to the template feature.

[0145] The second score threshold T4 is less than or equal to the first score threshold T3. The first similarity threshold T1 is greater than or equal to the second similarity threshold T2.

[0146] The output manner of the classification result includes but is not limited to: outputting category information and giving an alarm, or only displaying category information, etc. This embodiment does not limit the output manner.

[0147] In summary, the image recognition method provided in this embodiment can solve the problem of reducing the accuracy of image recognition when a low-quality image is input into the image recognition network by obtaining the target image to be recognized; inputting the target image into the feature extraction model and the quality assessment model respectively to output the image feature of the target image through the feature extraction model and output the quality score of the target image through the quality assessment model; determining the template feature matching the image feature in the preset template feature library; and determining the recognition result of the target image based on the quality score and the template feature. Since the quality of the input image can be evaluated based on the quality assessment model, in the case of low quality, it can be determined that the accuracy of the image recognition result is not high, so that the image recognition result is not output, ensuring the accuracy of the output image recognition result. At the same time, the quality assessment model does not need to be trained using pre-annotated quality score labels. Therefore, the human resources consumed in annotating the image quality score can be saved, and the training efficiency of the quality assessment model can be improved.

[0148] In addition, by dividing the feature map output by the feature extraction layer into at least two sub-feature maps through the chunking layer, even if the area corresponding to a certain sub-feature map is occluded, image recognition can still be performed through other sub-feature maps, which can improve the success rate of image recognition.

[0149] Figure 5It is a block diagram of a model training device for image recognition provided by an embodiment of the present application. The device at least includes the following modules: a data acquisition module 510, a first training module 520, and a second training module 530.

[0150] The data acquisition module 510 is configured to acquire a training data set, where the training data set includes multiple sample images and a class label corresponding to each sample image;

[0151] The first training module 520 is configured to train a preset first machine learning network based on the training data set to obtain a feature extraction model; the feature extraction model is used to extract image features from an input target image during image recognition to determine a template feature matching the image features in a preset template feature library;

[0152] The second training module 530 is configured to, for N sample images corresponding to each class label in the training data set, train a preset second machine learning network based on the N sample images and the feature extraction model to obtain a quality assessment model; the quality assessment model is used to determine a quality score of the target image during image recognition to determine the recognition result of the target image in combination with the quality score and the template feature; where N is a positive integer.

[0153] For related details, refer to the above method embodiment.

[0154] It should be noted that: when the model training device for image recognition provided in the above embodiment performs model training for image recognition, only the above division of each functional module is used for illustration. In practical applications, the above functions can be assigned to different functional modules according to needs, that is, the internal structure of the model training device for image recognition is divided into different functional modules to complete all or part of the functions described above. In addition, the model training device for image recognition provided in the above embodiment and the method embodiment for model training for image recognition belong to the same concept. For the specific implementation process, refer to the method embodiment and will not be elaborated here.

[0155] Figure 6 It is a block diagram of an image recognition device provided by an embodiment of the present application. The device at least includes the following modules: an image acquisition module 610, a data extraction module 620, a feature matching module 630, and an image recognition module 640.

[0156] The image acquisition module 610 is configured to acquire a target image to be recognized;

[0157] A data extraction module 620 is configured to input the target image into a feature extraction model and a quality assessment model respectively, so as to output the image features of the target image through the feature extraction model and output the quality score of the target image through the quality assessment model; the feature extraction model and the quality assessment model are trained based on the model training method for image recognition in the above embodiments;

[0158] A feature matching module 630 is configured to determine a template feature matching the image features in a preset template feature library;

[0159] An image recognition module 640 is configured to determine the recognition result of the target image based on the quality score and the template feature.

[0160] For relevant details, refer to the above method embodiments.

[0161] It should be noted that: when the image recognition device provided in the above embodiments performs image recognition, only the division of the above functional modules is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the image recognition device is divided into different functional modules to complete all or part of the functions described above. In addition, the image recognition device provided in the above embodiments and the image recognition method embodiments belong to the same concept. For the specific implementation process, refer to the method embodiments and will not be elaborated here.

[0162] Figure 7 It is a block diagram of an electronic device provided by an embodiment of the present application. The device at least includes a processor 501 and a memory 702.

[0163] The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0164] The memory 702 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 702 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 701 to implement the model training method or image recognition method for image recognition provided in the method embodiments of the present application.

[0165] In some embodiments, the electronic device may optionally further include: a peripheral device interface and at least one peripheral device. The processor 701, the memory 702, and the peripheral device interface may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface through a bus, signal lines, or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply, etc.

[0166] Of course, the electronic device may also include fewer or more components, and this embodiment does not limit this.

[0167] Optionally, the present application further provides a computer-readable storage medium, and a program is stored in the computer-readable storage medium, and the program is loaded and executed by a processor to implement the model training method or image recognition method for image recognition in the above method embodiments.

[0168] Optionally, the present application further provides a computer product, which includes a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the model training method or the image recognition method for image recognition in the above method embodiments.

[0169] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0170] The above embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A method for training a model for image recognition, characterized in that, The method includes: Obtaining a training data set, where the training data set includes multiple sample images and class labels corresponding to each sample image; Training a preset first machine learning network based on the training data set to obtain a feature extraction model; the feature extraction model is used to extract image features from an input target image during image recognition to determine a template feature matching the image features in a preset template feature library; For N sample images corresponding to each class label in the training data set, training a preset second machine learning network based on the N sample images and the feature extraction model to obtain a quality assessment model; the quality assessment model is used to determine a quality score of the target image during image recognition to determine the recognition result of the target image in combination with the quality score and the template feature; where N is a positive integer; The feature extraction model includes a feature extraction layer and a block layer connected to the feature extraction layer, and the block layer is used to divide the feature map output by the feature extraction layer into at least two sub-feature maps according to a preset rule; correspondingly, the quality assessment model outputs a quality score corresponding to each sub-feature map; The training of the preset second machine learning network based on the N sample images and the feature extraction model to obtain a quality assessment model includes: Inputting each sample image into the feature extraction model and the second machine learning network respectively to obtain at least two sub-feature maps corresponding to the sample image and prediction results of quality scores corresponding to each sub-feature map; Weighting the sub-feature maps based on the prediction results of the quality scores corresponding to each sub-feature map, and connecting different weighted sub-features of the same sample image to obtain a weighted fusion feature of the sample image; Inputting the weighted fusion feature into the classification loss and the feature distribution distillation loss in the loss function to obtain a loss value, and training the second machine learning network according to the loss value to obtain the quality assessment model.

2. The method according to claim 1, characterized in that, Before inputting each sample image into the feature extraction model and the second machine learning network respectively, it further includes: Generating a first sample image set, a second sample image set, and a third sample image set based on the N sample images; the third sample image set covers sample images collected in each image acquisition scenario; Correspondingly, inputting the weighted fusion feature into the feature distribution distillation loss in the loss function to obtain a loss value, and training the second machine learning network according to the loss value to obtain the quality assessment model includes: Determining a first distance distribution between the first sample image set and the second sample image set based on the weighted fusion feature; Determining a second distance distribution between the first sample image set and the third sample image set based on the weighted fusion feature; Training the second machine learning network based on the distance distribution difference between the first distance distribution and the second distance distribution to obtain the quality assessment model.

3. The method according to claim 2, wherein The block layer is also connected to at least two classification networks, the classification networks corresponding one by one to the sub-feature maps, and the weights corresponding to each classification of the sub-feature maps in the classification networks being different; Training the second machine learning network based on the distance distribution difference between the first distance distribution and the second distance distribution to obtain the quality assessment model, including: Obtaining the weights of the classification networks corresponding to each sub-feature map in each sample image; Determining the predicted category of the sample image based on the sub-feature map and the weights; Training the second machine learning network based on the classification difference between the predicted category and the category label corresponding to the sample image, as well as the distance distribution difference, to obtain the quality assessment model.

4. The method according to claim 2, wherein Generating the first sample image set, the second sample image set, and the third sample image set based on the N sample images, including: Dividing the N sample images into three sample image sets, where two of the sample image sets are the first sample image set and the second sample image set respectively; Performing image expansion on the sample image set other than the two sample image sets according to different image acquisition scenarios to obtain the third sample image set.

5. An image recognition method, characterized in that, The method includes: Obtaining a target image to be recognized; Inputting the target image into the feature extraction model and the quality assessment model respectively, so as to output the image features of the target image through the feature extraction model, and output the quality score of the target image through the quality assessment model; the feature extraction model and the quality assessment model are trained based on the model training method described in any one of claims 1 to 4; Determining the template feature matching the image features in a preset template feature library; Determining the recognition result of the target image based on the quality score and the template feature.

6. The method according to claim 5, wherein The feature extraction model includes a feature extraction layer and a block layer connected to the feature extraction layer, the block layer being used to divide the feature map output by the feature extraction layer into at least two sub-feature maps according to a preset rule; correspondingly, the quality assessment model outputs the quality score corresponding to each sub-feature map; Correspondingly, determining the recognition result of the target image based on the quality score and the template feature, including: If the similarity between each sub-feature map and the corresponding template feature is greater than a first similarity threshold, and the quality score of at least one sub-feature map is greater than a first score threshold, then output the classification result corresponding to the template feature.

7. The method according to claim 6, characterized in that The method further includes: If the similarity between each sub-feature map and the corresponding template feature is greater than a second similarity threshold, and the quality score of each sub-feature map is less than a second score threshold, then do not output the classification result corresponding to the template feature.

8. An electronic device, characterized in that, The device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement the model training method for image recognition described in any one of claims 1 to 4; or, to implement the image recognition method described in any one of claims 5 to 7.

9. A computer-readable storage medium, characterized in that, A program is stored in the storage medium, and when the program is executed by a processor, it is used to implement the model training method for image recognition according to any one of claims 1 to 4; or, to implement the image recognition method according to any one of claims 5 to 7.

Images