Image recognition method, device, computer equipment and storage medium

Abstract

The present application relates to an image recognition method, apparatus, computer device, and storage medium. The method comprises: obtaining an image to be recognized; performing feature extraction on the image to be recognized to obtain first-level features and second-level features of the image to be recognized; obtaining first-level category recognition results and second-level category recognition results for the image to be recognized based on the first-level features and the second-level features, respectively, wherein the first-level category recognition results include each identified first-level category, and the second-level category recognition results include each identified second-level category, where the second-level category is a subcategory of the first-level category; and determining the category of the image to be recognized based on the first-level category recognition results and the second-level category recognition results. This method can improve the accuracy of image sub-classification recognition.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image recognition method, apparatus, computer device, and storage medium. Background Art

[0002] With the development of deep learning technology, image recognition technology has emerged, which can be used to solve image classification problems. For example, when applied to product identification, image recognition technology can identify the category to which a product belongs. Product categories are generally divided into major categories, minor categories, etc. based on granularity. Traditional technologies use network models to learn the category features of images at a certain granularity to identify the image category at that granularity.

[0003] However, the feature differences between subcategories under different major categories may be relatively subtle, making image subclassification more difficult. The accuracy of traditional technologies for image subclassification recognition needs to be improved. Summary of the Invention

[0004] Based on this, it is necessary to provide an image recognition method, device, computer equipment and storage medium that can improve the accuracy of image sub-classification recognition in response to the above technical problems.

[0005] An image recognition method, comprising:

[0006] Obtain the image to be recognized;

[0007] Performing feature extraction on the image to be identified to obtain first-level features and second-level features of the image to be identified;

[0008] Obtaining a first-level category recognition result and a second-level category recognition result for the image to be recognized based on the first-level features and the second-level features, respectively, wherein the first-level category recognition result includes each identified first-level category, and the second-level category recognition result includes each identified second-level category, where the second-level category is a subcategory of the first-level category;

[0009] The category of the image to be identified is determined according to the first-level category identification result and the second-level category identification result.

[0010] An image recognition device, comprising:

[0011] An acquisition module, used to acquire an image to be identified;

[0012] A feature extraction module, configured to extract features from the image to be identified, and obtain first-level features and second-level features of the image to be identified;

[0013] an identification module, configured to obtain a first-level category identification result and a second-level category identification result of the image to be identified based on the first-level features and the second-level features, respectively, wherein the first-level category identification result includes each identified first-level category, and the second-level category identification result includes each identified second-level category, where the second-level category is a subcategory of the first-level category;

[0014] A determination module is used to determine the category of the image to be identified based on the first-level category identification result and the second-level category identification result.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0016] Obtain the image to be recognized;

[0017] Performing feature extraction on the image to be identified to obtain first-level features and second-level features of the image to be identified;

[0018] Obtaining a first-level category recognition result and a second-level category recognition result for the image to be recognized based on the first-level features and the second-level features, respectively, wherein the first-level category recognition result includes each identified first-level category, and the second-level category recognition result includes each identified second-level category, where the second-level category is a subcategory of the first-level category;

[0019] The category of the image to be identified is determined according to the first-level category identification result and the second-level category identification result.

[0020] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0021] Obtain the image to be recognized;

[0022] Performing feature extraction on the image to be identified to obtain first-level features and second-level features of the image to be identified;

[0023] Obtaining a first-level category recognition result and a second-level category recognition result for the image to be recognized based on the first-level features and the second-level features, respectively, wherein the first-level category recognition result includes each identified first-level category, and the second-level category recognition result includes each identified second-level category, where the second-level category is a subcategory of the first-level category;

[0024] The category of the image to be identified is determined according to the first-level category identification result and the second-level category identification result.

[0025] The above-described image recognition method, apparatus, computer device, and storage medium perform two-level feature extraction on an image to obtain first-level and second-level category recognition results for the image. The image's category is then determined based on the first-level and second-level category recognition results, where the second-level category is a subcategory of the first-level category. Consequently, when classifying an image, the image's final category can be determined using both the sub-category and major category recognition results. This addresses the difficulty of fine-grained classification due to minimal feature differences between sub-categories within different major categories, reduces classification errors within major categories, and thus improves image recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 1 is a flow chart of an image recognition method according to an embodiment;

[0027] Figure 2 1 is a flow chart of a training process of a twin network in one embodiment;

[0028] Figure 3 Schematic diagram of the structure of a twin network in one embodiment;

[0029] Figure 4 is a structural block diagram of an image recognition device in one embodiment;

[0030] Figure 5 is a diagram of the internal structure of a computer device in one embodiment;

[0031] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0033] In one embodiment, Figure 1 As shown, an image recognition method is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The server can be implemented as an independent server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps S102 to S108.

[0034] S102: Obtain an image to be recognized.

[0035] The image to be identified is the image that needs to be classified. The image is classified by identifying the image category. For example, when the image to be identified is a product image, the image can be classified based on different granularities (such as major categories, minor categories, etc.). The specific classification granularity can be set according to actual needs.

[0036] S104: Extract features of the image to be identified to obtain first-level features and second-level features of the image to be identified.

[0037] During implementation, first-level feature extraction is performed on the image to be identified to obtain the first-level features of the image to be identified, and second-level feature extraction is performed on the image to be identified to obtain the second-level features of the image to be identified. First-level feature extraction on the image to be identified refers to extracting features of the image to be identified in the first-level category dimension, where the first-level features refer to features extracted from the image to be identified that correspond to the first-level category dimension. Second-level feature extraction on the image to be identified refers to extracting features of the image to be identified in the second-level category dimension, where the second-level features refer to features extracted from the image to be identified that correspond to the second-level category dimension.

[0038] S106, based on the first-level features and the second-level features, respectively, obtain the first-level category recognition result and the second-level category recognition result of the image to be recognized, the first-level category recognition result includes the recognized first-level categories, and the second-level category recognition result includes the recognized second-level categories, and the second-level category is a subcategory of the first-level category.

[0039] In implementation, a first-level category recognition result of the image to be recognized is obtained based on the first-level features of the image to be recognized, and a second-level category recognition result of the image to be recognized is obtained based on the second-level features of the image to be recognized. The first-level category recognition result is the result obtained by classifying and recognizing the image to be recognized in the first-level category dimension, and the second-level category recognition result is the result obtained by classifying and recognizing the image to be recognized in the second-level category dimension.

[0040] The second-level category is a subcategory of the first-level category. It can be understood that the first-level category is a large category with a coarser granularity, and the second-level category is a small category with a finer granularity. A large category can contain multiple small categories. For example, for the large category of beverages, it can include bottled beverages, canned beverages, bagged beverages and other small categories. Similarly, for the large category of detergents, it can include bottled detergents, bagged detergents and other small categories. It should be noted that the specific categories in the large and small categories can be set according to actual conditions and there is no limitation on this.

[0041] S108 : Determine the category of the image to be identified based on the first-level category identification result and the second-level category identification result.

[0042] The category of the image to be identified may be the category to which the image to be identified belongs in the second-level category dimension. It is understood that once the second-level category to which the image to be identified belongs is determined, the first-level category to which the image to be identified belongs may also be determined. For example, if the second-level category to which the image to be identified belongs is determined to be bottled beverages, the first-level category to which the image to be identified belongs may also be determined to be beverages.

[0043] In the above-mentioned image recognition method, a two-level feature extraction process is performed on an image to obtain first-level and second-level category recognition results. The image's category is then determined based on these first-level and second-level category recognition results, where the second-level category is a subcategory of the first-level category. Consequently, when classifying an image, the final category can be determined using both the sub-category and the major category recognition results. This addresses the difficulty of fine-grained classification due to minimal feature differences between sub-categories within different major categories, reduces classification errors within major categories, and thus improves image recognition accuracy.

[0044] In one embodiment, the first-level category recognition result also includes a first probability corresponding to each first-level category, and the second-level category recognition result also includes a second probability corresponding to each second-level category. For any first-level category, the first probability corresponding to the first-level category represents the probability that the image to be recognized belongs to the first-level category. For any second-level category, the second probability corresponding to the second-level category represents the probability that the image to be recognized belongs to the second-level category.

[0045] In implementation, a first-level feature library and a second-level feature library can be pre-established, and the first-level features and second-level features of images of known categories can be extracted by corresponding feature extraction methods, and stored in the first-level feature library and the second-level feature library respectively. It can be understood that each feature in the first-level feature library corresponds to a known first-level category, and each feature in the second-level feature library corresponds to a known second-level category. The first-level features of the image to be identified are matched with each feature in the first-level feature library to obtain the first probability corresponding to each known first-level category, where the first probability corresponding to the first-level category is the matching degree between the first-level features of the image to be identified and the features corresponding to the first-level category. The second-level features of the image to be identified are matched with each feature in the second-level feature library to obtain the second probability corresponding to each known second-level category, where the second probability corresponding to the second-level category is the matching degree between the second-level features of the image to be identified and the features corresponding to the second-level category.

[0046] In one embodiment, the step of determining the category of the image to be identified based on the first-level category recognition results and the second-level category recognition results may specifically include the following steps: for any second-level category, determining the first probability corresponding to the first-level category to which the second-level category belongs from each first probability, as the third probability corresponding to the second-level category, and determining the recognition probability of each second-level category based on the third probability corresponding to the second-level category and the second probability; and determining the category of the image to be identified based on the recognition probability of each second-level category.

[0047] For any second-level category, the recognition probability of the second-level category represents the probability that the image to be recognized belongs to the second-level category. For example, the first-level categories in the first-level category recognition result include A and B, and the corresponding first probabilities are P A and P B The second-level categories in the second-level category recognition results include a1, a2, b1 and b2, and the corresponding second probabilities are P a1 、P a2 、P b1 and P b2 , where a1 and a2 are subcategories of A, and b1 and b2 are subcategories of B.

[0048] For the second-level category a1, the first-level category it belongs to is A, and the first probability P corresponding to the first-level category A is A That is the third probability corresponding to the second-level category a1. According to the corresponding third probability P A and the second probability P a1 Determine the recognition probability of the second-level category a1. Similarly, for the second-level category b1, the first-level category it belongs to is B, and the first probability P corresponding to the first-level category B is B That is the third probability corresponding to the second-level category b1. According to the corresponding third probability P B and the second probability P b1 Determine the recognition probability of the second-level category b1. Based on this, the recognition probabilities of the second-level categories a1, a2, b1, and b2 can be obtained respectively. Then, based on the recognition probabilities of the second-level categories a1, a2, b1, and b2, the second-level category to which the image to be recognized belongs can be determined.

[0049] In one embodiment, the step of determining the category of the image to be identified based on the recognition probabilities of the second-level categories may specifically include: determining the second-level category corresponding to the maximum recognition probability among the recognition probabilities of the second-level categories as the category of the image to be identified.

[0050] It can be understood that the greater the recognition probability of a second-level category, the more likely the image to be recognized belongs to that second-level category. Therefore, the second-level category corresponding to the highest recognition probability is determined as the category of the image to be recognized. It should be noted that once the second-level category to which the image to be recognized belongs is determined, the first-level category to which the image to be recognized belongs can also be determined. The first-level category to which the image to be recognized belongs is the first-level category to which the second-level category to which it belongs belongs.

[0051] In one embodiment, the step of determining the recognition probability of the second-level category based on the third probability and the second probability corresponding to the second-level category may specifically be to add the third probability corresponding to the second-level category to the second probability to obtain the recognition probability of the second-level category.

[0052] For example, in the aforementioned embodiment, the recognition probabilities of the second-level categories a1, a2, b1, and b2 are P respectively. A +P a1 、P A +P a2 、P B +P b1 and P B +P b2 Therefore, by correcting the second probability of the second level category by the first probability of the first level category, the problem of final classification error caused by the small difference in probability between the second level categories under different first level categories can be solved, thereby improving image recognition accuracy.

[0053] For example, the second probability P corresponding to the second-level categories a1, a2, b1 and b2 a1 、P a2 、P b1 and P b2 , respectively 0.15, 0.35, 0.4, 0.1, the first probability P corresponding to the first-level categories A and B A and P B , are 0.6 and 0.4 respectively. If we only consider the second probability of the second-level category, the second probability P corresponding to the second-level category b1 is b1 (0.4) is the largest, then the second-level category to which the image belongs is b1, and correspondingly, the first-level category to which it belongs is B. However, considering the first probability of the first-level category, the first probability P corresponding to the first-level category A is A (0.6) is greater than the first probability P corresponding to the first-level category B B (0.4), indicating that the first-level category to which the image belongs should be A, but only considering the second probability of the second-level category, the first-level category to which the image belongs is B. It can be seen that the classification result of the first-level category is wrong. The reason for the error may be the second probability P corresponding to the second-level category b1. b1(0.4) The second probability P corresponding to the second-level category a2 a2 (0.35) is relatively close, making it difficult to distinguish accurately.

[0054] If the second probability and the corresponding third probability of each second-level category are added together, the recognition probabilities corresponding to the second-level categories a1, a2, b1 and b2 are 0.75, 0.95, 0.8 and 0.5 respectively, among which the recognition probability P corresponding to the second-level category a2 is a2 (0.95) is the largest, then the second-level category to which the image belongs is determined to be a2, and correspondingly, the first-level category to which it belongs is A. The recognition result obtained based on this can limit the classification errors of the first-level category, making the final classification result more accurate.

[0055] It should be noted that the method for determining the recognition probability of the second-level category is not limited to the method in the above embodiment. For example, the third probability and the second probability corresponding to the second-level category can be weighted separately and then added together to obtain the recognition probability of the second-level category. The corresponding weight can be set according to the actual situation, or it can be determined according to the category probability in the recognition result. For example, when there are multiple second-level categories with relatively close second probabilities in the identified second-level categories (for example, the difference between the second probabilities is less than a preset threshold), it can be considered that it is difficult to accurately classify only by the second probability of the second-level category. At this time, a higher weight can be assigned to the third probability corresponding to the second-level category to help improve the classification accuracy.

[0056] In one embodiment, the step of performing feature extraction on the image to be identified and obtaining the first-level features and second-level features of the image to be identified can specifically be: performing feature extraction on the image to be identified through a feature extraction network and obtaining the first-level features and second-level features of the image to be identified; the feature extraction network is a branch network in the trained twin network.

[0057] The twin network includes two parallel branch networks (the first branch network and the second branch network respectively), and each branch network includes two-level networks (the first level network and the second level network respectively), wherein the first level network is used to extract the features of the image in the first level category dimension, and the second level network is used to extract the features of the image in the second level category dimension. The feature extraction network can be any branch network in the twin network, that is, it can be the first branch network or the second branch network. The image to be identified is input into any branch network in the twin network, and feature extraction is performed through the first level network and the second level network of the branch network to obtain the first level features and the second level features of the image to be identified.

[0058] During training, the outputs of the two first-level networks are connected to a classification network (the first classification network). The first classification network is used to fuse the image features output by the two first-level networks and identify the first-level category of the image based on the fused features. The outputs of the two second-level networks are connected to another classification network (the second classification network). The second classification network is used to fuse the image features output by the two second-level networks and identify the second-level category of the image based on the fused features. Two sample images are input into the two branch networks respectively, and the two-level networks learn the image features separately. For each feature, the two-level networks focus on different angles, which helps to strengthen the network's ability to learn and express image features.

[0059] In one embodiment, Figure 2 As shown, the training process of the twin network includes the following steps S202 to S212.

[0060] S202 , obtaining a sample image and its first-level label category and second-level label category, where the sample image includes a first sample image and a second sample image.

[0061] The sample images are images of known true categories. The first-level label category of the sample images refers to the true category of the sample images in the first-level category dimension, and the second-level label category of the sample images refers to the true category of the sample images in the second-level category dimension. The first and second sample images are randomly selected and can belong to the same or different categories. In implementation, data augmentation can also be performed on the samples. Specifically, richer sample images can be obtained through processes such as image flipping, random cropping, and pixel normalization.

[0062] S204, performing first-level feature extraction on the first sample image through the first-level network of the first branch network to obtain first-level features of the first sample image, and performing first-level feature extraction on the second sample image through the first-level network of the second branch network to obtain second-level features of the second sample image.

[0063] like Figure 3As shown, a structural diagram of a twin network in an embodiment is provided. The first-level network of the first branch network includes a first backbone network and a first-level feature re-extraction network. Among them, the first backbone network can use the SENet network, and the SE module is introduced in the SENet network. The processing process of the first backbone network can be specifically as follows: first, a convolution operation is performed on the first sample image to obtain a feature map, and a compression (Squeeze) operation is performed on the feature map to obtain a global feature at the channel level, and then an activation (Excitation) operation is performed on the global feature to learn the relationship between each channel and obtain the weights of different channels. Then, each weight is multiplied by the features of the corresponding channel in the above feature map to obtain an output feature map. It can be understood that the SE module performs attention (attention) or gating (gating) operations on the channel dimension. This attention mechanism allows the network to pay more attention to channel features with larger information content and suppress unimportant channel features. The first-level feature re-extraction network in the first branch network is connected to the output feature map of the first backbone network, and may include a convolution layer, a global average pooling layer, and a fully connected layer. After performing convolution processing, global average pooling processing, and fully connected processing on the output feature map, the first-level feature (B6_1) of the first sample image is output.

[0064] The first-level network of the second branch network includes a second backbone network and a first-level feature re-extraction network. Among them, the structure and processing process of the second backbone network can be the same as those of the first backbone network, and will not be repeated here. The second backbone network and the first backbone network can also reduce network parameters and improve network training speed by sharing weights. The first-level feature re-extraction network in the second branch network is connected to the output feature map of the second backbone network, and can include a convolution layer, a global average pooling layer and a fully connected layer. After the output feature map is convolutionally processed, globally averaged pooled and fully connected, the first-level features (B6_2) of the second sample image are output.

[0065] S206 , fusing the first-level features of the first sample image and the second sample image through the first classification network to obtain first-level fused features, and performing mapping based on the first-level fused features to obtain first-level predicted categories of the first sample image and the second sample image.

[0066] Among them, the step of fusing the first-level features of the first sample image and the second sample image to obtain the first-level fused features can specifically include the following steps: multiplying the first-level features of the first sample image and the second sample image by corresponding parameters respectively to obtain the first-level weighted features of the first sample image and the second sample image; splicing the first-level weighted features of the first sample image and the second sample image to obtain the first-level fused features.

[0067] In one embodiment, the number of samples of the category to which the first sample image belongs is smaller than the number of samples of the category to which the second sample image belongs. The parameters corresponding to the first-level features of the first sample image increase with increasing iterations, while the parameters corresponding to the first-level features of the second sample image decrease with increasing iterations. Accordingly, as the number of iterations increases, the network will relatively strengthen the learning of features of categories with fewer samples and relatively weaken the learning of features of categories with more samples, thereby addressing the problem of sample data imbalance.

[0068] In one embodiment, the step of multiplying the first-level features of the first sample image and the second sample image by corresponding parameters to obtain the first-level weighted features of the first sample image and the second sample image can specifically include the following steps: multiplying the first-level features of the first sample image by a first parameter to obtain the first-level weighted features of the first sample image, and multiplying the first-level features of the second sample image by a second parameter to obtain the first-level weighted features of the second sample image, wherein the first parameter is determined based on the ratio of the current number of iterations to the total number of iterations, and the second parameter is determined based on the difference between a preset value and the first parameter. Specifically, the ratio of the current number of iterations to the total number of iterations can be used as the parameter corresponding to the first-level feature of the first sample image, i.e., the first parameter (represented by a), the preset value is set to 1, and 1-a is used as the parameter corresponding to the first-level feature of the second sample image, i.e., the second parameter.

[0069] like Figure 3 As shown, the first classification network includes a feature fusion layer and a fully connected layer. The first-level feature (B6_1) of the first sample image is multiplied by parameter a to obtain the first-level weighted feature of the first sample image. The first-level feature (B6_2) of the second sample image is multiplied by parameter 1-a to obtain the first-level weighted feature of the second sample image. The first-level weighted features of the first sample image and the second sample image are concatenated (concat) to output the first-level fusion feature (B11_1). The first-level fusion feature (B11_1) is fully connected and activated (Softmax) to output the first-level predicted category of the first sample image and the second sample image.

[0070] S208, performing second-level feature extraction on the first sample image through the second-level network of the first branch network to obtain the second-level features of the first sample image, and performing second-level feature extraction on the second sample image through the second-level network of the second branch network to obtain the second-level features of the second sample image.

[0071] like Figure 3As shown, the second-level network of the first branch network includes a first backbone network and a second-level feature re-extraction network, wherein the first backbone network serves as a shared part of the second-level network and the first-level network of the first branch network. The second-level feature re-extraction network in the first branch network is connected to the output feature map of the first backbone network, and may include a convolution layer, a global average pooling layer, and a fully connected layer. After performing convolution processing, global average pooling processing, and fully connected processing on the output feature map, the second-level feature (B10_1) of the first sample image is output.

[0072] The second-level network of the second branch network includes a second backbone network and a second-level feature re-extraction network. The second backbone network serves as a shared part of the second-level network and the first-level network of the second branch network. The second-level feature re-extraction network in the second branch network is connected to the output feature map of the second backbone network and may include a convolution layer, a global average pooling layer, and a fully connected layer. After performing convolution processing, global average pooling processing, and fully connected processing on the output feature map, the second-level feature (B10_2) of the second sample image is output.

[0073] S210, fusing the second-level features of the first sample image and the second sample image through a second classification network to obtain second-level fused features, and performing mapping based on the second-level fused features to obtain second-level predicted categories of the first sample image and the second sample image.

[0074] Among them, the step of fusing the second-level features of the first sample image and the second sample image to obtain the second-level fused features can specifically include the following steps: multiplying the second-level features of the first sample image and the second sample image by corresponding parameters respectively to obtain the second-level weighted features of the first sample image and the second sample image; splicing the second-level weighted features of the first sample image and the second sample image to obtain the second-level fused features.

[0075] In one embodiment, the number of samples of the category to which the first sample image belongs is smaller than the number of samples of the category to which the second sample image belongs. The parameters corresponding to the second-level weighted features of the first sample image increase with increasing iterations, while the parameters corresponding to the second-level weighted features of the second sample image decrease with increasing iterations. Accordingly, as the number of iterations increases, the network will relatively strengthen the learning of features for categories with fewer samples and relatively weaken the learning of features for categories with more samples, thereby addressing the problem of sample data imbalance.

[0076] In one embodiment, the step of multiplying the second-level features of the first sample image and the second sample image by corresponding parameters to obtain the second-level weighted features of the first sample image and the second sample image can specifically include the following steps: multiplying the second-level features of the first sample image by a third parameter to obtain the second-level weighted features of the first sample image, and multiplying the second-level features of the second sample image by a fourth parameter to obtain the second-level weighted features of the second sample image, wherein the third parameter is determined based on the ratio of the current number of iterations to the total number of iterations, and the fourth parameter is determined based on the difference between a preset value and the third parameter. Specifically, the ratio of the current number of iterations to the total number of iterations can be used as the parameter corresponding to the second-level feature of the first sample image, i.e., the third parameter (represented by a), the preset value is set to 1, and 1-a is used as the parameter corresponding to the second-level feature of the second sample image, i.e., the fourth parameter. It can be understood that the third parameter and the fourth parameter in this embodiment are the same as the first parameter and the second parameter in the previous embodiment.

[0077] like Figure 3 As shown, the second classification network includes a feature fusion layer and a fully connected layer. The second-level feature (B10_1) of the first sample image is multiplied by parameter a to obtain the second-level weighted feature of the first sample image. The second-level feature (B10_2) of the second sample image is multiplied by parameter 1-a to obtain the second-level weighted feature of the second sample image. The second-level weighted features of the first sample image and the second sample image are concatenated (concat) to output the second-level fused feature (B11_2). The second-level fused feature (B11_2) is fully connected and activated (Softmax) to output the second-level predicted category of the first sample image and the second sample image.

[0078] S212, based on the error between the first-level prediction category and the corresponding first-level label category, and the error between the second-level prediction category and the corresponding second-level label category, adjust the parameters of the twin network until the iteration end condition is met to obtain a trained twin network.

[0079] Among them, the iteration end condition can be reaching a preset total number of iterations. When the iteration end condition is not met, the error between the first-level prediction category of the sample image and the corresponding first-level label category is calculated by the loss function to obtain a first loss value (loss1), and the error between the second-level prediction category of the sample image and the corresponding second-level label category is calculated by the loss function to obtain a second loss value (loss2). The first loss value (loss1) and the second loss value (loss2) are added to obtain a total loss value (loss). Based on the total loss value (loss), the network parameters of the twin network are adjusted, and then the iteration is returned and re-iterated until the iteration end condition is met to obtain a trained twin network.

[0080] It can be understood that the first loss value (loss1) is jointly determined by the error between the first-level predicted category of the first sample image and the corresponding first-level label category, and the error between the first-level predicted category of the second sample image and the corresponding first-level class label category. The second loss value (loss2) is jointly determined by the error between the second-level predicted category of the first sample image and the corresponding second-level label category, and the error between the second-level predicted category of the second sample image and the corresponding second-level label category.

[0081] For example, if the first-level label category of the first sample image is A (i.e., the true probability of category A is 1), and the probability of belonging to A in the first-level predicted category of the first sample image is 0.3 (i.e., the predicted probability of category A is 0.3), then the error between the first-level predicted category of the first sample image and the corresponding first-level label category is 0.7; if the first-level label category of the second sample image is B (i.e., the true probability of category B is 1), and the probability of belonging to B in the first-level predicted category of the second sample image is 0.4 (i.e., the predicted probability of category B is 0.4), then the error between the first-level predicted category of the second sample image and the corresponding first-level label category is 0.6; the loss function and the above two errors are used to perform corresponding calculations to obtain the first loss value (loss1). Among them, the loss function can be any loss function that is currently available or may appear in the future. The second loss value (loss2) can be obtained using a similar method, which will not be repeated here.

[0082] In the above embodiment, through the mechanism of combining the twin network with the two-level classification strategy, the feature learning and expression capabilities of the network can be enhanced to solve the problem of difficulty in fine classification due to the small feature differences between small categories under different major categories, reduce classification errors in major categories, and strengthen the learning of category features with a small number of samples by weighted fusion of the features output by the two branch networks to solve the problem of inability to accurately identify categories with a small number of samples due to unbalanced sample data, thereby improving network recognition accuracy.

[0083] It should be noted that the twin network structure used in this embodiment is not limited to Figure 3 The structure shown can be adjusted accordingly, for example, by reducing or increasing the number of convolutional layers, replacing the backbone network with a convolutional network, using feature addition as a feature fusion method, etc., which can also achieve the purpose of this embodiment.

[0084] It should be understood that, although the various steps in the various flow charts that the above-described embodiments relate to are shown in sequence according to the indications of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the various flow charts that the above-described embodiments relate to can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0085] In one embodiment, Figure 4 As shown, an image recognition device is provided, including: an acquisition module 410, a feature extraction module 420, a recognition module 430 and a determination module 440, wherein:

[0086] The acquisition module 410 is used to acquire an image to be recognized.

[0087] The feature extraction module 420 is used to extract features of the image to be identified, and obtain first-level features and second-level features of the image to be identified.

[0088] The recognition module 430 is used to obtain the first-level category recognition result and the second-level category recognition result of the image to be recognized based on the first-level features and the second-level features, respectively. The first-level category recognition result includes the recognized first-level categories, and the second-level category recognition result includes the recognized second-level categories. The second-level category is a subcategory of the first-level category.

[0089] The determination module 440 is configured to determine the category of the image to be identified based on the first-level category identification result and the second-level category identification result.

[0090] In one embodiment, the first-level category recognition result also includes a first probability corresponding to each first-level category, and the second-level category recognition result also includes a second probability corresponding to each second-level category; when the determination module 440 determines the category of the image to be recognized based on the first-level category recognition result and the second-level category recognition result, it is specifically used to: for any second-level category, determine the first probability corresponding to the first-level category to which the second-level category belongs from each first probability, as the third probability corresponding to the second-level category, and determine the recognition probability of each second-level category based on the third probability and the second probability corresponding to the second-level category; determine the category of the image to be recognized based on the recognition probability of each second-level category.

[0091] In one embodiment, when determining the recognition probability of the second-level category based on the third probability and the second probability corresponding to the second-level category, the determination module 440 is specifically configured to add the third probability corresponding to the second-level category to the second probability to obtain the recognition probability of the second-level category.

[0092] In one embodiment, when determining the category of the image to be identified based on the recognition probabilities of the second-level categories, the determination module 440 is specifically configured to: determine the second-level category corresponding to the maximum recognition probability among the recognition probabilities of the second-level categories as the category of the image to be identified.

[0093] In one embodiment, when the feature extraction module 420 performs feature extraction on the image to be identified and obtains the first-level features and second-level features of the image to be identified, it is specifically used to: perform feature extraction on the image to be identified through a feature extraction network to obtain the first-level features and second-level features of the image to be identified; the feature extraction network is a branch network in the trained twin network.

[0094] In one embodiment, the device further includes a training module for training a trained twin network. The twin network includes a first branch network and a second branch network, each of which includes a first-level network and a second-level network; the outputs of the two first-level networks are connected to the first classification network, and the outputs of the two second-level networks are connected to the second classification network. The training module includes: an acquisition unit, a first feature extraction unit, a first classification unit, a second feature extraction unit, a second classification unit, and an adjustment unit.

[0095] The acquisition unit is used to acquire a sample image and its first-level label category and second-level label category, where the sample image includes a first sample image and a second sample image.

[0096] The first feature extraction unit is used to perform first-level feature extraction on the first sample image through the first-level network of the first branch network to obtain the first-level features of the first sample image, and to perform first-level feature extraction on the second sample image through the first-level network of the second branch network to obtain the second-level features of the second sample image.

[0097] The first classification unit is used to fuse the first-level features of the first sample image and the second sample image through a first classification network to obtain first-level fused features, and perform mapping based on the first-level fused features to obtain first-level predicted categories of the first sample image and the second sample image.

[0098] The second feature extraction unit is used to perform second-level feature extraction on the first sample image through the second-level network of the first branch network to obtain the second-level features of the first sample image, and to perform second-level feature extraction on the second sample image through the second-level network of the second branch network to obtain the second-level features of the second sample image.

[0099] The second classification unit is used to fuse the second-level features of the first sample image and the second sample image through a second classification network to obtain second-level fused features, and perform mapping based on the second-level fused features to obtain second-level predicted categories of the first sample image and the second sample image.

[0100] An adjustment unit is used to adjust the parameters of the twin network based on the error between the first-level prediction category and the corresponding first-level label category, and the error between the second-level prediction category and the corresponding second-level label category, until the iteration end condition is met to obtain a trained twin network.

[0101] In one embodiment, when the first classification unit fuses the first-level features of the first sample image and the second sample image to obtain the first-level fused features, it is specifically used to: multiply the first-level features of the first sample image and the second sample image by corresponding parameters respectively to obtain the first-level weighted features of the first sample image and the second sample image; and splice the first-level weighted features of the first sample image and the second sample image to obtain the first-level fused features.

[0102] In one embodiment, when the second classification unit fuses the second-level features of the first sample image and the second sample image to obtain the second-level fused features, it is specifically used to: multiply the second-level features of the first sample image and the second sample image by corresponding parameters respectively to obtain the second-level weighted features of the first sample image and the second sample image; and splice the second-level weighted features of the first sample image and the second sample image to obtain the second-level fused features.

[0103] In one embodiment, the number of samples of the category to which the first sample image belongs is smaller than the number of samples of the category to which the second sample image belongs, and the parameters corresponding to the first-level features and the second-level features of the first sample image increase as the number of iterations increases, while the parameters corresponding to the first-level features and the second-level features of the second sample image decrease as the number of iterations increases.

[0104] In one embodiment, when the first classification unit multiplies the first-level features of the first sample image and the second sample image by corresponding parameters respectively to obtain the first-level weighted features of the first sample image and the second sample image, it is specifically used to: multiply the first-level features of the first sample image by the first parameter to obtain the first-level weighted features of the first sample image, and multiply the first-level features of the second sample image by the second parameter to obtain the first-level weighted features of the second sample image; wherein the first parameter is determined according to the ratio of the current number of iterations to the total number of iterations, and the second parameter is determined according to the difference between a preset value and the first parameter.

[0105] In one embodiment, when the second classification unit multiplies the second-level features of the first sample image and the second sample image by corresponding parameters respectively to obtain the second-level weighted features of the first sample image and the second sample image, it is specifically used to: multiply the second-level features of the first sample image by the first parameter to obtain the second-level weighted features of the first sample image, and multiply the second-level features of the second sample image by the second parameter to obtain the second-level weighted features of the second sample image; wherein the first parameter is determined according to the ratio of the current number of iterations to the total number of iterations, and the second parameter is determined according to the difference between a preset value and the first parameter.

[0106] For the specific definition of the image recognition device, please refer to the definition of the image recognition method above and will not be repeated here. Each module in the above-mentioned image recognition device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0107] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements an image recognition method.

[0108] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image recognition method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0109] Those skilled in the art will understand that Figure 5 or Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0110] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0111] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0112] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0113] It should be understood that the terms "first," "second," and the like in the above embodiments are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features being described. In describing a numerical range, the term "plurality" means more than one, i.e., equal to or greater than two.

[0114] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0115] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An image recognition method, characterized in that: The method comprises: Obtain the image to be recognized; Performing feature extraction on the image to be identified through a feature extraction network to obtain first-level features and second-level features of the image to be identified; the feature extraction network is a branch network in the trained twin network; Obtaining a first-level category recognition result and a second-level category recognition result for the image to be recognized based on the first-level features and the second-level features, respectively, wherein the first-level category recognition result includes each identified first-level category, and the second-level category recognition result includes each identified second-level category, where the second-level category is a subcategory of the first-level category; The category of the image to be identified is determined according to the first-level category identification result and the second-level category identification result.

2. The method according to claim 1, characterized in that The first-level category recognition result further includes a first probability corresponding to each of the first-level categories, and the second-level category recognition result further includes a second probability corresponding to each of the second-level categories; Determining the category of the image to be identified based on the first-level category identification result and the second-level category identification result includes: For any second-level category, determining, from each of the first probabilities, a first probability corresponding to the first-level category to which the second-level category belongs, as a third probability corresponding to the second-level category; and determining an identification probability for each second-level category based on the third probability corresponding to the second-level category and the second probability; The category of the image to be identified is determined according to the recognition probability of each of the second-level categories.

3. The method according to claim 2, characterized in that Determining the recognition probability of the second-level category according to the third probability corresponding to the second-level category and the second probability includes: The third probability corresponding to the second-level category is added to the second probability to obtain the recognition probability of the second-level category.

4. The method according to claim 2, characterized in that Determining the category of the image to be identified based on the recognition probability of each of the second-level categories includes: The second-level category corresponding to the maximum recognition probability among the recognition probabilities of the second-level categories is determined as the category of the image to be recognized.

5. The method according to claim 1, wherein The twin network includes a first branch network and a second branch network, and the first branch network and the second branch network both include a first-level network and a second-level network; the outputs of the two first-level networks are connected to the first classification network, and the outputs of the two second-level networks are connected to the second classification network; The training process of the twin network includes: Acquire a sample image and its first-level label category and second-level label category, wherein the sample image includes a first sample image and a second sample image; Performing first-level feature extraction on the first sample image through the first-level network of the first branch network to obtain first-level features of the first sample image, and performing first-level feature extraction on the second sample image through the first-level network of the second branch network to obtain first-level features of the second sample image; fusing the first-level features of the first sample image and the second sample image through the first classification network to obtain first-level fused features, and performing mapping based on the first-level fused features to obtain first-level predicted categories of the first sample image and the second sample image; Performing second-level feature extraction on the first sample image through the second-level network of the first branch network to obtain second-level features of the first sample image, and performing second-level feature extraction on the second sample image through the second-level network of the second branch network to obtain second-level features of the second sample image; fusing the second-level features of the first sample image and the second sample image through the second classification network to obtain second-level fused features, and performing mapping based on the second-level fused features to obtain second-level predicted categories of the first sample image and the second sample image; Based on the error between the first-level prediction category and the corresponding first-level label category, and the error between the second-level prediction category and the corresponding second-level label category, the parameters of the twin network are adjusted until the iteration end condition is met to obtain a trained twin network.

6. The method according to claim 5, characterized in that Fusing the first-level features of the first sample image and the second sample image to obtain first-level fused features, including: Multiplying the first-level features of the first sample image and the second sample image by corresponding parameters respectively to obtain first-level weighted features of the first sample image and the second sample image; splicing the first-level weighted features of the first sample image and the second sample image to obtain a first-level fusion feature; and / or, Fusing the second-level features of the first sample image and the second sample image to obtain second-level fused features, including: Multiplying the second-level features of the first sample image and the second sample image by corresponding parameters respectively to obtain second-level weighted features of the first sample image and the second sample image; The second-level weighted features of the first sample image and the second sample image are concatenated to obtain a second-level fusion feature.

7. The method according to claim 6, characterized in that The number of samples of the category to which the first sample image belongs is smaller than the number of samples of the category to which the second sample image belongs, and the parameters corresponding to the first-level features and the second-level features of the first sample image increase as the number of iterations increases, while the parameters corresponding to the first-level features and the second-level features of the second sample image decrease as the number of iterations increases.

8. The method according to claim 6, characterized in that Multiplying the first-level features of the first sample image and the second sample image by corresponding parameters respectively to obtain the first-level weighted features of the first sample image and the second sample image, including: Multiplying the first-level features of the first sample image by a first parameter to obtain the first-level weighted features of the first sample image, and multiplying the first-level features of the second sample image by a second parameter to obtain the first-level weighted features of the second sample image; and / or, Multiplying the second-level features of the first sample image and the second sample image by corresponding parameters respectively to obtain second-level weighted features of the first sample image and the second sample image, including: Multiplying the second-level feature of the first sample image by a first parameter to obtain a second-level weighted feature of the first sample image, and multiplying the second-level feature of the second sample image by a second parameter to obtain a second-level weighted feature of the second sample image; The first parameter is determined according to the ratio of the current number of iterations to the total number of iterations, and the second parameter is determined according to the difference between a preset value and the first parameter.

9. An image recognition device, characterized in that: The device comprises: An acquisition module, used to acquire an image to be identified; A feature extraction module is used to extract features of the image to be identified through a feature extraction network to obtain first-level features and second-level features of the image to be identified; the feature extraction network is a branch network in the trained twin network; an identification module, configured to obtain a first-level category identification result and a second-level category identification result of the image to be identified based on the first-level features and the second-level features, respectively, wherein the first-level category identification result includes each identified first-level category, and the second-level category identification result includes each identified second-level category, where the second-level category is a subcategory of the first-level category; A determination module is used to determine the category of the image to be identified based on the first-level category identification result and the second-level category identification result.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images