Identification method and device

Abstract

This application discloses an identification method, device, electronic device, and storage medium. The method involves extracting features from an image to be identified to obtain a feature map of the image. The feature map contains feature information of multiple regions in the image to be identified, and the feature information of different regions has different effects on the identification result of the image to be identified. The feature information of multiple regions in the feature map of the image to be identified is processed to obtain the identification result of the image to be identified.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] The difference and challenge of fine-grained image recognition compared to general image recognition lies in the finer granularity of the image's category. For example, general image recognition identifies whether an animal in an image is a dog or a kangaroo, while fine-grained image recognition identifies whether a dog in an image is a husky or an Eskimo dog.

[0003] Current fine-grained image recognition requires extracting global features and multiple local features from the image to be recognized, then fusing the global and local features and using the fused features for recognition. This recognition method results in low efficiency for fine-grained image recognition. Summary of the Invention

[0004] The purpose of this application is to provide an identification method, apparatus, electronic device, and storage medium, including the following technical solutions:

[0005] An identification method, the method comprising:

[0006] Feature extraction is performed on the image to be recognized to obtain a feature map of the image to be recognized; the feature map contains feature information of multiple regions in the image to be recognized, and the feature information of different regions has different effects on the recognition result of the image to be recognized;

[0007] The feature information of multiple regions of the image to be identified in the feature map is processed to obtain the recognition result of the image to be identified.

[0008] Optionally, the above method includes the process of extracting features from the image to be identified and processing the feature information of multiple regions in the feature map of the image to be identified, which includes:

[0009] The feature extraction module of the target recognition model extracts features from the image to be recognized, and obtains the feature map of the image to be recognized.

[0010] The recognition module of the target recognition model processes the feature information of multiple regions of the image to be recognized in the feature map to obtain the recognition result of the image to be recognized.

[0011] The target recognition model is trained based on multi-level images of the sample images; the first-level image in the multi-level images is the sample image, and the non-first-level images in the multi-level images are obtained by masking the previous-level image of the non-first-level images.

[0012] Optionally, the target recognition model is trained in the following manner:

[0013] For any level image in the multi-level images, the feature extraction module in the recognition model corresponding to the level image extracts features of the level image to obtain the feature map of the level image. The recognition module in the recognition model corresponding to the level image processes the feature information of multiple regions of the level image in the feature map of the level image to obtain the recognition result of the level image.

[0014] Wherein, if any level image is a non-first level image in the multi-level images, the any level image is obtained by masking the previous level image based on the feature map of the previous level image of the any level image;

[0015] With the goal of making the recognition result of the first-level image in the multi-level image approach the category label of the sample image, and making the recognition result of each non-first-level image in the multi-level image approach the recognition result of the first-level image in the multi-level image, the parameters of the recognition model corresponding to each level of image are updated until the target recognition model is obtained.

[0016] Optionally, the process of masking the parent image of any given level image based on the feature map of the parent image of any given level image includes the following steps:

[0017] The feature map of the previous level image of any level image is processed by the category response model to obtain the category response map of the previous level image of any level image.

[0018] The image above the sample image is masked based on the category label of the sample image and the category response map of the image above the previous level of the image above the previous level to obtain the image above the previous level.

[0019] Optionally, in the above method, the masking process of the parent image of the arbitrary level image based on the category label of the sample image and the category response map of the parent image of the arbitrary level image includes:

[0020] Based on the category response map of the previous level image of any given level image and the category label, a target region is determined in the category response map of the previous level image of any given level image; the correlation between the elements in the target region and the category label is greater than the correlation between the elements in the non-target region and the category label.

[0021] The region corresponding to the target region in the previous level image of any given level image is masked to obtain the given level image.

[0022] Optionally, in the above method, determining the target region in the category response map of the parent image of the any-level image based on the category response map of the parent image of the any-level image and the category label includes:

[0023] Obtain the category response sub-map corresponding to the channel of the category represented by the category label from the category response map of the previous level image of any level image;

[0024] Elements in the category response subgraph whose values ​​are greater than or equal to the target value are identified as target elements; the regions containing all target elements constitute the target region.

[0025] Optionally, in the above method, when the category response model processes the feature map of the previous level image of any level image, it is used to:

[0026] Based on the weights of each channel of the feature map of the previous level image of the above-level image corresponding to the same category, the sub-feature maps of each channel of the feature map of the previous level image of the above-level ...

[0027] Different categories correspond to different channels in the category response map of the previous level image of any given level image.

[0028] The above methods may optionally include:

[0029] Based on the weights of each channel of the feature map of the previous level image of any given level image corresponding to the same category, and the feature map of the previous level image of any given level image, the probability distribution of each category corresponding to the previous level image of any given level image is obtained.

[0030] With the goal of making the probability distribution of each category in the previous level image of any level image approach the category label, the parameters of the category response model are updated. The parameters of the category response model include the weights of each channel of the feature map of the previous level image of any level image corresponding to the same category.

[0031] Optionally, the method described above involves updating the parameters of the recognition model corresponding to each level of the image, with the goal of ensuring that the recognition result of the first-level image in the multi-level image approximates the category label of the sample image, and that the recognition results of each non-first-level image in the multi-level image approximate the recognition result of the first-level image in the multi-level image. This update includes:

[0032] Obtain the recognition result of the first-level image in the multi-level image and the first loss value of the category label, and the recognition result of each non-first-level image in the multi-level image and the second loss value of the recognition result of the first-level image in the multi-level image;

[0033] The first loss value and each of the second loss values ​​are weighted and summed to obtain the comprehensive loss value; wherein, if the number of iterations is less than the target number of iterations, the weight of the second loss value is positively correlated with the number of iterations; if the number of iterations is greater than or equal to the target number of iterations, the weight of the second loss value is the target weight;

[0034] With the goal of minimizing the overall loss value, the parameters of the recognition model corresponding to each level of image are updated.

[0035] An identification device, the device comprising:

[0036] The extraction unit is used to extract features from the image to be recognized to obtain a feature map; the feature map contains feature information of multiple regions in the image to be recognized, and the feature information of different regions has different effects on the recognition result of the image to be recognized;

[0037] The processing unit is used to process the feature information of multiple regions of the image to be identified in the feature map to obtain the recognition result of the image to be identified.

[0038] An electronic device, comprising:

[0039] Memory, used to store programs;

[0040] A processor is configured to invoke and execute the program in the memory, thereby implementing the various steps of the identification method as described in any of the preceding claims.

[0041] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the identification method as described in any of the preceding claims.

[0042] As can be seen from the above scheme, the recognition method, apparatus, electronic device, and storage medium provided in this application extract features from the image to be recognized, obtaining a feature map of the image to be recognized. This feature map contains feature information of multiple regions in the image to be recognized, and the feature information of different regions has different effects on the recognition result of the image to be recognized. The feature information of multiple regions in the image to be recognized in the above feature map is processed to obtain the recognition result of the image to be recognized. This application obtains the features of multiple regions with different effects on the recognition result of the image to be recognized through a single feature extraction, eliminating the need for multiple feature extractions, thereby improving the recognition efficiency of fine-grained images. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figures 1a-1c The three sets of images to be subjected to fine-grained image recognition are provided for the embodiments of this application;

[0045] Figure 2 A flowchart illustrating one implementation of the identification method provided in this application embodiment;

[0046] Figure 3 This is a flowchart illustrating an implementation of masking a higher-level image based on a feature map of the higher-level image of any of the above-mentioned images, as provided in an embodiment of this application.

[0047] Figure 4 A flowchart illustrating an implementation of masking a parent image of any given level image based on category labels of sample images and category response maps of the parent image of any given level image, as provided in this application embodiment.

[0048] Figure 5 A schematic diagram of a network architecture for obtaining a target recognition model based on multi-level image training of sample images, provided in an embodiment of this application;

[0049] Figure 6 A schematic diagram of the structure of the identification device provided in the embodiments of this application;

[0050] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0052] Compared to images to be subjected to general image recognition, images to be subjected to fine-grained image recognition are characterized by significant intra-class differences and subtle inter-class differences. For example... Figures 1a-1c As shown, these are three sets of images to be subjected to fine-grained image recognition according to an embodiment of this application. Figure 1aThe birds in the three images shown for fine-grained image recognition belong to the same fine-grained category. Figure 1b The food items in the three images shown for fine-grained image recognition belong to the same fine-grained category. Figure 1c The cars in the three images shown belong to different fine-grained categories, from... Figure 1a and 1b It can be seen that objects belonging to the same fine-grained category differ significantly in their overall appearance. Figure 1c It can be seen that objects belonging to different fine-grained categories are very similar in overall appearance. For example, SUVs and vans are often confused, leading to recognition errors. Images to be recognized with fine granularity share common semantic parts. For example, birds have beaks, wings, and claws; cars have tires, windshields, and bumpers. Current fine-grained image recognition methods, in addition to extracting global features from the image to be recognized, also find common semantic parts, extract features from each found part as local features, and finally fuse the global and local features for final recognition. This recognition method has significant drawbacks:

[0053] 1. When performing feature extraction, it is necessary to locate multiple semantic parts and then perform global and local fusion strategies, which greatly improves the feature extraction time.

[0054] 2. If the recognition method is implemented through a trained recognition model, the training of the recognition model requires the annotation of semantic parts in the sample images or the use of weak supervision to locate multiple semantic part regions.

[0055] like Figure 2 The diagram shown is a flowchart of one implementation of the identification method provided in this application, which may include:

[0056] Step S201: Perform feature extraction on the image to be recognized to obtain a feature map of the image to be recognized. This feature map contains feature information of multiple regions in the image to be recognized, and the feature information of different regions has different effects on the recognition result of the image to be recognized.

[0057] The image to be identified can be a color image, such as an RGB image.

[0058] The image to be identified can also be a grayscale image.

[0059] Suppose that the feature map of the image to be identified contains feature information of N regions (N is a positive integer greater than 2) in the image to be identified, and the above N regions constitute the image to be identified. Among them, the i-th region (i = 1, 2, 3, ..., N) can be an independent connected region, or it can be composed of at least two independent sub-regions.

[0060] The influence of feature information from different regions on the recognition result of the image to be recognized varies. Specifically, the greater the influence of feature information from any region on the recognition result, the greater the contribution of feature information from that region to the recognition result; conversely, the smaller the influence of feature information from any region on the recognition result, the smaller the contribution of feature information from that region to the recognition result.

[0061] Step S202: Process the feature information of multiple regions of the image to be identified in the feature map to obtain the recognition result of the image to be identified.

[0062] Optionally, the feature map of the image to be recognized can be pooled to obtain the pooling result; the pooling result can then be fully connected to obtain the recognition result of the image to be recognized.

[0063] The recognition method provided in this application obtains features of multiple regions with different effects on the recognition result of the image to be recognized through a single feature extraction, eliminating the need for multiple feature extractions and thus improving the recognition efficiency of fine-grained images.

[0064] In an optional embodiment, the process of extracting features from the image to be identified and processing the feature information of multiple regions of the image to be identified in the feature map can be implemented by a target recognition model.

[0065] The target recognition model can be trained based on multi-level images of the sample images. The first-level image in the multi-level images of the sample images is the sample image, and the non-first-level images in the multi-level images of the sample images are obtained by masking the previous-level image of the non-first-level image.

[0066] Optionally, the target recognition model includes a feature extraction module and a recognition module.

[0067] The feature extraction module of the target recognition model can be used to extract features from the image to be recognized, thus obtaining a feature map of the image. As an example, the feature extraction module can be an Embedding network, or it can be a Transformer network.

[0068] The recognition module of the target recognition model processes the feature information of multiple regions of the image to be recognized in the feature map to obtain the recognition result of the image to be recognized. As an example, the recognition module may include a pooling network and a fully connected network; wherein, the pooling network is used to perform pooling processing on the feature map of the image to be recognized to obtain the pooling processing result; the fully connected network is used to perform full connection processing on the pooling processing result to obtain the recognition result of the image to be recognized.

[0069] In an optional embodiment, the process of obtaining the target recognition model through multi-level image training based on sample images may include:

[0070] For any level of the multi-level images in the sample image, the feature extraction module in the recognition model corresponding to that level of image extracts features to obtain the feature map of that level of image. The recognition module in the recognition model corresponding to that level of image then processes the feature information of multiple regions in the feature map of that level of image to obtain the recognition result of that level of image.

[0071] If the image at any level is a non-first-level image in a multi-level image of the sample image, the image at any level is obtained by masking the image at the previous level based on the feature map of the image at the previous level.

[0072] The goal is to make the recognition result of the first-level image in the multi-level images of the sample image approach the category label of the sample image, and to make the recognition result of each non-first-level image in the multi-level images of the sample image approach the recognition result of the first-level image in the multi-level images of the sample image. The parameters of the recognition model corresponding to each level of image are updated until the target recognition model is obtained.

[0073] In the training process of the recognition model, this application does not require the annotation of semantic parts in the sample images, nor does it require the use of weak supervision to locate multiple semantic part regions. Therefore, training samples are easy to obtain, and the training process is simplified.

[0074] Optionally, the recognition result of the first-level image in the multi-level images and the first loss value of the category label can be obtained, as well as the recognition result of each non-first-level image in the multi-level images and the second loss value of the recognition result of the first-level image in the multi-level images.

[0075] The first loss value can be the cross-entropy between the recognition result of the first-level image in the multi-level images of the sample image and the category label of the sample image. It can be expressed by the formula:

[0076] L g (x,y)=L ce (h(f(x;θ);φ1),y) (1)

[0077] Among them, L g (x, y) represents the first loss value; x represents the first-level image; y represents the class label of the sample image; L ce(,) represents cross-entropy; h(f(x;θ);φ1) represents the output of the recognition module in the recognition model corresponding to the first-level image; φ1 represents the parameters of the recognition module in the recognition model corresponding to the first-level image; f(x;θ) represents the feature map of the first-level image; θ represents the parameters of the feature extraction module in the recognition model corresponding to the first-level image.

[0078] The second loss value between the recognition result of any non-first-level image and the recognition result of the first-level image in the aforementioned multi-level images can be the KL dispersion between the recognition result of any non-first-level image and the recognition result of the first-level image in the aforementioned multi-level images. This can be expressed by the formula:

[0079]

[0080] in, The second loss value represents the recognition result of any non-first-level image compared to the recognition result of the first-level image among the above multi-level images; x represents the above first-level image; D represents any non-first-level image; kl Let f(x; θ) represent the KL divergence; h(f(x; θ); φ1) represent the output of the recognition module in the recognition model corresponding to the first-level image; φ1 represent the parameters of the recognition module in the recognition model corresponding to the first-level image; f(x; θ) represent the feature map of the first-level image; θ represents the parameters of the feature extraction module in the recognition model corresponding to the first-level image. φ represents the output of the recognition module in the recognition model corresponding to any of the above non-first-level images; p This represents the parameters of the recognition module in the recognition model corresponding to any of the above non-first-level images; This represents the feature map of any of the above non-first-level images.

[0081] The first loss value and each of the second loss values ​​are weighted and summed to obtain the comprehensive loss value. If the number of iterations is less than the target number of iterations, the weight of the second loss value is positively correlated with the number of iterations; that is, as the number of iterations increases, the weight of the second loss value also increases. If the number of iterations is greater than or equal to the target number of iterations, the weight of the second loss value is the target weight; that is, after the number of iterations is greater than or equal to the target number of iterations, the weight of the second loss value no longer increases but remains unchanged. This can be expressed by the formula:

[0082]

[0083] Where L represents the overall loss value; m represents the number of non-first-level images; ω d This represents the weight of the second loss value corresponding to the j-th non-first-level image.

[0084] Optionally, the weights of the second loss values ​​are the same for all non-first-level images. The weights of the second loss can be calculated using the following formula:

[0085]

[0086] Where e is the current iteration number, α is a hyperparameter, and β is an integer.

[0087] With the goal of minimizing the overall loss value, the parameters of the recognition model corresponding to each level of image are updated.

[0088] In this application, the parameters of the feature extraction modules in the recognition models corresponding to different image levels are the same, while the parameters of the recognition modules in the recognition models corresponding to different image levels can be the same or different. Specifically, the recognition performance of the target recognition model trained with different parameters in the recognition models corresponding to different image levels is better than the recognition performance of the target recognition model trained with the same parameters in the recognition models corresponding to different image levels.

[0089] When the parameters of the recognition modules in the recognition models corresponding to different levels of images are the same, the recognition model corresponding to any level of image can be used as the target recognition model when the training termination condition is met.

[0090] When the parameters of the recognition modules in the recognition models corresponding to different levels of images are different, the recognition model corresponding to the first level of images can be used as the target recognition model when the training termination condition is met.

[0091] In an optional embodiment, a flowchart illustrating the masking process of the parent image of any given level image based on the feature map of the parent image of any given level image is shown below. Figure 3 As shown, it may include:

[0092] Step S301: Process the feature map of the previous level image of any of the above level images using the category response model to obtain the category response map of the previous level image of any of the above level images.

[0093] The feature map of the previous level image can be processed using the category response model corresponding to the previous level image, thus obtaining the category response map of the previous level image. The parameters of the category response model are different for different levels of images.

[0094] Step S302: Based on the category label of the sample image and the category response map of the previous level image of any of the above-mentioned level images, perform masking processing on the previous level image of any of the above-mentioned level images to obtain the above-mentioned level images.

[0095] The salient regions can be determined in the category response map based on the category labels of the sample images. These are the regions that have a significant impact on the recognition results of the previous level image of any of the above-mentioned level images. Then, the regions corresponding to the salient regions in the previous level image of any of the above-mentioned level images are masked to obtain the image of any level.

[0096] Optionally, a flowchart illustrating the masking process for the parent image of any given level image based on the category label of the sample image and the category response map of the parent image of any given level image is shown below. Figure 4 As shown, it may include:

[0097] Step S401: Based on the category response map of the previous level image of any of the above-mentioned level images and the category label of the sample image, determine the target region in the category response map of the previous level image of any of the above-mentioned level images; the correlation between the elements in the target region and the category label of the sample image is greater than the correlation between the elements in the non-target region and the category label of the sample image.

[0098] Optionally, a category response sub-map corresponding to the channel represented by the category label of the sample image can be obtained from the category response map of the image above any of the above-mentioned image levels.

[0099] Elements in the category response subgraph whose values ​​are greater than or equal to the target value are identified as target elements; the regions containing all target elements constitute the aforementioned target regions.

[0100] Optionally, the target value can be a preset value. That is, the target value is fixed.

[0101] Optionally, the target value can be determined based on the values ​​of each element in the category response subgraph corresponding to the channel representing the category of the sample image's category label. As an example, target elements can be determined in the category response subgraph corresponding to the channel representing the category of the sample image's category label, in ascending order of element value. Once the proportion of the target element in the total number of elements in the obtained category response subgraph reaches the target percentage, the determination of target elements stops, and the minimum value among the target elements is determined as the target value. In other words, the value of the target element is greater than the value of the non-target element, and the proportion of the target element in the category response subgraph is the target percentage.

[0102] Step S402: Mask the region corresponding to the target region in the previous level image of any of the above-mentioned level images to obtain the above-mentioned level images.

[0103] Optionally, an initial mask (i.e., an initial mask) of the same size as the previous level image of any of the above-mentioned level images can be obtained. The value of each element in the initial mask is 1. The elements in the initial mask corresponding to the target region are set to zero to obtain the target mask (i.e., the target mask). The target mask is then subjected to a Hadamard product operation with the previous level image of any of the above-mentioned level images, that is, the elements at corresponding positions are multiplied to obtain the above-mentioned level image.

[0104] Optionally, when the category response model processes the feature map of the previous level image of any of the above-mentioned level images, it can be used for:

[0105] Based on the weights of each channel of the feature map of the previous level image of the above-mentioned image corresponding to the same category, the sub-feature maps of each channel of the feature map of the previous level image of the above-mentioned image are weighted and summed to obtain the category response sub-map of one channel of the category response map of the previous level image of the above-mentioned image.

[0106] Different categories correspond to different channels in the category response map of the previous level image of any of the above-mentioned level images.

[0107] The process of determining the category response subgraph for a single channel described above can be expressed by the following formula:

[0108]

[0109] Among them, M c This represents the category response subgraph for the channel corresponding to category c; S d This represents the feature map of the d-th channel of the feature map of the image above any of the above-mentioned image levels; Θ d,c This represents the weight of the d-th channel of the feature map of the previous level image, corresponding to category c. Θ d,c This is a parameter that needs to be learned.

[0110] Furthermore, the category response model can also be used for:

[0111] Based on the weights of each channel of the feature map of the previous level image of any of the above-mentioned level images corresponding to the same category, and the feature map of the previous level image of any of the above-mentioned level images, the probability distribution of each category corresponding to the previous level image of any of the above-mentioned level images is obtained.

[0112] Optionally, the feature map of the previous level image of any of the above-mentioned level images can be pooled (e.g., global average pooling or global maximum pooling) to obtain the fused features of each channel. Based on the weights of each channel of the feature map of the previous level image of any of the above-mentioned level images corresponding to the same category, the fused features of each channel are weighted and summed to obtain the probability of the previous level image of any of the above-mentioned level images corresponding to the same category.

[0113] With the goal of making the probability distribution of each category in the previous level image of any of the above-mentioned images approach the category label of the sample image, the parameters of the category response model are updated. The parameters of the category response model include the weights of each channel of the feature map of the previous level image of any of the above-mentioned images corresponding to the same category.

[0114] Optionally, the probability distribution of each category in the previous level image of any of the above-mentioned images and the third loss value of the category label of the sample image can be calculated, and the parameters of the category response model can be updated with the goal of making the third loss value smaller and smaller.

[0115] As an example, the cross-entropy loss function can be used to calculate the third loss value between the probability distribution of each category in the previous level image of any of the above-mentioned images and the category labels of the sample images:

[0116] L l (x,y)=L ce (GAP(f(x;θ))*Θ,y) (6)

[0117] Among them, L l (x, y) represents the third loss value; x represents the previous level image of any of the above images; y represents the class label of the sample image; f(x; θ) represents the feature map of the previous level image of any of the above images; θ represents the parameters of the feature extraction module; GAP represents global average pooling; L ce (,) represents the cross-entropy; Θ represents the weights of each channel of the feature map of the previous level image in the category response model, corresponding to each category.

[0118] Assuming the feature map of the previous level image of any of the above levels is of size H×W and has D channels, then Θ is a matrix of size D×C, where C is the total number of categories.

[0119] Based on the above formula, when the feature extraction module is a Transformer network, it is necessary to convert the two-dimensional feature map T×D output by the feature extraction module into a three-dimensional feature map H×W×D.

[0120] like Figure 5 The diagram shown illustrates a network architecture for obtaining a target recognition model based on multi-level image training of sample images, as provided in an embodiment of this application. The diagram includes N recognition models, where N = m + 1 are denoted as Mode_1 to Mode_N respectively. The feature extraction module of the q-th (q = 1, 2, 3, ..., N) recognition model Mode_q is an Embedding_q network, and the recognition module is a Head_q network.

[0121] Based on Figure 5 When training the network architecture shown, for any sample image, the sample image (i.e., the first level image in the multi-level image of the sample image) is first input into the recognition model Mode_1. The Embedding_1 network in Mode_1 extracts features from the sample image to obtain the feature map of the sample image. The Head_1 network in Mode_1 processes the feature map of the sample image to obtain the recognition result of the sample image (as shown in the bar chart on the right side of Mode_1, which represents the probability distribution of each category of the sample image 1_1).

[0122] The feature map of any sample image is processed by the category response model 1 corresponding to Mode_1 to obtain category response map 1. In addition, the category response model 1 also obtains the probability distribution 1_2 of each category corresponding to any sample image based on the weights of each channel of the feature map of any sample image corresponding to the same category, and the feature map of any sample image.

[0123] Based on the category label of any sample image and category response map 1, the target region 1 is determined in category response map 1.

[0124] Mask 1 is generated based on target region 1.

[0125] Masking any sample image using mask 1 yields the second-level image, which is the first non-first-level image. Figure 5 The denoted 'Hadamard' is represented in the text.

[0126] The second-level image is input into the recognition model Mode_2. The Embedding_2 network in Mode_2 extracts features from the second-level image to obtain the feature map of the second-level image. The Head_2 network in Mode_2 processes the feature map of the second-level image to obtain the recognition result of the second-level image (as shown in the bar chart on the right side of Mode_2, which represents the probability distribution of each category of the second-level image 2_1).

[0127] The feature map of the second-level image is processed using the category response model 2 corresponding to Mode_2 to obtain the category response. Figure 2 In addition, the type response model 2 also obtains the probability distribution 2_2 of each category corresponding to the second-level image based on the weights of each channel of the feature map of the second-level image corresponding to the same category, and the feature map of the second-level image.

[0128] Based on the category label and category response of any given sample image Figure 2 In category response Figure 2 Target area 2 was identified.

[0129] Mask 2 is generated based on target region 2.

[0130] Mask the second-level image using mask 2 to obtain the third-level image, which is the second non-first-level image.

[0131] The third-level image is input into the recognition model Mode_3. The Embedding_3 network in Mode_3 extracts features from the third-level image to obtain the feature map of the third-level image. The Head_3 network in Mode_3 processes the feature map of the third-level image to obtain the recognition result of the third-level image (as shown in the bar chart on the right side of Mode_3, which represents the probability distribution of each category of the third-level image 3_1).

[0132] The feature map of the third-level image is processed using the category response model 3 corresponding to Mode_3 to obtain the category response. Figure 3 In addition, the type response model 3 also obtains the probability distribution 3_2 of each category corresponding to the third-level image based on the weights of each channel of the feature map of the third-level image corresponding to the same category, and the feature map of the third-level image.

[0133] Based on the category label and category response of any given sample image Figure 3 In category response Figure 3 The target area 3 was identified.

[0134] Mask 3 is generated based on target region 3.

[0135] Mask the third-level image using mask 3 to obtain the fourth-level image, which is the third non-first-level image.

[0136] This process continues until the Nth level image is obtained. The Nth level image is then input into the recognition model Mode_N. The Embedding_N network in Mode_N extracts features from the Nth level image to obtain its feature map. The Head_N network in Mode_N processes the feature map of the Nth level image to obtain the recognition result of the Nth level image (as shown in the bar chart on the right side of Mode_N, which represents the probability distribution N_1 of each category corresponding to the Nth level image).

[0137] With the primary objective of ensuring that the recognition result of any sample image approaches the category label of that sample image, and that the recognition results of all non-first-level images approach the recognition result of that sample image, the parameters of each recognition model (i.e., Mode_1 to Mode_N) are updated. To facilitate a clear view of the differences between the recognition results of each non-first-level image and the recognition result of any sample image, this application fits a probability distribution curve based on the recognition result output by model Mode_q, and displays the difference between the recognition result output by model Mode_1 and the recognition result output by the recognition model corresponding to any non-first-level image in a curve format, such as... Figure 5 As shown, the graph to the right of Mode_2 displays the curve fitted from the recognition result output by Mode_2 and the curve fitted from the recognition result output by Mode_1; the graph to the right of Mode_3 displays the curve fitted from the recognition result output by Mode_3 and the curve fitted from the recognition result output by Mode_1; the graph to the right of Mode_N displays the curve fitted from the recognition result output by Mode_N and the curve fitted from the recognition result output by Mode_1. The primary objective is to minimize the difference between the recognition result of any given sample image and the category label of that given sample image, and to minimize the difference between the recognition results of each non-first-level image and the recognition results of the first-level image.

[0138] The second objective is to update the parameters of each category response model so that the probability distribution of each level of image corresponding to each category obtained by each category response model approaches the category label of any sample image. In other words, in this application, the recognition model and the category response model are learned based on different objectives.

[0139] Corresponding to the method embodiments, this application also provides an identification device, such as... Figure 6 The diagram shown is a structural schematic of an identification device provided in an embodiment of this application, which may include:

[0140] Extraction unit 601 and processing unit 602; wherein,

[0141] The extraction unit 601 is used to extract features from the image to be recognized to obtain a feature map; the feature map contains feature information of multiple regions in the image to be recognized, and the feature information of different regions has different effects on the recognition result of the image to be recognized;

[0142] The processing unit 602 is used to process the feature information of multiple regions of the image to be identified in the feature map to obtain the recognition result of the image to be identified.

[0143] The recognition device provided in this application embodiment obtains features of multiple regions with different effects on the recognition result of the image to be recognized through a single feature extraction, eliminating the need for multiple feature extractions and thus improving the recognition efficiency of fine-grained images.

[0144] In an alternative embodiment, wherein,

[0145] The extraction unit 601 is used to extract features from the image to be identified through the feature extraction module of the target recognition model to obtain the feature map of the image to be identified.

[0146] The processing unit 602 is used to process the feature information of multiple regions of the image to be identified in the feature map through the recognition module of the target recognition model to obtain the recognition result of the image to be identified;

[0147] The target recognition model is trained based on multi-level images of the sample images; the first-level image in the multi-level images is the sample image, and the non-first-level images in the multi-level images are obtained by masking the previous-level image of the non-first-level images.

[0148] In an optional embodiment, a training unit is further included for:

[0149] For any level image in the multi-level images, the feature extraction module in the recognition model corresponding to the level image extracts features of the level image to obtain the feature map of the level image. The recognition module in the recognition model corresponding to the level image processes the feature information of multiple regions of the level image in the feature map of the level image to obtain the recognition result of the level image.

[0150] Wherein, if any level image is a non-first level image in the multi-level images, the any level image is obtained by masking the previous level image based on the feature map of the previous level image of the any level image;

[0151] With the goal of making the recognition result of the first-level image in the multi-level image approach the category label of the sample image, and making the recognition result of each non-first-level image in the multi-level image approach the recognition result of the first-level image in the multi-level image, the parameters of the recognition model corresponding to each level of image are updated until the target recognition model is obtained.

[0152] In an optional embodiment, when the training unit performs masking processing on the parent image of the given image based on the feature map of the parent image of the given image, it is used for:

[0153] The feature map of the previous level image of any level image is processed by the category response model to obtain the category response map of the previous level image of any level image.

[0154] The image above the sample image is masked based on the category label of the sample image and the category response map of the image above the previous level of the image above the previous level to obtain the image above the previous level.

[0155] In an optional embodiment, when the training unit performs masking processing on the parent image of any level image based on the class label of the sample image and the class response map of the parent image of any level image, it is used to:

[0156] Based on the category response map of the previous level image of any given level image and the category label, a target region is determined in the category response map of the previous level image of any given level image; the correlation between the elements in the target region and the category label is greater than the correlation between the elements in the non-target region and the category label.

[0157] The region corresponding to the target region in the previous level image of any given level image is masked to obtain the given level image.

[0158] In an optional embodiment, when the training unit determines the target region in the category response map of the parent image of the any level image based on the category label, it is used to:

[0159] Obtain the category response sub-map corresponding to the channel of the category represented by the category label from the category response map of the previous level image of any level image;

[0160] Elements in the category response subgraph whose values ​​are greater than or equal to the target value are identified as target elements; the regions containing all target elements constitute the target region.

[0161] In an alternative embodiment, wherein,

[0162] The target value is a preset value;

[0163] or,

[0164] The value of the target element is greater than the value of the non-target element, the proportion of the number of target elements in the total number of elements in the category response subgraph is the target percentage, and the target value is the minimum value among all target elements.

[0165] In an optional embodiment, when the category response model processes the feature map of the previous level image of any level image, it is used to:

[0166] Based on the weights of each channel of the feature map of the previous level image of the above-level image corresponding to the same category, the sub-feature maps of each channel of the feature map of the previous level image of the above-level ...

[0167] Different categories correspond to different channels in the category response map of the previous level image of any given level image.

[0168] In an optional embodiment, the category response model is further used for:

[0169] Based on the weights of each channel of the feature map of the previous level image of any given level image corresponding to the same category, and the feature map of the previous level image of any given level image, the probability distribution of each category corresponding to the previous level image of any given level image is obtained.

[0170] The training unit is also used to update the parameters of the category response model with the goal of making the probability distribution of each category in the previous level image of the image approach the category label. The parameters of the category response model include the weights of each channel of the feature map of the previous level image of the image corresponding to the same category.

[0171] In an optional embodiment, the training unit, aiming to make the recognition result of the first-level image in the multi-level images approach the category label, and the recognition results of each non-first-level image in the multi-level images approach the recognition result of the first-level image in the multi-level images, updates the parameters of the recognition model corresponding to each level of images by:

[0172] Obtain the recognition result of the first-level image in the multi-level image and the first loss value of the category label, and the recognition result of each non-first-level image in the multi-level image and the second loss value of the recognition result of the first-level image in the multi-level image;

[0173] The first loss value and each of the second loss values ​​are weighted and summed to obtain the comprehensive loss value; wherein, if the number of iterations is less than the target number of iterations, the weight of the second loss value is positively correlated with the number of iterations; if the number of iterations is greater than or equal to the target number of iterations, the weight of the second loss value is the target weight;

[0174] With the goal of minimizing the overall loss value, the parameters of the recognition model corresponding to each level of image are updated.

[0175] Corresponding to the method embodiments, this application also provides an electronic device, a schematic diagram of which is shown below. Figure 7As shown, it may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4.

[0176] In this embodiment, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4.

[0177] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0178] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device.

[0179] The memory 3 stores a program, and the processor 1 can call the program stored in the memory 3. The program is used for:

[0180] Feature extraction is performed on the image to be recognized to obtain a feature map of the image to be recognized; the feature map contains feature information of multiple regions in the image to be recognized, and the feature information of different regions has different effects on the recognition result of the image to be recognized;

[0181] The feature information of multiple regions of the image to be identified in the feature map is processed to obtain the recognition result of the image to be identified.

[0182] Optionally, the refined and extended functions of the program can be found in the description above.

[0183] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0184] Feature extraction is performed on the image to be recognized to obtain a feature map of the image to be recognized; the feature map contains feature information of multiple regions in the image to be recognized, and the feature information of different regions has different effects on the recognition result of the image to be recognized;

[0185] The feature information of multiple regions of the image to be identified in the feature map is processed to obtain the recognition result of the image to be identified.

[0186] Optionally, the refined and extended functions of the program can be found in the description above.

[0187] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0191] It should be understood that in the embodiments of this application, the claims, various embodiments, and features can be combined with each other to solve the aforementioned technical problems.

[0192] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0193] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An identification method, the method comprising: The target recognition model extracts features from the image to be recognized to obtain a feature map of the image to be recognized. The feature map contains feature information of multiple regions in the image to be recognized, and the feature information of different regions has different effects on the recognition result of the image to be recognized. The target recognition model processes the feature information of multiple regions of the image to be recognized in the feature map to obtain the recognition result of the image to be recognized. The target recognition model is trained based on multi-level images of the sample images; the first-level image in the multi-level images is the sample image, and the non-first-level images in the multi-level images are obtained by masking the previous-level image of the non-first-level images. During training, the goal is to make the recognition result of the first-level image in the multi-level images approach the category label of the sample image, and to make the recognition result of each non-first-level image in the multi-level images approach the recognition result of the first-level image in the multi-level images. The parameters of the recognition model corresponding to each level of image are updated until the target recognition model is obtained.

2. The method of claim 1, wherein, The process of extracting features from the image to be identified, and processing the feature information of multiple regions in the feature map of the image to be identified, includes: The feature extraction module of the target recognition model extracts features from the image to be recognized, and obtains the feature map of the image to be recognized. The recognition module of the target recognition model processes the feature information of multiple regions of the image to be recognized in the feature map to obtain the recognition result of the image to be recognized.

3. The method according to claim 2, wherein the target recognition model is trained in the following manner: For any level image in the multi-level images, the feature extraction module in the recognition model corresponding to the level image extracts features of the level image to obtain the feature map of the level image. The recognition module in the recognition model corresponding to the level image processes the feature information of multiple regions of the level image in the feature map of the level image to obtain the recognition result of the level image. wherein, If any level image is a non-first level image in the multi-level images, the any level image is obtained by masking the previous level image based on the feature map of the previous level image.

4. The method according to claim 3, wherein the process of masking the parent image of the first-level image based on the feature map of the parent image of the first-level image includes, The feature map of the previous level image of any level image is processed by the category response model to obtain the category response map of the previous level image of any level image. The image above the sample image is masked based on the category label of the sample image and the category response map of the image above the previous level of the image above the previous level to obtain the image above the previous level.

5. The method according to claim 4, wherein the masking of the parent image of the arbitrary level image based on the category label of the sample image and the category response map of the parent image of the arbitrary level image comprises: Based on the category response map of the previous level image of any given level image and the category label, a target region is determined in the category response map of the previous level image of any given level image; the correlation between the elements in the target region and the category label is greater than the correlation between the elements in the non-target region and the category label. The region corresponding to the target region in the previous level image of any given level image is masked to obtain the given level image.

6. The method according to claim 5, wherein determining the target region in the category response map of the parent image of the any-level image based on the category response map of the parent image of the any-level image and the category label comprises: Obtain the category response sub-map corresponding to the channel of the category represented by the category label from the category response map of the previous level image of any level image; Elements in the category response subgraph whose values ​​are greater than or equal to the target value are identified as target elements. The target region is comprised of the area containing all target elements.

7. The method according to claim 4, wherein when the category response model processes the feature map of the previous level image of any level image, it is used to: Based on the weights of each channel of the feature map of the previous level image of the above-level image corresponding to the same category, the sub-feature maps of each channel of the feature map of the previous level image of the above-level ... Different categories correspond to different channels in the category response map of the previous level image of any given level image.

8. The method according to claim 7, wherein the category response model is further used for: Based on the weights of each channel of the feature map of the previous level image of any given level image corresponding to the same category, and the feature map of the previous level image of any given level image, the probability distribution of each category corresponding to the previous level image of any given level image is obtained. With the goal of making the probability distribution of each category in the previous level image of any level image approach the category label, the parameters of the category response model are updated. The parameters of the category response model include the weights of each channel of the feature map of the previous level image of any level image corresponding to the same category.

9. The method according to claim 3, wherein updating the parameters of the recognition model corresponding to each level of image with the goal of the recognition result of the first level image in the multi-level image approaching the category label of the sample image, and the recognition results of each non-first level image in the multi-level image approaching the recognition result of the first level image in the multi-level image, includes: Obtain the recognition result of the first-level image in the multi-level image and the first loss value of the category label, and the recognition result of each non-first-level image in the multi-level image and the second loss value of the recognition result of the first-level image in the multi-level image; The first loss value and each of the second loss values ​​are weighted and summed to obtain the comprehensive loss value; wherein, if the number of iterations is less than the target number of iterations, the weight of the second loss value is positively correlated with the number of iterations; if the number of iterations is greater than or equal to the target number of iterations, the weight of the second loss value is the target weight; With the goal of minimizing the overall loss value, the parameters of the recognition model corresponding to each level of image are updated.

10. An identification device, the device comprising: The extraction unit is used to extract features from the image to be recognized using a target recognition model to obtain a feature map; the feature map contains feature information of multiple regions in the image to be recognized, and the feature information of different regions has different effects on the recognition result of the image to be recognized; The processing unit is used to process the feature information of multiple regions of the image to be identified in the feature map through the target recognition model to obtain the recognition result of the image to be identified; The target recognition model is trained based on multi-level images of the sample images; the first-level image in the multi-level images is the sample image, and the non-first-level images in the multi-level images are obtained by masking the previous-level image of the non-first-level images. During training, the goal is to make the recognition result of the first-level image in the multi-level images approach the category label of the sample image, and to make the recognition result of each non-first-level image in the multi-level images approach the recognition result of the first-level image in the multi-level images. The parameters of the recognition model corresponding to each level of image are updated until the target recognition model is obtained.

Images