Bird identification method and system

Abstract

The present invention provides a bird recognition method and system. The method generates image description texts of sample images and images to be recognized, and also obtains species description texts corresponding to each bird species. Subsequently, based on the similarity between image features of the sample image and image features of the image to be recognized, and the similarity between text features of the image description text of the sample image and text features of the image description text of the image to be recognized, several nearest neighbor sample images of the image to be recognized and species description texts corresponding to the nearest neighbor sample images are determined. Furthermore, based on the similarity between text features of the species description texts corresponding to each bird species, the nearest neighbor species description text of the species description text corresponding to the nearest neighbor sample image is determined. Thus, based on the correlation between the image description text of the image to be recognized and the nearest neighbor species description text, the species of the bird in the image to be recognized is determined, thereby improving the bird recognition accuracy including zero-shot scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a bird recognition method and system. Background Art

[0002] With the rapid development of deep learning, the accuracy of image recognition using deep learning algorithms on specific datasets has continued to improve. Image recognition has been applied to various real-world scenarios, with bird recognition being a particularly important application. Image recognition tasks are generally categorized into coarse-grained and fine-grained recognition. When applied to species identification, coarse-grained recognition involves cross-species recognition, such as identifying cats and dogs, while fine-grained recognition involves identifying subcategories within a species, such as different bird species.

[0003] Currently, bird recognition tasks typically use supervised training of image recognition models, allowing them to learn the visual features of specific bird species in the dataset and identify images of that specific species from bird images. However, collecting sample bird images is difficult, resulting in a small number of training samples and insufficient supervision information. This leads to poor recognition rates when the similarity between sub-species is high. Furthermore, real images of some bird species (such as endangered birds) are difficult to collect, leading to zero-shot bird recognition (i.e., images of bird species that do not appear in the training set are present in the test set). Summary of the Invention

[0004] The present invention provides a bird identification method and system to solve the defect of poor recognition rate in the prior art.

[0005] The present invention provides a bird identification method, comprising:

[0006] Extracting the image features of each sample image in the training set and the image features of the image to be identified;

[0007] Generate image description text for each sample image in the training set and image description text for the image to be identified;

[0008] Extracting text features of the species description text corresponding to each bird species and text features of the image description text of the sample image and the image to be identified;

[0009] Determining a plurality of nearest neighbor sample images of the image to be identified and category description texts corresponding to the nearest neighbor sample images based on similarities between image features of the sample image and image features of the image to be identified and similarities between text features of the image description text of the sample image and text features of the image description text of the image to be identified;

[0010] Determining a plurality of nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image based on similarities between text features of the species description texts corresponding to the respective bird species;

[0011] The species of the bird in the image to be identified is determined based on the degree of association between the image description text of the image to be identified and the nearest neighbor species description text.

[0012] According to a bird identification method provided by the present invention, determining the species of the bird in the image to be identified based on the correlation between the image description text of the image to be identified and the nearest neighbor species description text specifically includes:

[0013] Determining a matching degree of any nearest neighbor category description text based on a correlation between text features of the image description text of the image to be identified and text features of any nearest neighbor category description text, and a correlation between image features of the image to be identified and text features of any nearest neighbor category description text;

[0014] The species label corresponding to the nearest neighbor species description text with the highest matching degree is determined as the species of the bird in the image to be identified.

[0015] According to a bird identification method provided by the present invention, the degree of association between the image features of the image to be identified and the text features of any of the nearest neighbor species description texts is determined based on the product of the image features of the image to be identified, the association matrix, and the text features of any of the nearest neighbor species description texts;

[0016] The correlation matrix is ​​learned based on the following steps:

[0017] Determining a current association value between any sample image and any category description text based on image features of any sample image, a product of a current value of the association matrix and a text feature of any category description text, and an actual correspondence between any sample image and any category description text, and accumulating the current association values ​​between any sample image and each category description text to obtain an association loss for any sample image;

[0018] The values ​​of the correlation matrix are adjusted to reduce the correlation loss of each sample image until a preset target condition is reached.

[0019] According to a bird identification method provided by the present invention, text features of any species description text are extracted based on the following steps:

[0020] Cluster the species description texts corresponding to each bird species to obtain multiple text clusters, and determine the cluster vector of each text cluster based on the bag-of-words model;

[0021] Extracting a sentence vector of each sentence in the any type of description text, and determining text features of the any type of description text based on the sentence vector of each sentence in the any type of description text and the cluster vector of the text cluster to which the any type of description text belongs.

[0022] According to a bird identification method provided by the present invention, determining the text features of the any species description text based on the sentence vector of each sentence in the any species description text and the cluster vector of the text cluster to which the any species description text belongs specifically includes:

[0023] Extract sentence vectors for image caption text of multiple bird images;

[0024] determining an average value of similarities between a sentence vector of any sentence in the any species description text and the sentence vectors of the image caption texts of the plurality of bird images as a visual score of the any sentence;

[0025] Based on the sentence vector and visual score of each sentence in any type of description text, weighting is performed to obtain the visual vector of the any type of description text;

[0026] The visual vector of the any type of description text and the cluster vector of the text cluster to which the any type of description text belongs are fused to obtain the text feature of the any type of description text.

[0027] According to a bird identification method provided by the present invention, extracting the image features of each sample image in the training set and the image features of the image to be identified specifically includes:

[0028] Continuously extracting features from any image based on multiple feature extraction layers of different sizes, obtaining feature maps corresponding to the multiple feature extraction layers of different sizes; the any image being a sample image or an image to be identified;

[0029] Classifying each feature point in the feature map corresponding to the feature extraction layer of any size based on the classification layer, and obtaining the probability that each feature point in the feature map corresponding to the feature extraction layer of any size belongs to each bird species;

[0030] Based on the probability that each feature point in the feature map corresponding to the feature extraction layer of any size belongs to each bird species, screening out the distinguishing feature points in the feature map corresponding to the feature extraction layer of any size;

[0031] The distinguishing feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes are fused to obtain the image features of any one of the images.

[0032] According to a bird recognition method provided by the present invention, fusing the distinguishing feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain the image features of any one of the images specifically includes:

[0033] splicing the distinguishing feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain spliced ​​feature points;

[0034] The spliced ​​feature points are reorganized based on a fully connected layer to obtain image features of any one of the images.

[0035] According to a bird identification method provided by the present invention, the image description text of any image is generated based on the following steps:

[0036] Fusing the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain a fused feature map of any one of the images;

[0037] Based on the target detection model, detecting the target parts of the bird on the fusion feature map of any one of the images to obtain target detection frames of multiple target parts in the any one of the images;

[0038] Based on the target detection frames of multiple target parts in the any one image, the feature values ​​of the target detection frame areas corresponding to the respective target parts in the fusion feature map of the any one image are enhanced to obtain the regional enhanced features of the any one image;

[0039] Based on the image description generation model, the image description text of any image is generated by utilizing the region enhancement features of any image.

[0040] According to a bird identification method provided by the present invention, the image description generation model is trained based on the following steps:

[0041] Obtaining a fused feature map describing a training image, and performing target part detection of a bird on the fused feature map describing the training image based on a target detection model to obtain target detection frames describing multiple target parts in the training image;

[0042] Based on the target detection frames describing multiple target parts in the training image, the feature values ​​of the target detection frame regions corresponding to the respective target parts in the fusion feature map describing the training image are enhanced to obtain the region enhanced features describing the training image;

[0043] Based on the image description generation model, the image description text describing the training image is generated by utilizing the region enhancement features of the description training image;

[0044] Calculating a description matching loss based on a degree of alignment between the image description text describing the training image and the target detection boxes describing multiple target parts in the training image, and a similarity between the image description text describing the training image and the sample description text describing the training image;

[0045] Parameters of the image description generation model are adjusted based on the description matching loss.

[0046] The present invention also provides a bird identification system, comprising:

[0047] An image feature extraction unit, used to extract image features of each sample image in the training set and image features of the image to be identified;

[0048] a description text generating unit, configured to generate image description text for each sample image in the training set and image description text for the image to be identified;

[0049] a text feature extraction unit, configured to extract text features of the species description text corresponding to each bird species and text features of the image description text of the sample image and the image to be identified;

[0050] a first nearest neighbor determination unit, configured to determine a plurality of nearest neighbor sample images of the image to be identified and category description texts corresponding to the nearest neighbor sample images based on similarities between image features of the sample image and image features of the image to be identified and similarities between text features of the image description text of the sample image and text features of the image description text of the image to be identified;

[0051] A second nearest neighbor determination unit is configured to determine a plurality of nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image based on similarities between text features of the species description texts corresponding to the respective bird species;

[0052] The type determination unit is used to determine the type of bird in the image to be identified based on the correlation between the image description text of the image to be identified and the nearest neighbor type description text.

[0053] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described bird identification methods is implemented.

[0054] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the above-mentioned bird identification methods.

[0055] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned bird identification methods.

[0056] The present invention provides a bird recognition method and system, which can provide more visual features for image features by generating image description texts of sample images and images to be recognized, and make up for the defect of losing some underlying visual features when extracting image features. In the case that the appearance similarity between bird sub-species is large, it helps to improve the accuracy of subsequent bird recognition; in addition, the species description text corresponding to each bird species is obtained, which can provide more and richer bird species information on the one hand, and more importantly, even if some specific species of bird images are not collected in the training set, the species description text corresponding to the species can be used to learn the characteristics of the species, especially its visual features, and then based on the image description text of the sample image Based on the similarity between the image features and the image features of the image to be identified, as well as the similarity between the text features of the image description text of the sample image and the text features of the image description text of the image to be identified, several nearest neighbor sample images of the image to be identified and the species description texts corresponding to the nearest neighbor sample images are determined, and based on the similarity between the text features of the species description texts corresponding to each bird species, several nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image are determined. Therefore, based on the correlation between the image description text of the image to be identified and the nearest neighbor species description text, the species of the bird in the image to be identified is determined, which can improve the bird recognition accuracy including zero-shot scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 1 is a flow chart of the bird identification method provided by the present invention;

[0059] Figure 2 1 is a flow chart of the bird species determination method provided by the present invention;

[0060] Figure 3 It is a flowchart of the text feature extraction method provided by the present invention;

[0061] Figure 4 It is a flow chart of the image feature extraction method provided by the present invention;

[0062] Figure 5It is a schematic structural diagram of the bird identification system provided by the present invention;

[0063] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0064] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0065] Figure 1 Schematic diagram of the bird identification method provided by the present invention, such as Figure 1 As shown, the method includes:

[0066] Step 110, extracting image features of each sample image in the training set and image features of the image to be identified;

[0067] Step 120, generating image description text for each sample image in the training set and image description text for the image to be recognized;

[0068] Step 130 , extracting text features of the species description text corresponding to each bird species and text features of the image description text of the sample image and the image to be identified;

[0069] Step 140: determining a plurality of nearest neighbor sample images of the image to be identified and category description texts corresponding to the nearest neighbor sample images based on the similarity between the image features of the sample image and the image features of the image to be identified, and the similarity between the text features of the image description text of the sample image and the text features of the image description text of the image to be identified;

[0070] Step 150, determining a plurality of nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image based on the similarity between the text features of the species description texts corresponding to the respective bird species;

[0071] Step 160 : Determine the species of the bird in the image to be identified based on the correlation between the image description text of the image to be identified and the nearest neighbor species description text.

[0072] Specifically, a training set containing sample images of multiple bird species and images to be identified for bird species identification are obtained, and image features are extracted from the images to obtain image features of each sample image and image features of the image to be identified. The extracted image features contain the image semantic information of the corresponding image. Considering that many visual features are lost in the image features extracted through multiple convolutional layers, and the visual features of different bird species are more subtle, the loss of visual features will lead to a decrease in the accuracy of bird identification. Therefore, to compensate for the loss of visual features, image description text can be generated for each sample image in the training set and image description text for the image to be identified. An image description generation model can be used to perform image understanding on the sample images and the image to be identified, and generate corresponding image description text. Because the image description text generated by the image description generation model through image understanding of the input image more intuitively expresses the visual information of the corresponding image, the image description generation method can be used to supplement the visual features lost during image feature extraction. Furthermore, we can obtain species descriptions for each bird species from knowledge bases (e.g., Wikipedia, bird encyclopedias, etc.) to provide more and richer bird species information. Importantly, even if images of a particular bird species are not currently collected in the training set, we can use the corresponding species descriptions to learn the characteristics of that species, especially its visual features. Therefore, for the above-mentioned species descriptions and image descriptions, we can use the same or similar methods to extract their text features as a supplement to the image features.

[0073] Subsequently, based on the similarity between the image features of the sample image and the image features of the image to be identified, as well as the similarity between the text features of the image description text of the sample image and the text features of the image description text of the image to be identified, several nearest neighbor sample images of the image to be identified and the species description text corresponding to the nearest neighbor sample images are determined. By combining the similarity between image features and the similarity between text features of image description text, several sample images that are visually similar to the image to be identified, especially those with highly similar visual features of the target (i.e., birds), can be screened from the training set as the nearest neighbor sample images of the image to be identified. At the same time, the corresponding species description text is obtained based on the species label of the nearest neighbor sample images. Here, weights can be set for the image feature similarity and the text feature similarity of the image description text. For each sample image, the similarity between the image features of the sample image and the image features of the image to be identified, as well as the similarity between the text features of the image description text of the sample image and the text features of the image description text of the image to be identified, are weighted to obtain the overall similarity between the sample image and the image to be identified. Based on the overall similarity between each sample image and the image to be identified, sample images with an overall similarity higher than a first similarity threshold are screened out as nearest neighbor sample images.

[0074] To minimize the error rate of image recognition, considering that some bird images are shot at various angles and have interference from environmental issues, species description texts provide a more accurate description of the corresponding bird species. Furthermore, in the case of zero-shot recognition, the bird in the image to be identified may not have appeared in the sample image, and the species description text can compensate for this deficiency. Therefore, based on the similarity between the text features of the species description texts corresponding to each bird species, several nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image are determined. Among them, the nearest neighbor species description text of the species description text corresponding to the nearest neighbor sample image, that is, the species description text whose text features have a similarity greater than a second similarity threshold to the text features of the species description text corresponding to the nearest neighbor sample image. The obtained nearest neighbor species description samples include species description samples of bird species that are visually similar to the bird in the image to be identified. Given that the bird in the image to be identified may not have appeared in the training set, in order to accurately identify the bird species in the image to be identified, the species of the bird in the image to be identified can be determined based on the correlation between the image description text of the image to be identified and the nearest neighbor species description texts. Here, the higher the correlation between the image description text of the image to be identified and any of the nearest neighbor species description texts, the closer the visual characteristics of the bird in the image to be identified are to the visual characteristics of the bird species described in the nearest neighbor species description text. Therefore, the bird species corresponding to the nearest neighbor species description text with the highest correlation with the image description text of the image to be identified can be selected as the species of the bird in the image to be identified.

[0075] It can be seen that the method provided by the embodiment of the present invention can provide more visual features for image features by generating image description texts of sample images and images to be identified, and make up for the defect of losing some underlying visual features when extracting image features. When the appearance similarity between bird sub-species is large, it is helpful to improve the accuracy of subsequent bird recognition; in addition, the species description text corresponding to each bird species is also obtained, which can provide more and richer bird species information on the one hand. More importantly, even if some specific species of bird images are not collected in the training set, the species description text corresponding to the species can be used to learn the characteristics of the species, especially its visual features, and then based on the image features of the sample image, the species description text can be used to learn the characteristics of the species, especially its visual features. Based on the similarity between the image features of the sample image and the image features of the image to be identified, as well as the similarity between the text features of the image description text of the sample image and the text features of the image description text of the image to be identified, several nearest neighbor sample images of the image to be identified and the species description texts corresponding to the nearest neighbor sample images are determined, and based on the similarity between the text features of the species description texts corresponding to each bird species, several nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image are determined. Therefore, based on the correlation between the image description text of the image to be identified and the nearest neighbor species description text, the species of the bird in the image to be identified is determined, which can improve the bird recognition accuracy including zero-shot scenarios.

[0076] Based on the above embodiments, Figure 2 As shown, the determining of the species of the bird in the image to be identified based on the correlation between the image description text of the image to be identified and the nearest neighbor species description text specifically includes:

[0077] Step 210: determining a matching degree of any nearest neighbor category description text based on a correlation between text features of the image description text of the image to be identified and text features of any nearest neighbor category description text, and a correlation between image features of the image to be identified and text features of any nearest neighbor category description text;

[0078] Step 220 : Determine the species label corresponding to the nearest neighbor species description text with the highest matching degree as the species of the bird in the image to be identified.

[0079] Specifically, since image features contain rich image semantic information, which helps to distinguish similar bird species, the matching degree of the nearest neighbor species description text can be determined by combining the correlation between the text features of the image description text of the image to be identified and the text features of any nearest neighbor species description text, as well as the correlation between the image features of the image to be identified and the text features of the nearest neighbor species description text. The matching degree of the nearest neighbor species description text represents the visual similarity between the bird species described in the nearest neighbor species description text and the bird in the image to be identified. Subsequently, the species label corresponding to the nearest neighbor species description text with the highest matching degree is determined as the species of the bird in the image to be identified.

[0080] Based on any of the above embodiments, the degree of correlation between the image features of the image to be identified and the text features of any of the nearest neighbor category description texts is determined based on the product of the image features of the image to be identified, the correlation matrix, and the text features of any of the nearest neighbor category description texts;

[0081] The correlation matrix is ​​learned based on the following steps:

[0082] Determining a current association value between any sample image and any category description text based on image features of any sample image, a product of a current value of the association matrix and a text feature of any category description text, and an actual correspondence between any sample image and any category description text, and accumulating the current association values ​​between any sample image and each category description text to obtain an association loss for any sample image;

[0083] The values ​​of the association matrix are adjusted to reduce the association loss of each sample image until a preset target condition is reached.

[0084] Specifically, because the image features of the image to be identified are spatial vectors in the image mode, while the text features of the nearest neighbor category description text are spatial vectors in the text mode, there is a semantic gap between the two, making it difficult to directly calculate (for example, using the cosine similarity calculation method) the correlation between the two. To address this issue, an embodiment of the present invention calculates the correlation between the image features of the image to be identified and the text features of any nearest neighbor category description text by constructing an association matrix to overcome the semantic gap between the two. Specifically, the correlation between the image features of the image to be identified and the text features of the nearest neighbor category description text can be determined based on the product of the image features of the image to be identified, the aforementioned association matrix, and the text features of the nearest neighbor category description text.

[0085] Here, the current association value between any sample image and the species description text can be determined based on the product of the image features of the sample image, the current value of the association matrix, and the text features of any species description text (assuming it is denoted as I*W*T, where I represents the image features of the sample image, W represents the association matrix, and T represents the text features of the species description text), as well as the actual correspondence between the sample image and the species description text (i.e., whether the bird species in the sample image is consistent with the bird species described in the species description text). The current association value between the sample image and the species description text is then accumulated to obtain the association loss for the sample image. If the actual correspondence between the sample image and any species description text is inconsistent, the current association value between the sample image and the species description text is 0; otherwise, the current association value between the sample image and the species description text is I*W*T. The association loss of each sample image is reduced by adjusting the values ​​of the association matrix until a preset target condition is met, such as when the sum of the association losses of all sample images is minimized or is less than a preset loss value. The values ​​of the association matrix can be adjusted using methods such as machine learning or dynamic programming until the preset target condition is met.

[0086] Based on any of the above embodiments, Figure 3 As shown in Figure 2, the text features of any type of description text are extracted based on the following steps:

[0087] Step 310 , clustering the species description texts corresponding to each bird species to obtain multiple text clusters, and determining a cluster vector for each text cluster based on a bag-of-words model;

[0088] Step 320: extract the sentence vector of each sentence in the any type of description text, and determine the text features of the any type of description text based on the sentence vector of each sentence in the any type of description text and the cluster vector of the text cluster to which the any type of description text belongs.

[0089] Specifically, considering that the similarities between species description texts of different species but similar appearances are mostly reflected in the description of appearance, the species description texts corresponding to each bird species can be clustered to obtain multiple text clusters, and based on the bag-of-words model, cluster vectors containing the semantics of the common appearance description corresponding to each text cluster are extracted to provide additional semantic information for the species description text. Subsequently, the sentence vector of each sentence in the species description text is extracted, and the text features of the species description text are determined based on the sentence vector of each sentence in the species description text and the cluster vector of the text cluster to which the species description text belongs. Among them, by combining the sentence vector of each sentence in any species description text and the cluster vector of the text cluster to which the species description text belongs, and utilizing the semantics of the common appearance description contained in the cluster vector, the sentence vector of the species description text can be provided with more semantic information about visual features, thereby improving the accuracy of the correlation between the species description text and the image description text or image features of the image to be identified.

[0090] Based on any of the foregoing embodiments, determining the text features of any type of descriptive text based on the sentence vector of each sentence in the any type of descriptive text and the cluster vector of the text cluster to which the any type of descriptive text belongs specifically includes:

[0091] Extract sentence vectors for image caption text of multiple bird images;

[0092] determining an average value of similarities between a sentence vector of any sentence in the any species description text and the sentence vectors of the image caption texts of the plurality of bird images as a visual score of the any sentence;

[0093] Based on the sentence vector and visual score of each sentence in any type of description text, weighting is performed to obtain the visual vector of the any type of description text;

[0094] The visual vector of the any type of description text and the cluster vector of the text cluster to which the any type of description text belongs are fused to obtain the text feature of the any type of description text.

[0095] Specifically, the image caption texts of multiple bird images can be obtained from the Internet, and the sentence vectors of the image caption texts of the bird images can be extracted, wherein the image caption texts contain descriptions of the appearance of the birds in the corresponding bird images. The average value of the similarity between the sentence vector of any sentence in any species description text and the sentence vectors of the image caption texts of the above-mentioned multiple bird images is determined as the visual score of the sentence. Here, the higher the similarity between the sentence vector of any sentence and the sentence vectors of the image caption texts of the above-mentioned multiple bird images, the more words containing appearance descriptions in the sentence, and thus the higher its visual score. Subsequently, based on the sentence vector and visual score of each sentence in the species description text, the visual vector of the species description text is weighted to enhance the proportion of semantic information corresponding to the appearance description part in the species description text, and then the visual vector of the species description text and the cluster vector of the text cluster to which the species description text belongs are fused to obtain the text features of the species description text.

[0096] Based on any of the above embodiments, Figure 4 As shown, the extraction of image features of each sample image in the training set and image features of the image to be identified specifically includes:

[0097] Step 410: Continuously extract features from any image based on multiple feature extraction layers of different sizes to obtain feature maps corresponding to the multiple feature extraction layers of different sizes; the any image is a sample image or an image to be identified;

[0098] Step 420: Classify each feature point in the feature map corresponding to the feature extraction layer of any size based on the classification layer to obtain the probability that each feature point in the feature map corresponding to the feature extraction layer of any size belongs to each bird species;

[0099] Step 430 , based on the probability that each feature point in the feature map corresponding to the feature extraction layer of any size belongs to each bird species, screening out the distinguishing feature points in the feature map corresponding to the feature extraction layer of any size;

[0100] Step 440 : Fusing the distinguishing feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain image features of any one of the images.

[0101] Specifically, due to the high similarity in appearance between bird subspecies, the image features extracted from the image also contain a large number of similar image semantics, making it difficult to distinguish the subspecies based on image features. To address this issue, as described in the above embodiment, image description text and species description text are introduced to provide more visual features. Furthermore, the image features can be processed to filter out feature points in more discriminative areas, thereby improving the accuracy of bird recognition.

[0102] Specifically, a continuous feature extraction process can be performed on any image based on multiple feature extraction layers of different sizes to obtain feature maps corresponding to the feature extraction layers of different sizes. The image can be a sample image or an image to be identified. Each feature point in the feature map corresponding to each feature extraction layer of each size is classified based on a pre-trained classification layer (e.g., a fully connected layer) to obtain the probability that each feature point in the feature map corresponding to each feature extraction layer of each size belongs to a specific bird species. The closer the probability of a feature point in the feature map corresponding to each feature extraction layer of each size belongs to a specific bird species is to an extreme value (i.e., 0 or 1), the more discriminative the feature point in the feature map is. Therefore, based on the probability of each feature point in the feature map corresponding to each feature extraction layer of each size belonging to a specific bird species, the discriminative feature points in the feature map corresponding to each feature extraction layer of each size can be screened out, i.e., feature points whose probability of belonging to a specific bird species is closer to an extreme value. The discriminative feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes are then fused to obtain the image features of the image.

[0103] Based on any of the foregoing embodiments, fusing the distinguishing feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain the image features of any of the images specifically includes:

[0104] splicing the distinguishing feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain spliced ​​feature points;

[0105] The spliced ​​feature points are reorganized based on a fully connected layer to obtain image features of any one of the images.

[0106] Specifically, when fusing the distinctive feature points in the feature maps corresponding to multiple feature extraction layers of different sizes, the distinctive feature points in the feature maps corresponding to multiple feature extraction layers of different sizes can be first spliced ​​to obtain spliced ​​feature points, and the distinctive feature points in the spliced ​​feature points can be reorganized using one or more fully connected layers to obtain the image features of the image.

[0107] Based on any of the above embodiments, the image description text of any of the images is generated based on the following steps:

[0108] Fusing the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain a fused feature map of any one of the images;

[0109] Based on the target detection model, detecting the target parts of the bird on the fusion feature map of any one of the images to obtain target detection frames of multiple target parts in the any one of the images;

[0110] Based on the target detection frames of multiple target parts in the any one image, the feature values ​​of the target detection frame areas corresponding to the respective target parts in the fusion feature map of the any one image are enhanced to obtain the regional enhanced features of the any one image;

[0111] Based on the image description generation model, the image description text of any image is generated by utilizing the region enhancement features of any image.

[0112] Specifically, to enhance the image perception capabilities of the image description generation model and enable more accurate and comprehensive bird descriptions, the visual description of the bird's parts in the image can be focused on, removing interference from other objects (such as the background). Specifically, feature maps corresponding to multiple feature extraction layers of different sizes can be fused to obtain a fused feature map for the image. The fused feature map is then used by the object detection model to detect the target parts of the bird in the image, obtaining object detection frames for multiple target parts (such as the back and beak). Based on these target detection frames, the feature values ​​of the feature points corresponding to each target detection frame region in the fused feature map can be enhanced to obtain regional enhanced features for the image. For example, the fused feature map can be transformed to the size of the original image. Based on each target detection frame, feature sub-maps corresponding to the target detection frame region can be extracted from the transformed fused feature map, and the feature values ​​of the feature points in the sub-maps can be increased. Subsequently, the image description text for the image can be generated based on the regional enhanced features of the image, using the image description generation model. Since the region enhancement feature highlights the image semantics of the target part, the image description generation model will focus on the visual description of the target part when perceiving the image and generating image description text, thereby improving the visual description ability of the image description text.

[0113] Based on any of the above embodiments, the image description generation model is trained based on the following steps:

[0114] Obtaining a fused feature map describing a training image, and performing target part detection of a bird on the fused feature map describing the training image based on a target detection model to obtain target detection frames describing multiple target parts in the training image;

[0115] Based on the target detection frames describing multiple target parts in the training image, the feature values ​​of the target detection frame regions corresponding to the respective target parts in the fusion feature map describing the training image are enhanced to obtain the region enhanced features describing the training image;

[0116] Based on the image description generation model, the image description text describing the training image is generated by utilizing the region enhancement features of the description training image;

[0117] Calculating a description matching loss based on a degree of alignment between the image description text describing the training image and the target detection boxes describing multiple target parts in the training image, and a similarity between the image description text describing the training image and the sample description text describing the training image;

[0118] Parameters of the image description generation model are adjusted based on the description matching loss.

[0119] Specifically, a large number of description training images are obtained as training sets for the image description generation model. For any description training image, a fused feature map of the description training image can be obtained in the same manner as for the image to be identified or the sample image. Based on the target detection model, the fused feature map of the description training image is used to detect the target parts of the bird, obtaining target detection frames of multiple target parts in the description training image. Then, based on the target detection frames of multiple target parts in the description training image, the feature values ​​of the target detection frame areas corresponding to each target part in the fused feature map of the description training image are enhanced to obtain regional enhancement features of the description training image. Based on the image description generation model, the regional enhancement features of the description training image are used to generate an image description text for the description training image. Subsequently, based on the degree of alignment between the image description text and the target detection frames of multiple target parts in the description training image, as well as the similarity between the image description text and the sample description text of the description training image, a description matching loss is calculated, and the parameters of the image description generation model are adjusted based on the description matching loss.

[0120] When calculating the degree of alignment between the image description text describing the training image and the target detection frames describing multiple target areas in the training image, the image description text describing the training image can be divided into multiple text segments, and text features of each text segment are extracted. The similarity between the text features of each text segment and the feature subgraphs corresponding to each target detection frame region is then calculated. For any text segment, the maximum similarity between the text features of the text segment and the feature subgraphs corresponding to each target detection frame region can be selected as the matching score of the text segment. The matching scores of each text segment are then accumulated to obtain the degree of alignment between the image description text describing the training image and the target detection frames describing multiple target areas in the training image. Here, when calculating the similarity between the text features of each text segment and the feature subgraphs corresponding to each target detection frame region, a relatively simple and fast vector dot product method can be used. When calculating the similarity between the image description text describing the training image and the sample description text, keywords in the sample description text can be selected, and the similarity between the two can be determined based on the number of keywords hit in the image description text.

[0121] The bird identification system provided by the present invention is described below. The bird identification system described below and the bird identification method described above can be referenced to each other.

[0122] Based on any of the above embodiments, Figure 5 Schematic diagram of the bird identification system provided by the present invention. Figure 5 As shown, the system includes: an image feature extraction unit 510 , a description text generation unit 520 , a text feature extraction unit 530 , a first nearest neighbor determination unit 540 , a second nearest neighbor determination unit 550 and a category determination unit 560 .

[0123] The image feature extraction unit 510 is used to extract the image features of each sample image in the training set and the image features of the image to be identified;

[0124] The description text generating unit 520 is used to generate image description texts for each sample image in the training set and image description texts for the image to be recognized;

[0125] The text feature extraction unit 530 is used to extract text features of the species description text corresponding to each bird species and text features of the image description text of the sample image and the image to be identified;

[0126] The first nearest neighbor determination unit 540 is configured to determine a plurality of nearest neighbor sample images of the image to be identified and category description texts corresponding to the nearest neighbor sample images based on the similarity between the image features of the sample image and the image features of the image to be identified and the similarity between the text features of the image description text of the sample image and the text features of the image description text of the image to be identified;

[0127] The second nearest neighbor determination unit 550 is configured to determine a plurality of nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image based on the similarity between the text features of the species description texts corresponding to the respective bird species;

[0128] The type determination unit 560 is configured to determine the type of bird in the image to be identified based on the degree of association between the image description text of the image to be identified and the nearest neighbor type description text.

[0129] The system provided by the present invention can provide more visual features for image features by generating image description text for sample images and images to be identified, thereby compensating for the defect of losing some underlying visual features during image feature extraction. When the appearance similarity between bird sub-species is large, this helps to improve the accuracy of subsequent bird identification. In addition, species description text corresponding to each bird species is obtained, which can provide more and richer bird species information. Importantly, even if bird images of certain specific species are not currently collected in the training set, the species description text corresponding to the species can be used to learn the characteristics of the species, especially its visual characteristics. Subsequently, based on the similarity between the image features of the sample image and the image features of the image to be identified, and the similarity between the text features of the image description text of the sample image and the text features of the image description text of the image to be identified, several nearest neighbor sample images of the image to be identified and the species description text corresponding to the nearest neighbor sample images are determined. Based on the similarity between the text features of the species description texts corresponding to each bird species, several nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image are determined. Thus, based on the correlation between the image description text of the image to be identified and the nearest neighbor species description texts, the species of the bird in the image to be identified is determined, which can improve the accuracy of bird identification, including zero-shot scenarios.

[0130] Based on any of the foregoing embodiments, determining the species of the bird in the image to be identified based on the correlation between the image description text of the image to be identified and the nearest neighbor species description text specifically includes:

[0131] Determining a matching degree of any nearest neighbor category description text based on a correlation between text features of the image description text of the image to be identified and text features of any nearest neighbor category description text, and a correlation between image features of the image to be identified and text features of any nearest neighbor category description text;

[0132] The species label corresponding to the nearest neighbor species description text with the highest matching degree is determined as the species of the bird in the image to be identified.

[0133] Based on any of the above embodiments, the degree of correlation between the image features of the image to be identified and the text features of any of the nearest neighbor category description texts is determined based on the product of the image features of the image to be identified, the correlation matrix, and the text features of any of the nearest neighbor category description texts;

[0134] The correlation matrix is ​​learned based on the following steps:

[0135] Determining a current association value between any sample image and any category description text based on image features of any sample image, a product of a current value of the association matrix and a text feature of any category description text, and an actual correspondence between any sample image and any category description text, and accumulating the current association values ​​between any sample image and each category description text to obtain an association loss for any sample image;

[0136] The values ​​of the correlation matrix are adjusted to reduce the correlation loss of each sample image until a preset target condition is reached.

[0137] Based on any of the above embodiments, the text features of any type of description text are extracted based on the following steps:

[0138] Cluster the species description texts corresponding to each bird species to obtain multiple text clusters, and determine the cluster vector of each text cluster based on the bag-of-words model;

[0139] Extracting a sentence vector of each sentence in the any type of description text, and determining text features of the any type of description text based on the sentence vector of each sentence in the any type of description text and the cluster vector of the text cluster to which the any type of description text belongs.

[0140] Based on any of the foregoing embodiments, determining the text features of any type of descriptive text based on the sentence vector of each sentence in the any type of descriptive text and the cluster vector of the text cluster to which the any type of descriptive text belongs specifically includes:

[0141] Extract sentence vectors for image caption text of multiple bird images;

[0142] determining an average value of similarities between a sentence vector of any sentence in the any species description text and the sentence vectors of the image caption texts of the plurality of bird images as a visual score of the any sentence;

[0143] Based on the sentence vector and visual score of each sentence in any type of description text, weighting is performed to obtain the visual vector of the any type of description text;

[0144] The visual vector of the any type of description text and the cluster vector of the text cluster to which the any type of description text belongs are fused to obtain the text feature of the any type of description text.

[0145] Based on any of the above embodiments, extracting the image features of each sample image in the training set and the image features of the image to be identified specifically includes:

[0146] Continuously extracting features from any image based on multiple feature extraction layers of different sizes, obtaining feature maps corresponding to the multiple feature extraction layers of different sizes; the any image being a sample image or an image to be identified;

[0147] Classifying each feature point in the feature map corresponding to the feature extraction layer of any size based on the classification layer, and obtaining the probability that each feature point in the feature map corresponding to the feature extraction layer of any size belongs to each bird species;

[0148] Based on the probability that each feature point in the feature map corresponding to the feature extraction layer of any size belongs to each bird species, screening out the distinguishing feature points in the feature map corresponding to the feature extraction layer of any size;

[0149] The distinguishing feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes are fused to obtain the image features of any one of the images.

[0150] Based on any of the foregoing embodiments, fusing the distinguishing feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain the image features of any of the images specifically includes:

[0151] splicing the distinguishing feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain spliced ​​feature points;

[0152] The spliced ​​feature points are reorganized based on a fully connected layer to obtain image features of any one of the images.

[0153] Based on any of the above embodiments, the image description text of any of the images is generated based on the following steps:

[0154] Fusing the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain a fused feature map of any one of the images;

[0155] Based on the target detection model, detecting the target parts of the bird on the fusion feature map of any one of the images to obtain target detection frames of multiple target parts in the any one of the images;

[0156] Based on the target detection frames of multiple target parts in the any one image, the feature values ​​of the target detection frame areas corresponding to the respective target parts in the fusion feature map of the any one image are enhanced to obtain the regional enhanced features of the any one image;

[0157] Based on the image description generation model, the image description text of any image is generated by utilizing the region enhancement features of any image.

[0158] Based on any of the above embodiments, the image description generation model is trained based on the following steps:

[0159] Obtaining a fused feature map describing a training image, and performing target part detection of a bird on the fused feature map describing the training image based on a target detection model to obtain target detection frames describing multiple target parts in the training image;

[0160] Based on the target detection frames describing multiple target parts in the training image, the feature values ​​of the target detection frame regions corresponding to the respective target parts in the fusion feature map describing the training image are enhanced to obtain the region enhanced features describing the training image;

[0161] Based on the image description generation model, the image description text describing the training image is generated by utilizing the region enhancement features of the description training image;

[0162] Calculating a description matching loss based on a degree of alignment between the image description text describing the training image and the target detection boxes describing multiple target parts in the training image, and a similarity between the image description text describing the training image and the sample description text describing the training image;

[0163] Parameters of the image description generation model are adjusted based on the description matching loss.

[0164] Figure 6 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor (processor) 610, a memory (memory) 620, a communication interface (Communications Interface) 630 and a communication bus 640, wherein the processor 610, the memory 620, and the communication interface 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 620 to execute the bird recognition method, which includes: extracting image features of each sample image in the training set and image features of the image to be recognized; generating image description text for each sample image in the training set and image description text for the image to be recognized; extracting text features of the species description text corresponding to each bird species and text features of the image description text of the sample image and the image to be recognized; based on the similarity between the image features of the sample image and the image features of the image to be recognized and the similarity between the text features of the image description text of the sample image and the text features of the image description text of the image to be recognized, determining several nearest neighbor sample images of the image to be recognized and species description texts corresponding to the nearest neighbor sample images; based on the similarity between the text features of the species description texts corresponding to the various bird species, determining several nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image; and determining the species of the bird in the image to be recognized based on the correlation between the image description text of the image to be recognized and the nearest neighbor species description text.

[0165] In addition, the logic instructions in the aforementioned memory 620 can be implemented in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0166] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the bird recognition method provided by the above-mentioned methods, which includes: extracting image features of each sample image in a training set and image features of the image to be recognized; generating image description text of each sample image in the training set and image description text of the image to be recognized; extracting text features of the species description text corresponding to each bird species and text features of the image description text of the sample image and the image to be recognized; based on the similarity between the image features of the sample image and the image features of the image to be recognized and the similarity between the text features of the image description text of the sample image and the text features of the image description text of the image to be recognized, determining several nearest neighbor sample images of the image to be recognized and the species description text corresponding to the nearest neighbor sample images; based on the similarity between the text features of the species description texts corresponding to the respective bird species, determining several nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image; and determining the species of the bird in the image to be recognized based on the correlation between the image description text of the image to be recognized and the nearest neighbor species description text.

[0167] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-mentioned bird identification methods, the methods comprising: extracting image features of each sample image in a training set and image features of an image to be identified; generating image description texts of each sample image in the training set and image description texts of the image to be identified; extracting text features of the species description texts corresponding to each bird species and text features of the image description texts of the sample image and the image to be identified; determining several nearest neighbor sample images of the image to be identified and species description texts corresponding to the nearest neighbor sample images based on the similarity between the image features of the sample image and the image features of the image to be identified and the similarity between the text features of the image description text of the sample image and the text features of the image description text of the image to be identified; determining several nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image based on the similarity between the text features of the species description texts corresponding to the respective bird species; and determining the species of the bird in the image to be identified based on the correlation between the image description text of the image to be identified and the nearest neighbor species description text.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0169] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A bird identification method, characterized in that: include: Extracting the image features of each sample image in the training set and the image features of the image to be identified; Generate image description text for each sample image in the training set and image description text for the image to be identified; Extracting text features of the species description text corresponding to each bird species and text features of the image description text of the sample image and the image to be identified; Determining a plurality of nearest neighbor sample images of the image to be identified and category description texts corresponding to the nearest neighbor sample images based on similarities between image features of the sample image and image features of the image to be identified and similarities between text features of the image description text of the sample image and text features of the image description text of the image to be identified; Determining a plurality of nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image based on similarities between text features of the species description texts corresponding to the respective bird species; The species of the bird in the image to be identified is determined based on the degree of association between the image description text of the image to be identified and the nearest neighbor species description text.

2. A bird identification method according to claim 1, characterized in that: The determining of the species of the bird in the image to be identified based on the correlation between the image description text of the image to be identified and the nearest neighbor species description text specifically includes: Determining a matching degree of any nearest neighbor category description text based on a correlation between text features of the image description text of the image to be identified and text features of any nearest neighbor category description text, and a correlation between image features of the image to be identified and text features of any nearest neighbor category description text; The species label corresponding to the nearest neighbor species description text with the highest matching degree is determined as the species of the bird in the image to be identified.

3. A bird identification method according to claim 2, characterized in that: The degree of correlation between the image features of the image to be identified and the text features of any of the nearest neighbor category description texts is determined based on the product of the image features of the image to be identified, the correlation matrix, and the text features of any of the nearest neighbor category description texts; The correlation matrix is ​​learned based on the following steps: Determining a current association value between any sample image and any category description text based on image features of any sample image, a product of a current value of the association matrix and a text feature of any category description text, and an actual correspondence between any sample image and any category description text, and accumulating the current association values ​​between any sample image and each category description text to obtain an association loss for any sample image; The values ​​of the correlation matrix are adjusted to reduce the correlation loss of each sample image until a preset target condition is reached.

4. A bird identification method according to claim 1, characterized in that: The text features of any type of descriptive text are extracted based on the following steps: Cluster the species description texts corresponding to each bird species to obtain multiple text clusters, and determine the cluster vector of each text cluster based on the bag-of-words model; Extracting a sentence vector of each sentence in the any type of description text, and determining text features of the any type of description text based on the sentence vector of each sentence in the any type of description text and the cluster vector of the text cluster to which the any type of description text belongs.

5. A bird identification method according to claim 4, characterized in that: Determining the text features of the any type of description text based on the sentence vector of each sentence in the any type of description text and the cluster vector of the text cluster to which the any type of description text belongs specifically includes: Extract sentence vectors for image caption text of multiple bird images; determining an average value of similarities between a sentence vector of any sentence in the any species description text and the sentence vectors of the image caption texts of the plurality of bird images as a visual score of the any sentence; Based on the sentence vector and visual score of each sentence in any type of description text, weighting is performed to obtain the visual vector of the any type of description text; The visual vector of the any type of description text and the cluster vector of the text cluster to which the any type of description text belongs are fused to obtain the text feature of the any type of description text.

6. A bird identification method according to claim 1, characterized in that: The step of extracting the image features of each sample image in the training set and the image features of the image to be identified specifically includes: Continuously extracting features from any image based on multiple feature extraction layers of different sizes, obtaining feature maps corresponding to the multiple feature extraction layers of different sizes; the any image being a sample image or an image to be identified; Classifying each feature point in the feature map corresponding to the feature extraction layer of any size based on the classification layer, and obtaining the probability that each feature point in the feature map corresponding to the feature extraction layer of any size belongs to each bird species; Based on the probability that each feature point in the feature map corresponding to the feature extraction layer of any size belongs to each bird species, screening out the distinguishing feature points in the feature map corresponding to the feature extraction layer of any size; The distinguishing feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes are fused to obtain the image features of any one of the images.

7. A bird identification method according to claim 6, characterized in that: The fusing of the distinguishing feature points in the feature maps corresponding to the feature extraction layers of different sizes to obtain the image features of any one of the images specifically includes: splicing the distinguishing feature points in the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain spliced ​​feature points; The spliced ​​feature points are reorganized based on a fully connected layer to obtain image features of any one of the images.

8. A bird identification method according to claim 6, characterized in that: The image description text of any image is generated based on the following steps: Fusing the feature maps corresponding to the multiple feature extraction layers of different sizes to obtain a fused feature map of any one of the images; Based on the target detection model, detecting the target parts of the bird on the fusion feature map of any one of the images to obtain target detection frames of multiple target parts in the any one of the images; Based on the target detection frames of multiple target parts in the any one image, the feature values ​​of the target detection frame areas corresponding to the respective target parts in the fusion feature map of the any one image are enhanced to obtain the regional enhanced features of the any one image; Based on the image description generation model, the image description text of any image is generated by utilizing the region enhancement features of any image.

9. A bird identification method according to claim 8, characterized in that: The image description generation model is trained based on the following steps: Obtaining a fused feature map describing a training image, and performing target part detection of a bird on the fused feature map describing the training image based on a target detection model to obtain target detection frames describing multiple target parts in the training image; Based on the target detection frames describing multiple target parts in the training image, the feature values ​​of the target detection frame regions corresponding to the respective target parts in the fusion feature map describing the training image are enhanced to obtain the region enhanced features describing the training image; Based on the image description generation model, the image description text describing the training image is generated by utilizing the region enhancement features of the description training image; Calculating a description matching loss based on a degree of alignment between the image description text describing the training image and the target detection boxes describing multiple target parts in the training image, and a similarity between the image description text describing the training image and the sample description text describing the training image; Parameters of the image description generation model are adjusted based on the description matching loss.

10. A bird identification system, characterized in that: include: An image feature extraction unit, used to extract image features of each sample image in the training set and image features of the image to be identified; a description text generating unit, configured to generate image description text for each sample image in the training set and image description text for the image to be identified; a text feature extraction unit, configured to extract text features of the species description text corresponding to each bird species and text features of the image description text of the sample image and the image to be identified; a first nearest neighbor determination unit, configured to determine a plurality of nearest neighbor sample images of the image to be identified and category description texts corresponding to the nearest neighbor sample images based on similarities between image features of the sample image and image features of the image to be identified and similarities between text features of the image description text of the sample image and text features of the image description text of the image to be identified; A second nearest neighbor determination unit is configured to determine a plurality of nearest neighbor species description texts of the species description text corresponding to the nearest neighbor sample image based on similarities between text features of the species description texts corresponding to the respective bird species; The type determination unit is used to determine the type of bird in the image to be identified based on the correlation between the image description text of the image to be identified and the nearest neighbor type description text.

Images