Wild bird type identification method

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a wild bird type identification method, which adopts hierarchical independent models, the effect of each model can be independently reasoned, optimized and evaluated, multi-class prediction of 9000 classes of birds is ensured in the use process in combination with logic, model parameters are stored in a vector database, and the model parameters are stored in the vector database. The method does not cause the occupation of useless resources, utilizes a small amount of occupied resources, achieves the purpose that the model is taken, used and released, relieves the pressure of a server, guarantees the recognition efficiency and accuracy, is compatible with multi-terminal deployment, adapts to various application scenes, can replace conventional manual observation, reduces the consumption of human resources, and improves the recognition efficiency. In this way, the technical problem that the recognition method in the prior art is difficult to ensure the stability of system performance and the effective utilization of resources while finely classifying birds in different application areas is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for identifying wild bird species. Background Art

[0002] With the increasing awareness of ecological environmental protection, the location and recording of unknown bird species is particularly important. In addition, when carrying out bird-repelling work in a specific area, the traditional manual survey and recording method is not only inefficient, but may also cause varying degrees of fright to the birds. At the same time, there are certain personal safety risks in field operations. Therefore, the use of intelligent technology to identify wild bird species can not only improve the accuracy and efficiency of bird information statistics, but also achieve technical connection on special hardware equipment systems, effectively expel birds in a specific area, and mark unknown bird species to achieve further detection and recording effects.

[0003] However, there are many known bird species, more than 9,000 in total, and their living environments, distributions, forms, and survival methods are different. In addition, some birds have very similar appearances. Therefore, in the recognition process, we need to identify different birds in different application areas. It is difficult for a single model to accurately classify bird species, and the classification model has too much impact on performance.

[0004] In summary, existing identification methods are difficult to meet the needs of fine classification of birds in different application areas while ensuring the stability of system performance and effective use of resources. Summary of the invention

[0005] The purpose of the present invention is to provide a method for identifying wild bird species, aiming to solve the technical problem that the identification methods in the prior art are difficult to meet the requirements of fine classification of birds in different application areas while ensuring the stability of system performance and effective use of resources.

[0006] To achieve the above object, a method for identifying wild bird species adopted by the present invention comprises the following steps:

[0007] Step 1: First create the image buffer and audio buffer;

[0008] Step 2: Build a model vector library;

[0009] Step 3: Get the video stream of the area to be detected, use OpenCV to read the frame image and store it in the image buffer, use ffmpeg to extract the audio and store it in the audio buffer, and perform preliminary bird family detection on the data in the image buffer. If birds are detected, continue with the subsequent steps, otherwise do not perform any operation;

[0010] Step 4: When a bird is detected in the image buffer in step 3, the genus and species model parameters of the corresponding order family in the target detection vector library are taken out for reading and loading, and the original image in the output box is used for bird genus and species detection, and the output results obtained are stored according to the time node;

[0011] Step 5: Then, the data in the audio buffer is converted into a Mel-spectrogram, and the bird call area of ​​the spectrum is detected, and the result is stored according to the time node for audio recognition;

[0012] Step 6: Then, the internal position detection model parameters of the model vector library are taken out for reading, the detection frame position is cropped to obtain the detection image, and the part detection is performed. After obtaining the detection frame, the image is cropped at the detection frame position, and the shape feature analysis is performed after obtaining these part images. According to the part information, the top five birds with the highest correlation are obtained in the neo4j relational database that stores bird attribute data, and the time node information is stored;

[0013] Step 7. Finally, the final bird probability distribution is obtained by weighted addition of target detection, audio detection, and attribute multi-classification storage values ​​to obtain the species of the bird with the highest probability.

[0014] Among them, the constructed model vector library uses three commonly used deep learning algorithms, yolo, PANN+DenseNet, and ResNet, to handle corresponding target detection tasks, voiceprint classification tasks, and attribute multi-classification.

[0015] Among them, the detection model used in the target detection task uses the ppyoloe-plus-m framework of paddle, and the backbone network is cspresnet-m; there are three major advantages of choosing this model. The first point is that cspresnet-m has a stronger learning ability than conventional convolutional neural networks; the second point is to remove the bottleneck layer. The calculation of the bottleneck layer generally takes up more computing power during inference. Removing the bottleneck layer uses multiple CNNs to amortize this computing power to reduce the model calculation time; the third point is to reduce memory usage. In the process of feature extraction, cspresnet-m uses cross-channel pooling to compress the computational consumption brought by the feature map; faster training speed is reflected in the second and third points; the data label uses the voc type, and in order to avoid non-maximum suppression (NMS), it is divided into three target detection models, namely, bird families, bird species, and audio types. The losses used are GIOU (generalized intersection-over-union) + DFL (distributed focus loss) + VFL (vertical federated learning), and the optimizer selects Momentum (momentum optimizer).

[0016] The GIOU formula is:

[0017]

[0018] U=S((x hat 2-x hat 1), (y hat 2-y hat 1))+S((x2-x1),(y2-y1));

[0019] In the formula, A i y represents the predicted box i The y value of the point and the real box i The area of ​​the extended wireframe is the area of ​​the extra area after the points intersect. U is the area of ​​the real frame plus the area of ​​the predicted frame. hat i With y hat i Indicates the value coordinate of the corresponding point of the prediction box, y i With x i Indicates the value coordinates of the corresponding point of the real box, Y hat represents the predicted box, and Y represents the real box;

[0020]

[0021] The DFL formula is:

[0022] DFL=-((y hat i+1 -y)×log(S i )+(yy i )×log(S i+1 ));

[0023] In the formula, y represents the true value, y hat i The box representing the i-th predicted value.

[0024] The VFL formula is:

[0025]

[0026] Where IACS represents the iou-aware classification score of the prediction box, O h It represents the IOU between the predicted box and the real box. Both α and γ represent scaling factors. γ reduces the contribution of negative samples to the model, and α controls the weight distribution of positive samples to the model.

[0027] Among them, the voiceprint classification model used in the voiceprint classification task uses the torch framework model, and the torch framework model structure is divided into two parts; in the absence of sufficient data as support, the PANNS model trained with AudioSet data is used as a pre-trained model to act as the feature extraction part, and the downstream task part uses DenseNet as the basic atomic unit with normalization layer, pooling layer, and linear layer to complete the voiceprint classification; the characteristic of DenseNet is that each layer can directly access the features of all previous layers, thereby achieving feature reuse, which can be understood as multi-scale fusion, while avoiding the gradient disappearance and gradient explosion problems. The loss used is CEL (cross entropy loss), and the optimizer is adam; the number of models is divided into 2, the first model label is: Ornithology, unknown, empty; the second model label is bird species.

[0028] The CEL formula is:

[0029]

[0030] In the formula, N represents the number of samples, M represents the number of categories, and y c represents the true c category, y ^ c Indicates the predicted category c.

[0031] Among them, the attribute multi-classification model chooses to use the original model structure ResNet101 in the tensorflow framework as feature extraction, adds a layer of sigmoid as the final output layer, and uses sigmoid to convert multi-category labels into multiple binary classification labels to reduce model complexity and improve model speed; the loss function is BCE (binary cross entropy loss), and the optimizer is adam; there are 32 model label definitions, namely: black, white, red, orange, yellow, green, cyan, blue, purple, gradient, solid color, round tail, flat tail, wedge tail, pointed tail, convex tail, concave tail, forked tail, swallow tail, unequal toes, curved unequal toes, opposite toes, different toes, front toes, syndactyly, webbed feet, concave webbed feet, fully webbed feet, semi-webbed feet, flap webbed feet, and shape-empty;

[0032] The BCE formula is:

[0033]

[0034] In the formula, N represents the number of samples, M represents the number of labels, and y ij represents the true label, Represents the model output.

[0035] Among them, during model training, data is mainly obtained through search engines, open source databases, and actual scene shooting. The ffmpeg tool is used to segment the video, and the case application is 10-second segmentation; write a processing script to extract frames and extract audio for the segmented video to determine whether there are birds in the image. If there are birds, it means valid data. The audio is converted into Mel spectrum / audio signal graph, the spectrum graph is observed and the audio is listened to; if there are bird sounds, it is valid data, and the relevant spectrum graph and audio are retained. The data annotation of target detection is divided into two groups. In the first group of video images, a bird needs to be annotated with two label boxes, namely xx order-xx family and xx genus-xx species; the second group is the spectrum graph to mark the area where the spectrum graph event occurs; image data enhancement mainly includes random flipping, random filling, transparency, noise generation, random cutting, etc.; audio classification annotation can be conventionally classified, and audio enhancement is to randomly change the volume, audio filling, speed up / slow down, and background fusion; then enter the training, and after the training is completed, export the corresponding models for different scenes.

[0036] A method for identifying wild bird species of the present invention adopts a hierarchical independent model. The effect of each model can be independently evaluated for inference optimization. Combined with logic, it ensures multi-category prediction of 9,000 bird categories during use. Model parameters are stored in a vector database, which will not cause useless resource occupation. By using a small amount of occupied resources, the model can be used and released immediately, which relieves the pressure on the server while ensuring recognition efficiency and accuracy. It is compatible with multi-terminal deployment and adapts to various application scenarios. It can be equipped with multiple monitoring areas to achieve 24-hour observation, which can replace conventional manual observation, reduce human resource consumption, and avoid human safety factors. Functions can be added later, compatible with other recognition algorithms, and provide a more comfortable integrated environment for external secondary development.

[0037] In this way, the technical problem that the identification methods in the prior art are difficult to satisfy the requirement of fine classification of birds in different application areas while ensuring the stability of system performance and effective use of resources is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flow chart of the wild bird species identification method of the present invention.

[0040] Figure 2 It is a schematic diagram of the preparation for the operation of the detection model of the present invention.

[0041] Figure 3 It is a flow chart of the detection model training of the present invention.

[0042] Figure 4 It is a comparison diagram of the audio signal Mel spectrum of the present invention. DETAILED DESCRIPTION

[0043] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0044] See also Figure 1 to Figure 4 ,in Figure 1 is a flow chart of the method for identifying wild bird species of the present invention, Figure 2 is a schematic diagram of the detection model operation preparation of the present invention, Figure 3 is a flow chart of the detection model training of the present invention, Figure 4 It is a comparison diagram of the audio signal Mel spectrum of the present invention.

[0045] The present invention provides a method for identifying wild bird species, comprising the following steps:

[0046] Step 1: First create the image buffer and audio buffer;

[0047] According to this specific implementation, the image buffer and the audio buffer are respectively used to temporarily store image and audio data extracted from the video stream.

[0048] Step 2: Build a model vector library;

[0049] For this specific implementation, the constructed model vector library uses three common deep learning algorithms, yolo, PANN+DenseNet, and ResNet, to process corresponding target detection tasks, voiceprint classification tasks, and attribute multi-classification.

[0050] The detection model used in the target detection task uses the ppyoloe-plus-m framework of paddle, and the backbone network is cspresnet-m; there are three major advantages of choosing this model. The first point is that cspresnet-m has a stronger learning ability than conventional convolutional neural networks; the second point is to remove the bottleneck layer. The calculation of the bottleneck layer generally takes up more computing power during inference. Removing the bottleneck layer uses multiple CNNs to amortize this computing power to reduce the model calculation time; the third point is to reduce memory usage. In the process of feature extraction, cspresnet-m uses cross-channel pooling to compress the computational consumption brought by the feature map; faster training speed is reflected in the second and third points; the data label uses the voc type, and in order to avoid non-maximum suppression (NMS), it is divided into three target detection models, namely, bird families, bird species, and audio types. The losses used are GIOU (generalized intersection-over-union) + DFL (distributed focal loss) + VFL (vertical federated learning), and the optimizer chooses Momentum (momentum optimizer).

[0051] The GIOU formula is:

[0052]

[0053] U=S((x hat 2-x hat 1), (y hat 2-y hat 1))+S((x2-x1),)y2-y1));

[0054] In the formula, A i y represents the predicted box i The y value of the point and the real box i The area of ​​the extended wireframe is the area of ​​the extra area after the points intersect. U is the area of ​​the real frame plus the area of ​​the predicted frame. hat i With y hat i Indicates the value coordinate of the corresponding point of the prediction box, y i With x i Indicates the value coordinates of the corresponding point of the real box, Y hat represents the predicted box, and Y represents the real box;

[0055]

[0056] The DFL formula is:

[0057] DFL=-((y hat i+1 -y)×log(S i )+(yy i )×log(S i+1));

[0058] In the formula, y represents the true value, y hat i The box representing the i-th predicted value.

[0059] The VFL formula is:

[0060]

[0061] Where IACS represents the iou-aware classification score of the prediction box, O h It represents the IOU between the predicted box and the real box. Both α and γ represent scaling factors. γ reduces the contribution of negative samples to the model, and α controls the weight distribution of positive samples to the model.

[0062] Among them, the voiceprint classification model used in the voiceprint classification task uses the torch framework model, and the torch framework model structure is divided into two parts; in the absence of sufficient data as support, the PANNS model trained with AudioSet data is used as a pre-trained model to act as the feature extraction part, and the downstream task part uses DenseNet as the basic atomic unit with normalization layer, pooling layer, and linear layer to complete the voiceprint classification; the characteristic of DenseNet is that each layer can directly access the features of all previous layers, thereby achieving feature reuse, which can be understood as multi-scale fusion, while avoiding the gradient disappearance and gradient explosion problems. The loss used is CEL (cross entropy loss), and the optimizer is adam; the number of models is divided into 2, the first model label is: Ornithology, unknown, empty; the second model label is bird species.

[0063] The CEL formula is:

[0064]

[0065] In the formula, N represents the number of samples, M represents the number of categories, and y c represents the true c category, y ^ c Indicates the predicted category c.

[0066] Among them, the attribute multi-classification model chooses to use the original model structure ResNet101 in the tensorflow framework as feature extraction, adds a layer of sigmoid as the final output layer, and uses sigmoid to convert multi-category labels into multiple binary classification labels to reduce model complexity and improve model speed; the loss function is BCE (binary cross entropy loss), and the optimizer is adam; there are 32 model label definitions, namely: black, white, red, orange, yellow, green, cyan, blue, purple, gradient, solid color, round tail, flat tail, wedge tail, pointed tail, convex tail, concave tail, forked tail, swallow tail, unequal toes, curved unequal toes, opposite toes, different toes, front toes, syndactyly, webbed feet, concave webbed feet, fully webbed feet, semi-webbed feet, flap webbed feet, and shape-empty;

[0067] The BCE formula is:

[0068]

[0069] In the formula, N represents the number of samples, M represents the number of labels, and y ij represents the true label, y ^ ij Represents the model output.

[0070] During model training, data is mainly obtained through search engines, open source databases, and actual scene shooting. The ffmpeg tool is used to segment the video, and the case application is 10-second segmentation. A processing script is written to extract frames and audio for the segmented video to determine whether there are birds in the image. If there are birds, it means valid data. The audio is converted into Mel spectrum / audio signal graph, the spectrum graph is observed and the audio is listened to. If there are bird sounds, it is valid data, and the relevant spectrum graph and audio are retained. The data annotation of target detection is divided into two groups. In the first group of video images, a bird needs to be annotated with two label boxes, namely xx order-xx family and xx genus-xx species. The second group is the spectrum graph to mark the area where the spectrum graph event occurs. Image data enhancement mainly includes random flipping, random filling, transparency, noise generation, random cutting, etc. Conventional classification is sufficient for audio classification, and audio enhancement is to randomly change the volume, audio filling, speed up / slow down, and background fusion. Then enter the training, and after the training is completed, the corresponding model is exported for different scenes.

[0071] Step 3: Get the video stream of the area to be detected, use OpenCV to read the frame image and store it in the image buffer, use ffmpeg to extract the audio and store it in the audio buffer, and perform preliminary bird family detection on the data in the image buffer. If birds are detected, continue with the subsequent steps, otherwise do not perform any operation;

[0072] Step 4: When a bird is detected in the image buffer in step 3, the genus and species model parameters of the corresponding order family in the target detection vector library are taken out for reading and loading, and the original image in the output box is used for bird genus and species detection, and the output results obtained are stored according to the time node;

[0073] Step 5: Then, the data in the audio buffer is converted into a Mel-spectrogram, and the bird call area of ​​the spectrum is detected, and the result is stored according to the time node for audio recognition;

[0074] Step 6: Then, the internal position detection model parameters of the model vector library are taken out for reading, the detection frame position is cropped to obtain the detection image, and the part detection is performed. After obtaining the detection frame, the image is cropped at the detection frame position, and the shape feature analysis is performed after obtaining these part images. According to the part information, the top five birds with the highest correlation are obtained in the neo4j relational database that stores bird attribute data, and the time node information is stored;

[0075] Step 7. Finally, the final bird probability distribution is obtained by weighted addition of target detection, audio detection, and attribute multi-classification storage values ​​to obtain the species of the bird with the highest probability.

[0076] A wild bird species identification method of the present invention adopts a hierarchical independent model, and the effect of each model can be independently evaluated for inference optimization. Combined with logic, it ensures multi-category prediction of 9,000 bird categories during use. Model parameters are stored in a vector database, which will not cause useless resource occupation. By using a small amount of occupied resources, the model can be used and released immediately, which relieves the pressure on the server while ensuring the recognition efficiency and accuracy. It is compatible with multi-terminal deployment and adapts to various application scenarios. At the same time, it can be equipped with multiple monitoring areas to achieve 24-hour observation, which can replace conventional manual observation, reduce human resource consumption, and avoid human safety factors. Functions can be added later, compatible with other recognition algorithms, and provide a more comfortable integrated environment for external secondary development.

[0077] In this way, the technical problem that the identification methods in the prior art are difficult to satisfy the requirement of fine classification of birds in different application areas while ensuring the stability of system performance and effective use of resources is solved.

[0078] The present invention has the following beneficial effects:

[0079] 1. Find the bird database to the maximum extent possible. The content information includes (name, order, family, genus, species), morphological diagram (flying, stopping, swimming...), appearance (head, beak, throat, abdomen, feet, tail, wings, feathers), appearance characteristics (color, style, shape), audio;

[0080] 2. Select the training server according to the needs. There are many use case models to ensure the progress of model training;

[0081] 3. The authenticity of the data must be high. For example, hand-drawn pictures and stick figures are invalid data.

[0082] Fourth, the hardware equipment needs to have a bird tracking function, because long-range image shooting will cause a large error between the model training data and the recognition data, resulting in the inability to obtain effective recognition information;

[0083] 5. The retention time of the buffer is fixed according to the server resources, and the memory required for subsequent image processing and model loading prediction is reserved;

[0084] 6. The audio conversion graph needs to be defined according to the type of data used for training to ensure the correct recognition of the model;

[0085] 7. Adding some negative samples to the model will improve the overall generalization ability of the model. Pay attention to the input length when processing audio.

[0086] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A method for identifying wild bird species, characterized in that: The steps include: Step 1: First create the image buffer and audio buffer; Step 2: Build a model vector library; Step 3: Get the video stream of the area to be detected, use OpenCV to read the frame image and store it in the image buffer, use ffmpeg to extract the audio and store it in the audio buffer, and perform preliminary bird family detection on the data in the image buffer. If birds are detected, continue with the subsequent steps, otherwise do not perform any operation; Step 4: When a bird is detected in the image buffer in step 3, the genus and species model parameters of the corresponding order family in the target detection vector library are taken out for reading and loading, and the original image in the output box is used for bird genus and species detection, and the output results obtained are stored according to the time node; Step 5: Then, the data in the audio buffer is converted into a Mel-spectrogram, and the bird call area of ​​the spectrum is detected, and the result is stored according to the time node for audio recognition; Step 6: Then, the internal position detection model parameters of the model vector library are taken out for reading, the detection frame position is cropped to obtain the detection image, and the part detection is performed. After obtaining the detection frame, the image is cropped at the detection frame position, and the shape feature analysis is performed after obtaining these part images. According to the part information, the top five birds with the highest correlation are obtained in the neo4j relational database that stores bird attribute data, and the time node information is stored; Step 7. Finally, the final bird probability distribution is obtained by weighted addition of target detection, audio detection, and attribute multi-classification storage values ​​to obtain the species of the bird with the highest probability.

2. The method for identifying wild bird species according to claim 1, characterized in that: The constructed model vector library uses three common deep learning algorithms, yolo, PANN+DenseNet, and ResNet, to handle corresponding target detection tasks, voiceprint classification tasks, and attribute multi-classification.

3. The method for identifying wild bird species according to claim 2, characterized in that: The detection model used in the target detection task uses the ppyoloe-plus-m framework of paddle, and the backbone network is cspresnet-m; the loss used is GIOU+DFL+VFL, and the optimizer selects Momentum.

4. The method for identifying wild bird species according to claim 3, characterized in that: The GIOU formula is: U=S((x hat 2-x hat 1),(y hat 2-y hat 1))+S((x2-x1),(y2-y1)); In the formula, A i y represents the predicted box i The y value of the point and the real box i The area of ​​the extended wireframe is the area of ​​the extra area after the points intersect. U is the area of ​​the real frame plus the area of ​​the predicted frame. hat i With y hat i Indicates the value coordinate of the corresponding point of the prediction box, y i With x i Indicates the value coordinates of the corresponding point of the real box, Y hat represents the predicted box, and Y represents the real box; 5. The method for identifying wild bird species according to claim 4, characterized in that: The DFL formula is: DFL=-((y hat i+1 -y)×log(S i )+(y-y i )×log(S i+1 )); In the formula, y represents the true value, y hat i The box representing the i-th predicted value.

6. The method for identifying wild bird species according to claim 5, characterized in that: The VFL formula is: Where IACS represents the iou-aware classification score of the prediction box, O h It represents the IOU between the predicted box and the real box. Both α and γ represent scaling factors. γ reduces the contribution of negative samples to the model, and α controls the weight distribution of positive samples to the model.

7. The method for identifying wild bird species according to claim 6, characterized in that: The voiceprint classification model used in the voiceprint classification task uses the torch framework model, and the torch framework model structure is divided into two parts; in the absence of sufficient data as support, the PANNS model trained with AudioSet data is used as a pre-trained model to act as the feature extraction part, and the downstream task part uses DenseNet as the basic atomic unit with normalization layer, pooling layer, and linear layer to complete the voiceprint classification; the characteristic of DenseNet is that each layer can directly access the features of all previous layers, thereby achieving feature reuse; CEL is used for loss, and adam is selected as the optimizer; the number of models is divided into 2, the first model label is: Ornithology, unknown, empty; the second model label is bird species.

8. The method for identifying wild bird species according to claim 7, characterized in that: The CEL formula is: In the formula, N represents the number of samples, M represents the number of categories, and y c represents the true c category, y ^ c Indicates the predicted category c.

9. The method for identifying wild bird species according to claim 8, characterized in that: The attribute multi-classification model uses the original model structure ResNet101 in the tensorflow framework as feature extraction, adds a layer of sigmoid as the final output layer, and uses sigmoid to convert multi-category labels into multiple binary classification labels to reduce model complexity and improve model speed; the loss function is BCE, and the optimizer is adam; there are 32 model label definitions, namely: black, white, red, orange, yellow, green, cyan, blue, purple, gradient, solid color, round tail, flat tail, wedge tail, pointed tail, convex tail, concave tail, forked tail, swallow tail, unequal toes, curved unequal toes, opposite toes, different toes, front toes, syndactyly, webbed feet, concave webbed feet, fully webbed feet, semi-webbed feet, flap webbed feet, and shape-empty; The BCE formula is: In the formula, N represents the number of samples, M represents the number of labels, and y ij represents the true label, y ^ ij Represents the model output.

Images