Wild animal monitoring image species classification method based on point context

Abstract

The invention relates to a wild animal monitoring image species classification method based on point context, and belongs to the technical field of image classification. The method comprises the steps of modeling correlation between monitoring point locations and species, distributing weights for samples according to the strength of the correlation, and decoupling point location classification and species classification tasks to learn environment features and species features. According to the method, environment information represented by point locations serves as context of species classification, the understanding of a model on the relationship between species and habitat environments is enhanced, and effective generalization of data outside distribution is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification methods, and in particular relates to a method for classifying species of wildlife monitoring images based on point context. Background Art

[0002] The scale of wildlife resource surveys and biodiversity monitoring is constantly expanding, and the number of monitoring images is growing. In recent years, my country's wildlife surveys, especially those on terrestrial vertebrates, have made rapid progress in both survey scale (geographical area, animal group coverage) and survey methods. Technologies such as infrared triggered cameras provide an effective means of data collection, and wildlife resource surveys and monitoring conducted by national parks, nature reserves at all levels, and forestry departments provide a rich and diverse source of data.

[0003] The distribution shift of wildlife monitoring data leads to a decline in model performance. Wildlife monitoring work spans a long time and covers a wide geographical area. The data collection process is affected by objective uncontrollable factors such as weather, light, and the natural environment of the monitoring site, as well as subjective differences such as the specific installation location and shooting parameters of the camera. As a result, wildlife monitoring images collected at different monitoring points or with different cameras have differences in data distribution. In addition, due to the different living habits of different wild animals and the disparity in population size, monitoring images are often biased in terms of sample size, and categories with smaller sample sizes may have higher research value (such as rare wild animals). The shift in category distribution can easily cause the model to be biased towards the head category, affecting the classification effect of the tail category, resulting in the loss of opportunities to discover and observe species.

[0004] Deep learning models usually assume that the training data and test data come from the same distribution, but in the classification of wildlife monitoring images, the data distribution of the training set and the test set are inconsistent. This distribution shift makes it difficult for the model to generalize to new application environments, and the large amount of labeled data and computing resources required to train a specific model for each monitoring environment or camera are not cost-effective. The challenge of generalizing to unknown data distributions hinders the large-scale application of species classification models. Therefore, how to improve the out-of-distribution generalization ability of wildlife monitoring image species classification models so that they can be applied to monitoring images in different environments is a problem that needs to be solved. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The purpose of the present invention is to propose a species classification method for wildlife monitoring images based on point context to solve the problems raised in the background technology. The present invention enhances the model's understanding of the relationship between species and habitat environment and achieves effective generalization of out-of-distribution data.

[0007] (II) Technical solution

[0008] In order to achieve the above object, the present invention adopts the following technical scheme:

[0009] The method for species classification of wildlife monitoring images based on point context includes the following steps:

[0010] S1. Input the wildlife monitoring image into the encoder for feature extraction, and analyze and process the acquired features to facilitate model learning;

[0011] S2, dividing the image features obtained in S1 into species features and point features, and designing a method for allocating sample weights according to species features and point features to enhance the learning process of the model;

[0012] S3, input the species characteristics and point characteristics into the species classifier and point classifier respectively, use the classifier to classify the input species characteristics and point characteristics to obtain the prediction results, calculate the species classification loss and point classification loss, and re-weight the classification loss and point classification loss to form the final LoCo loss;

[0013] S4. Based on the operations described in S1 to S3, a wildlife monitoring image classification model is constructed, and the learning and training of the model is completed; the trained model is used to complete the estimation of the sample loss weights of each species in different environments and the species classification work.

[0014] Preferably, the S1 specifically includes the following contents:

[0015] The correlation between habitat environment and species category is modeled based on the uncertainty of data distribution. The strength of the correlation is used to describe the difficulty of the sample, and difficult samples with repeated pseudo-correlation features are distinguished. The corresponding sample weights are assigned, and the learning of environmental information represented by the point and species characteristics are decoupled. The point is used as the context information of the species classification task, and then generalization out of distribution is achieved, forming point context loss, which guides the model to perceive point differences and learn generalizable features.

[0016] Preferably, the method for allocating sample weights according to species characteristics and site characteristics in S2 specifically includes the following contents:

[0017] The species uncertainty of point distribution and the point uncertainty of species distribution are proposed to describe the prior knowledge contained in the point information. The calculation formula is as follows:

[0018]

[0019] Among them, U t is the species’ point distribution uncertainty, U c is the species category uncertainty of the point, P tFor a given point Y c The species label distribution at P c For a given species Y c The point distribution at that time, N is the number of species, M is the number of camera points;

[0020] The point information is used as the context to guide the model to learn the relationship between species and background or habitat environment during training. The correlation is used to enhance the learning process of the species classification model. The normalized mutual information correlation score is calculated. The specific calculation formula is:

[0021]

[0022] Where p(x,y) is the joint probability distribution function of X and Y; p(x) and p(y) are marginal probability distribution functions; MI is the mutual information, H(·) is the entropy calculation function, and NMI is the normalized mutual information.

[0023] Preferably, the species classification loss and the point classification loss are calculated in S3, and the classification loss and the point classification loss are re-weighted to form the final LoCo loss, which specifically includes the following contents:

[0024] Use weighted loss for optimization and select softmax as the weight scaling function. The specific calculation formula is:

[0025]

[0026] Among them, w t is the weight of species classification loss, w c is the weight of the point classification loss, Λ(·) represents the weight scaling function, w t ′ and w c ′ together constitute the input vector of softmax, adjust the weights of the above two tasks to make the model focus on species classification, and introduce a hyperparameter to control the degree of weight scaling at the sample level;

[0027] The supervision process of CoLoCo is decoupled to make it LoCo, which clearly separates the environmental features used for point classification and the animal features used for species classification. The feature extraction process, model optimization process and the specific input of the classifier are:

[0028] x t ,x c =φ(x)

[0029] p t ,p c =f t (concat(x t ,detach(xc ))),f c (x c )

[0030] L 1c (p t ,p c ,y t ,y c )=L t (p t ,y t )+L c (p c ,y c )

[0031] Among them, x t is the species characteristic, x c is the point feature, represents the encoder that extracts features from the input image x, f t is the species classifier, f c is the point classifier, p t and p c represents the result after classification; y represents the supervised label, the concat(·) operation represents feature concatenation, and the detach(·) operation represents the cancellation of the back propagation of species classification loss in point features to decouple the optimization process of species features and point features.

[0032] (III) Beneficial effects

[0033] (1) To address the problem that the distribution bias of wildlife monitoring data reduces the generalization performance of the model, the present invention uses point labels to learn environmental information to improve the robustness of the model to out-of-distribution data. Then, the correlation priors between point labels and species labels observed in wildlife monitoring data are qualitatively and quantitatively analyzed, and the nature of the pseudo-correlation and its two-sided nature are pointed out.

[0034] (2) This paper proposes two uncertainties to model the relationship between point context and species category, and formally describes the proposed point context-guided species classification method.

[0035] (3) The present invention verifies the actual effect of the method through experiments from various angles. Compared with the best baseline method, the Macro F1 score on the out-of-distribution test sets of the iWildCam and TerraInc datasets is improved by 3.1% and 1.9%, respectively, while the accuracy is improved by 0.5% and 3.2%, respectively. Finally, in closed-set scenarios, the method improves the recognition effect of the model for known categories and the generalization performance of out-of-distribution data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings involved in the embodiments are briefly introduced. Obviously, the drawings in the following description are only schematic illustrations of some embodiments of the present invention. For those skilled in the art, other forms of drawings can also be constructed based on these drawings without creative work.

[0037] Figure 1 This is a flow chart of the method for species classification of wildlife monitoring images based on point context proposed by the present invention. DETAILED DESCRIPTION

[0038] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0039] The present invention provides a method for species classification of wildlife monitoring images based on point context, which models the correlation between monitoring points and species, assigns weights to samples according to the strength of the correlation, and decouples point classification and species classification tasks to learn environmental characteristics and species characteristics. By using the environmental information represented by the points as the context for species classification, the method enhances the model's understanding of the relationship between species and habitat environment, and improves the model's recognition effect on known categories and generalization performance on out-of-distribution data in closed-set scenarios.

[0040] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0041] The following describes a method for classifying species in wildlife monitoring images based on point context according to an embodiment of the present invention with reference to the accompanying drawings.

[0042] Embodiment 1:

[0043] like Figure 1 As shown, the wildlife monitoring image species classification method includes the following steps:

[0044] First, the input image is put into the encoder for feature extraction. The correlation between habitat environment and species category is modeled based on the uncertainty of data distribution. The strength of the correlation is used to describe the difficulty of the sample. The difficult samples that are challenging due to repeated pseudo-correlation features are distinguished, and the corresponding sample weights are assigned. The environmental information represented by the point is decoupled from the learning of species characteristics. The point is used as the context information of the species classification task, thereby achieving out-of-distribution generalization, forming point context loss, and guiding the model to perceive point differences and learn generalizable features.

[0045] Secondly, the obtained features are composed of species features and site features. The prior knowledge contained in the site information is described according to the two proposed uncertainties, including the species uncertainty of site distribution and the site uncertainty of species distribution. The site information is used as the context to guide the model to learn the relationship between species and background or habitat environment during training. The proposed site context-guided species classification method is formally described, explaining how the species classification model uses the observed correlation to enhance the learning process, and calculating the normalized mutual information correlation score:

[0046]

[0047] Among them, p(x,y) is the joint probability distribution function of X and Y, p(x), p(y) are the marginal probability distribution functions. MI is the mutual information, H(·) is the entropy calculation function, and NMI is the normalized mutual information.

[0048] Then, the species classifier and the point classifier classify the features to obtain the prediction results, and the two parts of the loss are reweighted to form the final LoCo loss. The detach(·) operation is used to cancel the back propagation of the species classification loss in the point features to decouple the optimization process of species features and point features, and avoid the task competition that occurs during the feature extraction process to hinder the expression of environmental information. Using a method that uses point locations, i.e. environmental information, as the context of the species classification problem, the uncertainty of the species point distribution and the uncertainty of the species category of the point are proposed, and the formula is:

[0049]

[0050] Among them, U t is the species’ point distribution uncertainty, U c is the species category uncertainty of the point, P t For a given point Y c The species label distribution at P c For a given species Y c Point distribution at the time, N is the number of species, and M is the number of camera points.

[0051] Using weighted loss for optimization, softmax is selected as the function of weight scaling, and the formula is:

[0052]

[0053] Among them, w t is the weight of species classification loss, w c is the weight of the point classification loss, and They are abbreviated as w t 'and w' c , together constitute the input vector of softmax, adjust the weights of the two tasks to make the model focus on species classification, and introduce a hyperparameter to control the degree of weight scaling at the sample level.

[0054] The CoLoCo decoupled from the supervision process is used. The CoLoCo decoupled from the supervision process becomes LoCo, which clearly separates the environmental features for point classification and the animal features for species classification. The feature extraction process, model optimization process and specific input of the classifier are:

[0055] x t ,x c =φ(x)

[0056] p t ,p c =f t (concat(x t ,detach(x c ))),f c (x c )

[0057] L 1c (p t ,p c ,y t ,y c )=L t (p t ,y t )+L c (p c ,y c )

[0058] Among them, x t is the species characteristic, x c is the point feature, f t is the species classifier, f c is a point classifier, and the result after classification is p t and p c .

[0059] Finally, the two proposed uncertainties were used to pre-estimate the sample loss weights of each species in different environments. In the loss calculation stage, the corresponding weights can be directly queried by the species labels and point labels to perform loss weighting.

[0060] Embodiment 2:

[0061] Based on Example 1, but different in that the present invention proposes a method of using point location, i.e., environmental information, as the context of species classification problems, aiming to improve the species classification ability of the model by utilizing the correlation between the image background (environment) and the foreground (wild animals). Next, the task definition and the formal description of the proposed method are given. Samples with monotonous background and lack of foreground change (i.e., categories distributed in fewer monitoring points and points with fewer species) are regarded as difficult samples for species classification, and a larger weight for species classification loss is given. The CoLoCo that decouples the supervision process is called LoCo (i.e., LocationContext Loss). The method in this chapter follows the widely used evaluation protocols such as the WILDS out-of-distribution generalization evaluation benchmark, i.e., using an out-of-distribution validation set containing 75 categories for model selection, and reporting the standard deviation and mean of the results through repeated experiments with 3 different random seeds. This method uses accuracy and Macro F1 score as the main evaluation indicators and compares with existing methods. Similar to WILDS, this chapter focuses on the Macro F1 score of the out-of-distribution test set to evaluate the out-of-distribution generalization performance of the model. Given that it is difficult to evaluate the recognition effect of tail categories based on the overall accuracy, this chapter also reports the accuracy on non-overlapping category subsets divided by sample frequency (i.e., the overall accuracy of the head category, middle category, and tail category test samples) to evaluate the robustness of the model to the category distribution shift of the samples.

[0062] The present invention verifies the actual effect of the method through experiments from various angles. Compared with the best baseline method, the Macro F1 score on the out-of-distribution test sets of the iWildCam and TerraInc datasets is improved by 3.1% and 1.9%, respectively, while the accuracy is improved by 0.5% and 3.2%, respectively. Finally, in closed-set scenarios, this method improves the recognition effect of the model for known categories and the generalization performance of out-of-distribution data (see Tables 1 to 3).

[0063]

[0064] Table 1

[0065]

[0066] Table 2

[0067]

[0068] Table 3

[0069] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0071] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0072] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0073] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0074] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. A method for species classification of wildlife monitoring images based on point context, characterized in that: The following steps are involved: S1. Input the wildlife monitoring image into the encoder for feature extraction, and analyze and process the acquired features to facilitate model learning; S2, dividing the image features obtained in S1 into species features and point features, and designing a method for allocating sample weights according to species features and point features to enhance the learning process of the model; S3, input the species characteristics and point characteristics into the species classifier and point classifier respectively, use the classifier to classify the input species characteristics and point characteristics to obtain the prediction results, calculate the species classification loss and point classification loss, and re-weight the classification loss and point classification loss to form the final LoCo loss; S4. Based on the operations described in S1 to S3, a wildlife monitoring image classification model is constructed, and the learning and training of the model is completed; the trained model is used to complete the estimation of the sample loss weights of each species in different environments and the species classification work.

2. The method for species classification of wildlife monitoring images based on point context according to claim 1 is characterized in that: The S1 specifically includes the following contents: The correlation between habitat environment and species category is modeled based on the uncertainty of data distribution. The strength of the correlation is used to describe the difficulty of the sample, and difficult samples with repeated pseudo-correlation features are distinguished. The corresponding sample weights are assigned, and the learning of environmental information represented by the point and species characteristics are decoupled. The point is used as the context information of the species classification task, and then generalization out of distribution is achieved, forming point context loss, which guides the model to perceive point differences and learn generalizable features.

3. The method for species classification of wildlife monitoring images based on point context according to claim 1, characterized in that: The method for allocating sample weights based on species characteristics and site characteristics described in S2 specifically includes the following: The species uncertainty of point distribution and the point uncertainty of species distribution are proposed to describe the prior knowledge contained in the point information. The calculation formula is as follows: Among them, U t is the species’ point distribution uncertainty, U c is the species category uncertainty of the point, P t For a given point Y c The species label distribution at P c For a given species Y c The point distribution at that time, N is the number of species, M is the number of camera points; The point information is used as the context to guide the model to learn the relationship between species and background or habitat environment during training. The correlation is used to enhance the learning process of the species classification model. The normalized mutual information correlation score is calculated. The specific calculation formula is: Where p(x,y) is the joint probability distribution function of X and Y; p(x) and p(y) are marginal probability distribution functions; MI is the mutual information, H(·) is the entropy calculation function, and NMI is the normalized mutual information.

4. The method for species classification of wildlife monitoring images based on point context according to claim 1, characterized in that: The species classification loss and point classification loss are calculated as described in S3, and the classification loss and point classification loss are reweighted to form the final LoCo loss, which specifically includes the following: Use weighted loss for optimization and select softmax as the weight scaling function. The specific calculation formula is: Among them, w t is the weight of species classification loss, w c is the weight of the point classification loss, Λ(·) represents the weight scaling function, w t ′ and w c ′ together constitute the input vector of softmax, adjust the weights of the above two tasks to make the model focus on species classification, and introduce a hyperparameter to control the degree of weight scaling at the sample level; The supervision process of CoLoCo is decoupled to make it LoCo, which clearly separates the environmental features used for point classification and the animal features used for species classification. The feature extraction process, model optimization process and the specific input of the classifier are: x t ,x c =φ(x) p t ,p c =f t (concat(x t ,detach(x c ))),f c (x c ) L 1c (p t ,p c ,y t ,y c )=L t (p t ,y t )+L c (p c ,y c ) Among them, x t is the species characteristic, x c is the point feature, represents the encoder that extracts features from the input image x, f t is the species classifier, f c is the point classifier, p t and p c represents the result after classification; y represents the supervised label, the concat(·) operation represents feature concatenation, and the detach(·) operation represents the cancellation of the back propagation of species classification loss in point features to decouple the optimization process of species features and point features.

Images