Snow leopard recognition method based on multi-modal deep learning and electronic device
By employing multimodal deep learning methods and utilizing infrared cameras to acquire multimodal information and images of snow leopards, combined with self-attention mechanisms and residual aggregation convolutional networks, efficient automatic identification of snow leopards was achieved, improving the identification accuracy and solving the problem of low efficiency in manual identification during snow leopard monitoring.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QINGHAI QILIAN MOUNTAIN NATURE RESERVE MANAGEMENT BUREAU
- Filing Date
- 2021-05-13
- Publication Date
- 2026-06-02
AI Technical Summary
In the current technology, there are few monitoring images of snow leopards and the efficiency of manual identification is low, making it difficult to automatically identify snow leopard images efficiently and consuming a lot of manpower and resources.
A multimodal deep learning-based approach is adopted, utilizing infrared cameras to acquire multimodal information and images. Through fully connected layers, twin autoencoders, and classification layers, combined with self-attention mechanisms and residual aggregation convolutional networks, visual and environmental features of snow leopards are extracted for identification.
It improved the accuracy of snow leopard identification, solved the problem of low efficiency in manual identification, and realized automated and accurate snow leopard monitoring.
Smart Images
Figure CN113283317B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning technology, specifically to a snow leopard recognition method and electronic device based on multimodal deep learning. Background Technology
[0002] The snow leopard (Panthera uncia) belongs to the order Carnivora, family Felidae, and genus Panthera. It is a Class I protected animal in my country and was listed as Vulnerable (VU) by the International Union for Conservation of Nature (IUCN) in 2017. Although belonging to the same Felidae family, snow leopards rarely interbreed with tigers (Panthera tigri) and leopards (Panthera pardus), and no hybrids have ever been produced. In contrast, lions (Panthera leo), tigers, leopards, and jaguars (Panthera onca) can interbreed, and there are records of hybrids. Therefore, the snow leopard was once classified as a separate genus (Uncia).
[0003] In the frigid environment of the plateau, snow leopards have evolved a series of morphological features adapted to the harsh conditions. They possess long, thick fur, the thickest and longest of all big cats. They molt twice a year, with the belly fur reaching up to 12cm in length during winter. Their leopard-patterned fur is mixed with smoky gray or creamy yellow, making them extremely similar to the lichens and bare rocks on the surrounding terrain, providing excellent camouflage.
[0004] Snow leopards are more active at night than during the day, with peak activity between 6:00 PM and 8:00 PM, exhibiting twilight activity. Their diurnal activity also varies seasonally, with higher concentrations of nocturnal activity in summer and autumn compared to winter and spring. Furthermore, snow leopard activity at night is influenced by lunar phases, with increased activity on brighter moonlit nights. Research indicates that snow leopard activity patterns are affected by habitat conditions such as terrain, human activity, and primary food sources, resulting in variations in spatial utilization patterns across different regions.
[0005] Monitoring snow leopards is a crucial part of their conservation and management; however, their unique habitat and high alertness make manual monitoring challenging. Using infrared trigger cameras for image monitoring is currently one of the most effective methods. However, due to the snow leopard's small population, the number of monitored images is relatively limited. Manually identifying and sifting snow leopard images from a large volume of monitoring data is labor-intensive and inefficient. Therefore, using computers to automatically identify snow leopard images can overcome these problems. Summary of the Invention
[0006] The purpose of this invention is to provide a snow leopard recognition method and electronic device based on multimodal deep learning.
[0007] Therefore, the technical solution provided by the present invention is as follows:
[0008] In a first aspect, the present invention provides a snow leopard identification method based on multimodal deep learning, comprising:
[0009] Acquire multimodal information and an image to be identified; the multimodal information includes shooting time, latitude and longitude, temperature and lunar phase, and the image to be identified includes a trigger image and a background image;
[0010] The multimodal information is input into the fully connected layer of the snow leopard recognition model, and a self-attention mechanism is introduced to obtain feature T1;
[0011] The trigger image and background image are input into the twin autoencoder of the snow leopard recognition model to obtain feature T2;
[0012] The features T1 and T2 are fused to obtain the fused feature T;
[0013] The fused feature T is input into the classification layer of the snow leopard recognition model to obtain the recognition result, thus realizing the recognition of snow leopards.
[0014] As one specific implementation method of this application, obtaining multimodal information specifically involves:
[0015] Images of snow leopard activity are captured using an infrared camera, and the shooting time, latitude and longitude, temperature, and lunar phase are extracted from these images.
[0016] As a preferred embodiment of this application, before inputting the multimodal information into the fully connected layer of the snow leopard recognition model, the method further includes:
[0017] The shooting time, latitude and longitude, temperature and lunar phase were normalized to form 12-bit binary codes.
[0018] As a specific implementation of this application, feature T1 is obtained as follows:
[0019] Four different types of multimodal information are input into the fully connected layer to obtain the output results;
[0020] Adding a self-attention mechanism yields four different importance weights for multimodal information;
[0021] The output result is multiplied by the importance weight and then fused to obtain feature T1.
[0022] As a specific implementation of this application, feature T2 is obtained as follows:
[0023] The trigger image and background image are respectively input into the encoding module of the twin autoencoder to obtain latent representations h1 and h2; the encoding module adopts an attention mechanism and a residual aggregation convolutional network;
[0024] The latent representations h1 and h2 are input into the decoding module of the twin autoencoder to obtain feature T2; the decoding module adopts a deconvolution + fully connected network.
[0025] Furthermore, as a preferred embodiment of this application, before acquiring multimodal information, the method includes training the snow leopard recognition model, specifically:
[0026] Multiple training samples are acquired, and a sample set is formed based on the multiple training samples; each training sample is a set consisting of four types of multimodal information, a trigger image, and a background image at any given time, and the sample set is formed by multiple training samples at different times;
[0027] The first training phase involves training the Siamese autoencoder using the trigger and background images from the training samples to obtain its initial parameters. During this training process, the loss function is the mean square error of the two inputs and outputs of the Siamese autoencoder.
[0028] The second training phase involves using complete training samples to jointly train the initially trained Siamese autoencoder, fully connected layer, and classification layer to optimize the parameters of the entire snow leopard recognition model. During this training process, the loss function is the sum of the weighted cross-entropy loss function and the loss function used to train the Siamese autoencoder.
[0029] In a second aspect, embodiments of the present invention provide an electronic device, including a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to invoke the program instructions to execute the method described in the first aspect.
[0030] By implementing embodiments of the present invention, visual features of snow leopards are obtained by monitoring images (trigger images and background images), and four other modal features reflect the activity habits of snow leopards. The two are combined to identify snow leopards. That is, the present invention uses deep learning and makes full use of the information features of the snow leopard's activity environment to identify snow leopards, thereby improving the accuracy of identification. Attached Figure Description
[0031] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.
[0032] Figure 1 A flowchart illustrating the snow leopard recognition method based on multimodal deep learning provided in this embodiment of the invention;
[0033] Figure 2This is a network diagram of the present invention;
[0034] Figure 3 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0036] The inventive concept of this invention is as follows: Snow leopards have evolved extremely high camouflage capabilities to adapt to their environment, making the extraction of visual features difficult. Furthermore, research has shown that different times and locations affect the appearance of snow leopards, and lunar phases and temperatures influence their activity. Therefore, time, location, lunar phase, and temperature can provide auxiliary information for identifying snow leopards using infrared cameras. Based on this, this invention proposes to employ deep learning and fully utilize the environmental information features of snow leopard activity to perform multimodal snow leopard identification, thereby improving the accuracy of snow leopard identification. It also solves the technical problem of the existing technology that requires significant manpower and resources to manually sift through a large number of monitoring images of snow leopards.
[0037] Based on this invention concept, please refer to Figure 1 and Figure 2 The snow leopard recognition method based on multimodal deep learning provided by this invention mainly includes:
[0038] S101, training a snow leopard recognition model.
[0039] like Figure 2 As shown, the snow leopard recognition model mainly includes a T1 feature extraction network, a Siamese autoencoder, and a classification layer (classification network). Its training process mainly includes:
[0040] Multiple training samples are acquired, and a sample set is formed based on the multiple training samples; each training sample is a set consisting of four types of multimodal information, a trigger image, and a background image at any given time, and the sample set is formed by multiple training samples at different times;
[0041] The initial parameters of the twin autoencoder are obtained by training the trigger image and background image in the training samples.
[0042] The entire snow leopard recognition model was trained using complete training samples. The initial trained twin autoencoder, fully connected layer, and classification layer were trained together to optimize the parameters of the entire model.
[0043] That is, in this embodiment, a joint training method is adopted:
[0044] Phase 1: The Siamese autoencoder is trained using only the trigger images and background images from the training samples to obtain initial parameters (this part does not require labels and belongs to unsupervised clustering). The training loss function is the sum of the mean squared errors (MSE) of the two inputs and outputs.
[0045] The second stage involves inputting a complete training sample and training it together with the T1 feature extraction network and the classification network to optimize the parameters of the entire model. At this stage, the loss function is the sum of the weighted cross-entropy loss function and the loss function from the first stage.
[0046] S102, acquire multimodal information and the image to be recognized.
[0047] The multimodal information includes shooting time, latitude and longitude, temperature and lunar phase, and the image to be identified includes a trigger image and a background image.
[0048] Specifically, images of snow leopard activity are captured using an infrared camera, and the following information is extracted from these images: capture time (0-24h), latitude and longitude (-180°—180°, -90°—90°), temperature (-40℃—80℃), and lunar phase (0—30).
[0049] Furthermore, during infrared camera monitoring, the shooting model is set to capture an untriggered image immediately after one or more trigger shots are completed, thus obtaining a triggered image and a background image. The label indicates whether the triggered image is a snow leopard.
[0050] S103, the multimodal information is input into the fully connected layer of the snow leopard recognition model, and a self-attention mechanism is introduced to obtain feature T1.
[0051] Specifically, the shooting time, latitude and longitude, temperature, and lunar phase were normalized and each generated into a 12-bit binary code (multimodal information 1-4), which was then input into a fully-connected layer (fc). To obtain the importance of different modalities representing the snow leopard's activity habits, a self-attention mechanism (SA) was added to obtain the importance of different modal information. This information was then multiplied by the original features and fused to obtain feature T1.
[0052] S104, input the trigger image and background image into the twin autoencoder of the snow leopard recognition model to obtain feature T2.
[0053] Specifically, in this embodiment, a twin autoencoder is designed to extract features from the trigger image and the background image respectively. Finally, two latent representations, h1 and h2, are input into a fully connected layer to output the difference feature T2 between the two. The two autoencoders in the twin autoencoder have identical structures and parameters, which helps to improve the difference feature. In the encoder part, a residual aggregation convolutional network is used, with dilated convolutions replacing traditional convolutions, and an attention mechanism is applied to enhance image feature extraction capabilities. The decoder part can employ deconvolution and fully connected layers to achieve effective image reconstruction.
[0054] S105, the features T1 and T2 are fused to obtain the fused feature T.
[0055] S106, input the fused feature T into the classification layer of the snow leopard recognition model to obtain the recognition result and realize the recognition of snow leopard.
[0056] The snow leopard identification method of this invention obtains the visual features of the snow leopard by monitoring images (trigger image and background image), and the other four modal features reflect the snow leopard's activity habits. The two are combined to identify the snow leopard. That is, this invention adopts a deep learning approach and makes full use of the snow leopard's activity environment information features to identify the snow leopard, thereby improving the accuracy of identification.
[0057] Furthermore, multimodal information is not currently used in animal identification, nor has it been selected for animal identification. Therefore, this invention fills this technological gap.
[0058] Based on the same inventive concept, and corresponding to the above-described method embodiments, this invention also provides an electronic device. For example... Figure 3 As shown, the electronic device may include one or more processors 101, one or more input devices 102, one or more output devices 103, and a memory 104. The processors 101, input devices 102, output devices 103, and memory 104 are interconnected via a bus 105. The memory 104 stores a computer program, which includes program instructions. The processor 101 is configured to invoke the program instructions to execute the method described in the embodiment of the snow leopard recognition method based on multimodal deep learning.
[0059] It should be understood that, in this embodiment of the invention, the processor 101 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.
[0060] Input device 102 may include a keyboard, etc., and output device 103 may include a display (LCD, etc.), a speaker, etc.
[0061] The memory 104 may include read-only memory and random access memory, and provides instructions and data to the processor 101. A portion of the memory 104 may also include non-volatile random access memory. For example, the memory 104 may also store device type information.
[0062] In specific implementations, the processor 101, input device 102, and output device 103 described in the embodiments of the present invention can execute the implementation methods described in the embodiments of the snow leopard recognition method based on multimodal deep learning provided in the embodiments of the present invention, which will not be repeated here.
[0063] It should be noted that for a more detailed description of the working process and beneficial effects of the electronic device, please refer to the aforementioned method implementation section, which will not be repeated here.
[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A snow leopard recognition method based on multi-modal deep learning, characterized in that, include: Acquire multimodal information and an image to be identified; the multimodal information includes shooting time, latitude and longitude, temperature and lunar phase, and the image to be identified includes a trigger image and a background image; The multimodal information is input into the fully connected layer of the snow leopard recognition model, and a self-attention mechanism is introduced to obtain feature T1; The trigger image and background image are input into the twin autoencoder of the snow leopard recognition model to obtain feature T2; The features T1 and T2 are fused to obtain the fused feature T; The fused feature T is input into the classification layer of the snow leopard recognition model to obtain the recognition result, thereby realizing the recognition of snow leopards; The specific feature T2 obtained is as follows: The trigger image and background image are respectively input into the encoding module of the twin autoencoder to obtain latent representations h1 and h2; the encoding module adopts an attention mechanism and a residual aggregation convolutional network; The latent representations h1 and h2 are input into the decoding module of the twin autoencoder to obtain feature T2; the decoding module uses a deconvolution + fully connected network. Before acquiring multimodal information, the method includes training the snow leopard recognition model, specifically: Multiple training samples are acquired, and a sample set is formed based on the multiple training samples; each training sample is a set consisting of four types of multimodal information, a trigger image, and a background image at any given time, and the sample set is formed by multiple training samples at different times; The first training phase involves training the Siamese autoencoder using the trigger and background images from the training samples to obtain the initial parameters of the Siamese autoencoder. During this training process, the loss function is the mean square error of the two inputs and outputs of the twin autoencoder; The second training phase involves using complete training samples to jointly train the initially trained Siamese autoencoder, fully connected layer, and classification layer to optimize the parameters of the entire snow leopard recognition model. During this training process, the loss function is the sum of the weighted cross-entropy loss function and the loss function used to train the Siamese autoencoder.
2. The snow leopard identification method of claim 1, wherein, The specific steps for obtaining multimodal information are as follows: Images of snow leopard activity are captured using an infrared camera, and the shooting time, latitude and longitude, temperature, and lunar phase are extracted from these images.
3. The snow leopard identification method of claim 2, wherein, Before inputting the multimodal information into the fully connected layer of the snow leopard recognition model, the method further includes: The shooting time, latitude and longitude, temperature and lunar phase were normalized to form 12-bit binary codes.
4. The snow leopard identification method of claim 1, wherein, The specific feature T1 obtained is as follows: Four different types of multimodal information are input into the fully connected layer to obtain the output results; Adding a self-attention mechanism yields four different importance weights for multimodal information; The output result is multiplied by the importance weight and then fused to obtain feature T1.
5. An electronic device, comprising: The system includes a processor, an input device, an output device, and a memory, which are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to invoke the program instructions to execute the method as described in claim 1.