Dynamic monitoring method for dolphin behaviors for dolphin protection

Abstract

The invention discloses a dynamic monitoring method for a dolphin behavior for dolphin protection, and the method comprises the steps: arranging a dolphin hydrophone collection device, a dolphin water surface video shooting device and a water environment monitoring device in a dolphin protection water area; the method comprises the following steps of: acquiring audio data sent by a cowfish to acquire five environmental metadata including water temperature, air temperature, dissolved oxygen content, average water level measured value of a water area of a protection area and fishing amount of fish and shrimp in the water body, and marking data according to a cowfish behavior image shot by a cowfish water surface image shooting device during audio acquisition of the cowfish and according to a cowfish behavior attention category; and training and deploying the audio fusion identification model of the cowfish, so that the behaviors of the cowfish can be identified more accurately. According to the method and the system, the behavior condition of the cowfish in the protection area under the time window can be judged by acquiring the easily acquired cowfish audio data in combination with the environment, so that the cowfish community condition in the protection area can be dynamically monitored, and the cowfish can be better protected.

Description

Technical Field

[0001] The present invention relates to the technical field of finless porpoise audio processing, and in particular to a finless porpoise behavior dynamic monitoring method for finless porpoise protection. Background Art

[0002] The Yangtze finless porpoise, commonly known as the "River pig", is one of the few freshwater mammals with extremely important scientific research value. As a creature at the top of the food chain in the Yangtze River, it is of great value in maintaining the ecological balance of the Yangtze River waters. However, due to the increase in human activities, the survival of the Yangtze finless porpoise has been seriously threatened. The number of Yangtze finless porpoises is not optimistic. In order to protect the Yangtze finless porpoise, various places along the Yangtze River have also taken a variety of measures. Among the protection measures for Yangtze finless porpoises, the establishment of nature reserves is very important, which greatly facilitates the protective research on Yangtze finless porpoises, especially the convenient acquisition of Yangtze finless porpoise audio and corresponding behavioral observations. For example, a Chinese patent (CN116778405A) discloses a method for monitoring Yangtze finless porpoises by drone patrol, which uses several drones to patrol side by side for dragnet inspection. During the dragnet inspection, the distance between adjacent drones is 0.9 to 1.1 km, and the drones fly at an altitude of 70 to 90 meters. The drones use cameras equipped with zoom lenses to capture images and identify Yangtze finless porpoises based on the images; during the dragnet inspection and fixed-point inspection, deep learning methods are used to detect and identify finless porpoises in the video; after the Yangtze finless porpoise is found by the dragnet inspection, a fixed-point inspection area is demarcated according to the position where the finless porpoise emerges from the water, and two drones are used to patrol and track the fixed-point inspection area; during the tracking inspection, the drones fly at an altitude of 70 to 80 meters; one of the drones conducts a fixed-point inspection after discovering the finless porpoise, and then the other drone uses a wide-angle lens to capture the whereabouts of the finless porpoise. Drones are used for tracking, and monitoring accuracy is higher, but the cost is still relatively high compared to hydroacoustic monitoring.

[0003] At present, conventional finless porpoise hydroacoustic technology mainly relies on a single audio as an identification factor. However, from a biological perspective, organisms are a product of the environment, and their reactions in different environments are generally different. As a freshwater mammal, the Yangtze finless porpoise is bound to have emotional reactions to the environment like higher mammals. Therefore, if environmental factors are combined for identification and analysis, it can not only enrich the input of the model and improve the model's ability to understand data, but also more accurately identify the reaction behavior corresponding to the finless porpoise audio. Therefore, based on this situation, we proposed a dynamic monitoring method for finless porpoise behavior using the finless porpoise living environment data, finless porpoise audio and the corresponding behaviors observed in the protected area. Summary of the invention

[0004] In view of the above existing problems, the present invention provides a method for dynamically monitoring the behavior of finless porpoises for the protection of finless porpoises. The data on the living environment of finless porpoises in the protected area, the audio of finless porpoises and the observed behavior are used to construct a data set, and a finless porpoise audio fusion recognition model is trained and deployed. The behavior of finless porpoises can be identified more accurately. In this way, the behavior of finless porpoises in the protected area within a time window (for example, within one hour) can be judged by obtaining the relatively easy-to-obtain audio data of finless porpoises in combination with the environment in which they are located, which is helpful for dynamically monitoring the situation of finless porpoise populations in the protected area, so that the finless porpoises can be better protected.

[0005] To achieve the above object, the present invention provides a method for dynamic monitoring of finless porpoise behavior for finless porpoise protection, comprising the following steps:

[0006] S1. Deploy finless porpoise hydroacoustic collection equipment, finless porpoise surface video shooting equipment and water environment monitoring equipment in the finless porpoise protection waters to collect the audio data emitted by the finless porpoises and collect the five environmental metadata of water temperature, air temperature, dissolved oxygen content, average water level measurement value of the protected area waters, and the catchable amount of fish and shrimp in the water body. According to the finless porpoise behavior images captured by the finless porpoise surface video shooting equipment during the finless porpoise audio collection, complete the data labeling with the finless porpoise behavior focus category C, and complete the cropping of the finless porpoise audio during labeling. Then repeat the above collection and labeling process to construct the initial data set of finless porpoise audio and corresponding environmental metadata;

[0007] S2. Use a bandpass filter to intercept the high-frequency band in the finless porpoise audio signal, and perform short-time Fourier transform (STFT) on the filtered audio signal to obtain a spectrogram X(k, t) of the finless porpoise audio signal, and perform normalization processing to obtain a normalized spectrogram L. Where X max =max(X(k,t)),X min =min(X(k,t)), L is the normalized spectrogram, and the elements in the matrix L are between 0 and 1;

[0008] S3, normalize the five environmental metadata, namely water temperature, air temperature, dissolved oxygen content, average water level measurement value of the protected area, and the catchable amount of fish and shrimp in the water body, and obtain the corresponding normalized values ​​D norn (i) The values ​​of i range from 1 to 5, corresponding to water temperature, air temperature, dissolved oxygen content, average water level measurement value of the protected area, and the catchable amount of fish and shrimp in the water body. min(i) is the minimum value corresponding to the environmental data category, D max(i) The maximum value of the corresponding environmental data category;

[0009] S4, repeating steps S2 and S3, after all the finless porpoise audios of the initial data set are processed, a normalized data set is obtained, which ensures that the data are distributed in the same range, avoids the deviation between different input scales during the model training process, and constructs a basic data set of finless porpoise audios;

[0010] S5, convert the normalized spectrogram of the finless porpoise audio into a matrix L of size M×N, The corresponding normalized environment data forms a one-dimensional vector Fuse the environment data vector e with the matrix L to obtain the fusion matrix L of k channels k , Fusion matrix L k The input is fed into a convolutional neural network model with self-attention and residual network. After multiple rounds of training, a finless porpoise audio fusion recognition model is obtained.

[0011] S6. Collect geographical data in the finless porpoise reserve through multi-beam sonar equipment, use 3D modeling tools based on 3D visualization modeling technology to build a visualized water terrain digital twin model based on the finless porpoise reserve, use passive sonar equipment, and use the time difference of orientation (TDOA) algorithm to determine the location of the finless porpoise in real time based on the sound waves of the finless porpoise, and display it in real time in the visualized water terrain digital twin model;

[0012] S7, convert the sound waves of the finless porpoise into the finless porpoise audio, deploy the finless porpoise audio fusion recognition model for inference, and generate the feature fusion matrix L by collecting the finless porpoise audio and the corresponding water temperature, air temperature, dissolved oxygen, total phosphorus concentration in water quality, and ammonia nitrogen concentration in water quality in real time. k , the fusion matrix L k After inference of the finless porpoise audio fusion recognition model, the finless porpoise behavior focus category is identified and displayed in real time on the visualized water terrain digital twin model.

[0013] Furthermore, it also includes step S8, setting a comprehensive warning signal strength P according to the expert experience database, and by judging the behavior category of the finless porpoise and its position in the visualized water terrain digital twin model, if the comprehensive warning signal strength P exceeds a predetermined threshold, a monitoring and early warning prompt will be issued and displayed in real time on the visualized water terrain digital twin model.

[0014] Further, in step S8, the comprehensive warning signal strength P, P = αB + βG, where B represents the score of the finless porpoise's behavioral characteristics, and the larger the value, the more abnormal the behavior; G represents the terrain feature score, and the larger the value, the greater the impact of the terrain on the finless porpoise's behavior; α and β represent the weight coefficients of the behavioral characteristics and terrain characteristics, respectively, which are set by expert experience and are used to adjust the impact of different characteristics on the comprehensive warning signal.

[0015] Furthermore, in step S1, the finless porpoise behavior focus category C includes foraging behavior, social behavior, courtship and pairing behavior, migration behavior and risk avoidance behavior.

[0016] Further, in step S1, the finless porpoise surface video shooting equipment includes a drone with a zoom shooting device, a shore-based zoom shooting device and a buoy with a zoom shooting device.

[0017] Furthermore, in step S1 and step S5, the finless porpoise audio is cut according to a time length of 1 s after passing through the finless porpoise hydroacoustic collection device.

[0018] Further, in step S5, the finless porpoise audio fusion recognition model includes a fusion matrix L k The efficient feature extraction module constructed as input first uses F1 3×3 convolution kernels with padding and stride of 1 to perform convolution operation, and applies ReLU activation function to the convolution layer output to introduce nonlinearity, outputting L k(1) , At the same time, L k Through F1 1×1 convolution kernels, we get L k′ , Then output Based on L′ k(1) Calculate attention weights and generate weight matrix The weight at each position is normalized by the softmax function, and the adjusted feature matrix is ​​finally output Where ⊙ represents element-by-element multiplication, and L′ is adjusted using the weight matrix A. k(1) Channel focusing;

[0019] Furthermore, in step S5, the finless porpoise audio fusion recognition model further includes a feature matrix L k(2) The deep feature extraction module constructed as input uses F2 3×3 convolution kernels with padding and stride of 1, performs convolution operation and applies ReLU activation function to the convolution layer output to introduce nonlinearity, outputting L k(3) , At the same time, L k(2) Through F2 1×1 convolution kernels, we get L k(2)′ , Then output the feature matrix

[0020] Furthermore, in step S5, the finless porpoise audio fusion recognition model further includes a feature matrix L′ k(3) The classification processing module constructed as input is L′ k(3)Each channel of is averaged and the output vector v is Then use the fully connected layer to convert v into a classification result and output c. Where C is the number of finless porpoise behavior categories.

[0021] Furthermore, in step S5, when the vector e is fused with the matrix L, each element e in the vector e i Add to all elements of matrix L to form 5 new matrices, k = 5, thus forming the fusion matrix L k , Where k is the number of channels, I M×N is the identity matrix of size M×N.

[0022] Due to the adoption of the above technology, the present invention has the following specific positive and beneficial effects compared with the prior art:

[0023] The present invention uses the Yangtze River finless porpoise living environment data, finless porpoise audio and corresponding observed finless porpoise behavior in the Yangtze River finless porpoise protection area to construct a data set, and then according to the finless porpoise's own audio characteristics, intercepts the 30KHz-150KHz frequency band audio to construct a normalized spectrogram, and at the same time normalizes the corresponding environmental metadata to construct a finless porpoise audio basic data set containing a normalized spectrogram, normalized environmental data and corresponding finless porpoise behavior focus category. According to the finless porpoise audio basic data set, by fusing the normalized spectrogram and the normalized environmental data, a multi-channel fusion feature matrix is ​​formed as input, and a convolutional neural network model is sequentially constructed. The efficient feature extraction module, the deep feature extraction module, and the classification processing module, by defining the loss function and back propagation, train a finless porpoise audio fusion recognition model specifically for the fusion recognition of finless porpoise audio and environmental data. After the finless porpoise audio fusion recognition model is deployed in the reserve, it can more accurately identify the behavior of finless porpoises than the previous method of relying solely on finless porpoise audio for recognition. In this way, by obtaining the relatively easy-to-obtain finless porpoise audio data and combining it with the environment, the behavior of finless porpoises in the reserve within a time window (for example, within one hour) can be judged, which is helpful for dynamically monitoring the finless porpoise community in the reserve, so that the finless porpoises can be better protected. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the collection and processing of the finless porpoise audio data of the present invention;

[0025] Figure 2 is a single channel example image converted from the normalized spectrogram matrix obtained in step S2 of the present invention;

[0026] Figure 3It is the training loss curve (Training Loss Curve) (left), training accuracy curve (Training Accuracy Curve) and validation accuracy curve (Validation Accuracy Curve) (right) of the finless porpoise audio fusion recognition model of the present invention when epoch=100. DETAILED DESCRIPTION

[0027] Embodiment 1:

[0028] like Figure 1 As shown, the present invention provides a method for dynamic monitoring of finless porpoise behavior for finless porpoise protection, characterized in that it comprises the following steps:

[0029] S1. In the finless porpoise protection water area, arrange finless porpoise hydroacoustic collection equipment (since the finless porpoise audio data to be processed is between 30KHz and 150KHz, the sampling frequency of the finless porpoise hydroacoustic collection equipment is set to above 300KHz for accurate sampling), finless porpoise surface video shooting equipment and water environment monitoring equipment to collect the audio data emitted by the finless porpoise with five environmental metadata, namely, water temperature, air temperature, dissolved oxygen content, average water level measurement value of the protected area water area, and catchable amount of fish and shrimp in the water body at the time of collection. According to the finless porpoise behavior images captured by the finless porpoise surface image shooting equipment during the finless porpoise audio collection, complete data labeling with finless porpoise behavior attention category C, and crop the data according to the time length of 1s during labeling to complete the cropping of the finless porpoise audio, and then repeat the above collection and labeling process to construct the initial data set of finless porpoise audio and corresponding environmental metadata;

[0030] S2. Use a bandpass filter to capture the high-frequency band in the finless porpoise audio signal. The transfer function of the bandpass filter is H bp (f) is: H bp (f) = H hp (f) H lp (f)

[0031] Where f is the input frequency, H hp (f) is the transfer function of the high-pass filter and is given by: where f lp is the low cutoff frequency, and its value is 30KHz, n is the order of the filter, and H lp (f) is the transfer function of the low-pass filter and is given by: where f hp is the high cut-off frequency, and its value is 150KHz, n is the order of the filter,

[0032] Perform short-time Fourier transform (STFT) on the filtered audio signal, the formula is:

[0033]

[0034] Where X(k,t) is the output matrix of STFT, which represents the signal component at time t and frequency index k. Each element in this matrix represents the spectrum information of the audio signal at a given time window t and frequency k. x(n+t) represents the time offset of the signal, which is the amplitude value of the audio signal at the nth sampling point in the time window t. In this way, the window of each transformation can slide over the entire signal length, so that the signal can be analyzed locally and step by step.

[0035] w(n) is a window function, which is applied to the data points of each window. Its function is to reduce data leakage at both ends of the window and improve the locality and smoothness of the spectrum.

[0036] e -i2πkn / N It is the core part of Fourier transform. The exponential part here represents the rotation factor in complex form, where k is the frequency index, N is the length of the window or the number of FFT points, and n is the time index in the current window.

[0037] After short-time Fourier transform (STFT), a spectrogram X(k, t) of the finless porpoise audio signal is obtained, and normalized processing is performed to obtain a normalized spectrogram L. Where X max =max(X(k,t)),X min =min(X(k,t)), L is the normalized spectrogram, and the elements in the matrix L are between 0 and 1;

[0038] S3, normalize the five environmental metadata, namely water temperature, air temperature, dissolved oxygen content, average water level measurement value of the protected area, and the catchable amount of fish and shrimp in the water body, and obtain the corresponding normalized values ​​D norn (i) The values ​​of i range from 1 to 5, corresponding to water temperature, air temperature, dissolved oxygen content, average water level measurement value of the protected area, and the catchable amount of fish and shrimp in the water body. min(i) is the minimum value corresponding to the environmental data category, D max(i) The maximum value of the corresponding environmental data category, according to the lowest and highest temperatures in the Yangtze River waters over the years, the D min(1) Value -20℃, D max(1) The value is 50℃, and the D min(2) Value -20℃, D max(2) The value is 50℃, and according to the maximum and minimum values ​​of dissolved oxygen in general water, the D min(3) The value is 0mg / L, D max(3)The value is 10 mg / L. The average water level of the protected area is measured by taking the average of the measured values ​​of multiple measuring points in the protected area, and taking the highest water level in history as the maximum value, and taking the lowest water level in history as the minimum value. The catchable amount of fish and shrimp in the water body is determined by the number of fish and shrimp fishing points in the protected area after regular (weekly or monthly) fixed-point fishing, and the estimated amount is determined according to the number of fishing samples. The maximum value in history is taken as the maximum value, and the minimum value in history is taken as the minimum value.

[0039] S4, repeating steps S2 and S3, after all the finless porpoise audios of the initial data set are processed, a normalized data set is obtained, which ensures that the data are distributed in the same range, avoids the deviation between different input scales during the model training process, and constructs a basic data set of finless porpoise audios;

[0040] S5, feature fusion stage, the normalized spectrogram of the finless porpoise audio is a matrix L of size M×N, The corresponding normalized environmental data forms a one-dimensional vector e, Fuse the environment data vector e with the matrix L to obtain the fusion matrix L of k channels k ,

[0041] S6, with the fusion matrix L k As input, an efficient feature extraction module of the finless porpoise audio fusion recognition model is constructed. That is, first, a convolution operation is performed using F1 3×3 convolution kernels with padding and stride of 1, and a ReLU activation function is applied to the convolution layer output to introduce nonlinearity, and the output L k(1) , At the same time, L k Through F1 1×1 convolution kernels, we get L k′ , Then output Based on L′ k(1) Calculate attention weights and generate weight matrix The weight at each position is normalized by the softmax function, and the adjusted feature matrix is ​​finally output Where ⊙ represents element-by-element multiplication, and L′ is adjusted using the weight matrix A. k(1) Channel focusing;

[0042] S7, with the feature matrix L k(2) As input, the deep feature extraction module of the finless porpoise audio fusion recognition model is constructed, that is, using F2 3×3 convolution kernels with padding and stride of 1, performing convolution operations and applying the ReLU activation function to the convolution layer output to introduce nonlinearity, and outputting L k(3) , At the same time, Lk(2) Through F2 1×1 convolution kernels, we get L k(2)′ , Then output the feature matrix

[0043] S8, with the feature matrix L' k(3) As input, the classification processing module of the finless porpoise audio fusion recognition model is constructed, that is, L′ k(3) Each channel is averaged and the output vector Then use the fully connected layer to convert v into a classification result and output c. Where C is the number of finless porpoise behavior categories;

[0044] S9, through steps S5 to S8, by fusing the normalized spectrogram and the normalized environmental data, a multi-channel fusion feature matrix is ​​formed as input, and an efficient feature extraction module, a deep feature extraction module, and a classification processing module of a convolutional neural network model are successively constructed. The optimizer adopts the Adam algorithm, and the loss function adopts the cross entropy loss (CrossEntropyLoss) function. After multiple rounds of training, a finless porpoise audio fusion recognition model is obtained, and then deployed for inference use. By collecting finless porpoise audio and five environmental metadata such as water temperature, air temperature, dissolved oxygen content, average water level measurement value of the protected area water area, and catchable amount of fish and shrimp in the water body at the corresponding time, the audio is cropped in the same way as the training data, and the time length is cropped according to 1s. The collected data is processed through steps S2 and S3, and feature fusion is performed in step S5. The fusion matrix L k After the finless porpoise audio fusion recognition model is inferred, the finless porpoise behavior category recognition is obtained, which is used for finless porpoise monitoring and protection;

[0045] S10. Collect geographical data in the finless porpoise reserve through multi-beam sonar equipment, use 3D modeling tools based on 3D visualization modeling technology to build a visualized water terrain digital twin model based on the finless porpoise reserve, use passive sonar equipment, and use the time difference of orientation (TDOA) algorithm to determine the position of the finless porpoise in real time based on the sound waves of the finless porpoise, and display it in real time in the visualized water terrain digital twin model;

[0046] S11, convert the sound waves of the finless porpoise into the finless porpoise audio, deploy the finless porpoise audio fusion recognition model for inference, and generate the feature fusion matrix L by collecting the finless porpoise audio and the corresponding water temperature, air temperature, dissolved oxygen, total phosphorus concentration in water quality, and ammonia nitrogen concentration in water quality in real time. k , the fusion matrix L kAfter the finless porpoise audio fusion recognition model is inferred, the finless porpoise behavior focus category recognition is obtained. The data input into the finless porpoise audio fusion recognition model is automatically marked with the location obtained by the time difference location (TDOA) algorithm, so that the corresponding finless porpoise, location, and behavior focus category data can be displayed in real time on the visualized water terrain digital twin model;

[0047] S12. The comprehensive warning signal strength P is set according to the expert experience database. By judging the behavior category of the finless porpoise and its position in the visualized water terrain digital twin model, if the comprehensive warning signal strength P exceeds the predetermined threshold, a monitoring warning prompt is issued and displayed in real time on the visualized water terrain digital twin model.

[0048] In step S12, the comprehensive warning signal strength P, P = αB + βG, where B represents the score of the finless porpoise's behavioral characteristics, and the larger the value, the more abnormal the behavior; G represents the terrain feature score, and the larger the value, the greater the impact of the terrain on the finless porpoise's behavior; α and β represent the weight coefficients of the behavioral characteristics and terrain characteristics, respectively, which are set by expert experience and are used to adjust the impact of different characteristics on the comprehensive warning signal;

[0049] Specifically, Among them B fi represents the score of the foraging behavior detected for the i-th time within an hour, B si represents the score of the social behavior detected for the i-th time within an hour, B ci represents the score of the courtship pairing behavior detected for the i-th time within an hour, B mi represents the migration behavior score detected for the i-th time within an hour, B ai represents the risk aversion score detected for the i-th time within an hour, w fi 、w si 、w ci 、w mi 、w ai They represent the corresponding weight coefficients, which are set by the experience of the expert database;

[0050] The behavior scoring rules are set according to the experience of the expert database as follows: fi , normal foraging value is 0.1, frequent foraging value is 0.5, abnormal foraging value is 0.9, for B si , the value of normal social interaction is 0.1, the value of frequent social interaction is 0.5, and the value of abnormal social interaction is 0.9. ci , the normal courtship value is 0.1, the frequent courtship value is 0.5, and the abnormal courtship value is 0.9. mi , the normal migration value is 0.1, the frequent migration value is 0.5, and the abnormal migration value is 0.9. ai , normal hedging takes a value of 0.1, frequent hedging takes a value of 0.5, and emergency hedging takes a value of 0.9;

[0051] Specifically, G=w t G t +w d G d +w e G e , where G t Indicates the score of shoals and mudflats, G d Denotes the deep water area score, G e represents the estuary area score, w t 、w d 、w e Represents the corresponding weight coefficient, which is set by the experience of the expert database;

[0052] Among them, for G t , the value of normal activity area is 0.1, the value of higher activity area is 0.5, and the value of high risk area is 0.9. d , the value of normal activity area is 0.1, the value of higher activity area is 0.5, and the value of high risk area is 0.9. e , the value for normal activity area is 0.1, the value for higher activity area is 0.5, and the value for high risk area is 0.9;

[0053] For example, the following behaviors were detected within an hour:

[0054] B f1 =0.6, B s1 =0.2, B c1 =0.7, B m1 =0.5, B a1 =0.9

[0055] B f2 =0.5, B s2 =0.3, B c2 =0.6, B m2 =0.4, B a2 =0.8,

[0056] B f3 =0.7, B s3 =0.5, B c3 =0.1, B m3 =0.6, B a3 =0.7

[0057] Terrain feature score: G t =0.4, G d =0.1, G e =0.6,

[0058] Behavior feature weight: w f =0.2, w s =0.1, w c=0.3, w m =0.2, w a =0.2, terrain feature weight: w t =0.4, w d =0.3, w e =0.3, then P = 0.5 × 0.6467 + 0.3 × 0.37 = 0.43445,

[0059] According to the comprehensive warning information value P, different warning levels can be set, for example: P < 0.3, corresponding to low-level warning (green), 0.3 ≤ P < 0.6, corresponding to medium-level warning (yellow), 0.6 ≤ P < 0.8, corresponding to high-level warning (orange), P ≥ 0.8, corresponding to emergency warning (red),

[0060] Then P = 0.43445. At this time, the visualized water terrain digital twin model can show that it is in an intermediate warning (yellow) state. It can dynamically monitor the situation of the finless porpoise community from the overall perspective of the entire finless porpoise reserve, making it convenient to respond to emergencies in a timely manner.

[0061] In step S1, the finless porpoise behavior focus category C is set. The main purpose is to let the finless porpoise audio fusion recognition model focus on the behaviors of the finless porpoise that are convenient for protective actions. The finless porpoise behavior focus category C includes foraging behavior, social behavior, courtship and pairing behavior, migration behavior and risk avoidance behavior. Among them, foraging behavior refers to the behavior of finding and catching food, social behavior refers to the interaction with other finless porpoises, such as playing and touching, etc., courtship and pairing behavior refers to the reproductive behavior of finless porpoises, migration behavior refers to the seasonal migration of finless porpoises according to changes in food availability and water conditions, and risk avoidance behavior refers to the evasive behavior displayed by finless porpoises in the face of threats from natural enemies or human activities.

[0062] In order to more accurately identify the types of behaviors reflected in the finless porpoise audio, the categories of finless porpoise behaviors that need attention can be further subdivided.

[0063] In step S1, the finless porpoise surface video shooting equipment includes a drone with a zoom shooting device, a shore-based zoom shooting device and a buoy with a zoom shooting device.

[0064] In step S5, when the vector e is fused with the matrix L, each element e in the vector e i Add to all elements of matrix L to form 5 new matrices, k = 5, thus forming the fusion matrix L k , Where k is the number of channels, I M×N is the identity matrix of size M×N.

[0065] Embodiment 2:

[0066] On the basis of the first embodiment, in step S5, when the vector e is merged with the matrix L, each element e in the vector e is no longer used. i is added to all elements of the matrix L, but to each element e in the vector e i Fill them separately to form a matrix M of size M×N i , M i =e i I M×N , thus generating 5 new matrices, plus matrix L, forming a fusion matrix L with channel number k = 6 k , Generate specific channels. During convolution, each element of the matrix L is integrated with all the elements of the vector e, which enriches the input of the model and improves the model's ability to understand the data.

[0067] Embodiment three:

[0068] On the basis of the first and second embodiments, in order to further enrich the input of the model, in step 5, when the vector e is fused with the matrix L, each element e in the vector e is used. i Add to all elements of the matrix L, and use each element e in the vector e i Fill them separately to form a matrix M of size M×N i , thus forming a fusion matrix L with channel number k = 11 k , It enriches the input of the model and improves the model's ability to understand the data.

Claims

1. A dynamic monitoring method for finless porpoise behavior for finless porpoise protection, Features: The steps include: S1. Deploy finless porpoise hydroacoustic collection equipment, finless porpoise surface video shooting equipment and water environment monitoring equipment in the finless porpoise protection waters to collect the audio data emitted by the finless porpoises and collect the five environmental metadata of water temperature, air temperature, dissolved oxygen content, average water level measurement value of the protected area waters, and the catchable amount of fish and shrimp in the water body. According to the finless porpoise behavior images captured by the finless porpoise surface video shooting equipment during the finless porpoise audio collection, complete the data labeling with the finless porpoise behavior focus category C, and complete the cropping of the finless porpoise audio during labeling. Then repeat the above collection and labeling process to construct the initial data set of finless porpoise audio and corresponding environmental metadata; S2, using a bandpass filter to intercept the high-frequency band in the finless porpoise audio signal, and performing short-time Fourier transform (STFT) on the filtered audio signal to obtain a spectrogram X(k, t) of the finless porpoise audio signal, and performing normalization processing to obtain a normalized spectrogram L, Where X max =max(X(k, t)),X min =min(X(k, t)), L is the normalized spectrogram, and the elements in the matrix L are all between 0 and 1; S3, complete the normalization processing of the five environmental metadata, namely water temperature, air temperature, dissolved oxygen content, average water level measurement value of the protected area, and the catchable amount of fish and shrimp in the water body, and obtain the corresponding normalized values ​​D norn (i) The values ​​of i range from 1 to 5, corresponding to water temperature, air temperature, dissolved oxygen content, average water level measurement value of the protected area, and the catchable amount of fish and shrimp in the water body. min(i) is the minimum value corresponding to the environmental data category, D max(i) The maximum value of the corresponding environmental data category; S4, repeating steps S2 and S3, after all the finless porpoise audios of the initial data set are processed, a normalized data set is obtained, which ensures that the data are distributed in the same range, avoids the deviation between different input scales during the model training process, and constructs a basic data set of finless porpoise audios; S5, convert the normalized spectrogram of the finless porpoise audio into a matrix L of size M×N, The corresponding normalized environmental data forms a one-dimensional vector e, Fuse the environment data vector e with the matrix L to obtain the fusion matrix L of k channels k , Fusion matrix L k The input is fed into a convolutional neural network model with self-attention and residual network. After multiple rounds of training, a finless porpoise audio fusion recognition model is obtained. S6. Collect geographical data in the finless porpoise reserve through multi-beam sonar equipment, use 3D modeling tools based on 3D visualization modeling technology to build a visualized water terrain digital twin model based on the finless porpoise reserve, use passive sonar equipment, and use the time difference of orientation (TDOA) algorithm to determine the location of the finless porpoise in real time based on the sound waves of the finless porpoise, and display it in real time in the visualized water terrain digital twin model; S7, convert the sound waves of the finless porpoise into the finless porpoise audio, deploy the finless porpoise audio fusion recognition model for inference, and generate the feature fusion matrix L by collecting the finless porpoise audio and the corresponding water temperature, air temperature, dissolved oxygen, total phosphorus concentration in water quality, and ammonia nitrogen concentration in water quality in real time. k , the fusion matrix L k After inference of the finless porpoise audio fusion recognition model, the finless porpoise behavior focus category is identified and displayed in real time on the visualized water terrain digital twin model.

2. A method for dynamic monitoring of finless porpoise behavior for finless porpoise protection according to claim 1, Features: It also includes step S8, setting the comprehensive warning signal strength P according to the expert experience database, and by judging the behavior category of the finless porpoise and its position in the visualized water terrain digital twin model. If the comprehensive warning signal strength P exceeds a predetermined threshold, a monitoring and warning prompt will be issued and displayed in real time on the visualized water terrain digital twin model.

3. A method for dynamic monitoring of finless porpoise behavior for finless porpoise protection according to claim 2, Features: In step S8, the comprehensive warning signal strength P, P = αB + βG, where B represents the score of the finless porpoise's behavioral characteristics, and the larger the value, the more abnormal the behavior; G represents the terrain feature score, and the larger the value, the greater the impact of the terrain on the finless porpoise's behavior; α and β represent the weight coefficients of behavioral characteristics and terrain characteristics, respectively, which are set by expert experience and are used to adjust the impact of different characteristics on the comprehensive warning signal.

4. The method for dynamic monitoring of finless porpoise behavior for finless porpoise protection according to claim 1, Features: In step S1, the finless porpoise behavior focus category C includes foraging behavior, social behavior, courtship and pairing behavior, migration behavior and risk avoidance behavior.

5. The method for dynamic monitoring of finless porpoise behavior for finless porpoise protection according to claim 1, Features: In step S1, the finless porpoise surface video shooting equipment includes a drone with a zoom shooting device, a shore-based zoom shooting device and a buoy with a zoom shooting device.

6. A method for dynamic monitoring of finless porpoise behavior for finless porpoise protection according to claim 1, Features: In step S1 and step S5, the finless porpoise audio is cut according to the time length of 1 s after passing through the finless porpoise hydroacoustic collection device.

7. The method for dynamic monitoring of finless porpoise behavior for finless porpoise protection according to claim 1, Features: In step S5, the finless porpoise audio fusion recognition model includes a fusion matrix L k As input, an efficient feature extraction module is constructed, that is, first using F 1 A 3×3 convolution kernel with padding and stride of 1 is used for convolution operation, and a ReLU activation function is applied to the convolution layer output to introduce nonlinearity, outputting L k(1) , At the same time, L k By F 1 1×1 convolution kernels are used for convolution operation to obtain L k′ , Then output L′ k(1) =L k(1) +L k′ , Based on L′ k(1) Calculate attention weights and generate weight matrix The weight at each position is normalized by the softmax function, and the adjusted feature matrix L is finally output k(2) =L′ k(1) ⊙A, Where ⊙ represents element-by-element multiplication, and L′ is adjusted using the weight matrix A. k(1) Channel focus.

8. A method for dynamic monitoring of finless porpoise behavior for finless porpoise protection according to claim 7, Features: In step S5, the finless porpoise audio fusion recognition model also includes a feature matrix L k(2) The deep feature extraction module constructed as input, that is, using F 2 A 3×3 convolution kernel with padding and stride of 1 is used to perform convolution and apply the ReLU activation function to the convolution layer output to introduce nonlinearity, outputting L k(3) , At the same time, L k(2) By F 2 1×1 convolution kernels are used for convolution operation to obtain L k(2)′ , Then output the feature matrix L′ k(3) =L k(3) +L k(2)′ , 9. A method for dynamic monitoring of finless porpoise behavior for finless porpoise protection according to claim 8, Features: In step S5, the finless porpoise audio fusion recognition model also includes a feature matrix L′ k(3) The classification processing module constructed as input is L′ k(3) Each channel of is averaged and the output vector v is Then use the fully connected layer to convert v into a classification result and output c. Where C is the number of finless porpoise behavior categories.

10. A method for dynamic monitoring of finless porpoise behavior for finless porpoise protection according to claim 9, Features: In step S5, when the vector e is fused with the matrix L, each element e in the vector e i Add to all elements of matrix L to form 5 new matrices, k = 5, thus forming the fusion matrix L k , L k =L+e k I M×N , Where k is the number of channels, I M×N is the identity matrix of size M×N.

Images