Method and system for monitoring cowfish through unmanned aerial vehicle cruising and real-time video
Through drone monitoring and real-time detection of deep learning models, the problem of low accuracy in identification of finless porpoises is solved, and high accuracy and stable finless porpoise monitoring effect is achieved.
Patent Information
- Application Number
- CN202411499749.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-02-25
AI Technical Summary
In the existing videos of drone monitoring the porpoise, the porpoise target is small and prone to interference, resulting in misjudgment and missed judgments in human eye identification and low accuracy.
UAVs are used for cruise monitoring, video images are taken and real-time detection is carried out through deep learning models. The specific steps include: labeling, preprocessing, training deep learning models, and using the trained models for real-time detection and statistics.
It improves the accuracy and stability of the identification of finless porpoises, reduces the occurrence of misjudgments and misjudgments, and realizes accurate statistics on the occurrence time of finless porpoises and the total time when they are exposed to the water.
Smart Images
Figure CN119516411A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of biological monitoring, and in particular to a method and system for monitoring finless porpoises using real-time video during unmanned aerial vehicle cruise. Background Art
[0002] The Yangtze River porpoise is a rare and protected animal, and monitoring of the porpoise is very important and meaningful. Using drones to monitor the Yangtze River porpoise is a non-destructive monitoring method that is worth promoting. However, the porpoise targets in the video are small and there are interferences when using drones for monitoring. Long-term human eye recognition is prone to eye fatigue, resulting in missed judgments and wrong judgments during the recognition process. Therefore, it is very necessary to develop a stable, reliable system with a high recognition rate. Summary of the invention
[0003] In view of the above-mentioned defects of the prior art, the present invention provides a method and system for real-time video monitoring of finless porpoises by drone cruise, so as to solve the problem of low accuracy of misjudgment and missed judgment of human eye recognition.
[0004] To achieve the above objectives, the present invention provides a method for monitoring finless porpoises by real-time video of a drone cruise, which comprises:
[0005] S1: The drone conducts patrol monitoring over the target waters, captures video images, and selects frames containing finless porpoises in the video images;
[0006] S2: labeling the picture frames including the finless porpoise to obtain a finless porpoise image dataset; preprocessing the images in the finless porpoise image dataset;
[0007] S3: Using the finless porpoise image dataset to train a deep learning model; the deep learning model includes a feature extraction network Backbone, a neck network Neck, and a prediction layer Prediction;
[0008] S4: Using the deep learning model constructed through training, real-time porpoise detection is performed on the real-time transmitted drone monitoring videos, porpoises are annotated in the detected videos, and the number of porpoises is counted.
[0009] S5: Using the time when the finless porpoise video frame image appears in the video, statistics are performed to obtain the time when the finless porpoise appears and the total time when the finless porpoise is out of water.
[0010] A further improvement of the present invention is that, in step S2, LabelImg labeling software is used to label the picture frame including the finless porpoise, and a feature information frame of the image is obtained after labeling and screening; the preprocessing step includes image enhancement and image cropping.
[0011] A further improvement of the present invention is that the feature extraction network Backbone includes an input layer Input, a Focus layer, a basic convolution block CBL, a CSP1-2 network, a basic convolution block CBL, a first CSP1-6 network, a basic convolution block CBL, a second CSP1-6 network, a basic convolution block CBL, a spatial pyramid pooling module SPP, and a CSP2-2 network connected in sequence; the output of the first CSP1-6 network is also used as an input of the first residual block Res, and the output of the second CSP1-6 network is also used as an input of the second residual block Res.
[0012] A further improvement of the present invention is that the neck network Neck includes a first basic convolution block CBL, a first upsampling layer, a first concatenation layer Concat, a CSP2-2 network, a second basic convolution block CBL, a second upsampling layer, a second concatenation layer Concat, a CSP2-2 network, a third basic convolution block CBL, a third concatenation layer Concat, a CSP2-2 network, a fourth basic convolution block CBL, a fourth concatenation layer Concat, and a CSP2-2 network, which are connected in sequence;
[0013] In the neck network Neck, the first basic convolution block CBL receives the output of the CSP2-2 network in the feature extraction network Backbone; the first concatenation layer Concat is used to concatenate the output of the first upsampling layer and the output of the second residual block Res; the second concatenation layer Concat is used to concatenate the output of the second upsampling layer and the output of the first residual block Res; the third concatenation layer Concat is used to concatenate the output of the second basic convolution block CBL and the output of the third basic convolution block CBL; the fourth concatenation layer Concat is used to concatenate the output of the first basic convolution block CBL and the output of the fourth basic convolution block CBL.
[0014] A further improvement of the present invention is that the prediction layer Prediction generates a prediction result according to the outputs of the three CSP2-2 networks, and when a finless porpoise is identified, a rectangular frame is used to mark the finless porpoise.
[0015] The present invention also provides a system for monitoring finless porpoises by using a drone for real-time video during cruise, which is used to implement the method for monitoring finless porpoises by using a drone for real-time video during cruise; the system comprises:
[0016] Unmanned aerial vehicles (UAVs) are used to conduct patrol monitoring over target waters and take video images;
[0017] The video recognition server is connected to the drone for communication and is used to receive the images sent back by the drone in real time, identify the finless porpoise in the image, and calculate the time and location of the finless porpoise's appearance.
[0018] A further improvement of the present invention is that the drone and the video recognition server communicate in real time via an HDMI 4G DVR device.
[0019] The advantages of the present invention are as follows:
[0020] (1) The present invention solves the problem of low accuracy in identifying finless porpoises, and has the characteristics of high robustness, strong generalization ability, and high accuracy; at the same time, it achieves statistics on the appearance time of finless porpoises, which is convenient for tracking and investigating the living habits of finless porpoises.
[0021] (2) The present invention designs a novel recognition model for the recognition of surface finless porpoises, and its network structure has a better recognition effect for finless porpoises in aerial surface images. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 Flowchart of the method for real-time video monitoring of finless porpoises for drone cruises;
[0023] Figure 2 The network structure of the recognition model adopted by the present invention;
[0024] Figure 3 It is the network structure of the basic convolution block CBL;
[0025] Figure 4 It is the network structure of Res residual block;
[0026] Figure 5 Schematic diagram of slicing operation;
[0027] Figure 6 It is the specific structure of the Focus layer;
[0028] Figure 7 This is a schematic diagram of the CSP1-X module;
[0029] Figure 8 This is a schematic diagram of the CSP2-X module;
[0030] Fig. 9 Schematic diagram of the spatial pyramid pooling module SPP. DETAILED DESCRIPTION
[0031] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0032] While some exemplary embodiments of the present invention have been described for the purpose of illustration, it should be understood that the present invention may be implemented in other ways not specifically shown in the drawings.
[0033] like Figure 1 As shown, this embodiment provides a method for monitoring finless porpoises by real-time video of a drone cruise, which includes the following steps:
[0034] The method for monitoring finless porpoises by using real-time video of drone cruise is characterized by comprising:
[0035] S1: The drone conducts patrol monitoring over the target waters (the Yangtze River in this embodiment). The video image is composed of each frame of pictures. The picture frames containing the finless porpoise are selected from them, and the video frame images containing the finless porpoise are taken as the data set P, where P = {P1, P2, P2..., P N}, P i For the i-th image, i∈{1,2,...,N}; complete the real-time transmission and reception of the video; the video is stored in the server computer or cloud database.
[0036] In addition, in this step, during the drone cruise monitoring process, the captured video images are transmitted back in real time using the HDMI 4GDVR device, and are received and stored in the server and cloud in real time.
[0037] S2: Label the picture frames including the finless porpoise to obtain a finless porpoise image dataset; and preprocess the images in the finless porpoise image dataset.
[0038] Specifically, in this step, the finless porpoise image dataset P is imported into the LabelImg annotation software, and the feature information boxes of the filtered images are annotated to obtain the finless porpoise image dataset PL, where PL = {PL1, PL2, PL2..., PL N}, PL i is the i-th image, i∈{1,2,...,N}. The i-th filtered image PL in the finless porpoise image dataset PL i Perform preprocessing, which mainly includes image enhancement and image cropping, to obtain the preprocessed image
[0039] S3: Using the finless porpoise image dataset to train a deep learning model; the deep learning model includes a feature extraction network Backbone, a neck network Neck, and a prediction layer Prediction;
[0040] S4: Using the deep learning model constructed through training, real-time porpoise detection is performed on the real-time transmitted drone monitoring videos, porpoises are annotated in the detected videos, and the number of porpoises is counted.
[0041] S5: Using the time when the finless porpoise video frame image appears in the video, statistics are performed to obtain the time when the finless porpoise appears and the total time when the finless porpoise is out of water.
[0042] The deep learning model used in this embodiment is introduced below with reference to the accompanying drawings. Figure 2 As shown in FIG. 1 , the deep learning model used in this embodiment includes a feature extraction network Backbone, a neck network Neck, and a prediction layer Prediction. During the training process, the preprocessed image Input to the input layer of the feature extraction network Backbone; the preprocessed image The processing process is as follows Figure 2 As shown. Specifically:
[0043] The feature extraction network Backbone includes an input layer Input, a Focus layer, a basic convolution block CBL, a CSP1-2 network, a basic convolution block CBL, a first CSP1-6 network, a basic convolution block CBL, a second CSP1-6 network, a basic convolution block CBL, a spatial pyramid pooling module SPP, and a CSP2-2 network connected in sequence. The output of the first CSP1-6 network is also used as the input of the first residual block Res, and the output of the second CSP1-6 network is also used as the input of the second residual block Res.
[0044] The neck network Neck includes a first basic convolution block CBL, a first upsampling layer, a first concatenation layer Concat, a CSP2-2 network, a second basic convolution block CBL, a second upsampling layer, a second concatenation layer Concat, a CSP2-2 network, a third basic convolution block CBL, a third concatenation layer Concat, a CSP2-2 network, a fourth basic convolution block CBL, a fourth concatenation layer Concat, and a CSP2-2 network, which are connected in sequence.
[0045] The input color image structure is 416*416*3; Focus is converted to 208*208*48, after CBL convolution block it is 104*104*96, after CSP1-2 and CBL processing it is 52*52*192, after CSP1-6 and CBL processing it is 26*26*384, and then after CSP1-6 and CBL processing it is 13*13*768.
[0046] In the neck network Neck, the first basic convolution block CBL receives the output of the CSP2-2 network in the feature extraction network Backbone; the first concatenation layer Concat is used to concatenate the output of the first upsampling layer and the second residual block Res; the second concatenation layer Concat is used to concatenate the output of the second upsampling layer and the first residual block Res; the third concatenation layer Concat is used to concatenate the output of the second basic convolution block CBL and the output of the third basic convolution block CBL; the fourth concatenation layer Concat is used to concatenate the output of the first basic convolution block CBL and the output of the fourth basic convolution block CBL.
[0047] Figure 2 It is composed of multiple small network blocks, including the Focus module newly added by YOLOv5, as well as the basic convolution block CBL, the residual block Res (used for the neckCSP network structure, hereinafter referred to as the CSP network structure), the spatial pyramid pooling module SPP, and the CSP network structure inherited from YOLOv4.
[0048] (1) Basic convolutional block CBL: CBL is a convolutional block composed of a convolution layer (Convolution), a normalization layer (BatchNormalization) and an activation layer (Leaky ReLU). Its main function is to extract features. Its structure is as follows: Figure 3 shown.
[0049] (2) Residual block Res: The residual block Res cleverly uses the residual network structure to effectively alleviate the problem of gradient vanishing. Therefore, the Res residual block is also indispensable in YOLOv5. The residual block structure is as follows: Figure 4 As shown. It includes two paths. The first path processes the input by using the basic convolution block CBL and the basic convolution block CBL in sequence; the second path directly uses the input, and the output of the residual block Res is the superposition of the above two paths.
[0050] (3) Focus module: The key of this module is to perform slicing operations on the feature map, such as Figure 5 As shown, a 4×4×3 input image will output a 2×2×12 feature map after 4 slicing operations.
[0051] The Focus structure mainly transforms the input original image (size of 416*416*3) into a feature map of 208*208*12 through 4 slices, and then undergoes a convolution operation with 48 convolution kernels to finally transform it into a feature map of 208*208*48. Its function is to integrate the width and height dimensional information of the original image into the channel dimension, thus increasing the number of channels of the input image. Figure 6 As shown:
[0052] (4) CSP-i network structure: It reduces the number of model parameters from the perspective of network construction, which helps to improve prediction speed while ensuring training accuracy. Figure 7 and Figure 8 shown.
[0053] (5) Spatial Pyramid Pooling Module SPP: This module mainly extracts multi-scale features by performing 1×1, 5×5, 9×9 and 13×13 maximum pooling (Maxpool) on the input image, and finally concatenates and aggregates the feature maps through the Concat operation to improve the model accuracy. The specific structure is as follows Fig. 9 shown.
[0054] The present invention also provides a system for monitoring finless porpoises by using a drone for real-time video during cruise, which is used to implement the method for monitoring finless porpoises by using a drone for real-time video during cruise; the system comprises:
[0055] Unmanned aerial vehicles (UAVs) are used to conduct patrol monitoring over target waters and take video images;
[0056] The video recognition server is connected to the drone for communication and is used to receive the images sent back by the drone in real time, identify the finless porpoise in the image, and calculate the time and location of the finless porpoise's appearance.
[0057] A further improvement of the present invention is that the drone and the video recognition server communicate in real time via an HDMI 4G DVR device.
[0058] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.
Claims
1. A method for monitoring finless porpoises by using real-time video of drone cruise, characterized in that include: S1: The drone conducts patrol monitoring over the target waters, captures video images, and selects frames containing finless porpoises in the video images; S2: labeling the picture frames including the finless porpoise to obtain a finless porpoise image dataset; preprocessing the images in the finless porpoise image dataset; S3: Using the finless porpoise image dataset to train a deep learning model; the deep learning model includes a feature extraction network Backbone, a neck network Neck, and a prediction layer Prediction; S4: Using the deep learning model constructed through training, real-time porpoise detection is performed on the real-time transmission of drone monitoring videos, porpoises are annotated in the detected videos, and the number of porpoises is counted; S5: Using the time when the finless porpoise video frame image appears in the video, statistics are performed to obtain the time when the finless porpoise appears and the total time when the finless porpoise is out of water.
2. The method for monitoring finless porpoises by real-time video of a drone cruise according to claim 1, characterized in that: In step S2, LabelImg labeling software is used to label the picture frame including the finless porpoise, and the feature information frame of the image is obtained after labeling and screening; the preprocessing steps include image enhancement and image cropping.
3. The method for monitoring finless porpoises by real-time video of a drone cruise according to claim 1, characterized in that: The feature extraction network Backbone includes an input layer Input, a Focus layer, a basic convolution block CBL, a CSP1-2 network, a basic convolution block CBL, a first CSP1-6 network, a basic convolution block CBL, a second CSP1-6 network, a basic convolution block CBL, a spatial pyramid pooling module SPP, and a CSP2-2 network connected in sequence; the output of the first CSP1-6 network is also used as the input of the first residual block Res, and the output of the second CSP1-6 network is also used as the input of the second residual block Res.
4. The method of monitoring finless porpoises by real-time video of a drone cruise according to claim 3, characterized in that: The neck network Neck includes a first basic convolution block CBL, a first upsampling layer, a first concatenation layer Concat, a CSP2-2 network, a second basic convolution block CBL, a second upsampling layer, a second concatenation layer Concat, a CSP2-2 network, a third basic convolution block CBL, a third concatenation layer Concat, a CSP2-2 network, a fourth basic convolution block CBL, a fourth concatenation layer Concat, and a CSP2-2 network, which are connected in sequence; In the neck network Neck, the first basic convolution block CBL receives the output of the CSP2-2 network in the feature extraction network Backbone; the first concatenation layer Concat is used to concatenate the output of the first upsampling layer and the output of the second residual block Res; the second concatenation layer Concat is used to concatenate the output of the second upsampling layer and the output of the first residual block Res; the third concatenation layer Concat is used to concatenate the output of the second basic convolution block CBL and the output of the third basic convolution block CBL; the fourth concatenation layer Concat is used to concatenate the output of the first basic convolution block CBL and the output of the fourth basic convolution block CBL.
5. The method of monitoring finless porpoises by real-time video of a drone cruise according to claim 4, characterized in that: The prediction layer generates prediction results based on the outputs of the three CSP2-2 networks. When a finless porpoise is identified, a rectangular frame is used to mark the finless porpoise.
6. A system for real-time video monitoring of finless porpoises by drone cruising, used to implement the method for real-time video monitoring of finless porpoises by drone cruising as claimed in any one of claims 1 to 5; characterized in that include: Unmanned aerial vehicles (UAVs) are used to conduct patrol monitoring over target waters and take video images; The video recognition server is connected to the drone for communication and is used to receive the images sent back by the drone in real time, identify the finless porpoise in the image, and calculate the time and location of the finless porpoise's appearance.
7. The system for monitoring finless porpoises by real-time video of a drone cruise according to claim 6, characterized in that: The drone and the video recognition server communicate in real time via an HDMI 4G DVR device.