A Bird Detection Method Based on Video Stream

Abstract

This invention discloses a bird detection method based on video streams, relating to the field of video processing technology. The method includes: acquiring a video stream to be detected; determining a first confidence level of intermediate frames in the video stream and identifying target frame images from the video stream based on the first confidence level; concatenating the target frame images into an initial feature matrix and performing a three-dimensional convolution aggregation operation on the initial feature matrix to obtain aggregated features as the first feature; performing multiple downsampling operations on the first feature and then performing feature extraction at multiple stages corresponding to the downsampling operations to obtain a second feature; and obtaining a confidence feature map and a position feature map based on the second feature, wherein the input of each stage is the output of the previous stage and the corresponding downsampling result; and determining the bird situation in the video stream to be detected based on the confidence feature and the position feature map. This method can reduce the use of computational resources and improve the detection accuracy of bird targets.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a bird detection method based on video streams. Background Technology

[0002] In recent years, with the gradual development of computer vision technology and deep learning (DL), people have begun to use cameras to detect various targets in real time, and use convolutional neural networks (CNNs) to quickly process frame images and predict the target location and category.

[0003] Traditional convolutional neural networks (CNNs) fall into two categories. One category, represented by early YOLO series algorithms, features single-stage detection. These algorithms are fast, processing frames quickly and directly predicting the location and category of the detected target. However, their accuracy is poor when dealing with small or blurred objects. The other category, represented by fast region-based convolutional networks (Fast R-CNN), features two-stage detection. These algorithms use region proposal networks to generate candidate regions, offering significantly improved accuracy compared to single-stage algorithms. However, they also require significantly more computation, placing higher demands on equipment for real-time detection. Furthermore, both of these algorithms are static image detectors, directly detecting each frame of the video and ignoring the temporal information between frames. This leads to unsatisfactory detection results and poor robustness to motion blur and occlusion.

[0004] Therefore, video stream-based object detection has emerged. Currently, video stream-based object detection is mainly divided into end-to-end methods based on Transformers and motion-aware detection methods. Transformer-based end-to-end methods primarily utilize spatiotemporal attention mechanisms to directly model the global dependencies of video sequences. Motion-aware detection methods include optical flow enhancement methods, trajectory prediction methods, and event camera fusion methods. Optical flow enhancement methods provide temporal information for video object detection by calculating the motion displacement (motion vector field) of pixels between adjacent frames of the video, thereby representing the motion trajectory of the target object. Trajectory prediction methods can jointly model detection and motion trajectories, thus using historical motion information to correct the current detection. Event camera fusion methods mainly utilize the high temporal resolution of event cameras to compensate for motion blur in traditional red-green-blue (RGB) cameras.

[0005] While Transformer-based end-to-end methods can directly establish global dependencies in video sequences, they require significant computational resources. Compared to Transformer methods, motion-aware detection methods have lower computational costs, but optical flow enhancement methods are highly dependent on the accuracy of optical flow estimation and are sensitive to fast-moving or occluded objects. In contrast, trajectory prediction-based methods can address short-term occlusion issues, but require pre-labeled trajectory data. In summary, both existing methods have their advantages and disadvantages, but they share a common drawback: both require a large number of adjacent frames as input to construct dependencies or characterize the motion trajectory of the target object.

[0006] When detecting birds in flight based on video streams, the prominence of bird features varies across single frames. Birds are more prominent when close to the camera, but less so when far away, and may even be obscured or blend into the background. In such cases, existing methods that process a large number of adjacent frames are clearly inadequate, wasting significant computational resources. Summary of the Invention

[0007] This invention proposes a video stream-based bird detection method to reduce the use of computing resources and improve the detection accuracy of birds.

[0008] To achieve the above objectives, this invention proposes a bird detection method based on video streams. The method includes: acquiring a video stream to be detected; determining a first confidence level of intermediate frames in the video stream to be detected, and determining a target frame image from the video stream to be detected based on the first confidence level; concatenating the target frame image into an initial feature matrix, and performing a three-dimensional convolution aggregation operation on the initial feature matrix to obtain an aggregated feature as a first feature; performing multiple downsampling operations on the first feature, and performing feature extraction at multiple stages corresponding to the multiple downsampling operations to obtain a second feature, and obtaining a confidence feature map and a position feature map based on the second feature, wherein the input of each stage is the output of the previous stage and the downsampling result of the corresponding stage; and determining the bird situation in the video stream to be detected based on the confidence feature and the position feature map.

[0009] In some embodiments, determining the first confidence level of the intermediate frame of the video stream to be detected includes: performing two-dimensional convolution processing, ReLU activation processing, and two-dimensional batch normalization processing on the intermediate frame in sequence, repeating the process at least once, and then performing two-dimensional adaptive average pooling processing and Hardtanh activation processing in sequence to obtain the first confidence level.

[0010] In some embodiments, determining the target frame image from the video stream to be detected based on the first confidence level includes: when the first confidence level is greater than a first threshold, using the intermediate frame as the target frame image; when the first confidence level is greater than a second threshold and less than or equal to the first threshold, using the first frame, intermediate frames, and last frame of the video stream to be detected as the target frame image; and when the first confidence level is less than or equal to the second threshold, using all frames of the video stream to be detected as the target frame image.

[0011] In some embodiments, the method further includes: determining that the number of target frame images is multiple before concatenating the target frame images into an initial feature matrix; when the number of target frame images is one, performing convolutional neural network processing on the target frame images to obtain convolutional features as the first feature.

[0012] In some embodiments, performing a three-dimensional convolution aggregation operation on the initial feature matrix to obtain the aggregated features includes: sequentially performing a three-dimensional convolution operation and a three-dimensional pooling operation on the initial feature matrix to obtain a first feature matrix; sequentially performing a three-dimensional convolution operation and a three-dimensional pooling operation on the first feature matrix to obtain a second feature matrix; performing a sigmoid activation operation on the second feature matrix to obtain a third feature matrix; performing a dot product between the second feature matrix and the third feature matrix to obtain a fourth feature matrix; superimposing the fourth feature matrix with the first feature matrix to obtain a fifth feature matrix; and sequentially performing a three-dimensional convolution operation and a three-dimensional pooling operation on the fifth feature matrix to obtain the aggregated features.

[0013] In some embodiments, in each stage, the input from the previous stage is subjected to matrix transformation and then input to the decoder after positional encoding. In the decoder, the result of the positional encoding operation is normalized to obtain a query vector, a key vector, and a value vector. The key vector and the value vector are spatially reduced, and a third feature is obtained based on the query vector and the reduced key vector and value vector using a self-attention mechanism. After passing through a feedforward network, normalization, and residual connections, upsampling is performed, and multiple convolution operations are performed on the upsampling result and the corresponding downsampling result to obtain the second feature.

[0014] In some embodiments, determining the bird situation in the video stream to be detected based on the confidence feature and the location feature map includes: outputting a second confidence level based on the confidence feature map through a first fully connected layer, and outputting a regression result based on the location feature map through a second fully connected layer; and determining the bird situation in the video stream to be detected based on the second confidence level and the regression result.

[0015] In some embodiments, the second confidence score and the regression result are obtained from the video stream to be detected using a multi-frame adaptive network. The expression for the loss function of the multi-frame adaptive network is as follows:

[0016] ;

[0017] Where L represents the loss function, Indicates confidence loss. α represents the position loss, and α represents the balance parameter.

[0018] In some embodiments, the confidence loss is expressed as follows:

[0019] ;

[0020] Where N represents the number of samples, This represents the true label of the i-th sample. This represents the second confidence score for the prediction of the i-th sample.

[0021] In some embodiments, the expression for the position loss is as follows:

[0022] ;

[0023] Where N represents the number of samples, and j is the parameter of the ground truth bounding box. This represents the true bounding box of the i-th sample. This represents the predicted bounding box of the i-th sample. express Smoothing loss function.

[0024] This invention discloses a video stream-based bird detection method. First, it determines the initial confidence level of intermediate frames in the video stream to be detected. Then, it dynamically inputs different numbers of target frame images based on the initial confidence level, thereby reducing computational resource usage. Next, the target frame images are concatenated into an initial feature matrix, and a three-dimensional convolution aggregation operation is performed on the initial feature matrix to make the bird target's features more prominent in a single frame. Subsequently, multiple stages of feature extraction and fusion are performed on the aggregated features, integrating multi-scale and multi-level feature information, thereby improving the detection accuracy of the bird target.

[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] Figure 1 This is a flowchart of a bird detection method based on video stream according to an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of the structure of a dynamic frame aggregation module according to an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of a three-dimensional convolutional attention pooling network according to an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of the cross-feature pyramid transform decoder module according to an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of the structure of a multi-frame adaptive network according to an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of the detection accuracy curve of a bird target in a video, which is an example of the present invention. Detailed Implementation

[0032] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0033] A bird detection method based on video stream according to an embodiment of the present invention is described below with reference to the accompanying drawings.

[0034] Figure 1 This is a flowchart of a bird detection method based on video stream according to an embodiment of the present invention.

[0035] like Figure 1 As shown, a bird detection method based on video streams includes:

[0036] S11, Obtain the video stream to be detected.

[0037] The video stream to be detected can be acquired through a camera, such as a camera on a drone. The video stream to be detected can contain multiple consecutive video frames, such as 5, 6, or 7, which can be set as needed.

[0038] S12, determine the first confidence level of the intermediate frame of the video stream to be detected, and determine the target frame image from the video stream to be detected based on the first confidence level.

[0039] In one implementation, the video stream to be detected includes an odd number of video frames, with the middle frame being a frame located in the middle of the video stream to be detected.

[0040] In this implementation, it is only necessary to determine a first confidence level and determine the target frame image from the video stream to be detected based on the first confidence level.

[0041] For example, determining the target frame image from the video stream to be detected based on a first confidence level includes: when the first confidence level is greater than a first threshold, using the intermediate frame as the target frame image; when the first confidence level is greater than a second threshold and less than or equal to the first threshold, using the first frame, intermediate frame, and last frame of the video stream to be detected as the target frame image; and when the first confidence level is less than or equal to the second threshold, using all frames of the video stream to be detected as the target frame image.

[0042] As another implementation, the video stream to be detected includes an even number of video frames, with the middle frame being two frames located in the middle of the video stream to be detected.

[0043] In this implementation, two first confidence levels need to be determined, and the target frame image needs to be determined from the video stream to be detected based on the two first confidence levels.

[0044] For example, determining a target frame image from a video stream to be detected based on two first confidence levels includes: when at least one of the two first confidence levels is greater than a first threshold, the frame corresponding to the first confidence level greater than the first threshold is taken as the target frame image; when both first confidence levels are less than or equal to the first threshold, and at least one is greater than a second threshold, if the frame corresponding to the first confidence level greater than the second threshold is closer to the first frame than the last frame of the video stream to be detected, the frame corresponding to the first confidence level greater than the second threshold, the frame preceding the first frame and the last frame of the video stream to be detected are taken as the target frame image; if the frame corresponding to the first confidence level greater than the second threshold is farther from the first frame than the last frame of the video stream to be detected, the frame corresponding to the first confidence level greater than the second threshold, the frame following the first frame and the last frame of the video stream to be detected are taken as the target frame image; when both first confidence levels are less than or equal to the second threshold, all frames of the video stream to be detected are taken as the target frame image.

[0045] Therefore, by dynamically inputting different numbers of consecutive frames based on the confidence level of the intermediate frames, the computational resources used for subsequent bird target detection can be greatly reduced.

[0046] In some embodiments of the present invention, determining the first confidence level of an intermediate frame of a video stream to be detected includes: performing two-dimensional convolution processing, ReLU activation processing, and two-dimensional batch normalization processing on the intermediate frame in sequence, repeating the process at least once, and then performing two-dimensional adaptive average pooling processing and Hardtanh activation processing in sequence to obtain the first confidence level.

[0047] For example, it can be adopted Figure 2The Dynamic Frame Aggregation (DFB) module shown obtains a first confidence level and classifies the target query (i.e., the input target frame image) into three categories based on this first confidence level. DFB is the foundation of the entire detection process, aiming to optimize resource allocation. For targets with high confidence, DFB reduces the number of consecutive frames input; for targets with low confidence, it increases the number of consecutive frames input, thereby reducing computational resources.

[0048] See Figure 2 The target image (i.e., the image corresponding to the intermediate frame; if there are two, they are input sequentially) is input into the DFB. First, the Dynamic Confidence Scorer extracts features from the input target image, including: first, a 2D convolutional layer (Conv2d) extracts image features, outputting a feature map with 64 channels; then, a ReLU activation layer is applied to introduce non-linearity and improve the DFB's ability to fit complex features; finally, a 2D batch normalization layer (BatchNorm2d) performs batch normalization on the 64-channel features to prevent gradient explosion and improve the model's generalization ability. The layer normalization calculation method is as follows:

[0049] ;

[0050] in, It is along the feature dimension The mean, It is along the feature dimension variance It is a very small constant, the purpose of which is to prevent the denominator from becoming 0.

[0051] Next, the Conv2d, ReLU, and BatchNorm2d operations are repeated to extract higher-level, more abstract feature representations. Then, a two-dimensional adaptive average pooling layer, AdaptiveAvgPool2d, is used to preserve global feature information. In multi-frame input scenarios, adaptive pooling can automatically adapt without manual adjustment of feature sizes, thus improving the network's versatility and compatibility. Since the detection target is only birds, the Hardtanh function is used to map the output value between 0 and 1, obtaining the target query confidence score. The Hardtanh function is more sensitive to large-scale regional changes, thus outputting a more accurate target image confidence score, obtaining the first confidence score.

[0052] Taking a single intermediate frame as an example, see [link / reference]. Figure 2After obtaining the first confidence level, the GradingModule classifies the target query into three categories based on the first confidence level: when the first confidence level is higher than the first threshold (e.g., 0.7), only the intermediate frames are used for subsequent feature fusion; when the confidence level is between the first and second thresholds (e.g., 0.4-0.7), the first frame, the intermediate frame, and the last frame are used for fusion; when the confidence level is lower than the second threshold (e.g., 0.4), all frames in the video stream to be detected are used for fusion.

[0053] S13, the target frame image is stitched into an initial feature matrix, and a three-dimensional convolution aggregation operation is performed on the initial feature matrix to obtain the aggregated feature as the first feature.

[0054] In some embodiments of the present invention, the number of target frame images is multiple. A three-dimensional convolution aggregation operation is performed on the initial feature matrix to obtain aggregated features, including: sequentially performing three-dimensional convolution and three-dimensional pooling operations on the initial feature matrix to obtain a first feature matrix; sequentially performing three-dimensional convolution and three-dimensional pooling operations on the first feature matrix to obtain a second feature matrix; performing a Sigmoid activation operation on the second feature matrix to obtain a third feature matrix; performing a dot product between the second and third feature matrices to obtain a fourth feature matrix; superimposing the fourth feature matrix with the first feature matrix to obtain a fifth feature matrix; and sequentially performing three-dimensional convolution and three-dimensional pooling operations on the fifth feature matrix to obtain aggregated features.

[0055] For example, it can be adopted Figure 3 The 3D convolutional attention pooling network CP3 shown yields aggregated features. Since the features of some birds are not prominent in a single frame, this invention uses 3D convolution to aggregate the features of birds across consecutive frames, thereby enhancing their characteristics. Compared to 2D convolution, 3D convolution operates in three dimensions—depth, height, and width—capturing features in both time and space simultaneously. When processing video data, it effectively extracts spatial information from each frame and the correlation between consecutive frames.

[0056] The CP3 module contains three 3D convolutional layers, each with a kernel size of [missing value]. ( Figure 3 Conv This allows for convolutions in both the temporal and spatial dimensions, with a stride of 1 in both. Each convolutional layer is followed by a 3D pooling layer, pool1, with a kernel size of [missing value]. To avoid premature merging of time signals and thus preserve early time information, the kernel size of the remaining pooling layers (ppl2, pool3) is [missing information]. .

[0057] First, the consecutive target frame images A multi-channel initial feature matrix X is concatenated, and then a 3D convolution operation is performed on the initial feature matrix to extract spatiotemporal features from multiple frames of data. Pooling is then performed to reduce data dimensionality, thereby reducing computational cost, resulting in the first feature matrix F1. F1 is then subjected to another 3D convolution and pooling operation to obtain the second feature matrix F2, which further extracts and refines features. F2 is then calculated using the sigmoid function to obtain the third feature matrix F3, which introduces non-linearity to highlight target information. The fourth feature matrix F4 is obtained by multiplying F2 and F3, and then F4 is superimposed on F1 to preserve background information. Finally, another 3D convolution and pooling operation is performed to obtain the final aggregated feature F6, which contains spatiotemporal features from multiple frames for subsequent processing.

[0058] S14. After downsampling the first feature multiple times, feature extraction is performed in multiple stages corresponding to the multiple downsampling steps to obtain the second feature. Based on the second feature, a confidence feature map and a location feature map are obtained. The input of each stage is the output of the previous stage and the downsampling result of the corresponding step.

[0059] In some embodiments of the present invention, in each stage, the input of the previous stage is subjected to matrix transformation and then input to the decoder after positional encoding. In the decoder, the result of the positional encoding operation is normalized to obtain a query vector, a key vector, and a value vector. The key vector and the value vector are spatially reduced, and a third feature is obtained based on the query vector and the reduced key vector and value vector using a self-attention mechanism. After passing through a feedforward network, normalization, and residual connections, upsampling is performed, and multiple convolution operations are performed on the upsampling result and the corresponding downsampling result to obtain the second feature.

[0060] For example, it can be adopted Figure 4The Cross-Feature Pyramid Transform Decoder (CFPD) module shown in the diagram obtains the second feature. Traditional feature pyramid models mainly consist of upsampling and downsampling. By fusing downsampled high-level semantic features with shallow localization detail features, a final high-resolution feature map with strong semantic information is output. However, due to the limited local receptive field of feature pyramids, they struggle to capture long-distance dependencies, restricting their ability to model complex spatial relationships and global context. Because of the multi-scale nature of birds in videos, different bird species exhibit significant differences in appearance, and even birds of the same species can have vastly different flight postures. Furthermore, some birds occupy few pixels in the frame, making them small targets, which results in a lack of robust shape and texture. Therefore, this invention combines feature pyramids and a Transformer decoder to predict bird targets in videos, thereby fully leveraging the advantages of both models, compensating for their shortcomings, better adapting to the characteristics of bird targets in videos, and improving the accuracy of bird target prediction in videos.

[0061] CFPD is responsible for feature extraction, enhancement, and fusion. By using this module, we can enhance the ability to capture complex spatial relationships and long-distance dependencies, integrate information at different scales, and reduce the loss of dense information during downscaling, which plays a crucial role in improving the accuracy of target detection.

[0062] Taking a three-stage downsampling process as an example, see [link / reference]. Figure 4 The feature embedding after CP3 aggregation serves as the lowest-resolution feature map of the pyramid structure. After three downsampling operations, a feature map with even lower resolution and a larger receptive field is obtained. The lowest-resolution feature map is used as input to stage 1 and fused with itself to obtain a feature map with stronger feature representation. This output is used as input to the next stage, processed by the decoder, and then fused with the higher-resolution features in the pyramid structure. This process is repeated until the highest-resolution feature map of the pyramid is obtained.

[0063] In each stage, the input feature maps first undergo a matrix reshape operation to rearrange and combine them. After positional encoding, they are input to the decoder. The self-attention mechanism receives three inputs: a query (Q) vector, a key (K) vector, and a value (V) vector, with dimensions of [missing information]. ,in and Represents the height and width of the i-th feature map. This represents the number of channels in the i-th feature map. Since calculating the attention weights involves numerous matrix multiplications and requires significant computational overhead, a spatial reduction block is used to reduce the spatial dimensions of the inputs K and V, resulting in an output with the following dimension:

[0064] ;

[0065] in, This is a reduction factor.

[0066] Subsequently, a self-attention mechanism is used to ensure that each location in the feature map can pay attention to feature information from other locations, capturing long-distance dependencies between features. After passing through a feedforward network, normalization, and residual connections, the features are further subjected to nonlinear transformations and feature extraction. Finally, upsampling is used to improve the resolution of the feature map, enabling it to be fused with the feature map of the next layer. The principle of the fusion block is to fuse two feature maps of the same size through a series of convolutional operations (such as Conv 1×1, Conv 3×3, etc.), integrating multi-scale and multi-level feature information. After three stages of processing, the final fused feature map, i.e., the second feature, is obtained.

[0067] After obtaining the second feature, the second feature is processed by two feedforward neural networks to obtain the confidence feature map of the prediction confidence and the position feature map of the target bird, so as to achieve accurate prediction of the bird target in the video.

[0068] S15, determine the bird situation in the video stream to be detected based on the confidence feature degree and the position feature map.

[0069] In some embodiments of the present invention, determining the bird situation in the video stream to be detected based on the confidence feature and the position feature map includes: outputting a second confidence based on the confidence feature map through a first fully connected layer, and outputting a regression result based on the position feature map through a second fully connected layer; determining the bird situation in the video stream to be detected based on the second confidence and the regression result.

[0070] As one implementation method, the aforementioned modules such as DFB, CP3, and CFPD can be integrated into a multi-frame adaptive network (MFA-net) for bird detection based on video streams. The multi-frame adaptive network can be trained before actual use. The specific structure of the multi-frame adaptive network is as follows: Figure 5As shown, MFA-net first uses the dynamic confidence scorer in DFB to score the confidence of intermediate frames in the input video stream to be detected, and divides the target image into three levels according to the confidence score. Different numbers of consecutive frames are input according to different levels for subsequent processing. The CP3 module performs feature aggregation and processing on the input consecutive frames through 3D convolution and pooling operations. The CFPD module extracts features from the aggregated feature map, thereby improving the accuracy of bird target detection in the frame image. Finally, two fully connected layers (FFN) output the confidence score (Con) and regression result (Reg), and finally complete the detection of bird targets, outputting the final detection result, that is, the bird situation in the video stream to be detected, such as whether there are birds and the location of the birds.

[0071] For example, see Figure 5 When there is only one target frame image, the computation is small, and the target frame image can be directly processed by a convolutional neural network (CNN) to obtain the convolutional feature as the first feature.

[0072] Optionally, see Figure 5 Before inputting the video stream to be detected into the DFB, the video stream can be processed by a convolutional neural network to provide an efficient and robust data representation for subsequent detection and tracking.

[0073] In some embodiments of the present invention, a multi-frame adaptive network is used to obtain the second confidence score and regression result based on the video stream to be detected. The multi-frame adaptive network only targets bird objects in the video; therefore, the loss function does not include class loss, but only confidence loss and regression position loss. The specific expressions are as follows:

[0074] ;

[0075] Where L represents the loss function, Indicates confidence loss. α represents the position loss, and α represents the balance parameter.

[0076] For example, the expression for confidence loss is as follows:

[0077] ;

[0078] Where N represents the number of samples, This represents the true label of the i-th sample. This represents the second confidence score for the prediction of the i-th sample.

[0079] For example, the expression for position loss is as follows:

[0080] ;

[0081] Where N represents the number of samples, and j is the parameter of the true bounding box (including the coordinates of the center point of the bounding box, as well as the height and width of the bounding box). This represents the true bounding box of the i-th sample. This represents the predicted bounding box of the i-th sample. express Smoothing loss function.

[0082] Through experiments, the detection accuracy curve of birds in the video was obtained, as shown below. Figure 6 As shown, the horizontal axis represents the number of training epochs of the multi-frame adaptive network; the vertical axis represents ap-50, indicating the average accuracy when the intersection-union ratio (IU) threshold between the predicted and ground truth bounding boxes is set to 0.5. Figure 6 As can be seen, the curve first increases monotonically with the number of training rounds and finally converges to 0.7047ap, indicating that the method proposed in this invention can effectively detect flying birds in videos.

[0083] In summary, the video stream-based bird detection method of this invention can effectively improve the detection accuracy of birds in videos and significantly reduce the waste of computational resources. Specifically, it utilizes a dynamic frame aggregation module to adjust the number of consecutive input frames to reduce computational load; it uses a 3D convolutional attention pooling module to aggregate the features of birds in consecutive frames, thereby enhancing the features of birds; and it uses a cross-feature pyramid transform decoder module to integrate information from feature maps at different scales, thereby enhancing the model's ability to capture complex spatial relationships and long-distance dependencies, and mitigating the loss of dense information during downscaling. Compared with traditional target detection methods, the method proposed in this invention uses fewer computational resources and has higher computational accuracy, playing an important role in the field of video bird target detection, and has broad application value and economic benefits.

[0084] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0085] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0086] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0087] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0088] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0089] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0090] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0091] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A bird detection method based on video stream, characterized in that, The method includes: Obtain the video stream to be detected; Determine the first confidence level of the intermediate frame of the video stream to be detected, and determine the target frame image from the video stream to be detected based on the first confidence level; The target frame images are stitched together to form an initial feature matrix, and a three-dimensional convolution aggregation operation is performed on the initial feature matrix to obtain the aggregated features as the first feature; After downsampling the first feature multiple times, feature extraction is performed in multiple stages corresponding to the multiple downsampling steps to obtain the second feature. Based on the second feature, a confidence feature map and a location feature map are obtained. The input of each stage is the output of the previous stage and the downsampling result of the corresponding step. The bird situation in the video stream to be detected is determined based on the confidence feature and the location feature map; The step of determining the first confidence level of the intermediate frame of the video stream to be detected includes: The intermediate frames are sequentially subjected to two-dimensional convolution, ReLU activation, and two-dimensional batch normalization, and this process is repeated at least once. Then, two-dimensional adaptive average pooling and Hardtanh activation are sequentially performed to obtain the first confidence score. Determining the target frame image from the video stream to be detected based on the first confidence level includes: When the first confidence level is greater than the first threshold, the intermediate frame is used as the target frame image; when the first confidence level is greater than the second threshold and less than or equal to the first threshold, the first frame, intermediate frame, and last frame of the video stream to be detected are used as the target frame image; when the first confidence level is less than or equal to the second threshold, all frames of the video stream to be detected are used as the target frame image.

2. The bird detection method based on video stream according to claim 1, characterized in that, The method further includes: Before stitching the target frame images into an initial feature matrix, it is determined that there are multiple target frame images; When the number of target frame images is one, the target frame image is processed by a convolutional neural network to obtain the convolutional feature as the first feature.

3. The bird detection method based on video stream according to claim 1, characterized in that, Performing a 3D convolution aggregation operation on the initial feature matrix to obtain the aggregated features includes: The initial feature matrix is ​​subjected to three-dimensional convolution and three-dimensional pooling operations in sequence to obtain the first feature matrix; The first feature matrix is ​​subjected to three-dimensional convolution and three-dimensional pooling operations in sequence to obtain the second feature matrix. Perform a Sigmoid activation operation on the second feature matrix to obtain the third feature matrix; The second feature matrix is ​​multiplied by the third feature matrix to obtain the fourth feature matrix; The fourth feature matrix is ​​superimposed on the first feature matrix to obtain the fifth feature matrix; The fifth feature matrix is ​​subjected to three-dimensional convolution and three-dimensional pooling operations in sequence to obtain the aggregated features.

4. The bird detection method based on video stream according to claim 1, characterized in that, In each stage, the input from the previous stage is subjected to matrix transformation and then input to the decoder after positional encoding. In the decoder, the position encoding operation result is normalized to obtain a query vector, a key vector, and a value vector. The key vector and the value vector are spatially reduced, and a third feature is obtained based on the query vector and the reduced key vector and value vector using a self-attention mechanism. After passing through a feedforward network, normalization, and residual connections, upsampling is performed, and multiple convolution operations are performed on the upsampling result and the corresponding downsampling result to obtain the second feature.

5. The bird detection method based on video stream according to claim 1, characterized in that, The step of determining the bird situation in the video stream to be detected based on the confidence feature and the location feature map includes: The first fully connected layer outputs a second confidence level based on the confidence feature map, and the second fully connected layer outputs a regression result based on the location feature map. The bird situation in the video stream to be detected is determined based on the second confidence level and the regression results.

6. The bird detection method based on video stream according to claim 1, characterized in that, The second confidence score and the regression result are obtained from the video stream to be detected using a multi-frame adaptive network. The expression for the loss function of the multi-frame adaptive network is as follows: ； Where L represents the loss function, Indicates confidence loss. Indicates position loss. This represents the equilibrium parameters.

7. The bird detection method based on video stream according to claim 6, characterized in that, The expression for the confidence loss is as follows: ； Where N represents the number of samples, This represents the true label of the i-th sample. This represents the second confidence score for the prediction of the i-th sample.

8. A bird detection method based on video stream according to claim 6, characterized in that, The expression for the position loss is as follows: ； Where N represents the number of samples, and j is the parameter of the ground truth bounding box. This represents the true bounding box of the i-th sample. This represents the predicted bounding box of the i-th sample. express Smoothing loss function.

Images