Real-time bird tracking method and device, electronic equipment and storage medium

Abstract

The invention provides a real-time bird tracking and identification method and device, electronic equipment and a storage medium, and the method comprises the steps: collecting real-time bird multi-modal data which comprises video data, audio data and environment data; based on the real-time bird multi-modal data, a bird target detection model is used for detection and recognition, and a real-time bird detection and recognition result is obtained; and based on the real-time bird detection and identification result and the real-time bird multi-modal data, performing multi-target tracking by using an adaptive Kalman filtering algorithm to obtain a real-time bird tracking result. Compared with the prior art, the method of the invention has the advantages that through the bird target detection model and the adaptive Kalman filtering algorithm, the real-time bird tracking accuracy is improved; according to the method, multi-modal data, bird types and tracking information can be deeply fused and analyzed, different environmental conditions and bird types can be adapted, real-time dynamic tracking of birds can be realized, and ecological research and wild animal protection are facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer deep learning, in particular to a real-time bird tracking method and device, electronic equipment and storage medium. BACKGROUND

[0002] Under the background of increasing demand for ecological research and wildlife protection, bird behavior research is of great significance to ecology, environmental protection and biodiversity protection, so that bird behavior monitoring technology is attracting more and more attention. By tracking birds, the activity rules and health status of birds can be better understood so as to take timely measures to protect birds and the ecological environment. Traditional bird behavior observation methods mainly rely on manual observation and recording, which is not only time-consuming and laborious, but also difficult to obtain continuous data for a long time and a large range.

[0003] The existing technology still has many deficiencies: the existing deep learning model has large number of parameters, which is difficult to run in real time on edge devices, the target tracking is easy to fail in complex environment, the deep fusion of time sequence features and multi-modal information is lacking, and the device endurance is limited in long-term field monitoring. The existence of these problems leads to the difficulty of efficient, long-term and large-scale bird tracking and behavior research, which limits the research progress in the fields of ecology, environmental protection and biodiversity protection.

[0004] Therefore, there is an urgent need for a bird tracking method and device with high computing efficiency, high tracking accuracy and real-time operation on mobile devices to meet the needs of ecological research, wildlife protection and bird watching enthusiasts. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a real-time bird tracking method, device, electronic equipment and storage medium.

[0006] In a first aspect, the present application provides a real-time bird tracking method, comprising:

[0007] Collecting real-time bird multi-modal data, the real-time bird multi-modal data comprising video data, audio data and environmental data;

[0008] Based on the real-time bird multi-modal data, detecting and identifying using a bird target detection model to obtain real-time bird detection and identification results;

[0009] Based on the real-time bird detection and identification results and the real-time bird multi-modal data, performing multi-target tracking using an adaptive Kalman filtering algorithm to obtain real-time bird tracking results.

[0010] In some optional embodiments, the real-time bird multi-modal data can also be preprocessed to obtain real-time bird multi-modal preprocessed data.

[0011] In some optional embodiments, the bird target detection model can also be used for detection and identification based on real-time bird multi-modal preprocessed data, to obtain real-time bird detection and identification results.

[0012] In some optional embodiments, the training process of the bird target detection model is as follows:

[0013] A student model is constructed based on MobileNetV3 as a basic network combined with a feature pyramid network, and the student model is a model with a parameter amount less than 10M;

[0014] A large convolutional neural network model is used as a teacher model, and the teacher model is a pre-trained model, and the teacher model is a model with a parameter amount greater than 10M;

[0015] The student model is supervised by the teacher model for knowledge distillation training to obtain a bird target detection model.

[0016] In some optional embodiments, the bird target detection model is used for detection and identification based on the real-time bird multi-modal data to obtain real-time bird detection and identification results, which includes:

[0017] Based on the real-time bird multi-modal data, a bird target detection model is used to obtain corresponding real-time bird species feature data;

[0018] The real-time bird species feature data is classified to obtain corresponding real-time bird classification data, and a bird species feature template library is established;

[0019] The channel attention mechanism is used to perform channel weighting on the real-time bird classification data to obtain key features of real-time bird detection;

[0020] It is determined whether the key features of real-time bird detection are consistent with the real-time bird multi-modal data, if yes, a first operation is performed, otherwise, data corresponding to the key features of real-time bird detection is obtained as updated real-time bird multi-modal data, and real-time bird species feature data is reacquired;

[0021] The first operation is to determine whether the key features of real-time bird detection match the bird species feature template library, if yes, a bird species classification label corresponding to the key features of real-time bird detection is output as a real-time bird detection and identification result, and the key feature data of real-time bird detection is weighted by spatial attention, if not, the classification parameters are adjusted, and the real-time bird species feature data is reclassified;

[0022] The key feature data of real-time bird detection is weighted by spatial attention to obtain spatial features of real-time bird detection;

[0023] determining whether the spatial feature of the real-time bird detection conforms to a bird morphological feature, if yes, obtaining bird target bounding box position information corresponding to the spatial feature of the real-time bird detection, and performing a second operation; if no, marking as an invalid target, and reusing a channel attention mechanism for channel weighting;

[0024] The second operation is: determining whether the bird target bounding box position information corresponds to a clear and complete contour boundary, if yes, outputting the corresponding bird target bounding box coordinates as a real-time bird detection and recognition result; if no, returning to reusing spatial attention for feature weighting.

[0025] In some optional embodiments, the real-time bird tracking result is obtained by using an adaptive Kalman filtering algorithm for multi-target tracking based on the real-time bird detection and recognition result and the real-time bird multi-modal data.

[0026] extracting features from the real-time bird detection and recognition result and the real-time bird multi-modal data to obtain multi-modal features;

[0027] performing multi-level dynamic fusion on the multi-modal features to obtain corresponding multi-modal fusion features;

[0028] obtaining a real-time bird tracking result by using an adaptive Kalman filtering algorithm for multi-target tracking based on the multi-modal fusion features;

[0029] The multi-modal features include appearance features, motion features, context features, and audio features of the bird.

[0030] In some optional embodiments, the multi-level dynamic fusion on the multi-modal features to obtain corresponding multi-modal fusion features includes:

[0031] directly connecting the multi-modal features for a first fusion to form multi-dimensional initial fusion features;

[0032] using channel attention and spatial attention mechanisms to perform a second weighting fusion on the initial fusion features to generate weighted fusion features;

[0033] using an adaptive selection mechanism to dynamically adjust the weights of the initial fusion features to obtain dynamic fusion features;

[0034] integrating the initial fusion features, the weighted fusion features, and the dynamic fusion features by using ensemble learning to obtain multi-modal fusion features.

[0035] In some optional embodiments, the dynamic adjustment of the weight of the initial fusion feature by using an adaptive selection mechanism to obtain a dynamic fusion feature comprises:

[0036] According to the motion feature in the initial fusion feature, corresponding motion state data is obtained, and it is determined whether the motion state data corresponds to a high-speed motion state. If yes, the weight of the motion feature in the initial fusion feature is adjusted to obtain corresponding dynamic fusion feature. If no, the initial fusion feature is directly used as the dynamic fusion feature;

[0037] It is determined whether the appearance feature in the initial fusion feature is consistent with the context feature. If yes, corresponding ambient light intensity data is obtained according to the environment feature in the context feature. If no, the environment feature consistent with the appearance feature is obtained as an updated context feature, and the multi-dimensional initial fusion feature is reformed in the first fusion stage;

[0038] It is determined whether the ambient light intensity data exceeds a normal ambient light threshold range. If yes, the color weight in the initial fusion feature is adjusted to obtain corresponding dynamic fusion feature. If no, the initial fusion feature is directly used as the dynamic fusion feature;

[0039] The light value of the normal ambient light threshold range is 100-1000 lux. When the light value is less than or equal to 100 lux, it is determined as a low light condition. When the light value is greater than or equal to 1000 lux, it is determined as a high light condition.

[0040] In some optional embodiments, based on the multi-modal fusion feature, an adaptive Kalman filtering algorithm is used for multi-target tracking to obtain a real-time bird tracking result, which comprises:

[0041] Based on the multi-modal fusion feature, a state vector of a real-time bird target is obtained, and a state space model is constructed;

[0042] According to the state space model, a corresponding predicted state vector and a covariance matrix are obtained respectively;

[0043] By using an adaptive process noise covariance matrix and a measurement noise covariance matrix, the predicted state vector and the covariance matrix are adjusted based on the Kalman filtering algorithm to obtain an updated state vector and covariance matrix;

[0044] It is determined whether the speed parameter in the updated state vector is within a preset normal speed threshold range. If yes, the updated state vector is output as a real-time bird motion state change result. If no, a corresponding acceleration parameter is obtained for determination;

[0045] The judging of the obtained corresponding acceleration parameter is that whether the acceleration parameter in the updated state vector exceeds a preset normal acceleration threshold range, if yes, the updated state vector is output as the real-time bird motion state change result, if not, the bird target is marked as being in an abnormal motion state, and the adaptive Kalman filtering algorithm is returned to adjust again.

[0046] The real-time bird motion state change result is subjected to multi-hypothesis tracking processing to obtain a real-time bird tracking result.

[0047] In a second aspect, the present application provides a real-time bird tracking device, comprising:

[0048] A data acquisition module is configured to acquire real-time bird multi-modal data, wherein the real-time bird multi-modal data comprises video data, audio data and environmental data.

[0049] A detection and recognition module is configured to detect and recognize the real-time bird multi-modal data based on a bird target detection model to obtain a real-time bird detection and recognition result.

[0050] A multi-target tracking module is configured to track multiple targets based on the real-time bird detection and recognition result and the real-time bird multi-modal data by using an adaptive Kalman filtering algorithm to obtain a real-time bird tracking result.

[0051] In a third aspect, the present application provides an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementations of the first aspect.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by one or more processors to implement the method described in any of the implementations of the first aspect.

[0053] In order to improve the efficiency and accuracy of bird tracking, the real-time bird tracking method, device, electronic device and storage medium provided by the embodiments of the present application first acquire real-time bird multi-modal data, wherein the real-time bird multi-modal data comprises video data, audio data and environmental data; then detect and recognize the real-time bird multi-modal data based on a bird target detection model to obtain a real-time bird detection and recognition result; finally, track multiple targets based on the real-time bird detection and recognition result and the real-time bird multi-modal data by using an adaptive Kalman filtering algorithm to obtain a real-time bird tracking result.

[0054] Compared with the closest prior art, the present application has the beneficial effects that:

[0055] The real-time bird tracking method provided by the application significantly reduces the computational complexity by means of deep separable convolution and model optimization technology, so that the device can run in real time on a mobile device or an embedded device, and the calculation speed is increased by 3-5 times compared with the traditional method; the adaptive Kalman filtering algorithm and the multi-hypothesis tracking strategy are combined to effectively improve the bird tracking accuracy in complex environments, and the average tracking success rate is increased by more than 20%; the adaptive learning mechanism enables the device to be continuously optimized and updated, and to adapt to different environmental conditions and bird species, and can be widely applied to the fields of ecological research, wildlife protection, bird watching activities and environmental monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a flowchart of the real-time bird tracking method of the application;

[0057] Figure 2 is a schematic diagram of the device of the real-time bird tracking method of the application. DETAILED DESCRIPTION

[0058] The specific embodiments of the application will be further described in detail below with reference to the accompanying drawings.

[0059] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings of the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0060] Embodiment 1:

[0061] The application provides a real-time bird tracking method, as shown in Figure 1 , comprising:

[0062] S1, collecting real-time bird multi-modal data, the real-time bird multi-modal data comprising video data, audio data and environmental data.

[0063] Mainly obtained by means of multi-modal sensors, by means of high-resolution cameras, directional microphone arrays, for collecting bird calls, dynamic images and other video data and audio data, by means of environmental sensors, for collecting temperature, humidity, illumination and other environmental data.

[0064] It should be noted that the present embodiment collects real-time data, and the multi-modal sensor can include but is not limited to image acquisition equipment, sound recording equipment and environmental monitoring equipment.

[0065] S2, based on the real-time bird multi-modal data, using a bird target detection model for detection and identification, obtaining real-time bird detection and identification results.

[0066] Mainly input the real-time bird multi-modal data into the bird target detection model, optimize the features by channel attention and spatial attention mechanism weighting feature channels and feature space regions, output the bird species classification label and the bird target bounding box coordinates as the real-time bird detection and identification results.

[0067] S3, based on the real-time bird detection and identification results and the real-time bird multi-modal data, using an adaptive Kalman filtering algorithm for multi-target tracking, obtaining real-time bird tracking results.

[0068] Mainly design an adaptive Kalman filtering algorithm for target tracking, dynamically adjust the process noise covariance matrix and the measurement noise covariance matrix, improve the adaptability to the nonlinear motion mode of birds, and the adaptive Kalman filtering algorithm combined with multi-hypothesis tracking processing can effectively handle the complex situations such as fast motion, partial occlusion and temporary disappearance of birds, improve the bird tracking accuracy in complex environment, and the average tracking success rate is improved by more than 20%.

[0069] As an implementable manner, in the above embodiment, step S2 based on the real-time bird multi-modal data, using a bird target detection model for detection and identification, obtaining real-time bird detection and identification results includes:

[0070] S2-1, based on the real-time bird multi-modal data, using a bird target detection model, obtaining corresponding real-time bird species feature data;

[0071] S2-2, classifying the real-time bird species feature data to obtain corresponding real-time bird classification data, and establishing a bird species feature template library;

[0072] S2-3, using channel attention mechanism to channel weight the real-time bird classification data, obtaining key features of real-time bird detection;

[0073] S2-4, judging whether the key features of real-time bird detection are consistent with the real-time bird multi-modal data, if yes, executing S2-5; if not, obtaining the data corresponding to the key features of real-time bird detection as updated real-time bird multi-modal data, and returning to S2-1;

[0074] S2-5, determining whether the key features of the real-time bird detection match the bird species feature template library, if yes, outputting the bird species classification label corresponding to the key features of the real-time bird detection as the real-time bird detection and recognition result, and performing S2-6; if not, adjusting the classification parameters and returning to S2-2 for re-classification;

[0075] S2-6, performing feature weighting on the key feature data of the real-time bird detection through spatial attention to obtain spatial features of the real-time bird detection;

[0076] S2-7, determining whether the spatial features of the real-time bird detection conform to the bird morphological features, if yes, obtaining the bird target bounding box position information corresponding to the spatial features of the real-time bird detection, and performing S2-8; if not, marking as invalid target and returning to S2-3 for feature optimization;

[0077] S2-8, determining whether the bird target bounding box position information corresponds to a clear and complete contour boundary, if yes, outputting the corresponding bird target bounding box coordinates as the real-time bird detection and recognition result; if not, returning to S2-6;

[0078] The real-time bird species feature data includes but is not limited to sound features (songs or songs), visual features (colors and textures, shapes and contours, sizes), behavior features (flight methods, flight speeds, activity patterns).

[0079] The key features of the real-time bird detection are morphological features, color features and texture features, which are calculated by a channel attention mechanism to obtain the importance weight of each feature channel, and the key features of feather color, key part shape and species identification are obtained. The spatial features of the real-time bird detection are the size, contour features and spatial position features of the body parts of the bird, which are calculated by a spatial attention mechanism to give a feature spatial weight distribution, determine the specific position of the bird and the details of its surrounding environment, help the network better understand the position and boundary of the target, and thus improve the accuracy of target detection.

[0080] As an implementable manner, in the above embodiment, the training of the bird target detection model in step S2 specifically includes the following processes:

[0081] Taking MobileNetV3 as the basic network, a student model (with a parameter quantity less than 10M) is constructed in combination with a feature pyramid network;

[0082] Taking a pre-trained EfficientNet-B7 model as a teacher model (a large convolutional neural network model with a parameter quantity greater than 10M);

[0083] Supervise the student model to perform knowledge distillation training with the teacher model to obtain the bird target detection model.

[0084] In the embodiment, the knowledge distillation loss function is:

[0085] In the formula, L_CE is a cross-entropy loss, KL is a KL divergence, z_s and z_t are output logits of the student model and the teacher model respectively, τ is a temperature parameter, and α is a balance factor.

[0086] The knowledge of the EfficientNet-B7 model (teacher model) is migrated to a lightweight model (learning model), the calculation complexity is significantly reduced while maintaining high recognition accuracy, channel attention and spatial attention mechanisms are introduced to enhance the model's ability to perceive key features, 8-bit fixed-point quantization technology is used to convert the 32-bit floating-point model to an 8-bit integer model, reducing storage and calculation overhead; model pruning is performed through L1 regularization and sensitivity analysis to remove redundant connections, further compressing the model size, and the bird target detection model is obtained through optimization, realizing fast detection and recognition of bird targets in video frames.

[0087] The embodiment provides a convolutional network structure of the bird target detection model:

[0088] The input layer receives real-time bird multi-modal data or real-time bird multi-modal preprocessed data;

[0089] The MobileNetV3 backbone network part:

[0090] The first stage is composed of a standard convolutional layer, which is used to extract initial features from the input image. The convolution kernel size, stride, input and output channel number and other parameters of the convolutional layer are set according to the requirements. In this embodiment, the convolution kernel size is 3x3, the stride is 2, which is used for preliminary down-sampling to reduce the size of the feature map, and the channel number is increased to extract basic features such as edges and textures to a certain extent;

[0091] Depthwise separable convolution stage: the core part of MobileNetV3, containing multiple depthwise separable convolution modules, each module is composed of depthwise convolution and pointwise convolution, depthwise convolution performs convolution operation on each input channel separately, emphasizing the extraction of spatial information of each feature channel, pointwise convolution is responsible for feature fusion between channels, changing the number of channels or performing nonlinear transformation, etc. These modules are stacked to continuously extract and transform features, gradually capturing more complex bird target features. In this embodiment, there is a depthwise separable convolution module first, the convolution kernel size is 3x3, the stride is 1, the input channel number is 16, and the output channel number is 24, followed by another module, the convolution kernel size is 5x5, the stride is 2, the input channel number is 24, and the output channel number is 40, etc. In this way, multiple feature maps of different levels are formed, and these feature maps have different scales and different levels of semantic information.

[0092] Feature pyramid network (FPN, Feature Pyramid Network) part:

[0093] Top-down path: starting from the highest layer feature map output by the MobileNetV3 backbone network, gradually up-sampling and fusing with the feature maps from lower layers in the backbone network. In this embodiment, the highest layer feature map is first up-sampled (such as using nearest neighbor interpolation or bilinear interpolation) to make its size consistent with the previous layer feature Figure 1 Then, the two are element-wise added or spliced, so that high-level semantic information is propagated to the shallow feature map, enhancing the semantic expression ability of the shallow feature, which helps to detect bird targets of different scales, especially for smaller bird targets, making up for the use of detailed information in shallow features, combining with the context semantics, and improving the detection accuracy.

[0094] Optionally, the FPN part can also be a bottom-up path: continue to extract features along the original feature extraction path of the backbone network, while further enriching the feature information using the fused features from the top-down path, forming a multi-scale feature pyramid structure that integrates different levels of semantic information, providing a more comprehensive feature basis for subsequent detection and recognition tasks.

[0095] Knowledge distillation part:

[0096] Teacher model (large pre-training model): It can be a large target detection model with excellent performance, such as a detection model based on a complex architecture such as ResNet (in this embodiment, the teacher model is an EfficientNet-B7 model). The teacher model is pre-trained on a large-scale bird image dataset and has high detection accuracy. In the knowledge distillation process, its main role is to provide knowledge guidance, including but not limited to predicted bird class probability distribution, bounding box regression parameters, and other information.

[0097] Student model (in this embodiment, it is the lightweight MobileNetV3-FPN model described above): During the training process, the output of the teacher model (such as the classification probability distribution after softmax and the bounding box coordinates) is used as a supervision signal together with the real label data to guide the student model to learn the knowledge of the teacher model by designing a suitable loss function (in this embodiment, a knowledge distillation loss function is used, in which the cross-entropy loss is used for the student model to directly learn the real label, and the distillation loss is used for the student model to imitate the output of the teacher model). The student model can learn the knowledge of the teacher model while maintaining lightweight, so that it can approach the performance of the teacher model as much as possible, reducing the computational complexity while maintaining high bird target detection and recognition accuracy.

[0098] Output layer: According to different task requirements, such as bird target classification and identification, target positioning (such as outputting bounding box coordinates), etc., the corresponding output neurons are designed. For example, for bird classification and identification, the number of output neurons corresponds to the number of bird species, and a softmax activation function is used to output the probability of each class; for target positioning, the coordinates of the bounding box are outputted, etc. In this embodiment, the bounding box coordinates of the bird target, the species classification label, and the detection confidence are outputted.

[0099] As an implementation manner, the above structure provided by the embodiment is a relatively general convolutional network framework for bird target detection and identification based on MobileNetV3, FPN, and knowledge distillation. In actual application, the specific convolution kernel size, channel number, layer number, and other parameters need to be carefully adjusted and optimized according to the dataset, computing resources, and target performance requirements, etc. At the same time, the hyperparameters (such as learning rate, batch size) in the training process are also crucial to ensure that the model can effectively learn and achieve the expected performance.

[0100] As an implementation manner, in the above embodiment, step S2 can also be based on the real-time bird multi-modal pre-processing data to detect and identify using the bird target detection model to obtain real-time bird detection and identification results.

[0101] As an implementation manner, in the above embodiment, step S3 can specifically include the following process:

[0102] S3-1, feature extraction is performed on the real-time bird detection and identification result and the real-time bird multi-modal data to obtain multi-modal features;

[0103] The appearance feature, motion feature, context feature, and audio feature of the bird are extracted from the bird detection and identification result and the real-time bird multi-modal data.

[0104] The appearance feature is a shape feature of the bird obtained through contour analysis and pose estimation; the color feature is obtained by extracting an HSV color histogram (HSV is an abbreviation of Hue, Saturation, and Value) and a color moment; and the texture feature is obtained by extracting a local binary pattern (LBP) and a Gabor filter feature.

[0105] The motion feature is extracted from consecutive video frames, mainly including a bird optical flow feature obtained by using a dense optical flow algorithm to calculate a pixel-level motion vector; a trajectory feature of the bird obtained by extracting a motion trajectory and speed acceleration information of a target center point; and a pose change feature of the bird obtained by extracting relative motion features of key parts such as wings and heads.

[0106] The context feature is context information of a surrounding environment of the bird, including a scene feature obtained by identifying an environment type such as a forest, grassland, and water area; a time feature obtained by extracting time information such as a time period and season; and an environment feature obtained by extracting environmental parameters such as temperature, humidity, and illumination.

[0107] The audio feature is extracted from audio data, mainly including a time domain feature such as an energy envelope and a zero-crossing rate; a frequency domain feature such as a mel-frequency cepstral coefficient (MFCC) and a spectral centroid; and a time-frequency feature such as a short-time Fourier transform (STFT).

[0108] S3-2, multi-level dynamic fusion is performed on the multi-modal features to obtain corresponding multi-modal fusion features;

[0109] S3-2-1, first fusion is performed on the multi-modal features by direct connection to form multi-dimensional initial fusion features;

[0110] S3-2-2, the initial fusion features are subjected to second weighted fusion by using a channel attention and a spatial attention mechanism to generate weighted fusion features;

[0111] S3-2-3, the weights of the initial fusion features are dynamically adjusted by using an adaptive selection mechanism to obtain dynamic fusion features;

[0112] According to the motion feature in the initial fusion feature, corresponding motion state data is acquired, it is judged whether the motion state data corresponds to a high-speed motion state, if yes, the motion feature weight in the initial fusion feature is adjusted to obtain corresponding dynamic fusion feature; if no, the initial fusion feature is directly used as the dynamic fusion feature;

[0113] It is judged whether the appearance feature in the initial fusion feature is consistent with the context feature, if yes, corresponding ambient light intensity data is acquired according to the environment feature in the context feature; if no, the environment feature consistent with the appearance feature is acquired as the updated context feature, and the multi-dimensional initial fusion feature is reformed in the first fusion stage;

[0114] It is judged whether the ambient light intensity data exceeds the normal ambient light threshold range, if yes, the color weight in the initial fusion feature is adjusted to obtain corresponding dynamic fusion feature; if no, the initial fusion feature is directly used as the dynamic fusion feature;

[0115] The formula for adjusting the weight is:

[0116] In the formula, w_i represents the weight of the i-th feature, f_θ represents the feature weight prediction network, and c represents the encoding of the current scene condition.

[0117] The light value of the normal ambient light threshold range is 100-1000 lux; when the light value is ≤100 lux, it is determined as a low light condition; when the light value is ≥1000 lux, it is determined as a high light condition.

[0118] S3-2-4, the initial fusion feature, the weighted fusion feature and the dynamic fusion feature are integrated by using ensemble learning to obtain a multi-modal fusion feature.

[0119] The spatial mutual information amount of the initial fusion feature and the weighted fusion feature is calculated, and the historical matching degree of the dynamic fusion feature and the target trajectory is evaluated, and the weight is distributed, when the mutual information amount >0.7 and the historical matching degree >0.8, the dynamic fusion feature is given 60%-70% weight, when the mutual information amount ≤0.7 or the historical matching degree ≤0.8, the random forest regressor is used to predict the optimal weight combination; the three types of features are weighted and summed according to the distributed weight, and the multi-modal fusion feature is integrated.

[0120] S3-3, based on the multi-modal fusion feature, an adaptive Kalman filter algorithm is used for multi-target tracking to obtain real-time bird tracking results;

[0121] S3-3-1, based on the multi-modal fusion feature, the state vector of the real-time bird target is acquired, and a state space model is constructed;

[0122] The state vector of the real-time bird target is: wherein (x, y) represents the target center coordinates, (w, h) represents the target width and height, (vx, vy) represents the center point speed, and (vw, vh) represents the width and height change rate.

[0123] The state transition equation is: wherein F_t is the state transition matrix, and w_t is the process noise, which is subject to a Gaussian distribution with a mean of 0 and a covariance of Q_t;

[0124] S3-3-2, using an adaptive Kalman filter algorithm, dynamically adjusting the state space model to obtain a real-time bird motion state change result;

[0125] S3-3-2-1, respectively obtaining a corresponding predicted state vector and covariance matrix according to the state space model;

[0126] S3-3-2-2, using an adaptive process noise covariance matrix and a measurement noise covariance matrix, adjusting the predicted state vector and covariance matrix based on the Kalman filter algorithm to obtain an updated state vector and covariance matrix;

[0127] S3-3-2-3, determining whether the speed parameter in the updated state vector is within a preset normal speed threshold range, if yes, outputting the updated state vector as the real-time bird motion state change result; if no, executing S3-3-2-4;

[0128] S3-3-2-4, determining whether the acceleration parameter in the updated state vector exceeds a preset normal acceleration threshold range, if yes, outputting the updated state vector as the real-time bird motion state change result; if no, marking the bird target as being in an abnormal motion state and returning to S3-3-2-2;

[0129] An adaptive process noise covariance matrix Q_t is designed to dynamically adjust the target motion state: wherein λ_t is an adaptive factor; and the uncertainty of the target motion is dynamically adjusted according to: wherein v_t represents the target speed, and α and β are adjustment parameters;

[0130] An adaptive measurement noise covariance matrix R_t is designed to dynamically adjust the reliability of the detection result: wherein σ_i^2 is inversely proportional to the detection confidence score_t (σ_i^2 = σ_base^2 / score_t); R_t represents the covariance matrix at time t, σ_x², σ_y², σ_w², and σ_h² represent the variances in the x, y, width, and height directions; and σ_base^2 is a basic variance.

[0131] Kalman filter: design multiple motion models (uniform model, uniform acceleration model, random walk model, etc.), and dynamically switch according to model likelihood:

[0132] where p(M_i|Z_{1:t}) represents the posterior probability of model M_i given the observation sequence Z_{1:t};

[0133] S3-3-3, the real-time bird motion state change result is processed by multiple hypothesis tracking, and a real-time bird tracking result is obtained.

[0134] The multiple hypothesis tracking processing is to maintain multiple possible tracking hypotheses, and the optimal hypothesis is selected through a hypothesis scoring mechanism, and the hypothesis scoring mechanism is: where H_i represents the i-th hypothesis, f_j represents the j-th scoring function (such as motion consistency, appearance similarity, etc.), and w_j is the weight.

[0135] Through the prediction trajectory and the appearance model, the target occlusion and temporary disappearance situation is processed:

[0136] It is judged whether it is short-term occlusion, if yes, the predicted state is continuously used for tracking; if not, the tracking is suspended, the target ID and the appearance model are reserved until the reappearance, and if the reappearance, the target ID is recovered through the appearance matching.

[0137] Further reference Figure 2 , as an implementation of the above method, the present application provides an embodiment of a real-time bird tracking device, which corresponds to the method embodiment shown in Figure 1 , and the device can be applied to various electronic devices.

[0138] As shown in Figure 2 , the real-time bird tracking device of the embodiment comprises a data acquisition module, a detection and identification module, and a multi-target tracking module.

[0139] The data acquisition module is configured to acquire real-time bird multi-modal data, and the real-time bird multi-modal data comprises video data, audio data, and environmental data.

[0140] The detection and identification module is configured to detect and identify the real-time bird multi-modal data based on the real-time bird multi-modal data, and obtain real-time bird detection and identification results by using a bird target detection model.

[0141] The multi-target tracking module is configured to track multiple targets based on the real-time bird detection and identification results and the real-time bird multi-modal data, and obtain real-time bird tracking results by using an adaptive Kalman filter algorithm.

[0142] In some optional embodiments, the detection and recognition module comprises:

[0143] a preprocessing unit configured to preprocess the real-time bird multi-modal data to obtain real-time bird multi-modal preprocessed data.

[0144] The preprocessing mainly includes denoising, stabilization, resolution adjustment, color correction, and the like for the video data; noise reduction, spectrum analysis, feature extraction, and the like for the audio data; filtering, normalization, and the like for the environmental data.

[0145] a detection and recognition unit configured to perform detection and recognition based on the real-time bird multi-modal data or the real-time bird multi-modal preprocessed data by using a bird target detection model to obtain real-time bird detection and recognition results.

[0146] In this embodiment, the specific processing of the real-time bird tracking apparatus and the technical effects brought by the specific processing can be respectively referred to the related descriptions of steps S1, S2, S3, and S4 in the corresponding embodiments, which will not be repeated here. Figure 1 The related descriptions of steps S1, S2, S3, and S4 in the corresponding embodiments will not be repeated here.

[0147] The embodiment further provides an electronic device, including: one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the real-time bird tracking method provided in the embodiment.

[0148] The embodiment further provides a computer-readable storage medium, and the computer-readable storage medium has a computer program stored thereon, when the computer program is executed by one or more processors, the real-time bird tracking method provided in the embodiment is implemented.

[0149] Those skilled in the art should understand that embodiments of the present application can be provided as a method, device, or computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0150] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0151] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0152] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0153] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, but are not intended to limit the present application. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced, and any modification or replacement without departing from the spirit and scope of the present application should be covered within the protection scope of the claims of the present application.

Claims

1. A real-time bird tracking method, characterized in that: include: Collecting real-time bird multimodal data, wherein the real-time bird multimodal data includes video data, audio data, and environmental data; Based on the real-time bird multimodal data, a bird target detection model is used to detect and identify the bird, thereby obtaining a real-time bird detection and identification result; Based on the real-time bird detection and recognition results and the real-time bird multimodal data, an adaptive Kalman filter algorithm is used to perform multi-target tracking to obtain a real-time bird tracking result.

2. The method according to claim 1, characterized in that The training process of the bird target detection model is as follows: A student model is constructed based on MobileNetV3 as the basic network and combined with a feature pyramid network. The student model has a parameter count of less than 10M. A large convolutional neural network model is used as a teacher model, wherein the teacher model is a pre-trained model and has a parameter count greater than 10M. The teacher model is used to supervise the student model to perform knowledge distillation training to obtain a bird target detection model.

3. The method according to claim 2, characterized in that The real-time bird detection and recognition results obtained by using a bird target detection model for detection and recognition based on the real-time bird multimodal data include: Based on the real-time bird multimodal data, using a bird target detection model, obtaining corresponding real-time bird species characteristic data; Classifying the real-time bird species characteristic data to obtain corresponding real-time bird classification data, and establishing a bird species characteristic template library; Using a channel attention mechanism to perform channel weighting on the real-time bird classification data to obtain key features for real-time bird detection; Determining whether the key features of the real-time bird detection are consistent with the real-time bird multimodal data, and if so, performing a first operation; otherwise, obtaining data corresponding to the key features of the real-time bird detection as updated real-time bird multimodal data, and re-acquiring real-time bird species characteristic data; The first operation is to determine whether the key features of the real-time bird detection match the bird species feature template library; if so, output the bird species classification label corresponding to the key features of the real-time bird detection as the real-time bird detection and recognition result, and perform feature weighting on the key feature data of the real-time bird detection through spatial attention; if not, adjust the classification parameters and reclassify the real-time bird species feature data; Performing feature weighting on the key feature data of the real-time bird detection through spatial attention to obtain spatial features of the real-time bird detection; Determine whether the spatial features of the real-time bird detection conform to the bird morphological features. If so, obtain the bird target bounding box position information corresponding to the spatial features of the real-time bird detection and perform the second operation; if not, mark it as an invalid target and reuse the channel attention mechanism for channel weighting; The second operation is to determine whether the position information of the bird target bounding box corresponds to a clear and complete contour boundary. If so, output the corresponding bird target bounding box coordinates as the real-time bird detection and recognition result; if not, return to reuse spatial attention for feature weighting.

4. The method according to claim 1, wherein The method of performing multi-target tracking using an adaptive Kalman filter algorithm based on the real-time bird detection and recognition results and the real-time bird multimodal data to obtain real-time bird tracking results includes: performing feature extraction on the real-time bird detection and recognition results and the real-time bird multimodal data to obtain multimodal features; Performing multi-level dynamic fusion on the multimodal features to obtain corresponding multimodal fusion features; Based on the multimodal fusion features, an adaptive Kalman filter algorithm is used to perform multi-target tracking to obtain real-time bird tracking results; The multimodal features include appearance features, movement features, context features and audio features of the bird.

5. The method according to claim 4, characterized in that The multimodal features are subjected to multi-level dynamic fusion to obtain corresponding multimodal fusion features including: The multimodal features are directly connected and fused for the first time to form a multi-dimensional initial fusion feature; Using the channel attention and spatial attention mechanisms, the initial fusion features are subjected to a second weighted fusion to generate weighted fusion features; Using an adaptive selection mechanism, the weights of the initial fusion features are dynamically adjusted to obtain dynamic fusion features; The initial fusion features, weighted fusion features and dynamic fusion features are integrated by using ensemble learning to obtain multimodal fusion features.

6. The method according to claim 5, characterized in that The adaptive selection mechanism is used to dynamically adjust the weights of the initial fusion features to obtain dynamic fusion features including: Obtaining corresponding motion state data based on the motion features in the initial fusion features, determining whether the motion state data corresponds to a high-speed motion state, and if so, adjusting the motion feature weights in the initial fusion features to obtain corresponding dynamic fusion features; if not, directly using the initial fusion features as dynamic fusion features; Determine whether the appearance features in the initial fusion features are consistent with the context features. If so, obtain the corresponding ambient light intensity data based on the environmental features in the context features. If not, obtain the environmental features that are consistent with the appearance features as the updated context features, and return to the first fusion stage to re-form the multi-dimensional initial fusion features. Determine whether the ambient light intensity data exceeds a normal ambient light threshold range; if so, adjust the color weights in the initial fusion features to obtain corresponding dynamic fusion features; if not, use the initial fusion features directly as dynamic fusion features; The illumination value of the normal ambient illumination threshold range is 100-1000 lux; when the illumination value is ≤100 lux, it is determined to be a low illumination condition; and when the illumination value is ≥1000 lux, it is determined to be a high illumination condition.

7. The method according to claim 4, characterized in that Based on the multimodal fusion features, an adaptive Kalman filter algorithm is used to perform multi-target tracking, and the real-time bird tracking results include: Acquire a state vector of a real-time bird target based on the multimodal fusion feature and construct a state space model; Obtain the corresponding predicted state vector and covariance matrix according to the state space model; Using the adaptive process noise covariance matrix and the measurement noise covariance matrix, based on the Kalman filter algorithm, the predicted state vector and the covariance matrix are adjusted to obtain an updated state vector and covariance matrix; Determine whether the speed parameter in the updated state vector is within the preset normal speed threshold range. If so, output the updated state vector as the real-time bird motion state change result; if not, obtain the corresponding acceleration parameter for judgment; The acquisition of the corresponding acceleration parameter for determination is as follows: determining whether the acceleration parameter in the updated state vector exceeds a preset normal acceleration threshold range; if so, outputting the updated state vector as a real-time bird motion state change result; if not, marking the bird target as being in an abnormal motion state, and returning to re-use the adaptive Kalman filter algorithm for adjustment; Multi-hypothesis tracking processing is performed on the real-time bird motion state change result to obtain a real-time bird tracking result.

8. A real-time bird tracking device, characterized in that: include: A data acquisition module, configured to acquire real-time bird multimodal data, wherein the real-time bird multimodal data includes video data, audio data, and environmental data; A detection and recognition module is used to detect and recognize birds based on the real-time bird multimodal data using a bird target detection model to obtain real-time bird detection and recognition results; The multi-target tracking module is used to perform multi-target tracking based on the real-time bird detection and recognition results and the real-time bird multimodal data using an adaptive Kalman filter algorithm to obtain real-time bird tracking results.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, wherein when the computer program is executed by one or more processors, the method according to any one of claims 1 to 7 is implemented.

Images