A method, medium and system for predicting the movement trajectory of a distant bird target

Abstract

The application discloses a long-distance bird target motion trajectory prediction method, medium and system, comprising: inputting each frame of video image containing a bird target collected by a left camera in a binocular camera into a motion target detection model respectively, outputting a detection image labeled with a bird target recognition box, the motion target detection model comprising: a parallel depth residual network and an optical flow neural network; inputting each frame of binocular image containing a bird target collected by the binocular camera into a Transformer-based aggregation network, outputting a depth map with a depth value of each pixel point, the binocular image comprising: a left view collected by the left camera and a right view collected by a right camera; based on multiple sets of detection images and depth maps, using an LSTM neural network trained based on a Kalman filter to predict the motion trajectory of the bird target, wherein each set of detection image and depth image is an image collected at the same time. The application can accurately predict the position of the bird target.

Description

Technical Field

[0001] This invention relates to the field of power safety technology, and in particular to a method, medium, and system for predicting and identifying the movement trajectory of distant bird targets. Background Technology

[0002] In recent years, with the continuous improvement of the ecosystem, the natural activities of birds have become increasingly frequent. Nesting, hunting, and other natural activities near substations can cause varying degrees of damage to electrical equipment and power lines. Birds such as magpies and swallows landing on substation equipment may accidentally trigger short circuits and trip the circuit breaker. Therefore, preventing bird intrusion is of paramount importance in the operation and maintenance management of power grid substations. How to efficiently, intelligently, and stably drive away intruding animals without harming them is a crucial problem that urgently needs to be solved.

[0003] Existing bird deterrent devices include wind-powered deterrents, ultrasonic deterrents, and laser deterrents. Wind-powered deterrents involve mounting a windmill on the device and placing it in the area where intruding animals need to be deterred. The wind power drives the windmill to rotate, thus repelling the animals. However, when there is no wind, the windmill deterrent will not work, and its effectiveness will be significantly reduced. Ultrasonic deterrents utilize the sensitive hearing of animals, emitting high-frequency ultrasonic waves beyond the range of human hearing to repel intruding animals. However, prolonged use of a fixed frequency will cause animals to adapt, leading to a decrease in deterrent effectiveness, and it is also less economical, making it unsuitable for large-scale deployment. Furthermore, prolonged exposure to ultrasonic environments may cause hearing damage to workers. Laser deterrents utilize green lasers that animals are sensitive to, mechanically scanning various areas of the substation to repel intruding animals. After analyzing and comparing various deterrent methods and devices, laser deterrents are environmentally friendly and have a better deterrent effect, gradually becoming one of the current research hotspots. However, existing laser deterrence devices rely on mechanical, regular scanning, which can cause birds to adapt and reduce their deterrence effectiveness. They also lack spatial positioning and the ability to distinguish between animal and human targets, resulting in low deterrence accuracy and the risk of accidentally injuring human eyes. Summary of the Invention

[0004] This invention provides a method, medium, and system for predicting the movement trajectory of distant bird targets, in order to solve the problem that existing technologies cannot accurately locate bird targets, resulting in poor effectiveness in driving away bird targets.

[0005] Firstly, a method for predicting the trajectory of distant bird targets is provided, including:

[0006] Each frame of video image containing bird targets, captured by the left eye camera in a binocular camera, is input into the moving target detection model, and a detection map with bird target bounding boxes is output. The moving target detection model includes a deep residual network and an optical flow neural network in parallel.

[0007] Each frame of binocular image containing bird targets, acquired by a binocular camera, is input into a Transformer-based aggregation network, which outputs a depth map with depth values ​​for each pixel. The binocular image includes a left view acquired by the left camera and a right view acquired by the right camera.

[0008] Based on multiple sets of detection maps and depth maps, an LSTM neural network trained with a Kalman filter is used to predict the trajectory of bird targets. Each set of detection maps and depth maps consists of images acquired at the same time.

[0009] In a second aspect, a computer-readable storage medium is provided, wherein computer program instructions are stored thereon; when executed by a processor, the computer program instructions implement the motion trajectory prediction method for distant bird targets as described in the first aspect embodiment.

[0010] Thirdly, a system for predicting the trajectory of distant bird targets is provided, comprising: a computer-readable storage medium as described in the second aspect embodiment.

[0011] Thus, in this embodiment of the invention, the motion target detection algorithm based on the collaboration of salient semantics and optical flow information can more effectively detect moving targets and is not affected by background motion, with relatively high detection accuracy; the use of binocular vision technology for spatial localization of bird targets effectively solves the problem of inaccurate spatial localization of target objects caused by parallax blur at long distances; and the LSTM neural network trained based on Kalman filter can accurately predict the motion pattern of the target. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a method for predicting the trajectory of distant bird targets according to an embodiment of the present invention;

[0014] Figure 2 This is a flowchart of a long-distance bird target repelling method according to an embodiment of the present invention;

[0015] Figure 3 This is a flowchart of step S101 of the present invention, which is a moving target detection algorithm based on the coordination of significant semantics and optical flow information.

[0016] Figure 4 This is a flowchart of step S102 of the embodiment of the present invention, which is a stereo matching algorithm based on deep learning.

[0017] Figure 5 This is a diagram of the three-dimensional coordinate prediction network structure based on LSTM and Kalman filter in step S103 of this embodiment of the invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This invention discloses a method for predicting the trajectory of distant bird targets. For example... Figure 1 As shown, the method of this embodiment of the invention specifically includes the following steps:

[0020] Step S101: Input each frame of video image containing bird targets acquired by the left eye camera in the binocular camera into the moving target detection model, and output a detection map with bird target recognition boxes.

[0021] Specifically, a binocular camera should be able to cover the target area.

[0022] Specifically, the moving target detection models include: parallel deep residual networks and optical flow neural networks.

[0023] The deep residual network is an existing network. The deep residual network in this embodiment of the invention is the ResNet-101 network. The ResNet-101 deep residual network includes: a first stacked layer structure, a second stacked layer structure, a third stacked layer structure, a fourth stacked layer structure, and a fifth stacked layer structure connected in sequence. The first stacked layer structure includes a 7×7 convolutional layer with 64 convolutional kernels, a stride of 2, a padding attribute of 3, and a ReLU activation function. The second stacked layer structure consists of a 3×3 max-pooling layer connected in sequence, a stride of 2, and three residual modules. The third stacked layer structure consists of four residual modules connected in sequence. The fourth stacked layer structure consists of twenty-three residual modules connected in sequence, and the fifth stacked layer structure consists of three residual modules connected in sequence. For each residual module, a three-layer convolutional network is used: 1×1 (dimensionality reduction), 3×3 (convolutional block with the smallest input-output dimension), and 1×1 (dimensionality increase convolution). When the input is x, the learned feature is denoted as H(x), and the residual obtained after learning is F(x) = H(x) - x. The original learned feature is:

[0024] F(x)+x.

[0025] y=F(x,[W i ])+x.

[0026] F = w1 * x.

[0027] x represents the input vector of the current layer, y represents the output vector of the current layer, and the formula F(x,[W]) is given by... i The residual mapping is represented by ]). When the number of channels x and F are equal, x and F(x) are added element-wise. If they are not equal, a 1×1 convolution is first used for projection mapping, and then element-wise addition is performed. The summation result is then applied again with the ReLU nonlinear activation function. At this point, it can be observed that for any feature x at depth L... L It is the sum of all the preceding residual modules.

[0028] For deep residual networks, to overcome the imbalance in pixel count between the foreground and background regions, a weighted pixel-wise cross-entropy loss with a softmax function E is used during optimization. The loss function is defined as follows:

[0029]

[0030] in, Let f be the loss function of the deep residual network, and let i and j be the foreground and background, respectively. g and background b g The pixel position, y ij For the video image X in frame t t The binary prediction of each pixel, where w is the foreground-to-background pixel ratio, E is the softmax function, and θ is the probability vector of the binary prediction.

[0031] The optical flow neural network FlowNetS is an existing network. FlowNetS is chosen as the backbone for optical flow estimation. Two consecutive frames of a video image are used as input to FlowNetS, and the two input images are stacked together for processing. An encoder-decoder architecture is chosen as the optical flow branch, with jump connections used for feature connections between the encoder and decoder. The encoder consists of 9 layers of convolutional layers with ReLU activation functions and a stride of 2, without any fully connected layers, allowing the network to take images of arbitrary sizes as input. The convolutional filter size decreases with deeper network layers: 7×7 for the first layer, 5×5 for the next two layers, and 3×3 from the fourth layer onwards. The number of feature maps increases in deeper layers, approximately doubling after each layer, with a stride of 2. The main component of the decoder is the "upconvolutional" layer, consisting of unpooling and convolution. To perform refinement, the "upconvolution" is applied to the feature maps and connected to the corresponding feature maps from the "shrinkage" part of the network and the upsampled coarser flow predictions. In this way, both the high-level information passed from the coarser feature maps and the fine local information provided by the lower-level feature maps are preserved. The resolution is increased twice at each step and repeated four times to obtain a prediction stream.

[0032] To optimize the network, the loss function of the optical flow neural network... Endpoint error loss (EPE) is used and is defined as follows:

[0033]

[0034] Among them, u ij v ij These are video images X from frame t, respectively. t up to the (t+1)th frame of video image X t+1 The horizontal and vertical components of the true optical flow field at pixel (i,j). and For the video image X predicted by the optical flow neural network in frame t t up to the (t+1)th frame of video image X t+1 The horizontal and vertical components of the optical flow field at pixel (i,j).

[0035] Based on the loss functions of the two networks mentioned above, a loss function for the moving target detection model is designed. as follows:

[0036]

[0037] Where λ is the weighting coefficient.

[0038] Based on the loss function described above, ResNet-101 and FlowNetS were used to train the two branches respectively. Each time, the network focused on one task on one branch, while obtaining useful representations from the other branch through feature propagation. After switching to the other branch, the better features learned from the previous stage were used in the currently optimized branch. By tightly combining salient semantic and optical flow information, the approximate region of the invading animal target was obtained.

[0039] Step S102: Input each frame of binocular image containing bird targets acquired by the binocular camera into the Transformer-based aggregation network, and output a depth map with the depth value of each pixel.

[0040] This step includes the following specific procedures:

[0041] Step 1: Pre-calibrate the binocular camera.

[0042] The calibration in this embodiment of the invention uses Zhang Zhengyou's planar template calibration method to obtain binocular camera parameters. This method is prior art, and the specific process is as follows:

[0043] 1. Solve the linear equations using homography conditions to obtain the initial values ​​of the camera's intrinsic and extrinsic parameters.

[0044] The projection of point M(X,Y,Z) on the chessboard template onto the image pixel coordinate system is m(u,v), and the two satisfy a linear transformation relationship: sm=A[RT]M.

[0045] Where s is the scaling factor, R and T are the rotation matrix and translation vector, and A is the camera's intrinsic parameter matrix, it can be represented as:

[0046]

[0047] Where α and β are the scaling factors f x and f y γ is the offset caused by the incomplete perpendicularity between the coordinate axes. Assuming the chessboard template is located on the world coordinate system Z=0, it can be represented as:

[0048]

[0049] Where, r i =[r i1 r i2 r i3 ] T Let represent the i-th column of the rotation matrix R.

[0050] Given a chessboard image, we can obtain a homography matrix H, let H = [h1 h2 h3], h i =[hi1 h i2 h i3 ] T If H is the i-th column, then we can obtain:

[0051] H=[h1 h2 h3]=λA[r1 r2 t].

[0052] Where λ is the proportionality coefficient, and r1 and r2 are unit orthogonal vectors that are mutually orthogonal, by and The following constraints can be obtained:

[0053]

[0054] make:

[0055]

[0056] It can be seen that matrix B is a symmetric matrix, and a 6-dimensional vector b is defined as [B 11 B 12 B 22 B 13 B 23 B 33 ] T It satisfies the following relationship:

[0057]

[0058] Among them, v ij =[h i1 h j1 h i1 h j2 +h i2 h j1 h i2 h j2 h i3 h j1 +h i1 h j3 h i3 h j2 +h i2 h j3 h i3 h j3 ] T .

[0059] If we obtain n images containing the chessboard template, we can obtain n homography matrices, n systems of equations, and thus:

[0060] Vb = 0.

[0061] If n≥3, a unique solution b is obtained. Based on b, matrix B is calculated, and all camera intrinsic parameters are decomposed to determine matrix A. Based on A, the rotation matrix R and translation vector T are calculated:

[0062]

[0063] Where λ=1 / ||A -1 h1||=1 / ||A -1 h2||.

[0064] 2. Nonlinear optimization is achieved by using maximum likelihood estimation for the initial values.

[0065] Due to factors such as noise and distortion, the obtained camera parameters contain errors, requiring nonlinear optimization of the parameter values ​​through maximum likelihood estimation. A chessboard template has m corner points, and n images containing the chessboard template in different orientations are captured. The corner points in each image are independent. Maximum likelihood estimation is used to obtain more accurate intrinsic and extrinsic camera parameters.

[0066]

[0067] Where, m ij R is the image point coordinate of the j-th corner point in the i-th image. i and T i These are the rotation matrix and translation vector of the i-th image relative to the world coordinate system, M and M, respectively. j These are the three-dimensional coordinates of the j-th corner point. These are the estimated pixel coordinates.

[0068] The maximum likelihood estimates A and R are obtained by minimizing the objective function using the nonlinear optimization algorithm (Lavenberg-Marquardt, LM). i and T i That is the optimal camera parameter value.

[0069] 3. Considering nonlinear distortion factors, further optimize the camera parameters.

[0070] In real-world imaging processes, camera lenses exhibit varying degrees of nonlinear distortion, with radial distortion having the most significant impact. Assuming the camera's distortion coefficients are the same along the x and y axes, the radial distortion model is as follows:

[0071]

[0072] Where (x,y) represents the ideal coordinates. The actual coordinates are represented by k1 and k2, which represent the first and second order radial distortion factors, respectively. The radial distortion model in the image pixel coordinate system is as follows:

[0073]

[0074] Where (u,v) represents the ideal coordinates. Indicates the actual coordinates.

[0075] The first-order and second-order radial distortion factors are as follows:

[0076]

[0077] Given n images containing a chessboard template, we obtain n systems of equations, as follows:

[0078] Dk = d.

[0079] Where, k = [k1 k2] T The distortion factor is solved using the least squares method:

[0080] k=(D T D) -1 D T d.

[0081] After obtaining the radial distortion parameters of the camera, the parameter values ​​are then optimized using maximum likelihood estimation, as follows:

[0082]

[0083] Among them, A and R i T i k1 and k2 are obtained through the above steps. These are the estimated pixel coordinates. The objective function is minimized using the nonlinear optimization algorithm LM, i.e. The obtained maximum likelihood estimates A and R i T i k1 and k2 are the optimal camera parameter values.

[0084] 4. After calibrating the monocular camera, the external parameters between the two cameras, including the rotation matrix and translation vector, can be calculated using the external parameters of the left and right cameras.

[0085] R1 and T1 are the rotation matrix and translation vector of the left camera relative to the world coordinate system, and R2 and T2 are the rotation matrix and translation vector of the right camera relative to the world coordinate system. Therefore, the rotation matrix R and translation vector T of the left camera relative to the right camera are:

[0086]

[0087] Step 2: Extract features from the left and right views containing bird targets captured by the calibrated binocular camera using a convolutional neural network, transforming them into left and right view feature maps at three resolutions respectively.

[0088] Specifically, this step uses three convolutional networks. All three convolutional networks have a kernel size of 3 and a stride of 2.

[0089] For example, inputting a 448*448 image F into a first convolutional neural network for feature extraction yields a feature map with half the original resolution. Inputting this half-resolution feature map into a second convolutional neural network for feature extraction yields a feature map with a quarter of the original resolution. Inputting this quarter-resolution feature map into a third convolutional neural network for feature extraction yields a feature map with an eighth of the original resolution. This results in feature maps with three different resolutions.

[0090] Step 3: Flatten or reshape each left-view feature map and right-view feature map into a sequence.

[0091] This step is existing technology and will not be described in detail here.

[0092] Step 4: After adding positional encoding to each sequence, input each sequence into the Transformer network for encoding, and output the encoded feature image corresponding to each sequence.

[0093] By adding positional encoding, the Transformer network is able to understand the positional relationships between pixels, and the shape of the returned value should be the same as the shape of the input sequence.

[0094] The Transformer network obtains encoded feature maps for each resolution. Encoding each sequence returns a new sequence whose elements encode the relationships between the original elements and other elements. Through the Transformer's multi-attention mechanism, the relationships between different parts of the image can be captured. This helps in understanding the semantic content of the image, such as the local and overall relationships of objects, and the foreground and background.

[0095] Step 5: Calculate the matching cost of the encoded feature images corresponding to the two sequences of the left and right views at each resolution using the Correlation operation.

[0096] Specifically, multi-scale matching costs (Cost Volumes) are constructed for the features of the left and right images at three different resolutions, obtaining the matching cost at each pixel location, representing the degree of matching between corresponding points in the two images under different disparity shifts. For each scale *s*, the formula for calculating the matching cost is as follows:

[0097]

[0098] Where, C s (d,h,w) represents the matching cost of disparity candidate d at position (h,w). Let (h,w) represent the feature vectors at positions (h,w) in the left and right views, respectively, where (i,j) are the pixel coordinates in the feature block, <·,·> denotes the calculation of the inner product, and N is the number of feature extraction channels.

[0099] Step 6: Aggregate the matching costs of the left and right views at three resolutions using an adaptive aggregation network to output a feature image with the location information of each pixel.

[0100] Specifically, the adaptive aggregation network includes sequentially connected same-scale aggregation modules and cross-scale aggregation modules.

[0101] The same-scale aggregation module comprises a first convolutional layer, a second convolutional layer, a third convolutional layer, and a residual module connected in sequence. The first convolutional layer has a 1×1 kernel, the second convolutional layer has a 3×3 kernel, and the third convolutional layer has a 1×1 kernel. Furthermore, the number of input and output channels is the same to ensure that the number of disparity candidates remains constant.

[0102] Same-scale aggregation refers to aggregating the cost of matching only at the same resolution. It uses sparse point-based feature representation for cost aggregation and borrows deformable convolution to improve the cost aggregation. The formula is as follows:

[0103]

[0104] in, K is the aggregation cost of the pixels of the disparity candidate d at point p. 2 It is the number of sampling points, K=3, ω k It is the aggregate weight of the k-th point, p k It is a fixed offset of the pixel, Δp k It is an additional regularization bias for learning, m k This refers to the adjustment mechanism of deformable convolution, namely the weight values.

[0105] The cross-scale aggregation module is a known module in existing technology and will not be described in detail here.

[0106] The cross-scale aggregation module is used to process large, textureless regions. The specific formula is as follows:

[0107]

[0108] in, It is the aggregation cost after cross-scale aggregation. (Right now ) is the aggregation cost after aggregation by the same-scale aggregation module at the k-th scale, f k It is a general functional representation that enables the aggregation cost to be adaptively aggregated at multiple scales, and f is calculated using a human pose estimation network. k The details are as follows:

[0109]

[0110] Where I represents the identity mapping function, and (sk)stride-23×3convs is used to sum with 2 s-k The downsampling resolution remains consistent, while This means that the bilinear upsampling is performed to the same resolution, and then a 1×1 convolution is used to align the feature channels.

[0111] Step 7: Use the softmax function to weight the aggregation matching cost to obtain the probability distribution of each disparity candidate. Then, perform a weighted average of the probability distributions of the disparity candidates to obtain the predicted disparity value.

[0112] Specifically, the formula for calculating disparity is as follows:

[0113]

[0114] Where Disparity(x,y) is the disparity value of pixel (x,y), D max Let C be the maximum disparity range, σ be the softmax function, and C be the maximum disparity range. d To examine the aggregation matching cost of candidate d, i.e.

[0115] Step 8: Use the softargmin function to convert the disparity values ​​into depth values ​​to obtain a depth map with depth values ​​for each pixel.

[0116] Specifically, the softargmin function is as follows:

[0117] Depth(x,y)=f_baseline / Disparity(x,y).

[0118] Where Depth(x,y) is the depth value of pixel (x,y), and f_baseline is the baseline distance of the stereo camera, representing the distance between the two cameras.

[0119] Step S103: Based on multiple sets of detection maps and depth maps, use an LSTM neural network trained with a Kalman filter to predict the trajectory of the bird target.

[0120] Each set of detection maps and depth maps consists of images acquired at the same time.

[0121] Specifically, this step includes the following process:

[0122] Step 1: For each frame of the detection map with bird target bounding boxes, calculate the pixel coordinates of the center point of the bird target bounding box.

[0123] Specifically, the formula for calculating the center point pixel coordinates is as follows:

[0124]

[0125] Where, x left y top To identify the coordinates of the top-left corner of the bounding box, x right y bottom The coordinates of the bottom right corner of the identification box.

[0126] Step 2: Fuse the pixel coordinates of the center point of the bird target recognition box with the depth value of the center point in the depth map acquired at the same time to obtain the three-dimensional coordinates of the center point.

[0127] The three-dimensional coordinates of the center point are (x, y, z), where x is the horizontal pixel, y is the vertical pixel, and z is the depth value.

[0128] Step 3: After sorting the 3D coordinates of the center point of each frame in ascending order of horizontal pixels, use an interpolation function to interpolate the vertical pixels and depth values ​​to obtain new vertical pixels and depth values.

[0129] Specifically, the interpolation algorithm is as follows:

[0130]

[0131] Where new_y is the interpolated y-axis coordinate (vertical pixel), new_z is the interpolated z-axis coordinate (depth value), and interp_func() is the interpolation function.

[0132] Step 4: Combine the existing 3D coordinates of the center point with the new 3D coordinates obtained through interpolation into a single 3D coordinate set.

[0133] This set is denoted as interpolated_data.

[0134] Step 5: Input the three-dimensional coordinate set into the LSTM neural network trained based on the Kalman filter, so that the LSTM neural network predicts the three-dimensional coordinates of the bird target at the next time step corresponding to the three-dimensional coordinate set, and outputs the three-dimensional coordinates of the bird target at the next time step.

[0135] The three-dimensional coordinates of the bird target at the next moment are (Xlaser, Ylaser, Zlaser).

[0136] Specifically, the input layer has a shape of (i, 3), representing two time steps (from 0 to i seconds), with three features (3D coordinate data) at each time step. Design a neural network whose input is the 3D coordinate data from 0 to i seconds, and whose output is the set of position coordinates at the (i+1)th second. Use the ReLU activation function and set `return_sequences = True` to output the entire sequence. The output layer is a fully connected layer with a dimension of 3, corresponding to the position coordinates at the (i+1)th second.

[0137] Specifically, before actual detection, an LSTM neural network is pre-trained based on a Kalman filter. This training process includes:

[0138] 1. The pixel coordinates of the center point of the bird target recognition box in the detection map of the first frame of the video in the training set are used as the initial state vector of the coordinates.

[0139] The pixel coordinates of the center point k0 are α = [x0, y0, z0].

[0140] 2. Initialize the covariance matrix of the Kalman filter.

[0141] The covariance matrix of a Kalman filter is typically a diagonal matrix, where the elements on the diagonal represent the variance of the corresponding state. The covariance matrix for the initial state vector [x0, y0, z0] is as follows:

[0142]

[0143] The diagonal elements are the initial variances of x0, y0, and z0, respectively.

[0144] 3. After iteratively updating the Kalman filter using the training set, output the first coordinates of the bird target at the next moment corresponding to the video in the training set.

[0145] Specifically, this step includes the following process:

[0146] (1) For each time step, state prediction is performed based on the state transition equation.

[0147] The state transition equation is: α t =A·α t-1 α t This is a prediction of the state at time step t when there is no observation data. Here is the state transition matrix, and Δt is the time interval between two time steps.

[0148] (2) Update the covariance matrix.

[0149] The update equation for the covariance matrix is:

[0150] P t =A·P t-1 ·A T .

[0151] Among them, A T P is the transpose of the state transition matrix. t-1 P t These are the covariance matrices before and after the update;

[0152] (3) Repeat the above steps until the training set has been traversed.

[0153] 4. Input the video from the training set into the LSTM neural network, so that the LSTM neural network outputs the second coordinate of the bird target at the next moment corresponding to the video.

[0154] 5. Calculate the average of the first and second coordinates to obtain the third coordinate.

[0155] Specifically, the calculation formula is as follows:

[0156]

[0157] in, The third coordinate is the output coordinate of both models. As the first coordinate, This is the second coordinate.

[0158] 6. Using the third coordinate and the ground truth coordinates of the bird target at the next moment corresponding to the video, calculate the value of the loss function of the LSTM neural network.

[0159] The loss function used is the root mean square error loss function, as follows:

[0160]

[0161] Where N is the number of samples, k i+1 This represents the true coordinates of the bird target at the next moment in the video, i.e., the actual center coordinates at second i.

[0162] 7. After calculating the gradient of the loss function of the LSTM neural network using the backpropagation algorithm, the weights of the LSTM neural network are updated using the stochastic gradient descent algorithm to complete the training.

[0163] Updating using the stochastic gradient descent algorithm can reduce the loss.

[0164] Furthermore, after step S103, the method of this embodiment of the invention further includes:

[0165] Step S104: Based on the movement trajectory of the bird target, fire a laser at the location of the bird target's movement trajectory to drive it away.

[0166] Specifically, based on the three-dimensional coordinates of the bird's position on its trajectory and the position information of the laser device, the direction vector of the laser device pointing to the position on the trajectory is calculated. The laser is then adjusted to aim at that position on the trajectory based on the direction vector to ensure that the laser beam is accurately aimed at the bird and driven away.

[0167] Specifically, the laser device can cover the target area. Before performing calculations, the laser device can be calibrated to determine its position coordinates, represented in Cartesian coordinates as (Xlaser, Ylaser, Zlaser), where Xlaser, Ylaser, and Zlaser represent the position of the laser device in the horizontal, vertical, and depth directions, respectively.

[0168] The location of the bird target is represented by Cartesian coordinates (X,Y,Z), where X, Y, and Z represent the target's position in the horizontal, vertical, and depth directions, respectively.

[0169] The formula for calculating the direction vector is as follows:

[0170]

[0171] Among them, the laser deterrent device utilizes a green laser that animals are sensitive to.

[0172] Through simulation, the location prediction accuracy of the example of this invention is 95.68%, the prediction accuracy of using a deep residual network alone is 90.67%, the prediction accuracy of using an optical flow neural network alone is 80.32%, and the prediction accuracy of using an LSTM network alone is 88.35%, indicating that the technical solution can improve the prediction accuracy.

[0173] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing computer program instructions; when the computer program instructions are executed by a processor, they implement the method for predicting the location of distant bird targets as described in the above embodiments.

[0174] Furthermore, embodiments of the present invention also provide a location prediction system for distant bird targets, comprising: a computer-readable storage medium as described in the above embodiments.

[0175] In summary, the embodiments of this invention employ a moving target detection algorithm based on the synergy of salient semantics and optical flow information, which can more effectively detect moving targets and is unaffected by background motion, resulting in relatively high detection accuracy. The use of binocular vision technology for spatial localization of bird targets effectively solves the problem of inaccurate spatial localization of targets caused by parallax blur at long distances. An LSTM neural network trained based on a Kalman filter can accurately predict the movement patterns of targets. A laser deterrent device utilizes a green laser that animals are sensitive to to mechanically scan various areas of the substation, achieving the purpose of repelling intruding animals.

[0176] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for predicting the trajectory of distant bird targets, characterized in that, include: Each frame of video image containing bird targets, captured by the left eye camera in a binocular camera, is input into the moving target detection model, and a detection map with bird target bounding boxes is output. The moving target detection model includes a deep residual network and an optical flow neural network in parallel. Each frame of binocular image containing bird targets, acquired by a binocular camera, is input into a Transformer-based aggregation network, which outputs a depth map with depth values ​​for each pixel. The binocular image includes a left view acquired by the left camera and a right view acquired by the right camera. Based on multiple sets of detection maps and depth maps, an LSTM neural network trained on a Kalman filter is used to predict the trajectory of bird targets. Each set of detection maps and depth maps consists of images acquired at the same time. The step of outputting a depth map having depth values ​​for each pixel includes: The binocular camera is pre-calibrated; The left and right views containing bird targets, acquired by the calibrated binocular camera, are processed by convolutional neural networks to extract features, and transformed into left and right view feature maps at three resolutions respectively. Flatten or reshape each left-view feature map and right-view feature map into a sequence; After adding positional encoding to each sequence, each sequence is input into the Transformer network for encoding, and the encoded feature image corresponding to each sequence is output. The matching cost of the encoded feature images corresponding to the two sequences of the left and right views at each resolution is calculated through the Correlation operation. The matching costs corresponding to the left and right views at three resolutions are aggregated through an adaptive aggregation network to output the aggregated matching cost. The adaptive aggregation network includes a same-scale aggregation module and a cross-scale aggregation module connected in sequence. The same-scale aggregation module includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a residual module connected in sequence. The aggregation formula for the same-scale aggregation module is as follows: , Parallax candidate d pixels in p The cost of aggregation at that location It is the number of sampling points. K =3, It is the first k Aggregate weights of points, It is a fixed offset of pixels. It is an additional regularization bias for learning. This is the adjustment mechanism for deformable convolution; The aggregation matching cost is weighted using the softmax function to obtain the probability distribution of each disparity candidate. Then, the probability distributions of the disparity candidates are weighted and averaged to obtain the predicted disparity value. The disparity values ​​are converted into depth values ​​using the softargmin function to obtain a depth map with depth values ​​for each pixel.

2. The method for predicting the trajectory of distant bird targets according to claim 1, characterized in that, The step of outputting the detection map with labeled bird target recognition boxes includes: Each frame of video image is input into a deep residual network for feature extraction. Simultaneously, two consecutive frames of video images from the acquired video images are input into an optical flow neural network for feature extraction. The moving target detection model outputs a detection map with labeled bird target recognition boxes that combine the features extracted by the deep residual network and the optical flow neural network. The video image input into the deep residual network is the previous frame of the two consecutive video images input into the optical flow neural network.

3. The method for predicting the trajectory of distant bird targets according to claim 1, characterized in that, The loss function of the moving target detection model include: ； in, The loss function for deep residual networks is... Let be the loss function of the optical flow neural network. These are the weighting coefficients; in, , i , j Foreground and background pixel position, For the first t Frame video image Binary prediction for each pixel, w The ratio of foreground to background pixels. E For the softmax function, θ This is the probability vector for binary prediction; in, , , From the first t Frame video image To the t +1 frame video image pixels The horizontal and vertical components of the actual optical flow field, and The first prediction of the optical flow neural network t Frame video image To the t +1 frame video image pixels The horizontal and vertical components of the optical flow field.

4. The method for predicting the trajectory of distant bird targets according to claim 1, characterized in that, The step of predicting the trajectory of a bird target includes: For each frame of the detection map labeled with bird target recognition boxes, calculate the pixel coordinates of the center point of the bird target recognition box; The pixel coordinates of the center point of the bird target recognition box are fused with the depth value of the center point in the depth map acquired at the same time to obtain the three-dimensional coordinates of the center point. x , y , z ),in, x For horizontal pixels, y For vertical pixels, z This is the depth value; After sorting the three-dimensional coordinates of the center point of each frame in ascending order of horizontal pixels, the interpolation function is used to interpolate the vertical pixels and depth values ​​to obtain new vertical pixels and depth values. The existing three-dimensional coordinates of the center point and the new three-dimensional coordinates obtained through interpolation are combined to obtain a set of three-dimensional coordinates; The three-dimensional coordinate set is input into an LSTM neural network trained based on a Kalman filter, which then predicts the three-dimensional coordinates of the bird target at the next time step and outputs the three-dimensional coordinates of the bird target at the next time step.

5. The method for predicting the trajectory of distant bird targets according to claim 4, characterized in that, Training the LSTM neural network based on a Kalman filter includes: The pixel coordinates of the center point of the bird target recognition box in the detection map of the first frame of the video in the training set are used as the initial state vector of the coordinates; Initialize the covariance matrix of the Kalman filter; After iteratively updating the Kalman filter using the training set, the first coordinates of the bird target at the next moment corresponding to the video in the training set are output. The video in the training set is input into the LSTM neural network, and the LSTM neural network outputs the second coordinate of the bird target at the next moment corresponding to the video. Calculate the average of the first and second coordinates to obtain the third coordinate; Using the third coordinate and the ground truth coordinates of the bird target at the next moment corresponding to the video, the loss function of the LSTM neural network is calculated. After calculating the gradient of the loss function of the LSTM neural network using the backpropagation algorithm, the weights of the LSTM neural network are updated using the stochastic gradient descent algorithm to complete the training.

6. The method for predicting the trajectory of distant bird targets according to claim 5, characterized in that, The iterative update of the Kalman filter using the training set includes: For each time step, state prediction is performed based on the state transition equation, wherein the state transition equation is: , For time step t Predicting the state without observational data. Here is the state transition matrix. The time interval between two time steps; Update the covariance matrix, where the update equation for the covariance matrix is: , This is the transpose of the state transition matrix. , These are the covariance matrices before and after the update; Repeat the above steps until the entire training set has been traversed.

7. The method for predicting the trajectory of distant bird targets according to claim 1, characterized in that, Following the step of predicting the trajectory of the bird target, the method further includes: Based on the movement trajectory of the bird target, a laser is emitted towards the location of the bird target's movement trajectory in order to drive it away.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, they implement the method for predicting the trajectory of a distant bird target as described in any one of claims 1 to 7.

9. A trajectory prediction system for distant bird targets, characterized in that, include: The computer-readable storage medium as described in claim 8.

Images