An intelligent bird detection and bird repelling method and device based on radar point cloud and visual image

Abstract

The present invention relates to an intelligent bird detection and bird repelling method and device based on radar point cloud and visual image. In order to overcome the fact that traditional bird repellents are easily adapted to birds and solve the problem that bird target detection technology based on visual images is subject to the influence of meteorological conditions and image quality, the fusion of millimeter wave radar and camera in coordinate system transformation and neural network input format is realized according to the strategy of feature layer fusion, and the front-end acquisition technology, mid-end recognition technology and back-end discrimination technology are proposed: the front-end acquisition technology can accurately realize the fusion of coordinate system, the mid-end recognition technology has more advantages in accuracy and robustness than the current mainstream target detection technology, and the back-end discrimination technology can meet the actual engineering application of accurate discrimination of bird behavior, and provide a criterion for whether to start the bird repelling device; finally, a set of intelligent bird detection and bird repelling devices is designed by integrating the functions and relationships of the above-mentioned technologies, which provides a strong guarantee for preventing the influence of bird damage and ensuring the safe operation of power transmission lines.

Description

Technical Field

[0001] The invention relates to an intelligent bird detection and bird repelling method and device based on radar point cloud and visual image. Background Art

[0002] According to statistics, the number of transmission line fault trips caused by bird activities is second only to lightning strikes and external damage, and is showing an increasing trend year by year. With the development of deep learning technology and the rise of drone aerial photography technology, detection technology based on machine vision and deep convolutional neural networks has also begun to be used in the field of bird damage prevention for transmission lines. Drones are used or patrol personnel are dispatched to shoot key parts of the tower and adaptively search out defective parts such as bird nests, insulators invaded by bird droppings, and parts of the line damaged by bird bites from a large number of inspection photos through machine learning algorithms. It is not affected by background, shooting angle, etc., and does not require manual data processing and experience judgment, which improves the efficiency of patrol personnel and the safety and reliability of the transmission network. However, the method of using camera images collected by drones or patrol personnel for recognition cannot meet the real-time requirements. On the other hand, the acquisition of images mostly depends on drones or patrol personnel using cameras to shoot, and it is undeniable that cameras are easily affected by weather factors. Weather such as rain, snow and fog, insufficient light at night and strong sunlight will lead to a decrease in the accuracy of target recognition, which cannot meet the requirements of all-weather performance.

[0003] Compared with video surveillance, millimeter-wave radar (hereinafter referred to as radar) using short-wave electromagnetic waves is a special radar technology. The radar emits a modulated wave signal, which is then reflected by the object on its propagation path. The radar system can determine the position, distance, speed and direction of the target in three-dimensional space by capturing the radar reflection signal. In addition, millimeter-wave radar has a high degree of integration and can penetrate fog, smoke and other obstructions better than infrared sensors. It is also more cost-effective than laser radar. However, millimeter-wave radar also has disadvantages such as fewer radar points, limited lateral resolution and inability to effectively depict the boundaries of obstacles. Millimeter-wave radar and video surveillance can meet the complementary advantages and disadvantages in target recognition technology, so realizing the integration of millimeter-wave radar and video surveillance and applying them in the field of detection is a very promising research direction. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent bird detection and bird repelling method and device based on radar point cloud and visual image, which can solve the problem that bird target detection technology based on visual image is subject to the influence of meteorological conditions and image quality.

[0005] To achieve the above object, the technical solution of the present invention is: an intelligent bird detection and bird repelling method based on radar point cloud and visual image, comprising:

[0006] S1, front-end acquisition;

[0007] S2, mid-range identification;

[0008] S3, back-end identification.

[0009] The present invention also provides an intelligent bird detection and bird repelling device based on radar point cloud and visual image, comprising:

[0010] Radar-camera fusion machine: The radar point cloud acquisition device and the visual image acquisition device are assembled together via the radar-camera fusion machine, and the radar-camera fusion machine transmits the collected image data to the edge computing terminal via the communication interface;

[0011] Edge computing terminal: The edge computing terminal used must have computing power; the edge computing terminal exchanges information with the host computer through the communication interface and connects with the drive device of the bird-repelling device through the I / O interface;

[0012] Host computer interaction: A computer that can directly issue operation commands. Through the host computer, the operation of the radar-camera fusion machine and the edge computing terminal can be remotely controlled and viewed;

[0013] Bird repellent device: includes a drive device and a jammer. In order to adapt to different identification conditions and target types, a conversion and upgrade interface is reserved.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] (1) Innovatively realize the fusion of millimeter wave radar and camera and apply it to the field of bird recognition. This technology has never been used in bird target detection. Compared with camera-based target detection, the use of millimeter wave radar and camera fusion technology can effectively improve the robustness and readiness of detection, overcome the influence of target recognition devices on environmental factors and image quality, and significantly improve detection capabilities.

[0016] (2) An innovative target behavior discrimination strategy is proposed and applied to the intelligent start-stop strategy of the bird repellent. The relevant strategy and formula have never been used in the development and design of intelligent bird repellent. This strategy combines the improved GIoU calculation and the three-frame difference algorithm to accurately distinguish whether the birds are staying or flying over the target area by calculating S3F-GIoU.

[0017] (3) Innovatively develop an intelligent bird detection and repelling device, which has never been used in the research of bird repelling and bird detection on power transmission lines. The proposed device can meet the front-end collection, mid-end detection and back-end discrimination in actual application and transmit the discrimination results to the bird repelling device for bird repelling. The developed device has a certain transfer interface, which provides a basis for realizing the design requirements of repelling different creatures. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of the front-end acquisition technology S1 of the present invention.

[0019] Figure 2 It is a radar point cloud attention neural network model based on feature layer fusion strategy.

[0020] Figure 3 It is the network structure of feature fusion.

[0021] Figure 4 This is a flow chart of the mid-end identification technology S2 of the present invention.

[0022] Figure 5 This is a flow chart of the back-end discrimination technology S3 of the present invention.

[0023] Figure 6 These are the design steps of the intelligent bird-repelling and bird-detecting device proposed by the present invention.

[0024] Figure 7 The intelligent bird repelling and bird detecting technology algorithm of the intelligent bird repelling and bird detecting device proposed in the present invention is realized.

[0025] Figure 8 The edge computing terminal NVIDIA Jetson TX2 with computing power is provided in the present invention.

[0026] Fig. 9 The radar-camera fusion machine style and shell size design of the present invention.

[0027] Fig.10 Some visual images (top) and point cloud images (bottom) in the dataset.

[0028] Fig.11 Part of the detection results of the mid-end recognition algorithm.

[0029] Fig.12 These are the test results under different bird activity conditions. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0031] The present invention provides an intelligent bird detection and bird repelling method based on radar point cloud and visual image, which can be divided into front-end acquisition technology S1, mid-end recognition technology S2 and back-end discrimination technology S3 according to its function and effect. The algorithm flow chart involved in the proposed technology is as follows: Figure 1 , Figure 4 and Figure 5 As shown, the specific process steps are:

[0032] (1) Front-end acquisition technology S1:

[0033] Step S11, setting configuration parameters of the radar module according to the target object, including but not limited to maximum distance resolution, maximum speed resolution and minimum reflection cross-sectional area, etc.;

[0034] Step S12: Calculate the camera's internal parameters f using the calibration checkerboard and the Matlab camera calibration toolbox. x 、f y , u0 and v0, calculate the camera coordinate system coordinate (X c ,Y c ,Z c ) and the image coordinate system coordinates (u, v);

[0035]

[0036] Among them, f x is the image width transformation ratio, f y is the image width transformation ratio, u0 and v0 are the offsets of the camera optical axis in the pixel coordinate system;

[0037] Step S13: design and manufacture a radar-camera fusion machine to fix the relative positions of the radar and the camera;

[0038] Step S14, placing corner reflectors at different positions in an open field and collecting radar point cloud data and visual images to construct a correction data set;

[0039] Step S15: Calibrate the rotation matrix R of the radar-camera fusion machine using a level 3×3 ;

[0040] Step S16: Use the OpenCV sliding window to perform mean fitting on the correction data set to calculate the translation matrix T of the fusion machine. 3×1 ;

[0041] Step S17: Calculate the radar coordinate system coordinate (X r ,Y r ,Z r ) and the image coordinate system coordinates (u, v);

[0042]

[0043] Among them, R 3×3 and T 3×1 They are the rotation matrix and translation matrix of the external parameters of the radar-camera fusion machine 0 1×3 is a 1×3 zero matrix, I 1×1 is a 1×1 identity matrix;

[0044] Step S18, convert the collected three-dimensional coordinates of the radar point cloud into a two-dimensional plane of the visual image through formula (2), and represent it with a square with a side length of 20 pixels. The speed, distance and reflection modulation wave peak value in the radar point cloud information are converted into the color filling value of the square through formulas (3) to (5), and finally converted into a point cloud image;

[0045]

[0046]

[0047]

[0048] Among them, (R, G, B) is the color filling value and R, G, B ∈ [0, 255], D o , P e and R a They are the velocity, distance and reflected modulation wave peak of the radar point cloud respectively;

[0049] Step S19, sharpening the collected visual image, and the sharpening kernel S adopts the enlarged Laplace expansion template as shown in formula (6);

[0050]

[0051] Step S10: The processed point cloud image and the processed visual image together form a set of data.

[0052] (2) Mid-range recognition technology S2:

[0053] Step S21, using a radar-camera fusion machine to collect point cloud images and visual images for training models to construct an initial data set;

[0054] Step S22: randomly extract a portion of the data for visual enhancement, wherein the visual enhancement method includes rotation, flipping and scaling, and the visual and point cloud images of this portion of the data will be rotated, flipped and scaled synchronously;

[0055] Step S23, randomly extracting a portion of visual images from the data for scene enhancement, the scene enhancement method is to add different masks: shadow bars, noise points, blurring, light spots, brightness adjustment and snow scene, and the enhanced data set is formed after processing;

[0056] Step S24: design a radar point cloud attention neural network model based on feature layer fusion strategy. The structure of the model is as follows: Figure 2 As shown, its main function is to identify birds and their locations under different meteorological conditions.

[0057] Furthermore, the model can be divided into four parts according to its function: feature fusion, backbone feature extraction, feature re-extraction and regression classifier:

[0058] (1) Feature Fusion: The structure is as follows Figure 3 As shown in the figure, the point cloud image collected by the radar and the visual image collected by the camera are fused. The point cloud image is converted into a visual image weighted matrix with a channel number of 1 through a certain convolution operation; the visual image is subjected to a certain convolution operation and multiplied with the visual image weighted matrix to obtain the fusion feature.

[0059] (2) Backbone feature extraction: The function of backbone feature extraction is to learn fusion features. Its backbone network is CSPDarknet, which consists of multiple ResBody with residual structure. The feature map with input dimension (320,320,64) is halved in size and doubled in channels after residual structure processing. The three outputs corresponding to the middle layer, middle and lower layer and bottom layer of the backbone network are (80,80,256), (40,40,512) and (20,20,1024) respectively. The output with lower resolution has a wider receptive field and is suitable for detecting large targets, while it is suitable for detecting small targets.

[0060] (3) Feature re-extraction: It further extracts features. By fusing the outputs of the three feature layers in the previous section, the features can contain more details and semantic information, which is conducive to extracting better features. The bottom-level output of the backbone feature extraction matches the dimensions of the middle and lower layer outputs after adjusting the channels and upsampling, and is stacked. The same applies to the stacking of the middle and lower layer outputs and the middle layer outputs. On the other side, downsampling enables the fusion of the features obtained by upsampling from bottom to top.

[0061] (4) Regression classifier: A lightweight decoupling head is used. The enhanced features obtained from the feature re-extraction step are processed by two convolutions with a 3×3 kernel and one convolution with a 1×1 kernel to obtain the regression parameter R corresponding to each feature point on the feature map. eg , Object Judgment bj and type judgment C ls .

[0062] (3) Back-end discrimination technology S3:

[0063] Step S31, placing the calibrated front-end acquisition device in an open area, and collecting test data at regular intervals;

[0064] Step S32: The collected test data is used for bird target detection through mid-end recognition technology and the bird prediction frame information is sequentially imported into the buffer;

[0065] Step S33, determining whether the buffer has three frames of data stored, if not, repeating steps S31 and S32 until the condition is met, if yes, executing step S34;

[0066] Step S34: the buffer data structure is a stack structure, the newly imported information is placed on the top of the stack and the bottom data is cleared, and the buffer only contains three adjacent frames of data;

[0067] Step S35, extracting bird identification frame information from three frames of data in the buffer respectively and arranging and combining them in sequence;

[0068] Step S36: Calculate the overlapping area I for each combination according to formula (7): ABC And according to formula (8), divide it by the enclosed area to obtain S 3F-IoU Finally, according to formula (9), subtract the difference between the area enclosed by the A, B and C boxes and the area of ​​the D box to obtain S 3F-GIoU ;

[0069] I ABC =|A∩B|+|B∩C|+|C∩A| (7)

[0070]

[0071]

[0072] Among them, A, B and C are the predicted boxes in any permutation and combination, and the D box is the smallest box that can contain the A, B and C boxes, and there are A, B, C, in is the function space, is a vector space, I ABC is the sum of the overlapping areas of frames A, B, and C, S 3F-IoU is the intersection-over-union ratio of boxes A, B, and C, S 3F-GIoU is the generalized intersection-over-union ratio of boxes A, B, and C;

[0073] Step S37: Determine the calculated S 3F-GIoU Is the value greater than or equal to 0.2? If yes, go to step S38; if not, go to step S39;

[0074] Step S38: if the result of the judgment is that the birds are staying in the target area, the bird-repelling device is activated to drive away the birds at a fixed point and in a fixed quantity;

[0075] Step S39: if the result of the determination is that birds have flown over the target area, the bird repellent device is turned off to save resources and prevent the disadvantage of the bird repellent device being easily adapted to by birds due to being activated for a long time.

[0076] The present invention also provides an intelligent bird detection and bird repelling device based on radar point cloud and visual image, the structure of which is as follows: Figure 6 As shown, the functions of each part are as follows:

[0077] (1) Radar-camera fusion machine: The radar point cloud acquisition device and the visual image acquisition device are assembled together via the radar-camera fusion machine, and the collected image data is mainly transmitted to the edge computing terminal via a communication interface, including but not limited to USB, Bluetooth, WIFI, HDMI, etc.;

[0078] (2) Edge computing terminal: Since the network structure and parameter quantity of the target recognition algorithm are relatively complex, the edge computing terminal used must have computing power; it must have a certain communication interface to exchange information with the host computer, and it can be connected to the drive device of the bird-repelling device through the I / O interface;

[0079] (3) Communication interface: In order to adapt to different working environments, the edge computing terminal can communicate with the host through communication interfaces such as standard serial port, Ethernet, USB, 4G / 5G and WIFI;

[0080] (4) Host computer interaction: A computer that can directly issue operation commands. Through the host computer, the operation of the radar-camera fusion machine and the edge computing terminal can be remotely controlled and viewed;

[0081] (5) Bird repellent device: mainly includes a driving device and a jammer. The jammer includes but is not limited to bird repellent devices that can generate interference such as ultrasonic waves, biological odors, sonic booms and lasers. At the same time, in order to adapt to different identification conditions and target types, conversion and upgrade interfaces are reserved.

[0082] The algorithm implementation flow chart of the proposed intelligent bird repelling and bird detecting device is as follows: Figure 7 As shown, the algorithm steps are as follows:

[0083] Step S41, placing the calibrated front-end acquisition device in an open area, and collecting test data at regular intervals;

[0084] Step S42: The collected test data is used for bird target detection through mid-end recognition technology and the bird prediction frame information is sequentially imported into the buffer;

[0085] Step S43, determining whether the buffer stores three frames of data, if not, repeating steps S41 and S42 until the condition is met, if yes, executing step S44;

[0086] Step S44, the buffer data structure is a stack structure, the newly imported information is placed on the top of the stack and the bottom data is cleared, and the buffer only contains three adjacent frames of data;

[0087] Step S45, extracting bird identification frame information from three frames of data in the buffer respectively and arranging and combining them in sequence;

[0088] Step S46: Calculate the overlapping area I for each combination according to formula (10): ABCAnd according to formula (11), divide it by the enclosed area to obtain S 3F-IoU Finally, according to formula (12), subtract the difference between the area enclosed by the A, B and C boxes and the area of ​​the D box to obtain S 3F -GIoU;

[0089] I ABC =|A∩B|+|B∩C|+|C∩A| (10)

[0090]

[0091]

[0092] Among them, A, B and C are the predicted boxes in any permutation and combination, and the D box is the smallest box that can contain the A, B and C boxes, and there are A, B, C, in is the function space, is a vector space, I ABC is the sum of the overlapping areas of frames A, B, and C, S 3F-IoU is the intersection-over-union ratio of boxes A, B, and C, S 3F-GIoU is the generalized intersection-over-union ratio of boxes A, B, and C;

[0093] Step S47: Determine the calculated S 3F-GIoU Is the value greater than or equal to 0.2? If yes, go to step S48; if not, go to step S49;

[0094] Step S48: if the result of the judgment is that the birds are staying in the target area, the bird-repelling device is activated to drive away the birds at a fixed point and in a fixed quantity;

[0095] Step S49: if the result of the determination is that birds fly over the target area, the bird repellent device is turned off to save resources and prevent the disadvantage of the bird repellent device being easily adapted to by birds due to being activated for a long time.

[0096] Step S40, determine whether a termination instruction from the host computer is received, if not, return to step S41 to execute the loop, if yes, end the identification and terminate the loop.

[0097] Case Analysis

[0098] In this example, the hardware models used are shown in Table 1: the radar point cloud acquisition device uses TIIWR1443BOOST EVM, the visual image uses a monocular camera, and the edge computing terminal uses Figure 8 The NVIDIA Jetson TX2 with computing power and multiple communication interfaces is shown in Figure 1. Fig. 9 The radar-camera fusion machine shown in FIG. 1 constructs a front-end acquisition device, and some point cloud images and visual images of the radar-image fusion data set are collected as shown in FIG. Fig.10 As shown in the figure, the position of the radar point cloud matches the target in the visual image, so the design and correction of the front-end acquisition device meet the design requirements.

[0099] Table 1 Hardware configuration of this example

[0100]

[0101] Some of the detection results of the proposed mid-range target recognition algorithm are shown in Fig.11 As shown in the figure, the proposed detection model can well adapt to the impact of different meteorological environments on the camera, and can give good bird recognition results in low visibility weather such as foggy days, rainy days and nights.

[0102] The comparison of the recognition accuracy of the proposed mid-range target recognition algorithm and the current mainstream target detection algorithm is shown in Table 2. Through analysis, it can be found that the proposed method uses dual input and radar point cloud attention mechanism, which broadens the information collection channel, improves the bird recognition accuracy and recognition robustness under different meteorological conditions.

[0103] Table 2 Performance comparison of different target detection algorithms

[0104]

[0105] Fig.12 The three frames of images and prediction box results are shown when the bird stays on the line, flies to the left, and flies to the right. The S 3F-GIoU The values ​​are shown in Table 3. When the bird stays on the transmission line, its position is relatively fixed, and the prediction frames of the three frames collected have a high degree of overlap, so S 3F-GIoU is relatively high, and increases with the stability of the bird's posture; on the contrary, when the bird flies over the target area, the overlap of the prediction frames of the three frames collected is very low, S 3F-GIoU The calculated values ​​are also low and decrease as the flight speed of the bird increases.

[0106] To further explore the ability of S to distinguish between bird species 3F-GIoU Threshold, statistical analysis of the collected 50 sets of three-frame image data and screening them into two situations: stay and fly. Through mean calculation, it is found that when the threshold is set to 0.2, more than 90% of bird activities can be distinguished. In summary, the basis for distinguishing bird activities is:

[0107] 1. When 0.2≤S 3F-GIoU When ≤1, it is judged that birds are staying in the target area and the bird repellent is turned on;

[0108] 2. When -1≤S 3F-GIoU When the value is less than 0.2, it is judged that birds are flying over the target area and there is no need to activate the bird repellent.

[0109] Table 3 S under different bird activities 3F-GIoU Calculated Value

[0110]

[0111] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. An intelligent bird detection and bird repelling method based on radar point cloud and visual image, characterized in that: include: S1, front-end acquisition; S2, mid-range identification; S3, back-end discrimination; The step S1 is specifically implemented as follows: Step S11, setting configuration parameters of the radar module according to the target object, including maximum distance resolution, maximum speed resolution and minimum reflection cross-sectional area; Step S12: Calculate the camera's internal parameters f using the calibration checkerboard and the Matlab camera calibration toolbox. x 、f y , u0 and v0, calculate the camera coordinate system coordinate (X c ,Y c ,Z c ) and the image coordinate system coordinates (u, v); Among them, f x is the image length transformation ratio, f y is the image width transformation ratio, u0 and v0 are the offsets of the camera optical axis in the pixel coordinate system; Step S13: design and manufacture a radar-camera fusion machine to fix the relative positions of the radar and the camera; Step S14, placing corner reflectors at different positions in an open field and collecting radar point cloud data and visual images to construct a correction data set; Step S15: Calibrate the rotation matrix R of the radar-camera fusion machine using a level 3×3 ; Step S16: Use the OpenCV sliding window to perform mean fitting on the correction data set to calculate the translation matrix T of the radar-camera fusion machine. 3×1 ; Step S17: Calculate the radar coordinate system coordinate (X r ,Y r ,Z r ) and the image coordinate system coordinates (u, v); Among them, R 3×3 and T 3×1 are the rotation matrix and translation matrix of the radar-camera fusion machine, respectively. 1×3 is a 1×3 zero matrix, I 1×1 is a 1×1 identity matrix; Step S18, convert the collected three-dimensional coordinates of the radar point cloud into a two-dimensional plane of the visual image through formula (2), and represent it with a square with a side length of 20 pixels. The speed, distance and reflection modulation wave peak value in the radar point cloud information are converted into the color filling value of the square through formulas (3) to (5), and finally converted into a point cloud image; Among them, (R, G, B) is the color filling value and R, G, B ∈ [0, 255], D o , P e and R a They are the velocity, distance and reflected modulation wave peak of the radar point cloud respectively; Step S19, sharpening the collected visual image, and the sharpening kernel S adopts the enlarged Laplace expansion template as shown in formula (6); Step S10, the processed point cloud image and the processed visual image together form a set of data; The step S2 is specifically implemented as follows: Step S21, using a radar-camera fusion machine to collect point cloud images and visual images for training models to construct an initial data set; Step S22: randomly extract a portion of the data for visual enhancement, wherein the visual enhancement method includes rotation, flipping and scaling, and the visual image and the point cloud image of this portion of the data will be rotated, flipped and scaled synchronously; Step S23, randomly extracting a portion of visual images from the data for scene enhancement, the scene enhancement method is to add different masks: shadow bars, noise points, blurring, light spots, brightness adjustment and snow scene, and the enhanced data set is formed after processing; Step S24, designing a radar point cloud attention neural network model based on a feature layer fusion strategy to identify birds and their locations under different meteorological conditions; The radar point cloud attention neural network model based on the feature layer fusion strategy is divided into four parts according to its function: feature fusion device, backbone feature extraction, feature re-extraction and regression classifier: Feature fusion: The point cloud image collected by the radar and the visual image collected by the camera are fused. The point cloud image is converted into a visual image weighted matrix with a channel number of 1 through a convolution operation. The visual image is convolved and multiplied with the visual image weighted matrix to obtain the fusion feature. Backbone feature extraction: The function of backbone feature extraction is to learn fusion features. Its backbone network is CSPDarknet, which consists of multiple ResBody with residual structure. The feature map with input dimension (320,320,64) is halved in size and doubled in channels after residual structure processing. The three outputs of the middle layer, middle and lower layer and bottom layer of the backbone network are (80,80,256), (40,40,512) and (20,20,1024) respectively. The output with lower resolution has a wider receptive field, which is suitable for the detection of large targets, and vice versa for the detection of small targets. Feature re-extraction: Further extraction of features, by fusing the three feature layer outputs of the backbone feature extraction link, can make the features contain more details and semantic information, which is conducive to extracting better features; the bottom layer output of the backbone feature extraction matches the dimension of the middle and lower layer output after adjusting the channel and upsampling and stacking, and the same applies to the stacking of the middle and lower layer output and the middle layer output; on the other side, downsampling realizes the fusion of the features obtained by upsampling from bottom to top; Regression classifier: A lightweight decoupling head is used. The enhanced features obtained from the feature re-extraction link are processed by two convolutions with a 3×3 kernel and one convolution with a 1×1 kernel to obtain the regression parameter R corresponding to each feature point on the feature map. eg , Object Judgment bj and type judgment C ls ; The step S3 is specifically implemented as follows: Step S31, placing the calibrated front-end acquisition device in an open area, and collecting test data at predetermined intervals; Step S32: The collected test data is subjected to bird target detection through step S2 and the bird prediction frame information is sequentially imported into the buffer; Step S33, determining whether the buffer has three frames of data stored, if not, repeating steps S31 and S32 until the condition is met, if yes, executing step S34; Step S34, the buffer data structure is a stack structure, the newly imported information is placed on the top of the stack and the bottom data is cleared, and the buffer only contains three adjacent frames of data; Step S35, extracting bird identification frame information from three frames of data in the buffer respectively and arranging and combining them in sequence; Step S36: Calculate the overlapping area I for each combination according to formula (7): ABC And according to formula (8), divide it by the enclosed area to obtain S 3F-IoU Finally, according to formula (9), subtract the difference between the area enclosed by the A, B and C boxes and the area of ​​the D box to obtain S 3F-GIoU ; I ABC =|A∩B|+|B∩C|+|C∩A| (7) Among them, A, B and C are the predicted boxes in any permutation and combination, and the D box is the smallest box that can contain the A, B and C boxes, and there are A, B, C, in is the function space, is a vector space, I ABC is the sum of the overlapping areas of frames A, B, and C, S 3F-IoU is the intersection-over-union ratio of boxes A, B, and C, S 3F-GIoU is the generalized intersection-over-union ratio of boxes A, B, and C; Step S37: Determine the calculated S 3F-GIoU Is the value greater than or equal to 0.2? If yes, go to step S38; if not, go to step S39; Step S38: if the result of the judgment is that the birds are staying in the target area S, the bird-repelling device is activated to drive away the birds at a fixed point and in a fixed quantity; Step S39: if the result of the determination is that birds have flown over the target area, the bird repellent device is turned off to save resources and prevent the disadvantage of the bird repellent device being easily adapted to by birds due to being activated for a long time.

2. An intelligent bird detection and bird repelling device based on radar point cloud and visual image, characterized in that: include: Radar-camera fusion machine: The radar point cloud acquisition device and the visual image acquisition device are assembled together via the radar-camera fusion machine, and the radar-camera fusion machine transmits the collected image data to the edge computing terminal via the communication interface; Edge computing terminal: The edge computing terminal used must have computing power; The edge computing terminal exchanges information with the host computer through a communication interface and is connected to the driving device of the bird-repelling device through an I / O interface; the edge computing terminal is loaded with a program for executing the method as claimed in claim 1; Host computer interaction: A computer that can directly issue operation commands. Through the host computer, the operation of the radar-camera fusion machine and the edge computing terminal can be remotely controlled and viewed; Bird repellent device: includes a drive device and a jammer. In order to adapt to different identification conditions and target types, a conversion and upgrade interface is reserved.

3. The intelligent bird detection and bird repelling device based on radar point cloud and visual image according to claim 2 is characterized in that: The communication interfaces used by the radar-camera fusion machine include USB, Bluetooth, WIFI, and HDMI.

4. The intelligent bird detection and bird repelling device based on radar point cloud and visual image according to claim 2 is characterized in that: The communication interfaces used by edge computing terminals include standard serial ports, Ethernet, USB, 4G / 5G and WIFI.

5. The intelligent bird detection and bird repelling device based on radar point cloud and visual image according to claim 2 is characterized in that: The jammers include bird repellers that can generate ultrasonic waves, biological odors, sonic booms, and laser interference.

Images