Vision-based flying object risk monitoring apparatus

Abstract

The present invention relates to a vision-based flying object risk monitoring apparatus. The apparatus comprises: a camera array, which is arranged around a tower, and an input end of which is angled upwards. By means of synthesizing a panoramic image, obtaining the position and image information of a flying object on the basis of the panoramic image, further obtaining the size and distance of the flying object, and generating a copying policy for a wind turbine in combination with the trajectory of the flying object obtained on the basis of the positions of the same flying object in a plurality of consecutive panoramic images, a more accurate prediction result is obtained, thereby achieving effects of reducing unnecessary downtime, etc.

Description

Vision-based flying object risk monitoring device Technical Field

[0001] The present invention relates to the field of wind power generation, and in particular to a vision-based flying object risk monitoring device. Background Art

[0002] With the development of power system dispatching technology and wind power generation prediction technology, the grid-connected dispatching problem of wind power generation has been gradually overcome. As a clean energy, wind power generation has been increasingly widely used.

[0003] With the increase in installed wind power and the proportion of wind power generation, the impact of wind power generation on the ecological environment has received more and more attention. For example, some wind turbines are installed in places where there are many birds. If birds hit the blades of the wind turbine, it may cause the death of the birds. If the birds hit are protected species, it will bring legal problems.

[0004] Therefore, those skilled in the art have devoted themselves to solving such problems.

[0005] For example, U.S. Patent No. 11751560B2 provides an imaging array for bird or bat detection and identification. A camera is placed on the nacelle of a wind turbine, along with a group of cameras on the tower. This array is used to detect birds and identify their flight paths. However, this prior art uses a single camera for bird identification, which can lead to significant errors and is prone to duplicate identification, making it difficult to generate consistent output results.

[0006] In addition, U.S. Patent No. US20130050400A1 also discloses a similar technology, which discloses a device and method for preventing flying animals from colliding with wind turbines. This device uses a camera placed on the nacelle of the wind turbine to use a visual solution to identify the size, distance, and trajectory of birds, thereby assessing the probability of collision between birds or other flying objects and the wind turbine. Unlike the solution provided by U.S. Patent No. US11751560B2, this solution uses panoramic images to improve recognition accuracy. However, in this existing technology, the camera is installed on the nacelle, which has a relatively large blind spot.

[0007] Summary of the Invention

[0008] The purpose of the present invention is to provide a vision-based flying object risk monitoring device.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] A vision-based flying object risk monitoring device, comprising:

[0011] The camera array is arranged around the tower, with the input end tilted upward;

[0012] The panoramic image synthesis module is configured to: obtain images captured by the camera array, and synthesize the images captured by the camera array into a panoramic image;

[0013] The flying object recognition module is configured to: recognize the flying object based on the panoramic image and obtain image information of the flying object;

[0014] The flying object size generating module is configured to: obtain the size of the flying object according to the image information of the flying object;

[0015] The distance acquisition module is configured to: obtain the distance between the flying object and the wind turbine according to the image information based on the flying object;

[0016] a trajectory generation module configured to: obtain a trajectory of the flying object according to a position of the same flying object in a plurality of consecutive panoramic images, and generate a flight direction and speed of the flying object based on the obtained trajectory of the flying object;

[0017] The strategy generation module is configured to determine a response strategy of the wind turbine based on the size, flight direction and speed of the flying object, and the distance between the flying object and the wind turbine.

[0018] The device further comprises:

[0019] The protected bird comparison module is configured to: compare the image information of the flying object with a pre-configured local bird database to generate a probability that the flying object is a protected bird;

[0020] The strategy generation module is specifically configured to generate a response strategy for the wind turbine based on the size, flight direction and speed of the flying object, the distance between the flying object and the wind turbine, and the probability that the flying object is protecting birds.

[0021] The step of identifying the flying object based on the panoramic image and obtaining the image information of the flying object specifically includes:

[0022] Based on the obtained panoramic image, the flying object is identified and located to obtain the position of the flying object;

[0023] The image information of the flying object is extracted based on the recognition result and the position of the flying object.

[0024] The obtaining of the size of the flying object according to the image information of the flying object is specifically: based on the obtained image information of the flying object, the size of the flying object is obtained according to its pixel information.

[0025] The obtaining of the distance between the flying object and the wind turbine based on the image information of the flying object specifically includes:

[0026] The distance between the flying object and the wind turbine is obtained by using one or more of the following estimated distances:

[0027] i) obtaining a first estimated distance based on the position and size of the flying object in combination with the focal length and resolution of the camera;

[0028] ii) obtaining a second estimated distance based on the position of the flying object using a binocular vision algorithm;

[0029] iii) Based on the position of the flying object, a third estimated distance is obtained using a depth estimation model.

[0030] The panoramic image synthesis module includes:

[0031] The image dedistortion unit is configured to: perform dedistortion processing on the image captured by the camera array;

[0032] The image synthesis unit is configured to: input the dedistorted images combined with the camera position parameters corresponding to each image into a trained image fusion model to obtain a panoramic image output by the image fusion model.

[0033] The image fusion model is configured to perform the following steps:

[0034] Identify clouds in the input images and register the clouds in each image;

[0035] Based on the cloud registration results, the overlapping areas in each image are identified;

[0036] Based on the identified overlapping areas, all input images are stitched together to obtain a panoramic image in combination with the positions of the cameras corresponding to each image.

[0037] The position of the flying object is obtained through an instance segmentation model.

[0038] The flight direction and speed of the flying object are obtained through a flying object motion trajectory prediction model, wherein the input obtained by the flying object motion trajectory prediction model is a binary sequence, and the binary sequence includes a time and the coordinates of the flying object corresponding to the time.

[0039] The camera array includes one or more annular camera assemblies, all of which are arranged around the tower, and each annular camera assembly includes multiple cameras.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. Panoramic images are used as the basis for recognition. The wind turbine response strategy is generated by taking into account the size, flight direction and speed of the flying object, placing the camera on the tower instead of the nacelle, and combining the distance between the flying object and the wind turbine. The resulting prediction results are more accurate and have the effect of reducing unnecessary downtime.

[0042] 2. Using clouds as a reference solves the image synthesis problem of the camera array installed on the tower, eliminating the need for cameras on the cabin. This solves the problem of field of view obstruction and eliminates the need for accessories on the cabin, improving convenience and safety.

[0043] 3. Using one or more estimated distances to comprehensively obtain the distance between the flying object and the wind turbine is more flexible.

[0044] 4. The input of the flying object trajectory prediction model is a binary sequence, which can improve the accuracy of flight direction and speed prediction.

[0045] 5. The camera array includes thermal imaging cameras, which can improve imaging quality at night. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] FIG1 is a schematic diagram of the arrangement of a camera array according to an embodiment of the present invention;

[0047] FIG2 is a schematic diagram of the main components of the framework of the present invention;

[0048] FIG3 is a schematic flow chart of the main steps in an embodiment;

[0049] FIG4 is a schematic diagram of a camera array disposed on a bracket outside a tower;

[0050] Among them: 1. Tower, 2. Camera. DETAILED DESCRIPTION

[0051] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0052] A vision-based flying object risk monitoring device, as shown in Figures 2 and 3, includes:

[0053] The camera array, as shown in Figures 1 and 4, is arranged around the tower 1, and the input end is tilted upward. The camera array is composed of multiple cameras 2. For example, it can be set on the tower 1 as shown in Figure 1, or some fixing frames can be set around the tower 1 as shown in Figure 4 to install the camera array.

[0054] The panoramic image synthesis module is configured to: obtain images captured by the camera array, and synthesize the images captured by the camera array into a panoramic image;

[0055] The flying object recognition module is configured to: recognize the flying object based on the panoramic image and obtain image information of the flying object;

[0056] The flying object size generating module is configured to: obtain the size of the flying object according to the image information of the flying object;

[0057] The distance acquisition module is configured to: obtain the distance between the flying object and the wind turbine according to the image information based on the flying object;

[0058] a trajectory generation module configured to: obtain a trajectory of the flying object according to a position of the same flying object in a plurality of consecutive panoramic images, and generate a flight direction and speed of the flying object based on the obtained trajectory of the flying object;

[0059] The strategy generation module is configured to determine a response strategy of the wind turbine based on the size, flight direction and speed of the flying object, and the distance between the flying object and the wind turbine.

[0060] Compared with the existing technology, the inventors found that if the camera can be placed only on the tower, the occlusion problem can be solved. However, if the camera is only placed on the tower, it is difficult to achieve panoramic image synthesis. Existing panoramic images all rely on flat ground for chessboard calibration. Facing the sky, it is difficult to find such a fixed reference object. If the cabin is used as a reference object, some images captured by the camera may not include the cabin. To this end, the present application solves this problem through the following means. Specifically, the images captured by the camera array are synthesized into a panoramic image, including:

[0061] Dedistorting images captured by the camera array;

[0062] The dedistorted images are combined with the camera position parameters corresponding to each image and input into the trained image fusion model to obtain the panoramic image output by the image fusion model.

[0063] In this embodiment, the image fusion model is configured to perform the following steps:

[0064] Identify clouds in the input images and register the clouds in each image;

[0065] Based on the cloud registration results, the overlapping areas in each image are identified;

[0066] Based on the identified overlapping areas, all input images are stitched together to obtain a panoramic image in combination with the positions of the cameras corresponding to each image.

[0067] Typically, a single camera is calibrated offline using methods such as Zhang's calibration to determine its intrinsic parameters. This step only requires a single offline calibration, and the calibration information can be reused for the same camera. The intrinsic parameters, including distortion parameters, are then used to flexibly perform batch dedistortion on the original image based on the calibrated distortion parameters. This allows for fast and accurate processing.

[0068] In this embodiment, the image fusion model inputs are: dedistorted photos taken by N cameras at different ranges and their relative positions, and the output label is: the entire panoramic photo. Of course, in other embodiments, if the number of cameras is sufficient, depth information of the photographed object can also be obtained, that is, the output of the image fusion model includes depth information.

[0069] Compared with frequent on-site calibration, in this embodiment, the sky cloud clusters used have low correlation with regional locations, and a single training run can be applied to wind fields in multiple regions. The algorithm has wider adaptability, higher accuracy, and is easy to operate.

[0070] In addition, in other embodiments, some reference objects may be fixed to assist in the synthesis of panoramic images. Specifically, after the camera array is fixed, a textured calibration plate may be placed in the common shooting area as a reference, and then the external reference between the cameras, i.e., the posture conversion between the cameras, is obtained.

[0071] In this type of embodiment, similarly, a single camera uses an offline calibration method to determine the camera's intrinsic parameter information, such as Zhang's calibration; this step only requires offline calibration once, and the calibration information can be reused for the same camera; the relative positions of the two cameras can be determined by placing a textured calibration plate in the common shooting area. According to the calibrated distortion parameters, the original graphics can be flexibly dedistorted in batches; the processing speed is fast and accurate. Once the relative positions between the cameras are known, the camera coordinates can be unified into one camera, and then the camera coordinates are converted to the coordinates of the tower center. According to the coordinate conversion and imaging coordinate conversion formulas, the image points are mapped into the panoramic image coordinate system to obtain a panoramic photo. Taking a certain picture as a reference, the color saturation of other pictures is pre-processed and unified to solve the color difference problem; at the same time, filtering and weighting methods are used to eliminate stitching gaps.

[0072] In this embodiment, identifying a flying object based on a panoramic image and obtaining image information of the flying object specifically includes:

[0073] Based on the obtained panoramic image, the flying object is identified and located to obtain the position of the flying object;

[0074] The image information of the flying object is extracted based on the recognition result and the position of the flying object.

[0075] The size of the flying object is obtained according to the image information of the flying object, specifically: based on the obtained image information of the flying object, the size of the flying object is obtained according to its pixel information.

[0076] In addition, in the present embodiment, obtaining the distance between the flying object and the wind turbine based on the image information of the flying object specifically includes:

[0077] The distance between the flying object and the wind turbine is obtained by using one or more of the following estimated distances:

[0078] i) obtaining a first estimated distance based on the position and size of the flying object in combination with the focal length and resolution of the camera;

[0079] ii) obtaining a second estimated distance based on the position of the flying object using a binocular vision algorithm;

[0080] iii) Based on the position of the flying object, a third estimated distance is obtained using a depth estimation model.

[0081] Specifically, the distance between the flying object and the wind turbine can be calculated using one of the statistical indicators such as arithmetic mean, weighted mean, median, mode, etc., or a combination of these.

[0082] In this embodiment, the positions of the flying objects are determined using an instance segmentation model. The stitched panoramic image is input and the output is the bounding box and pixel-level mask information for each flying object in the image. Examples of instance segmentation models include Mask R-CNN, U-Net, and Yolov5.

[0083] The flight direction and speed of an aircraft are determined by a trajectory prediction model. The model's input is a sequence of two-tuples, consisting of a time and the corresponding aircraft coordinates. This model is a time series prediction model, trained on previously captured videos of aircraft. The model uses past aircraft trajectories to predict future flight direction and speed. This model can use LSTM, GRU, or a Transformer-based time series prediction model.

[0084] Furthermore, in some embodiments, the apparatus further comprises:

[0085] The protected bird comparison module is configured to: compare the image information of the flying object with a pre-configured local bird database to generate a probability that the flying object is a protected bird;

[0086] The strategy generation module is specifically configured to generate a response strategy for the wind turbine based on the size, flight direction and speed of the flying object, the distance between the flying object and the wind turbine, and the probability that the flying object is protecting birds.

[0087] Specifically, an image similarity calculation model can be used to match images of flying objects with a library of images of local protected birds to determine the probability that the object is a protected bird. This model training process uses a dataset of common local birds and protected birds, such as a twin network. The probability of an object striking a wind turbine blade and the probability of the object being a protected bird are then provided as input to the control system, which then adjusts the wind turbine's response strategy based on this input. In most embodiments, these response strategies include shutting down the turbine and driving it away.

[0088] In addition, in this embodiment, the camera array includes one or more annular camera combinations, all of which are arranged around the tower, and each annular camera includes multiple cameras. In addition, each annular camera can adopt the arrangement as shown in Figure 4, or the innermost layer can adopt the arrangement as shown in Figure 1, and the remaining layers can adopt the arrangement as shown in Figure 4.

[0089] Furthermore, in some embodiments, some of the cameras in the camera array may be thermal imaging cameras, which can improve the quality of imaging at night, thereby enabling normal identification of flying objects at night.

[0090] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A vision-based flying object risk monitoring device, characterized in that, it includes: A camera array, arranged around the tower barrel, and the input end is tilted upward; A panoramic image synthesis module, configured to: acquire the images captured by the camera array and synthesize them into a panoramic image based on the images captured by the camera array; A flying object recognition module, configured to: recognize a flying object according to the panoramic image and obtain the image information of the flying object; A flying object size generation module, configured to: obtain the size of the flying object according to the image information of the flying object; A distance acquisition module, configured to: obtain the distance between the flying object and the wind turbine according to the image information of the flying object; A trajectory generation module, configured to: obtain the trajectory of the flying object according to the positions of the same flying object in consecutive panoramic images, and generate the flying direction and speed of the flying object based on the obtained trajectory of the flying object; A strategy generation module, configured to: generate a response strategy for the wind turbine based on the obtained size, flying direction and speed of the flying object, and the distance between the flying object and the wind turbine.

2. The vision-based flying object risk monitoring device according to claim 1, characterized in that, the device further includes: A protected bird comparison module, configured to: compare with a pre-configured local bird database based on the image information of the flying object and generate the probability that the flying object is a protected bird; The strategy generation module is specifically configured to: generate a response strategy for the wind turbine based on the obtained size, flying direction and speed of the flying object, the distance between the flying object and the wind turbine, and the probability that the flying object is a protected bird.

3. The vision-based flying object risk monitoring device according to claim 1, characterized in that, recognizing the flying object according to the panoramic image and obtaining the image information of the flying object specifically includes: Based on the obtained panoramic image, recognizing the flying object and positioning the flying object to obtain the position of the flying object; Extracting the image information of the flying object based on the recognition result of the flying object and the position of the flying object.

4. The vision-based flying object risk monitoring device according to claim 1, characterized in that, obtaining the size of the flying object according to the image information of the flying object is specifically: based on the obtained image information of the flying object, obtaining the size of the flying object according to its pixel information.

5. The vision-based flying object risk monitoring device according to claim 3, characterized in that, obtaining the distance between the flying object and the wind turbine according to the image information of the flying object specifically includes: Obtaining the distance between the flying object and the wind turbine by using one or more of the following estimated distances: i) Obtaining a first estimated distance based on the position and size of the flying object in combination with the focal length and resolution of the camera; ii) Obtaining a second estimated distance based on the position of the flying object by using a binocular vision algorithm; iii) Obtaining a third estimated distance based on the position of the flying object by using a depth estimation model.

6. The vision-based flying object risk monitoring device according to claim 1, characterized in that, the panoramic image synthesis module includes: An image de-distortion unit, configured to: perform de-distortion processing on the images captured by the camera array; An image synthesis unit, configured to: input the image after distortion removal processing and the camera position parameters corresponding to each image into a trained image fusion model, and obtain a panoramic image output by the image fusion model.

7. A vision-based flying object risk monitoring device according to claim 6, wherein, the image fusion model is configured to perform the following steps: Identify the cloud clusters in the input images, and register the cloud clusters in each image; Identify the overlapping regions in each image based on the cloud cluster registration results; Based on the identified overlapping regions, stitch all the input images together according to the positions of the corresponding cameras of each image to obtain a panoramic image.

8. A vision-based flying object risk monitoring device according to claim 3, wherein, the position of the flying object is obtained through an instance segmentation model.

9. A vision-based flying object risk monitoring device according to claim 1, wherein, the flight direction and speed of the flying object are obtained through a flying object motion trajectory prediction model, wherein the input of the flying object motion trajectory prediction model is a sequence of binary tuples, and the binary tuple includes a time and the coordinates of the flying object corresponding to the time.

10. A vision-based flying object risk monitoring device according to claim 8, wherein, the camera array includes one or more annular camera combinations, all the annular camera combinations are arranged around the tower barrel in a surrounding manner, and each annular camera includes a plurality of cameras.

11. A vision-based flying object risk monitoring device according to any one of claims 1-10, wherein, the camera array includes a thermal imaging camera.

Images