Multimodal-based remote sensing offshore windmill identification method

Abstract

The invention provides a remote sensing offshore windmill identification method based on multiple modes. The method comprises the following steps: S1, obtaining an optical image and an SAR image of the same area; s2, performing radiometric calibration, atmospheric correction and geometric correction processing on the optical image, performing complex data conversion, filtering and geocoding operation on the SAR image, and then performing resampling on the optical image and the SAR image; s3, respectively cutting image blocks taking a central point as a cutting center from the optical image and the SAR image processed in the S2 to form a training sample set; s4, inputting the training sample set into the multi-modal offshore windmill identification model for training so as to improve the fitting degree of the multi-modal offshore windmill identification model, and obtaining the trained multi-modal offshore windmill identification model; and S5, obtaining the position and orientation information of the offshore windmill by performing morphological analysis and geometric calculation on the bright spots of the blades of the offshore windmill on the SAR image. According to the method, offshore windmill identification is carried out on the remote sensing image, and the detection precision and efficiency are high.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a multi-modal remote sensing offshore windmill identification method. Background Art

[0002] As the global demand for renewable energy continues to grow, offshore wind power generation has become one of the important sources of new energy. Offshore wind turbines play a vital role in reducing emissions and reducing dependence on fossil energy by utilizing strong and stable wind power at sea to generate electricity. However, the large-scale construction and operation and maintenance of offshore wind turbines also bring corresponding technical challenges, especially in monitoring and managing the location and status of wind turbines. In order to achieve automated and precise monitoring of offshore wind turbines, remote sensing technology has become an important part of the solution.

[0003] Traditional offshore wind turbine monitoring mainly relies on optical images, which can provide clear visual information during the day or in good weather conditions, and is suitable for precise positioning and appearance monitoring of offshore wind turbines. However, optical images perform poorly in cloud cover, at night or in bad weather, affecting their all-weather monitoring capabilities. In contrast, synthetic aperture radar (SAR) images, with their all-day, all-weather imaging advantages, can obtain ground information without being affected by weather conditions, and have gradually become one of the effective tools for offshore wind turbine monitoring. SAR images are particularly good at object detection in marine environments and can capture the characteristics of offshore wind turbines through their high-resolution images. However, single SAR images are still inferior to optical images in terms of resolution and detail presentation, especially in the recognition of fine morphology and structural features of objects.

[0004] Existing offshore wind turbine identification patents usually focus on wind turbine detection under a single mode, either based on optical images or SAR images. Although these methods perform well in their respective application scenarios, it is difficult to achieve stable and high-precision monitoring results under all conditions due to the advantages and disadvantages of the two types of images. Summary of the invention

[0005] In view of this, the purpose of the present invention is to propose a multi-modal remote sensing offshore windmill identification method to solve the technical problem that the existing offshore windmill identification technology cannot simultaneously apply optical images and SAR images.

[0006] The technical means adopted by the present invention are as follows:

[0007] A remote sensing offshore windmill identification method based on multi-modality comprises the following steps:

[0008] S1. Obtain optical images and SAR images of the same area in the remote sensing image database;

[0009] S2, performing radiometric calibration, atmospheric correction and geometric correction on the optical image, performing complex data conversion, filtering and geocoding operations on the SAR image, and then resampling the optical image and the SAR image to make the optical image and the SAR image have the same resolution;

[0010] S3, marking the center point and radius of the offshore windmill in the optical image processed by S2 as a marking file of the training sample set; cutting out image blocks with the center point as the cutting center from the optical image and the SAR image processed by S2, respectively, to form a training sample set;

[0011] S4, inputting the training sample set into the multimodal offshore wind turbine recognition model for training to improve the fitting degree of the multimodal offshore wind turbine recognition model, and obtaining a trained multimodal offshore wind turbine recognition model;

[0012] S5. Input the target image into the trained multimodal offshore windmill recognition model, and output the offshore windmill recognition result, which includes the location information of the windmill in the remote sensing image. Based on the recognition result, the bright spots of the offshore windmill blades in the SAR image are subjected to morphological analysis and geometric calculation to obtain the offshore windmill position and orientation information.

[0013] Furthermore, in S4, the multimodal offshore wind turbine recognition model consists of an optical image branch and a SAR image branch, including an input layer, a convolution layer, a pooling layer, a feature fusion layer and an output layer.

[0014] Furthermore, S4 specifically includes the following steps:

[0015] S41, inputting the training sample set into the convolution layer through the input layer of the multimodal offshore wind turbine recognition model for feature extraction to obtain a feature map;

[0016] S42, performing a maximum pooling operation on the feature map, and reducing the size of the feature map by pooling;

[0017] S43, splicing the feature maps processed by S42 in the channel dimension to generate a fused feature map;

[0018] S44. Generate candidate regions on the fused feature map, generate position regression vectors and target classification scores through sliding window technology, perform ROI pooling operation on the candidate boxes generated by the region proposal network, standardize candidate boxes of different sizes to a fixed size, classify the pooled feature map after passing the fully connected layer, adjust the position and size of the bounding box through the regressor, and obtain the final windmill detection result.

[0019] Furthermore, S3 specifically includes the following steps:

[0020] S31. According to the position of the windmill, mark the center point (x, y) of each windmill and the marked windmill radius r1 in the SAR image to determine the range of the windmill and generate a marking file;

[0021] S32, respectively cut the optical image and the SAR image with the center point (x, y) and the cutting radius r2 to generate image blocks.

[0022] Furthermore, in S5, the steps of performing morphological analysis and geometric calculation are as follows:

[0023] S51. Use threshold segmentation and morphological operation to extract bright spot features in SAR images. Set a threshold T and mark the pixels in the SAR image that are greater than T as bright spots. The formula is as follows:

[0024]

[0025] Among them, I(x,y) is the pixel value of the original SAR image, and I′(x,y) is the binary image;

[0026] S52. Remove the noise by morphological opening operation and keep the three bright spots of the windmill blades. Let the structural element be B. The opening operation is defined as:

[0027]

[0028] in, represents the corrosion operation, represents the expansion operation, I open Represents the image after opening operation;

[0029] S53, obtaining the coordinates of three bright spots through S52, namely (x1, y1), (x2, y2), and (x3, y3), to determine the arrangement of the three bright spots;

[0030] S54. Calculate the direction of the windmill based on the arrangement determined in S53.

[0031] Furthermore, the method for determining the arrangement of three bright spots is as follows:

[0032] If the first rule is met, it is a straight line arrangement; if the second rule is met, it is an isosceles triangle arrangement;

[0033] The judgment conditions of the first rule are as follows:

[0034] For any two (x1, y1) and (x2, y2) the slope is:

[0035]

[0036] Calculate the slope m between the other two points 13 and m 23 , if the three points are arranged in a straight line, the slope should satisfy:

[0037] |m 12 -m 13 |≈0 and |m 12 -m 23 |≈0

[0038] The judgment conditions of the second rule are as follows:

[0039] The distance between points (x1,y1) and (x2,y2) is:

[0040]

[0041] Calculate the distance d between the other two points 13 and d 23 , if the three points are arranged in an isosceles triangle, the distance should satisfy:

[0042] d 12 ≈d 13 or 12 ≈d 23 or 13 ≈d 23 .

[0043] Further, S54 specifically includes the following steps:

[0044] For the first rule, the windmill orientation is calculated as follows:

[0045]

[0046] Among them, θ is the windmill direction;

[0047] For the second rule, the windmill orientation is calculated as follows:

[0048] Calculate the centroid G of the triangle using the following formula:

[0049]

[0050] Let the base be the line segment between (x1,y1) and (x2,y2), then the midpoint of the base is The windmill orientation is calculated as follows:

[0051]

[0052] Among them, G x and G y represents the x- and y-coordinates of the centroid of the isosceles triangle, M x and M yRespectively represent the x-coordinate and y-coordinate of the midpoint of the base.

[0053] The present invention also provides a storage medium, characterized in that the storage medium includes a stored program, wherein when the program is run, any of the above-mentioned multi-modal remote sensing offshore wind turbine identification methods is executed.

[0054] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes any one of the multi-modal remote sensing offshore wind turbine identification methods described above through the computer program.

[0055] Compared with the prior art, the present invention has the following advantages:

[0056] The present invention enhances the robustness and accuracy of windmill detection by integrating the features of optical images and SAR images and making full use of the complementarity of the two types of data. Optical images can provide clear visual information, while SAR images can clearly capture the reflection characteristics of windmills under complex weather conditions, such as clouds or at night. The combined use of the two overcomes the limitations of single-modality images, especially in low-visibility conditions, and improves the overall performance of windmill detection.

[0057] The present invention uses the center point and radius of the windmill to cut out the local area of ​​the SAR and optical images, thereby achieving accurate alignment of the images, simplifying the traditional geometric registration operation, reducing the amount of calculation and improving the processing efficiency. At the same time, the method ensures the consistency of the windmill targets in different modal images.

[0058] The present invention proposes a method for determining the direction of a windmill by using the bright spot features in SAR images. By performing morphological analysis and geometric calculation on the bright spots of windmill blades, the direction information of the windmill can be accurately determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0060] Figure 1 It is a flow chart of the multi-modal remote sensing offshore windmill identification method of the present invention;

[0061] Figure 2 This is a schematic diagram of the offshore windmill recognition results using optical images;

[0062] Figure 3 Schematic diagram of offshore windmill identification results from SAR images. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0064] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0065] like Figure 1 As shown, the present invention provides a multi-modal remote sensing offshore windmill identification method, comprising the following steps:

[0066] S1. Obtain optical images and SAR images of the same area in the remote sensing image database;

[0067] Specifically, optical and SAR images in Liaoning area were screened out from the remote sensing image database, including 100 GF-2 images and 50 GF-3 images.

[0068] S2, perform radiation calibration, atmospheric correction, and geometric correction on optical images, perform complex data conversion, filtering, and geocoding operations on SAR images, and resample the two to have the same resolution;

[0069] Specifically, the GF-2 images were subjected to radiometric calibration, atmospheric correction, and geometric correction, and the GF-3 images were subjected to complex data conversion, EnLee filtering, and geocoding operations to ensure the geometric consistency between the optical image and the SAR image. The preprocessed GF-2 and GF-3 images were resampled to 2 meters to have the same resolution.

[0070] S3, by marking the center point and radius of the offshore windmill in the optical image as a marking file, and then cutting the image block with the center point as the cutting center from the optical image and the SAR image respectively, to form a sample set;

[0071] Furthermore, in step S3, specifically:

[0072] S31. Using software (such as QGIS), according to the location of the windmill, mark the center point (x, y) of each windmill and the marked windmill radius r1 in the SAR image to determine the range of the windmill and generate a marking file;

[0073] S32, cutting images on the optical image and the SAR image at the same time with the center point (x, y) and the cutting radius r2 to generate image blocks;

[0074] S4, inputting the training sample set into the multimodal offshore wind turbine recognition model for training, and obtaining a multimodal offshore wind turbine recognition model with high fitting degree;

[0075] Specifically, the training process of the multimodal offshore wind turbine recognition model is as follows:

[0076] Input layer: The size of the optical image input by the optical image branch is 512×512×3, representing an RGB three-channel image; SAR image branch: The size of the input SAR image is 512×512×1, representing a single-channel grayscale image.

[0077] The convolution layer uses a 7×7 convolution kernel with a stride of 2, performs a convolution operation, and the size of the output feature map is 256×256×64. A 3×3 pooling kernel with a stride of 2 is used for the maximum pooling operation. Pooling reduces the size of the feature map and the output is 128×128×64.

[0078] Use the standard residual module, introduce skip connections to ensure that the gradient can be propagated forward more effectively, use a 3×3 convolution kernel, and use a total of 4 stages. The specific configuration is:

[0079] Stage 1: 3 residual modules, outputting a 128×128×256 feature map;

[0080] Stage 2: 4 residual modules, outputting a 64×64×512 feature map;

[0081] Stage 3: 6 residual modules, outputting a 32×32×1024 feature map;

[0082] Stage 4: 3 residual modules, outputting a 16×16×2048 feature map;

[0083] The feature fusion layer concatenates the feature maps (16×16×2048) extracted from the optical image branch and the SAR image branch in the channel dimension to generate a fused feature map of size 16×16×4096, and applies a 3×3 convolution operation to the concatenated feature map to compress the feature map to a dimension of 16×16×1024. This convolution layer achieves deep fusion of the two modal information and further removes redundant information.

[0084] Generate candidate regions on the fused feature map, generate position regression vectors and target classification scores through sliding window technology, perform ROI pooling operation on the candidate boxes generated by the region proposal network, standardize candidate boxes of different sizes to a fixed size, and classify the pooled feature map after passing the fully connected layer. Adjust the position and size of the bounding box through the regressor to obtain the final windmill detection result.

[0085] S5. Input the target image into the trained multimodal offshore windmill recognition model, and output the offshore windmill recognition result, which includes the location information of the windmill in the remote sensing image. Based on the recognition result, the bright spots of the offshore windmill blades in the SAR image are subjected to morphological analysis and geometric calculation to obtain the offshore windmill position and orientation information.

[0086] Specifically, in step S5, the bright spots of the wind turbine blades on the sea in the SAR image are subjected to morphological analysis and geometric calculation to obtain the orientation information of the offshore wind turbine. The specific steps are as follows:

[0087] S51. Use threshold segmentation and morphological operation to extract bright spot features in SAR images. Set a threshold T and mark the pixels in the SAR image that are greater than T as bright spots. The formula is as follows:

[0088]

[0089] Among them, I(x,y) is the pixel value of the original SAR image, and I′(x,y) is the binarized image.

[0090] S52. Remove the noise through morphological opening operation and keep the three bright spots of the windmill blades. Let the structural element be B. The opening operation is defined as:

[0091]

[0092] in, represents the corrosion operation, represents the expansion operation, I open Represents the image after opening operation;

[0093] S53, by using S52, the coordinates of the three bright spots are obtained, which are (x1, y1), (x2, y2), and (x3, y3), respectively, to determine the arrangement of the three points. If the first rule is met, it is a straight line arrangement; if the second rule is met, it is an isosceles triangle arrangement.

[0094] The judgment conditions of the first rule are as follows:

[0095] For any two (x1, y1) and (x2, y2) the slope is:

[0096]

[0097] Similarly, calculate the slope m between the other two points 13 and m 23 , if the three points are arranged in a straight line, the slope should satisfy:

[0098] |m 12 -m 13 |≈0 and |m 12 -m 23 |≈0

[0099] The judgment conditions of the second rule are as follows:

[0100] The distance between points (x1,y1) and (x2,y2) is:

[0101]

[0102] Similarly, calculate the distance d between the other two points 13 and d 23 , if the three points are arranged in an isosceles triangle, the distance should satisfy:

[0103] d 12 ≈d 13 or 12 ≈d 23 or 13 ≈d 23

[0104] S54 For the first rule, the windmill orientation is calculated as follows:

[0105]

[0106] Among them, θ is the direction of the windmill;

[0107] For the second rule, the windmill orientation is calculated as follows:

[0108] Calculate the centroid G of the triangle using the following formula:

[0109]

[0110] Let the base be the line segment between (x1,y1) and (x2,y2), then the midpoint of the base is The windmill orientation is calculated as follows:

[0111]

[0112] Among them, G x and G y represents the x- and y-coordinates of the centroid of the isosceles triangle, M x and M y Respectively represent the x-coordinate and y-coordinate of the midpoint of the base.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal remote sensing offshore windmill identification method, characterized in that: The steps include: S1. Obtain optical images and SAR images of the same area in the remote sensing image database; S2, performing radiometric calibration, atmospheric correction and geometric correction on the optical image, performing complex data conversion, filtering and geocoding operations on the SAR image, and then resampling the optical image and the SAR image to make the optical image and the SAR image have the same resolution; S3, marking the center point and radius of the offshore windmill in the optical image processed by S2 as a marking file of the training sample set; Cut out image blocks with the center point as the cutting center from the optical image and SAR image processed by S2 to form a training sample set; S4, inputting the training sample set into the multimodal offshore wind turbine recognition model for training to improve the fitting degree of the multimodal offshore wind turbine recognition model, and obtaining a trained multimodal offshore wind turbine recognition model; S5, inputting the target image into the trained multimodal offshore windmill recognition model, and outputting the offshore windmill recognition result, where the recognition result includes the location information of the windmill in the remote sensing image; Based on the recognition results, the position and orientation information of the offshore windmills are obtained by performing morphological analysis and geometric calculation on the bright spots of the offshore windmill blades in the SAR images.

2. The multi-modal remote sensing offshore windmill identification method according to claim 1 is characterized in that: In S4, the multimodal offshore windmill recognition model consists of an optical image branch and a SAR image branch, including an input layer, a convolution layer, a pooling layer, a feature fusion layer and an output layer.

3. The multi-modal remote sensing offshore windmill identification method according to claim 2 is characterized in that: S4 specifically includes the following steps: S41, inputting the training sample set into the convolution layer through the input layer of the multimodal offshore wind turbine recognition model for feature extraction to obtain a feature map; S42, performing a maximum pooling operation on the feature map, and reducing the size of the feature map by pooling; S43, splicing the feature maps processed by S42 in the channel dimension to generate a fused feature map; S44. Generate candidate regions on the fused feature map, generate position regression vectors and target classification scores through sliding window technology, perform ROI pooling operation on the candidate boxes generated by the region proposal network, standardize candidate boxes of different sizes to a fixed size, classify the pooled feature map after passing the fully connected layer, adjust the position and size of the bounding box through the regressor, and obtain the final windmill detection result.

4. The multi-modal remote sensing offshore windmill identification method according to claim 1 is characterized in that: S3 specifically includes the following steps: S31. According to the position of the windmill, mark the center point (x, y) of each windmill and the radius r1 of the marked windmill in the SAR image to determine the range of the windmill and generate a marking file; S32, respectively cut the optical image and the SAR image with the center point (x, y) and the cutting radius r2 to generate image blocks.

5. The multi-modal remote sensing offshore windmill identification method according to claim 1 is characterized in that: In S5, the steps for morphological analysis and geometric calculation are as follows: S51. Use threshold segmentation and morphological operation to extract bright spot features in SAR images. Set a threshold T and mark the pixels in the SAR image that are greater than T as bright spots. The formula is as follows: Among them, I(x,y) is the pixel value of the original SAR image, and I′(x,y) is the binary image; S52. Remove the noise by morphological opening operation and keep the three bright spots of the windmill blades. Let the structural element be B. The opening operation is defined as: in, represents the corrosion operation, ⊕ represents the expansion operation, I open Represents the image after opening operation; S53, obtaining the coordinates of three bright spots through S52, namely (x1, y1), (x2, y2), and (x3, y3), to determine the arrangement of the three bright spots; S54. Calculate the direction of the windmill based on the arrangement determined in S53.

6. The multi-modal remote sensing offshore windmill identification method according to claim 5 is characterized in that: The method for determining the arrangement of three bright spots is as follows: If the first rule is met, it is a straight line arrangement; if the second rule is met, it is an isosceles triangle arrangement; The judgment conditions of the first rule are as follows: For any two (x1, y1) and (x2, y2) the slope is: Calculate the slope m between the other two points 13 and m 23 , if the three points are arranged in a straight line, the slope should satisfy: |m 12 -m 13 |≈0 and |m 12 -m 23 |≈0 The judgment conditions of the second rule are as follows: The distance between points (x1,y1) and (x2,y2) is: Calculate the distance d between the other two points 13 and d 23 , if the three points are arranged in an isosceles triangle, the distance should satisfy: d 12 ≈d 13 or 12 ≈d 23 or 13 ≈d 23 .

7. The multi-modal remote sensing offshore windmill identification method according to claim 6 is characterized in that: S54 specifically includes the following steps: For the first rule, the windmill orientation is calculated as follows: Among them, θ is the windmill direction; For the second rule, the windmill orientation is calculated as follows: Calculate the centroid G of the triangle using the following formula: Let the base be the line segment between (x1,y1) and (x2,y2), then the midpoint of the base is The windmill orientation is calculated as follows: Among them, G x and G y represents the x- and y-coordinates of the centroid of the isosceles triangle, M x and M y Respectively represent the x-coordinate and y-coordinate of the midpoint of the base.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the multi-modal remote sensing offshore wind turbine identification method described in any one of claims 1 to 7 is executed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the multi-modal remote sensing offshore wind turbine identification method according to any one of claims 1 to 7 by running the computer program.

Images