Multi-task learning method for large-area wind driven generator identification

Abstract

The invention discloses a multi-task learning method for large-area wind driven generator identification. The method comprises the steps of constructing an enhanced feature fusion module, introducing shadow features and linear manual features to enhance depth visual features, and utilizing spatial distribution of wind driven generators to enhance attention to important areas. The fused features are subjected to scene classification and multi-task learning of target detection, a dichotomy model under a geographic space constraint condition is constructed for scene classification, land utilization data is introduced to improve the precision of scene classification, the false detection rate in a large-range detection process is reduced, and the detection speed is improved. And shallow feature learning is guided through the incremental learning module, so that the small target detection effect is improved. And based on the self-made data set training model, performing multi-task learning of the wind driven generator in a large-range scene. The method can improve the recognition and positioning speed and precision of the wind driven generator in a large-scale scene, provides a reliable basis for the development of wind power energy, evaluates the environmental impact, and guarantees the sustainable and friendly development of the wind power energy.

Description

Technical Field

[0001] The present invention belongs to the fields of remote sensing, deep learning, wind power energy and target detection, and specifically relates to a multi-task learning method for identifying wind turbines in a large area. Background Art

[0002] At present, the global warming situation is getting worse and worse. Energy-rich countries have made commitments and formulated a series of energy-saving and emission-reduction goals. my country attaches great importance to energy issues and has put forward a series of effective energy-saving and environmental protection policies, including corresponding guidance and requirements for new energy power generation technologies. Compared with other power generation technologies, wind power generation has the advantages of low cost, simple equipment, and fewer influencing factors, but the current wind power generation potential and development layout cannot be accurately quantified. Therefore, the target detection research of wind turbines is extremely important. For the detection and identification of wind turbines in a large area, the workload of manual survey is huge and it is not feasible. Remote sensing data has the advantage of not being restricted by time and location, and remote sensing data from various regions around the world can be directly downloaded. Therefore, the automatic recognition of remote sensing images can be used to replace the traditional manual method to achieve fast and efficient automatic detection of wind turbines in a large area.

[0003] At present, remote sensing-based wind turbine detection is mainly divided into two categories: (1) Traditional methods: such as reverse detection, template matching, and target detection based on prior knowledge. These methods require a lot of prior knowledge and complex detection processes. For example, reverse detection requires prior knowledge such as solar altitude angle, azimuth angle, and tower height. Template matching methods require the use of artificially collected samples to generate templates and calculate similarity. These methods rely on a lot of prior knowledge, resulting in poor generalization performance of the model. (2) Deep learning methods: Deep learning technology uses multi-layer neural networks to extract target features. The detection framework is generally divided into three parts: feature extraction, feature fusion, and classification prediction. The two processes of target classification and positioning are performed. However, due to resources such as terrain, installation costs, and wind conditions, wind turbines are sparsely distributed. They are mostly distributed in open mountains, cultivated land, coastal areas, and far away from urban centers. In large-scale remote sensing images, wind turbines account for a small proportion. When detected in tile form, most of them belong to non-target areas, which will consume a lot of computing resources to classify and locate non-target areas, and may even lead to false detection. In addition, the proportion of wind turbine target pixels in remote sensing images is small, and most of them are small target objects. When using deep learning models to extract features, small target features are difficult to extract, which can easily lead to target misses.

[0004] In view of the above problems, this paper proposes a new YOLO-based deep learning model, which introduces shadow and linear features to enhance the information of wind turbines, and uses geographic constraints and incremental learning methods for multi-task learning, which is used for rapid positioning and accurate extraction of wind turbines in large-scale remote sensing scenes. It provides a basis for spatial planning decisions of wind turbines, evaluates environmental impacts, ensures that energy needs are met, and achieves friendly and sustainable development. Summary of the invention

[0005] In order to improve the detection efficiency and accuracy of wind turbines in large-scale remote sensing scenes, the present invention aims to provide a multi-task learning method for large-area wind turbine identification. Based on the YOLO basic model, shadow features and linear manual features are introduced, and the cross-attention enhancement module is used to enhance the deep visual features to improve the feature expression ability of the model. Multi-task learning of scene classification and target detection is constructed. The detection task is supervised by the scene classification task in the training stage, and the detection task is assisted in the prediction and reasoning stage to achieve rapid detection of wind turbines in large-scale scenes. Incremental learning modules are constructed in the detection task to enhance shallow features and improve the detection accuracy of small targets. The extraction speed and accuracy of wind turbines in large-scale scenes are improved to provide a reliable basis for wind power energy development planning.

[0006] In order to solve the technical problem, the technical solution of the present invention is:

[0007] A multi-task learning method for large-area wind turbine identification, the method comprising:

[0008] A wind turbine sample data set is obtained, and an enhanced multi-task network model based on YOLO is constructed; the multi-task network model is trained using a training set and a validation set in the sample data set; and the trained optimal network model is used to identify and extract a large range of wind turbines.

[0009] Further, the obtaining of the wind turbine sample data set includes:

[0010] Download remote sensing images. The original remote sensing images cover various land cover types such as coastal areas, cultivated land, and shrub forests. Use Python methods to crop RGB images into image blocks of 640×640×3;

[0011] LabelImg software is used to label the cropped image blocks, and an extensible markup language file containing the wind turbine category information and rectangular bounding box coordinates is created. The category and bounding box coordinate information in the xml file is extracted to generate a text file.

[0012] Use the Mosaic method to perform data enhancement processing and process the corresponding label files at the same time to expand the sample data and enhance the generalization ability of the model;

[0013] The sample data set is randomly divided into a training set and a validation set in a 7:3 ratio. The training set is used to train the multi-task network model, and the validation set is used to test the training accuracy and generalization ability of the model.

[0014] Further, the construction of an enhanced multi-task network model based on YOLO includes:

[0015] Multi-scale feature extraction: Based on the backbone of the YOLO network, the multi-scale deep features of the input image block are extracted in 80×80, 40×40, and 20×20 dimensions as the basis for classification and positioning;

[0016] Manual feature fusion: Extract shadow index and linear features as important auxiliary information for wind turbine scenes; use the cross-attention mechanism to fuse manual features, i.e., shadow and linear features, with deep visual features to improve feature expression capabilities;

[0017] Scene classification: Use geographic spatial constraints, i.e. land use data, to fuse with visual features and output scene classification results through a multi-layer perceptron; use cross-attention mechanism and global pooling to improve classification accuracy;

[0018] Incremental learning: Deep features guide shallow features layer by layer, integrating 20×20, 40×40 and 80×80 features to improve the ability to detect small targets; using cross-attention to gradually enhance shallow features, making the model more accurate in detecting small targets of wind turbines.

[0019] Furthermore, the construction of an enhanced multi-task network model based on YOLO specifically includes:

[0020] The YOLO original backbone feature extraction module is used to extract multi-scale deep features of the cropped image blocks. The extracted feature sizes are 80×80, 40×40, and 20×20.

[0021] The shadow index feature and linear feature of the input image are extracted. The wind turbine recognition scene includes shadow information. The spatial relationship between the shadow and the wind turbine body is used to assist in determining the wind turbine location information through shadow features. Wind turbines are built along the edge of independent linear roads in cultivated land boundaries or shrub forests. The linear features are extracted through Hough transform to effectively improve the positioning accuracy of wind turbines. First, a simple splicing method is used to fuse two manual features. The fused manual features need to use a convolution function with a convolution kernel of 1×1 to adjust the feature size to match the multi-scale deep features. The cross-attention method is used to introduce them into the multi-scale deep features, that is, the manual features generate Q sequences, the visual features generate K and V sequences, Q and K generate attention weight matrices, and then multiply them with V to obtain the fused features. The fused features are enhanced by the self-attention method. The fusion process can be defined as formula (1-3):

[0022] SI=|RED+GREEN-2×BLUE| (1)

[0023]

[0024] Feature out =Attention self (F′) (3)

[0025] In the formula, RED, GREEN and BLUE represent the information of the red, green and blue bands of the image respectively, SI represents the shadow index feature, Hough represents the Hough transform, Input represents the input image, and Feature vision represents visual features, ⊕ represents concatenation operation, Indicates element-by-element multiplication, Attention cross and Attention self They represent the cross attention mechanism and the self-attention mechanism respectively, and F′ represents the process value;

[0026] The fused features are used for scene classification tasks under geographic space constraints, and the 2-meter resolution land use data is used as a constraint to classify the scene tasks. First, the cross-attention mechanism is used to fuse the visual features and the single-channel land use data, that is, the land use data generates the Q sequence, the visual features generate the K and V sequences, and the Q and K generate the attention weight matrix, which is then multiplied with V to obtain the fused features. The fused features are compressed by a 1×1 convolution operation, and then the global average pooling and global maximum pooling operations are used to obtain one-dimensional features as the input of the multi-layer perceptron, and the multi-layer perceptron outputs the scene classification results. The classification process can be defined as formula (4-5):

[0027] Feature′=Attention cross (Feature vision ,Feature LU ) (4)

[0028] Class = MLP(GAP(Conv 1×1 (Feature′))⊕GMP(Conv 1×1 (Feature′))) (5)

[0029] In the formula, Feature LU Land use data features, GAP and GMP represent global average pooling and global maximum pooling respectively, MLP represents multi-layer perceptron, Conv 1×1 It represents the convolution function with a convolution kernel of 1×1, ⊕ represents the concatenation operation, and Feature′ represents the process value;

[0030] The multi-scale features after fusion of manual features improve the detection ability of small targets under the incremental learning module. The enhanced feature scales are 80×80, 40×40 and 20×20. Shallow features are used for target detection. The incremental learning module is designed to guide deep features layer by layer to enhance shallow features. First, the deep features with a size of 20×20 are subjected to two 3×3 convolutions to generate category features and bounding box features. The feature size is adjusted using a 1×1 standard convolution. Then, the cross-attention mechanism is used to fuse the enhanced features with a size of 40×40, and the enhanced features are concatenated to generate a feature with a size of 40×40. 40×40 ′; Enhanced Feature 40×40 ′After two 3×3 convolutions, the category features and bounding box features are generated again, and the feature size is adjusted using a 1×1 standard convolution, and then they are respectively combined with the shallow feature Feature 80×80 The fusion is performed through the cross attention mechanism, and the fused features are concatenated to obtain an enhanced shallow feature with a size of 80×80. 80×80 ′, and then enters the detection head as input to output the category and bounding box results; the incremental learning method uses deep information to guide shallow feature learning layer by layer, and effectively uses deep information to improve the detection accuracy of small targets; the incremental learning process can be defined as formula (6-9):

[0031] (Feature cls ,Feature Bbox )=Conv 3×3 (Conv 3×3 (Feature 20×20 )) (6)

[0032] Feature 40×40 ′=Attention cross (Feature 40×40 ,Feature cls )

[0033] ⊕Attention cross (Feature 40×40 ,Feature Bbox ) (7)

[0034] (Feature cls ′,Feature Bbox ′)=Conv 3×3 (Conv 3×3 (Feature 40×40 ′)) (8)

[0035] Feature80×80 ′=Attention cross (Feature 40×40 ′,Feature clss ′)

[0036] ⊕Attention cross (Feature 40×40 ′,Feature Bbox ′) (9)

[0037] In the formula, Conv 3×3 Indicates a convolution function with a convolution kernel of 3×3, Attention cross represents the cross attention mechanism, and ⊕ represents the concatenation operation.

[0038] Further, the multi-task network model is trained using the training set and the validation set, including:

[0039] Training data preparation: During training, the enhanced remote sensing image data is used, combined with a text file containing category and bounding box information, to further extract the shadow features and linear features of the image. Then, the multi-task network based on YOLO is trained.

[0040] Loss function design: The loss function of the model consists of two parts: scene classification loss: used to evaluate the performance of the model on the scene classification task, using the binary cross entropy loss function to measure the accuracy of classification; object detection loss: including category loss and bounding box regression loss. The bounding box loss is defined by calculating the overlap between the predicted box and the true box, i.e., the intersection-over-union ratio, as well as the shape and position differences of the box.

[0041] Training optimization: During the training process, set the number of training times. In the last few iterations, stop Mosaic data enhancement to avoid distribution differences in background data during training, so that the model can better adapt to the test set data and reduce the risk of overfitting. Use the stochastic gradient descent algorithm SGD to optimize the model. By continuously adjusting the model parameters, the training error can be gradually reduced to improve the accuracy of the model.

[0042] Further, the multi-task network model is trained using the training set and the validation set, specifically including:

[0043] Using the 640×640×3 remote sensing image after image data enhancement and the text file containing category and bounding box information, as well as the shadow features extracted by band information and the linear features extracted by Hough transform, the YOLO-based multi-task model is trained according to step B. The loss function of the multi-task network is composed of the loss functions of the scene classification part and the target detection part, as shown in formula (10):

[0044] L=L Sence +L Detect (10)

[0045] Among them, L Sence is the loss function of the scene classification part, L Detect is the loss function of the target detection part, and L is the sum of the loss functions of the model, which is used to represent the overall error of the model;

[0046] The loss function of the scene classification part is a binary classifier loss function, which is expressed by a binary cross entropy loss function, as shown in formula (11):

[0047]

[0048] Where y i * is the true category corresponding to the pixel, y i The pixel result predicted by the model;

[0049] The loss function of the target detection part is divided into a category loss function and a target bounding box regression loss function. The bounding box loss is composed of the intersection-over-union loss CIoU loss and the distribution focus loss DFL loss between the real box and the predicted box, as shown in formula (12-16):

[0050] L Detect =L cls +L reg (12)

[0051]

[0052] L reg =L DFL +L CIoU (14)

[0053]

[0054]

[0055] Where p j * is the true category of the bounding box j. If the candidate box is positive, then p j * That is 1. If the candidate box is negative, then p j * is 0, p j is the bounding box prediction probability; t j and t j * They are the four coordinate parameters of the predicted bounding box and the coordinate parameters of the true value box related to the predicted bounding box. Softmax represents the activation function. Nreg Represents the number of positive samples; IoU is the intersection of the predicted box and the real box, ρ 2 (b,b * ) The Euclidean distance between the center point of the predicted box and the real box, c is the diagonal length of the minimum containing box, υ is the difference in aspect ratio, which is used to measure the shape difference between the predicted box and the real box, and α is the adjustment factor to balance the center point and shape error;

[0056] Set the number of epochs for model training. Stop Mosaic data augmentation in the last 10 epochs to train the model in a more realistic single-image background to reduce difference distribution, improve the performance of the model on the test set, and reduce the risk of overfitting. Use the stochastic gradient descent SGD optimization algorithm to optimize and update the model parameters, that is, the connection weights between layers in the network, and gradually reduce the loss function, so that the model can better fit the training data.

[0057] Furthermore, the trained optimal network model is applied to identify and extract a wide range of wind turbines, including:

[0058] Test data processing: The image of the test area to be identified is cropped without overlap into image blocks of 640×640×3;

[0059] Model input and feature extraction: The cropped image blocks are input into the enhanced multi-task network model, and the YOLO backbone network is used to extract multi-scale deep features, while integrating the image's shadow features and linear manual features;

[0060] Task execution, scene classification task: classify wind turbine scenes under geographic space constraints; target detection task: improve the detection capability of small target wind turbines by fusing deep features and shallow features layer by layer through incremental learning modules;

[0061] Result output: The model finally outputs the positioning and identification results of wind turbines in a large-scale scene.

[0062] Compared with the prior art, the advantages of the present invention are:

[0063] (1) After extracting multi-scale deep features from the main features, the present invention introduces shadow index and linear features, and uses the spatial relationship between the shadow and the main body of the wind turbine to assist in determining the location information of the wind turbine through shadow features. Wind turbines are usually built along the boundaries of cultivated land or near independent linear roads in shrub forests. The distribution area of ​​wind turbines can be effectively located through shadow and linear features, irrelevant information can be suppressed, and the positioning accuracy of wind turbines can be improved.

[0064] (2) The present invention proposes multi-task learning for scene classification, uses the fused enhanced features to perform scene classification tasks under geographical constraints, uses 2-meter land use data features to improve the accuracy of scene classification, and uses a multi-layer perceptron to detect whether an image block contains a wind turbine. In addition, the detection task of the scene classification supervision model can be used in the training stage, and auxiliary detection can be performed according to the scene classification results in the prediction stage to avoid detecting image blocks without targets, thereby improving detection efficiency and accuracy.

[0065] (3) The incremental learning module proposed in the present invention in the detection task mainly uses deep features through incremental learning to guide shallow feature learning, fully utilizes deep information, effectively improves the expression of small target features in shallow features, and improves the detection accuracy of small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 , enhanced multi-task network model;

[0067] Figure 2 , manual feature enhancement fusion module;

[0068] Figure 3 ,Scene classification module under geographic space constraints;

[0069] Figure 4 , incremental learning module. DETAILED DESCRIPTION

[0070] The specific implementation mode of the present invention is described below in conjunction with embodiments:

[0071] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0072] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0073] Embodiment 1:

[0074] In order to improve the recognition and positioning speed and accuracy of wind turbines in large-scale scenes, the present invention proposes a multi-task learning method for large-area wind turbine recognition. First, a sample data set is produced, which is an original remote sensing image with a resolution of 2m, including three bands of red, green and blue. The original image is cropped to a size of 640×640, and a label text file (txt file) containing the category and bounding box coordinates is produced. Shadow features and linear manual features are calculated based on the image, and then the feature fusion module is used to fuse the manual features and multi-scale features. The fused features can perform scene classification tasks under geographic space constraints to improve the detection speed, and on the other hand, an incremental learning module is constructed to perform target detection tasks to improve the detection accuracy of small targets.

[0075] A multi-task learning method for large-area wind turbine identification includes the following steps:

[0076] Step S1: Create a wind turbine sample data set;

[0077] Step S2: Build an enhanced multi-task network model based on YOLO;

[0078] Step S3: training the multi-task network model using the training set and the validation set;

[0079] Step S4: Apply the trained optimal network model to identify and extract a wide range of wind turbines.

[0080] S1: Create a wind turbine sample dataset: Create a wind turbine sample dataset, which includes RGB images of size 640×640×3 and corresponding image label files. The method is as follows:

[0081] S101: Download remote sensing images. The original remote sensing images cover various land cover types such as coastal areas, cultivated land, shrub forests, etc. Use Python method to crop the RGB images into image blocks of 640×640×3.

[0082] S102: Label the cropped image blocks using LabelImg software to create an extensible markup language file (xml file) containing wind turbine category information and rectangular bounding box coordinates, and extract the category and bounding box coordinate information in the xml file to generate a text file (txt file).

[0083] S103: Use the Mosaic method to perform data enhancement processing and process the corresponding label files at the same time to expand the sample data and enhance the generalization ability of the model.

[0084] S104: The sample data set is randomly divided into a training set and a validation set in a 7:3 ratio. The training set is used to train the multi-task network model, and the validation set is used to test the training accuracy and generalization ability of the model.

[0085] S2. Build an enhanced multi-task network model based on YOLO, such as Figure 1 shown.

[0086] S201: YOLO is a basic network for completing wind turbine scene classification and positioning tasks. The present invention uses the YOLO original backbone feature extraction module to extract multi-scale depth features of the cropped image blocks. The extracted feature sizes are 80×80, 40×40 and 20×20.

[0087] S202: Extract shadow index features and linear features of the input image. Wind turbine recognition scenes usually contain shadow information. The spatial relationship between the shadow and the main body of the wind turbine is used to assist in determining the location information of the wind turbine through shadow features, thereby improving the robustness of the model. In addition, wind turbines are usually built along the boundaries of cultivated land or the edges of independent linear roads in shrubbery. The linear features extracted by Hough transform can effectively improve the positioning accuracy of wind turbines. First, a simple splicing method is used to fuse two manual features. The fused manual features need to use a convolution function with a convolution kernel of 1×1 to adjust the feature size to match the multi-scale deep features. The cross-attention method is used to introduce them into the multi-scale deep features, that is, the manual features generate Q sequences, the visual features generate K and V sequences, Q and K generate attention weight matrices, and then multiply them with V to obtain the fused features. The fused features are enhanced by the self-attention method, thereby improving the expression ability of the multi-scale deep features (such as Figure 2 As shown). The fusion process can be defined as formula (1-3):

[0088] SI=|RED+GREEN-2×BLUE| (1)

[0089]

[0090] Feature out =Attention self (F′) (3)

[0091] In the formula, RED, GREEN and BLUE represent the information of the red, green and blue bands of the image respectively, SI represents the shadow index feature, Hough represents the Hough transform, Input represents the input image, and Feature vision represents visual features, ⊕ represents concatenation operation, Indicates element-by-element multiplication, Attention cross and Attention selfThey represent the cross attention mechanism and self-attention mechanism respectively, and F′ represents the process value.

[0092] S203: The fused features are used to perform scene classification tasks under geographic space constraints, and the land use data with a resolution of 2 meters is used as a constraint to perform scene task classification. First, the cross-attention mechanism is used to fuse the visual features and the single-channel land use data, that is, the land use data generates the Q sequence, the visual features generate the K and V sequences, and the Q and K generate the attention weight matrix, which is then multiplied with V to obtain the fused features. The fused features are compressed through a 1×1 convolution operation, and then the global average pooling and global maximum pooling operations are used to obtain one-dimensional features as the input of the multi-layer perceptron, and the multi-layer perceptron outputs the scene classification results. The classification process can be defined as formula (4-5):

[0093] Feature′=Attention cross (Feature vision ,Feature LU ) (4)

[0094] Class = MLP(GAP(Conv 1×1 (Feature′))⊕GMP(Conv 1×1 (Feature′))) (5)

[0095] In the formula, Feature LU Land use data features, GAP and GMP represent global average pooling and global maximum pooling respectively, MLP represents multi-layer perceptron, Conv 1×1 It represents the convolution function with a convolution kernel of 1×1, ⊕ represents the concatenation operation, and Feature′ represents the process value.

[0096] S204: The multi-scale features after fusion of manual features improve the detection ability of small targets under the incremental learning module. The enhanced feature scales are 80×80, 40×40 and 20×20. Since the proportion of wind turbine pixels in 2-meter resolution images is small, this patent uses shallow features for target detection. At the same time, because deep information also contains effective information, it can enhance shallow features. Therefore, an incremental learning module is designed to guide deep features layer by layer to enhance shallow features. First, the deep features with a size of 20×20 are subjected to two 3×3 convolutions to generate category features and bounding box features. The feature size is adjusted using a 1×1 standard convolution. Then, the cross-attention mechanism is used to fuse the enhanced features with a size of 40×40. The enhanced features are spliced ​​to generate a feature with a size of 40×40. 40×40 ′. Enhanced Feature 40×40′After two 3×3 convolutions, the category features and bounding box features are generated again, and the feature size is adjusted using a 1×1 standard convolution, and then they are respectively combined with the shallow feature Feature 80×80 The fusion is performed through the cross attention mechanism, and the fused features are concatenated to obtain an enhanced shallow feature with a size of 80×80. 80×80 ′, and then enters the detection head as input to output the category and bounding box results. The incremental learning method uses deep information to guide shallow feature learning layer by layer, effectively using deep information to improve the detection accuracy of small targets (such as Figure 4 As shown). The incremental learning process can be defined as formula (6-9):

[0097] (Feature cls ,Feature Bbox )=Conv 3×3 (Conv 3×3 (Feature 20×20 )) (6)

[0098] Feature 40×40 ′=Attention cross (Feature 40×40 ,Feature cls )

[0099] ⊕Attention cross (Feature 40×40 ,Feature Bbox ) (7)

[0100] (Feature cls ′,Feature Bbox ′)=Conv 3×3 (Conv 3×3 (Feature 40×40 ′)) (8)

[0101] Feature 80×80 ′=Attention cross (Feature 40×40 ′,Feature cls ′)

[0102] ⊕Attention cross (Feature 40×40 ′,Feature Bbox ′) (9)

[0103] Where Conv 3×3 Indicates a convolution function with a convolution kernel of 3×3, Attentioncross represents the cross attention mechanism, and ⊕ represents the concatenation operation.

[0104] S3: Use the training set and validation set to train the multi-task network model. The specific steps include:

[0105] S301: Using the 640×640×3 remote sensing image after image data enhancement and the text file containing the category and bounding box information, as well as the shadow features extracted by band information and the linear features extracted by Hough transform, the YOLO-based multi-task model is trained according to step B. The loss function of the multi-task network is composed of the loss functions of the scene classification part and the target detection part, as shown in formula (10):

[0106] L=L Sence +L Detect (10)

[0107] Among them, L Sence is the loss function of the scene classification part, L Detect is the loss function of the target detection part, and L is the sum of the loss functions of the model, which is used to represent the overall error of the model.

[0108] The loss function of the scene classification part is a binary classifier loss function, which is expressed by a binary cross entropy loss function, as shown in formula (11):

[0109]

[0110] Where y i * is the true category corresponding to the pixel, y i The pixel result predicted by the model.

[0111] The loss function of the target detection part is divided into a category loss function and a target bounding box regression loss function. The bounding box loss is composed of the intersection-over-union loss CIoU loss and the distribution focus loss DFL loss between the real box and the predicted box, as shown in formula (12-16):

[0112] L Detect =L cls +L reg (12)

[0113]

[0114] L reg =L DFL +L CIoU (14)

[0115]

[0116]

[0117] Where p j * is the true category of the bounding box j. If the candidate box is positive, then p j * That is 1. If the candidate box is negative, then p j * is 0, p j Predict the probability for the bounding box. j and t j * They are the four coordinate parameters of the predicted bounding box and the coordinate parameters of the true value box related to the predicted bounding box. Softmax represents the activation function. N reg Represents the number of positive samples. The intersection-over-union ratio of the IoU prediction box and the real box, ρ 2 (b,b * ) The Euclidean distance between the center point of the predicted box and the real box, c is the diagonal length of the minimum containing box, and v is the difference in aspect ratio, which is used to measure the shape difference between the predicted box and the real box. α is the adjustment factor to balance the center point and shape errors.

[0118] S302: Set the number of epochs for model training. Stop Mosaic data augmentation in the last 10 epochs to train the model in a more realistic single-image background to reduce the difference distribution, improve the performance of the model on the test set, and reduce the risk of overfitting. Use the Stochastic Gradient Descent (SGD) optimization algorithm to optimize and update the parameters of the model, that is, the connection weights between layers in the network, and gradually reduce the loss function, so that the model can better fit the training data.

[0119] S4: Apply the trained enhanced multi-task network model to locate and identify a large range of wind turbines, including:

[0120] S401: The images of the test area are cropped into image blocks of 640×640×3 without overlap, and input into the enhanced multi-task network model. Multi-scale deep information is extracted through the YOLO backbone network, and shadow features and linear manual features are fused. The fused features are used for scene classification tasks under geographic spatial constraints on the one hand, and target detection tasks are performed under the incremental learning module on the other hand. Finally, the positioning and recognition results of wind turbines in large-scale scenes are output.

[0121] Embodiment 2:

[0122] A multi-task learning method for large-area wind turbine identification, the specific operation steps are as follows:

[0123] A. Create a wind turbine sample data set:

[0124] A1. Download high-resolution remote sensing images of Beijing, Fujian Province, Shanxi Province, etc., including complex backgrounds such as coastal areas, cultivated land, and shrub forests, and perform radiation calibration, atmospheric correction, geometric correction, orthorectification, data fusion, etc., to finally obtain RGB three-band images with a spatial resolution of 2m.

[0125] A2. Crop the image to a size of 640×640×3 and use the labeling software LabelImg to create sample labels. The generated label file is in XML format and then converted into a text file containing the category and bounding box coordinate information.

[0126] A3. Perform the same Mosaic data augmentation processing on the cropped remote sensing images and the corresponding label files to expand the sample size and enhance the generalization ability of the model.

[0127] A4. Divide the data-augmented samples into a training set and a validation set in a ratio of 7:3. Input them into the network model. The training set is used to train the multi-task network model, while the validation set is used to test the accuracy of the model.

[0128] B. Build an enhanced multi-task network model based on YOLO

[0129] B1. The feature extraction of the built multi-task network model mainly relies on the backbone feature extraction part of the YOLO model, and the extracted multi-scale features, shadow features and linear features are fused ( Figure 2 ), based on the shadow features and linear features extracted from the original image, a manual feature enhancement module is used to fuse the multi-scale features extracted from the backbone to enhance the performance of important features and suppress irrelevant information.

[0130] B2. Process the enhanced features for scene classification tasks ( Figure 3 ), firstly, the cross attention mechanism is used to fuse the land use data features and visual features. The fused features are reduced in dimension through 1×1 standard convolution, and then one-dimensional features are obtained through global average pooling and global maximum pooling. The global features are extracted while maintaining the core information of the feature map. The processed features are used for scene classification using a multi-layer perceptron, and the output result is whether the image block contains a wind turbine target.

[0131] B3. The enhanced multi-scale features use the incremental learning module to enhance the shallow features and improve the detection accuracy of small targets ( Figure 4). The deep features with a size of 20×20 are subjected to two convolution operations with a convolution kernel of 3×3 to obtain the category and bounding box features. The sizes of these two features are adjusted to 40×40 through a 1×1 standard convolution. The middle-level features are then enhanced using the cross-attention mechanism. The enhanced middle-level features are then subjected to two 3×3 standard convolutions to obtain the category features and bounding box features. The feature size is adjusted to 80×80 using a 1×1 standard convolution. The shallow features are enhanced using cross-attention, and then feature splicing is performed to perform the target detection task. The features obtained through the incremental learning module twice are then input into the detection head to obtain the category and bounding box results.

[0132] C. Use the wind turbine dataset to train the network model:

[0133] C1. The multi-task model proposed in the present invention is trained using the prepared wind turbine sample data set, and the classification and positioning accuracy of the model is tested using the validation set. The experimental parameter settings are shown in Table 1, and the experimental server performance is shown in Table 2.

[0134] Table 1 Network parameter settings

[0135]

[0136] Table 2 Server configuration

[0137]

[0138] The experiment set the epoch to 300, and Mosaic data enhancement was turned off in the last 10 epochs to improve the model's adaptability to the real data distribution and ensure the optimization of the final performance. The experiment used four indicators, Precision, Recall, mAP50 and mAP50-95, to test the accuracy of the model. After multiple iterations, the weight model with the highest accuracy was selected to identify and locate a large range of wind turbines.

[0139] D. Finally, use the trained network model to identify wind turbines in 5,000 square kilometers in Hunan Province:

[0140] The remote sensing images of Hunan Province were cropped without overlap to a size of 640×640×3 as the input of the network model. The deep features were extracted through the YOLO backbone network, and the band information and Hough transform were used to extract manual features. The fused features were used to perform scene classification tasks under geographic spatial constraints, and target detection tasks under the incremental learning module. Finally, the scene classification binary results of the study area and the identification and positioning results of wind turbines were obtained.

[0141] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

[0146] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A multi-task learning method for large-area wind turbine identification, characterized in that: The method comprises: A wind turbine sample data set is obtained, and an enhanced multi-task network model based on YOLO is constructed; the multi-task network model is trained using a training set and a validation set in the sample data set; and the trained optimal network model is used to identify and extract a large range of wind turbines.

2. A multi-task learning method for large-area wind turbine identification according to claim 1, characterized in that: The step of obtaining a wind turbine sample data set includes: Download remote sensing images. The original remote sensing images cover various land cover types such as coastal areas, cultivated land, and shrub forests. Use Python methods to crop RGB images into image blocks of 640×640×3; LabelImg software is used to label the cropped image blocks, and an extensible markup language file containing the wind turbine category information and rectangular bounding box coordinates is created. The category and bounding box coordinate information in the xml file is extracted to generate a text file. Use the Mosaic method to perform data enhancement processing and process the corresponding label files at the same time to expand the sample data and enhance the generalization ability of the model; The sample data set is randomly divided into a training set and a validation set in a 7:3 ratio. The training set is used to train the multi-task network model, and the validation set is used to test the training accuracy and generalization ability of the model.

3. A multi-task learning method for large-area wind turbine identification according to claim 1, characterized in that: The construction of an enhanced multi-task network model based on YOLO includes: Multi-scale feature extraction: Based on the backbone of the YOLO network, the multi-scale deep features of the input image block are extracted in 80×80, 40×40, and 20×20 dimensions as the basis for classification and positioning; Manual feature fusion: Extract shadow index and linear features as important auxiliary information for wind turbine scenes; use the cross-attention mechanism to fuse manual features, i.e., shadow and linear features, with deep visual features to improve feature expression capabilities; Scene classification: Use geographic spatial constraints, i.e. land use data, to fuse with visual features and output scene classification results through a multi-layer perceptron; use cross-attention mechanism and global pooling to improve classification accuracy; Incremental learning: Deep features guide shallow features layer by layer, integrating 20×20, 40×40 and 80×80 features to improve the ability to detect small targets; using cross-attention to gradually enhance shallow features, making the model more accurate in detecting small targets of wind turbines.

4. A multi-task learning method for large-area wind turbine identification according to claim 3, characterized in that: The construction of the enhanced multi-task network model based on YOLO specifically includes: The YOLO original backbone feature extraction module is used to extract multi-scale deep features of the cropped image blocks. The extracted feature sizes are 80×80, 40×40, and 20×20. The shadow index feature and linear feature of the input image are extracted. The wind turbine recognition scene includes shadow information. The spatial relationship between the shadow and the wind turbine body is used to assist in determining the wind turbine location information through shadow features. Wind turbines are built along the edge of independent linear roads in cultivated land boundaries or shrub forests. The linear features are extracted through Hough transform to effectively improve the positioning accuracy of wind turbines. First, a simple splicing method is used to fuse two manual features. The fused manual features need to use a convolution function with a convolution kernel of 1×1 to adjust the feature size to match the multi-scale deep features. The cross-attention method is used to introduce them into the multi-scale deep features, that is, the manual features generate Q sequences, the visual features generate K and V sequences, Q and K generate attention weight matrices, and then multiply them with V to obtain the fused features. The fused features are enhanced by the self-attention method. The fusion process can be defined as formula (1-3): SI=|RED+GREEN-2×BLUE| (1) Feature out =Attention self (F′) (3) In the formula, RED, GREEN and BLUE represent the information of the red, green and blue bands of the image respectively, SI represents the shadow index feature, Hough represents the Hough transform, Input represents the input image, and Feature vision Represents visual features, Represents a splicing operation, Indicates element-by-element multiplication, Attention cross and Attention self They represent the cross attention mechanism and the self-attention mechanism respectively, and F′ represents the process value; The fused features are used for scene classification tasks under geographic space constraints, and the 2-meter resolution land use data is used as a constraint to classify the scene tasks. First, the cross-attention mechanism is used to fuse the visual features and the single-channel land use data, that is, the land use data generates the Q sequence, the visual features generate the K and V sequences, and the Q and K generate the attention weight matrix, which is then multiplied with V to obtain the fused features. The fused features are compressed by a 1×1 convolution operation, and then the global average pooling and global maximum pooling operations are used to obtain one-dimensional features as the input of the multi-layer perceptron, and the multi-layer perceptron outputs the scene classification results. The classification process can be defined as formula (4-5): Feature′=Attention cross (Feature vision ,Feature LU ) (4) In the formula, Feature LU Land use data features, GAP and GMP represent global average pooling and global maximum pooling respectively, MLP represents multi-layer perceptron, Conv 1×1 represents a convolution function with a convolution kernel of 1×1. represents the splicing operation, and Feature′ represents the process value; The multi-scale features after fusion of manual features improve the detection ability of small targets under the incremental learning module. The enhanced feature scales are 80×80, 40×40 and 20×20. Shallow features are used for target detection. The incremental learning module is designed to guide deep features layer by layer to enhance shallow features. First, the deep features with a size of 20×20 are subjected to two 3×3 convolutions to generate category features and bounding box features. The feature size is adjusted using a 1×1 standard convolution. Then, the cross-attention mechanism is used to fuse the enhanced features with a size of 40×40, and the enhanced features are concatenated to generate a feature with a size of 40×40. 40×40 ′; Enhanced Feature 40×40 ′After two 3×3 convolutions, the category features and bounding box features are generated again, and the feature size is adjusted using a 1×1 standard convolution, and then they are respectively combined with the shallow feature Feature 80×80 The fusion is performed through the cross attention mechanism, and the fused features are concatenated to obtain an enhanced shallow feature with a size of 80×80. 80×80 ′, and then enters the detection head as input to output the category and bounding box results; the incremental learning method uses deep information to guide shallow feature learning layer by layer, and effectively uses deep information to improve the detection accuracy of small targets; the incremental learning process can be defined as formula (6-9): (Feature cls ,Feature Bbox )=Conv 3×3 (Conv 3×3 (Feature 20×20 )) (6) Feature 40×40 ′=Attention cross (Feature 40×40 ,Feature cls )⊕Attention cross (Feature 40×40 ,Feature Bbox ) (7) (Feature cls ′,Feature Bbox ′)=Conv 3×3 (Conv 3×3 (Feature 40×40 ′)) (8) Feature 80×80 ′=Attention cross (Feature 40×40 ′,Feature cls ′)⊕Attention cross (Feature 40×40 ′,Feature Bbox ′) (9) In the formula, Conv 3×3 Indicates a convolution function with a convolution kernel of 3×3, Attention cross represents the cross attention mechanism, and ⊕ represents the concatenation operation.

5. The multi-task learning method for large-area wind turbine identification according to claim 1 is characterized in that: The multi-task network model is trained using a training set and a validation set, including: Training data preparation: During training, the enhanced remote sensing image data is used, combined with a text file containing category and bounding box information, to further extract the shadow features and linear features of the image. Then, the multi-task network based on YOLO is trained. Loss function design: The loss function of the model consists of two parts: scene classification loss: used to evaluate the performance of the model on the scene classification task, using the binary cross entropy loss function to measure the accuracy of classification; object detection loss: including category loss and bounding box regression loss. The bounding box loss is defined by calculating the overlap between the predicted box and the true box, i.e., the intersection-over-union ratio, as well as the shape and position differences of the box. Training optimization: During the training process, set the number of training times. In the last few iterations, stop Mosaic data enhancement to avoid distribution differences in background data during training, so that the model can better adapt to the test set data and reduce the risk of overfitting. Use the stochastic gradient descent algorithm SGD to optimize the model. By continuously adjusting the model parameters, the training error can be gradually reduced to improve the accuracy of the model.

6. A multi-task learning method for large-area wind turbine identification according to claim 5, characterized in that: The multi-task network model is trained using the training set and the validation set, specifically including: Using the 640×640×3 remote sensing image after image data enhancement and the text file containing category and bounding box information, as well as the shadow features extracted by band information and the linear features extracted by Hough transform, the YOLO-based multi-task model is trained according to step B. The loss function of the multi-task network is composed of the loss functions of the scene classification part and the target detection part, as shown in formula (10): L=L Sence +L Detect (10) Among them, L Sence is the loss function of the scene classification part, L Detect is the loss function of the target detection part, and L is the sum of the loss functions of the model, which is used to represent the overall error of the model; The loss function of the scene classification part is a binary classifier loss function, which is expressed by a binary cross entropy loss function, as shown in formula (11): In the formula is the true category corresponding to the pixel, y i The pixel result predicted by the model; The loss function of the target detection part is divided into a category loss function and a target bounding box regression loss function. The bounding box loss is composed of the intersection-over-union loss CIoU loss and the distribution focus loss DFL loss between the real box and the predicted box, as shown in formula (12-16): L Detect =L cls +L reg (12) L reg =L DFL +L CIoU (14) In the formula is the true category of the bounding box j. If the candidate box is positive, then is 1. If the candidate box is negative, then is 0, p j is the bounding box prediction probability; t j and They are the four coordinate parameters of the predicted bounding box and the coordinate parameters of the true value box related to the predicted bounding box. Softmax represents the activation function. N reg Represents the number of positive samples; IoU is the intersection of the predicted box and the real box, ρ 2 (b,b * ) The Euclidean distance between the center point of the predicted box and the real box, c is the diagonal length of the minimum containing box, v is the difference in aspect ratio, which is used to measure the shape difference between the predicted box and the real box, and α is the adjustment factor to balance the center point and shape errors; Set the number of epochs for model training. Stop Mosaic data augmentation in the last 10 epochs to train the model in a more realistic single-image background to reduce difference distribution, improve the performance of the model on the test set, and reduce the risk of overfitting. Use the stochastic gradient descent SGD optimization algorithm to optimize and update the model parameters, that is, the connection weights between layers in the network, and gradually reduce the loss function, so that the model can better fit the training data.

7. A multi-task learning method for large-area wind turbine identification according to claim 1, characterized in that: Apply the trained optimal network model to identify and extract a wide range of wind turbines, including: Test data processing: The image of the test area to be identified is cropped without overlap into image blocks of 640×640×3; Model input and feature extraction: The cropped image blocks are input into the enhanced multi-task network model, and the YOLO backbone network is used to extract multi-scale deep features, while integrating the image's shadow features and linear manual features; Task execution, scene classification task: classify wind turbine scenes under geographic space constraints; target detection task: improve the detection capability of small target wind turbines by fusing deep features and shallow features layer by layer through incremental learning modules; Result output: The model finally outputs the positioning and identification results of wind turbines in a large-scale scene.

Images