Real-time fan blade image segmentation method and system
By introducing a dual-channel information fusion structure and lightweight module in the semantic segmentation network, the problem of low high-resolution image segmentation efficiency is solved, and efficient and accurate segmentation of fan blade images is achieved.
Patent Information
- Application Number
- CN202210417498.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-20
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-04-20
AI Technical Summary
When processing large and high-resolution drone images, the existing semantic segmentation network has high computational complexity, resulting in low operating efficiency of segmentation algorithms and low accuracy, making it difficult to meet the needs of real-time fan blade image segmentation.
Adopt deep semantic segmentation network based on dual-channel information fusion structure, including high-resolution branches and low-resolution branches. Through lightweight dense connection modules, feature fusion modules and spatial attention modules, the amount of parameters is reduced and segmentation accuracy and efficiency is improved. .
The calculation complexity is reduced, the segmentation operation efficiency and accuracy are improved, and the accurate and efficient segmentation of fan blade images is achieved, which solves the problem of low efficiency of segmentation algorithms in the prior art.
Smart Images

Figure CN114842026B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image segmentation, and in particular to a real-time method and system for wind turbine blade image segmentation. Background Art
[0002] With the development of wind power generation technology, the popularity of wind power generation through wind turbines is increasing. In order to ensure the normal operation of wind turbines, it is necessary to inspect the wind turbine blades so that corresponding treatment measures can be taken in time when the wind turbine blades are damaged. At present, the inspection of wind turbine blades based on drones generally inputs the blade image containing background information taken by the drone into the blade image recognition model for calculation. However, due to the complex location environment of the wind turbine, the background information easily interferes with the recognition effect of the blade image. Therefore, it is necessary to first segment the wind turbine blades from the background in the image to improve the accuracy of defect detection.
[0003] Among them, the purpose of image segmentation is to label each pixel as a category. Image semantic segmentation is developed from traditional image segmentation methods. Traditional image segmentation methods, such as threshold method, k-Means clustering method, region method and edge detection method, aim to find the boundary contour of the target and do not care about the category to which the target belongs. Semantic segmentation not only needs to accurately find the contour of the target, but also needs to accurately determine the category to which the target belongs, that is, to give its semantics. Thanks to the rapid development of deep learning and the great improvement of computer storage and computing power, the semantic segmentation method based on deep convolutional neural network has become one of the important methods of image semantic segmentation.
[0004] Semantic segmentation based on deep convolutional neural networks can be considered a pixel-level classification task, which can be roughly divided into two directions: high-precision semantic segmentation and real-time semantic segmentation. For example, Unet and DeepLab series are semantic segmentation networks with relatively large parameters and high precision. High-precision semantic segmentation pursues high accuracy and is basically not limited by inference time. Real-time semantic segmentation requires little inference time while also ensuring basic accuracy, and its development is relatively lagging behind.
[0005] In the related art, when performing real-time semantic segmentation on wind turbine blade images, the model distillation and model pruning methods are usually used to optimize the trained model. However, the above segmentation method requires the establishment of a model before optimization, and the processing steps are complex and the amount of calculation is large. In actual applications, when faced with large-scale and high-resolution drone images, the segmentation algorithm has low operating efficiency due to the large number of parameters and calculations, which is not conducive to completing the image segmentation task in a timely manner, and the segmentation accuracy is low. Summary of the invention
[0006] The present application aims to solve one of the technical problems in the related art at least to some extent.
[0007] To this end, the first purpose of this application is to propose a real-time wind turbine blade image segmentation method, which solves the problem of low segmentation algorithm operation efficiency caused by the number of parameters and calculations in the existing semantic segmentation network when facing large-scale high-resolution drone images. The computational complexity of segmenting wind turbine blade images is reduced, and the segmentation operation efficiency and segmentation accuracy are improved.
[0008] The second object of the present application is to provide a real-time wind turbine blade image segmentation system;
[0009] A third object of the present application is to provide a non-transitory computer-readable storage medium.
[0010] To achieve the above-mentioned purpose, a first embodiment of the present application is to propose a real-time wind turbine blade image segmentation method, the method comprising the following steps:
[0011] Constructing a deep semantic segmentation network based on a two-way information fusion structure, the deep semantic segmentation network includes a high-resolution branch, a low-resolution branch, a feature fusion module, a spatial attention module and a segmentation module, the high-resolution branch and the low-resolution branch have multiple bidirectional connections and each branch includes multiple lightweight dense connection modules LDCM;
[0012] Obtain a sample set of wind turbine blade image data, input any image in the sample set into the deep semantic segmentation network, perform feature extraction and generate a corresponding first feature map through each lightweight dense connection module LDCM in each branch, and obtain a feature map output by each branch, wherein the low-resolution branch is controlled to perform multiple downsampling so that the low-resolution branch extracts deep feature information of the image;
[0013] The spatial attention module is used to obtain context information of the feature map output by the low-resolution branch, the feature fusion module is used to fuse the information output by the two branches, and the fused feature information is input into the segmentation module to obtain an initial segmentation result;
[0014] The deep semantic segmentation network is iteratively trained based on the initial segmentation results of different images in the sample set until the accuracy of the segmentation results output by the deep semantic segmentation network reaches a preset accuracy threshold to complete the training of the deep semantic segmentation network, and the wind turbine blade image to be segmented is input into the trained deep semantic segmentation network to obtain the image segmentation result output by the trained deep semantic segmentation network.
[0015] Optionally, in one embodiment of the present application, each of the lightweight dense connection modules LDCM includes a preset number of lightweight bottleneck sub-modules and a feature fusion sub-module, and the feature extraction and generation of the corresponding first feature map by each of the lightweight dense connection modules LDCM in each branch include: extracting features in the image by each of the lightweight bottleneck sub-modules in any lightweight dense connection module LDCM, and generating a second feature map corresponding to each of the lightweight bottleneck sub-modules; inputting the second feature map extracted by each of the lightweight bottleneck sub-modules into the feature fusion sub-module to fuse all the second feature maps; performing channel attention calculation by the feature fusion sub-module to obtain channel features of all the fused second feature maps, and suppressing the channel features whose information amount in all the fused second feature maps is lower than a preset threshold, and generating a first feature map corresponding to any of the lightweight dense connection modules LDCM.
[0016] Optionally, in one embodiment of the present application, the lightweight bottleneck submodule includes: a first 1×1 convolution layer, a channel random mixing layer, a 1×3 depth-separable convolution layer, a 3×1 depth-separable convolution layer, a second 1×1 convolution layer and a residual layer.
[0017] Optionally, in one embodiment of the present application, channel attention calculation is performed through the feature fusion submodule to obtain channel features of all the fused second feature maps, including: obtaining all the fused second feature maps, and performing a standard convolution operation, batch normalization processing and ReLU activation operation on all the fused second feature maps in sequence to obtain the original feature map; performing global average pooling on the original feature map to obtain a weight feature vector for each channel in the original feature map; performing a 1×1 convolution operation, batch normalization processing and ReLU activation operation on the weight feature vector of each channel in sequence to learn the nonlinear relationship of each channel, and activating it through a sigmoid function to obtain the weight of each channel; multiplying the weight of each channel by the original feature map and adding it to the original feature map to obtain the channel features of each channel.
[0018] Optionally, in one embodiment of the present application, the context information of the feature map output by the low-resolution branch is obtained through the spatial attention module, including: performing three 1×1 convolutions on the feature map input by the low-resolution branch to generate a first new feature map, a second new feature map and a third new feature map; multiplying the first new feature map and the second new feature map and inputting them into the softmax layer to obtain a spatial correlation coefficient matrix; multiplying the correlation coefficient matrix by the third new feature map to obtain a fourth new feature map to restore the size of the feature map input by the low-resolution branch; multiplying the fourth new feature map by a learnable parameter and multiplying it with the feature map input by the low-resolution branch to obtain the global context information of the feature map output by the low-resolution branch.
[0019] Optionally, in one embodiment of the present application, the segmentation module includes: a third 1×1 convolutional layer, a batch normalization layer, an activation layer and an 8-fold bilinear interpolation layer.
[0020] To achieve the above-mentioned purpose, the second aspect of the present application also proposes a real-time wind turbine blade image segmentation system, comprising the following modules:
[0021] A construction module is used to construct a deep semantic segmentation network based on a two-way information fusion structure, wherein the deep semantic segmentation network includes a high-resolution branch, a low-resolution branch, a feature fusion module, a spatial attention module, and a segmentation module, wherein the high-resolution branch and the low-resolution branch have multiple bidirectional connections and each branch includes multiple lightweight dense connection modules LDCM;
[0022] A generation module, used to obtain a sample set of wind turbine blade image data, input any image in the sample set into the deep semantic segmentation network, perform feature extraction and generate a corresponding first feature map through each lightweight dense connection module LDCM in each branch, and obtain a feature map output by each branch, wherein the low-resolution branch is controlled to perform multiple downsampling so that the low-resolution branch extracts deep feature information of the image;
[0023] A first acquisition module is used to obtain the context information of the feature map output by the low-resolution branch through the spatial attention module, perform feature fusion on the information output by the two branches through the feature fusion module, and input the fused feature information into the segmentation module to obtain an initial segmentation result;
[0024] The second acquisition module is used to iteratively train the deep semantic segmentation network based on the initial segmentation results of different images in the sample set until the accuracy of the segmentation results output by the deep semantic segmentation network reaches a preset accuracy threshold to complete the training of the deep semantic segmentation network, and input the wind turbine blade image to be segmented into the trained deep semantic segmentation network to obtain the image segmentation result output by the trained deep semantic segmentation network.
[0025] Optionally, in one embodiment of the present application, a generation module is specifically used to: extract features in an image through each of the lightweight bottleneck submodules in any lightweight dense connection module LDCM, and generate a second feature map corresponding to each of the lightweight bottleneck submodules; input the second feature map extracted by each of the lightweight bottleneck submodules into the feature fusion submodule to fuse all the second feature maps; perform channel attention calculation through the feature fusion submodule to obtain channel features of all the fused second feature maps, and suppress the channel features in all the fused second feature maps whose information amount is lower than a preset threshold, to generate a first feature map corresponding to any of the lightweight dense connection modules LDCM.
[0026] In order to implement the above-mentioned embodiments, the third aspect of the present application further proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the real-time wind blade image segmentation method in the above-mentioned embodiments is implemented.
[0027] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects: On the one hand, the present application reduces the parameters of the deep semantic segmentation network through dual-path design, bottleneck modules, deep separable convolution and asymmetric convolution and other modules, reduces the computational complexity, simplifies the segmentation processing steps, shortens the time of image semantic segmentation, improves the efficiency of semantic segmentation, and is conducive to improving the real-time performance of image segmentation. On the other hand, the accuracy of the semantic segmentation network is improved through high- and low-resolution dual-path information fusion, attention modules and feature fusion modules, so that the proposed real-time semantic segmentation network can accurately and efficiently segment the wind turbine blade part in the image. As a result, the problem of low efficiency of the segmentation algorithm caused by the amount of parameters and calculations in the existing semantic segmentation network facing large-scale high-resolution drone images is solved, the computational complexity of segmenting wind turbine blade images is reduced, and the segmentation operation efficiency and segmentation accuracy are improved.
[0028] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0030] Figure 1 A flow chart of a real-time wind turbine blade image segmentation method proposed in an embodiment of the present application;
[0031] Figure 2 A schematic diagram of the structure of a deep semantic segmentation network proposed in an embodiment of the present application;
[0032] Figure 3 A schematic diagram of the structure of a lightweight dense connection module proposed in an embodiment of the present application;
[0033] Figure 4 A schematic diagram of the structure of a lightweight bottleneck submodule proposed in an embodiment of the present application;
[0034] Figure 5 A flow chart of a channel feature acquisition method based on a feature fusion module proposed in an embodiment of the present application;
[0035] Figure 6 A structural diagram of a feature fusion submodule proposed in an embodiment of the present application;
[0036] Figure 7 A schematic diagram of the principle of obtaining context information by a spatial attention module proposed in an embodiment of the present application;
[0037] Figure 8 A structural schematic diagram of a real-time wind blade image segmentation system proposed in an embodiment of the present application. DETAILED DESCRIPTION
[0038] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0039] It should be noted that in real-time semantic segmentation, a more efficient network structure design can directly design a semantic segmentation network that meets real-time requirements, without optimizing the model again to meet the needs of the application. And efficient network structure design can usually be divided into encoder-decoder based structure, dual-path information fusion based structure, and lightweight backbone network based structure. Among them, based on the encoder-decoder structure, if you want to maintain a good accuracy rate, you need to continuously upsample and fuse multi-scale feature maps in the decoding stage, which will bring more calculations. Therefore, this application adopts a structure based on dual-path information fusion and a lightweight backbone network structure design.
[0040] A real-time wind blade image segmentation method and system proposed in an embodiment of the present invention will be described in detail below with reference to the accompanying drawings.
[0041] Figure 1 A flow chart of a real-time wind turbine blade image segmentation method proposed in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises the following steps:
[0042] Step S101, construct a deep semantic segmentation network based on a dual-path information fusion structure, the deep semantic segmentation network includes a high-resolution branch, a low-resolution branch, a feature fusion module, a spatial attention module and a segmentation module, there are multiple bidirectional connections between the high-resolution branch and the low-resolution branch and each branch includes multiple lightweight dense connection modules LDCM.
[0043] Among them, the deep semantic segmentation network constructed by the present application has a dual-path information fusion structure, that is, the deep semantic segmentation network starts from the same trunk and is divided into two parallel network branches of different resolutions. The main branch (i.e., the high-resolution branch) generates a high-resolution feature map containing more detailed information, and the secondary branch (i.e., the low-resolution branch) generates a low-resolution feature map containing deeper detailed information. In addition, the deep semantic segmentation network of the present application has multiple bidirectional connections between the two branches to achieve effective fusion of information in the two branches.
[0044] In one embodiment of the present application, when constructing a deep semantic segmentation network, a dual-path information fusion structure can be constructed based on the PyTorch deep learning framework, wherein PyTorch is an open source Python machine learning library, which can be approximately regarded as a deep neural network with automatic differentiation function. The present application constructs a dual-path information structure based on the existing PyTorch deep learning framework, and then realizes the construction of the network structure of the deep semantic segmentation network by setting a feature fusion module, a spatial attention module and a segmentation module.
[0045] Specifically, each branch of the deep semantic segmentation network of the present application, namely the high-resolution branch and the low-resolution branch, includes multiple lightweight dense connection modules (Light Dense Connection Module, abbreviated as LDCM), and the features in the input image can be extracted through the lightweight dense connection module LDCM.
[0046] It should be noted that in the embodiment of the present application, the number of lightweight densely connected modules LDCM set in the two branches and the scaling scale of the sampling operation performed by each LDCM can be different, so as to achieve different branches finally outputting feature maps of different resolutions.
[0047] As an example, this application proposes an overall network structure of a deep semantic segmentation network, such as Figure 2 As shown, the high-resolution branch of the network includes five lightweight densely connected modules LDCM, and the low-resolution branch includes three lightweight densely connected modules LDCM. Each LDCM scales the image size to different scales (i.e. Figure 2 The number after each LDCM in the figure indicates the scale of the feature map size of the LDCM). For example, inputting a 512*512 image into Figure 2 After LDCM1 / 2 in the LDCM, it is downsampled to 256*256 through the LDCM. The connection method of each LDCM is as follows Figure 2 As shown, the size scaling scale of each LDCM can be set in advance as needed, for example, Figure 2 The scaling scale of the last three LDCMs of the high-resolution branch is set to 1 / 8 to generate a high-resolution feature map, and the low-resolution branch can be controlled to perform downsampling operations. The downsampling degree of each LDCM in the low-resolution branch is high to extract rich deep features.
[0048] When actually performing feature extraction, continue to refer to Figure 2 In the deep semantic segmentation network of this embodiment, 7 bidirectional connections are set between the high-resolution branch and the low-resolution branch, which can fuse the features extracted by different LDCM1 in the two branches, which is conducive to the effective information fusion of the two branches. In addition, the spatial attention module 2 is set in the low-resolution branch, the feature fusion module 3 is set in the high-resolution branch, and the feature fusion module is connected to the segmentation module 4, so that the subsequent feature fusion module sends the fused information to the segmentation module to obtain the segmentation result.
[0049] Step S102, obtain a sample set of wind turbine blade image data, input any image in the sample set into a deep semantic segmentation network, perform feature extraction and generate a corresponding first feature map through each lightweight dense connection module LDCM in each branch, and obtain the feature map output by each branch, wherein the low-resolution branch is controlled to perform multiple downsampling so that the low-resolution branch can extract deep feature information of the image.
[0050] Among them, the fan blade image data sample set is the training data set required for training the constructed deep semantic segmentation network. The fan blade image data sample set contains a large number of fan blade images, and each image may contain different blade shapes and background information as segmentation training data.
[0051] The first feature map is a feature map corresponding to the lightweight dense connection module LDCM output after each lightweight dense connection module LDCM performs feature extraction. The first feature map can be used for subsequent corresponding branch calculations to output the feature map corresponding to the branch.
[0052] In an embodiment of the present application, a fan blade image data set can be generated by collecting historical image data of multiple fan blades in different time periods, and then a fan blade image data sample set can be retrieved from a database. When performing network training, any image in the sample set is input into a deep semantic segmentation network, and features are extracted through each lightweight densely connected module LDCM in the two branches to generate a first feature map corresponding to each LDCM.
[0053] In specific implementation, the low-resolution branch is controlled to perform multiple downsampling. For example, continue to refer to Figure 2 In the example of , the first LDCM in the low-resolution branch is four times downsampled relative to the second LDCM in the high-resolution branch. Specifically, the second LDCM in the high-resolution branch sends its own feature map to the first LDCM in the low-resolution branch through a bidirectional connection, and then the first LDCM in the low-resolution branch performs four times downsampling on the received feature map. The information interaction and sampling method of each lightweight densely connected module LDCM in the deep semantic segmentation network is shown in the figure, which will not be repeated here.
[0054] In order to facilitate understanding of the specific implementation process of feature extraction by the lightweight dense connection module LDCM in the present application, a detailed description is given below in combination with the structure of the lightweight dense connection module LDCM in the present application.
[0055] In one embodiment of the present application, Figure 3 As shown, each lightweight dense connection module LDCM includes a preset number of lightweight bottleneck submodules 11 and a feature fusion submodule 12. When the lightweight dense connection module LDCM performs feature extraction and generates the corresponding first feature map, the features in the image are first extracted through each lightweight bottleneck submodule in the LDCM to generate the second feature map corresponding to each lightweight bottleneck submodule, that is, the lightweight bottleneck submodule extracts features, and each lightweight bottleneck submodule can generate a feature map corresponding to the lightweight bottleneck type, that is, the second feature map. Then, the second feature map extracted by each lightweight bottleneck submodule is input into the feature fusion submodule, and all the second feature maps are fused, that is, the feature map extracted by each lightweight bottleneck submodule is input into the feature fusion submodule through dense connection, and all the feature maps are fused. Finally, the channel attention calculation is performed through the feature fusion submodule to obtain the channel features of all the fused second feature maps, and the channel features whose information content is lower than the preset threshold in all the fused second feature maps are suppressed, so as to generate the first feature map corresponding to any lightweight dense connection module LDCM. Therefore, through the channel attention calculation in the feature fusion submodule, the semantic segmentation network pays more attention to the channel features with the largest amount of information, while suppressing those unimportant channel features.
[0056] It should be noted that the number of lightweight bottleneck submodules in each lightweight densely connected module LDCM can be set according to actual needs. For example, Figure 2 In the example, three lightweight bottleneck sub-modules are set in each lightweight densely connected module LDCM of the low-resolution branch, and the numbers of lightweight bottleneck sub-modules included in the LDCM in the high-resolution branch are 3, 4, 6, 4 and 3 respectively.
[0057] In order to more clearly illustrate the specific implementation process of extracting features from the lightweight bottleneck submodule, the structure of the lightweight bottleneck submodule in the embodiment of the present application is described in detail below. Figure 4 As shown, the lightweight bottleneck submodule in this embodiment includes: a first 1×1 convolutional layer 10, a channel random mixing layer 20, a 1×3 depth-separable convolutional layer 30, a 3×1 depth-separable convolutional layer 40, a second 1×1 convolutional layer 50 and a residual layer 60.
[0058] In the lightweight bottleneck submodule of this example, the residual layer 60 can reduce gradient diffusion, which is beneficial to the execution of network training. The bottleneck type is implemented by two 1×1 convolution operations, which can more conveniently change the dimension of the network structure, flexibly design the network structure, and reduce the amount of calculation.
[0059] It should also be noted that the structure of the lightweight bottleneck submodule of the embodiment of the present application can reduce the number of parameters and the amount of calculation for image segmentation. For example, to facilitate the description of the calculation process of the lightweight bottleneck submodule, assume that the number of channels c in the image input to the lightweight bottleneck submodule is i , using c o The convolution operation is performed on all channels using the convolution kernel. For the standard 3×3 convolution kernel, the convolution operation is performed on all channels, and its parameter quantity is 3×3×c i ×c o The asymmetric convolution decomposes the standard 3×3 convolution kernel into 3×1 and 1×3 convolution kernels, and its parameter volume is 3×3×c i ×c o The depth-wise separable convolution is responsible for performing convolution operations on each channel, with a parameter size of 3×3×c i The present application combines asymmetric convolution and depth-separable convolution, and sets the lightweight bottleneck submodule in the above example, and the number of parameters is 3×3×c i Therefore, compared with the standard convolution kernel, the present application can effectively reduce the number of parameters and the amount of calculation. In addition, for the problem of lack of information interaction between channels when performing convolution operations on each channel separately, the present application introduces channel mixing technology in the lightweight bottleneck type to improve the information interaction between channels, thereby reducing the amount of calculation while ensuring the segmentation performance of the deep semantic segmentation network.
[0060] In order to more clearly illustrate the specific implementation process of the feature fusion submodule in the embodiment of the present application performing channel attention calculation to obtain relevant information of channel features, the present application also proposes a channel feature acquisition method based on the feature fusion submodule. Figure 5 This is a flow chart of a channel feature acquisition method based on a feature fusion module proposed in an embodiment of the present application. Figure 6 This is a schematic diagram of the structure of a feature fusion submodule proposed in an embodiment of the present application. The calculation method of the channel feature is described below in conjunction with the structure of the feature fusion submodule. Figure 5 As shown, the method comprises the following steps:
[0061] Step S501, obtain all the fused second feature maps, and perform a standard convolution operation, batch normalization processing and ReLU activation operation on all the fused second feature maps in sequence to obtain the original feature map.
[0062] Specifically, the second feature maps output by each lightweight bottleneck submodule in the lightweight dense connection module LDCM are first fused through the feature fusion submodule, and the second feature maps can be spliced through the feature map splicing layer 100 in the feature fusion submodule. For example, for multiple input feature maps (x 1 , …, x c ), and concatenate them together to get X through the concatenation function: X = concatenate(x 1 , …, x c ), where c is the number of lightweight bottleneck submodules in this LDCM.
[0063] Then, all the fused second feature maps are obtained, and the concatenated feature maps are processed by standard convolution, batch normalization (BN) and Relu activation layer 200 to obtain the original feature map. Referring to the example of step S501, the original feature map X′=Relu(BN(conv(X cat ))), where X cat That is X in step S501.
[0064] Step S502, performing global average pooling on the original feature map to obtain a weighted feature vector for each channel in the original feature map.
[0065] Specifically, global average pooling is performed in the global average pooling layer 300 to obtain the weight feature vector W of each channel feature map: Where H is the height of the feature map, W is the width of the feature map, i and j are the indexes of the height and width of the feature map, respectively. The height and width of the feature map can be automatically determined by each module when generating the feature map, and the indexes of the height and width are used to determine the position of each pixel in the feature map, that is, the mean of each pixel in the current feature map is calculated by this formula to obtain the weight feature vector of the feature map.
[0066] In step S503, the weight feature vector of each channel is sequentially subjected to a 1×1 convolution operation, batch normalization processing, and ReLU activation operation to learn the nonlinear relationship of each channel, and activated by a sigmoid function to obtain the weight of each channel.
[0067] Specifically, in the 1×1 convolution layer, batch normalization and Relu activation operation layer 400, 1×1 convolution, batch normalization, and Relu activation operations are performed in sequence to learn the nonlinear relationship between multiple feature maps, and then input into the sigmoid layer 500, σ activation is performed through the sigmoid function to obtain the weight W′ of each channel feature map: W′=σ(Relu(BN(conv(W)))).
[0068] Step S504, multiplying the weight of each channel by the original feature map and adding the result to the original feature map to obtain the channel feature of each channel.
[0069] Specifically, the learned weight coefficient of each channel is multiplied back to the original feature map and summed, that is, O = W′⊙X′+′X′. The calculated result O contains the correlation information between channels, that is, the channel characteristics of each channel.
[0070] Therefore, after calculating the features of each channel in the spliced second feature map by this method, the information content of the channel features is compared with the preset information content threshold. When the information content of the channel features is lower than the information content threshold, it means that the information content of the channel features is low, and the impact on image segmentation is small and can be ignored, thereby making the deep semantic segmentation network pay more attention to the channel features with the largest amount of information and suppress those unimportant channel features.
[0071] It should also be noted that after each lightweight densely connected module LDCM in each branch generates the corresponding first feature map, it can be calculated according to the connection relationship of the LDCM in each branch to obtain the feature map corresponding to each branch. Figure 2 In the low-resolution branch, the feature map output by the LDCM1 / 8 module of the high-resolution branch is downsampled by 8 times, and then added to the feature map output by the LDCM1 / 64 module of the low-resolution branch, and the obtained feature map is used as the final output feature map of the low-resolution branch.
[0072] Step S103, obtaining the context information of the feature map output by the low-resolution branch through the spatial attention module, performing feature fusion on the information output by the two branches through the feature fusion module, and inputting the fused feature information into the segmentation module to obtain the initial segmentation result.
[0073] In the embodiment of the present application, the spatial attention module can be set in the low-resolution branch, and the feature map processed by each LDCM in the low-resolution branch is processed, for example, Figure 2 The output of the low-resolution branch LDCM 1 / 64 in the example is input into the spatial attention module to obtain the context information of the feature map output by the low-resolution branch.
[0074] In order to facilitate understanding of the specific implementation process of the spatial attention module in this application to obtain context information, the following Figure 7 The schematic diagram of the principle of the spatial attention module obtaining contextual information is shown in detail.
[0075] In this embodiment, first, the feature map input by the low-resolution branch is subjected to three 1×1 convolutions to generate the first new feature map, the second new feature map, and the third new feature map, as shown in FIG. Figure 7 As shown in Figure 1, B, C, and D are all new feature maps generated from the low-resolution branch input feature map X through 1×1 convolution. Then, the first new feature map and the second new feature map are multiplied and input into the softmax layer to obtain the spatial correlation coefficient matrix. Figure 7 In the example, the result of multiplying B and C is fed into the softmax layer to obtain the spatial correlation coefficient matrix S. Then, the correlation coefficient matrix is multiplied by the third new feature map to obtain the fourth new feature map to restore the size of the feature map input by the low-resolution branch. Figure 7 Finally, the fourth new feature map is multiplied by the learnable parameter and multiplied with the feature map of the low-resolution branch input to obtain the global context information of the feature map output by the low-resolution branch. Figure 7 For example, the calculation process of the spatial attention module is shown in the following formula:
[0076] S = softmax(B·C)
[0077]
[0078] in, is a learnable parameter, and the final result Y is obtained s The global context information is preserved.
[0079] Furthermore, the information output by the two branches is subjected to feature fusion through a feature fusion module. The information output by the two branches includes: a high-resolution feature map output by the high-resolution branch, and a low-resolution feature map with rich deep features and context information output by the low-resolution branch. The feature fusion module can perform feature fusion on the information output by the two branches by feature splicing.
[0080] Furthermore, the feature fusion module inputs the fused dual-path information into the segmentation module, and the segmentation module obtains the initial segmentation result of the current input wind turbine blade image. In specific implementation, as a possible implementation method, the segmentation module includes: a third 1×1 convolution layer, a batch normalization layer, an activation layer, and an 8-fold bilinear interpolation layer, and the fused dual-path information is processed in turn through the above layers to obtain the segmentation result.
[0081] Step S104, iteratively train the deep semantic segmentation network based on the initial segmentation results of different images in the sample set until the accuracy of the segmentation results output by the deep semantic segmentation network reaches a preset accuracy threshold to complete the training of the deep semantic segmentation network, and input the wind turbine blade image to be segmented into the trained deep semantic segmentation network to obtain the image segmentation result output by the trained deep semantic segmentation network.
[0082] Specifically, different images in the fan blade image data sample set are sequentially input into the deep semantic segmentation network, and the initial segmentation result of each image is obtained in the manner described in the above steps S102 to S103. The deep semantic segmentation network is then iteratively trained in combination with the segmentation results of each image until the accuracy of the segmentation result output by the deep semantic segmentation network reaches a preset accuracy threshold, i.e., the deep semantic segmentation network is trained.
[0083] In one embodiment of the present application, the training method of the deep semantic segmentation network can refer to the training method of the neural network in the related art, and is not limited here. As one possible implementation method, each image in the wind turbine blade image data set can be manually annotated in advance to generate a corresponding label file, and then according to the label file of each image in the data set and the corresponding initial detection result, the loss is calculated and the parameters of the deep semantic segmentation network are updated through a preset target loss function, and the deep semantic segmentation network is iteratively trained for a preset number of times through the random gradient descent method until the accuracy of the image segmentation result of the deep semantic segmentation network reaches a preset accuracy threshold.
[0084] Furthermore, after the deep semantic segmentation network is trained, the image of the wind turbine blade to be segmented is input into the trained deep semantic segmentation network, and the result output by the trained deep semantic segmentation network is the result of segmenting the current wind turbine blade image.
[0085] In summary, the real-time wind blade image segmentation method of the embodiment of the present application, on the one hand, reduces the parameters of the deep semantic segmentation network through dual-path design, bottleneck modules, deep separable convolution and asymmetric convolution and other modules, reduces the computational complexity, simplifies the segmentation processing steps, shortens the time of image semantic segmentation, improves the efficiency of semantic segmentation, and is conducive to improving the real-time performance of image segmentation. On the other hand, the accuracy of the semantic segmentation network is improved through high- and low-resolution dual-path information fusion, attention modules and feature fusion modules, so that the proposed real-time semantic segmentation network can accurately and efficiently segment the wind blade part in the image. As a result, this method solves the problem of low operating efficiency of the segmentation algorithm of the existing semantic segmentation network due to the amount of parameters and calculations in the face of large-scale high-resolution drone images, reduces the computational complexity when segmenting wind blade images, and improves the segmentation operation efficiency and segmentation accuracy.
[0086] In order to implement the above embodiment, the present application also proposes a real-time wind turbine blade image segmentation system. Figure 8 A structural diagram of a real-time wind turbine blade image segmentation system proposed in an embodiment of the present application is shown in FIG. Figure 8 As shown, the system includes a construction module 100 , a generation module 200 , a first acquisition module 300 and a second acquisition module 400 .
[0087] Among them, the construction module 100 is used to construct a deep semantic segmentation network based on a dual-path information fusion structure. The deep semantic segmentation network includes a high-resolution branch, a low-resolution branch, a feature fusion module, a spatial attention module and a segmentation module. There are multiple bidirectional connections between the high-resolution branch and the low-resolution branch and each branch includes multiple lightweight dense connection modules LDCM.
[0088] A generation module 200 is used to obtain a sample set of wind turbine blade image data, input any image in the sample set into a deep semantic segmentation network, perform feature extraction and generate a corresponding first feature map through each lightweight dense connection module LDCM in each branch, and obtain a feature map output by each branch, wherein the low-resolution branch is controlled to perform multiple downsampling so that the low-resolution branch can extract deep feature information of the image.
[0089] The first acquisition module 300 is used to obtain the context information of the feature map output by the low-resolution branch through the spatial attention module, perform feature fusion on the information output by the two branches through the feature fusion module, and input the fused feature information into the segmentation module to obtain the initial segmentation result.
[0090] The second acquisition module 400 is used to iteratively train the deep semantic segmentation network based on the initial segmentation results of different images in the sample set, until the accuracy of the segmentation results output by the deep semantic segmentation network reaches a preset accuracy threshold to complete the training of the deep semantic segmentation network, and input the wind turbine blade image to be segmented into the trained deep semantic segmentation network to obtain the image segmentation results output by the trained deep semantic segmentation network.
[0091] Optionally, in one embodiment of the present application, each lightweight dense connection module LDCM includes a preset number of lightweight bottleneck sub-modules and a feature fusion sub-module, and the generation module 200 is specifically used to: extract features in the image through each of the lightweight bottleneck sub-modules in any lightweight dense connection module LDCM, and generate a second feature map corresponding to each lightweight bottleneck sub-module; input the second feature map extracted by each lightweight bottleneck sub-module into the feature fusion unit, and fuse all the second feature maps; perform channel attention calculation through the feature fusion sub-module to obtain channel features of all the fused second feature maps, and suppress the channel features in all the fused second feature maps whose information amount is lower than a preset threshold, and generate a first feature map corresponding to any lightweight dense connection module LDCM.
[0092] Optionally, in one embodiment of the present application, the lightweight bottleneck submodule includes: a first 1×1 convolution layer, a channel random mixing layer, a 1×3 depth-separable convolution layer, a 3×1 depth-separable convolution layer, a second 1×1 convolution layer and a residual layer.
[0093] Optionally, in one embodiment of the present application, the generation module 200 is also used to: obtain all the second feature maps after fusion, and perform a standard convolution operation, batch normalization processing and ReLU activation operation on all the second feature maps after fusion in sequence to obtain the original feature map; perform global average pooling on the original feature map to obtain the weight feature vector of each channel in the original feature map; perform a 1×1 convolution operation, batch normalization processing and ReLU activation operation on the weight feature vector of each channel in sequence to learn the nonlinear relationship of each channel, and activate it through a sigmoid function to obtain the weight of each channel; multiply the weight of each channel by the original feature map and add it to the original feature map to obtain the channel feature of each channel.
[0094] Optionally, in one embodiment of the present application, the first acquisition 300 is specifically used to: perform three 1×1 convolutions on the feature map of the low-resolution branch input to generate a first new feature map, a second new feature map and a third new feature map; multiply the first new feature map and the second new feature map and input them into the softmax layer to obtain a spatial correlation coefficient matrix; multiply the correlation coefficient matrix by the third new feature map to obtain a fourth new feature map to restore the size of the feature map of the low-resolution branch input; multiply the fourth new feature map by a learnable parameter and multiply it with the feature map of the low-resolution branch input to obtain the global context information of the feature map of the low-resolution branch output.
[0095] Optionally, in one embodiment of the present application, the segmentation module includes: a third 1×1 convolutional layer, a batch normalization layer, an activation layer and an 8-fold bilinear interpolation layer.
[0096] It should be noted that the above explanation of the embodiment of the real-time wind turbine blade image segmentation method is also applicable to the system of this embodiment, and will not be repeated here.
[0097] In summary, the real-time wind blade image segmentation system of the embodiment of the present application, on the one hand, reduces the parameters of the deep semantic segmentation network through dual-path design, bottleneck modules, deep separable convolution and asymmetric convolution and other modules, reduces the computational complexity, simplifies the segmentation processing steps, shortens the time of image semantic segmentation, improves the efficiency of semantic segmentation, and is conducive to improving the real-time performance of image segmentation. On the other hand, the accuracy of the semantic segmentation network is improved through high- and low-resolution dual-path information fusion, attention modules and feature fusion modules, so that the proposed real-time semantic segmentation network can accurately and efficiently segment the wind blade part in the image. As a result, the system solves the problem of low operating efficiency of the segmentation algorithm of the existing semantic segmentation network due to the amount of parameters and calculations in the face of large-scale high-resolution drone images, reduces the computational complexity when segmenting wind blade images, and improves the segmentation operation efficiency and segmentation accuracy.
[0098] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the real-time wind blade image segmentation method as described in any of the above embodiments is implemented.
[0099] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0100] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0101] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0102] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0103] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0104] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0105] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0106] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A real-time fan blade image segmentation method, characterized in that: The following steps are involved: Constructing a deep semantic segmentation network based on a two-way information fusion structure, the deep semantic segmentation network includes a high-resolution branch, a low-resolution branch, a feature fusion module, a spatial attention module and a segmentation module, the high-resolution branch and the low-resolution branch have multiple bidirectional connections and each branch includes multiple lightweight dense connection modules LDCM; Obtain a sample set of wind turbine blade image data, input any image in the sample set into the deep semantic segmentation network, perform feature extraction and generate a corresponding first feature map through each of the lightweight dense connection modules LDCM in each branch, and obtain the feature map output by each branch, wherein the low-resolution branch is controlled to perform multiple downsampling so that the low-resolution branch extracts deep feature information of the image, each of the lightweight dense connection modules LDCM includes a preset number of lightweight bottleneck submodules and a feature fusion submodule, and each of the lightweight dense connection modules LDCM in each branch performs feature extraction and generates a corresponding The first feature map includes: extracting features in the image through each lightweight bottleneck submodule in any lightweight dense connection module LDCM, generating a second feature map corresponding to each lightweight bottleneck submodule; inputting the second feature map extracted by each lightweight bottleneck submodule into the feature fusion submodule to fuse all the second feature maps; performing channel attention calculation through the feature fusion submodule to obtain channel features of all the fused second feature maps, and suppressing the channel features whose information amount in all the fused second feature maps is lower than a preset threshold, to generate the first feature map corresponding to any lightweight dense connection module LDCM; The spatial attention module is used to obtain context information of the feature map output by the low-resolution branch, the feature fusion module is used to fuse the information output by the two branches, and the fused feature information is input into the segmentation module to obtain an initial segmentation result; The deep semantic segmentation network is iteratively trained based on the initial segmentation results of different images in the sample set until the accuracy of the segmentation results output by the deep semantic segmentation network reaches a preset accuracy threshold to complete the training of the deep semantic segmentation network, and the wind turbine blade image to be segmented is input into the trained deep semantic segmentation network to obtain the image segmentation result output by the trained deep semantic segmentation network.
2. The segmentation method according to claim 1, characterized in that: The lightweight bottleneck submodule includes: a first Convolutional layer, channel random mixing layer, Depthwise separable convolutional layers, Depthwise separable convolutional layer, second Convolutional layers and residual layers.
3. The segmentation method according to claim 1, characterized in that: The performing channel attention calculation by the feature fusion submodule to obtain the channel features of all the fused second feature maps includes: Obtaining all the fused second feature maps, and sequentially performing a standard convolution operation, a batch normalization process, and a ReLU activation operation on all the fused second feature maps to obtain an original feature map; Performing global average pooling on the original feature map to obtain a weight feature vector for each channel in the original feature map; The weight feature vector of each channel is processed once in turn Convolution operation, batch normalization and ReLU activation operation are used to learn the nonlinear relationship of each channel, and the sigmoid function is used for activation to obtain the weight of each channel; The weight of each channel is multiplied by the original feature map and then added to the original feature map to obtain the channel feature of each channel.
4. The segmentation method according to claim 1, characterized in that: The obtaining, through the spatial attention module, context information of the feature map output by the low-resolution branch includes: The feature map of the low-resolution branch input is processed three times Convolution generates a first new feature map, a second new feature map, and a third new feature map; Multiplying the first new feature map and the second new feature map and inputting the resultant into a softmax layer to obtain a spatial correlation coefficient matrix; Multiplying the correlation coefficient matrix by the third new feature map to obtain a fourth new feature map, so as to restore the size of the feature map input by the low-resolution branch; The fourth new feature map is multiplied by a learnable parameter and multiplied with the feature map input by the low-resolution branch to obtain global context information of the feature map output by the low-resolution branch.
5. The segmentation method according to claim 1, characterized in that: The segmentation module includes: a third Convolutional layers, batch normalization layers, activation layers, and 8x bilinear interpolation layers.
6. A real-time wind turbine blade image segmentation system, characterized in that: include: A construction module is used to construct a deep semantic segmentation network based on a two-way information fusion structure, wherein the deep semantic segmentation network includes a high-resolution branch, a low-resolution branch, a feature fusion module, a spatial attention module, and a segmentation module, wherein the high-resolution branch and the low-resolution branch have multiple bidirectional connections and each branch includes multiple lightweight dense connection modules LDCM; A generation module, used to obtain a sample set of wind turbine blade image data, input any image in the sample set into the deep semantic segmentation network, perform feature extraction and generate a corresponding first feature map through each of the lightweight dense connection modules LDCM in each branch, and obtain a feature map output by each branch, wherein the low-resolution branch is controlled to perform multiple downsampling so that the low-resolution branch extracts deep feature information of the image, and each of the lightweight dense connection modules LDCM includes a preset number of lightweight bottleneck submodules and a feature fusion submodule. The generation module is specifically used to: extract features in the image through each of the lightweight bottleneck submodules in any lightweight dense connection module LDCM, and generate a second feature map corresponding to each lightweight bottleneck submodule; input the second feature map extracted by each of the lightweight bottleneck submodules into the feature fusion submodule to fuse all the second feature maps; perform channel attention calculation through the feature fusion submodule to obtain channel features of all the fused second feature maps, and suppress the channel features whose information amount in all the fused second feature maps is lower than a preset threshold, and generate a first feature map corresponding to any of the lightweight dense connection modules LDCM; A first acquisition module is used to obtain the context information of the feature map output by the low-resolution branch through the spatial attention module, perform feature fusion on the information output by the two branches through the feature fusion module, and input the fused feature information into the segmentation module to obtain an initial segmentation result; The second acquisition module is used to iteratively train the deep semantic segmentation network based on the initial segmentation results of different images in the sample set until the accuracy of the segmentation results output by the deep semantic segmentation network reaches a preset accuracy threshold to complete the training of the deep semantic segmentation network, and input the wind turbine blade image to be segmented into the trained deep semantic segmentation network to obtain the image segmentation result output by the trained deep semantic segmentation network.
7. The segmentation system according to claim 6, characterized in that: The lightweight bottleneck submodule includes: a first Convolutional layer, channel random mixing layer, Depthwise separable convolutional layers, Depthwise separable convolutional layer, second Convolutional layers and residual layers.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the real-time wind blade image segmentation method as described in any one of claims 1 to 5 is implemented.
