A method and device for detecting the surface state of a blade

Abstract

The present application provides a method and device for detecting the surface state of a blade, which are applied to the field of wind power technology. The method for detecting the surface state of a blade includes: obtaining a to-be-detected image corresponding to the to-be-detected blade and a support set; wherein the support set includes sample images corresponding to each blade on the wind turbine; inputting the to-be-detected image and the support set into a pre-trained feature extraction network to obtain a first image feature and a second image feature output by the feature extraction network; inputting the first image feature and the second image feature into a pre-trained graph neural network to obtain a classification result output by the graph neural network; wherein the loss function corresponding to the graph neural network is a differential loss function; determining the surface state of the to-be-detected blade according to the classification result. Since the differential loss function is introduced during the training of the graph neural network, the dynamic changes of the samples can be learned, and even if the surface damage samples of the blade are insufficient, a detection result with higher accuracy can be obtained.

Description

Technical Field

[0001] This application relates to the technical field of wind power, and more specifically, to a method and device for detecting the surface state of a blade. Background Art

[0002] The blade is a wind energy capture device of the fan in the offshore wind power system, and the quality of its performance and the length of its service life directly affect the normal operation of the fan and the system efficiency. Different from electrical components such as generators and inverters in the offshore wind power system that can provide rich operation data for condition monitoring and defect detection, since the blade is a pure mechanical component and only has a rotational speed signal available for acquisition, this brings great inconvenience to the blade state detection.

[0003] In the prior art, generally, deep learning is used to detect the blade. However, when using the deep learning method, due to the insufficient surface damage samples of the blade, the accuracy of detecting the surface damage of the blade is relatively low. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a method and device for detecting the surface state of a blade, so as to solve the technical problem of relatively low accuracy in detecting the surface damage of the blade due to insufficient surface damage samples of the blade in the prior art.

[0005] In a first aspect, the embodiments of this application provide a method for detecting the surface state of a blade, including: obtaining a to-be-detected image corresponding to the to-be-detected blade and a support set; wherein, the support set includes sample images corresponding to each blade on the fan; inputting the to-be-detected image and the support set into a pre-trained feature extraction network, and obtaining a first image feature corresponding to the to-be-detected image and a second image feature corresponding to the support set output by the feature extraction network; inputting the first image feature and the second image feature into a pre-trained graph neural network, and obtaining a classification result output by the graph neural network; wherein, the loss function corresponding to the graph neural network is a differential loss function; determining the surface state of the to-be-detected blade according to the classification result.

[0006] In the above solution, based on the pre-trained feature extraction network, the image features of the to-be-detected image and the support set can be extracted, and then the extracted image features are input into the graph neural network, and the corresponding classification result can be obtained. According to the above classification result, the detection of the surface state of the blade can be realized. Among them, since the differential loss function is introduced during the training of the graph neural network, the dynamic changes of the samples can be learned, and even if the surface damage samples of the blade are insufficient, a relatively high-accuracy detection result can be obtained.

[0007] In an alternative embodiment, before obtaining the to-be-detected image and the support set corresponding to the to-be-detected blade, the method further includes: obtaining a first sample image corresponding to at least one blade on the wind turbine and a plurality of second sample images of other blades on the wind turbine at multiple times; inputting the first sample image and the second sample images into a to-be-trained feature extraction network to obtain a first sample feature corresponding to the first sample image and a plurality of second sample features corresponding to the second sample images output by the feature extraction network; calculating a matching loss value between the first sample feature and the second sample features, and calculating a distance loss value between the plurality of second sample features; updating the parameters of the feature extraction network according to the matching loss value and the distance loss value. In the above solution, when training the feature extraction network, a multi-task framework can be adopted for training; among them, the main task is the blade surface feature matching task, which is used to distinguish the differences between two different blades, and the auxiliary task is the differential task, which is used to reflect the differential characteristics of the features in the time dimension. Therefore, training the feature extraction network in the above manner can improve the performance of the feature extraction network.

[0008] In an alternative embodiment, before obtaining the to-be-detected image and the support set corresponding to the to-be-detected blade, the method further includes: obtaining a registration set and a query set; wherein, the registration set includes a first blade image, and the query set includes a second blade image and a third blade image; inputting the first blade image and the second blade image into a pre-trained feature extraction network to obtain a first blade feature corresponding to the first blade image and a second blade feature corresponding to the second blade image output by the feature extraction network; inputting the first blade feature and the second blade feature into a to-be-trained graph neural network to obtain a first distance between the first blade image and the second blade image; calculating a differential loss value according to the first distance and a second distance between the first blade image and the third blade image; updating the parameters of the graph neural network according to the differential loss value. In the above solution, during the process of training the graph neural network, meta-learning can be adopted for training, and a differential loss function is introduced, and this differential loss function is used to reflect the differential characteristics of the features in the time dimension. Therefore, training the graph neural network in the above manner can improve the performance of the graph neural network; in addition, since the dynamic changes of the samples can be learned, even if there are insufficient blade surface damage samples, a detection result with higher accuracy can be obtained.

[0009] In an alternative embodiment, before determining the surface state of the blade to be detected according to the classification result, the method further includes: inputting the image to be detected into a pre-trained level set graph network to obtain a judgment result output by the level set graph network; the determining the surface state of the blade to be detected according to the classification result includes: determining the surface state of the blade according to the classification result and the judgment result. In the above solution, the state of the blade surface can also be detected based on the level set graph network, further improving the accuracy of detecting damage to the blade surface.

[0010] In an alternative embodiment, before inputting the image to be detected into a pre-trained level set graph network to obtain a judgment result output by the level set graph network, the method further includes: training the level set graph network to be trained using blade damage images and non-blade damage images. In the above solution, during the training of the level set graph network, in addition to using blade damage images, non-blade damage images can also be introduced, so as to enrich the distribution of source domain objects and transfer the surface damage feature knowledge of other objects to the fan blade object. Therefore, the generalization of the level set graph network can be improved.

[0011] In an alternative embodiment, the non-blade damage images include: road surface damage images, aircraft skin damage images, wall damage images, wood damage images, biscuit packaging damage images, fabric damage images, and steel damage images.

[0012] In a second aspect, an embodiment of the present application provides a device for detecting the surface state of a blade, including: a first acquisition module, configured to acquire an image to be detected corresponding to the blade to be detected and a support set; wherein, the support set includes sample images corresponding to each blade on the fan; a first input module, configured to input the image to be detected and the support set into a pre-trained feature extraction network to obtain a first image feature corresponding to the image to be detected and a second image feature corresponding to the support set output by the feature extraction network; a second input module, configured to input the first image feature and the second image feature into a pre-trained graph neural network to obtain a classification result output by the graph neural network; wherein, the loss function corresponding to the graph neural network is a differential loss function; a determination module, configured to determine the surface state of the blade to be detected according to the classification result.

[0013] In the above solution, based on the pre-trained feature extraction network, the image features of the image to be detected and the support set can be extracted. Then, the extracted image features are input into the graph neural network, and the corresponding classification results can be obtained. According to the above classification results, the detection of the blade surface state can be realized. Among them, since the differential loss function is introduced in the process of training the graph neural network, the dynamic changes of the samples can be learned. Even if the number of blade surface damage samples is insufficient, a detection result with higher accuracy can be obtained.

[0014] In an alternative embodiment, the blade surface state detection device further includes: a second acquisition module, configured to acquire a first sample image corresponding to at least one blade on the wind turbine and a plurality of second sample images of other blades on the wind turbine at multiple times; a third input module, configured to input the first sample image and the second sample images into a feature extraction network to be trained, and obtain a first sample feature corresponding to the first sample image and a plurality of second sample features corresponding to the second sample images output by the feature extraction network; a first calculation module, configured to calculate a matching loss value between the first sample feature and the second sample features, and calculate a distance loss value between the plurality of second sample features; a first update module, configured to update the parameters of the feature extraction network according to the matching loss value and the distance loss value. In the above solution, when training the feature extraction network, a multi-task framework can be adopted for training; among them, the main task is the blade surface feature matching task, which is used to distinguish the differences between two different blades, and the auxiliary task is the differential task, which is used to reflect the differential characteristics of the features in the time dimension. Therefore, by training the feature extraction network in the above manner, the performance of the feature extraction network can be improved.

[0015] In an alternative embodiment, the blade surface state detection device further includes: a third acquisition module configured to acquire a registration set and a query set; wherein, the registration set includes a first blade image, and the query set includes a second blade image and a third blade image; a fourth input module configured to input the first blade image and the second blade image into a pre-trained feature extraction network to obtain a first blade feature corresponding to the first blade image and a second blade feature corresponding to the second blade image output by the feature extraction network; a fifth input module configured to input the first blade feature and the second blade feature into a graph neural network to be trained to obtain a first distance between the first blade image and the second blade image; a second calculation module configured to calculate a differential loss value according to the first distance and a second distance between the first blade image and the third blade image; a second update module configured to update the parameters of the graph neural network according to the differential loss value. In the above solution, during the training of the graph neural network, meta-learning can be adopted for training, and a differential loss function is introduced, which is used to reflect the differential characteristics of features in the time dimension. Therefore, by training the graph neural network in the above manner, the performance of the graph neural network can be improved; in addition, since the dynamic changes of samples can be learned, even if the number of blade surface damage samples is insufficient, a detection result with higher accuracy can be obtained.

[0016] In an alternative embodiment, the blade surface state detection device further includes: a sixth input module configured to input the image to be detected into a pre-trained level set graph network to obtain a judgment result output by the level set graph network; the determination module is specifically configured to: determine the blade surface state according to the classification result and the judgment result. In the above solution, the state of the blade surface can also be detected based on the level set graph network, further improving the accuracy of detecting the damage on the blade surface.

[0017] In an alternative embodiment, the blade surface state detection device further includes: a training module configured to train a level set graph network to be trained by using blade damage images and non-blade damage images. In the above solution, during the training of the level set graph network, in addition to using blade damage images, non-blade damage images can also be introduced, so as to enrich the distribution of source domain objects and transfer the surface damage feature knowledge of other objects to the fan blade object. Therefore, the generalization of the level set graph network can be improved.

[0018] In an alternative embodiment, the non-blade damage images include: road surface damage images, aircraft skin damage images, wall damage images, wood damage images, biscuit packaging damage images, fabric damage images, and steel damage images.

[0019] In a third aspect, an embodiment of the present application provides a computer program product, including computer program instructions which, when read and run by a processor, execute the blade surface state detection method as described in the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory, and a bus; the processor and the memory complete communication with each other through the bus; the memory stores computer program instructions executable by the processor, and the processor can execute the blade surface state detection method as described in the first aspect by invoking the computer program instructions.

[0021] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium which stores computer program instructions that, when run on a computer, cause the computer to execute the blade surface state detection method as described in the first aspect.

[0022] To make the above objects, features, and advantages of the present application more obvious and understandable, specific embodiments of the present application are hereinafter given and described in detail in conjunction with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] To more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1 A schematic diagram of a fan provided by an embodiment of the present application;

[0025] Figure 2 A flowchart of a blade surface state detection method provided by an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the training process of a feature extraction network provided by an embodiment of the present application;

[0027] Figure 4 A structural block diagram of a blade surface state detection device provided by an embodiment of the present application;

[0028] Figure 5 A structural block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] China is rich in offshore wind energy resources. The exploitable amount of wind energy resources in the sea area within 25 km from the shore is 190 million KW, and the exploitable amount in the sea area within 50 km from the shore is 360 million KW. China has a natural advantage in the development of offshore wind power. Among them, since China's wind power industry entered the high-speed development period, it has gone through three development stages: before 2013, the focus was on developing the wind-rich areas in the Three-North regions (Northeast, North China, and Northwest); from 2013 to 2018, the main development was on the mountain wind farms in the central and southwestern regions and the coastal wind farms in the southeastern region; since 2018, large-scale development of offshore wind power has begun.

[0030] With the continuous increase in the number of offshore wind turbines put into operation, the problems of fault risks and maintenance cost control caused by the operation characteristics of offshore wind power systems themselves and the harsh environment have become increasingly prominent. For example, in November 2015, a 2.3 MW Siemens wind turbine in the Paludans Flak offshore wind farm in Denmark collapsed, and the entire wind farm was shut down for maintenance, causing huge economic losses; recently, the impeller of a 3.6 MW Siemens wind turbine in the Anholt offshore wind farm in Denmark separated from the nacelle and fell into the sea, and the cause of the accident is still under investigation.

[0031] The equipment structure of offshore wind power systems is complex, and the number of failure modes is huge. The failure rate of wind turbines is relatively high in adverse marine environments. Therefore, in order to ensure the normal use of offshore wind power systems, it is necessary to regularly detect the operating status of offshore wind power systems. However, due to the great influence of weather, ocean, geology and other environments on the construction and maintenance of offshore wind farms, using large floating cranes for detection is time-consuming and laborious, accompanied by high lifting ship costs and great lifting risks. Therefore, a more convenient method is needed to realize the detection of offshore wind power systems.

[0032] Among them, the blade is the wind energy capture device of the wind turbine in the offshore wind power system. The quality of its performance and the length of its service life directly affect the normal operation of the wind turbine and the system efficiency. The main factors leading to blade damage include: 1) Wind turbine blades are generally made of glass fiber composite materials, and defects and damages are inevitable during the forming and transportation processes; 2) Micro-site selection results in uneven wind reaching the impeller surface, which in turn causes the blade to bear unbalanced loads for a long time; 3) The influence of environmental factors such as saltwater corrosion; 4) Extreme weather such as lightning strikes or icing damages the internal structure of the blade.

[0033] During the manufacturing process, typical defects such as pores, delamination, and inclusions may occur in the blades. Among them, pore defects are mainly caused by factors such as poor infiltration of resin and fiber, incomplete air displacement, etc.; delamination defects are mainly due to insufficient resin dosage, secondary forming, etc.; the generation of inclusion defects is mainly due to the mixing of foreign substances during the processing. During operation, defects such as cracks, fractures, and matrix aging may occur in the blades. Among them, external impact is the main cause of cracks; fractures are usually caused by the accumulation of defect damage, and the blades of the fan will not suddenly fracture under normal operating conditions; matrix aging is caused by the long-term operation of the fan blades in harsh conditions such as sand, rain, and salt spray corrosion.

[0034] In the prior art, deep learning is generally used to detect the blades. However, when using the deep learning method, due to the insufficient surface damage samples of the blades, the accuracy of detecting the surface damage of the blades is relatively low.

[0035] The inventor found that there are three differences in the dynamic changes of the surface characteristics of the blades compared with other tasks: First, the appearances of all blades are very similar, especially brand-new blades are almost identical, and obvious differences will only appear on the blade surface after running for a period of time; Second, the blade surface is gradually changing, and the change speed is relatively slow. When establishing a data set, it is necessary to select an appropriate time scale to establish sample pairs, and the time difference scale also needs to be reasonably adjusted; Third, a single fan has multiple blades. Before analyzing the changes in the surface characteristics of the blades, it is also necessary to determine which blade, that is, to clarify the blade identification.

[0036] Based on the above analysis, the embodiment of the present application provides a method for detecting the surface state of the blade. The method for detecting the surface state of the blade is a large-time-scale meta-learning differential network detection method. Through this method for detecting the surface state of the blade, even if the surface damage samples of the blade are insufficient, a relatively high-accuracy detection result can be obtained. It can be understood that the method for detecting the internal state of the blade provided by the embodiment of the present application can be applied not only to the blades of the fan in the offshore wind power system, but also to the blades in other scenarios. The embodiment of the present application does not make specific limitations on this.

[0037] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application.

[0038] Before introducing the method for detecting the surface state of the blade provided by the embodiment of the present application, the fan structure applicable to the embodiment of the present application will be introduced first. Please refer to Figure 1 , Figure 1A schematic diagram of a wind turbine provided by an embodiment of the present application. The wind turbine 100 includes three blades 101, a nacelle 102, and a tower 103. Among them, the blades 101 are arranged on the nacelle 102, and the nacelle 102 can control the rotation and flipping of the blades 101; the nacelle 102 is arranged at one end of the tower 103, and the other end of the tower 103 is arranged on the ground.

[0039] It can be understood that the embodiments of the present application do not make specific limitations on the number of blades 101, the specific position where the wind turbine 100 is arranged, etc. Figure 1 The wind turbine 100 shown is only an example provided by the embodiments of the present application, and those skilled in the art can make appropriate adjustments according to the actual situation, in combination with Figure 1 for appropriate adjustment.

[0040] Furthermore, on the basis of the above embodiments, an image acquisition device 104 can be arranged on the top of the nacelle 102 for acquiring images of the blades 101. As an implementation manner, the above image acquisition device 104 can be a starlight-level camera and has functions such as active pan-tilt control, infrared night vision, heating function, and windshield wiper.

[0041] Furthermore, on the basis of the above embodiments, the deployment method of the computing unit can be selected according to the scale of the wind farm. For example, for a small-scale wind farm, the ring network communication pressure is small. Therefore, the image acquisition device 104 on the wind turbine 100 can directly transmit the data back to the central server deployed in the central control room, and the central server performs unified operations on the data, thereby reducing the hardware cost; for a large-scale wind farm, the ring network communication pressure is large. Therefore, a computing unit can be directly deployed on the wind turbine 100, and the computing unit processes the data acquired by the image acquisition device 104 and then uploads it to the server for storage.

[0042] Furthermore, on the basis of the above embodiments, since the image acquisition device 104 on the top of the wind turbine 100 may not be able to obtain high-quality complete blade images, a segmented shooting strategy can be used, that is, whenever a blade 101 rotates into the acquisition range of the image acquisition device 104, the pan-tilt controls the image acquisition device 104 to move accordingly and shoots the blade 101.

[0043] Based on the above wind turbine 100, the blade surface state detection method provided by the embodiments of the present application is introduced below. This blade internal state detection method can be used to detect the surface state of the blades 101 in the above wind turbine 100.

[0044] Please refer to Figure 2 , Figure 2 A flowchart of a blade surface state detection method provided by an embodiment of the present application. This blade surface state detection method can include the following steps:

[0045] Step S201: Obtain the image to be detected corresponding to the blade to be detected and the support set.

[0046] Step S202: Input the image to be detected and the support set into a pre-trained feature extraction network to obtain the first image feature corresponding to the image to be detected and the second image feature corresponding to the support set output by the feature extraction network.

[0047] Step S203: Input the first image feature and the second image feature into a pre-trained graph neural network to obtain the classification result output by the graph neural network.

[0048] Step S204: Determine the surface state of the blade to be detected according to the classification result.

[0049] Specifically, in the above step S201, the blade to be detected refers to the blade that needs to be in a state of detection currently. It can be understood that the blade to be detected can be one or more blades on the wind turbine, or all the blades on the wind turbine.

[0050] The image to be detected corresponding to the blade to be detected refers to the image of the blade to be detected collected by an image acquisition device arranged at the top of the wind turbine. And the support set includes the sample images corresponding to each blade on the wind turbine. By comparing the image to be detected with the sample images, it can be determined which blade on the wind turbine the blade to be detected corresponds to, and whether there is damage on the surface of the blade to be detected can be determined.

[0051] It should be noted that the specific implementation manner of obtaining the image to be detected and the support set in the embodiments of the present application is not specifically limited, and those skilled in the art can make appropriate adjustments according to the actual situation. For example, the image to be detected or the support set sent by an external device can be received; or the image to be detected or the support set can be read from the cloud, etc.

[0052] In the above step S202, the image to be detected can be input into a pre-trained feature extraction network to obtain the first image feature output by the feature extraction network; at the same time, the support set can be input into the same feature extraction network to obtain the second image feature output by the feature extraction network.

[0053] Among them, the specific implementation manner of training the feature extraction network will be described in detail in the subsequent embodiments and will not be introduced here for the time being.

[0054] In the above step S203, the first image feature and the second image feature can be input into a pre-trained graph neural network to obtain the classification result output by the graph neural network. Among them, the loss function corresponding to the graph neural network is a differential loss function, and this differential loss function reflects the differential characteristics of the first image feature and the second image feature in the time dimension.

[0055] Similarly, the specific implementation manner of training the graph neural network will be described in detail in the subsequent embodiments and will not be introduced here for the time being.

[0056] In the above step S204, according to the above classification result, it can be determined which blade in the support set the blade to be detected corresponds to, and it can also be determined whether there is damage on the surface of the blade to be detected.

[0057] In the above solution, based on the pre-trained feature extraction network, the image features of the image to be detected and the support set can be extracted, and then the extracted image features are input into the graph neural network to obtain the corresponding classification result. According to the above classification result, the detection of the surface state of the blade can be realized. Among them, since a differential loss function is introduced in the process of training the graph neural network, the dynamic changes of the samples can be learned, and even if the number of damaged blade surface samples is insufficient, a detection result with higher accuracy can be obtained.

[0058] Further, on the basis of the above embodiments, before the above step S201, the feature extraction network can be trained first. The process of training the feature extraction network can include the following steps:

[0059] Step 1), obtain at least one first sample image corresponding to the blades on the wind turbine and multiple second sample images of other blades on the wind turbine at multiple times.

[0060] Step 2), input the first sample image and the second sample images into the feature extraction network to be trained, and obtain the first sample feature corresponding to the first sample image and multiple second sample features corresponding to the second sample images output by the feature extraction network.

[0061] Step 3), calculate the matching loss value between the first sample feature and the second sample features, and calculate the distance loss value between the multiple second sample features.

[0062] Step 4), update the parameters of the feature extraction network according to the matching loss value and the distance loss value.

[0063] Specifically, when training the feature extraction network, a multi-task framework can be adopted for training. As an implementation, the main task can be the task of matching the surface features of the blades, and the auxiliary task can be the differential task. Adding the auxiliary task can make the feature distribution more uniform in the feature space. Among them, the learning objectives of both the main task and the auxiliary task are: the longer the sampling interval time of the same blade, the greater the feature difference.

[0064] Taking the example of a wind turbine with three blades, the above training process will be introduced. Please refer to Figure 3 , Figure 3 which is a schematic diagram of the training process of a feature extraction network provided by an embodiment of the present application.

[0065] Among them, the input of the feature extraction network includes: the first sample image corresponding to blade 1, the first sample image corresponding to blade 2, and three second sample images collected at three different times corresponding to blade 3; the output of the feature extraction network includes: the first sample feature corresponding to blade 1, the first sample feature corresponding to blade 2, and three second sample features corresponding to blade 3.

[0066] First, the main task will be introduced. The main task branch is a binary classification task for predicting whether two pictures are the same picture. As an implementation, the output layer of the main task branch can be a fully connected layer, denoted as

[0067] When training the main task branch, it is necessary to divide the time-series dataset S of the blade surface pictures according to the blade identification and combine them into sample pairs Then, standard metric learning or siamese network loss is used for training. Among them, the time-series dataset S of the blade surface pictures can be expressed as:

[0068] S = {b l，m，n |l = 1, 2,..., L; m = 1, 2,..., M; n = 1, 2,..., N};

[0069] Among them, L is the number of blades, M is the number of sampling times, N is the number of pictures sampled each time, and m increases with time. By extracting data from S in different ways, it can be used for the pre-training of the feature extraction network and the training of the graph neural network respectively.

[0070] It is understandable that during the training of the main task branch, the generation process of a single sample is as follows: Two samples are randomly selected from the spaces of l1, l2 ∈ {1, 2, …, N}, m1, m2 ∈ {M - 3, M - 2, …, M}, n1, n2 ∈ {1, 2, …, N}. When l1 = l2 and m1 = m2, the label is label = 1; when l1 ≠ l2, label = 0; in other cases, the sample is discarded. As an implementation, since the features on the surface of the new blade are very similar, m1, m2 ∈ {M - 3, M - 2, …, M} can be selected to make the feature discrimination more significant.

[0071] Concatenate the features and the features and input them into the matching network to obtain the matching output result as As an implementation, the main task branch can be trained using binary cross - entropy loss. Among them, the binary cross - entropy loss function is defined as:

[0072] loss match = BCE(pred, label);

[0073] Secondly, the auxiliary task is introduced. The auxiliary task branch is a distance metric branch, and its input is a triple satisfying m1 ≤ m3 < m3.

[0074] As an implementation, the auxiliary task branch can be trained using a differential loss function. Among them, denote the feature distance metric function as Then the differential loss function is defined as:

[0075]

[0076] Among them, the meaning of the above formula is The distance between and the feature should be greater than The distance between and, which reflects the differential characteristics of the feature in the time dimension.

[0077] Finally, combining the main task and the auxiliary task, the total loss for training the feature extraction network can be expressed as:

[0078] loss total = w1·loss match + w2·loss diff ;

[0079] Among them, w1 and w2 are the weight values of the two branches.

[0080] In the above solution, when training the feature extraction network, a multi-task framework can be adopted for training. Among them, the main task is the leaf surface feature matching task, which is used to distinguish the differences between two different leaves, and the auxiliary task is the differential task, which is used to reflect the differential characteristics of the features in the time dimension. Therefore, training the feature extraction network in the above manner can improve the performance of the feature extraction network.

[0081] Further, on the basis of the above embodiment, before the above step S201, the graph neural network can be trained first. The process of training the graph neural network may include the following steps:

[0082] Step 1), obtain a registration set and a query set. Among them, the registration set includes the first leaf image, and the query set includes the second leaf image and the third leaf image.

[0083] Step 2), input the first leaf image and the second leaf image into the pre-trained feature extraction network to obtain the first leaf feature corresponding to the first leaf image and the second leaf feature corresponding to the second leaf image output by the feature extraction network.

[0084] Step 3), input the first leaf feature and the second leaf feature into the graph neural network to be trained to obtain the first distance between the first leaf image and the second leaf image.

[0085] Step 4), calculate the differential loss value according to the first distance and the second distance between the first leaf image and the third leaf image.

[0086] Step 5), update the parameters of the graph neural network according to the differential loss value.

[0087] Specifically, the main idea of the meta-learning differential network is to use the meta-learning framework to train the network to learn the dynamic change process of the leaf appearance features. In the embodiment of the present application, first, the feature extraction network can be used to extract the leaf surface features, and then the features of the registration set and the query set are uniformly input into the graph neural network (Graph Neural Network, GNN) for training.

[0088] As an implementation manner, the structure of the graph neural network can be consistent with the small sample learning task of 3 way 3 shot. However, in the embodiment of the present application, a differential loss function is introduced to train the graph neural network to learn the dynamic change relationship of the samples.

[0089] It can be understood that in order to train the above graph neural network, a meta-task data set needs to be constructed. The steps of constructing the meta-task data set are as follows:

[0090] The first step is to initialize the data set ind = 0.

[0091] In the second step, generate random numbers l1, l2, l3 ∈ {1, 2, …, N} such that l1 ≠ l2 ≠ l3.

[0092] In the third step, generate a random number m ij ∈ {1, 2, …, M} and n ij ∈ {1, 2, …, N}, where i = 1, 2, 3, j = 1, 2, …, 5, and satisfy m i1 < m i2 < m i3 < m i4 < m i5 .

[0093] In the fourth step, generate the ind-th group of sample pairs (p ind , q ind , o ind ), where Update the support set and the query set T = T ∪ {(p ind , q ind , o ind )}.

[0094] In the fifth step: increment ind by one, jump to step 2, and repeat the above steps until enough training samples are generated.

[0095] Next, the graph neural network can be trained. The above meta-task dataset (p ind , q ind , o ind ) can be divided into a registration set P = {p i，j |i = 1, 2, 3; j = 1, 2, 3} and a query set Query = {q k , o k |k = 1, 2, 3}, and input into the feature extraction network. The features obtained by the feature extraction network are respectively and Taking (v i，j , u k ) as the initial input nodes of the graph and randomly initializing the weights, the final output result can be obtained where The edges of represent the distances between nodes. In the same way, can be obtained

[0096] Denote the distance metric function of the feature graph edges as The differential loss function can be expressed as:

[0097]

[0098] where the meaning of the above formula is that it is desired that qk The distance to the p i，j feature should be less than o k The distance to the p i，j between them.

[0099] In the above solution, during the process of training the graph neural network, meta-learning can be adopted for training, and a differential loss function is introduced. This differential loss function is used to reflect the differential characteristics of features in the time dimension. Therefore, by training the graph neural network in the above manner, the performance of the graph neural network can be improved; in addition, since the dynamic changes of the samples can be learned, even if the number of damaged samples on the blade surface is insufficient, a detection result with relatively high accuracy can be obtained.

[0100] Furthermore, on the basis of the above embodiments, before the above step S204, the blade surface state detection method provided by the embodiments of the present application may further include the following steps:

[0101] Input the image to be detected into a pre-trained level set graph network to obtain a judgment result output by the level set graph network.

[0102] Correspondingly, the above step S204 may specifically include the following steps:

[0103] Determine the blade surface state according to the classification result and the judgment result.

[0104] Specifically, the level set map (LSM) is a method for annotating the object contour.

[0105] The level set graph network is a type of active contour method. It adopts the same structure as semantic segmentation networks such as U-Net and directly trains the network using supervision information. In some tasks, the level set graph network can complete the instance segmentation task of a single category, and the computational complexity is much smaller than that of the instance segmentation network. In the embodiments of the present application, the instance segmentation ability of the level set graph network is used to make the extracted features focus on the object and the damaged part, and instance-level domain alignment is performed on this basis.

[0106] As an implementation manner, the level set graph network can be divided into two parts: the main task and the auxiliary task. The main task is a classification task to determine whether there is damage to the current object; while the auxiliary task is to extract the LSM. Among them, the main task and the auxiliary task branches share an encoder. The encoder and the auxiliary task branch form a Unet structure. The main task classifier is a fully connected layer, and the domain adaptation is designed for the features output by the encoder.

[0107] In the above solution, the state of the blade surface can also be detected based on the level set graph network, further improving the accuracy of detecting damage on the blade surface.

[0108] Further, on the basis of the above embodiments, before the step of inputting the image to be detected into the pre-trained level set graph network to obtain the judgment result output by the level set graph network, the level set graph network can be trained first. The process of training the level set graph network may include the following steps:

[0109] Use the blade damage images and non-blade damage images to train the level set graph network to be trained.

[0110] Specifically, in the process of training the level set graph network, in addition to using blade damage images, non-blade damage images can also be introduced. It can be understood that the specific implementation manner of the non-blade damage images in the embodiments of the present application is not specifically limited, and those skilled in the art can make appropriate adjustments according to the actual situation. For example, the non-blade damage images may include: road surface damage images, aircraft skin damage images, wall damage images, wood damage images, biscuit packaging damage images, fabric damage images, and steel damage images.

[0111] In the above solution, in the process of training the level set graph network, in addition to using blade damage images, non-blade damage images can also be introduced, so as to enrich the distribution of source domain objects and transfer the surface damage feature knowledge of other objects to the fan blade object. Therefore, the generalization of the level set graph network can be improved.

[0112] Please refer to Figure 4 , Figure 4 , which is a structural block diagram of a blade surface state detection device provided by an embodiment of the present application. The blade surface state detection device 400 includes: a first acquisition module 401, configured to acquire an image to be detected corresponding to the blade to be detected and a support set; wherein, the support set includes sample images corresponding to each blade on the fan; a first input module 402, configured to input the image to be detected and the support set into a pre-trained feature extraction network to obtain a first image feature corresponding to the image to be detected and a second image feature corresponding to the support set output by the feature extraction network; a second input module 403, configured to input the first image feature and the second image feature into a pre-trained graph neural network to obtain a classification result output by the graph neural network; wherein, the loss function corresponding to the graph neural network is a differential loss function; a determination module 404, configured to determine the blade surface state of the blade to be detected according to the classification result.

[0113] In the above solution, based on a pre-trained feature extraction network, the image features of the image to be detected and the support set can be extracted. Then, the extracted image features are input into the graph neural network, and the corresponding classification results can be obtained. According to the above classification results, the detection of the blade surface state can be realized. Among them, since a differential loss function is introduced during the training of the graph neural network, the dynamic changes of the samples can be learned. Even if the number of blade surface damage samples is insufficient, a detection result with high accuracy can be obtained.

[0114] Further, on the basis of the above embodiment, the blade surface state detection device 400 further includes: a second acquisition module, configured to acquire a first sample image corresponding to at least one blade on the wind turbine and a plurality of second sample images of other blades on the wind turbine at multiple times; a third input module, configured to input the first sample image and the second sample images into a feature extraction network to be trained, and obtain a first sample feature corresponding to the first sample image and a plurality of second sample features corresponding to the second sample images output by the feature extraction network; a first calculation module, configured to calculate a matching loss value between the first sample feature and the second sample features, and calculate a distance loss value between the plurality of second sample features; a first update module, configured to update the parameters of the feature extraction network according to the matching loss value and the distance loss value.

[0115] In the above solution, when training the feature extraction network, a multi-task framework can be adopted for training; among them, the main task is the blade surface feature matching task, which is used to distinguish the differences between two different blades, and the auxiliary task is the differential task, which is used to reflect the differential characteristics of the features in the time dimension. Therefore, training the feature extraction network in the above manner can improve the performance of the feature extraction network.

[0116] Further, on the basis of the above embodiments, the blade surface state detection device 400 further includes: a third acquisition module, configured to acquire a registration set and a query set; wherein, the registration set includes a first blade image, and the query set includes a second blade image and a third blade image; a fourth input module, configured to input the first blade image and the second blade image into a pre-trained feature extraction network, to obtain a first blade feature corresponding to the first blade image and a second blade feature corresponding to the second blade image output by the feature extraction network; a fifth input module, configured to input the first blade feature and the second blade feature into a graph neural network to be trained, to obtain a first distance between the first blade image and the second blade image; a second calculation module, configured to calculate a differential loss value according to the first distance and a second distance between the first blade image and the third blade image; a second update module, configured to update the parameters of the graph neural network according to the differential loss value.

[0117] In the above solution, during the training of the graph neural network, meta-learning can be adopted for training, and a differential loss function is introduced, which is used to reflect the differential characteristics of features in the time dimension. Therefore, by adopting the above method to train the graph neural network, the performance of the graph neural network can be improved; in addition, since the dynamic changes of the samples can be learned, even if the number of blade surface damage samples is insufficient, a detection result with high accuracy can be obtained.

[0118] Further, on the basis of the above embodiments, the blade surface state detection device 400 further includes: a sixth input module, configured to input the image to be detected into a pre-trained level set graph network, to obtain a judgment result output by the level set graph network; the determination module is specifically configured to: determine the blade surface state according to the classification result and the judgment result.

[0119] In the above solution, the state of the blade surface can also be detected based on the level set graph network, further improving the accuracy of detecting the damage to the blade surface.

[0120] Further, on the basis of the above embodiments, the blade surface state detection device 400 further includes: a training module, configured to train the level set graph network to be trained by using blade damage images and non-blade damage images.

[0121] In the above solution, during the training of the level set graph network, in addition to using blade damage images, non-blade damage images can also be introduced, so as to enrich the distribution of source domain objects and transfer the surface damage feature knowledge of other objects to the fan blade object. Therefore, the generalization of the level set graph network can be improved.

[0122] Further, based on the above embodiments, the non-blade damage images include: road surface damage images, aircraft skin damage images, wall damage images, wood damage images, biscuit packaging damage images, fabric damage images, and steel damage images.

[0123] Please refer to Figure 5 , Figure 5 which is a structural block diagram of an electronic device provided by an embodiment of the present application. The electronic device 500 includes: at least one processor 501, at least one communication interface 502, at least one memory 503, and at least one communication bus 504. Among them, the communication bus 504 is used to realize the direct connection and communication of these components. The communication interface 502 is used to communicate with other node devices for signaling or data. The memory 503 stores machine-readable instructions executable by the processor 501. When the electronic device 500 runs, the processor 501 communicates with the memory 503 through the communication bus 504. When the machine-readable instructions are called by the processor 501, the above-mentioned blade surface state detection method is executed.

[0124] For example, the processor 501 of the embodiment of the present application can read a computer program from the memory 503 through the communication bus 504 and execute the computer program to implement the following method: Step S201: Obtain a to-be-detected image and a support set corresponding to the to-be-detected blade. Step S202: Input the to-be-detected image and the support set into a pre-trained feature extraction network to obtain a first image feature corresponding to the to-be-detected image and a second image feature corresponding to the support set output by the feature extraction network. Step S203: Input the first image feature and the second image feature into a pre-trained graph neural network to obtain a classification result output by the graph neural network. Step S204: Determine the blade surface state of the to-be-detected blade according to the classification result.

[0125] Among them, the processor 501 includes one or more, which can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 501 can be a general-purpose processor, including a Central Processing Unit (CPU), a Micro Controller Unit (MCU), a Network Processor (NP), or other conventional processors; it can also be a dedicated processor, including a Neural-network Processing Unit (NPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuits (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Moreover, when there are multiple processor 501s, a part of them can be general-purpose processors and another part can be dedicated processors.

[0126] The memory 503 includes one or more, which can be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electric Erasable Programmable Read-Only Memory (EEPROM), etc.

[0127] It can be understood that Figure 5 The structure shown is only schematic, and the electronic device 500 may also include more or fewer components than those Figure 5 shown, or have a configuration different from that Figure 5 shown. Figure 5Each component shown in can be implemented using hardware, software, or a combination thereof. In the embodiments of the present application, the electronic device 500 can be, but is not limited to, physical devices such as desktop computers, laptop computers, smartphones, smart wearable devices, vehicle-mounted devices, etc., and can also be virtual devices such as virtual machines. Additionally, the electronic device 500 does not necessarily have to be a single device, and can also be a combination of multiple devices, such as a server cluster, and so on.

[0128] The embodiments of the present application also provide a computer program product, including a computer program stored on a computer-readable storage medium. The computer program includes computer program instructions. When the computer program instructions are executed by a computer, the computer is capable of executing the steps of the blade surface state detection method in the above embodiments, for example, including: obtaining a to-be-detected image and a support set corresponding to the to-be-detected blade; wherein, the support set includes sample images corresponding to each blade on the wind turbine; inputting the to-be-detected image and the support set into a pre-trained feature extraction network to obtain a first image feature corresponding to the to-be-detected image and a second image feature corresponding to the support set output by the feature extraction network; inputting the first image feature and the second image feature into a pre-trained graph neural network to obtain a classification result output by the graph neural network; wherein, the loss function corresponding to the graph neural network is a differential loss function; determining the blade surface state of the to-be-detected blade according to the classification result.

[0129] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores computer program instructions. When the computer program instructions are run by a computer, the computer executes the blade surface state detection method described in the foregoing method embodiments.

[0130] In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some communication interfaces. The indirect coupling or communication connection of the apparatus or unit can be in an electrical, mechanical, or other form.

[0131] In addition, the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] Furthermore, each functional module in various embodiments of the present application may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

[0133] It should be noted that if a function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, and other various media that can store program codes.

[0134] In this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0135] The above description is only for the embodiments of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting the surface state of a blade, characterized in that, Including: Obtain a to-be-detected image corresponding to a to-be-detected blade and a support set; wherein, the support set includes sample images corresponding to each blade on the wind turbine; Input the to-be-detected image and the support set into a pre-trained feature extraction network to obtain a first image feature corresponding to the to-be-detected image and a second image feature corresponding to the support set output by the feature extraction network; Input the first image feature and the second image feature into a pre-trained graph neural network to obtain a classification result output by the graph neural network; wherein, the loss function corresponding to the graph neural network is a differential loss function; Determine the blade surface state of the to-be-detected blade according to the classification result.

2. The method for detecting the surface state of a blade according to claim 1, wherein Before obtaining the to-be-detected image corresponding to the to-be-detected blade and the support set, the method further includes: Obtain a first sample image corresponding to at least one blade on the wind turbine and multiple second sample images of other blades on the wind turbine at multiple times; Input the first sample image and the second sample images into a to-be-trained feature extraction network to obtain a first sample feature corresponding to the first sample image and multiple second sample features corresponding to the second sample images output by the feature extraction network; Calculate a matching loss value between the first sample feature and the second sample features, and calculate a distance loss value between the multiple second sample features; Update the parameters of the feature extraction network according to the matching loss value and the distance loss value.

3. The method for detecting the surface state of a blade according to claim 1, characterized in that Before obtaining the to-be-detected image corresponding to the to-be-detected blade and the support set, the method further includes: Obtain a registration set and a query set; wherein, the registration set includes a first blade image, and the query set includes a second blade image and a third blade image; Input the first blade image and the second blade image into a pre-trained feature extraction network to obtain a first blade feature corresponding to the first blade image and a second blade feature corresponding to the second blade image output by the feature extraction network; Input the first blade feature and the second blade feature into a to-be-trained graph neural network to obtain a first distance between the first blade image and the second blade image; Calculate a differential loss value according to the first distance and a second distance between the first blade image and the third blade image; Update the parameters of the graph neural network according to the differential loss value.

4. The blade surface state detection method according to any one of claims 1-3, characterized in that, Before determining the blade surface state of the to-be-detected blade according to the classification result, the method further includes: Input the to-be-detected image into a pre-trained level set graph network to obtain a judgment result output by the level set graph network; The determining the blade surface state of the to-be-detected blade according to the classification result includes: Determine the blade surface state according to the classification result and the judgment result.

5. The blade surface state detection method according to claim 4, wherein Before inputting the to-be-detected image into a pre-trained level set graph network to obtain a judgment result output by the level set graph network, the method further includes: Train the to-be-trained level set graph network by using blade damage images and non-blade damage images.

6. The method for detecting the surface state of a blade according to claim 5, wherein The non-blade damage images include: road surface damage images, aircraft skin damage images, wall damage images, wood damage images, biscuit packaging damage images, fabric damage images, and steel damage images.

7. A device for detecting the surface state of a blade, characterized in that, Comprising: A first acquisition module, configured to acquire a to-be-detected image corresponding to a to-be-detected blade and a support set; wherein, the support set includes sample images corresponding to each blade on the wind turbine. A first input module, configured to input the to-be-detected image and the support set into a pre-trained feature extraction network, to obtain a first image feature corresponding to the to-be-detected image and a second image feature corresponding to the support set output by the feature extraction network. A second input module, configured to input the first image feature and the second image feature into a pre-trained graph neural network, to obtain a classification result output by the graph neural network; wherein, the loss function corresponding to the graph neural network is a differential loss function. A determination module, configured to determine the blade surface state of the to-be-detected blade according to the classification result.

8. A computer program product, characterized in that, Comprising computer program instructions, when the computer program instructions are read and run by a processor, the blade surface state detection method according to any one of claims 1-6 is executed.

9. An electronic device, characterized in that, Comprising: A processor, a memory, and a bus; The processor and the memory complete communication with each other through the bus; The memory stores computer program instructions executable by the processor, and the processor can execute the blade surface state detection method according to any one of claims 1-6 by invoking the computer program instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, when the computer program instructions are run by a computer, the computer is enabled to execute the blade surface state detection method according to any one of claims 1-6.

Images