A fan blade detection method based on wall-climbing robot and image recognition

Abstract

The application discloses a fan blade detection method based on a wall-climbing robot and image recognition, which comprises the following steps: deploying a wall-climbing robot in a blade cavity, wherein the wall-climbing robot is provided with an advancing camera, a left sweeping camera and a right sweeping camera; the wall-climbing robot works in a web area in the blade cavity, collects video frame data while climbing on the web; image splicing is performed on the video frame data of each camera to obtain a spliced image; and a target detection method is used to detect faults in the spliced image. The advancing camera and the sweeping camera of the wall-climbing robot are used to shoot pictures, the panoramic shooting function of the blade cavity is realized according to the camera pose and the image splicing algorithm, the fault information in the blade cavity is diagnosed according to the image recognition algorithm, the fault position is located according to the record of the advancing direction of the wall-climbing robot, and the distance sensor is arranged on the sweeping camera, so that the pixel scale of the camera can be roughly estimated and used for estimating the size of the fault.

Description

Technical Field

[0001] This invention relates to the field of wind turbine monitoring technology, and in particular to a wind turbine blade detection method based on a wall-climbing robot and image recognition. Background Technology

[0002] In the field of wind power generation, blades are one of the most important major components and require regular inspections. The main diagnostic method is manual inspection, including annual and semi-annual inspections. The main techniques involve using telescopes to inspect for external defects and taking photos, as well as manually entering the blades to inspect for internal defects and taking photos. Manually diagnosing blade faults through visual means, such as light transmission, cracks, wrinkles, and whitening, requires a certain level of expertise to determine the severity of the fault.

[0003] With the increasing number of blade accidents, more and more attention is being paid to blade inspection, and corresponding blade inspection methods are gradually maturing. The more mature external blade inspection has gradually shifted from manual telescope inspection to drone inspection, which can inspect the outside of the blade, but this method cannot detect internal blade faults.

[0004] Robotic inspection methods have also emerged for inspecting the interior of wind turbine blades. One such robot (CN119664595A) uses a Knum wheel chassis to inspect and photograph the interior of the nacelle and blades. However, internal inspection is still in the exploratory and research stage. Due to the complex internal structure of the blades, there are still many technical challenges to overcome, and a truly feasible technical solution has not yet been formed. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a wind turbine blade detection method based on a wall-climbing robot and image recognition. The method utilizes the forward-facing camera and sweeping camera of the wall-climbing robot to capture images. Based on the camera pose and image stitching algorithm, a panoramic view of the blade cavity is achieved. Then, an image recognition algorithm is used to diagnose fault information within the blade cavity. The fault location is determined based on the recorded movement of the wall-climbing robot. The sweeping camera, equipped with a distance sensor, can roughly estimate the camera's pixel scale, which is used to estimate the magnitude of the fault.

[0006] The embodiments of the present invention provide the following solutions:

[0007] This invention provides a method for detecting wind turbine blades based on a wall-climbing robot and image recognition. The method includes:

[0008] S1. Deploy a wall-climbing robot inside the blade cavity. The wall-climbing robot is equipped with a forward-facing camera, a left-sweeping camera, and a right-sweeping camera.

[0009] S2. The wall-climbing robot operates in the web area inside the blade cavity, collecting video frame data while climbing the web.

[0010] S3. Perform image stitching on the video frame data of each shot to obtain a stitched image;

[0011] S4. Use target detection methods to detect faults in the stitched images.

[0012] In an optional embodiment, the wall-climbing robot described in step S1 climbs the wall using a negative pressure adsorption walking method.

[0013] In an alternative embodiment, the forward-facing camera described in step S1 is calibrated before operation.

[0014] In one optional embodiment, the web region described in step S2 includes three cavities: an anterior edge cavity, a middle cavity, and a posterior edge cavity.

[0015] In one optional embodiment, the scanning surface of the leading edge cavity includes the windward side, the leeward side, and the web surface of the leading edge cavity. Video frame data for the windward and leeward sides of the leading edge cavity are acquired by a left and right sweeping camera, while video frame data for the web surface is acquired by a forward-facing lens. The scanning surface of the intermediate cavity includes the leading web surface, the windward side, the leeward side, and the trailing web surface of the intermediate cavity. Video frame data for the leading web surface, the windward side, and the leeward side of the intermediate cavity are acquired by a left and right sweeping camera, while video frame data for the trailing web surface is acquired by a forward-facing lens. The scanning surface of the trailing edge cavity includes the windward side, the leeward side, and the web surface of the trailing edge cavity. Video frame data for the windward and leeward sides of the trailing edge cavity are acquired by a left and right sweeping camera, while video frame data for the web surface is acquired by a forward-facing lens.

[0016] In one optional embodiment, the stitched image in step S3 includes a stitched image of the leading edge cavity, a stitched image of the middle cavity, and a stitched image of the trailing edge cavity.

[0017] In one optional embodiment, the stitching process of the leading edge cavity image is as follows: the video frame data of the windward side and the leeward side of the leading edge cavity are segmented with the mold seam as the boundary to obtain the segmented image of the windward side and the segmented image of the leeward side; the video frame data of the web surface of the leading edge cavity are segmented with the web boundary line as the boundary to obtain the segmented image of the web surface; the segmented image of the windward side, the segmented image of the leeward side, and the segmented image of the web surface are stitched together to obtain the stitched image of the leading edge cavity.

[0018] In one optional embodiment, the stitching process of the intermediate cavity image is as follows: the video frame data of the windward side and the leeward side of the intermediate cavity are segmented with the mold seam as the boundary to obtain the windward side segmented image and the leeward side segmented image of the intermediate cavity; the video frame data of the front edge web surface and the rear edge web surface of the intermediate cavity are segmented with the boundary line of their respective webs as the boundary to obtain the front edge web surface segmented image and the rear edge web surface segmented image of the intermediate cavity; the windward side segmented image, the leeward side segmented image, the front edge web surface segmented image, and the rear edge web surface segmented image of the intermediate cavity are stitched together to obtain the intermediate cavity image.

[0019] In an optional embodiment, when a fault is detected during the fault detection process in step S4, the location and area of ​​the fault are calculated and output using the distance sensors of the left and right sweep cameras.

[0020] The beneficial effects of this invention based on its technical solution are as follows:

[0021] This invention proposes a practical and effective solution that utilizes the forward camera and sweep camera of a wall-climbing robot to capture images. Based on the camera pose and image stitching algorithm, a panoramic shooting function of the blade cavity is achieved. Then, the fault information inside the blade cavity is diagnosed based on the image recognition algorithm. Based on the record of the wall-climbing robot's travel direction, the fault location is located. The sweep camera is equipped with a distance sensor, which can roughly estimate the camera's pixel scale to estimate the size of the fault. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the process of the present invention.

[0024] Figure 2 This is a schematic diagram of the wall-climbing robot.

[0025] Figure 3 This is a schematic diagram of the wall-climbing robot in operation.

[0026] Figure 4 This is a diagram illustrating the status of the remote control.

[0027] Figure 5 This is a schematic diagram showing the shooting fields of the left and right sweep cameras.

[0028] Figure 6 This is a schematic diagram of the internal cavity structure of the blade.

[0029] Figure 7 This is a schematic diagram of the stitching of the leading edge cavity images.

[0030] Figure 8 A schematic diagram illustrating the segmentation of video frame data from the left and right sweep cameras in the anterior edge cavity.

[0031] Figure 9 This is a schematic diagram of the video frame data from the left and right sweep cameras of the segmented leading edge cavity.

[0032] Figure 10 A schematic diagram showing the stitching effect of segmenting the leading edge windward side image and the leading edge leeward side image.

[0033] Figure 11 This is a schematic diagram of the stitching of images of the trailing edge cavity.

[0034] Figure 12 This is a schematic diagram of the stitching of images of the intermediate cavity.

[0035] Figure 13 Schematic diagram of the stitched image of the leading edge cavity.

[0036] Figure 14 This is a schematic diagram illustrating the fault identification effect.

[0037] Figure 15 This is a schematic diagram illustrating the principle of fault area calculation.

[0038] In the diagram, 1-Climbing robot, 2-Forward camera, 3-Left sweep camera, 4-Right sweep camera, 5-Front fill light, 6-Side fill light, 7-Windward surface of the leading edge cavity, 8-Leftward surface of the leading edge cavity, 9-Web surface of the leading edge cavity, 10-Leftward surface of the middle cavity, 11-Windward surface of the trailing edge cavity, 12-Leftward surface of the trailing edge cavity, 13-Web surface of the trailing edge cavity, 14-Leading edge mold joint, 15-Trailing edge mold joint. Detailed Implementation

[0039] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the protection scope of the embodiments of the present invention.

[0040] This embodiment provides a wind turbine blade detection method based on a wall-climbing robot and image recognition, referring to... Figure 1 The method includes:

[0041] S1. Reference Figure 2A wall-climbing robot 1 is deployed inside the blade cavity. This robot is equipped with a forward-facing camera 2, a left-sweeping camera 3, and a right-sweeping camera 4, and may also include a front supplementary light 5 and a side supplementary light 6. The robot can climb the wall using a negative pressure adsorption method. The forward-facing camera is calibrated before operation, following the calibration method used for the front-facing camera in a car's AVM (Around View Monitor) system. This yields a top-down view of the blade's belly scan image. Based on the 2-meter mark on the robot's running distance on flat ground, a 2-meter marker frame for the forward-facing camera is obtained on the image.

[0042] S2. The wall-climbing robot operates in the web area within the blade cavity, simultaneously climbing the web and acquiring video frame data. The robot's operation and the remote control's status are as follows: Figure 3 and Figure 4 As shown. The field of view of the left and right sweep cameras are as follows. Figure 5 The blue-purple area is shown.

[0043] Reference Figure 6 The ventral region comprises three cavities: the anterior cavity, the intermediate cavity, and the posterior cavity. The scanning surfaces of the anterior cavity include the windward side (7), the leeward side (8), and the ventral surface (9). Video frame data for the windward and leeward sides of the anterior cavity are acquired by the left and right sweep cameras, while video frame data for the ventral surface is acquired by the forward-facing lens. The scanning surfaces of the intermediate cavity include the anterior ventral surface, the windward side, the leeward side (10), and the posterior ventral surface. The video frame data of the windward side and the leeward side of the central cavity are acquired by the left and right sweep cameras, while the video frame data of the rear edge web surface of the central cavity is acquired by the forward lens. The scanning surfaces of the rear edge cavity include the windward side 11, the leeward side 12, and the web surface 13 of the rear edge cavity. The video frame data of the windward side and the leeward side of the rear edge cavity are acquired by the left and right sweep cameras, while the video frame data of the web surface of the rear edge cavity is acquired by the forward lens.

[0044] S3. Perform image stitching on the video frame data of each shot to obtain a stitched image, which includes the stitched image of the leading cavity, the stitched image of the middle cavity, and the stitched image of the trailing cavity.

[0045] The stitching process for the leading edge cavity image is as follows: The video frame data from the windward and leeward sides of the leading edge cavity are stitched together. Figure 7 The yellow and blue parts are divided into two segments, with the front edge mold seam 14 as the boundary, to obtain the front edge windward side segmentation image and the front edge leeward side segmentation image. Figure 8 (a) and (b) are video frame data from the left and right sweep cameras of the segmented leading edge cavity, respectively. Figure 9(a) and (b) are video frame data from the left and right sweep cameras of the segmented leading edge cavity, respectively. The video frame data of the leading edge cavity's web surface are segmented using the web boundary line as the dividing line, resulting in a segmented image of the leading edge web. Finally, the segmented images of the leading edge windward side and the leading edge leeward side are stitched together, as shown in the image. Figure 10 As shown, the image is finally stitched together with the segmented image of the anterior ventral plate to obtain the stitched image of the anterior cavity.

[0046] Reference Figure 11 The stitching process of the trailing cavity image is similar to that of the leading cavity. The video frame data of the windward side and the leeward side of the trailing cavity (the yellow and blue parts in the figure) are segmented separately with the trailing mold seam 15 as the boundary.

[0047] Reference Figure 12 The stitching process for the intermediate cavity images is as follows: Video frame data from the windward and leeward sides of the intermediate cavity are segmented, with the mold closing seam as the boundary, to obtain segmented images of the windward and leeward sides of the intermediate cavity. The stitched effect of these two segmented images is shown below. Figure 13 As shown, the video frame data of the anterior and posterior web surfaces of the central cavity are segmented using their respective web boundaries as the dividing line, resulting in segmented images of the anterior and posterior web surfaces of the central cavity. These segmented images are then stitched together to obtain a stitched image of the central cavity.

[0048] During the scanning and stitching process, the image stitching of a single scanned surface still uses distortion correction algorithms, feature point extraction algorithms (such as SuperPoint), and image stitching algorithms (such as DeepPanorama) to stitch together the pre-segmented scanned surface images that overlap with each other.

[0049] S4. Fault detection is performed on the stitched image using a target detection method. This embodiment uses the YOLOv8 segmentation model, which can identify the outer contour of the fault and mark the target detection box. An example of fault identification is shown below. Figure 7 As shown, the area of ​​the faulty pixel can be obtained from the outer envelope point set obtained by the segmentation algorithm. S p Calculate the minimum outer envelope rectangle of the fault in the image, and then find the longer side of the minimum outer envelope rectangle to obtain the pixel length of the fault. l p The fault identification effect is as follows: Figure 14 As shown.

[0050] Specifically, when a fault is detected, the distance sensors of the left and right sweep cameras are used to calculate and output the fault location and area. (Refer to...) Figure 15 When calibrating the camera, the distance between the calibration object and the camera can be used as a reference.d s Size of the calibration object l s pixel distance from the calibration object l ps Obtain the pixel scale per unit distance r s The calculation formula is as follows:

[0051] .

[0052] The distance between the shooting surface and the camera can be obtained from the distance sensor on the sweep camera. d r The known algorithm calculates the length of the faulty pixel as follows: l p The actual length of the fault can be estimated. l and the actual area of ​​the fault S The calculation formula is as follows:

[0053] ,

[0054] .

[0055] The following table shows an example of the output diagnostic results:

[0056]

[0057] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (modules, systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0061] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0062] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A wind turbine blade inspection method based on a wall-climbing robot and image recognition, characterized in that, The method comprises: S1, deploying a wall-climbing robot inside the blade cavity, wherein the wall-climbing robot is provided with a forward camera, a left-sweep camera and a right-sweep camera; the wall-climbing robot climbs the wall in a negative pressure adsorption walking mode; S2, the wall-climbing robot works in the web area in the blade cavity, collects video frame data while climbing the wall on the web; the web area comprises three cavities, namely a leading edge cavity, a middle cavity and a trailing edge cavity; the scanning surface of the leading edge cavity comprises a leading edge cavity windward surface, a leading edge cavity leeward surface and a leading edge cavity web surface, wherein the video frame data of the leading edge cavity windward surface and the leading edge cavity leeward surface are collected by the left-sweep camera and the right-sweep camera, and the video frame data of the leading edge cavity web surface are collected by the forward camera; the scanning surface of the middle cavity comprises a middle cavity leading edge web surface, a middle cavity windward surface, a middle cavity leeward surface and a middle cavity trailing edge web surface, wherein the video frame data of the middle cavity leading edge web surface, the middle cavity windward surface and the middle cavity leeward surface are collected by the left-sweep camera and the right-sweep camera, and the video frame data of the middle cavity trailing edge web surface are collected by the forward camera; the scanning surface of the trailing edge cavity comprises a trailing edge cavity windward surface, a trailing edge cavity leeward surface and a trailing edge cavity web surface, wherein the video frame data of the trailing edge cavity windward surface and the trailing edge cavity leeward surface are collected by the left-sweep camera and the right-sweep camera, and the video frame data of the trailing edge cavity web surface are collected by the forward camera; S3, image stitching is performed on the video frame data of each camera to obtain a stitched image; the stitching process of the leading edge cavity stitched image is as follows: the video frame data of the leading edge cavity windward surface and the leading edge cavity leeward surface are respectively segmented with the mold joint as the boundary to obtain a leading edge windward surface segmented image and a leading edge leeward surface segmented image; the video frame data of the leading edge cavity web surface are segmented with the web boundary line as the boundary to obtain a leading edge web segmented image; the leading edge windward surface segmented image, the leading edge leeward surface segmented image and the leading edge web segmented image are spliced to obtain the leading edge cavity stitched image; the stitching process of the middle cavity stitched image is as follows: the video frame data of the middle cavity windward surface and the middle cavity leeward surface are respectively segmented with the mold joint as the boundary to obtain a middle cavity windward surface segmented image and a middle cavity leeward surface segmented image; the video frame data of the middle cavity leading edge web surface and the middle cavity trailing edge web surface are segmented with the respective web boundary lines as the boundary to obtain a middle cavity leading edge web segmented image and a middle cavity trailing edge web segmented image; the middle cavity windward surface segmented image, the middle cavity leeward surface segmented image, the middle cavity leading edge web segmented image and the middle cavity trailing edge web segmented image are spliced to obtain the middle cavity stitched image; S4, a target detection method is used to detect faults in the stitched image.

2. The wind turbine blade inspection method based on wall-climbing robot and image recognition according to claim 1, characterized in that: The forward camera in step S1 is calibrated before work.

3. The wind turbine blade inspection method based on wall-climbing robot and image recognition according to claim 1, characterized in that: The stitched image in step S3 comprises a leading edge cavity stitched image, a middle cavity stitched image and a trailing edge cavity stitched image.

4. The wall-climbing robot and image recognition based fan blade detection method according to claim 1, characterized in that: When a fault is detected in the fault detection process in step S4, the distance sensors of the left-sweep camera and the right-sweep camera are used to calculate the fault position and area and output.

Images