Processing method for unmanned aerial vehicle to collect fan blade image

Abstract

This invention provides a processing method for collecting wind turbine blade images using a drone. The method involves data processing, including semantic segmentation of each frame in a blade image sequence to obtain a blade mask; determining coarse registration transformation parameters between adjacent frames based on the blade mask; aligning the adjacent frames based on the coarse registration transformation parameters to obtain a first segmentation region; enhancing the contrast of the image region corresponding to the blade mask in each first segmentation region to obtain a first enhanced image; extracting feature points from the image region of the first enhanced image and performing feature matching; determining fine registration transformation parameters based on the linkage of coarse and fine matching; and correcting failed matching parts in the fine matching through sequence consistency detection, interpolation repair, and contour secondary correction; and projecting and fusing multiple frames in the blade image sequence according to the fine registration transformation parameters to generate a panoramic image of the blade.

Description

Technical Field

[0001] This invention relates to data processing technology, and more particularly to a processing method for acquiring images of wind turbine blades using a drone. Background Technology

[0002] With the rapid development of the wind power industry, regular maintenance and inspection of wind turbine blades are crucial for ensuring the safe operation of the units. Currently, using drones equipped with high-definition cameras for close-range blade inspection has become the mainstream method. Due to the enormous size of wind turbine blades, drones typically need to acquire multiple frames of high-resolution local images along the blade's length. To achieve complete observation and precise location of surface defects such as cracks, corrosion, and lightning strike points, these sequential local views must be stitched together into a complete panoramic image of the blade.

[0003] However, most existing image stitching techniques rely directly on feature point matching, such as SIFT and SURF algorithms combined with RANSAC filtering to calculate the transformation relationship between images. In the wind turbine blade scenario, this single-dimensional matching strategy faces severe challenges. The blade surface is usually covered with a uniform coating, resulting in extremely sparse texture features. This low-texture and repetitive texture characteristic makes it difficult for traditional algorithms to extract enough and evenly distributed effective feature points, easily leading to a large number of mismatches in the initial matching stage. This can cause the algorithm to get stuck in a local optimum, resulting in stitching breaks or misalignments. Especially when the drone has attitude jitter or large parallax, direct registration without global geometric constraints often cannot guarantee the spatial order correctness of the image sequence.

[0004] Therefore, how to utilize the geometric features of the leaf mask to perform two matching operations, thereby ensuring splicing accuracy while improving the robustness and convergence of the algorithm, has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a processing method for collecting wind turbine blade images by UAVs. It can perform two matching operations using the geometric features of the blade mask, thereby improving the robustness and convergence of the algorithm while ensuring stitching accuracy.

[0006] A first aspect of the present invention provides a processing method for acquiring images of wind turbine blades using a drone, comprising:

[0007] Semantic segmentation is performed on each frame of the leaf image sequence to obtain the leaf mask corresponding to each frame of the image;

[0008] Based on the leaf mask, coarse registration transformation parameters between adjacent frame images are determined, and the adjacent frame images are aligned based on the coarse registration transformation parameters to obtain the first segmentation region;

[0009] Contrast enhancement is performed on the image regions corresponding to the leaf masks in each first segmentation region to obtain a first enhanced image;

[0010] Feature points are extracted from the image region of the first enhanced image, and feature matching is performed. Based on the feature matching results, fine registration transformation parameters between the adjacent frame images are determined.

[0011] The multiple frames of images in the leaf image sequence are projected and fused according to the fine registration transformation parameters to generate a panoramic image of the leaf.

[0012] Optionally, in one possible implementation of the first aspect, determining the coarse registration transformation parameters between the adjacent frame images based on the leaf mask includes:

[0013] Based on the blade mask, determine the blade region in the adjacent frame image, and calculate the width information and overlapping region information of the blade region;

[0014] Based on the width information and the overlapping region information, the scaling parameters and translation parameters between the adjacent frame images are calculated, and the scaling parameters and translation parameters are used as the coarse registration transformation parameters.

[0015] Optionally, in one possible implementation of the first aspect, determining the fine registration transformation parameters between the adjacent frame images based on the feature matching result includes:

[0016] Based on the feature matching results, obtain the initial feature matching point pairs;

[0017] The erroneous matching points in the initial feature matching point pairs are removed based on the random sampling consistency algorithm, and the affine transformation matrix between adjacent frame images is calculated based on the selected correct matching points. The affine transformation matrix is ​​then determined as the fine registration transformation parameter.

[0018] Optionally, in one possible implementation of the first aspect, the step of projecting and fusing multiple frames of images in the leaf image sequence according to the fine registration transformation parameters to generate a panoramic image of the leaf includes:

[0019] The images in the leaf image sequence are projected onto the global canvas according to the fine registration transformation parameters to obtain the projected image.

[0020] The brightness of the overlapping area in the projected image is adjusted to obtain a brightness-adjusted projected image;

[0021] The brightness-adjusted projected image is fused to generate the panoramic image of the blade.

[0022] Optionally, in one possible implementation of the first aspect, adjusting the brightness of the overlapping region in the projected image to obtain a brightness-adjusted projected image includes:

[0023] Calculate the brightness distribution information of the overlapping region;

[0024] A brightness adjustment map is generated based on the brightness distribution information;

[0025] The brightness of the overlapping region is adjusted according to the brightness adjustment map to obtain a brightness-adjusted projected image.

[0026] Optionally, in one possible implementation of the first aspect, it also includes:

[0027] When an abnormal area is identified in the color ring of the wind turbine blade in the blade panoramic image, a first wind turbine model is constructed based on the corresponding blade panoramic image.

[0028] The blade model with the abnormal area in the first wind turbine model is regarded as the abnormal blade;

[0029] Based on the location of the abnormal area in the abnormal blade and the blade position, the spraying parameters of the spraying drone are determined;

[0030] Based on the spraying parameters, the spraying drone is controlled to spray the abnormal areas at the color ring.

[0031] Optionally, in one possible implementation of the first aspect, determining the spraying parameters of the spraying drone based on the location of the abnormal area in the abnormal blade and the blade position includes:

[0032] The first wind turbine model is processed into coordinates to obtain a dividing plane, and the blade position of the abnormal blade is determined based on the dividing plane;

[0033] When the blade position is determined to be at the first spraying position, the corresponding abnormal blade is taken as the first blade, and the first spraying parameters of the spraying drone are determined according to the location of the abnormal area in the first blade.

[0034] When it is determined that the blade position is located at the second spraying position, the corresponding abnormal blade is taken as the second blade, and the second spraying parameters of the spraying drone are determined according to the location of the abnormal area in the second blade.

[0035] Optionally, in one possible implementation of the first aspect, the coordinate processing of the first wind turbine model to obtain a dividing plane, and the determination of the blade position of the abnormal blade based on the dividing plane, includes:

[0036] Using the center point of the fairing in the first wind turbine model as the origin of the coordinate system, a vertical axis perpendicular to the ground is constructed to obtain the first coordinate system;

[0037] The plane containing the horizontal and vertical axes in the first coordinate system is used as the dividing plane. Abnormal blades located above the dividing plane are used as the first spraying position, and abnormal blades located below the dividing plane are used as the second spraying position.

[0038] The location of the abnormal blade is obtained based on the first spraying location and the second spraying location.

[0039] Optionally, in one possible implementation of the first aspect, determining the first spraying parameters of the spraying drone based on the regional location of the abnormal area in the first blade includes:

[0040] Connect the midpoint of the tip of the first blade and the midpoint of the root of the first blade to obtain the first direction line. The direction from the tip of the blade to the root of the first blade is taken as the first spraying direction of the first blade.

[0041] The spraying width of the abnormal area is determined based on the first spraying direction, and the first included angle between the first blade and the dividing plane is obtained;

[0042] When the location of the abnormal area is determined to be on the upper surface of the first blade, the abnormal area is taken as the first upper abnormal area, the preset upper angle interval in the upper angle comparison table is determined as the first upper angle interval, and the preset number of times corresponding to the first upper angle interval is retrieved as the number of times the upper surface of the first upper abnormal area is sprayed.

[0043] When it is determined that the location of the abnormal area is on the lower surface of the first blade, the abnormal area is taken as the first lower abnormal area, the preset lower angle interval in the lower angle comparison table is determined as the first lower angle interval, and the preset number of superpositions corresponding to the first lower angle interval is retrieved as the number of spray superpositions for the first lower abnormal area.

[0044] The number of times the upper surface is sprayed and the number of times the spraying is superimposed are used to obtain the number of times the lower surface of the first lower abnormal zone is sprayed.

[0045] The first spraying parameters of the spraying drone are obtained based on the first spraying direction, spraying width, number of spraying times on the upper surface and number of spraying times on the lower surface.

[0046] Optionally, in one possible implementation of the first aspect, determining the second spraying parameters of the spraying drone based on the regional location of the abnormal area in the second blade includes:

[0047] Connect the midpoint of the tip of the second blade to the midpoint of the root of the blade to obtain the second direction line. The direction from the tip of the blade to the root of the blade along the second direction line is the second spraying direction of the first blade.

[0048] The spraying width of the abnormal area is determined based on the second spraying direction, and the second included angle between the second blade and the dividing plane is obtained;

[0049] When the location of the abnormal area is determined to be on the upper surface of the second blade, the abnormal area is taken as the second upper abnormal area. The preset upper angle interval in the upper angle comparison table is determined as the second upper angle interval. The preset number of times corresponding to the second upper angle interval is retrieved is taken as the number of times the upper surface of the second upper abnormal area is sprayed.

[0050] When it is determined that the abnormal area is located on the lower surface of the second blade, the abnormal area is taken as the second lower abnormal area. The preset lower angle interval in the lower angle comparison table is determined as the second lower angle interval. The preset number of superpositions corresponding to the second lower angle interval is retrieved as the number of spray superpositions for the second lower abnormal area.

[0051] The number of times the upper surface is sprayed and the number of times the spraying is superimposed are used to obtain the number of times the lower surface of the second lower abnormal zone is sprayed.

[0052] The second spraying parameters of the spraying drone are obtained based on the second spraying direction, spraying width, number of spraying times on the upper surface, and number of spraying times on the lower surface.

[0053] A second aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program, the computer program being stored in the memory, and the processor executing the computer program to perform the methods described in the first aspect of the present invention and various possible methods related to the first aspect.

[0054] A third aspect of the present invention provides a storage medium storing a computer program, which, when executed by a processor, is used to implement the first aspect of the present invention and various methods possibly involved in the first aspect.

[0055] The beneficial effects of this invention are as follows:

[0056] 1. This invention separates the leaf from a complex background by obtaining a mask through semantic segmentation of the leaf image. The method first uses the width and overlap information of the leaf mask for coarse registration, placing the image in a roughly correct position. Then, contrast enhancement is applied to the overlapping areas after coarse registration, revealing more clearly visible texture details. Feature points are then extracted from these details for fine matching. This coarse-to-fine approach, enhancing texture before matching, avoids the problem of failing to extract effective matching points due to the smoothness of the leaf surface, thus improving the success rate of stitching.

[0057] 2. This invention takes into account both the location of the blade and the location of the abnormal area, and customizes the spraying treatment for each wind turbine blade. Even for the same abnormal area on the same blade, the spraying parameters will be different because the stopping position of the wind turbine blade is different. We will customize the spraying treatment according to the actual stopping position and the location of the damage on the wind turbine blade, thereby avoiding dripping and paint slippage. Attached Figure Description

[0058] Figure 1 This is a flowchart of the processing method for acquiring wind turbine blade images by a drone, provided by the present invention.

[0059] Figure 2 This is a schematic diagram illustrating a spraying drone performing spraying according to the present invention;

[0060] Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided by the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein.

[0063] It should be understood that in the various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0064] It should be understood that in this invention, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0065] It should be understood that in this invention, "multiple" refers to two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Contains A, B, and C", "Contains A, B, and C" means that all three A, B, and C are contained; "Contains A, B, or C" means that one of A, B, and C is contained; "Contains A, B, and / or C" means that any one, two, or three of A, B, and C are contained.

[0066] It should be understood that in this invention, "B corresponding to A", "B corresponding to A", "A and B correspond", or "B and A correspond" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Matching A and B is defined as a similarity between A and B that is greater than or equal to a preset threshold.

[0067] Depending on the context, "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection."

[0068] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0069] This invention provides a processing method for acquiring images of wind turbine blades using a drone, such as... Figure 1 As shown, steps S1-S5 are included:

[0070] S1, perform semantic segmentation on each frame of the leaf image sequence to obtain the leaf mask corresponding to each frame of the image.

[0071] It's important to note that when a drone photographs leaves from the air, the lens inevitably captures the sky, clouds, ground towers, or distant scenery. When stitching the photos, if the entire image is used directly to calculate the position, the computer can easily mistake these irrelevant background elements for alignment, resulting in misaligned or non-existent leaf stitching. This step involves first isolating the leaves from the complex background, allowing subsequent processing to eliminate background interference.

[0072] It is worth mentioning that the blade image sequence refers to a set of photos taken by a drone around the wind turbine blades in sequence. Semantic segmentation refers to an image processing technology that can classify and label every pixel in the photo. This is an existing technology and will not be elaborated here. The blade mask refers to a black and white shape image generated after segmentation. On this image, the area of ​​the blade is marked with one color, such as white, while the background, such as the sky, is marked with another color, such as black, which can be removed later.

[0073] Understandably, we will read the sequence of photos taken by the drone one by one. The device runs a dedicated recognition program, such as DeepLabv3 or UNet algorithms, which scans each photo and identifies which pixels belong to the leaf and which belong to the background. After processing, the device generates a corresponding black-and-white mask image for each original photo, clearly outlining the specific contours and positions of the leaf within the photo.

[0074] S2, Based on the leaf mask, determine the coarse registration transformation parameters between adjacent frame images, and perform alignment processing on the adjacent frame images based on the coarse registration transformation parameters to obtain the first segmentation region.

[0075] It's important to note that wind turbine blades often have very smooth surfaces with almost no texture or detail. Attempting to align them by finding tiny feature points from the outset is likely to fail due to missing or incorrect points. However, the overall blade outline is very clear. Therefore, we initially disregarded details and used the blade outline shape extracted by S1 to calculate the approximate movement and scaling required to align the image. This ensured a correct overall direction for subsequent fine-tuning, preventing incorrect placement from the outset.

[0076] Among them, the coarse registration transformation parameters refer to a set of basic mathematical values, mainly including the scaling ratio and translation distance. They are not calculated from image details, but rather derived from the geometry of the leaf mask. They are used to describe the approximate positional relationship between the two photos. The first segmentation region refers to the image that has been roughly superimposed on the previous photo after the photo has been moved and scaled according to the above parameters.

[0077] Understandably, after the server receives the leaf masks from two adjacent photos, it first analyzes the shape relationship between the two masks. By calculating the geometric features of the masks, the device estimates approximately how far the drone moved and how much it zoomed in or out when taking these two photos. The device uses these estimated values ​​as coarse registration transformation parameters, directly applying them to the subsequent photo, moving and scaling it to a position that is closer to the previous photo, thus generating the first segmentation region.

[0078] In some embodiments, step S2 (determining the coarse registration transformation parameters between adjacent frame images based on the leaf mask) includes:

[0079] Based on the blade mask, determine the blade region in the adjacent frame image, and calculate the width information and overlapping region information of the blade region;

[0080] Based on the width information and the overlapping region information, the scaling parameters and translation parameters between the adjacent frame images are calculated, and the scaling parameters and translation parameters are used as the coarse registration transformation parameters.

[0081] Among them, the leaf region refers to the part of the mask image marked in white that represents the leaf entity, the width information refers to the pixel width of the leaf cross-section measured in the mask image, the overlapping area information refers to the area or shape features of the overlapping part after two adjacent mask images are superimposed, the scaling parameter refers to the factor used to adjust the image size, and the translation parameter refers to the pixel distance used to move the image up, down, left, and right.

[0082] Understandably, the server compares two masked images. First, the device measures the width of the blades in both images. If the blades are wider in the latter image, it means the drone is closer, so the device calculates a scaling parameter to shrink the image slightly to match the former. Simultaneously, the device calculates the overlapping portion of the two images. Based on the misalignment of the overlapping area, it calculates how many pixels the image needs to be moved in which direction—this is the translation parameter. Finally, the device combines these two parameters as the basis for guiding coarse registration.

[0083] S3, perform contrast enhancement on the image regions corresponding to the leaf masks in each first segmentation region to obtain the first enhanced image.

[0084] It's worth noting that wind turbine blades are typically made with a very smooth surface and a uniform coating color to reduce air resistance, much like a sheet of white paper. On such a smooth surface, it's difficult for a computer to find unique markings like spots or scratches for subsequent fine alignment. This step involves adding a filter to the blade image. By specifically increasing the contrast between light and dark areas in the blade region, tiny textures, paint particles, or minor dirt spots that are almost invisible to the naked eye are brought to the surface, artificially creating more detail so that the subsequent process can capture these features and stitch the images together accurately.

[0085] The first segmentation region is the region image after alignment processing. Contrast enhancement refers to an image processing technique, such as the CLAHE algorithm, which is an existing technology and will not be elaborated here. It can make blurry details in the image clear and sharp. The first enhanced image refers to the new photo with richer surface texture details generated after this sharpening process.

[0086] Understandably, we first use a set of default settings to sharpen the area. After processing, the server checks and attempts to extract and count feature points from the image. If the number of extracted feature points is less than a pre-set threshold, the enhancement is deemed insufficient. Then, the device automatically modifies the enhancement parameters, such as increasing the contrast limit, and re-enhances the area of ​​the original image using the new parameters. This process may repeat several times until a sufficient number of feature points are extracted, at which point the device outputs the final, satisfactory image as the first enhanced image.

[0087] S4, extract feature points from the image region of the first enhanced image and perform feature matching, and determine the fine registration transformation parameters between the adjacent frame images based on the feature matching results.

[0088] It's important to note that the previous coarse registration only roughly positioned the photos based on the leaf's outline, but it wasn't a perfect match. If done directly, misalignment or ghosting might occur at the seams. After the enhancement process in the previous step, many previously invisible texture details have become apparent in the photos. This step utilizes these details, allowing the computer to find identical tiny marks in both photos. By aligning these marks, a very precise adjustment value is calculated, enabling fine-tuning of the image.

[0089] The first enhanced image is a photo with enhanced contrast and clearer texture; feature points are special pixels that are easy to identify, such as stains, scratches, texture transitions, etc. that can be recognized by the computer; feature matching refers to the process of finding the feature point that represents the same physical location in two photos; fine registration transformation parameters refer to a set of precise mathematical values ​​used to guide the photo to make tiny movements, rotations, or stretches to achieve pixel-level alignment.

[0090] Understandably, the server scans two adjacent enhanced images. It captures hundreds or thousands of unique feature points within the leaf-like regions of the images. Next, the device uses an existing matching program like LightGlue to compare the points in the previous image with those in the next, identifying pairs of identical points. Finally, based on the positional differences of these point pairs, the server calculates how much fine-tuning is needed in the next image to perfectly overlap with the previous one, and determines this fine-tuning value as the fine-registration transform parameter.

[0091] Subsequently, the parts that failed to match in the fine matching can be corrected by combining sequence consistency detection with interpolation repair and contour secondary correction.

[0092] In some embodiments, step S4 (determining the fine registration transformation parameters between adjacent frame images based on feature matching results) includes:

[0093] Based on the feature matching results, obtain the initial feature matching point pairs;

[0094] The erroneous matching points in the initial feature matching point pairs are removed based on the random sampling consistency algorithm, and the affine transformation matrix between adjacent frame images is calculated based on the selected correct matching points. The affine transformation matrix is ​​then determined as the fine registration transformation parameter.

[0095] It's important to note that computers can make mistakes when automatically matching point pairs. For example, two seemingly similar dirt spots on a blade might actually be in different locations. If the computer mistakenly identifies them as the same spot and attempts to align them, the entire image will be distorted, resulting in a crooked composite. Therefore, we discard point pairs that are incorrectly matched due to misjudgment, retaining only the truly correct point pairs for position calculation.

[0096] The initial feature matching point pairs refer to all point pairs found by the server, which include both correct and incorrect ones.

[0097] Understandably, after receiving the initial large set of point pairs, the server won't immediately accept everything. It will activate the RANSAC algorithm (Random Sample Consensus), which, like a sampling check, first randomly selects a few point pairs and calculates a temporary transformation rule. Then, it applies this rule to other point pairs to see how many pairs conform to it. Any point pairs that don't conform to the majority rule are judged as mismatches and discarded. After multiple rounds of this filtering, the remaining points are all correct matches. Finally, the device uses these correct points to calculate a final affine transformation matrix. This matrix can refine the fine registration transformation parameters of the image stitching. That is, using the coarse registration matrix as the initial value / prior, RANSAC is used to repeatedly sample on the matching points to estimate and filter inliers. Then, the transformation parameters are refined using the inliers through local nonlinear least squares refinement, iterating multiple times until the set of inliers and the transformation converge.

[0098] S5, Project and fuse multiple frames of images in the leaf image sequence according to the fine registration transformation parameters to generate a panoramic image of the leaf.

[0099] It's important to note that while the computer calculated how each photo should be moved and rotated for precise alignment in the previous steps, these photos were still independent files in the computer's memory. Therefore, this step uses the precisely calculated values ​​to transfer the scattered partial photos one by one to a unified large canvas, and removes the seams caused by differences in lighting between the photos, ultimately synthesizing a complete panoramic image of the leaf.

[0100] Understandably, the server will execute a fusion process, processing the overlapping pixels and smoothly blending the textures and colors of multiple images together to ultimately output a seamless, connected full-page image of the leaf.

[0101] In some embodiments, step S5 (projecting and fusing multiple frames of images in the leaf image sequence according to the fine registration transformation parameters to generate a panoramic image of the leaf) includes:

[0102] The images in the leaf image sequence are projected onto the global canvas according to the fine registration transformation parameters to obtain the projected image.

[0103] It's important to note that during drone shooting, the angle of sunlight may change, or clouds may drift by, causing adjacent photos to differ in brightness. Simply stacking them together would result in a noticeable bright line at the seam, affecting viewing and subsequent defect identification. Therefore, we used projection to orthogonalize the photos, analyzed the brightness difference between the two images, and used mathematical techniques to brighten the darker areas or darken the brighter areas, creating a natural transition before finally merging them into a single image.

[0104] The global canvas is a virtual planar coordinate system established in storage that is large enough to hold all the photos;

[0105] In some embodiments, the step of (adjusting the brightness of the overlapping region in the projected image to obtain a brightness-adjusted projected image) includes:

[0106] Calculate the brightness distribution information of the overlapping region;

[0107] A brightness adjustment map is generated based on the brightness distribution information;

[0108] The brightness of the overlapping region is adjusted according to the brightness adjustment map to obtain a brightness-adjusted projected image.

[0109] It's important to note that in areas where photos overlap, the intensity of light is often uneven; the left side might be brighter than the right, or there might be a shadow in the middle. Simply increasing or decreasing the brightness across the entire area won't eliminate the patchwork effect.

[0110] Therefore, we scan each pixel in the overlapping area, comparing the brightness difference at the same location from two different photos. The device then calculates an instruction map the same size as the overlapping area based on this difference. Each point in this map records a correction value; for example, indicating that the top-left pixel is too dark and needs to be brightened, while the bottom-right pixel is too bright and needs to be darkened. Finally, the server performs brightness adjustments on each pixel in the overlapping area, thus smoothing out the previously uneven brightness.

[0111] Specifically, the two images of the overlapping area are first converted from RGB color mode to HSV or Lab mode. This is because we only want to adjust the brightness, not change the leaf color. Then, the brightness values ​​of the overlapping area of ​​the two images are compared point by point. The resulting image is very coarse, containing both differences in light intensity and details such as stains and scratches on the leaves. To preserve only the changes in light intensity without destroying the leaf texture, Gaussian blur or mean filtering is used to strongly blur the coarse image. After blurring, the specific stain details are no longer visible, leaving only a smooth, gradual trend of light and dark, which yields the brightness distribution information. This is existing technology and will not be elaborated upon here. Furthermore, after obtaining the brightness distribution information, the image is essentially a matrix, with each point storing a value. For example, a value of +10 means that the brightness of the pixel at that location needs to be increased by 10; if it is 0.8, it means that the brightness here should be 80% of the original, which is a brightness adjustment mapping. Subsequently, there should be no sudden changes at the boundary between the overlapping and non-overlapping areas. By using feathering techniques or linear interpolation, the values ​​at the edges of this mapping image are gradually transitioned to 0 to prevent a noticeable seam from appearing after stitching. Specifically, by calculating the difference or ratio of pixel brightness in the overlapping area of ​​the two images, and performing low-pass filtering, such as Gaussian blurring, on the difference image to filter out high-frequency texture details, brightness distribution information reflecting low-frequency illumination changes is obtained. This smoothed distribution information is then used to construct a pixel-level gain mapping image for point-by-point exposure compensation of the image.

[0112] The brightness of the overlapping area in the projected image is adjusted to obtain a brightness-adjusted projected image;

[0113] The brightness-adjusted projected image is fused to generate the panoramic image of the blade.

[0114] Among them, brightness adjustment can be a local brightness mapping technique, which is a process of correcting exposure differences; the projected image after brightness adjustment refers to a photograph that has undergone uniform illumination processing and subsequent merging.

[0115] Understandably, the server first uses an affine transformation matrix to map all photos onto a global canvas. Next, the device analyzes each overlapping region, comparing the pixel brightness of pixels from two different photos within the overlapping area. It calculates a brightness difference distribution map and uses this map to point-by-point correct the brightness of the photos, ensuring that the brightness of the two images is completely consistent at the overlapping edges. Finally, the device uses algorithms such as multi-band fusion to compress the corrected multi-layered images into a single layer, ensuring natural texture transitions and generating the final panoramic image of the leaf.

[0116] Based on the above embodiments, A1-A4 are also included:

[0117] A1, when an abnormal area is identified in the color ring of the wind turbine blade in the panoramic view of the blade, a first wind turbine model is constructed based on the corresponding panoramic view of the blade.

[0118] It should be noted that the color rings on the wind turbine blades serve as reminders and warnings, alerting aircraft and birds to take evasive action. They are usually used as markings or specific coating areas on the blades, and if any abnormalities occur, such as peeling off, maintenance is required.

[0119] Among them, the color ring refers to a specific color ring area on the wind turbine blade used for identification, such as red or yellow, and the abnormal area can be the area where the color has been detected to have fallen off within the color ring area.

[0120] Understandably, pixel value analysis of the color ring area on the blades reveals anomalies, triggering the modeling process. Using a panoramic image of the blades and standard structural data of the wind turbine, a first wind turbine model is constructed. A 3D model is then built based on images captured by a drone (this is existing technology and will not be elaborated upon here). In this process, the anomaly will also be reflected in the color ring area of ​​the first wind turbine model.

[0121] A2, the blade model with the abnormal area in the first wind turbine model is regarded as the abnormal blade.

[0122] The blade model is the wind turbine blade in the first wind turbine model.

[0123] A3. Based on the location of the abnormal area in the abnormal blade and the position of the blade, determine the spraying parameters of the spraying drone.

[0124] It should be noted that current technologies using drones for spray painting repair typically rely solely on GPS coordinates to guide the drone to the target point and employ a fixed set of parameters. This ignores the decisive influence of gravity and blade spatial attitude on paint adhesion quality. Due to the randomness of wind turbine shutdown locations, blades may be in different orientations, such as vertical, horizontal, or tilted. Defects may be located on the upper surface of the blade where gravity assists adhesion, or on the lower surface where gravity hinders dripping. If the fixed parameters of existing technologies are used, excessive paint thickness can easily lead to sagging on the lower surface or in areas with large tilt angles, or insufficient adhesion can cause dripping.

[0125] In some embodiments, step A3 (determining the spraying parameters of the spraying drone based on the location of the abnormal area in the abnormal blade and the blade position) includes A31-A33:

[0126] A31, the first wind turbine model is processed into coordinates to obtain a dividing plane, and the blade position of the abnormal blade is determined based on the dividing plane.

[0127] In some embodiments, step A31 (coordinate processing of the first wind turbine model to obtain a dividing plane, and determining the blade position of the abnormal blade based on the dividing plane) includes A311-A313:

[0128] A311, using the center point of the fairing in the first wind turbine model as the origin of the coordinate system, construct a vertical axis perpendicular to the ground to obtain the first coordinate system.

[0129] Understandably, in the first wind turbine model, geometric feature recognition can be performed by extracting the center of the fairing as the origin of the coordinate system, constructing a spatial coordinate system perpendicular to the ground, so that the plane corresponding to xy is parallel to the ground, thus obtaining the first coordinate system.

[0130] A312, obtain the plane where the horizontal axis and vertical axis are located in the first coordinate system as the dividing plane, take the abnormal blade located above the dividing plane as the first spraying position, and take the abnormal blade located below the dividing plane as the second spraying position.

[0131] It should be noted that the effect of gravity on the coating is drastically different when the blade is facing upwards and downwards. When the blade is upwards, the fluid tends to flow along the axial hub of the blade, that is, from the blade tip to the blade root; when the blade is downwards, the fluid tends to accumulate towards the blade tip, flowing from the blade root to the blade tip.

[0132] Therefore, we take the plane containing the horizontal and vertical axes in the first coordinate system as the dividing plane, take the abnormal blades located above the dividing plane as the first spraying position (i.e., above), and take the abnormal blades located below the dividing plane as the second spraying position (i.e., below).

[0133] A313, based on the first spraying position and the second spraying position, the blade position of the abnormal blade is obtained.

[0134] A32, when the blade position is determined to be at the first spraying position, the corresponding abnormal blade is taken as the first blade, and the first spraying parameters of the spraying drone are determined according to the regional position of the abnormal area in the first blade.

[0135] In some embodiments, step A32 (determining the first spraying parameters of the spraying drone based on the regional location of the abnormal area in the first blade) includes A321-A326:

[0136] A321, connect the midpoint of the tip of the first blade and the midpoint of the root of the first blade to obtain the first direction line, and take the direction from the tip of the blade to the root of the first blade as the first spraying direction of the first blade.

[0137] It should be noted that existing technologies use fixed parameters and preset travel paths to spray the abnormal areas, without taking into account the actual location of the wind turbine blades and the location of the abnormal areas to carry out targeted spraying treatments for the corresponding locations.

[0138] Among them, the blade tip refers to the end of the blade that is furthest from the hub; the blade root refers to the root where the blade connects to the hub.

[0139] Therefore, we will connect the midpoint of the tip of the first blade and the midpoint of the root of the first blade to obtain a first direction line, and take the direction from the tip of the blade to the root of the first blade as the first spraying direction of the first blade.

[0140] A322, determine the spray width of the abnormal area based on the first spraying direction, and obtain the first included angle between the first blade and the dividing plane.

[0141] Understandably, the server will analyze the width change of the abnormal area in the first spraying direction, that is, from the tip of the blade to the root of the blade, to obtain the spraying width. Subsequently, the abnormal area will be sprayed with the change of the spraying width. Alternatively, the maximum width can be selected as the spraying width for overall coverage spraying. This is existing technology and will not be elaborated here. The server will also obtain the first angle between the first blade and the dividing plane.

[0142] A323, when the location of the abnormal area is determined to be on the upper surface of the first blade, the abnormal area is taken as the first upper abnormal area, the preset upper angle interval in the upper angle comparison table is determined as the first upper angle interval, and the preset number of times corresponding to the first upper angle interval is retrieved as the number of times the upper surface of the first upper abnormal area is sprayed.

[0143] It's important to note that when the abnormal area is located on the upper surface of the blade, although gravity helps the paint adhere to the surface, the larger the blade's tilt angle (the first included angle), the steeper the slope, and the faster the paint slides down. A single spray is unlikely to achieve sufficient thickness and is prone to runoff. Conversely, the smaller the angle and the gentler the slope, the easier it is for the paint to accumulate. Therefore, when the angle is large, the number of sprays should be increased to achieve a smaller, more frequent application, preventing paint runoff. For example, the number of sprays for each angle range can be determined based on a preset spray volume, thus determining the amount to be sprayed each time. Knowing the number of sprays is sufficient.

[0144] The upper surface can be the surface facing the sky, and the upper angle comparison table refers to a relational table pre-stored in the database. It has a one-to-one correspondence between preset upper angle intervals and preset number of times, reflecting the positive correlation between the tilt angle and the recommended number of spraying times.

[0145] Understandably, when the abnormal area is located on the upper surface of the first blade, this area is affected by the first angle. The larger the angle, the greater the influence of gravity, and it is easy to slip off. Therefore, multiple small sprays are required. Thus, we will use different spraying times for different tilt angles for customized treatment.

[0146] It is worth mentioning that when the fan blades are at a right angle, there is no need to consider the upper and lower surfaces. At this time, the two surfaces are the same, so the corresponding number of spraying times can be directly retrieved from the preset upper angle range.

[0147] A324, when it is determined that the location of the abnormal area is on the lower surface of the first blade, the abnormal area is taken as the first lower abnormal area, the preset lower angle interval in the lower angle comparison table is determined as the first lower angle interval, and the preset number of superpositions corresponding to the first lower angle interval is retrieved as the number of spray superpositions for the first lower abnormal area.

[0148] It should be noted that the lower surface, the side facing the ground, presents a more challenging mode for spraying. Here, gravity not only generates a downward tangential force causing sagging, but also a vertically downward normal force attempting to drip paint off the blade surface. Compared to the upper surface, paint adhesion on the lower surface is significantly more difficult, resulting in a higher loss rate; in other words, in addition to the difficulties of spraying on the upper surface, there is also the risk of dripping.

[0149] The lower surface can be the surface facing the ground. The lower angle comparison table refers to a relation table stored in the database in advance, which has a one-to-one correspondence between the preset lower angle range and the preset number of superpositions. It is easy to understand that when spraying the abnormal area on the lower surface of the same wind turbine blade, not only will the same paint slippage as on the upper surface occur, but dripping will also occur, which may drip onto other wind turbine blades. The smaller the angle, the greater the risk of dripping.

[0150] Therefore, we will retrieve the corresponding number of spray coating layers from the lower angle comparison table based on the first included angle.

[0151] A325, based on the sum of the number of times the upper surface is sprayed and the number of times the spraying is superimposed, the number of times the lower surface of the first lower abnormal zone is obtained.

[0152] It is not difficult to understand that the spraying strategy for the lower surface is essentially a combination of basic anti-sagging and additional anti-dripping. The number of spraying times for the lower surface in the first lower anomalous zone is obtained based on the sum of the number of surface spraying times and the number of spraying superposition times.

[0153] A326, based on the first spraying direction, spraying width, number of spraying times on the upper surface and number of spraying times on the lower surface, the first spraying parameters of the spraying drone are obtained.

[0154] It is easy to understand that the subsequent spraying drone will spray a preset amount of paint on the abnormal area above the upper surface in the first spraying direction, and spray a preset amount of paint on the abnormal area below the lower surface in the same spraying direction.

[0155] A33, when it is determined that the blade position is located at the second spraying position, the corresponding abnormal blade is taken as the second blade, and the second spraying parameters of the spraying drone are determined according to the location of the abnormal area in the second blade.

[0156] In some embodiments, step A33 (determining the second spraying parameters of the spraying drone based on the location of the abnormal area in the second blade) includes:

[0157] A331, connect the midpoint of the tip of the second blade and the midpoint of the root of the blade to obtain the second direction line, and use the direction from the tip of the blade to the root of the blade along the second direction line as the second spraying direction of the first blade.

[0158] It's easy to understand that the second blade is sprayed in a different direction than the first blade. The first blade is sprayed from the tip to the root, while the second blade is sprayed from the root to the tip. Subsequent treatments are carried out in the second spraying direction. For the upper surface, the principle is the same as A323. The larger the angle, the greater the risk of paint slipping in the second spraying direction. The principle is the same for the lower surface. On top of the upper surface, there is a risk of dripping. Therefore, by layering the spraying times, increasing the number of spraying times and reducing the amount of paint sprayed, multiple small-volume customized sprayings can be achieved.

[0159] A332, determine the spray width of the abnormal area based on the second spraying direction, and obtain the second included angle between the second blade and the dividing plane.

[0160] A333, when the location of the abnormal area is determined to be on the upper surface of the second blade, the abnormal area is taken as the second upper abnormal area, the preset upper angle interval in the upper angle comparison table is determined as the second upper angle interval, and the preset number of times corresponding to the second upper angle interval is retrieved as the number of times the upper surface of the second upper abnormal area is sprayed.

[0161] A334, when it is determined that the location of the abnormal area is on the lower surface of the second blade, the abnormal area is taken as the second lower abnormal area, the preset lower angle interval in the lower angle comparison table is determined as the second lower angle interval, and the preset number of superpositions corresponding to the second lower angle interval is retrieved as the number of spray superpositions for the second lower abnormal area.

[0162] A335, based on the sum of the number of times the upper surface is sprayed and the number of times the spraying is superimposed, the number of times the lower surface of the second lower abnormal zone is obtained.

[0163] A336, based on the second spraying direction, spraying width, number of spraying times on the upper surface and number of spraying times on the lower surface, obtains the second spraying parameters of the spraying drone.

[0164] A4, based on the spraying parameters, control the spraying drone to spray the abnormal area at the color ring.

[0165] See Figure 2 The spraying drone is controlled to spray abnormal areas of the color ring on the blades.

[0166] See Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. The electronic device 30 includes: a processor 31, a memory 32, and a computer program; wherein...

[0167] The memory 32 is used to store the computer program, and the memory may also be flash memory. The computer program is, for example, an application program or functional module that implements the above method.

[0168] Processor 31 is configured to execute the computer program stored in the memory to implement the various steps performed by the device in the above method. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0169] Alternatively, the memory 32 can be either standalone or integrated with the processor 31.

[0170] When the memory 32 is a device independent of the processor 31, the device may further include:

[0171] Bus 33 is used to connect the memory 32 and the processor 31.

[0172] The present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the methods provided in the various embodiments described above.

[0173] The readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of computer programs from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application-Specific Integrated Circuit (ASIC). Alternatively, the ASIC can be located in a user equipment. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0174] The present invention also provides a program product including executable instructions stored in a readable storage medium. At least one processor of the device can read the executable instructions from the readable storage medium, and the at least one processor executes the executable instructions to cause the device to implement the methods provided in the various embodiments described above.

[0175] In the embodiments of the above-described device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A processing method for acquiring images of wind turbine blades using a drone, characterized in that, include: Semantic segmentation is performed on each frame of the leaf image sequence to obtain the leaf mask corresponding to each frame of the image; Based on the leaf mask, coarse registration transformation parameters between adjacent frame images are determined, and the adjacent frame images are aligned based on the coarse registration transformation parameters to obtain the first segmentation region; Contrast enhancement is performed on the image regions corresponding to the leaf masks in each first segmentation region to obtain a first enhanced image; Feature points are extracted from the image region of the first enhanced image, and feature matching is performed. Based on the feature matching results, fine registration transformation parameters between the adjacent frame images are determined. Based on the fine registration transformation parameters, multiple frames of images in the leaf image sequence are projected and fused to generate a panoramic image of the leaf. When an abnormal area is identified in the color ring of the wind turbine blade in the blade panoramic image, a first wind turbine model is constructed based on the corresponding blade panoramic image. The blade model with the abnormal area in the first wind turbine model is regarded as the abnormal blade; Based on the location of the abnormal area in the abnormal blade and the blade position, the spraying parameters of the spraying drone are determined, including: The first wind turbine model is coordinate-based to obtain a dividing plane. Based on the dividing plane, the blade position of the abnormal blade is determined, including: Using the center point of the fairing in the first wind turbine model as the origin of the coordinate system, a vertical axis perpendicular to the ground is constructed to obtain the first coordinate system; The plane containing the horizontal and vertical axes in the first coordinate system is used as the dividing plane. Abnormal blades located above the dividing plane are used as the first spraying position, and abnormal blades located below the dividing plane are used as the second spraying position. The location of the abnormal blade is obtained based on the first spraying location and the second spraying location; When the blade position is determined to be at the first spraying position, the corresponding abnormal blade is taken as the first blade, and the first spraying parameters of the spraying drone are determined according to the location of the abnormal area in the first blade. When it is determined that the blade position is located at the second spraying position, the corresponding abnormal blade is taken as the second blade, and the second spraying parameters of the spraying drone are determined according to the location of the abnormal area in the second blade. Based on the spraying parameters, the spraying drone is controlled to spray the abnormal areas at the color ring.

2. The method according to claim 1, characterized in that, The step of determining the coarse registration transformation parameters between adjacent frame images based on the leaf mask includes: Based on the blade mask, determine the blade region in the adjacent frame image, and calculate the width information and overlapping region information of the blade region; Based on the width information and the overlapping region information, the scaling parameters and translation parameters between the adjacent frame images are calculated, and the scaling parameters and translation parameters are used as the coarse registration transformation parameters.

3. The method according to claim 1, characterized in that, The step of determining the fine registration transformation parameters between adjacent frame images based on feature matching results includes: Based on the feature matching results, obtain the initial feature matching point pairs; The erroneous matching points in the initial feature matching point pairs are removed based on the random sampling consistency algorithm, and the affine transformation matrix between adjacent frame images is calculated based on the selected correct matching points. The affine transformation matrix is ​​then determined as the fine registration transformation parameter.

4. The method according to claim 1, characterized in that, The step of projecting and fusing multiple frames of images from the leaf image sequence according to the fine registration transformation parameters to generate a panoramic image of the leaf includes: The images in the leaf image sequence are projected onto the global canvas according to the fine registration transformation parameters to obtain the projected image. The brightness of the overlapping area in the projected image is adjusted to obtain a brightness-adjusted projected image; The brightness-adjusted projected image is fused to generate the panoramic image of the blade.

5. The method according to claim 4, characterized in that, The step of adjusting the brightness of the overlapping region in the projected image to obtain a brightness-adjusted projected image includes: Calculate the brightness distribution information of the overlapping region; A brightness adjustment map is generated based on the brightness distribution information; The brightness of the overlapping region is adjusted according to the brightness adjustment map to obtain a brightness-adjusted projected image.

6. The method according to claim 1, characterized in that, The step of determining the first spraying parameters of the spraying drone based on the location of the abnormal area in the first blade includes: Connect the midpoint of the tip of the first blade and the midpoint of the root of the first blade to obtain the first direction line. The direction from the tip of the blade to the root of the first blade is taken as the first spraying direction of the first blade. The spraying width of the abnormal area is determined based on the first spraying direction, and the first included angle between the first blade and the dividing plane is obtained; When the location of the abnormal area is determined to be on the upper surface of the first blade, the abnormal area is taken as the first upper abnormal area, the preset upper angle interval in the upper angle comparison table is determined as the first upper angle interval, and the preset number of times corresponding to the first upper angle interval is retrieved as the number of times the upper surface of the first upper abnormal area is sprayed. When it is determined that the location of the abnormal area is on the lower surface of the first blade, the abnormal area is taken as the first lower abnormal area, the preset lower angle interval in the lower angle comparison table is determined as the first lower angle interval, and the preset number of superpositions corresponding to the first lower angle interval is retrieved as the number of spray superpositions for the first lower abnormal area. The number of times the upper surface is sprayed and the number of times the spraying is superimposed are used to obtain the number of times the lower surface of the first lower abnormal zone is sprayed. The first spraying parameters of the spraying drone are obtained based on the first spraying direction, spraying width, number of spraying times on the upper surface and number of spraying times on the lower surface.

7. The method according to claim 1, characterized in that, The step of determining the second spraying parameters of the spraying drone based on the location of the abnormal area in the second blade includes: Connect the midpoint of the tip of the second blade to the midpoint of the root of the blade to obtain the second direction line. The direction from the tip of the blade to the root of the blade along the second direction line is the second spraying direction of the first blade. The spraying width of the abnormal area is determined based on the second spraying direction, and the second included angle between the second blade and the dividing plane is obtained; When the location of the abnormal area is determined to be on the upper surface of the second blade, the abnormal area is taken as the second upper abnormal area. The preset upper angle interval in the upper angle comparison table is determined as the second upper angle interval. The preset number of times corresponding to the second upper angle interval is retrieved is taken as the number of times the upper surface of the second upper abnormal area is sprayed. When it is determined that the abnormal area is located on the lower surface of the second blade, the abnormal area is taken as the second lower abnormal area. The preset lower angle interval in the lower angle comparison table is determined as the second lower angle interval. The preset number of superpositions corresponding to the second lower angle interval is retrieved as the number of spray superpositions for the second lower abnormal area. The number of times the upper surface is sprayed and the number of times the spraying is superimposed are used to obtain the number of times the lower surface of the second lower abnormal zone is sprayed. The second spraying parameters of the spraying drone are obtained based on the second spraying direction, spraying width, number of spraying times on the upper surface, and number of spraying times on the lower surface.

Images