Wind power blade infrared image panoramic stitching method, device, storage medium and product

Abstract

The application discloses a kind of wind power blade infrared image panorama splicing method, equipment, storage medium and product, the splicing method includes to the visible light image and infrared image of each frame are carried out coarse registration;Visible light image and infrared image after coarse registration are respectively carried out background elimination, obtain visible light foreground mask image and infrared foreground mask image, blade visible light image and blade infrared image after eliminating background;Visible light foreground mask image and infrared foreground mask image are carried out fine registration, obtain relative displacement;Multiple blade visible light images are spliced, obtain pixel increment;According to relative displacement and pixel increment, the pixel increment when two adjacent blade infrared images are spliced is calculated;According to the pixel increment when two adjacent blade infrared images are spliced, corresponding two adjacent blade infrared images are spliced, and then wind power blade infrared panorama image is obtained.The application realizes the splicing of wind power blade infrared thermal image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a wind turbine blade infrared image panoramic stitching method assisted by visible light alignment, a device, a storage medium and a product. BACKGROUND

[0002] As a key component for obtaining wind energy, the wind turbine blade works all-weather in harsh natural environment and complex climate conditions, and defects such as cracks, debonding and pits may occur on the wind turbine blade, which poses a serious safety hazard. Traditional inspection mainly relies on manual inspection methods such as telescope detection, high-altitude round inspection and maintenance platform inspection, which have problems such as poor safety, low efficiency and strong subjectivity.

[0003] At present, the use of unmanned aerial vehicles to inspect wind turbine blades has become a mainstream trend at home and abroad, but the length of a wind turbine blade is usually about 100 m, and a single shot of an unmanned aerial vehicle cannot capture the overall appearance of the blade. Therefore, during the inspection process, the camera position needs to be continuously translated, and continuous multiple shots need to be taken to ensure overall coverage of the blade. In order to accurately analyze the defect position, area and overall condition of the blade, the blade images collected by the unmanned aerial vehicle need to be stitched.

[0004] Traditional visible light detection can only detect surface defects of an object, while infrared thermal imaging detection technology can detect internal water accumulation, delamination and debonding of a wind turbine blade, which has important application significance. Unlike visible light images that have obvious features, high contrast and clear texture, the infrared images collected are generally characterized by low resolution, small gray difference value, sparse texture features and blurred blade edge details due to the hardware limitations of infrared detectors. Traditional visible light image registration and stitching algorithms, such as feature-based image stitching algorithms and image gray matching-based image stitching algorithms, cannot achieve good results in the field of infrared image stitching, and cannot meet the panoramic stitching of wind turbine blade infrared images. SUMMARY

[0005] The present application aims to provide a wind turbine blade infrared image panoramic stitching method, device, storage medium and product, which can solve the problem that traditional visible light image registration and stitching algorithms do not work well in the field of infrared image stitching, resulting in the inability to obtain panoramic stitched images of wind turbine blade infrared images.

[0006] The present application solves the above technical problems by the following technical solutions: a wind turbine blade infrared image panoramic stitching method, comprising:

[0007] obtaining multiple frames of visible light images and infrared images of a wind turbine blade; wherein each frame of visible light image corresponds to an infrared image, and there is an overlap between adjacent two frames of images;

[0008] Coarsely register the visible light image and the infrared image of each frame to obtain the infrared image coarsely registered with the visible light image;

[0009] Respectively perform background elimination on the coarsely registered visible light image and infrared image to obtain a visible light foreground mask image and an infrared foreground mask image, and a leaf visible light image and a leaf infrared image after background elimination;

[0010] Perform fine registration on the visible light foreground mask image and the infrared foreground mask image to obtain a relative displacement of the visible light foreground mask image and the infrared foreground mask image in fine registration;

[0011] Perform splicing on a plurality of leaf visible light images after background elimination to obtain a pixel increment in splicing of adjacent two leaf visible light images;

[0012] Calculate a pixel increment in splicing of corresponding adjacent two leaf infrared images according to the relative displacement and the pixel increment in splicing of adjacent two leaf visible light images;

[0013] Perform splicing on corresponding adjacent two leaf infrared images after background elimination according to the pixel increment in splicing of adjacent two leaf infrared images to obtain a wind power blade infrared panoramic image.

[0014] Further, before obtaining the plurality of visible light images and infrared images of the wind power blade, a single target calibration is performed on a visible light camera for collecting the visible light images and an infrared thermal imaging camera for collecting the infrared images by using Zhang Zhengyou calibration method to obtain an intrinsic matrix of the visible light camera, an intrinsic matrix of the infrared thermal imaging camera, and distortion correction parameters of the visible light image and the infrared image.

[0015] Further, the specific implementation process of obtaining the plurality of visible light images and infrared images of the wind power blade is as follows:

[0016] Lock the wind power blade to be inspected;

[0017] Use a drone to carry the visible light camera and the infrared thermal imaging camera, and control the drone to fly in a straight line along a single blade direction and parallel to the blade surface and collect images.

[0018] Further, the coarse registration of the visible light image and the infrared image of each frame includes:

[0019] According to the intrinsic matrix of the visible light camera and the distortion correction parameters of the visible light image obtained by calibration, perform a distortion elimination operation on the visible light image;

[0020] According to the intrinsic matrix of the infrared thermal imaging camera and the distortion correction parameters of the infrared image obtained by calibration, perform a distortion elimination operation on the infrared image;

[0021] The registration conversion matrix and the offset are calculated according to the visible light image and the infrared image after the distortion removal operation; wherein the calculation formula of the registration conversion matrix is:

[0022]

[0023] wherein R C represents the registration conversion matrix, δ s represents the scaling ratio, and respectively represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the chessboard calibration board in the visible light image, and respectively represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the chessboard calibration board in the infrared image; the chessboard calibration board is located in front of the visible light camera and the infrared thermal imaging camera;

[0024] The calculation formula of the offset is:

[0025] x d =x vis -x inf , y d =y vis -y inf ;

[0026] wherein x d and y d represent the horizontal and vertical offsets of the infrared image, (x vis , y vis ) and (x inf , y inf ) respectively represent the pixel coordinates of the center of any square in the chessboard calibration board in the visible light image and the infrared image;

[0027] The infrared image coarsely registered with the visible light image is calculated according to the registration conversion matrix and the offset, and the specific formula is:

[0028]

[0029] wherein (Inf ix , Inf iy ) represents the pixel coordinates of the infrared image after the distortion removal operation, and (Inf ix ', Inf') represents the pixel coordinates of the infrared image coarsely registered with the visible light image.

[0030] Further, the visible light image and the infrared image after the coarse registration are respectively subjected to background removal, including:

[0031] The visible light image and the infrared image after coarse registration are respectively subjected to background segmentation to obtain a visible light foreground mask image and an infrared foreground mask image;

[0032] The visible light foreground mask image and the infrared foreground mask image are respectively subjected to edge contour smoothing operation;

[0033] The foreground part of the visible light foreground mask image after smoothing operation is set to 1 respectively, and then multiplied by the visible light image to obtain a leaf visible light image after background elimination; the foreground part of the infrared foreground mask image after smoothing operation is set to 1 respectively, and then multiplied by the infrared image after coarse registration to obtain a leaf infrared image after background elimination.

[0034] Further, the visible light foreground mask image and the infrared foreground mask image are subjected to fine registration, including:

[0035] The visible light foreground mask image and the infrared foreground mask image are respectively subjected to edge detection to obtain a leaf visible light edge image and a leaf infrared edge image;

[0036] The leaf visible light edge image and the leaf infrared edge image are respectively subjected to boundary extraction to obtain a first boundary coordinate list and a second boundary coordinate list; wherein the first boundary coordinate list is a boundary coordinate list of the leaf visible light edge image, and the second boundary coordinate list is a boundary coordinate list of the leaf infrared edge image;

[0037] A first width information list is calculated according to the first boundary coordinate list, and a second width information list is calculated according to the second boundary coordinate list;

[0038] A relative displacement of the visible light foreground mask image and the infrared foreground mask image during fine registration is calculated according to the first width information list and the second width information list, and the specific calculation formula is:

[0039] h k =y i ″,y i ″←min|w i ″-w′0|;

[0040] wherein h k represents the relative displacement of the visible light foreground mask image and the infrared foreground mask image during fine registration of the kth frame; w0′ represents the topmost width in the first width information list; w i ″ represents the width of the i-th row in the second width information list; y i ″ represents the ordinate of the i-th row in the second width information list.

[0041] Further, a specific calculation formula of the pixel increment when the two adjacent frames of blade infrared images are spliced is:

[0042] P k,k-1 =Q k,k-1 +(h k -h k-1 );

[0043] Wherein, P k,k-1 represents the pixel increment when the kth frame of blade infrared image after background elimination and the (k-1)th frame of blade infrared image after background elimination are spliced; Q k,k-1 represents the pixel increment when the kth frame of blade visible light image after background elimination and the (k-1)th frame of blade visible light image after background elimination are spliced; h k represents the relative displacement when the kth frame of visible light foreground mask image and infrared foreground mask image are precisely registered; h k-1 represents the relative displacement when the (k-1)th frame of visible light foreground mask image and infrared foreground mask image are precisely registered.

[0044] Based on the same concept, the present application also provides an electronic device, comprising a memory, a processor and a computer program / instruction stored in the memory, wherein the processor executes the computer program / instruction to realize the wind power blade infrared image panorama splicing method as described above.

[0045] Based on the same concept, the present application also provides a computer readable storage medium, which stores a computer program / instruction, wherein the computer program / instruction is executed by a processor to realize the wind power blade infrared image panorama splicing method as described above.

[0046] Based on the same concept, the present application also provides a computer program product, which comprises a computer program / instruction, wherein the computer program / instruction is executed by a processor to realize the wind power blade infrared image panorama splicing method as described above.

[0047] Advantages

[0048] Compared with the prior art, the present application has the following advantages:

[0049] The present application realizes the splicing of the wind power blade infrared image by means of the splicing of the wind power blade visible light image with more abundant gray scale feature information through the registration operation between the heterogeneous images, solves the problem that the wind power blade infrared image with low resolution, sparse features and fuzzy details is difficult to splice, and effectively realizes the splicing of the wind power blade infrared thermal image with more sparse texture features, and the splicing result is stable and reliable. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the present application, the drawings required to be used in the following embodiment description will be briefly introduced. Obviously, the drawings described in the following description are only a part of the embodiments of the present application, and all other drawings obtained by those skilled in the art without creative effort based on these drawings also belong to the protection scope of the present application.

[0051] Figure 1 is a flow chart of the wind turbine blade infrared image panoramic stitching method in the embodiment of the present application;

[0052] Figure 2 is a schematic diagram of the collection of visible light images and infrared images in the embodiment of the present application; wherein, UAV represents a UAV;

[0053] Figure 3 is a schematic diagram of the coarse registration process in the embodiment of the present application;

[0054] Figure 4 is a schematic diagram of the effect of background elimination of the visible light image in the embodiment of the present application;

[0055] Figure 5 is a schematic diagram of the effect of background elimination of the infrared image in the embodiment of the present application;

[0056] Figure 6 is a schematic diagram of the fine registration process in the embodiment of the present application;

[0057] Figure 7 is a schematic diagram of the effect of stitching of the visible light image in the embodiment of the present application;

[0058] Figure 8 is a schematic diagram of the effect of stitching of the infrared image in the embodiment of the present application;

[0059] Figure 9 is a wind turbine blade infrared panoramic image in the embodiment of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the present application will be described clearly and completely in the following description in combination with the drawings in the embodiment of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort also belong to the protection scope of the present application.

[0061] The technical solutions of the present application will be described in detail in the following specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0062] As Figure 1As shown, the wind turbine blade infrared image panoramic stitching method provided by the embodiment of the application comprises the following steps:

[0063] Step 1: Calibration of the visible light camera and the infrared thermal imaging camera.

[0064] The visible light camera is used to collect the visible light image of the wind turbine blade, and the infrared thermal imaging camera is used to collect the infrared image of the wind turbine blade. Before collecting the dual light images (i.e. the visible light image and the infrared image) of the wind turbine blade, the Zhang Zhengyou calibration method is used to calibrate the single target of the visible light camera and the infrared thermal imaging camera respectively. The specific calibration process is performed on OpenCV, and the specific process is as follows: a plurality of checkerboard calibration board images are shot from different angles, the findChessboardCorners function is used to judge and search the checkerboard corner points in the images, then the cornerSubPix function is used to further determine the coordinates of the corner points of the checkerboard calibration board, and finally the calibrateCamera function is called to calculate the intrinsic matrix of the visible light camera and the infrared thermal imaging camera, and the distortion correction parameters of the visible light image and the infrared image.

[0065] Due to the different lens positions and focal lengths of the visible light camera and the infrared thermal imaging camera, the imaging sizes and positions of the space objects shot by the two lenses are also different. Therefore, the checkerboard calibration board is placed in front of the two cameras, and it is ensured that the complete calibration board image can be captured when the two cameras are imaging, so as to obtain the square imaging size and the relative distance of the calibration board of the visible light camera and the infrared thermal imaging camera, so as to calculate the coordinate mapping relationship of the coarse registration alignment of the heterogeneous images, i.e. the registration conversion matrix R C .

[0066] Step 2: Obtain multiple frames of visible light images and infrared images of the wind turbine blade.

[0067] The present application uses a UAV to carry a visible light camera and an infrared thermal imaging camera to collect the visible light image and the infrared image of the wind turbine blade. As shown, Figure 2 Before image collection, the wind turbine blade to be inspected is locked, and then the UAV is controlled to fly along the direction of a single blade and parallel to the surface of the blade to collect images. The images can be collected in the direction from the root to the tip of the wind turbine blade, or the images can be collected in the direction from the tip to the root of the wind turbine blade. The visible light camera and the infrared thermal imaging camera collect images at the same time and have the same time interval, so each frame of image includes one frame of visible light image and one frame of infrared image, and the visible light image and the infrared image of the same frame correspond to the same part of the wind turbine blade. In order to facilitate stitching and ensure the integrity of the stitched image, there is an overlap between adjacent two frames of images.

[0068] To simplify the trajectory control scheme of the UAV, the wind turbine blade to be inspected is first locked in a vertically downward direction. Then, the pitch angle of the UAV is adjusted to keep the UAV at a pitch angle of 0°. Finally, the UAV is controlled to fly in a straight line perpendicular to the ground and collect images.

[0069] Step 3: Perform coarse registration on the visible light image and infrared image of each frame to obtain the infrared image coarsely registered with the visible light image.

[0070] For each frame of the visible light image and infrared image, the coarse registration operation in step 3, the background removal operation in step 4, and the fine registration operation in step 5 are performed. In a specific embodiment of the present invention, the coarse registration of each frame of the visible light image and infrared image includes:

[0071] Step 3.1: Based on the intrinsic parameter matrix of the visible light camera and the distortion correction parameters of the visible light image obtained in Step 1, perform distortion correction operation on the visible light image;

[0072] Step 3.2: Based on the intrinsic parameter matrix of the infrared thermal imaging camera and the distortion correction parameters of the infrared image obtained in Step 1, perform distortion removal operation on the infrared image; the distortion removal operation can be implemented by calling the undistort function in OpenCV.

[0073] Step 3.3: Calculate the registration transformation matrix and offset based on the distortion-corrected visible light and infrared images; the formula for calculating the registration transformation matrix is ​​as follows:

[0074]

[0075] Among them, R C Denotes the registration transformation matrix, δ s Indicates the scaling ratio. and Let i and i-1 represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the visible light image, respectively. and These represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the infrared image, respectively, as shown below. Figure 3 As shown. Registration transformation matrix R C The scaling factor δ is calculated based on the image size and relative distance of the checkerboard calibration board in the visible light camera and the infrared thermal imaging camera. Before calculating the registration transformation matrix, the scaling factor δ is calculated based on the pixel difference between the coordinates of the centers of any two adjacent black squares in the checkerboard calibration board in the visible light image and the infrared image, respectively. s .

[0076] There is a certain relative offset between the visible light image and the infrared image. Therefore, it is necessary to calculate the image offset based on the pixel coordinates of the center of any square in the checkerboard calibration plate in the visible light image and the infrared image. The specific formula for calculating the offset is as follows:

[0077] x d =x vis -x inf (3)

[0078] y d =y vis -y inf (4)

[0079] Where, x d and y d This represents the offset of the infrared image in the horizontal and vertical directions, (x) vis ,y vis ), (x inf ,y inf ) represent the pixel coordinates of the center of any square in the checkerboard calibration board in the visible light image and the infrared image, respectively.

[0080] Step 3.4: Calculate the infrared image coarsely registered with the visible light image based on the registration transformation matrix and offset, thus achieving coarse registration between the visible light image and the infrared image.

[0081] In this embodiment, the specific formula for the infrared image coarsely registered with the visible light image is as follows:

[0082]

[0083] Among them, (Inf ix Inf iy (Inf) represents the pixel coordinates of the infrared image after distortion correction. ix (',Inf') represents the pixel coordinates of the infrared image coarsely registered with the visible light image.

[0084] Step 4: Perform background removal on the coarsely registered visible light image and infrared image to obtain visible light foreground mask image and infrared foreground mask image, as well as visible light image and infrared image of the leaf after background removal.

[0085] like Figure 4 and Figure 5 As shown, the wind turbine blade images captured by the visible light camera and the infrared thermal imaging camera contain a large amount of background information. This complex background information can affect the subsequent stitching results, necessitating background removal operations for both the visible light and infrared images. In a specific embodiment of this invention, background removal is performed on the coarsely registered visible light and infrared images, including:

[0086] Step 4.1: background segmentation is performed on the coarse-registered visible light image and infrared image respectively to obtain a visible light foreground mask image A and an infrared foreground mask image B;

[0087] Step 4.2: edge contour smoothing operation is performed on the visible light foreground mask image A and the infrared foreground mask image B respectively.

[0088] Step 4.3: the foreground part of the visible light foreground mask image after the smoothing operation is set to 1 respectively, and then multiplied with the visible light image, so that the pixel value of the blade part in the visible light image remains the original value, thereby obtaining a blade visible light image after background elimination, as shown in Figure 4 .

[0089] Step 4.4: the foreground part of the infrared foreground mask image after the smoothing operation is set to 1 respectively, and then multiplied with the coarse-registered infrared image, so that the pixel value of the blade part in the infrared image remains the original value, thereby obtaining a blade infrared image after background elimination, as shown in Figure 5 .

[0090] In this embodiment, U-net network is used to perform background segmentation on the coarse-registered visible light image and infrared image. U-net network is an effective semantic segmentation framework. Before using U-net network to perform background segmentation on the coarse-registered visible light image and infrared image, a training sample data set is constructed to train the U-net network. The training sample data set includes multiple samples, each sample includes a coarse-registered infrared image and its label or a visible light image and its label. Labeling the visible light image and the coarse-registered infrared image using Labelme software can obtain pixel-level labels.

[0091] In order to make the edge contour of the visible light foreground mask image A and the infrared foreground mask image B obtained by background segmentation more smooth and natural, it is also necessary to eliminate the small gap between the visible light foreground mask image A and the image boundary, and set a 7x7 all-1 matrix as the convolution kernel of the closing operation, to perform the closing operation of inflation first and then corrosion on the visible light foreground mask image A to fill and connect the gaps of the image edge, so as to bridge the narrow discontinuities and elongated gullies between the visible light foreground mask image A and the image boundary, and make the transition of the wind power blade boundary more natural. Similarly, the infrared foreground mask image B is also subjected to the closing operation of inflation first and then corrosion.

[0092] Step 5: fine registration is performed on the visible light foreground mask image and the infrared foreground mask image to obtain the relative displacement of the visible light foreground mask image and the infrared foreground mask image in fine registration.

[0093] The fine registration of the dual-optical images of the present application is achieved according to the search matching of the leaf width information in the visible light foreground mask image and the infrared foreground mask image. In the detailed embodiments of the present application, the fine registration of the visible light foreground mask image and the infrared foreground mask image comprises:

[0094] Step 5.1: edge detection is performed on the visible light foreground mask image and the infrared foreground mask image respectively to obtain a leaf visible light edge image and a leaf infrared edge image.

[0095] In this embodiment, Canny edge detection operator is used to perform edge extraction on the visible light foreground mask image and the infrared foreground mask image respectively, which specifically comprises: after the image is smoothed by using Gaussian filtering, the gradient amplitude and direction of the image pixels are calculated. Specifically, Sobel horizontal operator S x and Sobel vertical operator S y are respectively convolved with the input image (i.e. the visible light foreground mask image or the infrared foreground mask image) to calculate the gradient amplitude E x and E y of each pixel point of the image in the horizontal direction and the vertical direction, and further the gradient amplitude E and the direction θ of each pixel point of the image are obtained, and the specific formula is as follows:

[0096]

[0097] E x = f(x, y) * S x (8)

[0098] E y = f(x, y) * S y (9)

[0099]

[0100] θ = arctan(E y / E x ) (11)

[0101] Wherein, f(x, y) represents the image region covered by the convolution kernel with the point (x, y) as the center when the gradient of each pixel point is calculated by convolution.

[0102] The non-maximum suppression method is applied to the image obtained by edge detection to filter out non-edge pixels, and then double-threshold detection is performed to remove false edges, and finally the edges are connected to obtain a complete leaf edge image, i.e. a leaf visible light edge image A' and a leaf infrared edge image B' are obtained respectively.

[0103] Step 5.2: Extract the boundaries of the visible light edge image and the infrared edge image of the leaf respectively to obtain a first boundary coordinate list and a second boundary coordinate list; wherein, the first boundary coordinate list is the boundary coordinate list of the visible light edge image of the leaf, and the second boundary coordinate list is the boundary coordinate list of the infrared edge image of the leaf.

[0104] For the visible light edge image A' of the blade, with the top left corner of the image as the origin, the horizontal direction as the X-axis, and the vertical direction as the Y-axis, traverse the image to obtain points with non-zero pixel values, and save the coordinates of these points and their corresponding pixel values ​​to a list; judge the coordinates of the points in the list. If the distance between any two points in each row of the list is less than the pixel threshold (e.g., 3 pixels), then the two points are considered to come from the same boundary; otherwise, the two points are considered to belong to different boundaries, and the two points are the left and right edge points of the corresponding row, respectively. Thus, all points on the left and right edges of each row of the blade are obtained; for each row, take the coordinates of the point with the largest pixel value among all points on the left and right edges of the blade to obtain the first boundary coordinate list, denoted as:

[0105] L1=[[y0,x 0l ,x 0r ],[y1,x 1l ,x 1r ],...,[y i ,x il ,x ir ],...,[y N ,x Nl ,x Nr (12)

[0106] Where L1 represents the list of first boundary coordinates, x il The x-coordinate represents the x-coordinate of the point with the largest pixel value on the left edge of the i-th row of the leaf. ir The x-coordinate of the point with the largest pixel value on the right edge of the i-th row of the leaf is represented by y. i Indicates with x il and x ir The corresponding ordinates (i.e., the ordinates of the points with the largest left and right edge pixel values ​​in the i-th row), where N represents the index of the last row, and there are a total of N+1 rows. Points with the same ordinate in the image form a row; that is, pixels in each row have the same ordinate that is not 0.

[0107] Similarly, for the infrared edge image B' of the blade, the second boundary coordinate list L2 can be obtained.

[0108] Step 5.3: Calculate the first width information list based on the first boundary coordinate list, and calculate the second width information list based on the second boundary coordinate list.

[0109] Based on the first boundary coordinate list L1 or the second boundary coordinate list L2, the width calculation formula w is used. i =x ir -x il Calculate the blade width for each row to obtain the first width information list L. vis =[[y0′,w0′],[y1′,w1′]...[y i ′,w i ′]...[y′ N ,w′ N Second width information list L inf =[[y′0′,w0″],[y1″,w1″]...[y i ",w i "]...[y′ N ′,w′ N ′]], where y i ′、y i "" represents the ordinate of the i-th row in the visible light edge image A' and the infrared edge image B' of the leaf, respectively, w i ′、w i "" represents the width of the i-th row in the visible light edge image A' and the infrared edge image B' of the leaf, respectively.

[0110] Step 5.4: Calculate the relative displacement of the visible light foreground mask image A and the infrared foreground mask image B during fine registration based on the first width information list and the second width information list.

[0111] like Figure 6 As shown, the first width information list L vis Using the width of the topmost element (i.e., the width of the first row, with the ordinate of the first row being 0) w0′ as a reference, the second width information list L is... inf The process involves iterating through the data to obtain the ordinate of the row corresponding to the blade infrared edge image B' that is closest to w0′. This ordinate represents the relative displacement of the blade infrared edge image B' relative to the blade visible light edge image A' during precise registration. The specific calculation formula is as follows:

[0112] h k =y i ", y i "←min|w i "-w′0| (13)

[0113] Among them, h k This represents the relative displacement between the visible light foreground mask image and the infrared foreground mask image during fine-tuning in the k-th frame; w0′ represents the width of the topmost (i.e., the first row) in the first width information list; w i "" indicates the width of the i-th row in the second width information list, where the width of the row is w.i " is the closest to w0'; y i " represents the vertical coordinate of the i-th row in the second width information list, that is, the width w i " corresponds to the vertical coordinate of the row.

[0114] For each frame of infrared foreground mask image (or each frame of infrared image), there is a relative displacement h k .

[0115] Step 6: Splicing the multiple frames of background-removed leaf visible light images to obtain the pixel increment when splicing the adjacent two frames of leaf visible light images.

[0116] In the specific embodiment of the present application, the NCC (Normalized Cross-Correlation) algorithm based on image gray scale is used to splice the multiple frames of background-removed leaf visible light images. The NCC (Normalized Cross-Correlation) algorithm based on image gray scale is a prior art, and its principle is: matching the adjacent two frames of images to obtain the best matching position, and then splicing and synthesizing the adjacent two frames of images according to the best matching position. The NCC algorithm obtains the similarity degree of the template image and the search image at different positions by using the evaluation function, and the position with the maximum similarity degree is the best matching position. The similarity calculation formula is:

[0117]

[0118] Wherein, T(m,n) represents the template image, S i,j represents the sub-image covered by the template image in the search image, m and n represent the width and height of the template image, and represent the gray scale mean of the template image and the sub-image, respectively.

[0119] The NCC algorithm based on image gray scale has a long splicing time and requires a high image contrast, so an improved NCC algorithm with boundary search is used. As shown in Figure 7 , the specific implementation process of the improved NCC algorithm with boundary search is as follows:

[0120] The background-removed leaf visible light image is preprocessed by histogram equalization (for details, see the literature: Wang Chunzhi, Niu Hongxia. Dust degradation image enhancement algorithm based on histogram equalization and MSRCR [J]. Computer Engineering, 2022, 48(09): 223-229. DOI: 10.19678 / j.issn.1000-3428.0062764.), which improves the image contrast;

[0121] For stitching two adjacent image frames, the next image is used as the image to be searched, and a portion of the overlapping area of ​​the previous image is cropped as a template image (e.g., ...). Figure 7 To be spliced Figure 1 (see the blue box), using downsampling to construct a two-layer pyramid sequence for the search image and the template image respectively;

[0122] Based on the first boundary coordinate list, the top layer image of the pyramid is first matched. During matching, the matching area is formed by expanding 10 pixels to the left and right of the right boundary of the leaf in the top layer image of the image to be searched. The top layer image of the template image is also based on the right boundary of the leaf. The correct matching position is found by traversing and searching within the matching area. Since the size of the top layer image of the pyramid is 1 / 4 of the original image, the amount of computation can be reduced during matching.

[0123] After determining the initial optimal matching position by matching the top-level image, the process is then applied to the next layer of the pyramid. A 6×6 pixel rectangle is expanded around the initial optimal matching position in the image to be searched. This process is repeated, traversing and searching around the right boundary of the blade to obtain the optimal matching position. During the traversal and search matching, the normalized cross-correlation calculation is performed using a search matching method along the blade boundary, which improves the matching speed.

[0124] Finally, the pixel increment Q between two adjacent frames is obtained based on the optimal matching position. k,k-1 (i.e., Q) The previous frame image (i.e., the (k-1)th frame) is placed at the pixel increment Q position of the next frame image (i.e., the kth frame) to achieve the stitching of two adjacent visible light images of the leaf.

[0125] Step 7: Calculate the pixel increment when stitching the corresponding infrared images of two adjacent blades based on the relative displacement and the pixel increment when stitching the visible light images of two adjacent blades.

[0126] In this embodiment, the specific formula for calculating the pixel increment when stitching together two adjacent frames of leaf infrared images is as follows:

[0127] P k,k-1 =Q k,k-1 +(h k -h k-1 (15)

[0128] Among them, P k,k-1 Q represents the pixel increment when stitching the background-removed infrared image of the leaf in frame k to the background-removed infrared image of the leaf in frame (k-1); k,k-1 h represents the pixel increment when stitching the visible light image of the leaf after background removal in frame k to the visible light image of the leaf after background removal in frame (k-1); k h represents the relative displacement between the visible light foreground mask image and the infrared foreground mask image during fine registration of the k-th frame;k-1 This represents the relative displacement between the visible light foreground mask image and the infrared foreground mask image during fine registration of the (k-1)th frame.

[0129] Step 8: Based on the pixel increment when stitching adjacent two frames of blade infrared images, stitch together the corresponding adjacent two frames of blade infrared images after removing the background to obtain a panoramic infrared image of the wind turbine blade.

[0130] like Figure 8 As shown, based on the pixel increment P when stitching together two adjacent frames of leaf infrared images k,k-1 The infrared image of the leaf after removing the background in frame k-1 is placed at the pixel increment P of the infrared image of the leaf after removing the background in frame k. k,k-1 The position is determined by stitching the (k-1)th frame of the leaf infrared image with the kth frame. The stitched result is then stitched with the next frame (i.e., the (k+1)th frame) of the leaf infrared image using the same method. Specifically, the (k+1)th frame's visible light image of the leaf after background removal is used as the search image. A portion of the overlapping area of ​​the corresponding visible light stitching result (i.e., the stitching result of the (k-1)th frame and the kth frame's visible light image of the leaf) is used as the template image for the next stitching. The corresponding pixel increment P is matched and mapped to calculate. The leaf infrared image is then stitched according to the pixel increment P. Steps 6 to 8 are executed iteratively to obtain the following result: Figure 9 The image shown is an infrared panoramic image of a wind turbine blade.

[0131] In another specific embodiment of the present invention, the pixel increment of each stitching when stitching the visible light images of the leaf can also be obtained first, that is, the pixel increment when stitching the first frame and the second frame, the pixel increment when stitching the stitching result of the first frame and the second frame and the third frame, the pixel increment when stitching the previous stitching result (that is, the stitching result of the first frame and the second frame and the third frame) and the fourth frame, ...; then the pixel increment when stitching the corresponding infrared images of the leaf can be calculated according to step 7; finally, the stitching of the corresponding two frames of infrared images of the leaf can be performed according to step 8.

[0132] Example 2

[0133] This invention also provides an electronic device, which includes a memory, a processor, and a computer program / instructions stored in the memory. The processor executes the computer program / instructions to implement the panoramic stitching method for infrared images of wind turbine blades in this application embodiment.

[0134] Although not shown, the electronic device includes a processor that can perform various appropriate operations and processes according to programs and / or data stored in a read-only memory (ROM) or programs and / or data loaded from a storage section into a random access memory (RAM). The processor can be one multi-core processor or can include a plurality of processors. In some embodiments, the processor can include a general-purpose main processor and one or more special-purpose co-processors, such as a central processing unit, a graphics processing unit (GPU), a neural processing unit (NPU), a digital signal processor (DSP), and the like. In the RAM, various programs and data required for device operations are also stored. The processor, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0135] The processor described above is used in common with the memory to execute programs / instructions stored in the memory, which, when executed by a computer, can implement the methods, steps, or functions described in the above embodiments.

[0136] Although not shown, the embodiments of the present application also provide a computer-readable storage medium having stored thereon computer programs / instructions, which, when executed by a processor, implement the wind power blade infrared image panoramic stitching method in the embodiments of the present application.

[0137] The computer-readable storage medium includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media do not include transitory media such as modulated data signals and carriers.

[0138] Although not shown, the embodiments of the present application also provide a computer program product, comprising: computer programs / instructions, which, when executed by a processor, implement the wind power blade infrared image panoramic stitching method in the embodiments of the present application.

[0139] The above merely provides the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of the changes or modifications within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for panoramic stitching of infrared images of wind turbine blades, characterized in that, The splicing method includes: Acquire multiple frames of visible light and infrared images of wind turbine blades; where each frame of visible light image corresponds to an infrared image, and there is overlap between adjacent frames; Coarse registration is performed on the visible light image and infrared image of each frame to obtain the infrared image coarsely registered with the visible light image; Background removal was performed on the coarsely registered visible light image and infrared image to obtain visible light foreground mask image and infrared foreground mask image, as well as visible light image and infrared image of the leaf after background removal; Fine registration is performed on the visible light foreground mask image and the infrared foreground mask image to obtain the relative displacement of the visible light foreground mask image and the infrared foreground mask image during fine registration; The pixel increment when stitching together two adjacent visible light images of leaves after removing the background is obtained by stitching together multiple frames of visible light images of leaves. The pixel increment when stitching the corresponding two adjacent frames of the leaf infrared image is calculated based on the relative displacement and the pixel increment when stitching the visible light images of two adjacent frames of the leaf. Based on the pixel increment when stitching two adjacent frames of blade infrared images, the corresponding two adjacent frames of blade infrared images after removing the background are stitched together to obtain a panoramic infrared image of the wind turbine blade. This includes coarse registration of the visible light and infrared images for each frame, including: Based on the intrinsic parameter matrix of the visible light camera obtained from calibration and the distortion correction parameters of the visible light image, a distortion correction operation is performed on the visible light image; Based on the calibrated intrinsic parameter matrix of the infrared thermal imaging camera and the distortion correction parameters of the infrared image, the infrared image is subjected to distortion correction operation. Calculate the registration transformation matrix and offset based on the visible light image and infrared image after distortion correction; Calculate the infrared image coarsely registered with the visible light image based on the registration transformation matrix and offset; This includes fine registration of the visible light foreground mask image and the infrared foreground mask image, including: Edge detection is performed on the visible light foreground mask image and the infrared foreground mask image respectively to obtain the visible light edge image and the infrared edge image of the leaf; Boundary extraction is performed on the visible light edge image and the infrared edge image of the leaf to obtain a first boundary coordinate list and a second boundary coordinate list; wherein, the first boundary coordinate list is the boundary coordinate list of the visible light edge image of the leaf, and the second boundary coordinate list is the boundary coordinate list of the infrared edge image of the leaf. Calculate a first width information list based on the first boundary coordinate list, and calculate a second width information list based on the second boundary coordinate list; The relative displacement during fine registration of the visible light foreground mask image and the infrared foreground mask image is calculated based on the first width information list and the second width information list.

2. The panoramic stitching method for infrared images of wind turbine blades according to claim 1, characterized in that, Before acquiring multiple visible light and infrared images of wind turbine blades, the Zhang Zhengyou calibration method is used to perform single-target calibration on the visible light camera used to acquire visible light images and the infrared thermal imaging camera used to acquire infrared images, respectively, to obtain the intrinsic parameter matrix of the visible light camera, the intrinsic parameter matrix of the infrared thermal imaging camera, and the distortion correction parameters of the visible light and infrared images.

3. The panoramic stitching method for infrared images of wind turbine blades according to claim 1, characterized in that, The specific implementation process for acquiring multiple frames of visible light and infrared images of wind turbine blades is as follows: Lock the wind turbine blades to be inspected; Using a drone equipped with a visible light camera and an infrared thermal imaging camera, the drone is controlled to fly in a straight line along the direction of a single blade and parallel to the blade surface to collect images.

4. The panoramic stitching method for infrared images of wind turbine blades according to claim 1, characterized in that, The formula for calculating the registration transformation matrix is ​​as follows: Among them, R C Denotes the registration transformation matrix, δ s Indicates the scaling ratio. and Let i and i-1 represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the visible light image, respectively. and These represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the checkerboard calibration plate in the infrared image, respectively; the checkerboard calibration plate is located directly in front of the visible light camera and the infrared thermal imaging camera. The formula for calculating the offset is: x d =x vis -x inf ,and d / and vis -and inf ; Where, x d and y d This represents the offset of the infrared image in the horizontal and vertical directions, (x) vis ,y vis ), (x inf ,y inf ) represent the pixel coordinates of the center of any square in the checkerboard calibration board in the visible light image and the infrared image, respectively; The specific formula for the infrared image coarsely registered with the visible light image is as follows: Among them, (Inf ix Inf iy (Inf′) represents the pixel coordinates of the infrared image after distortion correction. ix ,Inf′) represents the pixel coordinates of the infrared image coarsely registered with the visible light image.

5. The panoramic stitching method for infrared images of wind turbine blades according to claim 1, characterized in that, Background removal was performed on the coarsely registered visible light and infrared images, including: Background segmentation is performed on the coarsely registered visible light image and infrared image to obtain visible light foreground mask image and infrared foreground mask image respectively; Edge contour smoothing operations are performed on the visible light foreground mask image and the infrared foreground mask image, respectively; The RGB channel pixels of the foreground portion of the smoothed visible light foreground mask image are set to 1, and then multiplied with the visible light image to obtain the visible light image of the leaf after removing the background; the RGB channel pixels of the foreground portion of the smoothed infrared foreground mask image are set to 1, and then multiplied with the coarsely registered infrared image to obtain the infrared image of the leaf after removing the background.

6. The panoramic stitching method for infrared images of wind turbine blades according to claim 1, characterized in that, The specific formula for calculating the relative displacement between the visible light foreground mask image and the infrared foreground mask image during fine registration is as follows: h k =y″ i ,y″ i ←min|w″ i -w′0|; Among them, h k w' represents the relative displacement during fine-tuning of the visible light foreground mask image and the infrared foreground mask image in the k-th frame; w'0 represents the width of the topmost element in the first width information list; w″ i This represents the width of the i-th row in the second width information list; y″ i This represents the ordinate of the i-th row in the second width information list.

7. The panoramic stitching method for infrared images of wind turbine blades according to any one of claims 1 to 6, characterized in that, The specific formula for calculating the pixel increment when stitching together two adjacent frames of leaf infrared images is as follows: P k,k-1 =Q k,k-1 +(h k -h k-1 ); Among them, P k,k-1 Q represents the pixel increment when stitching the background-removed infrared image of the leaf in frame k to the background-removed infrared image of the leaf in frame (k-1); k,k-1 h represents the pixel increment when stitching the visible light image of the leaf after background removal in frame k to the visible light image of the leaf after background removal in frame (k-1); k h represents the relative displacement between the visible light foreground mask image and the infrared foreground mask image during fine registration of the k-th frame; k-1 This represents the relative displacement between the visible light foreground mask image and the infrared foreground mask image during fine registration of the (k-1)th frame.

8. An electronic device comprising a memory, a processor, and a computer program / instructions stored in the memory, characterized in that, The processor executes the computer program / instructions to implement the panoramic stitching method for infrared images of wind turbine blades as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the panoramic stitching method for infrared images of wind turbine blades as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the panoramic stitching method for infrared images of wind turbine blades as described in any one of claims 1 to 7.

Images