Wind power blade infrared image panoramic splicing method and device, storage medium and product

Abstract

The invention discloses a wind power blade infrared image panoramic splicing method and device, a storage medium and a product. The splicing method comprises the following steps: performing coarse registration on each frame of visible light image and infrared image; respectively removing backgrounds of the visible light image and the infrared image after coarse registration to obtain a visible light foreground mask image, an infrared foreground mask image, and a blade visible light image and a blade infrared image after the backgrounds are removed; performing fine registration on the visible light foreground mask image and the infrared foreground mask image to obtain relative displacement; splicing the multiple frames of leaf visible light images to obtain a pixel increment; according to the relative displacement and the pixel increment, calculating the pixel increment when the two adjacent leaf infrared images are spliced; and splicing the corresponding two adjacent frames of blade infrared images according to the pixel increment when the two adjacent frames of blade infrared images are spliced, so as to obtain a wind power blade infrared panoramic image. According to the invention, the splicing of the infrared thermal images of the wind power blade is realized.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a method, device, storage medium and product for panoramic stitching of infrared images of wind turbine blades with visible light auxiliary alignment. Background Art

[0002] As a key component for obtaining wind energy, wind turbine blades will have cracks, debonding, pits and other defects due to working in harsh natural environments and complex climatic conditions all day long, which poses a serious safety hazard. Traditional inspections mainly rely on manual inspection methods such as telescope inspections, high-altitude detours and visual inspections, and maintenance platform inspections, which have problems such as poor safety, low efficiency, and strong subjectivity.

[0003] At present, the use of drones to inspect wind turbine blades has become a mainstream trend at home and abroad, but the length of wind turbine blades is usually about 100 meters, and a single shot by a drone cannot capture the entire blade. Therefore, during the inspection process, the camera position needs to be continuously translated and multiple shots are taken continuously to ensure the overall coverage of the blade. In order to accurately analyze the defect location, area and overall condition of the blade, the blade images collected by the drone need to be spliced.

[0004] Traditional visible light detection can only detect surface defects of objects, while infrared thermal imaging detection technology can detect defects such as water accumulation, delamination, and debonding inside wind turbine blades, which has important application significance. Different from visible light images with obvious features, high contrast, and clear texture, due to the hardware limitations of infrared detectors themselves, the collected infrared images generally have the characteristics of low resolution, small grayscale difference value, sparse texture features, and blurred blade edge details. Traditional visible light image registration and stitching algorithms, such as image stitching algorithms based on feature registration and image stitching algorithms based on image grayscale matching, are difficult to achieve good results in stitching infrared images of wind turbine blades, and cannot meet the requirements of panoramic stitching of infrared images of wind turbine blades. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device, storage medium and product for panoramic stitching of infrared images of wind turbine blades. Traditional visible light image registration and stitching algorithms are not effective in the field of infrared image stitching, resulting in the problem that panoramic stitching images of wind turbine blade infrared images cannot be obtained.

[0006] The present invention solves the above technical problems through the following technical solutions: a method for panoramic stitching of infrared images of wind turbine blades, comprising:

[0007] Acquire multiple frames of visible light images and infrared images of wind turbine blades; wherein each frame of the visible light image corresponds to the infrared image, and there is overlap between two adjacent frames of images;

[0008] Roughly registering the visible light image and the infrared image of each frame to obtain an infrared image roughly registered with the visible light image;

[0009] The background of the visible light image and the infrared image after rough registration is removed respectively to obtain the visible light foreground mask image and the infrared foreground mask image, and the visible light image and the infrared image of the leaf after the background is removed;

[0010] Precisely aligning the visible light foreground mask image and the infrared foreground mask image to obtain a relative displacement of the visible light foreground mask image and the infrared foreground mask image during precise alignment;

[0011] Splicing multiple frames of visible light images of leaves after background removal to obtain pixel increments when two adjacent frames of visible light images of leaves are spliced;

[0012] Calculate the pixel increment when stitching two adjacent frames of infrared images of the leaf according to the relative displacement and the pixel increment when stitching two adjacent frames of visible light images of the leaf;

[0013] According to the pixel increment when two adjacent frames of blade infrared images are stitched, the corresponding two adjacent frames of blade infrared images after background removal are stitched to obtain an infrared panoramic image of the wind turbine blade.

[0014] Furthermore, before obtaining multiple frames of visible light images and infrared images of wind turbine blades, the Zhang Zhengyou calibration method is used to perform single-target calibration on the visible light camera used to collect visible light images and the infrared thermal imaging camera used to collect infrared images, respectively, to obtain the intrinsic parameter matrix of the visible light camera, the intrinsic parameter matrix of the infrared thermal imaging camera, and the distortion correction parameters of the visible light image and the infrared image.

[0015] Furthermore, the specific implementation process of acquiring multiple frames of visible light images and infrared images of wind turbine blades is as follows:

[0016] Lock the wind turbine blade to be inspected;

[0017] Using a drone equipped with a visible light camera and an infrared thermal imaging camera, the drone is controlled to fly in a straight line along the direction of a single blade and parallel to the blade surface to collect images.

[0018] Furthermore, the visible light image and the infrared image of each frame are roughly registered, including:

[0019] Dedistorting the visible light image according to the calibrated intrinsic parameter matrix of the visible light camera and the distortion correction parameters of the visible light image;

[0020] Dedistorting the infrared image according to the calibrated intrinsic parameter matrix of the infrared thermal imaging camera and the distortion correction parameters of the infrared image;

[0021] The registration transformation matrix and the offset are calculated based on the visible light image and the infrared image after the dedistortion operation; wherein the calculation formula of the registration transformation matrix is:

[0022]

[0023] Among them, R C represents the registration transformation matrix, δ s Indicates the zoom ratio, and Respectively represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the checkerboard calibration plate in the visible light image, and Respectively represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the checkerboard calibration plate in the infrared image; the checkerboard calibration plate is located directly in front of the visible light camera and the infrared thermal imaging camera;

[0024] The calculation formula of the offset is:

[0025] x d =x vis -x inf ,y d =y vis -y inf ;

[0026] Among them, x d and d Indicates the horizontal and vertical offset of the infrared image, (x vis ,y vis )、(x inf ,y inf ) represent the pixel coordinates of the center of any square in the checkerboard calibration plate in the visible light image and the infrared image respectively;

[0027] The infrared image roughly registered with the visible light image is calculated according to the registration transformation matrix and the offset. The specific formula is:

[0028]

[0029] Among them, (Inf ix ,Inf iy ) represents the pixel coordinates of the infrared image after the dedistortion operation, (Inf ix ′,Inf′) represents the pixel coordinates of the infrared image roughly aligned with the visible light image.

[0030] Furthermore, background removal is performed on the coarsely registered visible light image and infrared image respectively, including:

[0031] Performing background segmentation on the coarsely registered visible light image and infrared image respectively to obtain a visible light foreground mask image and an infrared foreground mask image;

[0032] Performing edge contour smoothing operations on the visible light foreground mask image and the infrared foreground mask image respectively;

[0033] The RGB channel pixels of the foreground part of the visible light foreground mask image after the smoothing operation are set to 1 respectively, and then multiplied with the visible light image to obtain the visible light image of the leaf after the background is removed; the RGB channel pixels of the foreground part of the infrared foreground mask image after the smoothing operation are set to 1 respectively, and then multiplied with the infrared image after the coarse alignment to obtain the infrared image of the leaf after the background is removed.

[0034] Furthermore, the visible light foreground mask image and the infrared foreground mask image are precisely registered, including:

[0035] Perform edge detection on the visible light foreground mask image and the infrared foreground mask image respectively to obtain a leaf visible light edge image and a leaf infrared edge image;

[0036] Extracting boundaries of the visible light edge image of the leaf and the infrared edge image of the leaf respectively to obtain a first boundary coordinate list and a second boundary coordinate list; wherein the first boundary coordinate list is a boundary coordinate list of the visible light edge image of the leaf, and the second boundary coordinate list is a boundary coordinate list of the infrared edge image of the leaf;

[0037] Calculate a first width information list according to the first boundary coordinate list, and calculate a second width information list according to the second boundary coordinate list;

[0038] The relative displacement of the visible light foreground mask image and the infrared foreground mask image during precise registration is calculated according to the first width information list and the second width information list. The specific calculation formula is:

[0039] h k =y i ″,y i ″←min|w i ″-w′0|;

[0040] Among them, h k represents the relative displacement of the visible light foreground mask image and the infrared foreground mask image of the kth frame during precise registration; w0′ represents the top width in the first width information list; w i ″ represents the width of the i-th row in the second width information list; y i ″ represents the vertical coordinate of the i-th row in the second width information list.

[0041] Furthermore, the specific calculation formula for the pixel increment when stitching two adjacent frames of leaf infrared images is:

[0042] P k,k-1 =Q k,k-1 +(h k -h k-1 );

[0043] Among them, P k,k-1 represents the pixel increment when the infrared image of the leaf after the background is removed in the kth frame is spliced ​​with the infrared image of the leaf after the background is removed in the k-1th frame; Q k,k-1 represents the pixel increment when the visible light image of the leaf after the background is removed in the kth frame is spliced ​​with the visible light image of the leaf after the background is removed in the k-1th frame; h k represents the relative displacement of the visible light foreground mask image and the infrared foreground mask image of the kth frame during precise registration; h k-1 It represents the relative displacement of the visible light foreground mask image and the infrared foreground mask image of the k-1th frame during precise registration.

[0044] Based on the same concept, the present invention also provides an electronic device, including a memory, a processor, and a computer program / instruction stored in the memory, wherein the processor executes the computer program / instruction to implement the above-mentioned method for panoramic stitching of infrared images of wind turbine blades.

[0045] Based on the same concept, the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the panoramic stitching method of infrared images of wind turbine blades as described above is implemented.

[0046] Based on the same concept, the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned method for panoramic stitching of infrared images of wind turbine blades.

[0047] Beneficial Effects

[0048] Compared with the prior art, the advantages of the present invention are:

[0049] The present invention realizes the stitching of infrared images of wind turbine blades by means of registration operations between heterogeneous images and stitching of visible light images of wind turbine blades with richer grayscale feature information, thereby solving the problem that infrared images of wind turbine blades with low resolution, sparse features and blurred details are difficult to stitch. The present invention can effectively realize the stitching of infrared thermal images of wind turbine blades with sparser texture features, and the stitching results are stable and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 is a flow chart of a method for panoramic stitching of infrared images of wind turbine blades according to an embodiment of the present invention;

[0052] Figure 2 is a schematic diagram of collecting visible light images and infrared images in an embodiment of the present invention; wherein UAV represents an unmanned aerial vehicle;

[0053] Figure 3 is a schematic diagram of a coarse registration process in an embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of the effect of background removal of a visible light image in an embodiment of the present invention;

[0055] Figure 5 Schematic diagram of the effect of background removal of infrared images in an embodiment of the present invention;

[0056] Figure 6 is a schematic diagram of a precise registration process in an embodiment of the present invention;

[0057] Figure 7 is a schematic diagram of the effect of splicing visible light images in an embodiment of the present invention;

[0058] Figure 8 is a schematic diagram of the effect of splicing infrared images in an embodiment of the present invention;

[0059] Fig. 9 This is an infrared panoramic image of a wind turbine blade in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following is a clear and complete description of the technical solutions in the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0062] like Figure 1As shown, a method for panoramic stitching of infrared images of wind turbine blades provided by an embodiment of the present invention comprises the following steps:

[0063] Step 1: Calibration of visible light camera and infrared thermal imaging camera.

[0064] The visible light camera is used to collect visible light images of wind turbine blades, and the infrared thermal imaging camera is used to collect infrared images of wind turbine blades. Before collecting the dual-light images (visible light images and infrared images) of wind turbine blades, the Zhang Zhengyou calibration method is used to perform single-target calibration on the visible light camera and the infrared thermal imaging camera. The specific calibration process is carried out on OpenCV. The specific process is: take several chessboard calibration plate images from different angles, use the findChessboardCorners function to determine and search for chessboard corners in the image, and then use the cornerSubPix function to further determine the coordinates of each corner point of the chessboard calibration plate. Finally, call the calibrateCamera function to calculate the internal parameter matrix of the visible light camera and the infrared thermal imaging camera, as well as the distortion correction parameters of the visible light image and the infrared image.

[0065] Since the lens positions and focal lengths of the visible light camera and the infrared thermal imaging camera are different, the image sizes and positions of the space objects captured by the two lenses are also different. Therefore, a checkerboard calibration plate is placed in front of the two cameras, and it is ensured that both cameras can capture the complete calibration plate image when imaging. The calibration plate square image size and relative distance of the visible light camera and the infrared thermal imaging camera are obtained, so as to calculate the coordinate mapping relationship of the rough registration alignment of heterogeneous images, that is, the registration transformation matrix R C .

[0066] Step 2: Acquire multiple frames of visible light images and infrared images of wind turbine blades.

[0067] The present invention uses a drone equipped with a visible light camera and an infrared thermal imaging camera to collect visible light images and infrared images of wind turbine blades. Figure 2 As shown, before image acquisition, the wind turbine blade to be inspected is locked first, and then the drone is controlled to fly in a straight line along the direction of a single blade and parallel to the blade surface and acquire images. Images can be acquired sequentially from the root to the wingtip of the wind turbine blade, or from the wingtip to the root of the wind turbine blade. The visible light camera and the infrared thermal imaging camera acquire images simultaneously and at the same acquisition time interval, so each frame of the image includes a visible light image and an infrared image. The visible light image and infrared image of the same frame correspond to the same part of the wind turbine blade. In order to facilitate splicing and ensure the integrity of the spliced ​​image, there is overlap between two adjacent frames of images.

[0068] In order to simplify the trajectory control scheme of the drone, the wind turbine blade to be inspected is first locked in a vertical downward direction, and then the pitch angle of the drone is adjusted to keep the pitch angle of the drone at 0°. The drone is then controlled to fly in a straight line perpendicular to the ground and collect images.

[0069] Step 3: Roughly align the visible light image and infrared image of each frame to obtain the infrared image roughly aligned with the visible light image.

[0070] For each frame of visible light image and infrared image, the coarse registration of step 3, the background removal of step 4 and the fine registration of step 5 are performed. In a specific embodiment of the present invention, the coarse registration of each frame of visible light image and infrared image includes:

[0071] Step 3.1: Dedistort the visible light image according to the intrinsic parameter matrix of the visible light camera and the distortion correction parameters of the visible light image calibrated in step 1;

[0072] Step 3.2: According to the intrinsic parameter matrix of the infrared thermal imaging camera and the distortion correction parameters of the infrared image calibrated in step 1, the infrared image is dedistorted; the dedistortion operation can be implemented by calling the undistort function in OpenCV.

[0073] Step 3.3: Calculate the registration transformation matrix and offset based on the visible light image and infrared image after the dedistortion operation; the calculation formula of the registration transformation matrix is:

[0074]

[0075] Among them, R C represents the registration transformation matrix, δ s Indicates the zoom ratio, and Respectively represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the checkerboard calibration plate in the visible light image, and They represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the checkerboard calibration plate in the infrared image, respectively. Figure 3 As shown. Registration transformation matrix R C It is calculated based on the image size and relative distance of the checkerboard calibration plate in the visible light camera and the infrared thermal imaging camera. Before calculating the registration transformation matrix, the scaling ratio δ is calculated based on the pixel difference between the coordinates of any two adjacent black square centers in the checkerboard calibration plate in the visible light image and the infrared image. s .

[0076] There will be a certain relative offset between the visible light image and the infrared image, so it is also necessary to calculate the image offset based on the pixel coordinate position of the center of any square in the checkerboard calibration plate in the visible light image and the infrared image. The specific calculation formula of the offset is:

[0077] x d =x vis -x inf (3)

[0078] y d =y vis -y inf (4)

[0079] Among them, x d and d Indicates the horizontal and vertical offset of the infrared image, (x vis ,y vis )、(x inf ,y inf ) represent the pixel coordinates of the center of any square in the checkerboard calibration plate in the visible light image and infrared image, respectively.

[0080] Step 3.4: Calculate the infrared image roughly aligned with the visible light image based on the alignment transformation matrix and the offset, and achieve rough alignment of the visible light image with the infrared image.

[0081] In this embodiment, the specific formula for roughly registering the infrared image with the visible light image is:

[0082]

[0083] Among them, (Inf ix ,Inf iy ) represents the pixel coordinates of the infrared image after the dedistortion operation, (Inf ix ′,Inf′) represents the pixel coordinates of the infrared image roughly aligned with the visible light image.

[0084] Step 4: Perform background removal on the coarsely registered visible light image and infrared image respectively to obtain a visible light foreground mask image and an infrared foreground mask image, and a visible light image and infrared image of the leaf after background removal.

[0085] like Figure 4 and Figure 5 As shown, the wind turbine blade images captured by the visible light camera and the infrared thermal imaging camera contain a large amount of background information. The complex background information will affect the subsequent stitching results, and the background removal operation needs to be performed on the visible light image and the infrared image respectively. In a specific embodiment of the present invention, the background removal operation is performed on the visible light image and the infrared image after rough registration, including:

[0086] Step 4.1: Perform background segmentation on the coarsely registered visible light image and infrared image to obtain a visible light foreground mask image A and an infrared foreground mask image B;

[0087] Step 4.2: performing edge contour smoothing operations on the visible light foreground mask image A and the infrared foreground mask image B respectively;

[0088] Step 4.3: Set the RGB channel pixels of the foreground part of the visible light foreground mask image after the smoothing operation to 1 respectively, and then multiply it with the visible light image to keep the pixel values ​​of the leaf part in the visible light image original, so as to obtain the visible light image of the leaf after removing the background, as shown in Figure 4 As shown;

[0089] Step 4.4: Set the RGB channel pixels of the foreground part of the infrared foreground mask image after the smoothing operation to 1 respectively, and then multiply it with the infrared image after the rough registration, so that the pixel values ​​of the leaf part in the infrared image remain the original values, so as to obtain the infrared image of the leaf after the background is removed, as shown in Figure 5 shown.

[0090] In this embodiment, the U-net network is used to perform background segmentation on the visible light image and the infrared image after rough registration. The U-net network is an effective semantic segmentation framework. Before using the U-net network to perform background segmentation on the visible light image and the infrared image after rough registration, a training sample data set must be constructed to train the U-net network. The training sample data set includes multiple samples, each of which includes an infrared image after rough registration and its label or a visible light image and its label. The visible light image and the infrared image after rough registration are annotated using Labelme software to obtain pixel-level labels.

[0091] In order to make the edge contours of the visible light foreground mask image A and the infrared foreground mask image B obtained by background segmentation smoother and more natural, it is also necessary to eliminate the small gap between the visible light foreground mask image A and the image boundary, and set a 7×7 all-1 matrix as the convolution kernel of the closed operation, and perform a closed operation of first dilation and then corrosion on the visible light foreground mask image A to fill and connect the gaps at the edge of the image, so as to bridge the narrow discontinuity and slender gullies between the visible light foreground mask image A and the image boundary, so as to make the transition of the wind turbine blade boundary more natural. Similarly, the infrared foreground mask image B is also subjected to a closed operation of first dilation and then corrosion.

[0092] Step 5: Perform precise registration on the visible light foreground mask image and the infrared foreground mask image to obtain the relative displacement when the visible light foreground mask image and the infrared foreground mask image are precisely registered.

[0093] The precise registration of the dual light images of the present invention is achieved by searching and matching the blade width information in the visible light foreground mask image and the infrared foreground mask image. In a specific embodiment of the present invention, the precise registration of the visible light foreground mask image and the infrared foreground mask image includes:

[0094] Step 5.1: Perform edge detection on the visible light foreground mask image and the infrared foreground mask image respectively to obtain a leaf visible light edge image and a leaf infrared edge image.

[0095] In this embodiment, the Canny edge detection operator is used to extract the edges of the visible light foreground mask image and the infrared foreground mask image, which specifically includes: after smoothing the image using Gaussian filtering, the gradient amplitude and direction of the image pixels are calculated. Specifically, the Sobel level operator S x and the vertical operator S y Convolve with the input image (i.e., visible light foreground mask image or infrared foreground mask image) respectively to calculate the horizontal and vertical gradient amplitude E of each pixel in the image x and E y , and further obtain the gradient amplitude E and direction θ of each pixel in the image. The specific formula is:

[0096]

[0097] E x =f(x,y)*S x (8)

[0098] E y =f(x,y)*S y (9)

[0099]

[0100] θ=arctan(E y / E x ) (11)

[0101] Among them, f(x,y) represents the image area covered by the convolution kernel centered at point (x,y) when convolution calculates the gradient of each pixel.

[0102] The non-maximum suppression method is applied to the image obtained by edge detection to filter out non-edge pixels, and then double threshold detection is performed to remove pseudo edges. Finally, the edges are connected to obtain a complete leaf edge image, that is, the leaf visible light edge image A' and the leaf infrared edge image B' are obtained respectively.

[0103] Step 5.2: Perform boundary extraction on the visible light edge image of the leaf and the infrared edge image of the leaf respectively to obtain a first boundary coordinate list and a second boundary coordinate list; wherein the first boundary coordinate list is the boundary coordinate list of the visible light edge image of the leaf, and the second boundary coordinate list is the boundary coordinate list of the infrared edge image of the leaf.

[0104] For the visible light edge image A' of the leaf, take the upper left corner of the image as the origin, the horizontal direction as the X-axis, and the vertical direction as the Y-axis, traverse the image to obtain points whose pixel values ​​are not 0, and save the coordinates of these points and the corresponding pixel values ​​in the list; judge the coordinates of the points in the list, if the distance between any two points in each row of the list is less than the pixel threshold (for example, 3 pixels), the two points are considered to come from the same boundary, otherwise the two points are considered to belong to different boundaries, and the two points are the left and right edge points of the corresponding rows, thereby obtaining all the points on the left and right edges of each row of the leaf; for each row, take the coordinates corresponding to the point with the largest pixel value among all the points on the left and right edges of the leaf, and obtain the first boundary coordinate list, which is recorded as:

[0105] L1=[[y0,x 0l ,x 0r ],[y1,x 1l ,x 1r ],...,[y i ,x il ,x ir ],...,[y N ,x Nl ,x Nr ]] (12)

[0106] Where L1 represents the first boundary coordinate list, x il Indicates the horizontal coordinate of the point with the largest pixel value on the left edge of the i-th row of leaves, x ir Indicates the horizontal coordinate of the point with the largest pixel value on the right edge of the i-th row of leaves, y i Represents x il and x ir The corresponding ordinates (i.e. the ordinates of the points with the largest pixel values ​​on the left edge of the i-th row and the points with the largest pixel values ​​on the right edge of the i-th row), N represents the number of the last row, with a total of N+1 rows. Points with the same ordinate in the image form a row, that is, the ordinates of the pixels in each row are the same and not 0.

[0107] Similarly, for the leaf infrared edge image B', a second boundary coordinate list L2 can be obtained.

[0108] Step 5.3: Calculate a first width information list according to the first boundary coordinate list, and calculate a second width information list according to the second boundary coordinate list.

[0109] According to the first boundary coordinate list L1 or the second boundary coordinate list L2, the width calculation formula w is used i =x ir -x il Calculate the leaf width of each row, thereby obtaining the first width information list L vis =[[y0′,w0′],[y1′,w1′]...[y i ′,w i ′]...[y′ N ,w′ N ]] and the second width information list L inf =[[y′0′,w0″],[y1″,w1″]...[y i ″,w i ″]...[y′ N ′,w′ N ′]], where y i ′、y i ″ respectively represent the ordinate of the i-th row in the leaf visible light edge image A' and the leaf infrared edge image B', w i ′、w i ″ respectively represent the width of the i-th row in the leaf visible light edge image A' and the leaf infrared edge image B'.

[0110] Step 5.4: Calculate the relative displacement of the visible light foreground mask image A and the infrared foreground mask image B during precise registration according to the first width information list and the second width information list.

[0111] like Figure 6 As shown, the first width information list L vis The top width (i.e., the width of the first row, the vertical coordinate of the first row is 0) w0′ is used as a reference to calculate the width of the second width information list L inf Traverse and obtain the ordinate of the row of the leaf infrared edge image B' that is closest to w0'. The ordinate of this row is the relative displacement of the leaf infrared edge image B' relative to the leaf visible light edge image A' during precise registration. The specific calculation formula is:

[0112] h k =y i ″, y i ″←min|w i ″-w′0| (13)

[0113] Among them, h k represents the relative displacement of the visible light foreground mask image and the infrared foreground mask image of the kth frame during precise registration; w0′ represents the width of the topmost (i.e., the first row) in the first width information list; w i ″ represents the width of the i-th row in the second width information list. The width of the row is wi ″ is closest to w0′; y i ″ represents the ordinate of the i-th row in the second width information list, i.e., the ordinate of the i-th row in the second width information list corresponding to the width w i ” The vertical coordinate of the row corresponding to .

[0114] For each frame of infrared foreground mask image (or each frame of infrared image), there is a relative displacement h k .

[0115] Step 6: stitch multiple frames of visible light images of leaves after background removal to obtain pixel increments when stitching two adjacent frames of visible light images of leaves.

[0116] In a specific embodiment of the present invention, the NCC (Normalized Cross-Correlation) algorithm based on image grayscale is used to splice multiple frames of visible light images of leaves after background removal. The NCC (Normalized Cross-Correlation) algorithm based on image grayscale is a prior art, and its principle is: match two adjacent frames of images to obtain the best matching position, and then splice and synthesize the two adjacent frames of images according to the best matching position. The NCC algorithm obtains the similarity between the template image and the image to be searched at different positions by using an evaluation function, and the position with the greatest similarity is the best matching position. The similarity calculation formula is:

[0117]

[0118] Among them, T(m,n) represents the template graph, S i,j It represents the sub-image covered by the template image in the image to be searched, m and n represent the width and height of the template image respectively. and Represent the grayscale mean of the template image and the sub-image respectively.

[0119] The NCC algorithm based on image grayscale takes a long time to stitch and has high requirements on image contrast, so the improved NCC algorithm based on boundary search is adopted. Figure 7 As shown in the figure, the specific implementation process of the improved NCC algorithm using boundary search is:

[0120] The visible light image of the leaves after background removal is preprocessed by histogram equalization (for details, see the reference: Wang Chunzhi, Niu Hongxia. Dust degradation image enhancement algorithm based on histogram equalization and MSRCR [J]. Computer Engineering, 2022, 48(09): 223-229. DOI: 10.19678 / j.issn.1000-3428.0062764.) to improve the image contrast;

[0121] For the splicing of two adjacent frames of images, the latter frame of image is used as the image to be searched, and a portion of the image is cut out from the overlapping part of the previous frame of image as the template image (such as Figure 7 Waiting for splicing Figure 1 The blue box), using downsampling method to treat the search image and template image respectively to construct a two-layer pyramid sequence;

[0122] According to the first boundary coordinate list, the top image of the pyramid is matched first; when matching, the right edge of the leaf in the top image of the search image is taken as the center, and the left and right are expanded by 10 pixels to form a matching area. The top image of the template image is also based on the right edge of the leaf, and the correct matching position is traversed and searched in the matching area. Since the size of the top image of the pyramid is 1 / 4 of the original image, the amount of calculation can be reduced during matching;

[0123] After completing the top-level image matching to determine the initial best matching position, the next layer of the pyramid image is matched, and a 6×6 pixel rectangular range is expanded with the initial best matching position as the center in the image to be searched. Similarly, the right edge of the leaf is used as the center for traversal and search, and precise matching is performed to obtain the best matching position. When traversing and searching for matching, the normalized cross-correlation calculation is performed along the leaf edge, which can improve the matching speed;

[0124] Finally, the pixel increment Q of two adjacent frames is obtained according to the best matching position. k,k-1 (ie Q), the previous frame image (ie the k-1th frame) is placed at the pixel increment Q position of the next frame image (ie the kth frame) to achieve the splicing of two adjacent frames of leaf visible light images.

[0125] Step 7: Calculate the pixel increment when stitching two adjacent frames of infrared images of the leaves according to the relative displacement and the pixel increment when stitching two adjacent frames of visible light images of the leaves.

[0126] In this embodiment, the specific calculation formula for the pixel increment when two adjacent frames of leaf infrared images are stitched is:

[0127] P k,k-1 =Q k,k-1 +(h k -h k-1 ) (15)

[0128] Among them, P k,k-1 represents the pixel increment when the infrared image of the leaf after the background is removed in the kth frame is spliced ​​with the infrared image of the leaf after the background is removed in the k-1th frame; Q k,k-1 represents the pixel increment when the visible light image of the leaf after the background is removed in the kth frame is spliced ​​with the visible light image of the leaf after the background is removed in the k-1th frame; h k represents the relative displacement of the visible light foreground mask image and the infrared foreground mask image of the kth frame during precise registration; hk-1 It represents the relative displacement of the visible light foreground mask image and the infrared foreground mask image of the k-1th frame during precise registration.

[0129] Step 8: According to the pixel increment when stitching two adjacent frames of blade infrared images, stitch the corresponding two adjacent frames of blade infrared images after removing the background to obtain an infrared panoramic image of the wind turbine blade.

[0130] like Figure 8 As shown in the figure, according to the pixel increment P when stitching two adjacent frames of leaf infrared images k,k-1 The pixel increment P of the infrared image of the leaf after removing the background in the k-1th frame is placed on the infrared image of the leaf after removing the background in the kth frame. k,k-1 Position, to achieve the stitching of the k-1th frame and the kth frame of the leaf infrared image. The stitching result is stitched with the next frame (i.e., the k+1th frame) of the leaf infrared image using the same method, that is, the leaf visible light image after removing the background of the k+1th frame is used as the image to be searched, and the corresponding visible light stitching result (i.e., the stitching result of the leaf visible light image of the k-1th frame and the kth frame) is used as the template image for stitching again, and the corresponding pixel increment P is matched and mapped to calculate, and the leaf infrared image is stitched according to the pixel increment P, and steps 6 to 8 are executed repeatedly to obtain the following: Fig. 9 The infrared panoramic image of wind turbine blades is shown.

[0131] In another specific embodiment of the present invention, the pixel increment of each splicing of the visible light images of the leaves can also be obtained first, that is, the pixel increment when the first frame is spliced ​​with the second frame, the pixel increment when the splicing result of the first frame and the second frame is spliced ​​with the third frame, the pixel increment when the previous splicing result (that is, the splicing result of the first frame and the second frame and the result of splicing the third frame) is spliced ​​with the fourth frame, ...; then the pixel increment when the corresponding leaf infrared image is spliced ​​is calculated according to step 7; finally, the corresponding two frames of leaf infrared images are spliced ​​according to step 8.

[0132] Example 2

[0133] An embodiment of the present invention also provides an electronic device, which includes: a memory, a processor, and a computer program / instructions stored in the memory, and the processor executes the computer program / instructions to implement the panoramic stitching method of infrared images of wind turbine blades in the embodiment of the present application.

[0134] Although not shown, the electronic device includes a processor, which can perform various appropriate operations and processes according to the program and / or data stored in the read-only memory (ROM) or the program and / or data loaded from the storage portion into the random access memory (RAM). The processor can be a multi-core processor, or it can include multiple processors. In some embodiments, the processor can include a general main processor and one or more special coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM, various programs and data required for device operation are also stored. The processor, ROM and RAM are connected to each other via a bus. The input / output (I / O) interface is also connected to the bus.

[0135] The processor and the memory are used together to execute the program / instructions stored in the memory. When the program / instructions are executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0136] Although not shown, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method for panoramic stitching of infrared images of wind turbine blades in the embodiment of the present application is implemented.

[0137] Readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0138] Although not shown, an embodiment of the present invention further provides a computer program product, including: a computer program / instruction, which, when executed by a processor, implements the method for panoramic stitching of infrared images of wind turbine blades in the embodiment of the present application.

[0139] What is disclosed above is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or modifications within the technical scope disclosed in the present invention, which should be covered within the protection scope of the present invention.

Claims

1. A method for panoramic stitching of infrared images of wind turbine blades, characterized in that: The splicing method comprises: Acquire multiple frames of visible light images and infrared images of wind turbine blades; wherein each frame of the visible light image corresponds to the infrared image, and there is overlap between two adjacent frames of images; Roughly registering the visible light image and the infrared image of each frame to obtain an infrared image roughly registered with the visible light image; The background of the visible light image and the infrared image after rough registration is removed respectively to obtain the visible light foreground mask image and the infrared foreground mask image, and the visible light image and the infrared image of the leaf after the background is removed; Precisely aligning the visible light foreground mask image and the infrared foreground mask image to obtain a relative displacement of the visible light foreground mask image and the infrared foreground mask image during precise alignment; Splicing multiple frames of visible light images of leaves after background removal to obtain pixel increments when two adjacent frames of visible light images of leaves are spliced; Calculate the pixel increment when stitching two adjacent frames of infrared images of the leaf according to the relative displacement and the pixel increment when stitching two adjacent frames of visible light images of the leaf; According to the pixel increment when two adjacent frames of blade infrared images are stitched, the corresponding two adjacent frames of blade infrared images after background removal are stitched to obtain an infrared panoramic image of the wind turbine blade.

2. The method for panoramic stitching of infrared images of wind turbine blades according to claim 1 is characterized in that: Before obtaining multi-frame visible light images and infrared images of wind turbine blades, the Zhang Zhengyou calibration method is used to perform single-target calibration on the visible light camera used to collect visible light images and the infrared thermal imaging camera used to collect infrared images, respectively, to obtain the intrinsic parameter matrix of the visible light camera, the intrinsic parameter matrix of the infrared thermal imaging camera, and the distortion correction parameters of the visible light image and the infrared image.

3. The method for panoramic stitching of infrared images of wind turbine blades according to claim 1, characterized in that: The specific implementation process of acquiring multiple frames of visible light images and infrared images of wind turbine blades is as follows: Lock the wind turbine blade to be inspected; Using a drone equipped with a visible light camera and an infrared thermal imaging camera, the drone is controlled to fly in a straight line along the direction of a single blade and parallel to the blade surface to collect images.

4. The method for panoramic stitching of infrared images of wind turbine blades according to claim 1, characterized in that: Perform coarse registration of visible light image and infrared image of each frame, including: Dedistorting the visible light image according to the calibrated intrinsic parameter matrix of the visible light camera and the distortion correction parameters of the visible light image; Dedistorting the infrared image according to the calibrated intrinsic parameter matrix of the infrared thermal imaging camera and the distortion correction parameters of the infrared image; The registration transformation matrix and the offset are calculated based on the visible light image and the infrared image after the dedistortion operation; wherein the calculation formula of the registration transformation matrix is: Among them, R C represents the registration transformation matrix, δ s Indicates the zoom ratio, and Respectively represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the checkerboard calibration plate in the visible light image, and Respectively represent the horizontal or vertical coordinates of the centers of two adjacent black squares i and i-1 in the checkerboard calibration plate in the infrared image; the checkerboard calibration plate is located directly in front of the visible light camera and the infrared thermal imaging camera; The calculation formula of the offset is: x d =x vis -x inf ,and d =and vis -and inf ; Among them, x d and d Indicates the horizontal and vertical offset of the infrared image, (x vis ,y vis )、(x inf ,y inf ) represent the pixel coordinates of the center of any square in the checkerboard calibration plate in the visible light image and the infrared image respectively; The infrared image roughly registered with the visible light image is calculated according to the registration transformation matrix and the offset. The specific formula is: Among them, (Inf ix ,Inf iy ) represents the pixel coordinates of the infrared image after the dedistortion operation, (Inf ix ′,Inf′) represents the pixel coordinates of the infrared image roughly aligned with the visible light image.

5. The method for panoramic stitching of infrared images of wind turbine blades according to claim 1, characterized in that: The background of the coarsely registered visible light image and infrared image is removed respectively, including: Performing background segmentation on the coarsely registered visible light image and infrared image respectively to obtain a visible light foreground mask image and an infrared foreground mask image; Performing edge contour smoothing operations on the visible light foreground mask image and the infrared foreground mask image respectively; The RGB channel pixels of the foreground part of the visible light foreground mask image after the smoothing operation are set to 1 respectively, and then multiplied with the visible light image to obtain the visible light image of the leaf after the background is removed; the RGB channel pixels of the foreground part of the infrared foreground mask image after the smoothing operation are set to 1 respectively, and then multiplied with the infrared image after the coarse alignment to obtain the infrared image of the leaf after the background is removed.

6. The method for panoramic stitching of infrared images of wind turbine blades according to claim 1, characterized in that: Perform precise registration of the visible light foreground mask image and the infrared foreground mask image, including: Perform edge detection on the visible light foreground mask image and the infrared foreground mask image respectively to obtain a leaf visible light edge image and a leaf infrared edge image; Extracting boundaries of the visible light edge image of the leaf and the infrared edge image of the leaf respectively to obtain a first boundary coordinate list and a second boundary coordinate list; wherein the first boundary coordinate list is a boundary coordinate list of the visible light edge image of the leaf, and the second boundary coordinate list is a boundary coordinate list of the infrared edge image of the leaf; Calculate a first width information list according to the first boundary coordinate list, and calculate a second width information list according to the second boundary coordinate list; The relative displacement of the visible light foreground mask image and the infrared foreground mask image during precise registration is calculated according to the first width information list and the second width information list. The specific calculation formula is: h k =y i ″,y i ″←min|w i ″-w′0|; Among them, h k represents the relative displacement of the visible light foreground mask image and the infrared foreground mask image of the kth frame during precise registration; w0′ represents the top width in the first width information list; w i ″ represents the width of the i-th row in the second width information list; y i ″ represents the vertical coordinate of the i-th row in the second width information list.

7. The method for panoramic stitching of infrared images of wind turbine blades according to any one of claims 1 to 6, characterized in that: The specific calculation formula for the pixel increment when stitching two adjacent frames of leaf infrared images is: P k,k-1 =Q k,k-1 +(h k -h k-1 ); Among them, P k,k-1 represents the pixel increment when the infrared image of the leaf after the background is removed in the kth frame is spliced ​​with the infrared image of the leaf after the background is removed in the k-1th frame; Q k,k-1 represents the pixel increment when the visible light image of the leaf after the background is removed in the kth frame is spliced ​​with the visible light image of the leaf after the background is removed in the k-1th frame; h k represents the relative displacement of the visible light foreground mask image and the infrared foreground mask image of the kth frame during precise registration; h k-1 It represents the relative displacement of the visible light foreground mask image and the infrared foreground mask image of the k-1th frame during precise registration.

8. An electronic device comprising a memory, a processor, and a computer program / instruction stored in the memory, characterized in that: The processor executes the computer program / instructions to implement the method for panoramic stitching of infrared images of wind turbine blades according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the method for panoramic stitching of infrared images of wind turbine blades as claimed in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for panoramic stitching of infrared images of wind turbine blades as claimed in any one of claims 1 to 7 is implemented.

Images