Tunnel linear array image adjacent sheet fusion method and system
By calculating the pixel energy field in tunnel detection and solving the global optimal stitching path, using a fusion method based on distance weights, the blur and ghosting problems during image fusion in tunnel detection are solved, and a seamless image stitching effect is achieved.
Patent Information
- Application Number
- CN202411141429.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the tunnel detection, due to the limited field of view and resolution of a single linear array camera, it is difficult to achieve complete identification and detection of large-area disease defects or equipment, and the difference in shooting angles of adjacent cameras leads to blur or ghosts when the image is fusion. Visual defects such as film.
A tunnel line array image adjacency image fusion method is proposed. By calculating the pixel energy field and solving the global optimal splicing path as the best suture, the two images are processed using the fusion method based on the distance weight to generate seamless splicing Fusion image.
It effectively eliminates the problems of fusion images blur or ghosting, retains the original image content, and achieves a seamless fusion effect.
Smart Images

Figure CN119027326A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of image processing, and in particular to a method for fusing adjacent images of tunnel linear array images. Background Art
[0002] For tunnel inspection tasks, a single linear array camera can only cover a portion of the tunnel's inner surface due to the limitations of field of view and resolution. This single-view, local field of view imaging mode makes it difficult to fully identify and detect large-area distributed defects or important facilities and equipment. In order to obtain panoramic image data of the entire inner surface of the tunnel and meet the needs of large-scale defect detection and equipment identification, it is necessary to fuse a group of images taken by multiple linear array cameras at the same time node to generate a complete orthophoto of the tunnel's inner wall.
[0003] After registering adjacent linear array images, how to process the pixels in the overlapping area is an urgent problem to be solved. In tunnel inspection tasks, the shooting angles of adjacent cameras will be quite different, so the same object under inspection will appear at completely different angles in the two images.
[0004] The simplest approach is to directly take the arithmetic average of the pixel values at the same position of all overlapping images. However, this simple linear averaging method cannot produce ideal fusion effects. It ignores the intrinsic structural information of the image, such as edges and textures, and can easily produce unnatural aliasing effects in these areas. At the same time, for uneven exposure or dynamic scenes, when there is a large difference in brightness or motion between the two images, simple linear averaging will cause visual defects such as image blur or ghosting. Summary of the invention
[0005] Purpose of the invention: To propose a method for fusing adjacent images of tunnel linear array images, and further propose a system for implementing the above method, so as to solve the above problems existing in the prior art.
[0006] The present invention proposes a method for fusing adjacent images of tunnel linear array images, and the steps are as follows:
[0007] Acquire a target image, where the target image is an image captured by two adjacent linear array cameras in a tunnel; perform a preprocessing operation on the target image to obtain two images to be fused;
[0008] Calculate the pixel energy field of the image to be fused;
[0009] Based on the pixel energy field, a global optimal stitching path of the images to be fused is solved, and the global optimal stitching path is used as an optimal stitching line;
[0010] Based on the optimal stitching line, the two images to be fused are processed using a distance weighted fusion method to complete the fusion of adjacent images of the tunnel linear array image.
[0011] In a further embodiment, pre-processing operations are performed on the target image, including direction correction, tilt correction, orthorectification, and image registration.
[0012] In a further embodiment, the process of calculating the pixel energy field of the image to be fused specifically includes:
[0013] Define the energy function to calculate the energy of the pixels in the overlapping area of the two images to be fused. The energy function expression is as follows:
[0014]
[0015] Where (x, y) is the pixel coordinate; E c (x, y) represents the color difference between the overlapping pixels of the two images to be fused; E g (x, y) represents the gradient difference between the overlapping pixels of the two images to be fused; α is the weight for controlling the color difference and gradient difference;
[0016] Analyze the pixel gradient information and pixel color information of the two images to be fused, and assign different crossing costs to different pixels. The crossing costs meet the following conditions:
[0017] The gradient value of the pixel point is positively correlated with the assigned crossing cost, and the color difference value of the pixel point is positively correlated with the assigned crossing cost.
[0018] In a further embodiment, the color difference E between the overlapping pixels of the two images to be fused is c (x,y) is calculated as follows:
[0019] E c (x,y)=I1(x,y)-I2(x,y)
[0020] Gradient Difference E g The calculation is as follows:
[0021] E g =G x (I1,I2)·G y (I1,I2)
[0022] In the formula, G x , G y They represent the gradient values in the horizontal and vertical directions respectively; I1 and I2 represent the pixel matrices of the two images respectively. The color difference can be obtained by subtracting the two matrices.
[0023] In a further embodiment, based on the pixel energy field, solving the global optimal stitching path of the image to be fused specifically includes:
[0024] Define the function M(x,y) to represent the horizontal seam with the lowest energy starting from the left side of the image to be fused and ending at pixel P(x,y);
[0025] At the far left of the image to be fused, all seams ending with these pixels are only one pixel long, so the lowest energy seam ending with these pixels is simply the energy of these pixels:
[0026] M(0,y)=E(0,y)
[0027] For all other pixels, look at the pixels in the previous column; take the lowest energy of the seams that end with those pixels, and add that to the energy of the current pixel:
[0028]
[0029] When the pixel being viewed is at the boundary of the upper edge or lower edge of the image to be fused, M(x-1,y+1) or M(x-1,y+1) is ignored respectively;
[0030] Finally, the energy of the lowest energy seam extending horizontally across the entire image to be fused is obtained, expressed as a two-dimensional cumulative cost matrix;
[0031] From left to right and from top to bottom, calculate the minimum cumulative cost to reach each pixel position:
[0032] min(M(W-1,y)),0≤y≤H-1
[0033] Where W and H are the width and height of the image respectively;
[0034] Select the minimum energy pixel M(W-1,y W-1 ), trace back the best seam pixel position of the previous column from right to left, and the calculation formula is as follows:
[0035]
[0036] Finally, a global optimal splicing path with the minimum total cost is obtained; the global optimal splicing path is used as the best stitching line.
[0037] In a further embodiment, the two images to be fused are processed using a distance weighted fusion method, specifically including:
[0038] A region with a width of D around the optimal stitching line is set for fusion, and the goal is to generate a seamlessly stitched fusion image I mosaic; For any position P in the overlapping area mosaic (x, y), calculate its distance d(x, y) to the edge of the overlapping area, and take the pixel values of the two images at this position as P1(x, y) and P2(x, y), and use the following formula for weighted fusion:
[0039]
[0040] When d(x,y)=0, point P mosaic (x, y) is located at the edge of the overlapping area, and the fusion result is completely taken from the image on the adjacent side;
[0041] when When mosaic (x,y) is located at the center of the overlapping area, and the contributions of the two images are used equally.
[0042] In a second aspect, the present invention provides a tunnel linear image adjacent image fusion system, which can automatically execute the tunnel linear image adjacent image fusion method disclosed in the first aspect. The fusion system includes a preprocessing unit, a calculation unit, a splicing path generation unit, and a fusion unit.
[0043] The preprocessing unit is used to obtain a target image photographed by two adjacent linear array cameras in a tunnel, and perform a preprocessing operation on the target image to obtain two images to be fused.
[0044] The calculation unit is used to calculate the pixel energy field of the image to be fused.
[0045] The stitching path generation unit is used to solve the global optimal stitching path of the to-be-fused images based on the pixel energy field, and use the global optimal stitching path as the best stitching line.
[0046] The fusion unit processes the two images to be fused based on the optimal stitching line to complete the fusion of adjacent images of the tunnel linear array.
[0047] In a third aspect, the present invention provides an electronic device, comprising a processor and a memory storing computer program instructions. When the processor executes the computer program instructions, the method for fusing adjacent images of tunnel linear array images disclosed in the first aspect is implemented.
[0048] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium. When the executable instruction is executed on an electronic device, the electronic device executes the method for fusing adjacent images of tunnel linear array images as disclosed in the first aspect.
[0049] Beneficial effect: The present invention calculates an optimal path with minimized energy in the overlapping area as the stitching seam of the two images, and uses a distance weighted fusion algorithm to fuse and stitch the two images, thereby effectively eliminating problems such as blurring or ghosting of the fused image, and achieving a seamless fusion effect while retaining the original image content. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 This is a flow chart of the method for fusing adjacent frames of tunnel linear array images.
[0051] Figure 2 2 is an example of two images to be fused in the embodiment.
[0052] Figure 3 It is the horizontal and vertical gradient difference map of the image to be fused in the embodiment.
[0053] Figure 4 Graph showing the color difference between two images in the overlapping area in the embodiment.
[0054] Figure 5 It is a pixel energy field prediction diagram in the embodiment.
[0055] Figure 6 is the cumulative cost map of the fused area in the embodiment.
[0056] Figure 7 It is the global optimal suture path graph solved on the cumulative cost graph in the embodiment. DETAILED DESCRIPTION
[0057] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.
[0058] This embodiment discloses a method for fusing adjacent images of tunnel linear array images. The process is shown in FIG. Figure 1 This method is aimed at fusing images taken by linear array cameras in tunnels. The images between two adjacent cameras will overlap. Before using this solution for fusion, a series of processing procedures such as direction, tilt, orthorectification, and image registration have been carried out (this process adopts the common methods in this field and will not be described here). This solution only targets the overlapping part of the two images and mainly includes three steps:
[0059] 1. Calculate pixel energy field
[0060] First, we need to define an energy function to quantify the visual cost of crossing each pixel, so that the final stitching line path can avoid the key structural areas in the image and ensure the stitching quality. The most common energy function is based on pixel gradient. By analyzing the pixel gradient information, pixels with larger gradient values are assigned higher crossing costs. In this way, the optimal stitching path obtained will naturally avoid the edges of objects and high-contrast areas, thereby retaining the main structural features of the image. At the same time, the color consistency between the two images should also be considered. Regions with large color differences are assigned higher crossing costs, so that the stitching line bypasses these areas, which can greatly reduce the seam marks caused by exposure differences or white balance imbalances. The energy E(x,y) of any pixel in the overlapping area can be calculated using formula (1):
[0061]
[0062] Where E c Indicates the color difference between the overlapping pixels of the two images, E g Represents the gradient difference, and α controls the weight of the two differences. This paper uses the Sobel operator to calculate the gradient, and other edge detection operators such as Roberts and Prewitt can also be considered. x and G y Represents the gradient values in the horizontal and vertical directions respectively.
[0063] The Sobel operator is an edge detection operator based on the first-order derivative. It introduces an operation similar to the local average, which has a smoothing effect on noise and can effectively eliminate the influence of noise. At the same time, the Sobel operator weights the influence of the pixel position, so it is more effective than the Prewitt operator and the Roberts operator. Formula (2) describes the Sobel operator template in the horizontal and vertical directions:
[0064]
[0065] Figure 2 The overlapping area of the two images to be fused is shown. Due to differences in imaging conditions, such as shooting angle and uneven exposure, there is a certain degree of parallax and color difference between the two images in the overlapping area. Figure 3 (a) and (b) respectively show the horizontal and vertical gradient difference maps calculated using the Sobel operator. Figure 4 The color difference of the two images in the overlapping area is shown. Linear or nonlinear fusion of the above multiple energy maps can obtain a comprehensive pixel energy field prediction, such as Figure 5 This energy map intuitively reflects the "travel cost" of each pixel position, and the brighter area represents a higher energy cost.
[0066] 2. Solving the Energy Minimization Path
[0067] Taking into account the computational cost and processing efficiency, the present invention adopts the idea of dynamic programming and can obtain the global optimal solution with lower time complexity.
[0068] First, define the function M(x,y) to represent the lowest energy horizontal seam starting from the left side of the image and ending at pixel P(x,y). On the far left side of the image, all seams ending with these pixels are only one pixel long, so the lowest energy seam ending with these pixels is the energy of these pixels:
[0069] M(0,y)=E(0,y)
[0070] For all other pixels, we need to look at the pixels in the previous column. Since the seams need to be connected, the candidates are only the top left, left, and bottom left pixels. We need to select the seam with the lowest energy that ends with these pixels, and then add the energy of the current pixel:
[0071]
[0072] When the pixel being viewed is at the boundary of the upper or lower edge of the image, M(x-1, y+1) or M(x-1, y+1) is ignored for the pixels at the upper or lower edge, respectively.
[0073] Finally, we get the energy of the lowest energy seam that extends horizontally across the entire image, which is a two-dimensional cumulative cost matrix. Just look at the pixels in the rightmost column of the image to find the lowest energy seam that ends at one of these pixels. Let the image be W wide and H high, and the minimum energy pixel in the last column is:
[0074] min(M(W-1,y)),0≤y≤H-1
[0075] First, a two-dimensional cumulative cost matrix is constructed. From left to right and from top to bottom, the minimum cumulative cost to reach each pixel position is calculated, and the optimal predecessor pixel is stored. After the calculation is completed, the rightmost column stores the global minimum cost value to reach each pixel position in the column. Simply select the pixel position with the minimum cost and trace back to its optimal predecessor in sequence to obtain a global optimal splicing path with the minimum total cost.
[0076] From the above process of calculating the two-dimensional cumulative cost matrix, we can see that each column depends only on the previous column. For a single pixel, the three nearest pixels are the upper left, left, and lower left pixels of the previous column. Therefore, for the minimum energy pixel M(W-1,y W-1 ), the best seam pixel position of the previous column can be traced back using the following formula:
[0077]
[0078] By performing the above process from right to left in the cumulative cost matrix, a global optimal splicing path with the minimum total cost can be obtained.
[0079] like Figure 6 The figure shows the cumulative cost map of the fused area, which is the key intermediate result of the dynamic programming solution for the optimal stitching path. Different colors are used to intuitively reflect the cumulative cost value of each position. It can be seen that the cost value is higher in the area where the two images are very different, showing a dark red color. At the same time, there are some "flowing" cumulative costs formed around the contours of the object, which reflects that the path naturally follows the surface of the object and preserves the overall shape and integrity of the object.
[0080] Through the dynamic programming algorithm introduced above, a globally optimal suture path can be solved on the cumulative cost graph of the overlapping area. Figure 7 As shown in the figure, the automatically calculated optimal stitching line is not a simple straight line, but it cleverly bypasses the areas that will produce parallax, texture and brightness mutations in the image, and adaptively selects a route with the least visual impact. This path solving strategy based on energy minimization allows the stitching line to closely follow the geometric and structural features of the image, effectively avoiding crossing objects or high-contrast areas, and greatly reducing the visual distortion and distortion caused by fusion.
[0081] 3. Influence Integration
[0082] Based on the best seam calculated above, the two images are processed using a distance-weighted fusion method. A weight distribution function is calculated for each pair of overlapping images, which is determined by the distance of the pixels in the spatial domain. Near the edge of the overlapping area, the pixel value of the image on the adjacent side should be trusted more, while in the center of the overlapping area, the contribution from the images on both sides should be considered equally. In this way, the overlapping area can be smoothly transitioned, and the obvious seam edges can be effectively avoided, while the details and quality of the original image can be retained to the maximum extent.
[0083] Assume that the two images to be fused are I1 and I2, and an area with a width of D around the optimal stitching line is artificially set for fusion. The goal is to generate a seamlessly stitched fused image I mosaic For any position P in the overlapping area mosaic (x, y), calculate its distance d(x, y) to the edge of the overlapping area, and take the pixel values of I1 and I2 at this position as P1(x, y) and P2(x, y), and use formula (3) for weighted fusion:
[0084]
[0085] When d(x,y)=0, point P mosaic When (x, y) is located at the edge of the overlapping area, the fusion result is completely taken from the image on the adjacent side. When mosaic (x, y) is located at the center of the overlapping area, and the contributions of the images on both sides are equally used. The final fused image is as follows Figure 7 shown.
[0086] In practical applications, the logic of the method for fusion of adjacent images of tunnel linear array images disclosed in the above embodiment can be implemented as a system. The system can be composed of a preprocessing unit, a calculation unit, a stitching path generation unit, and a fusion unit. The preprocessing unit is used to obtain the target image taken by two adjacent linear array cameras in the tunnel, and perform preprocessing operations on the target image to obtain two images to be fused. The calculation unit is used to calculate the pixel energy field of the image to be fused. The stitching path generation unit is used to solve the global optimal stitching path of the image to be fused based on the pixel energy field, and use the global optimal stitching path as the optimal stitching line. The fusion unit processes the two images to be fused based on the optimal stitching line to complete the fusion of adjacent images of the tunnel linear array images. When the system is run, the method for fusion of adjacent images of tunnel linear array images disclosed in the above embodiment can be automatically executed, which will not be repeated here.
[0087] In practical applications, the operation logic of the tunnel linear image adjacent image fusion method disclosed in the above embodiment can also be compiled into a set of executable instructions, which are written into a storage medium and run on an electronic device.
[0088] More specific examples of the computer-readable storage medium mentioned in this embodiment may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device.
[0089] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.
Claims
1. A method for fusing adjacent images of tunnel linear array images, characterized in that: The steps include: Acquire a target image, where the target image is an image captured by two adjacent linear array cameras in a tunnel; perform a preprocessing operation on the target image to obtain two images to be fused; Calculate the pixel energy field of the image to be fused; Based on the pixel energy field, a global optimal stitching path of the images to be fused is solved, and the global optimal stitching path is used as an optimal stitching line; Based on the optimal stitching line, the two images to be fused are processed using a distance weighted fusion method to complete the fusion of adjacent images of the tunnel linear array image.
2. The method for fusing adjacent images of tunnel linear array images according to claim 1 is characterized in that: Preprocessing operations are performed on the target image, including direction correction, tilt correction, orthorectification, and image registration.
3. The method for fusing adjacent images of tunnel linear array images according to claim 1 is characterized in that: Calculate the pixel energy field of the image to be fused, including: Define the energy function to calculate the energy of the pixels in the overlapping area of the two images to be fused. The energy function expression is as follows: Where (x, y) is the pixel coordinate; E c (x, y) represents the color difference between the overlapping pixels of the two images to be fused; E g (x, y) represents the gradient difference between the overlapping pixels of the two images to be fused; α is the weight for controlling the color difference and gradient difference; Analyze the pixel gradient information and pixel color information of the two images to be fused, and assign different crossing costs to different pixels. The crossing costs meet the following conditions: The gradient value of the pixel point is positively correlated with the assigned crossing cost, and the color difference value of the pixel point is positively correlated with the assigned crossing cost.
4. The method for fusing adjacent images of tunnel linear array images according to claim 3 is characterized in that: The color difference E between the overlapping pixels of the two images to be fused c (x,y) is calculated as follows: AND c (x,y)=I1(x,y)-I2(x,y) Gradient Difference E g The calculation is as follows: E g =G x (I1,I2) G y (I1,I2) In the formula, G x , G y Represent the gradient values in the horizontal and vertical directions respectively; I1 and I2 represent the pixel matrices of the two images respectively.
5. The method for fusing adjacent images of tunnel linear array images according to claim 4 is characterized in that: Based on the pixel energy field, the global optimal stitching path of the image to be fused is solved, which specifically includes: Define the function M(x,y) to represent the horizontal seam with the lowest energy starting from the left side of the image to be fused and ending at pixel P(x,y); At the far left of the image to be fused, all seams ending with these pixels are only one pixel long, so the lowest energy seam ending with these pixels is simply the energy of these pixels: M(0,y)=E(0,y) For all other pixels, look at the pixels in the previous column; take the lowest energy of the seams that end with those pixels, and add that to the energy of the current pixel: When the pixel being viewed is at the boundary of the upper edge or lower edge of the image to be fused, M(x-1,y+1) or M(x-1,y+1) is ignored respectively; Finally, the energy of the lowest energy seam extending horizontally across the entire image to be fused is obtained, expressed as a two-dimensional cumulative cost matrix; From left to right and from top to bottom, calculate the minimum cumulative cost to reach each pixel position: min(M(W-1,y)),0≤y≤H-1 Where W and H are the width and height of the image respectively; Select the minimum energy pixel M(W-1,y W-1 ), trace back the best seam pixel position of the previous column from right to left, and the calculation formula is as follows: Finally, a global optimal splicing path with the minimum total cost is obtained; the global optimal splicing path is used as the best stitching line.
6. The method for fusing adjacent images of tunnel linear array images according to claim 5 is characterized in that: The two images to be fused are processed using a distance weighted fusion method, including: A region with a width of D around the optimal stitching line is set for fusion, and the goal is to generate a seamlessly stitched fusion image I mosaic ; For any position P in the overlapping area mosaic (x, y), calculate its distance d(x, y) to the edge of the overlapping area, and take the pixel values of the two images at this position as P1(x, y) and P2(x, y), and use the following formula for weighted fusion: When d(x,y)=0, point P mosaic (x, y) is located at the edge of the overlapping area, and the fusion result is completely taken from the image on the adjacent side; when When mosaic (x,y) is located at the center of the overlapping area, and the contributions of the two images are used equally.
7. A tunnel linear array image adjacent image fusion system, characterized in that: include: Pre-processing unit; The preprocessing unit is used to obtain a target image captured by two adjacent linear array cameras in a tunnel, and perform a preprocessing operation on the target image to obtain two images to be fused; A calculation unit; the calculation unit is used to calculate the pixel energy field of the image to be fused; A splicing path generation unit; The stitching path generation unit is used to solve the global optimal stitching path of the image to be fused based on the pixel energy field, and use the global optimal stitching path as the best stitching line; Fusion unit; The fusion unit processes the two images to be fused based on the optimal stitching line to complete the fusion of adjacent images of the tunnel linear array image; The tunnel linear array image adjacent image frame fusion system can automatically execute the tunnel linear array image adjacent image frame fusion method as described in any one of claims 1 to 6.
8. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for fusing adjacent images of tunnel linear array images according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the electronic device, the electronic device executes the tunnel linear array image adjacent image fusion method according to any one of claims 1 to 6.
Citation Information
Cited By
Railway tunnel panorama synthesis monitoring method, device, equipment and medium
CN119417831A
Image-based photovoltaic construction condition monitoring method
CN120013945A
