Image splicing method and system
By combining Canny's Devernay subpixel algorithm and epipolar matching technology with the Laplacian pyramid algorithm, the problem of color and brightness differences in image stitching was solved, achieving high-precision and high-quality image stitching.
Patent Information
- Application Number
- CN202410847790.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-27
- Publication Date
- 2025-12-30
AI Technical Summary
Existing image stitching methods suffer from significant color and brightness differences at the stitching points, resulting in low stitching accuracy and affecting visual quality.
The Devernay subpixel algorithm based on Canny is used for subpixel-level edge extraction. Combined with epipolar matching technology and Laplacian pyramid algorithm, the center point of the line structured light is accurately located, the homography matrix is calculated for image stitching, and high-quality stitching is achieved through multi-resolution fusion.
It improves the precision and accuracy of image stitching, generates high-quality stitched images, and reduces color and brightness differences at the stitching points.
Smart Images
Figure CN121235901A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image processing technology, and relates to an image stitching method, and particularly to an image stitching method and system. Background Technology
[0002] When a single industrial camera cannot capture the entire object due to limitations in camera resolution and shooting accuracy, or for objects that are long, narrow, or large, multiple cameras are needed to sample the image in segments and then stitch them together into a single image. However, existing image stitching methods often result in noticeable color and brightness differences at the stitching points, leading to low stitching accuracy and affecting the visual quality of the resulting stitched image. Summary of the Invention
[0003] The purpose of this application is to provide an image stitching method and system to solve the problem that existing image stitching methods usually have obvious visible color and brightness differences at the image stitching points, resulting in low image stitching accuracy and affecting the visual effect of the obtained stitched image.
[0004] In a first aspect, this application provides an image stitching method, the image stitching method comprising: acquiring adjacent first and second images, and a first line structured light image corresponding to the first image and a second line structured light image corresponding to the second image; acquiring the overlapping region of the first image and the second image, and the overlapping region of the first line structured light image and the second line structured light image; performing sub-pixel level edge extraction on the overlapping region of the first line structured light image to obtain a first edge point set, and calculating a first center point coordinate set; performing sub-pixel level edge extraction on the overlapping region of the second line structured light image to obtain a second edge point set, and calculating a second center point coordinate set; performing epipolar matching on the first center point coordinate set and the second center point coordinate set according to the epipolar matching strategy to obtain a corresponding center point pair set; calculating the center point pair set to obtain a homography matrix, which serves as the homography matrix of the first image and the second image; and projecting according to the homography matrix to obtain a stitched image of the first image and the second image.
[0005] In one implementation of the first aspect, performing subpixel-level edge extraction on the overlapping region of the first line structured light image to obtain a first edge point set includes: performing image preprocessing on the overlapping region of the first line structured light image and the overlapping region of the first image to obtain a preprocessed image; performing image gradient calculation based on the preprocessed image to obtain image gradient information; performing edge point extraction based on the image gradient information to obtain an image edge point set; and performing edge processing based on the image edge point set to obtain an effective edge point set.
[0006] In one implementation of the first aspect, performing subpixel-level edge extraction on the overlapping region of the second line structured light image to obtain a second edge point set includes: performing image preprocessing on the overlapping region of the second line structured light image and the overlapping region of the second image to obtain a preprocessed image; performing image gradient calculation based on the preprocessed image to obtain image gradient information; performing edge point extraction based on the image gradient information to obtain an image edge point set; and performing edge processing based on the image edge point set to obtain an effective edge point set.
[0007] In one implementation of the first aspect, calculating the image gradient based on the preprocessed image includes: extracting feature points from the preprocessed image; and calculating the image gradient of the feature points in the horizontal direction using an edge detection algorithm; wherein the edge detection algorithm is: I S (x+1, y)-I S (x-1, y)→g x (x, y) where I S (x+1, y) represents the image pixel value at (x+1, y), I S (x-1, y) represents the image pixel value at (x-1, y), g x (x, y) represents the gradient value in the horizontal direction at the point (x, y).
[0008] In one implementation of the first aspect, calculating the image gradient based on the preprocessed image includes: extracting feature points from the preprocessed image; and calculating the image gradient of the feature points in the vertical direction according to an edge detection algorithm; wherein the edge detection algorithm is: I S (x, y+1)-I S (x, y-1)→g y (x, y) where I S (x, y+1) represents the image pixel value at (x, y+1), I S (x, y-1) represents the image pixel value at (x, y-1), g y (x, y) represents the gradient value in the vertical direction at the point (x, y).
[0009] In one implementation of the first aspect, edge point extraction based on the image gradient information includes: obtaining gradient values of several adjacent feature points based on the image gradient information; obtaining corresponding gradient magnitude values based on the gradient values of each feature point; and calculating corresponding edge points based on the gradient magnitude values of each feature point.
[0010] In one implementation of the first aspect, edge processing based on the image edge point set to obtain a valid edge point set includes: obtaining the gradient value of each edge point based on the image edge point set; comparing the gradient value of each edge point with a preset dual threshold; the preset dual threshold includes a high threshold and a low threshold; if the gradient value of each edge point is higher than the high threshold, then the current edge point is marked as a strong edge point and used as a valid edge point; if the gradient value of each edge point is between the high threshold and the low threshold, then the current edge point is marked as a weak edge point, and neighbor pixel detection is performed on the current edge point; if a strong edge point exists among the neighbor pixels of the current edge point, then the current edge point is used as a valid edge point; if the gradient value of each edge point is lower than the low threshold, then the current edge point is an invalid edge point.
[0011] In one implementation of the first aspect, matching the first set of center point coordinates and the second set of center point coordinates according to the epipolar matching strategy to obtain the corresponding set of center points includes: matching the center point coordinates of each center point in the first set of center point coordinates with the corresponding center point coordinates in the second set of center point coordinates according to the epipolar matching strategy; and taking the center point coordinates in the first set of center point coordinates and the corresponding matched center point coordinates in the second set of center point coordinates as a pair of center points.
[0012] In one implementation of the first aspect, the image stitching method further includes: fusing the stitched images according to the Laplacian pyramid algorithm to obtain a fused stitched image.
[0013] Secondly, this application provides an image stitching system, comprising: an acquisition module, configured to acquire adjacent first and second images, a first line structured light image corresponding to the first image, and a second line structured light image corresponding to the second image; and to acquire the overlapping region of the first image and the second image, and the overlapping region of the first line structured light image and the second line structured light image; an edge point extraction module, configured to perform sub-pixel level edge extraction on the overlapping region of the first line structured light image to obtain a first edge point set, and calculate a first center point coordinate set; and to perform sub-pixel level edge extraction on the overlapping region of the second line structured light image to obtain a second edge point set, and calculate a second center point coordinate set; an epipolar matching module, configured to perform epipolar matching on the first center point coordinate set and the second center point coordinate set according to the epipolar matching strategy to obtain a corresponding center point set; a calculation module, configured to calculate the center point set to obtain a homography matrix, which serves as the homography matrix of the first image and the second image; and a stitching module, configured to project according to the homography matrix to obtain a stitched image of the first image and the second image.
[0014] Thirdly, this application provides an electronic device, the electronic device comprising: a memory and a processor; the memory for storing a computer program; the processor connected to the memory for executing the computer program stored in the memory, so that the electronic device performs the image stitching method described above.
[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program thereon, characterized in that the program, when executed by an electronic device, implements the image stitching method described above.
[0016] As described above, the image stitching method and system of this application have the following beneficial effects:
[0017] 1. This application uses the Devernay subpixel algorithm based on Canny to perform subpixel-level edge extraction on line structured light, which is used to accurately locate the center point set of line structured light, thereby assisting in the stitching of images without line structured light and improving the accuracy of image stitching.
[0018] 2. Based on epipolar matching technology, this application can accurately determine the center point position on adjacent field-of-view images, thereby achieving geometric alignment of images, effectively improving the accuracy of image stitching, and thus stitching out high-quality stitched images.
[0019] 3. This application utilizes the Laplacian pyramid algorithm for multi-resolution image fusion, enabling the aligned images to be smoothly merged, thereby effectively stitching together high-quality image results. Attached Figure Description
[0020] Figure 1 The image shown is a schematic diagram of a continuously cast billet containing multi-line structured light, as described in an embodiment of this application.
[0021] Figure 2 The image shown is a schematic diagram of a continuously cast billet without multi-line structured light, as described in an embodiment of this application.
[0022] Figure 3 The diagram shown is a flowchart illustrating the image stitching method described in an embodiment of this application.
[0023] Figure 4 The diagram shows a flowchart illustrating the subpixel-level edge extraction process for the overlapping region of the first line structured light image as described in an embodiment of this application.
[0024] Figure 5 The diagram shows a flowchart illustrating the subpixel-level edge extraction process for the overlapping region of the second line structured light image as described in an embodiment of this application.
[0025] Figure 6The diagram shown is a schematic representation of the process for edge point extraction based on the image gradient information as described in an embodiment of this application.
[0026] Figure 7 The diagram shown is a schematic representation of the process for edge filtering based on the image edge point set as described in an embodiment of this application.
[0027] Figure 8 The diagram shows a flowchart illustrating the matching process between the first set of center point coordinates and the second set of center point coordinates as described in this embodiment.
[0028] Figure 9 The diagram shown is a flowchart illustrating an image stitching method according to another embodiment of this application.
[0029] Figure 10 The image shown is a schematic diagram of the fused spliced image of a continuously cast billet containing multi-line structured light.
[0030] Figure 11 The image shown is a schematic diagram of the fused spliced image of a continuously cast billet without multi-line structured light.
[0031] Figure 12 The diagram shown is a structural schematic of the image stitching system described in an embodiment of this application.
[0032] Figure 13 The diagram shown is a structural schematic of the electronic device described in an embodiment of this application.
[0033] Component designation explanation
[0034] 10 Image stitching system
[0035] 110 Acquisition Module
[0036] 120 Edge Point Extraction Module
[0037] 130 Polar Matching Module
[0038] 140 Calculation Module
[0039] 150 Image Stitching Module
[0040] 20 Electronic devices
[0041] 210 Memory
[0042] 220 processor
[0043] Steps S1 to Sn Detailed Implementation
[0044] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.
[0045] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0046] The following embodiments of this application provide an image stitching method and system that solves the problem that existing image stitching methods often have obvious color and brightness differences at the stitching points, which affect the visual effect of the obtained stitched image.
[0047] The image stitching method and system described in the following embodiments of this application are applicable to edge processing of regular and irregular graphic images. The following embodiments of this application use edge processing of regular graphic images as an example for illustration. This application is not limited thereto. Edge processing of any graphic image is applicable to the image stitching method and system described in this application.
[0048] In one embodiment, this method is applicable when the subject is too long and too wide, requiring images taken by two or more cameras simultaneously to cover the minimum width of the subject, i.e., it requires n×m (n≥2 and m≥2) cameras to simultaneously photograph the same subject.
[0049] This application first obtains the overlapping regions of two adjacent images of several camera fields of view based on a calibration board, and sets the overlapping field of view regions as ROI regions. Subsequently, the image processing process only processes the images in the ROI regions.
[0050] Next, edge points of the overlapping region between two adjacent images are extracted. The corresponding homography matrix is calculated based on these edge points, and projection is performed using this homography matrix to obtain a stitched image of the two adjacent images. Improper edge point extraction can lead to inaccurate coordinates, resulting in noticeable color and brightness differences at the stitching point. This invention uses the Devernay sub-pixel algorithm based on Canny to perform sub-pixel-level edge extraction on line structured light, enabling precise localization of the center point set of the line structured light. The difference between line structured light images and images without line structured light lies in the presence or absence of structured light; their homography transformation matrices are the same. Therefore, this application uses sub-pixel-level edge extraction on line structured light to precisely locate the center point set of the line structured light, and calculates the corresponding homography transformation matrix, which is then applied to the image without line structured light to complete the stitching of the image without line structured light.
[0051] Please see Figure 1 and Figure 2 This application describes the use of multi-line structured light projected onto a continuously cast billet to capture an image of the billet containing the multi-line structured light (e.g., ...). Figure 1 ) and images of continuously cast billets before projection without multi-line structured light (such as Figure 2 Image processing is performed, and the resulting image is converted into a grayscale image. Then, the endpoints of the multi-line structured light center line are extracted using the Devernay sub-pixel algorithm based on Canny, forming a set of edge contour points of the continuous casting billet. A horizontal straight line is drawn by connecting the first and second edge points on the lower edge of the horizontal surface of the continuous casting billet from the left. A vertical straight line is drawn by connecting the first and second edge points on the lower left edge of the vertical surface of the continuous casting billet from the left. The two straight lines intersect at the corner points of the continuous casting billet, thereby realizing the positioning of the corner points and edge contour point set of the continuous casting billet.
[0052] Subsequently, this invention stitches images acquired by two cameras based on the epipolar constraint principle, employing epipolar matching technology to accurately determine the corresponding positions of these center points in the right camera's field of view. Next, based on the selected set of correct center point pairs, a homography matrix is calculated, describing the image transformation relationship from one view to another. Then, this homography matrix is used to perform a homography transformation on the original image that does not contain multiple structured rays to achieve geometric alignment of the images. Finally, to smoothly fuse the aligned images, the Laplacian Pyramid algorithm is used for multi-resolution image fusion, thereby effectively stitching together a high-quality image result.
[0053] ROI (Region of Interest) refers to the area in an image or video that requires special attention or processing.
[0054] The Devernay subpixel algorithm, derived from Canny, is an algorithm designed to improve edge detection accuracy. It achieves more precise edge localization by optimizing the Canny algorithm at the subpixel level. The Canny algorithm itself is widely used for edge detection, extracting edges from images through steps such as Gaussian filtering, non-maximum suppression, and double thresholding. The Devernay algorithm, however, is an improvement on the Canny algorithm, enabling subpixel-level edge localization.
[0055] The core idea of the Devernay subpixel algorithm is to use quadratic interpolation based on the magnitudes of three adjacent points along the gradient direction to find the subpixel-level edge location. Specifically, the algorithm calculates the gradient magnitudes of three points perpendicular to the edge direction and obtains the precise subpixel-level location by solving a quadratic equation.
[0056] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0057] like Figure 3 As shown in the figure, this embodiment provides an image stitching method, which includes the following steps.
[0058] Step S1: Obtain adjacent first and second images, as well as the first line structured light image corresponding to the first image and the second line structured light image corresponding to the second image;
[0059] Step S2: Obtain the overlapping region of the first image and the second image, and the overlapping region of the first line structured light image and the second line structured light image;
[0060] Step S3: Perform sub-pixel level edge extraction on the overlapping area of the first line structured light image to obtain the first edge point set, and calculate the first center point coordinate set.
[0061] Step S4: Perform sub-pixel level edge extraction on the overlapping area of the second line structured light image to obtain the second edge point set, and calculate the second center point coordinate set;
[0062] Step S5: Perform epipolar matching on the first center point coordinate set and the second center point coordinate set according to the epipolar matching strategy to obtain the corresponding center point pair set; calculate the homography matrix on the center point pair set to obtain the homography matrix as the first image and the second image;
[0063] Step S6: Project the image based on the homography matrix to obtain a stitched image of the first image and the second image.
[0064] It should be noted that the labels S1 to S6 are used only to identify different steps, and not to restrict the execution order of these steps.
[0065] The following refers to the appendix in the embodiments of this application. Figures 4 to 11 The image stitching method described in the embodiments of this application will be described in detail.
[0066] Step S1: Obtain adjacent first and second images, as well as the first line structured light image corresponding to the first image and the second line structured light image corresponding to the second image;
[0067] Step S2: Obtain the overlapping region of the first image and the second image, and the overlapping region of the first line structured light image and the second line structured light image;
[0068] In one embodiment, since the camera field of view is fixed, a calibration plate can be placed in the overlapping field of view of the two cameras to determine the position of the overlapping field of view in the pixel coordinate system of each camera. The overlapping field of view region is then set as the Region of Interest (ROI), and subsequent image processing only processes the image within the ROI region.
[0069] Step S3: Perform sub-pixel level edge extraction on the overlapping area of the first line structured light image to obtain the first edge point set, and calculate the first center point coordinate set.
[0070] like Figure 4 As shown, in one embodiment of this application, performing sub-pixel level edge extraction on the overlapping region of the first line structured light image to obtain a first edge point set S3 includes the following steps.
[0071] Step S31: Perform image preprocessing on the overlapping region of the first line structured light image and the overlapping region of the first image to obtain a preprocessed image;
[0072] Step S32: Perform image gradient calculation based on the preprocessed image to obtain image gradient information;
[0073] Step S33: Extract edge points based on the image gradient information to obtain an image edge point set;
[0074] Step S34: Perform edge filtering processing based on the image edge point set to obtain an effective edge point set.
[0075] Step S4: Perform sub-pixel level edge extraction on the overlapping area of the second line structured light image to obtain the second edge point set, and calculate the second center point coordinate set;
[0076] like Figure 5As shown, in one embodiment of this application, step S4, which involves sub-pixel level edge extraction of the overlapping region of the second line structured light image to obtain a second edge point set, includes:
[0077] Step S41: Perform image preprocessing on the overlapping region of the second line structured light image and the overlapping region of the second image to obtain a preprocessed image;
[0078] Step S42: Perform image gradient calculation based on the preprocessed image to obtain image gradient information;
[0079] Step S43: Extract edge points based on the image gradient information to obtain an image edge point set;
[0080] Step S44: Perform edge filtering processing based on the image edge point set to obtain an effective edge point set.
[0081] In one embodiment, the edge extraction of the line structured light is performed at the subpixel level based on the Devernay subpixel algorithm. After obtaining the edge points of the line structured light, the average value of the edge points in the same normal direction is taken to obtain the subpixel coordinates of the center point of the line structured light.
[0082] In one embodiment, please refer to Figure 1 and Figure 2 Images of the continuously cast billet containing multi-line structured light were captured by projecting multi-line structured light onto the billet (see [link to documentation]). Figure 2 ) and an image of the continuously cast billet before projection without multi-line structured light (see [link]). Figure 1 Image subtraction is performed to obtain a multi-line structured light point set image. This image is then converted to grayscale and convolved with a Gaussian kernel function to achieve Gaussian smoothing filtering, minimizing the impact of noise and reducing the error rate. The process of convolving the grayscale image with a Gaussian kernel function is as follows:
[0083] I*K s →I s Formula (1)
[0084] Where I is the grayscale image to be processed, K s Let I be the Gaussian kernel function. s This is the image obtained after filtering.
[0085] In one embodiment of this application, calculating the image gradient based on the preprocessed image includes:
[0086] Step S21A: Extract feature points from the preprocessed image;
[0087] Step S22A: Calculate the image gradient of the feature points in the horizontal direction according to the edge detection algorithm; the edge detection algorithm is:
[0088] I S (x+1, y)-I S (x-1, y)→g x (x, y) Formula (2)
[0089] Among them, I S (x+1, y) represents the image pixel value at point (x+1, y), I S (x-1, y) represents the image pixel value at (x-1, y), g x (x, y) represents the gradient value in the horizontal direction at the point (x, y), and the point (x, y) represents the image I. s One pixel on the screen.
[0090] In one embodiment of this application, calculating the image gradient based on the preprocessed image includes:
[0091] Step S21B: Extract feature points from the preprocessed image;
[0092] Step S21B: Calculate the image gradient of the feature points in the vertical direction according to the edge detection algorithm; the edge detection algorithm is:
[0093] I S (x, y+1)-I S (x, y-1)→g y (x, y) formula (3)
[0094] Among them, I S (x, y+1) represents the image pixel value at (x, y+1), I S (x, y-1) represents the image pixel value at (x, y-1), and gy(x, y) represents the gradient value in the vertical direction at point (x, y).
[0095] In one embodiment, the gradient values of an image often contain rich image information. Generally, when edges exist in an image, they will have large gradient values. Therefore, the Canny / Devernay edge detection algorithm mainly calculates the gradient values of image I in both the horizontal and vertical directions. S The gradient value of (x, y) is represented as follows:
[0096] I S (x+1, y)-I S (x-1, y)→g x (x, y) formula (2)
[0097] I S(x, y+1)-I S (x, y-1)→ gy(x, y) formula (3)
[0098] Where gx(x,y) is the gradient value in the horizontal direction at point (x,y), and gy(x,y) is the gradient value in the vertical direction at point (x,y).
[0099] (g x (x, y), g y (x, y))→g(x, y) formula (4)
[0100] Where g(x, y) represents image I S The gradient vector field of (x, y) at the point (x, y).
[0101] like Figure 6 As shown, in one embodiment of this application, edge point extraction based on the image gradient information includes:
[0102] Step S331: Obtain the gradient values of several adjacent feature points based on the image gradient information;
[0103] Step S332: Obtain the corresponding gradient magnitude value based on the gradient value of each feature point;
[0104] Step S333: Calculate the corresponding edge points based on the gradient magnitude of each feature point.
[0105] In one embodiment, to improve positioning accuracy, the present invention extracts a sub-pixel level set of edge points. Devernay defines the point with the largest interpolation of multiple adjacent gradient magnitudes as an edge point. Therefore, edge points can be determined by calculating the quadratic interpolation of the gradient magnitudes of multiple adjacent points along the gradient direction. The calculation process is illustrated here using three adjacent feature points as an example, as follows:
[0106] A, B, and C are images I. S For three adjacent points on (x, y), their gradient magnitudes are denoted as ||g(A)||, ||g(B)||, and ||g(C)||, respectively. There may exist a sub-pixel position η whose gradient magnitude is greater than ||g(A)||, ||g(B)||, and ||g(C)||. Therefore, it can be expressed as:
[0107]
[0108] To reduce the error, the following corrections are made:
[0109] A point (x, y) is designated as a horizontal edge point when the gradient satisfies the following three conditions:
[0110]
[0111] When the gradient at point (x, y) satisfies the following three conditions (in formula 7), the point is determined as the vertical edge point:
[0112]
[0113] When ||g x (x,y)||=||g y When (x,y)||, the default is the horizontal edge point.
[0114] In one embodiment, the image edge point set obtained through the above steps is an independent set of edge pixels. Here, it is necessary to group pixel sets belonging to the same edge contour together to form a contour. Taking pixels A and B as an example, pixels grouped into the same link should meet the following two conditions:
[0115] ① When the angle between the gradient directions is less than 90 degrees, the mathematical expression is: g(A)·g(B)>0;
[0116] ② This continuous link can divide the dark area onto the same side of the curve, which can be achieved through vector... Verify whether it is orthogonal to one of the two possible gradient directions of point A.
[0117] Where g(A) represents image I S The gradient vector field at pixel A, g(B) represents the image I. S The gradient vector field at pixel B.
[0118] like Figure 7 As shown, in one embodiment of this application, edge filtering processing based on the image edge point set includes:
[0119] Step S341: Obtain the gradient value of each edge point based on the image edge point set;
[0120] Step S342: Compare the gradient value of each edge point with a preset dual threshold; the preset dual threshold includes a high threshold and a low threshold;
[0121] Step S343: If the gradient value of each edge point is higher than the high threshold, then the current edge point is marked as a strong edge point and used as a valid edge point.
[0122] Step S344: If the gradient value of each edge point is between the high threshold and the low threshold, then mark the current edge point as a weak edge point and perform neighborhood pixel detection on the current edge point. If there are strong edge points among the neighboring pixels of the current edge point, then take the current edge point as a valid edge point.
[0123] Step S345: If the gradient value of each edge point is lower than the low threshold, then the current edge point is an invalid edge point.
[0124] In one embodiment, to address edge pixels caused by noise and color variations, a dual-threshold method is used. Let the high threshold be T2 and the low threshold be T1. If the gradient value of an edge pixel is higher than T2, it is marked as a strong edge pixel; if the gradient value is between T1 and T2, it is marked as a weak edge pixel; if the gradient value is lower than T1, it is suppressed.
[0125] Strong edge points can be considered as real edge points, while weak edge points may be caused by changes in noise or other factors. Therefore, we detect the 8 connected neighbor pixels of weak edge points. If there are strong edge points in the neighbor pixels, then the weak edge point is considered as a real edge point and is retained.
[0126] Step S5: Perform epipolar matching on the first center point coordinate set and the second center point coordinate set according to the epipolar matching strategy to obtain the corresponding center point set; calculate the homography matrix on the center point set to obtain the homography matrix of the first image and the second image;
[0127] like Figure 8 As shown, in one embodiment of this application, step S5, which involves matching the first set of center point coordinates and the second set of center point coordinates according to the epipolar matching strategy to obtain the corresponding set of center points, includes the following steps.
[0128] Step S51: For each center point coordinate in the first center point coordinate set, match the corresponding center point coordinate in the second center point coordinate set according to the epipolar matching strategy;
[0129] Step S52: Take the center point coordinates in the first center point coordinate set and the corresponding matching center point coordinates in the second center point coordinate set as a center point pair.
[0130] In one embodiment, the epipolar matching constraint of the binocular camera is given by the following formula (8).
[0131]
[0132] Where p is the coordinate of a pixel in the three-dimensional space where the camera is located, p r Let p be the pixel coordinates of the right camera in the field of view, and F be the fundamental matrix of the camera. l Let p be the pixel coordinates of p in the left camera's field of view. According to formula (8), for a point p in the left camera's field of view... l Its corresponding point p in the right camera's field of view r It must be located relative to p lTherefore, after extracting the center point coordinates of the line structured light from the ROI region, epipolar matching technology is used to accurately determine the corresponding positions of these center points in the right camera's field of view.
[0133] In one embodiment, this application uses homography transformation to make the two images basically aligned, and uses the projection matrix H to represent the correspondence between the two images. The optimal transformation matrix obtained by the RANSAC algorithm is used as the projection matrix H. Let p1(x1,y1) and p2(x2,y2) be the matching points (i.e., center points) in the two images, respectively. Then the mapping relationship between them is as follows (9):
[0134]
[0135] Step S6: Project the image based on the homography matrix to obtain a stitched image of the first image and the second image.
[0136] like Figure 9 As shown, in one embodiment of this application, the image stitching method further includes:
[0137] Step S7: Merge the stitched images according to the Laplacian pyramid algorithm to obtain the merged stitched image.
[0138] In one embodiment, this application employs the Laplacian pyramid algorithm for image fusion, with the specific steps as follows:
[0139] Step S71: Construct the Gaussian pyramid;
[0140] Step S72: Subtract the image data of the adjacent layer of the Gaussian pyramid to obtain the Laplace pyramid;
[0141] Step S73: Add the Laplacian pyramids of the same layers with overlapping areas of the stitched images, and then stack the fused images layer by layer from the bottom layer upwards to obtain the fused image. The final image is as follows. Figure 10 , Figure 11 As shown.
[0142] Please see Figure 10 and Figure 11 . Figure 10 The image displayed is a fused spliced image of a continuously cast billet containing multi-line structured light. Figure 11 The image displayed is a fused spliced image of a continuously cast billet without multi-line structured light.
[0143] It should be noted that the Laplacian pyramid algorithm is an image processing technique, mainly used in applications such as multi-scale image analysis, image fusion, and image restoration.
[0144] The Laplacian pyramid enables multi-scale analysis of image information by constructing multi-level image representations. This algorithm excels in image fusion, for example, in HDR (High Dynamic Range) synthesis of multi-exposure images, where it can effectively merge image details at different exposure levels to generate HDR images with a wide dynamic range. Furthermore, the Laplacian pyramid can perform inverse upsampling and downsampling operations, preserving image detail during downsampling and restoring this information when needed.
[0145] The scope of protection of the image stitching method described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.
[0146] This application also provides an image stitching system that can implement the image stitching method described in this application. However, the implementation apparatus of the image stitching method described in this application includes, but is not limited to, the structure of the image stitching system listed in this embodiment. All structural modifications and substitutions of the prior art made based on the principles of this application are included within the protection scope of this application.
[0147] like Figure 12 As shown, this embodiment provides an image stitching system. The image stitching system 10 includes: an acquisition module 110, an edge point extraction module 120, an epipolar matching module 130, a calculation module 140, and an image stitching module 150.
[0148] The acquisition module 110 is used to acquire adjacent first images and second images, as well as a first line structured light image corresponding to the first image and a second line structured light image corresponding to the second image; and to acquire the overlapping area of the first image and the second image, as well as the overlapping area of the first line structured light image and the second line structured light image.
[0149] The edge point extraction module 120 is used to perform sub-pixel level edge extraction on the overlapping area of the first line structured light image to obtain a first edge point set and calculate a first center point coordinate set; and to perform sub-pixel level edge extraction on the overlapping area of the second line structured light image to obtain a second edge point set and calculate a second center point coordinate set.
[0150] The epipolar matching module 130 is used to perform epipolar matching on the first set of center point coordinates and the second set of center point coordinates according to the epipolar matching strategy to obtain the corresponding set of center points.
[0151] The calculation module 140 is used to calculate the homography matrix of the center point set to obtain the homography matrix, which serves as the homography matrix of the first image and the second image.
[0152] The image stitching module 150 is used to project according to the homography matrix to obtain a stitched image of the first image and the second image.
[0153] It should be noted that the functions or operations of the acquisition module 110, edge point extraction module 120, epipolar matching strategy 130, homography matrix calculation module 140, and image stitching module 150 described in this application embodiment correspond one-to-one with the steps in the above image stitching method, and therefore will not be repeated here.
[0154] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.
[0155] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.
[0156] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0157] like Figure 13 As shown, this embodiment provides an electronic device, the electronic device 20 including: a memory 210 and a processor 220.
[0158] The memory 210 is used to store computer programs;
[0159] The processor 220 is connected to the memory 210 and is used to execute the computer program stored in the memory 210 so that the electronic device performs the image stitching method described above.
[0160] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).
[0161] This application embodiment may also provide a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application embodiment are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.
[0162] When the computer program product is executed by a computer, the computer performs the method described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing method is required, the computer program product can be downloaded and executed on the computer.
[0163] In summary, the image stitching method and system described in this application have the following beneficial effects:
[0164] 1. This application uses the Devernay subpixel algorithm based on Canny to perform subpixel-level edge extraction on line structured light, which is used to accurately locate the center point set of line structured light, thereby assisting in the stitching of images without line structured light and improving the accuracy of image stitching.
[0165] 2. Based on epipolar matching technology, this application can accurately determine the center point position on adjacent field-of-view images, thereby achieving geometric alignment of images, effectively improving the accuracy of image stitching, and thus stitching out high-quality stitched images.
[0166] 3. This application utilizes the Laplacian pyramid algorithm for multi-resolution image fusion, enabling the aligned images to be smoothly merged, thereby effectively stitching together high-quality image results.
[0167] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.
[0168] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.
Claims
1. An image stitching method, characterized by, The image stitching method comprises: obtaining adjacent first and second images, and first and second line structured light images corresponding to the first and second images respectively; obtaining overlapping regions of the first and second images, and overlapping regions of the first and second line structured light images; performing sub-pixel level edge extraction on the overlapping region of the first line structured light image to obtain a first edge point set, and calculating a first center point coordinate set; performing sub-pixel level edge extraction on the overlapping region of the second line structured light image to obtain a second edge point set, and calculating a second center point coordinate set; performing epipolar matching on the first and second center point coordinate sets according to the epipolar matching strategy to obtain a corresponding center point pair set; and calculating the center point pair set to obtain a homography matrix as the homography matrix of the first and second images; projecting according to the homography matrix to obtain a stitched image of the first and second images.
2. The image stitching method of claim 1, wherein, The sub-pixel level edge extraction on the overlapping region of the first line structured light image to obtain a first edge point set comprises: performing image preprocessing on the overlapping region of the first line structured light image and the overlapping region of the first image to obtain a preprocessed image; performing image gradient calculation according to the preprocessed image to obtain image gradient information; performing edge point extraction according to the image gradient information to obtain an image edge point set; performing edge screening processing according to the image edge point set to obtain an effective edge point set.
3. The image stitching method of claim 1, wherein, The sub-pixel level edge extraction on the overlapping region of the second line structured light image to obtain a second edge point set comprises: performing image preprocessing on the overlapping region of the second line structured light image and the overlapping region of the second image to obtain a preprocessed image; performing image gradient calculation according to the preprocessed image to obtain image gradient information; performing edge point extraction according to the image gradient information to obtain an image edge point set; performing edge screening processing according to the image edge point set to obtain an effective edge point set.
4. The image stitching method of claim 2 or 3, characterized in that, The image gradient calculation according to the preprocessed image comprises: extracting feature points from the preprocessed image; performing image gradient calculation from the horizontal direction on the feature points according to an edge detection algorithm; the edge detection algorithm is: I S (x+1,y)-I S (x-1,y)→g x (x,y) where I S (x+1,y) represents the image pixel value at (x+1, y), I S (x-1,y) represents the image pixel value at (x-1, y), g x (x,y) represents the gradient value in the horizontal direction at point (x, y).
5. The image stitching method of claim 2 or 3, wherein, The image gradient calculation according to the preprocessed image comprises: extracting feature points from the preprocessed image; performing image gradient calculation from the vertical direction on the feature points according to an edge detection algorithm; the edge detection algorithm is: I S (x,y+1)-I S (x,y-1)→g y (x,y) where I S (x,y+1) denotes the image pixel value at (x, y + 1), I S (x,y-1) denotes the image pixel value at (x, y - 1), g y (x,y) denotes the gradient value in the vertical direction at point (x, y).
6. The image stitching method of claim 2 or 3, wherein, The edge point extraction according to the image gradient information comprises: obtaining gradient values of a plurality of adjacent feature points according to the image gradient information; obtaining gradient module values corresponding to each of the feature points respectively according to the gradient values of the feature points; calculating the corresponding edge points according to the gradient module values corresponding to each of the feature points.
7. The image stitching method of claim 2 or 3, wherein, The edge screening processing according to the image edge point set comprises: obtaining gradient values of each of the edge points according to the image edge point set; The gradient value of each edge point is compared with preset double thresholds respectively, and the preset double thresholds include a high threshold and a low threshold; If the gradient value of each edge point is higher than the high threshold, the current edge point is marked as a strong edge point as a valid edge point; If the gradient value of each edge point is between the high threshold and the low threshold, the current edge point is marked as a weak edge point, and neighborhood pixel detection is performed on the current edge point, and if there is a strong edge point in the neighborhood pixel points of the current edge point, the current edge point is regarded as a valid edge point; If the gradient value of each edge point is lower than the low threshold, the current edge point is an invalid edge point.
8. The image stitching method of claim 1, wherein, The matching of the first center point coordinate set and the second center point coordinate set according to the epipolar matching strategy to obtain a corresponding center point set includes: For each center point coordinate in the first center point coordinate set, a corresponding center point coordinate is matched in the second center point coordinate set according to the epipolar matching strategy; The center point coordinate in the first center point coordinate set and the corresponding matched center point coordinate in the second center point coordinate set are taken as a center point pair.
9. The image stitching method of claim 1, wherein, The image stitching method further includes: Fusion of the stitched image according to a Laplacian pyramid algorithm to obtain a fused stitched image.
10. An image stitching system characterized by, The image stitching system includes: An acquisition module is configured to acquire adjacent first and second images, and a first line structured light image corresponding to the first image and a second line structured light image corresponding to the second image, and to acquire an overlapping region of the first and second images and an overlapping region of the first and second line structured light images; An edge point extraction module is configured to perform sub-pixel level edge extraction on the overlapping region of the first line structured light image to obtain a first edge point set and to calculate a first center point coordinate set, and to perform sub-pixel level edge extraction on the overlapping region of the second line structured light image to obtain a second edge point set and to calculate a second center point coordinate set; An epipolar matching module is configured to perform epipolar matching of the first center point coordinate set and the second center point coordinate set according to the epipolar matching strategy to obtain a corresponding center point pair set; A calculation module is configured to calculate the center point pair set to obtain a homography matrix as a homography matrix of the first and second images; An image stitching module is configured to perform projection according to the homography matrix to obtain a stitched image of the first and second images.