Two-stage zoom camera multi-image stitching method and system based on SuperGlue
By decomposing image stitching into two steps using the SuperPoint and SuperGlue algorithms and combining it with optical flow field registration, the problems of misalignment and ghosting in multi-view, multi-image stitching are solved, achieving efficient and natural image stitching results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NINGBO INST OF MATERIALS TECH & ENG CHINESE ACAD OF SCI
- Filing Date
- 2023-04-23
- Publication Date
- 2026-05-29
AI Technical Summary
Traditional image stitching methods suffer from problems such as misalignment and ghosting in multi-view, multi-image stitching, and the stitching process is complex. In particular, feature points are difficult to extract and computationally complex in sparse texture scenes, resulting in distorted or deformed stitching results.
The SuperPoint and SuperGlue algorithms are used for feature extraction and matching, and the image stitching is decomposed into two steps: first, large field-of-view stitching is performed, and then fine stitching of clear images is performed. Optical flow field registration is combined to reduce errors.
It improves the robustness and speed of stitching in sparse texture scenes, reduces the error introduced by the homography matrix, and achieves efficient and natural image stitching.
Smart Images
Figure CN116468609B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the stitching of multiple images from multiple perspectives, and particularly to a method and system for stitching multiple images from a two-stage zoom camera based on SuperGlue and optical flow field, belonging to the field of image processing technology. Background Technology
[0002] With the rapid development of computer vision technology, cameras are frequently used in various fields such as security and traffic monitoring, medical imaging, and industrial inspection. For some large-scale scenes, the limited field of view of cameras makes it difficult to capture a complete image, often necessitating image stitching methods to obtain images with a wide field of view. However, image stitching tasks involving large variations in viewing angle and a large number of images are challenging, prone to problems such as misalignment and ghosting, and the stitching process is also quite complex. Therefore, researching an efficient stitching method for multiple viewing angles and multiple images is of great significance.
[0003] Traditional image stitching methods typically employ SIFT or Orb algorithms for feature extraction, followed by feature matching to calculate the image transformation matrix for alignment and stitching. This is the most practical and widely used method for image stitching. However, for scenes with sparse textures, this method may struggle to obtain a sufficient number of feature points and achieve accurate feature matching. Furthermore, for multi-planar scenes, a single homography transformation matrix may be insufficient to align the entire image. A more representative solution is based on spatial transformation methods, such as the APAP algorithm, which aligns overlapping regions by dividing the image into a grid and using a large number of local homography matrices. However, the computational process is too complex, requiring the adjustment of numerous parameters, and non-overlapping regions are prone to distortion.
[0004] In the field of multi-image stitching, traditional methods usually involve stitching images one by one in sequence. This process is cumbersome and prone to cumulative errors, resulting in obviously distorted or deformed stitching results that are not natural. Summary of the Invention
[0005] To address the aforementioned issues, this invention proposes an efficient multi-image stitching method based on a variable-focus gimbal camera. The image stitching process is decomposed into two steps: first, large-field-of-view image stitching is performed; second, fine-grained stitching of clear images is completed based on this. The stitching of each image is independent of the others, simplifying the process. SuperPoint and SuperGlue algorithms are employed for feature extraction and matching, offering greater robustness and speed in sparse texture scenes. Optical flow field registration is introduced to reduce the unavoidable stitching errors caused by individual homography matrices.
[0006] To achieve the aforementioned objectives, the present invention employs the following solution:
[0007] One aspect of the present invention provides a method for stitching multiple images from a two-segment zoom camera based on SuperGlue, the image stitching method comprising the following steps:
[0008] Step 1: Lower the magnification of the gimbal camera and take several images with a wide field of view. Use a feature matching algorithm to stitch the images together and generate a template base image.
[0009] Step 2: Upsample and divide the template base image into blocks to generate template base image blocks;
[0010] Step 3: Calculate and record the shooting point position using the camera's field of view geometry model and the area to be photographed, combined with the gimbal camera's field of view angle; automatically acquire clear images and ensure that all acquired clear images cover the specified area.
[0011] Step 4: Select a clear image and select the template base map block with roughly the same coverage area. Use the SuperPoint and SuperGlue algorithms to extract and match features between the clear image and the template base map block. After removing mismatched points, restore the pixel positions of the template base map block in the matching points to the pixel positions in the original template base map, and calculate the homography transformation matrix.
[0012] Step 5: Use the homography transformation matrix to transform the clear image to the coordinate system of the original template base image, obtain the template base image of the area covered by the transformed clear image, and calculate the optical flow between the template base image of the covered area and the transformed clear image.
[0013] Step 6: Perform pixel-by-pixel registration on the transformed clear image based on the optical flow to align it with the template base image;
[0014] Step 7: Repeat steps 4 through 6 until all clear images have been transformed onto the coordinate system of the template base image, and finally fuse the images.
[0015] In one embodiment, calculating the location of the shooting point in step three specifically includes:
[0016] Assuming the camera is pointing directly downwards, the spatial coordinates of the four corners of the camera's field of view on the imaging plane at a distance h are respectively... Where α represents the horizontal field of view and β represents the vertical field of view; when the gimbal rotates left and right or up and down by an angle θ, points A, B, C, and D are uniformly represented as v0, v′0=T*R(z)*R(x)*v0 represents the coordinates after rotation and translation, then v′0=T*R(z)*R(x)*v0, where R(z) and R(x) represent the rotation matrices around the z-axis and x-axis respectively, and the intersection of v′0=T*R(z)*R(x)*v0 with the ground is the coordinate of the four intersection points of the camera's field of view.
[0017] In one embodiment, the optical flow calculation process in step five is as follows:
[0018] I(x, y, z) = I(x + dx, y + dy, t + dt) represents a pixel that moves a distance of (dx, dy) from one frame to the next, taking a time of dt; this is expressed using Taylor expansion.
[0019]
[0020] Further
[0021]
[0022] set up Let x and y represent the velocity vectors of the optical flow along the x and y axes, respectively, so we get...
[0023] I x u+I y v+I t =0
[0024] Among them, I x I y I t The partial derivatives of the gray values of the image pixels in the X, Y, and T directions can be obtained from the image data. (u, v) are the light loss along the X and Y axes, respectively.
[0025] In one embodiment, the process of performing pixel-by-pixel registration of the transformed clear image based on the optical flow in step six to align it with the template base image uses the following calculation formula:
[0026] F(x+u,y+v)=P(x,y);
[0027] Where P represents the pixel coordinates of the clear image before the alignment template base image, F represents the pixel values of the clear image after the alignment template base image, x and y represent pixel coordinates, and u and v represent the optical flow magnitudes in the X and Y directions of the pixel position.
[0028] In one embodiment, the image fusion in step seven is specifically calculated using the following formula:
[0029] P(x, y)=w1*P1(x, y)+w2*P2(x, y)
[0030] Where P represents the pixel value of the fused image, P1 and P2 represent the pixel values of the overlapping regions of the two clear images, and w1 and w2 represent the weights of the pixel values of the two images.
[0031] Another aspect of the present invention provides a multi-image stitching system for a two-segment zoom camera based on SuperGlue, characterized in that the image stitching system comprises:
[0032] The image acquisition and template base image generation module is used to reduce the magnification of the gimbal camera, capture several images with a wide field of view, and use a feature matching algorithm to stitch the images together to generate a template base image.
[0033] A template base image block generation module is used to upsample and divide the template base image into blocks to generate template base image blocks.
[0034] The clear image acquisition module is used to calculate and record the position of the shooting point by combining the field of view geometry model of the camera and the area to be shot with the field of view angle of the gimbal camera, and automatically acquire clear images so that all acquired clear images cover the specified area.
[0035] The transformation matrix calculation module is used to select a clear image and a template base map block with roughly the same coverage area. The SuperPoint and SuperGlue algorithms are used to extract and match features between the clear image and the template base map block. After removing mismatched points, the pixel positions of the template base map block in the matching points are restored to the pixel positions in the original template base map, and the homography transformation matrix is calculated.
[0036] An optical flow calculation module is used to transform the sharp image to the coordinate system of the original template base image using the homography transformation matrix, obtain the template base image of the covered area of the transformed sharp image, and calculate the optical flow between the template base image of the covered area and the transformed sharp image.
[0037] A pixel matching module, wherein the pixel matching is used to perform pixel-by-pixel registration on the transformed clear image according to the optical flow, so that it is aligned with the template base image;
[0038] An image fusion module is used to fuse images after all clear images have been transformed to the coordinate system of the template base image.
[0039] In one embodiment, the clear image acquisition module calculates the location of the shooting point, specifically including:
[0040] Assuming the camera is pointing directly downwards, the spatial coordinates of the four corners of the camera's field of view on the imaging plane at a distance h are respectively... Where α represents the horizontal field of view and β represents the vertical field of view; when the gimbal rotates left and right or up and down by an angle θ, points A, B, C, and D are uniformly represented as v0, v′0=T*R(z)*R(x)*v0 represents the coordinates after rotation and translation, then v′0=T*R(z)*R(x)*v0, where R(z) and R(x) represent the rotation matrices around the z-axis and x-axis respectively, and the intersection of v′0=T*R(z)*R(x)*v0 with the ground is the coordinate of the four intersection points of the camera's field of view.
[0041] In one embodiment, the optical flow calculation module calculates the optical flow as follows:
[0042] I(x, y, z) = I(x + dx, y + dy, t + dt) represents a pixel that moves a distance of (dx, dy) from one frame to the next, taking a time of dt; this is expressed using Taylor expansion.
[0043]
[0044] Further
[0045]
[0046] set up Let x and y represent the velocity vectors of the optical flow along the x and y axes, respectively, so we get...
[0047] I x u+I y v+I t =0
[0048] Among them, I x I y I t The partial derivatives of the gray values of the image pixels in the X, Y, and T directions can be obtained from the image data. (u, v) are the light loss along the X and Y axes, respectively.
[0049] In one embodiment, the pixel matching module performs pixel-by-pixel registration of the transformed sharp image based on the optical flow to align it with the template base image using the following calculation formula:
[0050] F(x+u,y+v)=P(x,y);
[0051] Where P represents the pixel coordinates of the clear image before the alignment template base image, F represents the pixel values of the clear image after the alignment template base image, x and y represent pixel coordinates, and u and v represent the optical flow magnitudes in the X and Y directions of the pixel position.
[0052] In one embodiment, the image module performs image fusion using the following calculation formula:
[0053] P(x, y)=w1*P1(x, y)+w2*P2(x, y)
[0054] Where P represents the pixel value of the fused image, P1 and P2 represent the pixel values of the overlapping regions of the two clear images, and w1 and w2 represent the weights of the pixel values of the two images.
[0055] Compared with the prior art, the present invention has at least the following advantages: (1) It decomposes the image stitching into two steps. First, it completes the image stitching with a large field of view, and then completes the fine stitching of clear images on this basis. The stitching of each image is independent of each other, simplifying the stitching process; (2) It uses SuperPoint and SuperGlue algorithms for feature extraction and matching, which is more robust and faster in sparse texture scenes; (3) It introduces optical flow field registration to reduce the unavoidable stitching error caused by the single homography matrix. Attached Figure Description
[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0057] Figure 1 This is a flowchart of a splicing method according to one embodiment of the present invention;
[0058] Figure 2 This is a geometric model of the gimbal shooting field of view in one embodiment of the present invention;
[0059] Figure 3 This is an example of a template base map in one embodiment of the present invention;
[0060] Figure 4 This is a feature matching comparison diagram between the Sift algorithm and the SuperGlue algorithm;
[0061] Figure 5a This is a clear illustrated example of an embodiment of the present invention. Figure 5b This is a schematic diagram of a splicing method according to one embodiment of the present invention;
[0062] Figure 6a , Figure 6b These are examples of local alignment before and after optical flow registration in one embodiment of the present invention. Detailed Implementation
[0063] To address the shortcomings of existing technologies, the inventors propose the technical solution of this invention, which effectively solves the aforementioned defects or problems of existing neural radiation field technologies. This invention is a three-dimensional reconstruction method for neural radiation fields based on depth dynamic sampling. The proposed method is comparable in performance to some of the most advanced neural radiation field reconstruction methods and offers excellent visual effects.
[0064] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Examples of these preferred embodiments are illustrated in the drawings. The embodiments of the present invention shown in and described with reference to the drawings are merely exemplary, and the present invention is not limited to these embodiments.
[0065] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.
[0066] In a typical embodiment of this application, the following steps are included:
[0067] 1) To achieve accurate stitching of a certain number of multi-view images captured by a gimbal camera, the gimbal camera magnification is first lowered. At this point, the field of view is larger, and several low-resolution images covering a specified area are captured. The gimbal camera magnification is a property of zoom cameras; the lower the magnification, the larger the field of view. Since images with a large field of view have small angular changes and rich features, a traditional feature matching method is used to calculate the homography transformation matrix for stitching, resulting in a template base image.
[0068] 2) Upsample the template base image obtained in step 1) and divide it into 2*2 or 4*4 blocks to obtain template base image blocks.
[0069] 3) Acquiring clear images: Using the camera's field-of-view geometry model and the area to be photographed (as shown in the figure), the position of the shooting point is calculated and recorded based on the gimbal camera's field of view. The calculation process is as follows:
[0070] Assume that when the camera is pointing directly downwards, the spatial coordinates of the four corners of the camera's field of view on the imaging plane at a distance h are A(h*). Where α represents the horizontal field of view and β represents the vertical field of view; when the gimbal rotates left and right or up and down by an angle θ, points A, B, C, and D are uniformly represented as v0, v′0=T*R(z)*R(x)*v0 represents the coordinates after rotation and translation, then v′0=T*R(z)*R(x)*v0, where R(z) and R(x) represent the rotation matrices around the z-axis and x-axis respectively, and the intersection of v′0=T*R(z)*R(x)*v0 with the ground is the coordinate of the four intersection points of the camera's field of view.
[0071] Therefore, by using the height, magnification, and rotation angle of the gimbal camera, the preset points of the camera's field of view can be obtained. These three sets of data can be used to set several preset points so that all captured images can cover the specified area, thereby automatically completing the acquisition of clear images.
[0072] 4) Select a clear image and choose a template base map patch with roughly the same coverage area. Use the SuperPoint and SuperGlue algorithms to extract features and match the clear image with the template base map patch. After removing mismatched points using the random consistency method, restore the pixel positions of the template base map patch in the matching points to their original pixel positions in the template base map, and calculate the homography transformation matrix. The homography transformation matrix represents an image linear geometric transformation matrix with eight degrees of freedom. Establish a linear equation using the coordinates of several pairs of matching points from the two images, and use the least squares method to fit the best result.
[0073] 5) Transform the sharp image using the homography transformation matrix calculated in step 4). Establish a template base image pixel coordinate system with the top left corner of the template base image as the origin. Transform the sharp image into the template base image coordinate system through operations such as rotation, scaling, translation, shearing, and mirroring. Obtain the template base image of the area covered by the transformed sharp image. Calculate the dense optical flow between the template base image and the transformed sharp image. Optical flow refers to the projection of a three-dimensional motion field onto the image, representing the magnitude and direction of the image pixel's motion at a certain moment. The calculation process of optical flow is as follows:
[0074] I(x, y, z) = I(x + dx, y + dy, t + dt) represents a pixel that moves a distance of (dx, dy) from one frame to the next, taking a time of dt; this is expressed using Taylor expansion.
[0075]
[0076] Further
[0077]
[0078] set up Let x and y represent the velocity vectors of the optical flow along the x and y axes, respectively, so we get...
[0079] I x u+I y v+I t =0
[0080] Among them, I x I y I t The partial derivatives of the gray values of the image pixels in the X, Y, and T directions can be obtained from the image data. (u, v) are the light loss along the X and Y axes, respectively.
[0081] 6) Using the optical flow obtained in step 5), perform pixel-by-pixel registration on the transformed clear image, as shown in the following formula, to align it with the template base image.
[0082] F(x+u,y+v)=P(x,y);
[0083] Where P is the pixel value of the clear image before the alignment template base image, F is the pixel value of the clear image after the alignment template base image, x and y represent pixel coordinates, and u and v represent the optical flow magnitude in the X and Y directions at that pixel position.
[0084] 7) Repeat steps 4), 5), and 6) until all clear images have been transformed onto the coordinate system of the template base image. Finally, use a weighted average method to fuse the images, as shown in the following formula:
[0085] P(x, y)=w1*P1(x, y)+w2*P2(x, y)
[0086] Where p represents the pixel value of the fused image, P1 and P2 represent the pixel values of the overlapping areas of the two clear images, and w1 and w2 represent the weights of the pixel values of the two images. The simplest way is to set them to 0.5 and 0.5, but they can also be set to other suitable values.
[0087] Specifically, such as Figure 1 As shown, the specific implementation process of the present invention, a method for efficient multi-image stitching of a two-segment zoom camera based on SuperGlue, is as follows:
[0088] 1) such as Figure 2 As shown, by capturing images from top to bottom using a gimbal camera, adjusting the gimbal's magnification and pitch and yaw angles can produce images of different clarity and positions.
[0089] 2) By adjusting the magnification of the gimbal camera, capture 2-3 low-resolution images with a wide field of view covering a specified area. Stitch these images together using the traditional SIFT+FLAN method to obtain a template base image, and then upsample and segment it, such as... Figure 3The template shown is a composite image of two wide field-of-view images; then, the shooting preset position is calculated according to the required sharpness and a clear image is automatically acquired, and all clear images cover the specified area.
[0090] 3) Select a clear image and a template base image patch, and use the SuperPonit and SuperGlue algorithms for feature extraction and matching. Use the random consistency method to remove mismatched points to obtain accurate matching point pairs, such as... Figure 4 As shown, the traditional SIFT method suffers from a large number of mismatches.
[0091] 4) Restore the pixel coordinates of the template base image blocks in the matching point pair back to their original positions in the template base image, and calculate the homography transformation matrix, such as... Figure 5a , Figure 5b As shown, the sharp image is transformed to the coordinate system of the original template base image according to the homography transformation matrix.
[0092] 5) Generate a mask by transforming the coordinates of the four vertices of the clear image, thereby obtaining the template base image of the area covered by the transformed clear image, and calculate the optical flow between it and the transformed clear image.
[0093] 6) The calculated optical flow includes the x and y position offsets of each pixel. Based on these offsets, the transformed sharp image is registered pixel by pixel to align it with the template base image, such as... Figure 6a , Figure 6b The image shows the local alignment before and after optical flow registration. It can be seen that after optical flow registration, the reprojected images are accurately aligned.
[0094] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions of each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
Claims
1. A method for stitching multiple images from a two-segment zoom camera based on SuperGlue, characterized in that, The image stitching method includes the following steps: Step 1: Lower the magnification of the gimbal camera and take several images with a wide field of view. Use a feature matching algorithm to stitch the images together and generate a template base image. Step 2: Upsample and divide the template base image into blocks to generate template base image blocks; Step 3: Calculate and record the shooting point position using the camera's field of view geometry model and the area to be photographed, combined with the gimbal camera's field of view angle; automatically acquire clear images and ensure that all acquired clear images cover the specified area. Step 4: Select a clear image and select the template base map block with roughly the same coverage area. Use the SuperPoint and SuperGlue algorithms to extract and match features between the clear image and the template base map block. After removing mismatched points, restore the pixel positions of the template base map block in the matching points to the pixel positions in the original template base map, and calculate the homography transformation matrix. Step 5: Use the homography transformation matrix to transform the clear image to the coordinate system of the original template base image, obtain the template base image of the area covered by the transformed clear image, and calculate the optical flow between the template base image of the covered area and the transformed clear image. Step 6: Perform pixel-by-pixel registration on the transformed clear image based on the optical flow to align it with the template base image; Step 7: Repeat steps 4 through 6 until all clear images have been transformed onto the coordinate system of the template base image, and finally fuse the images.
2. The image stitching method according to claim 1, characterized in that, Step three, calculating the location of the shooting point, specifically includes: Assuming the camera is pointing directly downwards, the spatial coordinates of the four corners of the camera's field of view on the imaging plane at a distance h are respectively... Where α represents the horizontal field of view and β represents the vertical field of view; when the gimbal rotates left and right or up and down by an angle θ, points A, B, C, and D are uniformly represented as v0, v′0=T*R(z)*R(x)*v0 represents the coordinates after rotation and translation, then v′0=T*R(z)*R(x)*v0, where R(z) and R(x) represent the rotation matrices around the z-axis and x-axis respectively, and the intersection of v′0=T*R(z)*R(x)*v0 with the ground is the coordinate of the four intersection points of the camera's field of view.
3. The image stitching method according to claim 1, characterized in that, The calculation process for optical flow in step five is as follows: I(x, y, z) = I(x+dx, y+dy, t+dt) represents a pixel that moves a distance of (dx, dy) from one frame to the next, and takes a time of dt. Taylor expansion Further set up Let x and y represent the velocity vectors of the optical flow along the x and y axes, respectively, so we get... I x u+I y v+I t =0 Among them, I x I y I t The partial derivatives of the gray values of the image pixels in the X, Y, and T directions can be obtained from the image data. (u, v) are the light loss along the X and Y axes, respectively.
4. The image stitching method according to claim 3, characterized in that, In step six, the process of registering the transformed clear image pixel-by-pixel according to the optical flow to align it with the template base image uses the following calculation formula: F(x+u,y+v)=P(x,y); Where P represents the pixel coordinates of the clear image before the alignment template base image, F represents the pixel values of the clear image after the alignment template base image, x and y represent pixel coordinates, and u and v represent the optical flow magnitudes in the X and Y directions of the pixel position.
5. The image stitching method according to claim 1, characterized in that, The image fusion process in step seven uses the following calculation formula: P(x, y)=w1*P1(x, y)+w2*P2(x, y) Where P represents the pixel value of the fused image, P1 and P2 represent the pixel values of the overlapping regions of the two clear images, and w1 and w2 represent the weights of the pixel values of the two images.
6. A multi-image stitching system for a two-segment zoom camera based on SuperGlue, characterized in that, The image stitching system includes: The image acquisition and template base image generation module is used to reduce the magnification of the gimbal camera, capture several images with a wide field of view, and use a feature matching algorithm to stitch the images together to generate a template base image. A template base image block generation module is used to upsample and divide the template base image into blocks to generate template base image blocks. The clear image acquisition module is used to calculate and record the position of the shooting point by combining the field of view geometry model of the camera and the area to be shot with the field of view angle of the gimbal camera, and automatically acquire clear images so that all acquired clear images cover the specified area. The transformation matrix calculation module is used to select a clear image and a template base map block with roughly the same coverage area. The SuperPoint and SuperGlue algorithms are used to extract and match features between the clear image and the template base map block. After removing mismatched points, the pixel positions of the template base map block in the matching points are restored to the pixel positions in the original template base map, and the homography transformation matrix is calculated. An optical flow calculation module is used to transform the sharp image to the coordinate system of the original template base image using the homography transformation matrix, obtain the template base image of the covered area of the transformed sharp image, and calculate the optical flow between the template base image of the covered area and the transformed sharp image. A pixel matching module, wherein the pixel matching is used to perform pixel-by-pixel registration on the transformed clear image according to the optical flow, so that it is aligned with the template base image; An image fusion module is used to fuse images after all clear images have been transformed to the coordinate system of the template base image.
7. The image stitching system according to claim 6, characterized in that, The clear image acquisition module calculates the location of the shooting point, specifically including: Assuming the camera is pointing directly downwards, the spatial coordinates of the four corners of the camera's field of view on the imaging plane at a distance h are respectively... Where α represents the horizontal field of view and β represents the vertical field of view; when the gimbal rotates left and right or up and down by an angle θ, points A, B, C, and D are uniformly represented as v0, v′0=T*R(z)*R(x)*v0 represents the coordinates after rotation and translation, then v′0=T*R(z)*R(x)*v0, where R(z) and R(x) represent the rotation matrices around the z-axis and x-axis respectively, and the intersection of v′0=T*R(z)*R(x)*v0 with the ground is the coordinate of the four intersection points of the camera's field of view.
8. The image stitching system according to claim 6, characterized in that, The optical flow calculation module calculates the optical flow as follows: I(x, y, z) = I(x+dx, y+dy, t+dt) represents a pixel that moves a distance of (dx, dy) from one frame to the next, and takes a time of dt. Taylor expansion Further set up Let x and y represent the velocity vectors of the optical flow along the x and y axes, respectively, so we get... I x u+I y v+I t =0 Among them, I x I y I t The partial derivatives of the gray values of the image pixels in the X, Y, and T directions can be obtained from the image data. (u, v) are the light loss along the X and Y axes, respectively.
9. The image stitching system according to claim 8, characterized in that, The pixel matching module performs pixel-by-pixel registration of the transformed clear image based on the optical flow to align it with the template base image using the following calculation formula: F(x+u,y+v)=P(x,y); Where P represents the pixel coordinates of the clear image before the alignment template base image, F represents the pixel values of the clear image after the alignment template base image, x and y represent pixel coordinates, and u and v represent the optical flow magnitudes in the X and Y directions of the pixel position.
10. The image stitching system according to claim 6, characterized in that, The image module performs image fusion using the following calculation formula: P(x, y)=w1*P1(x, y)+w2*P2(x, y) Where P represents the pixel value of the fused image, P1 and P2 represent the pixel values of the overlapping regions of the two clear images, and w1 and w2 represent the weights of the pixel values of the two images.