An image stitching method, device and equipment

Abstract

The application provides an image splicing method, device and equipment, and applies to the technical field of image processing. The method can be applied to a binocular camera, the binocular camera comprising a first camera and a second camera, and an image collected by the first camera and an image collected by the second camera partially overlap. The method comprises obtaining a first image collected by the first camera and a second image collected by the second camera when the target region is photographed by using the binocular camera. The second image is subjected to perspective transformation by using a predetermined projection mapping matrix to obtain a transformed image. The first image and the transformed image are fused to obtain an initial fused image. The first overlapped image of the initial fused image and the second overlapped image of the initial fused image are spliced to obtain a spliced image. The problem that a traditional image splicing method has a complex calibration process, is easily affected by environmental factors, and has a high response delay of the binocular camera due to a slow data transmission speed can be solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image stitching method, apparatus and device. Background Technology

[0002] With the continuous advancement of computer vision and image processing technologies, image stitching technology has been widely applied in various fields. Especially in scenarios with high environmental perception requirements, such as industrial inspection, medical imaging, virtual reality, and autonomous driving, image stitching technology can effectively expand the visual range and provide more comprehensive scene information. Typically, a single camera, limited by lens focal length and installation location, has a limited field of view, making it difficult to meet users' needs for wide-angle or panoramic images. To overcome this limitation, binocular camera image stitching technology has emerged. This technology registers and fuses images captured by two cameras to generate a continuous image covering a larger field of view.

[0003] However, in practical applications, traditional image stitching techniques still have many shortcomings. For example, traditional image stitching methods require accurate acquisition of camera intrinsic and extrinsic parameters during camera calibration, a process that is complex and easily affected by environmental factors. Simultaneously, image matching and fusion algorithms involve large computational loads, resulting in slow data transmission speeds and high response latency for binocular cameras. Summary of the Invention

[0004] The purpose of this application is to provide an image stitching method, apparatus, and device to solve the problems of complex calibration process and susceptibility to environmental factors in traditional image stitching methods, as well as high response delay of binocular cameras due to slow data transmission speed.

[0005] In a first aspect, embodiments of this application provide an image stitching method applied to a binocular camera, which includes a first camera and a second camera. The images captured by the first camera and the second camera cover an overlapping area. The method includes: acquiring a first image captured by the first camera and a second image captured by the second camera when photographing a target area using the binocular camera; performing a perspective transformation on the second image using a predetermined projection mapping matrix to obtain a transformed image; fusing the first image and the transformed image to obtain an initial fused image; and stitching a first overlapping image and a second overlapping image of the initial fused image together to obtain a stitched image. The first and second overlapping images are images of the overlapping area. The stitched image includes images of the overlapping area and images outside the overlapping area of ​​the first image and the transformed image.

[0006] The image stitching method provided in this application utilizes perspective transformation and fusion algorithms to process the second image during the image stitching process and then fuses it with the first image, achieving fast and high-quality image stitching. This method not only reduces computational load and improves data transmission speed but also effectively solves the problem of obvious stitching artifacts in traditional stitching methods, resulting in more natural edge transitions and a more complete and aesthetically pleasing visual effect in the stitched image. Furthermore, the calibration process of this projection mapping matrix does not require complex calculations of camera parameters, simplifying the calibration process, reducing sensitivity to environmental factors, and improving the accuracy and stability of the calibration.

[0007] One possible implementation involves transforming the image, including its size and the pixel value of each pixel. The transformed image is obtained by performing a perspective transformation on the second image using a pre-determined projection mapping matrix. This includes: performing a perspective transformation on the vertex coordinates of the second image using the projection mapping matrix to obtain the transformed vertex coordinates; determining the image size of the transformed image based on the transformed vertex coordinates; performing a perspective transformation on the first pixel coordinates of any pixel in the transformed image using the projection mapping matrix to obtain the second pixel coordinates of the pixel; and determining the pixel value of each pixel in the transformed image based on the pixel value of each pixel in the second image and the second pixel coordinates of all pixels in the transformed image.

[0008] One possible implementation involves stitching together a first overlapping image and a second overlapping image of the initial fused image to obtain a stitched image. This includes: determining the left boundary coordinates of the overlapping region between the second and first images based on the vertex coordinates of the second image; determining the width of the overlapping region based on the image width of the first image and the left boundary coordinates; determining the fusion weight corresponding to any overlapping position of the first and second overlapping images within the overlapping region based on the left boundary coordinates, the width of the overlapping region, and the position coordinates of the overlapping position; determining the pixel value of each pixel in the overlapping region based on the fusion weight of each overlapping position, the coordinate values ​​of each pixel in the first and second overlapping images, and the pixel values ​​of all pixels in the overlapping region; determining the overlapping image of the overlapping region based on the pixel values ​​of all pixels in the overlapping region; and determining the stitched image based on the image outside the overlapping region of the first and transformed images and the overlapping image of the overlapping region.

[0009] One possible implementation involves establishing the projection mapping matrix as follows: acquiring a first chessboard image of the chessboard region captured by a first camera and a second chessboard image of the chessboard region captured by a second camera. The chessboard region is located in an overlapping area. For any one of the first and second chessboard images, corrected corner points of the chessboard region are extracted from all valid corner points of the chessboard image. A world coordinate system is constructed based on the corner point positions of the corrected corner points in the chessboard region, and the corner point coordinates of all corner points in the chessboard region in the world coordinate system are determined. Based on the corner point coordinates of all corner points in the first and second chessboard images, and the correspondence between all corner points in the first and second chessboard images, the projection mapping matrix is ​​determined.

[0010] One possible implementation involves extracting corrected corner points from all valid corner points of a checkerboard image, including: performing a masking process on the checkerboard image to obtain a mask image; selecting a target image from the checkerboard image that corresponds to the mask image; performing corner detection on the target image to obtain all corner points in the target image; and selecting corrected corner points from all corner points in the target image.

[0011] One possible implementation involves selecting correction corner points from all corner points in the target image, including: for any corner point in the target image, extracting the corner point with the largest response value within a preset pixel range centered on that corner point as the target corner point; for any one of the extracted target corner points, constructing a local image region of the target size centered on that target corner point; flipping the local image region based on a preset flip angle to obtain a flipped image region; calculating the difference value between each pixel in the local image region and the flipped image region; and determining the pixel with the largest difference value as the correction corner point.

[0012] One possible implementation involves calculating the difference between each pixel in the local image region and the flipped image region, including:

[0013] P = (i+m,j+n)).

[0014] Where m and n represent the image range of the local image region and the flipped image region, respectively. This is used to represent the pixel value of each pixel in a local image region. (i+m,j+n) is used to represent the pixel value of each pixel in the flipped image region.

[0015] One possible implementation involves determining the projection mapping matrix based on the corner coordinates of all corner points in the first chessboard image, the corner coordinates of all corner points in the second chessboard image, and the correspondence between all corner points in the first and second chessboard images, including:

[0016] .

[0017] in, Here are the corner coordinates of the corner points of the first chessboard grid image. The coordinates of the corner point of the second chessboard image that corresponds to the corner point of the first chessboard image. This is the projection mapping matrix.

[0018] Secondly, embodiments of this application provide an image stitching device applied to a binocular camera, which includes a first camera and a second camera. The images acquired by the first camera and the second camera include overlapping areas that are covered by each other. The device includes: an acquisition module, a transformation module, a fusion module, and a stitching module.

[0019] The acquisition module is used to acquire a first image captured by the first camera and a second image captured by the second camera when shooting a target area using a binocular camera.

[0020] The transformation module is used to perform perspective transformation on the second image using a pre-determined projection mapping matrix to obtain a transformed image.

[0021] The fusion module is used to fuse the first image with the transformed image to obtain an initial fused image.

[0022] The stitching module is used to stitch together the first overlapping image and the second overlapping image of the initial fused image to obtain a stitched image. The first overlapping image and the second overlapping image are images of the overlapping region. The stitched image includes the image of the overlapping region as well as the image outside the overlapping region between the first image and the transformed image.

[0023] Thirdly, embodiments of this application provide an image stitching device that has the function of implementing the image stitching method of the first aspect or any possible implementation of the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the image stitching method described in the first aspect or any possible implementation thereof.

[0025] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, enable the computer to execute the image stitching method described in the first aspect or any possible implementation thereof.

[0026] The technical effects of any of the second to fifth aspects can be referred to the technical effects of the possible implementations in the first aspect, and will not be repeated here. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 A schematic diagram of the structure of a binocular camera provided in an embodiment of this application;

[0029] Figure 2 An example image of a binocular camera provided in this application embodiment;

[0030] Figure 3 A schematic flowchart of an image stitching method provided in an embodiment of this application;

[0031] Figure 4 A schematic diagram of an image stitching device provided in an embodiment of this application;

[0032] Figure 5 This is a schematic diagram of an image stitching system provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0034] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0035] In related technologies, image stitching methods typically rely on precise camera calibration to obtain the camera's intrinsic and extrinsic parameters, and then calculate the homography matrix or projection mapping matrix between images to achieve image alignment and stitching. However, this image stitching method has several shortcomings, mainly in the following aspects:

[0036] Camera calibration requires precise acquisition of the camera's intrinsic and extrinsic parameters. This typically involves complex mathematical models and numerous data acquisition and processing steps, such as using calibration boards (e.g., checkerboard or dot arrays) for multi-view shooting and calculating the camera's intrinsic parameters (focal length, principal point coordinates, etc.) and extrinsic parameters (rotation matrix and translation vector) through optimization algorithms. This process is not only time-consuming and labor-intensive but also easily affected by environmental factors (e.g., changes in lighting, reflection interference, etc.), leading to decreased calibration accuracy and consequently affecting the quality and accuracy of the stitched images.

[0037] Image matching and fusion are two crucial steps in image stitching. Traditional image matching algorithms (such as feature point matching and optical flow methods) typically require complex feature extraction and matching calculations, which are computationally intensive, especially when processing high-resolution or multi-view images, significantly increasing computation time. Furthermore, image fusion algorithms (such as weighted averaging and multi-resolution fusion) also require fine processing of overlapping areas to avoid obvious stitching artifacts. The complexity of these algorithms leads to slow data transmission speeds, particularly in applications with high real-time requirements (such as autonomous driving and industrial online inspection), where the high response latency of binocular cameras makes it difficult to meet practical needs.

[0038] While traditional image stitching methods can achieve image alignment and fusion to a certain extent, they still have shortcomings in terms of the visual effect of the stitched images. For example, obvious stitching marks may appear in the overlapping areas of the stitched images, affecting the naturalness and aesthetics of the images. In addition, due to the large amount of computation, the generation speed of stitched images is slow, making it difficult to meet the needs of applications with high real-time requirements.

[0039] Based on this, embodiments of this application provide an image stitching method applied to a binocular camera, which includes a first camera and a second camera. The images captured by the first camera and the second camera cover an overlapping area. The method includes: acquiring a first image captured by the first camera and a second image captured by the second camera when photographing a target area using the binocular camera; performing a perspective transformation on the second image using a predetermined projection mapping matrix to obtain a transformed image; fusing the first image and the transformed image to obtain an initial fused image; and stitching together a first overlapping image and a second overlapping image of the initial fused image to obtain a stitched image. The first and second overlapping images are images of the overlapping area. The stitched image includes images of the overlapping area and images outside the overlapping area of ​​the first image and the transformed image.

[0040] The image stitching method provided in this application utilizes perspective transformation and fusion algorithms to process the second image during the image stitching process and then fuses it with the first image, achieving fast and high-quality image stitching. This method not only reduces computational load and improves data transmission speed but also effectively solves the problem of obvious stitching artifacts in traditional stitching methods, resulting in more natural edge transitions and a more complete and aesthetically pleasing visual effect in the stitched image. Furthermore, the calibration process of this projection mapping matrix does not require complex calculations of camera parameters, simplifying the calibration process, reducing sensitivity to environmental factors, and improving the accuracy and stability of the calibration.

[0041] The methods provided in the embodiments of this application will now be described in conjunction with the specific accompanying drawings.

[0042] On one hand, embodiments of this application provide a binocular camera. For example... Figure 1 As shown, the binocular camera 100 may include a first camera 101 and a second camera 102.

[0043] The first camera 101 is used to capture a portion of the scene in the target area when the binocular camera 100 is used to photograph the target area, thereby obtaining a first image. The second camera 102 is used to capture a portion of the scene in the target area when the binocular camera 100 is used to photograph the target area, thereby obtaining a second image. The first image captured by the first camera 101 and the second image captured by the second camera 102 partially overlap in the overlapping area.

[0044] The binocular camera 100 is used to stitch the first image and the second image together using the image stitching method provided in this application embodiment after acquiring the first image and the second image, to obtain a stitched image.

[0045] For example, such as Figure 2 As shown, Figure 2This is an example image of a binocular camera used in an embodiment of this application. Taking this example image as an example, the target area is the shooting area covered by the binocular camera 100. This shooting area includes a first area and a second area. The first area is captured by the first camera 101, covering a portion of the scene in the target area. The second area is captured by the second camera 102, covering a portion of the scene in the target area. There is an overlapping area between the first and second areas.

[0046] On the one hand, embodiments of this application provide an image stitching method, which can be performed by... Figure 1 The illustrated stereo camera operation. (Example) Figure 1 As shown, the method may include the following steps.

[0047] S301, when using a binocular camera to photograph a target area, acquire a first image captured by the first camera and a second image captured by the second camera.

[0048] Specifically, when using a stereo camera to photograph a target area, the first and second cameras in the stereo camera system first focus on the target area. After focusing on the target area, the first and second cameras simultaneously capture images of the target area. The first camera captures a first portion of the target area, generating a first image. The second camera captures a second portion of the target area, generating a second image. The first and second images each contain partial scene information of the target area, and there is an overlapping area between the first and second images.

[0049] S302, perform perspective transformation on the second image using a predetermined projection mapping matrix to obtain the transformed image.

[0050] The transformation of the image includes the image size and the pixel value of each pixel.

[0051] One possible implementation involves using a projection mapping matrix to perform a perspective transformation on the vertex coordinates of the second image, obtaining the transformed vertex coordinates. Based on the transformed vertex coordinates, the image size of the transformed image is determined. For any pixel in the transformed image, a perspective transformation is performed on the pixel's first pixel coordinates using the projection mapping matrix, obtaining the pixel's second pixel coordinates. Based on the pixel value of each pixel in the second image and the second pixel coordinates of all pixels in the transformed image, the pixel value of each pixel in the transformed image is determined.

[0052] For example, after acquiring the first image imgL captured by the first camera and the second image imgR captured by the second camera, a perspective transformation is performed on the coordinates of the four vertices of the second image imgR using a projection mapping matrix, resulting in the transformed coordinates of the four vertices as follows: , , , Based on the coordinates of the Q transformed vertices, the size of the transformed image `imgTrans` can be determined. This leads to the creation of a blank space with a size of The imgTrans image.

[0053] The formula for perspective transformation is as follows:

[0054]

[0055] Based on this perspective transformation formula, the second pixel coordinates of the pixel after perspective transformation can be obtained by normalization:

[0056]

[0057] in, Let (x, y) be a pre-established projection mapping matrix. (x, y) represents the first pixel coordinates of each pixel in the first image. (x', y') represents the second pixel coordinates of each pixel in the transformed image.

[0058] Furthermore, the coordinates of the second pixel in the imgTrans image are subjected to perspective transformation and combined with bilinear interpolation to obtain the corresponding coordinate position of imgR.

[0059] The formula for perspective transformation is as follows:

[0060]

[0061]

[0062] The formula after combining bilinear interpolation is as follows:

[0063] ,

[0064] in, Let x, y, be the coordinates of the four adjacent pixels (x, y). Then the corresponding interpolation weights are: .

[0065] Furthermore, by processing all pixels in the imgTrans image and all pixels in the imgR image using the following formula, the pixel values ​​of all pixels in the imgTrans image can be obtained.

[0066] +

[0067] in, This indicates that the coordinates in the imgTrans image are... The pixel value of the pixel.

[0068] The projection mapping matrix is ​​established based on the checkerboard area marked in the overlapping region.

[0069] Specifically, the process of establishing the projection mapping matrix involves acquiring a first chessboard image of the chessboard region captured by a first camera and a second chessboard image of the chessboard region captured by a second camera. The chessboard region is located in an overlapping area. For either the first or second chessboard image, the corrected corner points of the chessboard region are extracted from all valid corner points of the chessboard image.

[0070] One possible implementation involves extracting corrected corner points from a checkerboard region of an image by performing a masking process on the checkerboard image to obtain a mask image. Then, a target image corresponding to the mask image is selected from the checkerboard image. Corner detection is performed on the target image to obtain all corner points. Corner points are pixels in the target image with response values ​​greater than a preset threshold. Finally, corrected corner points are selected from all corner points in the target image.

[0071] In this possible implementation, the process of selecting correction corner points from all corner points in the target image can be as follows: For any corner point in the target image, extract the corner point with the largest response value from a preset pixel range centered on that corner point as the target corner point. For any of the extracted target corner points, construct a local image region of the target size centered on that target corner point. Flip the local image region based on a preset flip angle to obtain a flipped image region. Calculate the difference value of each pixel in the local image region and the flipped image region. Determine the pixel with the largest difference value as the correction corner point.

[0072] Furthermore, after extracting the corrected corner points of the checkerboard region from all valid corner points of the checkerboard image, a world coordinate system is constructed based on the corner positions of the corrected corner points in the checkerboard region, and the corner coordinates of all corner points in the checkerboard region in the world coordinate system are determined. Based on the corner coordinates of all corner points in the first checkerboard image, the corner coordinates of all corner points in the second checkerboard image, and the correspondence between all corner points in the first checkerboard image and all corner points in the second checkerboard image, the projection mapping matrix is ​​determined.

[0073] For example, a binocular camera acquires a first checkerboard image. Second chessboard image The checkerboard pattern is located in the overlapping area of ​​the two cameras, meaning that the checkerboard area can be captured by both the first and second cameras.

[0074] A semantic segmentation network is used to obtain mask images of the chessboard grid positions, resulting in the mask image corresponding to the first chessboard grid image. The mask image corresponding to the second checkerboard image The mask image ranges from 0 to 1. The mask image... The first target image is obtained by setting the area outside the first checkerboard image processing mask to zero. Simultaneously, the mask image The second target image is obtained by setting the area outside the second checkerboard image processing mask to zero. This process uses a mask image to preserve the pixel values ​​of the checkerboard area while setting the pixel values ​​of other areas to zero. This separates the checkerboard from the background, highlighting the checkerboard area and facilitating subsequent corner detection. This process can be implemented using the following formula.

[0075]

[0076] Where (i, j) are used to represent the pixel coordinates of each pixel in the first mask image, the first target image, and the first checkerboard image, respectively.

[0077] It should be noted that the calculation method for the second image is the same as that for the first image, and this application will not elaborate on this.

[0078] Furthermore, the Harris method was used to separately analyze... and Process the image to obtain all corner points of the chessboard pattern.

[0079] by For example, firstly, regarding Perform Sobel processing to obtain and Construct matrix M, and calculate each element of M within the window using the following formula.

[0080]

[0081] To determine whether an element within a window is a corner, a response value R is proposed. The calculation process for this response value R is as follows.

[0082]

[0083] in, , Let be the determinant of the matrix. = , Let be the trace of the matrix. These are empirical parameters. It can be 0.04.

[0084] Furthermore, by comparing the value of R with a preset threshold, it can be determined whether a pixel is a corner point; this preset threshold can be 0. If so, then the pixel is a corner point. If so, then that pixel is an edge point. If the pixel is flat, then the pixel is a flat region.

[0085] Furthermore, when using the Harris corner detection method, multiple qualifying corners may appear at a given corner location. These multiple qualifying corners need to be filtered to ensure that only one corrected corner is calculated at each checkerboard location. First, multiple corners may be detected at the black-and-white boundary of the checkerboard. To ensure the accuracy of subsequent processing, these corners need to be filtered to remove redundant corners. This filtering method involves retaining only one corner within a ±5 pixel range around each detected corner, discarding the rest to obtain the valid corners for the checkerboard area. This ensures that multiple corners do not appear within a local image region.

[0086] Then, using each of the selected valid corner points as the center, select a size of... Pixel area crop. For this area. The crop is processed as follows.

[0087] This area Rotate the crop function 90 degrees to get the flipped area. .right crop and The difference value of each effective corner point can be obtained by calculating the difference of each pixel using the following formula.

[0088] P = (i+m,j+n)),

[0089] Furthermore, the corner point with the largest difference value among all valid corner points is selected as the correction corner point.

[0090] After selecting the corrected corner points, a world coordinate system is constructed based on the position of the corrected corner points in the checkerboard. The valid corner points of the checkerboard determined by the first and second cameras are mapped to the constructed world coordinate system, and the corner points of the first and second cameras are mapped to the same world coordinate system to determine the one-to-one correspondence between the corner points of the first and second images, so as to ensure that the first and second images can be accurately aligned.

[0091] Using the registered corner points, the projection mapping matrix H is calculated. Assume the corner points of the first camera are... (x1, y1, 1), the corner points of the second camera are (x2, y2, 1). and The mapping relationship between them is as follows.

[0092]

[0093] Specifically,

[0094]

[0095] Finally, by using the least squares method and the effective corner points of all registered first and second images, the value of H can be determined, i.e., the elements in H can be found. .

[0096] This process effectively filters and corrects corner points, constructs a world coordinate system, and calculates the projection mapping matrix, providing a foundation for accurate image stitching from binocular cameras. Furthermore, determining this projection mapping matrix eliminates the need for complex camera parameter calculations, simplifying the calibration process, reducing sensitivity to environmental factors, and improving the accuracy and stability of the calibration.

[0097] S303, fuse the first image with the transformed image to obtain the initial fused image.

[0098] For example, copying the imgTrans image to a blank image. Then copy the imgL file to the middle. From the image, we can obtain the initial fused image after fusing the first image and the transformed image. .

[0099] S304, the first overlapping image of the initial fused image and the second overlapping image of the initial fused image are stitched together to obtain a stitched image.

[0100] The first and second overlapping images are images of the overlapping region. The stitched image includes images of the overlapping region as well as images outside the overlapping region between the first and transformed images.

[0101] Specifically, based on the vertex coordinates of the second image, the left boundary coordinates of the overlapping region between the second and first images are determined. Then, combining the image width of the first image and the left boundary coordinates, the width of the overlapping region is calculated. For any overlapping position of the first and second overlapping images within the overlapping region, the fusion weight corresponding to the overlapping position is determined based on the left boundary coordinates, the width of the overlapping region, and the position coordinates of the overlapping position. Based on the fusion weight of each overlapping position, the coordinate values ​​of each pixel in the first and second overlapping images, the pixel value of each pixel in the overlapping region is determined. Based on the pixel values ​​of all pixels in the overlapping region, the overlapping image of the overlapping region is determined. Based on the image outside the overlapping region of the first and transformed images and the overlapping image of the overlapping region, the stitched image is determined.

[0102] For example, the first overlapping image and the second overlapping image are stitched together using a gradient mask. The specific calculation formula is as follows:

[0103] First, the width of the overlapping region is calculated by combining the image width of the first image and the coordinates of the left boundary position.

[0104] Overlap = w -

[0105] Here, Overlap represents the width of the overlapping region. W represents the image width of the first image. Used to represent the coordinates of the left boundary of the overlapping region.

[0106] Furthermore, for any overlapping position of the first overlapping image and the second overlapping image in the overlapping region, the fusion weight corresponding to the overlapping position is determined based on the coordinates of the left boundary position, the width of the overlapping region, and the position coordinates of the overlapping position.

[0107]

[0108] in, Used to represent fusion weights. This is used to represent the x-coordinate of each pixel in the overlapping region. The value of x ranges from [min( , ),min( , )+Overlap.

[0109] Then, based on the fusion weight at each overlapping position, the coordinates of each pixel in the first overlapping image, and the coordinates of each pixel in the second overlapping image, the pixel value of each pixel in the overlapping region is determined. Based on the pixel values ​​of all pixels in the overlapping region, the overlapping image of the overlapping region is determined.

[0110]

[0111] in, This is an overlapping image.

[0112] Then, the images in the first image excluding the overlapping images, the images in the second image excluding the overlapping images, and the overlapping images are stitched together to obtain the stitched image.

[0113] The above primarily describes the solutions provided in the embodiments of this application from the perspective of the working principle of the device. It is understood that, in order to achieve the above functions, the image stitching device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] This application embodiment can divide the image stitching device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module.

[0115] It should be noted that the module division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. When dividing functional modules according to their respective functions, Figure 4 A schematic diagram illustrating a possible composition of the image stitching device involved in the above and embodiment examples is shown. For example... Figure 4 As shown, the image stitching device 400 may include: an acquisition module 401, a transformation module 402, a fusion module 403, and a stitching module 404.

[0116] The acquisition module 401 is used to support the image stitching device 400 in performing operations. Figure 3 S301 in the illustrated image stitching method.

[0117] Transformation module 402 is used to support the image stitching device 400 in performing its functions. Figure 3 S302 in the illustrated image stitching method.

[0118] The fusion module 403 is used to support the image stitching device 400 in performing its functions. Figure 3S303 in the illustrated image stitching method.

[0119] The stitching module 404 is used to support the image stitching device 400 in performing its functions. Figure 3 S304 in the illustrated image stitching method.

[0120] One possible implementation involves transforming the image, including its size and the pixel value of each pixel. Specifically, the image stitching apparatus provided in this application can be used to perform perspective transformation on the vertex coordinates of a second image using a projection mapping matrix to obtain the transformed vertex coordinates. Based on the transformed vertex coordinates, the image size of the transformed image is determined. For any pixel in the transformed image, a perspective transformation is performed on the first pixel coordinates using the projection mapping matrix to obtain the second pixel coordinates. Based on the pixel value of each pixel in the second image and the second pixel coordinates of all pixels in the transformed image, the pixel value of each pixel in the transformed image is determined.

[0121] One possible implementation involves the device specifically used to determine the left boundary coordinates of the overlapping region between the second and first images based on the vertex coordinates of the second image. The width of the overlapping region is determined based on the image width of the first image and the left boundary coordinates. For any overlapping position between the first and second overlapping images within the overlapping region, a fusion weight is determined based on the left boundary coordinates, the width of the overlapping region, and the position coordinates of the overlapping position. The pixel value of each pixel in the overlapping region is determined based on the fusion weight of each overlapping position, the coordinate values ​​of each pixel in the first and second overlapping images, and the coordinate values ​​of each pixel in the second overlapping image. The overlapping image of the overlapping region is determined based on the pixel values ​​of all pixels in the overlapping region. Finally, a stitched image is determined based on the image outside the overlapping region of the first and transformed images and the overlapping image of the overlapping region.

[0122] One possible implementation involves the device acquiring a first chessboard image of the chessboard region captured by a first camera and a second chessboard image of the same region captured by a second camera. The chessboard regions are located in an overlapping area. For either the first or second chessboard image, correction corner points of the chessboard region are extracted from all valid corner points of the chessboard image. A world coordinate system is constructed based on the corner positions of the correction corner points in the chessboard region, and the corner coordinates of all corner points in the chessboard region within the world coordinate system are determined. Based on the corner coordinates of all corner points in the first and second chessboard images, and the correspondence between all corner points in the first and second chessboard images, a projection mapping matrix is ​​determined.

[0123] One possible implementation involves using the device to perform masking on a checkerboard image to obtain a mask image of the checkerboard image. A target image corresponding to the mask image is then selected from the checkerboard image. Corner detection is performed on the target image to obtain all corner points. Corner points are pixels in the target image whose response value is greater than a preset threshold. Corrected corner points are then selected from all corner points in the target image.

[0124] One possible implementation involves the device extracting the corner point with the largest response value from a preset pixel range, centered on any corner point in a target image, as the target corner point. For any of the extracted target corner points, a local image region of the target size is constructed, centered on the target corner point. The local image region is then flipped based on a preset flip angle to obtain a flipped image region. The difference value between each pixel in the local image region and the flipped image region is calculated. The pixel with the largest difference value is determined as the correction corner point.

[0125] One possible implementation is that the device can be used to calculate the difference value of each pixel in the local image region and the flipped image region using the following formula.

[0126] P = (i+m,j+n)).

[0127] Where m and n represent the image range of the local image region and the flipped image region, respectively. This is used to represent the pixel value of each pixel in a local image region. (i+m,j+n) is used to represent the pixel value of each pixel in the flipped image region.

[0128] One possible implementation is that the device can be used to determine the projection mapping matrix based on the corner coordinates of all corner points in the first chessboard image, the corner coordinates of all corner points in the second chessboard image, and the correspondence between all corner points in the first chessboard image and all corner points in the second chessboard image, using the following formula.

[0129] .

[0130] in, Here are the corner coordinates of the corner points of the first chessboard grid image. The coordinates of the corner point of the second chessboard image that corresponds to the corner point of the first chessboard image. This is the projection mapping matrix.

[0131] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0132] The image stitching device 400 provided in this application embodiment is used to perform the above-described... Figure 3 The image stitching method shown can achieve the same effect as the image stitching method described above.

[0133] This application also provides an image stitching device that can perform the image stitching method and related steps described in the above method embodiments.

[0134] This application also provides a computer-readable storage medium storing instructions thereon, which, when executed, perform the image stitching method and related steps described in the above method embodiments.

[0135] This application also provides a computer program product that, when run on a computer, causes the computer to execute the image stitching method and related steps described in the above method embodiments.

[0136] In some embodiments, the methods shown in this application can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0137] This application also provides an image stitching system 500, such as... Figure 5 As shown, the image stitching system 500 includes at least one processor 501 and at least one interface circuit 502.

[0138] As an example, when the image stitching system 500 includes a processor and an interface circuit, the processor can be... Figure 5 The processor 501 shown in the solid box (or the processor 501 shown in the dashed box) can be an interface circuit. Figure 5 The interface circuit 502 is shown in the solid box (or the dashed box). When the image stitching system 500 includes two processors and two interface circuits, the two processors include... Figure 5 The processor 501 shown in the solid box and the processor 501 shown in the dashed box, these two interface circuits include Figure 5 Interface circuit 502 is shown in both solid and dashed boxes. No limitations are imposed on this.

[0139] The processor 501 and the interface circuit 502 can be interconnected via a line. For example, the interface circuit 502 can be used to receive signals. Alternatively, the interface circuit 502 can be used to send signals to other devices (such as the processor 501). For instance, the interface circuit 502 can read computer instructions stored in memory and send those instructions to the processor 501. The processor 501 executes the instructions and, in conjunction with input / output devices, implements the various steps in the above embodiments, such as implementing... Figure 3 The illustrated method embodiments demonstrate the various steps performed. Of course, this image stitching system may also include other discrete components, and this application embodiment does not specifically limit this.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0142] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0144] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to it, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image stitching method, characterized in that, Applied to a stereo camera, the stereo camera comprising a first camera and a second camera, wherein the image acquired by the first camera partially overlaps with the image acquired by the second camera; the method includes: When using the binocular camera to photograph the target area, a first image captured by the first camera and a second image captured by the second camera are obtained; The second image is transformed using a predetermined projection mapping matrix to obtain a transformed image; The first image and the transformed image are fused to obtain an initial fused image; The first overlapping image of the initial fused image and the second overlapping image of the initial fused image are stitched together to obtain a stitched image; the first overlapping image and the second overlapping image are images of the overlapping region; the stitched image includes the image of the overlapping region and the image outside the overlapping region of the first image and the transformed image; The process of establishing the projection mapping matrix includes: acquiring a first chessboard image of the chessboard region captured by the first camera and a second chessboard image of the chessboard region captured by the second camera; the chessboard region is located in the overlapping region; for any one of the first and second chessboard images, performing masking processing on the chessboard image to obtain a mask image of the chessboard image; filtering out the target image corresponding to the mask image from the chessboard image; performing corner detection on the target image to obtain all corner points in the target image; the corner point is a pixel in the target image whose response value is greater than a preset threshold; For any corner point in the target image, extract the corner point with the largest response value from a preset pixel range centered on the corner point as the target corner point; for any target corner point among all extracted target corner points, construct a local image region of the target size centered on the target corner point; flip the local image region based on a preset flip angle to obtain a flipped image region; calculate the difference value of each pixel point between the local image region and the flipped image region; determine the pixel point with the largest difference value as the corrected corner point; A world coordinate system is constructed based on the corner positions of the corrected corner points in the chessboard area, and the corner coordinates of all corner points in the chessboard area in the world coordinate system are determined; the projection mapping matrix is ​​determined based on the corner coordinates of all corner points in the first chessboard image, the corner coordinates of all corner points in the second chessboard image, and the correspondence between all corner points in the first chessboard image and all corner points in the second chessboard image.

2. The method according to claim 1, characterized in that, The transformed image includes the image size and the pixel value of each pixel; the step of performing perspective transformation on the second image using a pre-determined projection mapping matrix to obtain the transformed image includes: The vertex coordinates of the second image are transformed using the projection mapping matrix to obtain the transformed vertex coordinates; The image size of the transformed image is determined based on the transformed vertex coordinates; For any pixel in the transformed image, the first pixel coordinates of the pixel are transformed by the projection mapping matrix to obtain the second pixel coordinates of the pixel. The pixel value of each pixel in the transformed image is determined based on the pixel value of each pixel in the second image and the second pixel coordinates of all pixels in the transformed image.

3. The method according to claim 1, characterized in that, The step of stitching together the first overlapping image of the initial fused image with the second overlapping image of the initial fused image to obtain a stitched image includes: Based on the vertex coordinates of the second image, determine the coordinates of the left boundary of the overlapping area between the second image and the first image; The width of the overlapping region is determined based on the image width of the first image and the coordinates of the left boundary position; For any overlapping position of the first overlapping image and the second overlapping image in the overlapping region, the fusion weight corresponding to the overlapping position is determined based on the left boundary position coordinates, the width of the overlapping region, and the position coordinates of the overlapping position. The pixel value of each pixel in the overlapping region is determined based on the fusion weight of each overlapping position, the coordinate value of each pixel in the first overlapping image, and the coordinate value of each pixel in the second overlapping image. The overlapping image of the overlapping region is determined based on the pixel values ​​of all pixels in the overlapping region; The stitched image is determined based on the image outside the overlapping area of ​​the first image and the transformed image and the overlapping image of the overlapping area.

4. The method according to claim 1, characterized in that, The calculation of the difference value of each pixel in the local image region and the flipped image region includes: P = (i+m,j+n)); Wherein, m and n represent the image range of the local image region and the flipped image region, respectively. , used to represent the pixel value of each pixel in the local image region, the (i+m, j+n) is used to represent the pixel value of each pixel in the flipped image region.

5. The method according to claim 1, characterized in that, The step of determining the projection mapping matrix based on the corner coordinates of all corner points in the first chessboard image, the corner coordinates of all corner points in the second chessboard image, and the correspondence between all corner points in the first chessboard image and all corner points in the second chessboard image includes: ； Among them, the The corner coordinates of the corner points of the first chessboard image are given. The coordinates of the corner point of the second chessboard image corresponding to the corner point of the first chessboard image; Let be the projection mapping matrix.

6. An image stitching device, characterized in that, An apparatus for use with a binocular camera, the binocular camera comprising a first camera and a second camera, wherein the images captured by the first camera and the images captured by the second camera include a common overlapping area; the apparatus comprises: The acquisition module is used to acquire a first image captured by the first camera and a second image captured by the second camera when shooting a target area using the binocular camera; The transformation module is used to perform perspective transformation on the second image using a predetermined projection mapping matrix to obtain a transformed image; The fusion module is used to fuse the first image and the transformed image to obtain an initial fused image; A stitching module is used to stitch together a first overlapping image of the initial fused image and a second overlapping image of the initial fused image to obtain a stitched image; the first overlapping image and the second overlapping image are images of the overlapping region; the stitched image includes images of the overlapping region and images outside the overlapping region of the first image and the transformed image; The process of establishing the projection mapping matrix includes: acquiring a first chessboard image of the chessboard region captured by the first camera and a second chessboard image of the chessboard region captured by the second camera; the chessboard region is located in the overlapping region; for any one of the first and second chessboard images, performing masking processing on the chessboard image to obtain a mask image of the chessboard image; filtering out the target image corresponding to the mask image from the chessboard image; performing corner detection on the target image to obtain all corner points in the target image; the corner point is a pixel in the target image whose response value is greater than a preset threshold; For any corner point in the target image, extract the corner point with the largest response value from a preset pixel range centered on the corner point as the target corner point; for any target corner point among all extracted target corner points, construct a local image region of the target size centered on the target corner point; flip the local image region based on a preset flip angle to obtain a flipped image region; calculate the difference value of each pixel point between the local image region and the flipped image region; determine the pixel point with the largest difference value as the corrected corner point; A world coordinate system is constructed based on the corner positions of the corrected corner points in the chessboard area, and the corner coordinates of all corner points in the chessboard area in the world coordinate system are determined; the projection mapping matrix is ​​determined based on the corner coordinates of all corner points in the first chessboard image, the corner coordinates of all corner points in the second chessboard image, and the correspondence between all corner points in the first chessboard image and all corner points in the second chessboard image.

7. An image stitching device, characterized in that, The image stitching device includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, and the processor executing the machine-executable instructions to implement the image stitching method according to any one of claims 1 to 5.

Images