Image stitching method, device, equipment, medium and program product

Abstract

The embodiment of the present application provides an image stitching method, device, equipment, medium and program product, the image stitching method includes: extracting feature points of a reference image and an image to be stitched for a video frame captured by a camera array; determining the world coordinates of each feature point based on an intrinsic parameter matrix and an extrinsic parameter matrix of camera calibration in the camera array; determining the homography transformation matrix of the image to be stitched relative to the reference image based on the world coordinates of the feature points, the intrinsic parameter matrix and the calibrated extrinsic parameter matrix; performing coordinate transformation on the image to be stitched based on the homography transformation matrix; completing the image stitching operation based on the optimal stitching seam between the target image to be stitched and the reference image, and obtaining a frame of video panorama. A method for determining the homography transformation matrix based on camera internal and external parameters is implemented, which weakens the dependence of image stitching on image features and is suitable for low-texture scenes.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image stitching method, device, equipment, medium and program product. Background Art

[0002] Image stitching technology is a technology that stitches several images with overlapping areas into a larger high-resolution image. Through image stitching, ordinary cameras can obtain a wider viewing angle, such as generating a 360° panoramic image.

[0003] Image stitching usually relies on the extraction and matching of image features. Based on the image features of the overlapping area, the homography matrix between cameras is calculated to convert the images captured by other cameras to the image coordinate system of the selected reference camera to complete image stitching.

[0004] However, the image stitching method that relies on image features cannot be applied to low-texture application scenarios and its application scenarios are limited. Summary of the invention

[0005] The embodiments of the present application provide an image stitching method, apparatus, device, medium and program product, which implement an image stitching method based on camera parameters, weaken the dependence on image features, and have a wide range of application scenarios.

[0006] In a first aspect, an embodiment of the present application provides an image stitching method, comprising:

[0007] Extract feature points of the reference image and the image to be stitched from a video frame captured by a camera array, wherein the camera array includes a plurality of cameras with the same intrinsic parameter matrix; determine the world coordinates of each feature point based on the intrinsic parameter matrix and an extrinsic parameter matrix of camera calibration in the camera array; determine the homography transformation matrix of the image to be stitched relative to the reference image based on the world coordinates of the feature points, the reference and calibrated extrinsic matrix; perform coordinate transformation on the image to be stitched based on the homography transformation matrix to obtain a target image to be stitched; and complete an image stitching operation based on an optimal stitching seam between the target image to be stitched and the reference image.

[0008] Optionally, the reference image is acquired by a reference camera in the camera array, and the stitched image is acquired by a target camera in the camera array, and the target camera is another camera in the camera array except the reference camera;

[0009] The determining the world coordinates of each of the feature points based on the intrinsic parameter matrix and the extrinsic parameter matrix of the camera calibration in the camera array includes:

[0010] The world coordinates of each of the feature points are determined based on the intrinsic parameter matrix, the extrinsic parameter matrix of the reference camera calibration, and the extrinsic parameter matrix of the target camera calibration.

[0011] Optionally, determining the homography transformation matrix of the image to be stitched relative to the reference image based on the world coordinates of the feature points, the intrinsic parameter matrix and the calibrated extrinsic parameter matrix includes:

[0012] Based on the nearest neighbor matching algorithm, feature point pairs whose world coordinates match in the image to be stitched and the reference image are determined; based on the image coordinates of each feature point in the feature point pairs on the corresponding image, and the intrinsic parameter matrix and extrinsic parameter matrix of the reference camera, the homography transformation matrix of the image to be stitched relative to the reference image is determined.

[0013] Optionally, determining the homography transformation matrix of the image to be stitched relative to the reference image based on the image coordinates of each feature point in the feature point pair on the corresponding image and the intrinsic parameter matrix and the extrinsic parameter matrix of the reference camera includes:

[0014] Based on a random sampling consensus algorithm, matching feature point pairs are screened; based on the image coordinates of each feature point in the screened feature point pairs on the corresponding image, a first transformation matrix is ​​determined; based on the first transformation matrix, and the intrinsic parameter matrix and extrinsic parameter matrix of the reference camera, the homography transformation matrix is ​​determined.

[0015] Optionally, after determining the homography transformation matrix of the image to be stitched relative to the reference image based on the world coordinates of the feature points, the intrinsic parameter matrix and the calibrated extrinsic parameter matrix, the method further comprises:

[0016] For the video frames subsequently captured by the camera array, based on the homography transformation matrix, coordinate transformation is performed on the subsequent images to be stitched to obtain the subsequent target images to be stitched; for the previous video frame captured by the camera array, a first optimal stitching seam between the previous target image to be stitched and the previous reference image is obtained; it is determined whether there is a moving object at the stitching seam between the subsequent reference image and the subsequent target stitching image; if so, an optimal stitching seam update operation is performed; if not, the first optimal stitching seam is used as the optimal stitching seam to complete the image stitching operation of the subsequent reference image and the subsequent target stitching image.

[0017] Optionally, the determining whether there is a moving object at a joint seam of a subsequent reference image and a subsequent target joint image includes:

[0018] Obtain the gradient value of the pixels at the stitching seam of the previous reference image and the previous target stitching image, recorded as the first gradient value; obtain the gradient value of the pixels at the stitching seam of the subsequent reference image and the subsequent target stitching image, recorded as the second gradient value; and determine whether there is a moving object at the stitching seam of the subsequent reference image and the subsequent target stitching image based on the first gradient value and the second gradient value.

[0019] Optionally, judging whether there is a moving object at a stitching seam of a subsequent reference image and a subsequent target stitching image according to the first gradient value and the second gradient value includes:

[0020] Based on the first gradient value and the second gradient value, the gradient change value of each pixel at the stitching seam of the subsequent reference image and the subsequent target stitching image is calculated to obtain a gradient change set; two gradient change values ​​are selected from the gradient change set as gradient change centers; the distance between the gradient change values ​​in the gradient change set and each of the gradient change centers is calculated; based on the distance, the gradient change values ​​in the gradient change set are divided into two groups; the gradient change center is updated to the average value of the gradient change values ​​in each group; it is determined whether the gradient change centers before and after the update are the same; if the gradient change centers before and after the update are the same, it is determined whether the number of gradient change values ​​in the group with a larger average gradient change value is greater than a preset number; if the number of gradient change values ​​in the group with a larger average gradient change value is greater than a preset number, it is determined that there is a moving object at the stitching seam of the subsequent reference image and the subsequent target stitching image; if the gradient change centers before and after the update are not the same, the step of calculating the distance between the gradient change values ​​in the gradient change set and each gradient change center is returned to.

[0021] Optionally, performing the optimal seam update operation includes:

[0022] Re-searching for an optimal stitching seam from stitching seams of a subsequent reference image and a subsequent target stitching image to obtain a second optimal stitching seam, and updating the optimal stitching seam to be the second optimal stitching seam; and stitching the subsequent reference image and the subsequent target stitching image based on the second optimal stitching seam.

[0023] Optionally, after completing the image stitching operation, the method further includes:

[0024] Displays the video panorama obtained by the image stitching operation.

[0025] In a second aspect, an embodiment of the present application further provides an image stitching device, comprising:

[0026] A feature point extraction module is used to extract feature points of the reference image and the image to be stitched for a video frame captured by a camera array, wherein the camera array includes a plurality of cameras with the same intrinsic parameter matrix; a world coordinate determination module is used to determine the world coordinates of each feature point based on the intrinsic parameter matrix and an extrinsic parameter matrix of camera calibration in the camera array; a transformation matrix determination module is used to determine the homography transformation matrix of the image to be stitched relative to the reference image based on the world coordinates of the feature points, the intrinsic parameter matrix and the calibrated extrinsic parameter matrix; an image transformation module is used to perform coordinate transformation on the image to be stitched based on the homography transformation matrix to obtain a target image to be stitched; and a stitching module is used to complete an image stitching operation based on an optimal stitching seam between the target image to be stitched and the reference image.

[0027] Optionally, the reference image is acquired by a reference camera in the camera array, and the stitched image is acquired by a target camera in the camera array, and the target camera is another camera in the camera array except the reference camera; the feature point extraction module is specifically used to:

[0028] The world coordinates of each of the feature points are determined based on the intrinsic parameter matrix, the extrinsic parameter matrix of the reference camera calibration, and the extrinsic parameter matrix of the target camera calibration.

[0029] Optionally, a transformation matrix determination module includes:

[0030] A feature point pair determination unit is used to determine, based on a nearest neighbor matching algorithm, feature point pairs whose world coordinates match in the image to be stitched and the reference image; a transformation matrix determination unit is used to determine a homography transformation matrix of the image to be stitched relative to the reference image based on the image coordinates of each feature point in the feature point pair on the corresponding image, and the intrinsic parameter matrix and extrinsic parameter matrix of the reference camera.

[0031] Optionally, the transformation matrix determination unit is specifically used to:

[0032] Based on a random sampling consensus algorithm, matching feature point pairs are screened; based on the image coordinates of each feature point in the screened feature point pairs on the corresponding image, a first transformation matrix is ​​determined; based on the first transformation matrix, and the intrinsic parameter matrix and extrinsic parameter matrix of the reference camera, the homography transformation matrix is ​​determined.

[0033] Optionally, the device further comprises:

[0034] A subsequent image transformation module is used to perform coordinate transformation on subsequent images to be stitched based on the homography transformation matrix for the video frames subsequently captured by the camera array to obtain subsequent target images to be stitched; a historical best stitching seam acquisition module is used to acquire the first best stitching seam of the previous target image to be stitched and the previous reference image for the previous video frame captured by the camera array; a stitching seam determination module is used to determine whether there is a moving object at the stitching seam of the subsequent reference image and the subsequent target stitching image; a first subsequent image stitching module is used to use the first best stitching seam as the best stitching seam to complete the image stitching operation of the subsequent reference image and the subsequent target stitching image if there is no moving object at the stitching seam of the subsequent reference image and the subsequent target stitching image; and a second subsequent image stitching module is used to perform the best stitching seam update operation if there is a moving object at the stitching seam of the subsequent reference image and the subsequent target stitching image.

[0035] Optionally, a seam determination module is specifically used for:

[0036] Based on the first gradient value and the second gradient value, the gradient change value of each pixel at the stitching seam of the subsequent reference image and the subsequent target stitching image is calculated to obtain a gradient change set; two gradient change values ​​are selected from the gradient change set as gradient change centers; the distance between the gradient change values ​​in the gradient change set and each of the gradient change centers is calculated; based on the distance, the gradient change values ​​in the gradient change set are divided into two groups; the gradient change center is updated to the average value of the gradient change values ​​in each group; it is determined whether the gradient change centers before and after the update are the same; if the gradient change centers before and after the update are the same, it is determined whether the number of gradient change values ​​in the group with a larger average gradient change value is greater than a preset number, so as to determine that there is a moving object at the stitching seam of the subsequent reference image and the subsequent target stitching image when the number of gradient change values ​​in the group with a larger average gradient change value is greater than the preset number; if the gradient change centers before and after the update are not the same, the step of calculating the distance between the gradient change values ​​in the gradient change set and each gradient change center is returned to.

[0037] Optionally, the second subsequent image stitching module is specifically used to:

[0038] If there is a moving object at the stitching seam between the subsequent reference image and the subsequent target stitching image, re-search the best stitching seam from the stitching seams of the subsequent reference image and the subsequent target stitching image to obtain a second best stitching seam, and update the best stitching seam to be the second best stitching seam; based on the second best stitching seam, stitch the subsequent reference image and the subsequent target stitching image.

[0039] Optionally, the device further comprises:

[0040] The panoramic image display module is used to display the video panoramic image obtained by the image stitching operation after the image stitching operation is completed.

[0041] In a third aspect, an embodiment of the present application further provides an image stitching device, comprising a memory and at least one processor; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the image stitching method provided by any embodiment of the present application.

[0042] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, an image stitching method as provided in any embodiment of the present application is implemented.

[0043] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the image stitching method provided in any embodiment of the present application.

[0044] The image stitching method, apparatus, device, medium and program product provided by the embodiments of the present application provide a method for determining a homography transformation matrix based on internal and external parameters of the camera for panoramic stitching of video frames captured by a camera array, specifically, based on the camera's intrinsic parameter matrix and a calibrated extrinsic parameter matrix, determining the world coordinates of feature points on the extracted reference image and the image to be stitched, and then determining the homography transformation matrix between the camera that captures the image to be stitched and the camera that captures the reference image based on the world coordinates of the feature points, the camera's intrinsic parameter matrix and the extrinsic parameter matrix, so that the image to be stitched is converted into the same image coordinate system as the reference image through the homography transformation matrix, providing a basis for image stitching; and then by searching for the best stitching seam, the reference image and the image to be stitched are stitched, and after the images of the same video frame captured by each camera in the camera array are stitched, a frame of video panoramic image is obtained. Compared with the method of relying on image feature matching to calculate the homography transformation matrix, the calculation method provided by the present application that combines the internal and external parameters of the camera has high accuracy and can better adapt to scenes with low texture or fewer feature points, thereby improving the quality of image stitching in low-texture scenes and reducing ghosting or artifacts in the stitched panorama. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a flow chart of an image stitching method provided in one embodiment of the present application;

[0048] Figure 3 A schematic diagram of a camera array provided for one embodiment of the present application;

[0049] Figure 4 For this application Figure 2 A schematic diagram of the process of step S202 in the illustrated embodiment;

[0050] Figure 5 A flowchart of an image stitching method provided by another embodiment of the present application;

[0051] Figure 6 For this application Figure 5 A schematic diagram of the process of step S502 in the illustrated embodiment;

[0052] Figure 7 A flowchart of an image stitching method provided in yet another embodiment of the present application;

[0053] Figure 8 A schematic diagram of the structure of an image stitching device provided in one embodiment of the present application;

[0054] Fig. 9 A schematic diagram of the structure of an image stitching device provided in one embodiment of the present application.

[0055] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0056] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0057] First, the terms involved in this application are explained:

[0058] World coordinate system: also known as the reference coordinate system, is a three-dimensional coordinate system used to represent the position of objects in the real world.

[0059] Camera coordinate system: a three-dimensional coordinate system used to describe the position of an object from the perspective of a camera. The optical center of the camera is the origin of the coordinate system, and the optical axis of the camera is one of the coordinate axes.

[0060] Image coordinate system: It is a two-dimensional coordinate system used to describe the coordinate system of the image of an object at the focal length of the camera. The origin is the intersection of the camera optical axis and the imaging surface, which is usually the center of the image.

[0061] Seam: The overlapping area when two images are newly stitched together.

[0062] Optimal stitching seam: A curve with the smallest difference determined according to the set rules in the new overlapping area of ​​the two images is used as the new stitching position of the two images.

[0063] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0064] The application scenarios of the embodiments of the present application are explained below:

[0065] In practical applications, high-resolution images with a wide or even panoramic field of view are often used, such as monitoring applications, medical imaging, etc. Since wide-angle cameras are expensive, camera arrays combined with image stitching technology are usually used to capture panoramic images.

[0066] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application, such as Figure 1 As shown, the camera array includes N cameras, and the arrangement of the N cameras is a static array arrangement, that is, the relative position relationship between the cameras is fixed. The camera array collects video through the camera head to obtain a sequence of video frames, and the video frame includes an image collected by each of the N cameras at a corresponding time.

[0067] In some embodiments, images or video frames captured by different cameras in the camera array may be displayed via different displays.

[0068] In other embodiments, the N images corresponding to the same video frame of the camera array can be stitched into a panoramic image through image stitching technology, which is recorded as a video panorama. The video panorama is displayed on a display to facilitate environmental monitoring, image analysis, etc. through the panorama.

[0069] Image stitching is usually divided into two stages. The first stage is mainly responsible for calculating the homography transformation matrix to obtain the stitching template; the second stage stitches subsequent video frames and updates the stitching template based on the stitching template.

[0070] In the related art, in the first stage, the features of the overlapping areas of the images are usually extracted, and the homography transformation matrix is ​​calculated by feature matching to transform the images of the remaining cameras in the camera array to the image coordinate system of the selected reference camera. Then, the panorama is stitched by finding the best stitching seam and multi-band fusion.

[0071] The above-mentioned image stitching method is suitable for scenes where image features are easy to extract and the image details are relatively large. For scenes where image features are difficult to extract, such as low-texture scenes, the accuracy of the homography transformation matrix calculation is low, resulting in ghosting or artifacts in the panorama.

[0072] For panoramic image stitching in low-texture application scenarios, the present application provides an image stitching method, which implements a homography transformation matrix calculation method based on camera internal and external parameters, weakens the dependence on image features, improves the accuracy of homography transformation matrix calculation, and improves the quality of panoramic images in low-texture scenes.

[0073] Figure 2 The following is a flow chart of an image stitching method provided by an embodiment of the present application. The image stitching method can be executed by a device with corresponding data processing capabilities, such as an image stitching device, such as Figure 2 As shown, the image stitching method provided in this embodiment includes the following steps:

[0074] Step S201 : extracting feature points of a reference image and an image to be stitched from video frames captured by a camera array.

[0075] The reference image and the image to be stitched are video frames captured by a camera array, such as the first video frame or a specified video frame. The camera array includes multiple cameras with the same intrinsic parameter matrix. The reference image is captured by a reference camera in the camera array, and the image to be stitched is captured by a target camera in the camera array. The target camera is another camera in the camera array except the reference camera.

[0076] In some embodiments, the camera array may be arranged in a matrix.

[0077] In other embodiments, the camera array may be arranged in a ring shape to surround the target scene to be photographed, thereby achieving 360° panoramic shooting.

[0078] The reference camera may be any camera in the camera array. The image captured by the reference camera is recorded as the reference image, and the images captured by the remaining cameras in the camera array are the images to be stitched.

[0079] Adjacent cameras in the camera array have overlapping fields of view, so that images captured by adjacent cameras include overlapping areas. The reference image and the image to be stitched have overlapping areas, such as images captured by adjacent cameras.

[0080] In some embodiments, the cameras in the camera array may be of the same model and have the same internal parameters, where the internal parameters of the camera include parameters related to the camera's own characteristics, such as focal length and pixel size.

[0081] For each video frame captured by the camera array, the image captured by the reference camera in the video frame can be used as the reference image, the image captured by the adjacent camera of the reference camera can be used as the image to be stitched, or the image captured by a camera that has an overlapping field of view with the reference camera can be used as the image to be stitched. The reference image and the image to be stitched are stitched using the image stitching method provided in the present application. When all the images corresponding to the video frame are stitched, a frame of video panorama corresponding to the video frame is obtained.

[0082] For example, Figure 3 A schematic diagram of a camera array provided for one embodiment of the present application, such as Figure 3 As shown, in Figure 3 In the example of a camera array including four cameras, camera C1 can be used as a reference camera, and cameras C2 and C3 can be used as target cameras to perform the first image stitching, stitching the images captured by cameras C1 to C3 together, then camera C3 can be used as a reference camera, and camera C4 can be used as a target camera to perform the second image stitching, stitching the images captured by cameras C1 to C4 together. If the camera's field of view is wide, so that the fields of view of cameras C1 to C4 overlap, then one of cameras C1 to C4 can be used as a reference camera, and the remaining cameras can be used as target cameras to perform image stitching once.

[0083] Specifically, feature points of images captured by cameras in the camera array (including reference images and images to be stitched) can be extracted based on a SURF (Speeded Up Robust Features) algorithm, a SIFT (Scale-invariant features transform) algorithm, or other feature extraction algorithms.

[0084] Step S202: determining the world coordinates of each of the feature points based on the intrinsic parameter matrix and the extrinsic parameter matrix of camera calibration in the camera array.

[0085] The camera's intrinsic parameter matrix is ​​used to represent the mapping from the camera coordinate system to the image coordinate system, while the extrinsic parameter matrix includes a rotation matrix and a translation matrix, which are used to represent the mapping from the camera coordinate system to the world coordinate system. Through the camera's intrinsic parameter matrix and extrinsic parameter matrix, the feature points on the image captured by the camera (in the image coordinate system) can be converted to points in the world coordinate system to obtain the world coordinates of the feature points.

[0086] The extrinsic matrix of the camera in the camera array may be calibrated in any manner, which is not limited in this application, such as by photographing a calibration plate and calibrating the extrinsic matrix of the camera based on the positions of feature points on the calibration plate in the image.

[0087] Optionally, determining the world coordinates of each of the feature points based on the intrinsic parameter matrix and an extrinsic parameter matrix of camera calibration in the camera array includes:

[0088] The world coordinates of each of the feature points are determined based on the intrinsic parameter matrix, the extrinsic parameter matrix of the reference camera calibration, and the extrinsic parameter matrix of the target camera calibration.

[0089] Step S203 : determining a homography transformation matrix of the image to be stitched relative to the reference image based on the world coordinates of the feature points, the intrinsic parameter matrix and the calibrated extrinsic parameter matrix.

[0090] Specifically, based on the world coordinates of the feature points in the two images (the reference image and one of the images to be stitched), the mapping relationship between the two images in the world coordinate system can be obtained, and then combined with the intrinsic parameter matrix and the extrinsic parameter matrix of the reference camera, the mapping relationship in the world coordinate system is converted into a mapping relationship in the image coordinate system, and the mapping relationship between the two images is obtained, that is, the homography transformation matrix is ​​obtained.

[0091] Furthermore, based on the world coordinates, matching feature point pairs in the two images can be found, such as feature points with the same world coordinates, that is, overlapping feature points; based on the coordinates of the feature point pairs in the image coordinate system, the mapping relationship between the two images in the world coordinate system can be calculated.

[0092] A feature point pair includes two feature points, one located on the reference image and the other located on the image to be stitched.

[0093] Optionally, determining a homography transformation matrix of the image to be stitched relative to the reference image based on the world coordinates of the feature points, the intrinsic parameter matrix, and the calibrated extrinsic parameter matrix includes:

[0094] Based on the nearest neighbor matching algorithm, feature point pairs whose world coordinates match in the image to be stitched and the reference image are determined; based on the image coordinates of each feature point in the feature point pairs on the corresponding image, and the intrinsic parameter matrix and extrinsic parameter matrix of the reference camera, the homography transformation matrix of the image to be stitched relative to the reference image is determined.

[0095] Through the nearest neighbor matching algorithm (KNN, K-Nearest Neighbor), find the feature points that match the feature points of the reference image from the feature points of the image to be stitched, obtain multiple feature point pairs, and determine the matching feature point pairs from the multiple feature point pairs, such as feature point pairs with the same world coordinates, or feature point pairs with close world coordinates. Based on the image coordinates of each feature point of the matching feature point pair on the corresponding image, as well as the intrinsic and extrinsic matrix of the reference camera, determine the homography transformation matrix of the target camera relative to the reference camera.

[0096] The feature point matching is performed through the KNN algorithm with high matching accuracy, which improves the accuracy of finding the matching feature point pairs and provides a reliable data basis for the calculation of the homography transformation matrix.

[0097] Step S204: performing coordinate transformation on the images to be stitched based on the homography transformation matrix to obtain a target image to be stitched.

[0098] Based on the calculated homography transformation matrix, coordinate transformation is performed on the pixel points on each image to be stitched to obtain a target image to be stitched, wherein the target image to be stitched and the reference image are in the same image coordinate system.

[0099] Step S205 , completing the image stitching operation based on the optimal stitching seam between the target image to be stitched and the reference image.

[0100] Through the image stitching operation, a set of reference images and target images to be stitched are stitched into a frame of video panorama.

[0101] Specifically, the best stitching seam between the target image to be stitched and the reference image is searched, and based on the best stitching seam, the reference image and the target image to be stitched are stitched to obtain a frame of video panorama.

[0102] In order to optimize the effect of image stitching, the stitching seam needs to be processed. Specifically, the best stitching seam is searched, the reference image and the target image to be stitched are segmented along the best stitching seam, the segmented reference image and the image to be stitched are stitched, and after the images of the video frame captured by each camera are stitched, the video panorama corresponding to the video frame is obtained.

[0103] Since the fields of view of the reference camera and the target camera overlap, there is a seam on the stitched image. The seam corresponds to the overlapping area of ​​the two images and can be a polygonal area or an irregular area.

[0104] Specifically, based on the pixel points on the target image to be stitched and the reference image corresponding to the same image coordinates, the overlapping area of ​​the two images can be determined, and the optimal stitching seam can be searched in the overlapping area.

[0105] For example, an image energy function may be defined and the best stitching seam may be searched in the overlapping area by minimizing the image energy function. The image energy function may be a function of parameters such as color and gradient.

[0106] Any algorithm may be used to search for and update the optimal seam, and this application does not limit this.

[0107] The best stitching seam is a curve with the smallest difference between the reference image and the target image to be stitched in the overlapping area. By searching for the best stitching seam, the stitching seam caused by the grayscale difference and geometric misalignment of the two images can be reduced, thus improving the image stitching quality.

[0108] Optionally, after completing the image stitching operation and obtaining a frame of video panorama, the method further includes:

[0109] The video panorama is displayed.

[0110] Optionally, after completing the image stitching operation, the method further includes:

[0111] Displays the video panorama obtained by the image stitching operation.

[0112] Each frame of the video panorama obtained by background processing can be sent to a display for display, so as to show the panorama to the user.

[0113] The image stitching method provided in the embodiment of the present application provides a method for determining a homography transformation matrix based on internal and external parameters of the camera for panoramic stitching of video frames captured by a camera array. Specifically, based on the intrinsic parameter matrix of the camera and the calibrated extrinsic parameter matrix, the world coordinates of feature points on the extracted reference image and the image to be stitched are determined, and then based on the world coordinates of the feature points, the intrinsic parameter matrix of the camera and the extrinsic parameter matrix, the homography transformation matrix between the camera that captures the image to be stitched and the camera that captures the reference image is determined, so that the image to be stitched is converted into the same image coordinate system as the reference image through the homography transformation matrix, providing a basis for image stitching; and then by searching for the best stitching seam, the reference image and the image to be stitched are stitched, and after the images of the same video frame captured by each camera in the camera array are stitched, a frame of video panoramic image is obtained. Compared with the method of relying on image feature matching to calculate the homography transformation matrix, the calculation method provided by the present application that combines the internal and external parameters of the camera has high accuracy and can better adapt to scenes with low texture or fewer feature points, thereby improving the quality of image stitching in low-texture scenes and reducing ghosting or artifacts in the stitched panorama.

[0114] Optional, Figure 4 For this application Figure 2 The flowchart of step S202 in the embodiment shown is as follows: Figure 4 As shown, step S202 specifically includes the following steps:

[0115] Step S401: determining feature point pairs whose world coordinates match in the image to be stitched and the reference image based on a nearest neighbor matching algorithm.

[0116] Step S402: Screening the matching feature point pairs based on the random sampling consensus algorithm.

[0117] RANdom SAmple Consensus (RANSAC) is an iterative algorithm used to filter outliers or outliers from a set of data. The inliers in the matching feature point pairs are obtained through screening based on the random sampling consensus algorithm.

[0118] By screening the matched feature point pairs through RANSAC, outliers in the feature point pairs are filtered out, the accuracy of the feature point pairs used to calculate the mapping relationship is improved, and then the accuracy of the homography transformation matrix is ​​improved.

[0119] Step S403: determining a first transformation matrix based on the image coordinates of each feature point in the filtered feature point pairs on the corresponding image.

[0120] Based on the mapping relationship between the image coordinates of each feature point, ie, the inner point, in the filtered feature point pair on the corresponding image, a first transformation matrix is ​​obtained.

[0121] Specifically, the mapping relationship between the reference image and the image to be spliced ​​in the world coordinate system can be obtained by least squares calculation based on the image coordinates of the multiple groups of inliers, that is, the first transformation matrix can be obtained. A group of inliers corresponds to a pair of feature points after screening.

[0122] Step S404: determining the homography transformation matrix based on the first conversion matrix, and the intrinsic parameter matrix and the extrinsic parameter matrix of the reference camera.

[0123] Since the intrinsic parameters of each camera in the camera array are the same, there are only translation and rotation transformations between the images captured by the reference camera and the target camera in the camera array. The relationship between the feature point pairs on the image to be stitched and the reference image in the camera coordinate system is:

[0124]

[0125]

[0126]

[0127] Among them, (X c1 , Y c1 , Z c1 ) is the coordinate of feature point p1 on the reference image in the camera coordinate system; (X c2 , Y c2 , Z c2 ) is the coordinate of the feature point p2 on the image to be stitched in the camera coordinate system; R c is the rotation matrix between the reference camera and the target camera; M is the first transformation matrix; T c is the translation matrix between the reference camera and the target camera; α and β are the angles of rotation of the target camera in the left and right and up and down directions relative to the reference camera; t1, t2, t3 are the distances moved by the target camera in the three directions of the world coordinate system XYZ relative to the reference camera. Feature points p1 and p2 are a set of feature pairs.

[0128] The relationship between the camera coordinate system and the world coordinate system is:

[0129]

[0130] Among them, (X c1 , Y c1 , Z c1) is the coordinate of the feature point p1 on the reference image in the world coordinate system; R and T are the rotation matrix and offset vector between the camera coordinate system and the world coordinate system, respectively, and R and T can be determined based on the extrinsic parameter matrix of the reference camera.

[0131] Substituting equation (4.4) into equation (4.1), we obtain

[0132]

[0133] Substitute the coordinates of multiple groups of feature points corresponding to the reference image and the image to be stitched in the camera coordinate system and the world coordinate system into equation (4.5), and obtain the matrix M′ by solving it to realize the transformation from the camera coordinate system of the image to be stitched to the world coordinate system of the reference image.

[0134] The coordinates of the feature points in the camera coordinate system can be obtained based on the camera's intrinsic matrix and the coordinates of the feature points in the image coordinate system. Based on the matrix M' and the intrinsic matrix and extrinsic matrix of the reference camera, the homography transformation matrix H can be obtained to achieve the conversion between the image to be stitched and the reference image.

[0135] After the images of the target video frames captured by the camera array are stitched using the image stitching method provided by the above embodiment to obtain a frame of video panorama, other video frames captured by the camera array, such as subsequent videos, can be stitched according to the stitching template obtained when the target video frames are stitched, including the homography transformation matrix and the optimal stitching seam.

[0136] The target video frame should preferably be a video frame with more obvious features in the image, such as a video frame containing a calibration plate or an image of other objects with more features.

[0137] In some embodiments, the target video frame may be the first frame captured by the camera array, or the first frame specified by the user.

[0138] During the test phase, the camera in the camera array can capture images containing additional feature objects (such as calibration plates) to obtain multiple test video frames, and one of the multiple test video frames is designated as the target video frame. Based on the target video frame, a stitching template is obtained, including a homography transformation matrix and an optimal stitching seam, so as to facilitate the image stitching of video frames subsequently collected by the camera array.

[0139] For the subsequently acquired video frames, when stitching, it is necessary to determine whether the best stitching seam needs to be updated. Specifically, it is determined whether the best stitching seam needs to be updated by determining whether there is a moving object in the stitching seam. If there is no moving object, stitching is performed according to the previous stitching template; if there is a moving object, the best stitching seam needs to be updated, and image stitching is performed based on the homography transformation matrix and the updated best stitching seam.

[0140] Figure 5 A flow chart of an image stitching method provided for another embodiment of the present application. The image stitching method provided in this embodiment is for image stitching processing of subsequently acquired video frames, that is, stitching processing of subsequent reference images and subsequent images to be stitched. The subsequent reference images are images of subsequent frames acquired by the camera corresponding to the reference image, and the subsequent images to be stitched are images of subsequent frames acquired by the camera corresponding to the images to be stitched.

[0141] like Figure 5 As shown, the image stitching method provided in this embodiment may also include the following steps:

[0142] Step S501 : for the video frames subsequently captured by the camera array, based on the homography transformation matrix, coordinate transformation is performed on subsequent images to be stitched, so as to obtain subsequent target images to be stitched.

[0143] The homography transformation matrix is ​​obtained through step S203 and related steps, that is, determined based on the world coordinates of the feature points of the reference image and the image to be stitched, the intrinsic parameter matrix of the camera in the camera array, and the calibrated extrinsic parameter matrix.

[0144] The subsequently acquired video frames are video frames acquired by the camera array after the video frames used in determining the homography transformation matrix. The reference images, images to be stitched, and target images to be stitched in the subsequently acquired video frames are recorded as subsequent reference images, subsequent images to be stitched, and subsequent target images to be stitched.

[0145] Step S502: for a previous video frame captured by the camera array, a first optimal stitching seam between a previous target image to be stitched and a previous reference image is obtained.

[0146] The previous video frame is a video frame before the subsequently acquired video frame, and the reference image, the image to be stitched, and the target image to be stitched in the previous video frame are recorded as the previous reference image, the previous image to be stitched, and the previous target image to be stitched.

[0147] The previous reference image is the previous frame image captured by the camera corresponding to the subsequent reference image, and the previous target image to be stitched is the image obtained after the coordinate transformation of the previous frame image captured by the camera corresponding to the subsequent image to be stitched is performed based on the homography transformation matrix.

[0148] The first optimal stitching seam is the optimal stitching seam used when the previous labeled image to be stitched is stitched with the previous reference image.

[0149] Step S503, determining whether there is a moving object at the joint seam of the subsequent reference image and the subsequent target joint image. If not, executing step S504; if yes, executing step S505.

[0150] Step S504: Using the first optimal stitching seam as the optimal stitching seam, completing the image stitching operation of a subsequent reference image and a subsequent target stitching image.

[0151] If there is no moving object at the stitching seam, the subsequent video frames are stitched directly based on the previously determined optimal stitching seam.

[0152] Step S505, performing an optimal seam update operation.

[0153] For the video frames subsequently captured by the camera array, when stitching multiple images corresponding to the video frames, the coordinates of the images to be stitched are first transformed based on the previously determined homography transformation matrix to obtain the overlapping area between the images to be stitched and the reference image, that is, the stitching seam. The specific method is similar to that for the target frame processing and will not be repeated here.

[0154] For the subsequently acquired video frames, it is necessary to determine whether there is a moving object at the stitching seam. If there is a moving object, the previous best stitching seam cannot be used and the best stitching seam needs to be searched again.

[0155] Optionally, the determining whether there is a moving object at a joint seam of a subsequent reference image and a subsequent target joint image includes:

[0156] Obtain the gradient value of the pixels at the stitching seam of the previous reference image and the previous target stitching image, recorded as the first gradient value; obtain the gradient value of the pixels at the stitching seam of the subsequent reference image and the subsequent target stitching image, recorded as the second gradient value; and determine whether there is a moving object at the stitching seam of the subsequent reference image and the subsequent target stitching image based on the first gradient value and the second gradient value.

[0157] Based on the change of pixel gradient in the image at the seam (determined by the difference of pixel gradient values ​​at the same position of the previous and next frames), it is possible to detect whether there is a moving object at the seam. Specifically, it is possible to determine whether there is a moving object based on the number of pixels with large gradient change values ​​at the seam. If the number of pixels with large gradient change values ​​at the seam exceeds a set threshold, it is determined that there is a moving object and the optimal seam needs to be updated.

[0158] In some embodiments, in order to improve the accuracy of judging the size of gradient change values, the gradient change values ​​can be classified into size or speed based on the K-means clustering algorithm to obtain a set of pixels with fast gradient changes. If the number of pixels in the pixel set exceeds a set threshold, it is determined that there is a moving object at the seam.

[0159] Specifically, based on the position of the moving object and the color, texture, optical flow trajectory and other features of the pixels at the seam, the optimal seam between the subsequent reference image and the subsequent target image to be stitched can be re-searched to avoid misalignment of the moving object.

[0160] Optionally, performing the optimal seam update operation includes:

[0161] Re-searching for an optimal stitching seam from stitching seams of a subsequent reference image and a subsequent target stitching image to obtain a second optimal stitching seam, and updating the optimal stitching seam to be the second optimal stitching seam; and stitching the subsequent reference image and the subsequent target stitching image based on the second optimal stitching seam.

[0162] Exemplarily, for cameras A and B in the camera array, after the homography transformation matrix and the optimal stitching seam are obtained for the first frame image captured by cameras A and camera B, the first frame image captured by cameras A and camera B is stitched; for the subsequent images captured by cameras A and camera B, such as the second frame image, an image transformation is performed based on the homography transformation matrix to obtain an overlapping area or a stitching seam, and it is determined whether there is a moving object at the stitching seam; if not, the second frame image captured by cameras A and camera B is stitched based on the optimal stitching seam obtained for the first frame; if so, an optimal stitching seam update operation is performed, and the second frame image captured by cameras A and camera B is stitched based on the updated optimal stitching seam.

[0163] In this embodiment, for subsequent video frames, the previously determined stitching template can be directly used when stitching images, without determining the overlapping area based on image features, which can better adapt to low-texture application scenarios. When there is a moving object at the stitching seam of two images in the subsequent video frame, the optimal stitching seam is dynamically updated, which improves the image stitching quality and reduces the phenomenon of splicing dislocation of moving objects.

[0164] Optional, Figure 6 For this application Figure 5 The flowchart of step S503 in the embodiment shown is as follows: Figure 6 As shown, step S502 specifically includes the following steps:

[0165] Step S601 : Obtain the gradient value of pixels at the joint seam of the previous reference image and the previous target stitched image, recorded as a first gradient value.

[0166] Step S602, obtaining the gradient value of the pixels at the joint seam of the subsequent reference image and the subsequent target joint image, recorded as the second gradient value.

[0167] Step S603 : Based on the first gradient value and the second gradient value, the gradient change value of each pixel at the joint seam of the subsequent reference image and the subsequent target stitched image is calculated to obtain a gradient change set.

[0168] Step S604: Select two gradient change values ​​from the gradient change set as gradient change centers.

[0169] From the gradient change set S t Randomly select two gradient change values ​​as the initial gradient change center. Gradient change set S t Represents the set of gradient change values ​​of the video frame collected at time t.

[0170] Gradient change set S t Medium pixel p i The gradient change value s it It can be defined as:

[0171]

[0172] Among them, g i0 is the pixel p in the first video frame (first frame) or the target video frame i The gradient of g it is the median pixel p of the video frame captured at time t i gradient.

[0173] When the gradient change set S t When there are moving objects in the gradient set S, the following steps can be used to convert the gradient change set S t The gradient change values ​​in are divided into two categories: fast gradient change and slow gradient change. The average value of the gradient change value of the category with fast gradient change is larger.

[0174] Step S605 , calculating the distance between the gradient change value in the gradient change set and the center of each gradient change.

[0175] For the first round of distance calculation, this step specifically calculates the distance between the remaining gradient change values ​​in the gradient change set except the gradient change center and each gradient change center.

[0176] The gradient includes two dimensions, such as the gradient in the x and y directions. The corresponding gradient change value also corresponds to two dimensions, and the distance between the gradient change values ​​can be expressed by the difference in each dimension, such as using the Euclidean distance.

[0177] Step S606: Divide the gradient change values ​​in the gradient change set into two groups based on the distance.

[0178] Each gradient change value corresponds to two distances, which are the distances between the gradient change value and two gradient change centers. The gradient change value can be divided into a group corresponding to the gradient center with the smallest distance.

[0179] Taking the gradient change centers c1 and c2 as an example, for pixel p iThe gradient change value s it , if its distance from the gradient change center c1 is smaller than that from the gradient change center c2, then the gradient change value s it Divide into a group where the gradient change center c1 is located.

[0180] Based on the smaller value in the distance, the gradient change set S t The gradient change values ​​in are divided into two groups.

[0181] Step S607, updating the gradient change center to the average value of each group of gradient change values.

[0182] Step S608, determine whether the gradient change centers before and after the update are the same; if so, execute step S609; then return to step S605, calculate the distance between the gradient change values ​​in the gradient change set and the updated gradient change center, re-group the gradient change values ​​and update the gradient change center.

[0183] Step S609, determining whether the number of gradient change values ​​in the group with a larger average value of gradient change values ​​is greater than a preset number. If yes, executing step S610; otherwise, executing step S611.

[0184] The preset number may be determined based on the total number of pixels at the joint, or a default value may be used.

[0185] In some embodiments, the preset number is the product of the total number of pixels at the joint seam and a preset coefficient, and the preset coefficient is in the interval (0, 1).

[0186] Exemplarily, the preset coefficient may be 0.3.

[0187] Step S610: determining whether a moving object exists at a joint of a subsequent reference image and a subsequent target joint image.

[0188] Step S611 , determining that there is no moving object at the joint seam of the subsequent reference image and the subsequent target joint image.

[0189] In this embodiment, based on the distribution of gradient change values ​​of each pixel point at the seam of the image, pixels with larger gradient change values ​​are dynamically determined, so that when the scale of the moving object is too small, the moving object can still be effectively detected, thereby improving the accuracy of moving object detection.

[0190] Figure 7 A flowchart of an image stitching method provided in yet another embodiment of the present application is shown in FIG. Figure 7 As shown, in this embodiment, the image stitching method includes the following steps:

[0191] Step S701, selecting a target frame to be captured by a camera array.

[0192] The target frame can be the first frame or a video frame specified by the user.

[0193] Step S702, executing an image stitching algorithm based on camera parameters to obtain a homography transformation matrix.

[0194] The method for determining the homography transformation matrix provided in the above-mentioned embodiment may be referred to, such as the method corresponding to step S201 to step S203, which will not be described in detail here.

[0195] Step S703, finding the best seam.

[0196] Step S704: transform the images corresponding to the subsequent video frames captured by the camera array based on the homography transformation matrix.

[0197] Step S705 , based on the K-means clustering algorithm, the gradient change values ​​are classified according to the gradient change values ​​of the pixels at the joint seams.

[0198] Step S706, based on the classification result, determine whether there is a moving object at the joint; if so, execute step S707; if not, execute step S708.

[0199] Specifically, based on the number of gradient change values ​​or the number of pixels in the category with faster gradient change in the classification result, it is determined whether there is a moving object at the splicing seam. If the number is greater than a preset number, it is determined that there is a moving object at the splicing seam, otherwise it is not present.

[0200] Step S707, updating the optimal stitching seam.

[0201] Step S708, generating a video panorama.

[0202] Based on the best stitching seam or the updated best stitching seam, the images are stitched to generate a video panorama.

[0203] Step S709, synthesizing the panoramic video.

[0204] The panoramic images of each video frame are synthesized in chronological order to obtain a panoramic video.

[0205] Figure 8 A schematic diagram of the structure of an image stitching device provided in one embodiment of the present application is shown in FIG. Figure 8 As shown, the image stitching device includes: a feature point extraction module 810, a world coordinate determination module 820, a transformation matrix determination module 830, an image transformation module 840 and a stitching module 850.

[0206] The feature point extraction module 810 is used to extract feature points of a reference image and an image to be stitched for a video frame captured by a camera array, wherein the camera array includes a plurality of cameras with the same intrinsic parameter matrix; the world coordinate determination module 820 is used to determine the world coordinates of each feature point based on the intrinsic parameter matrix and the extrinsic parameter matrix of the camera calibration in the camera array; the transformation matrix determination module 830 is used to determine the homography transformation matrix of the image to be stitched relative to the reference image based on the world coordinates of the feature points, the intrinsic parameter matrix and the calibrated extrinsic parameter matrix; the image transformation module 840 is used to perform coordinate transformation on the image to be stitched based on the homography transformation matrix; the stitching module 850 is used to complete the image stitching operation based on the optimal stitching seam between the target image to be stitched and the reference image.

[0207] Optionally, the reference image is acquired by a reference camera in the camera array, and the stitched image is acquired by a target camera in the camera array, where the target camera is another camera in the camera array except the reference camera. The feature point extraction module 810 is specifically used for:

[0208] The world coordinates of each of the feature points are determined based on the intrinsic parameter matrix, the extrinsic parameter matrix of the reference camera calibration, and the extrinsic parameter matrix of the target camera calibration.

[0209] Optionally, the transformation matrix determination module 830 includes:

[0210] A feature point pair determination unit is used to determine, based on a nearest neighbor matching algorithm, feature point pairs whose world coordinates match in the image to be stitched and the reference image; a transformation matrix determination unit is used to determine a homography transformation matrix of the image to be stitched relative to the reference image based on the image coordinates of each feature point in the feature point pair on the corresponding image, and the intrinsic parameter matrix and extrinsic parameter matrix of the reference camera.

[0211] Optionally, the transformation matrix determination unit is specifically used to:

[0212] Based on a random sampling consensus algorithm, matching feature point pairs are screened; based on the image coordinates of each feature point in the screened feature point pairs on the corresponding image, a first transformation matrix is ​​determined; based on the first transformation matrix, and the intrinsic parameter matrix and extrinsic parameter matrix of the reference camera, the homography transformation matrix is ​​determined.

[0213] Optionally, the device further comprises:

[0214] A subsequent image transformation module is used to perform coordinate transformation on subsequent images to be stitched based on the homography transformation matrix for the video frames subsequently captured by the camera array to obtain subsequent target images to be stitched; a historical best stitching seam acquisition module is used to acquire the first best stitching seam of the previous target image to be stitched and the previous reference image for the previous video frame captured by the camera array; a stitching seam determination module is used to determine whether there is a moving object at the stitching seam of the subsequent reference image and the subsequent target stitching image; a first subsequent image stitching module is used to use the first best stitching seam as the best stitching seam to complete the image stitching operation of the subsequent reference image and the subsequent target stitching image if there is no moving object at the stitching seam of the subsequent reference image and the subsequent target stitching image; and a second subsequent image stitching module is used to perform the best stitching seam update operation if there is a moving object at the stitching seam of the subsequent reference image and the subsequent target stitching image.

[0215] Optionally, a seam determination module is specifically used for:

[0216] Based on the first gradient value and the second gradient value, the gradient change value of each pixel at the stitching seam of the subsequent reference image and the subsequent target stitching image is calculated to obtain a gradient change set; two gradient change values ​​are selected from the gradient change set as gradient change centers; the distance between the gradient change values ​​in the gradient change set and each of the gradient change centers is calculated; based on the distance, the gradient change values ​​in the gradient change set are divided into two groups; the gradient change center is updated to the average value of the gradient change values ​​in each group; it is determined whether the gradient change centers before and after the update are the same; if the gradient change centers before and after the update are the same, it is determined whether the number of gradient change values ​​in the group with a larger average gradient change value is greater than a preset number, so as to determine that there is a moving object at the stitching seam of the subsequent reference image and the subsequent target stitching image when the number of gradient change values ​​in the group with a larger average gradient change value is greater than the preset number; if the gradient change centers before and after the update are not the same, the step of calculating the distance between the gradient change values ​​in the gradient change set and each gradient change center is returned to.

[0217] Optionally, the second subsequent image stitching module is specifically used to:

[0218] If there is a moving object at the stitching seam between the subsequent reference image and the subsequent target stitching image, re-search the best stitching seam from the stitching seams of the subsequent reference image and the subsequent target stitching image to obtain a second best stitching seam, and update the best stitching seam to be the second best stitching seam; based on the second best stitching seam, stitch the subsequent reference image and the subsequent target stitching image.

[0219] Optionally, the device further comprises:

[0220] The panoramic image display module is used to display the video panoramic image obtained by the image stitching operation after the image stitching operation is completed.

[0221] The image stitching device provided in the embodiment of the present application can execute the image stitching method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0222] Fig. 9 A schematic diagram of the structure of an image stitching device provided in one embodiment of the present application is shown in FIG. Fig. 9 As shown, the image stitching device includes: a memory 910 and at least one processor 920.

[0223] The memory 910 stores computer-executable instructions; and at least one processor 920 executes the computer-executable instructions stored in the memory, so that the at least one processor executes to implement the image stitching method provided in any embodiment of the present application.

[0224] The memory 910 and the processor 920 are connected via a bus 930 .

[0225] The relevant explanations can be understood by referring to the relevant descriptions and effects corresponding to the steps provided in the relevant embodiments of the image stitching method of the present application, and no further elaboration will be made here.

[0226] An embodiment of the present application provides a computer-readable storage medium on which computer execution instructions are stored. When a processor executes the computer execution instructions, the image stitching method provided by any embodiment of the present application is implemented.

[0227] Among them, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0228] An embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the image stitching method provided by any embodiment of the present application is implemented.

[0229] The processor may be an integrated circuit chip having the ability to process signals. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0230] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0231] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the disclosure disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0232] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image stitching method, characterized in that: include: Extracting feature points of a reference image and an image to be stitched from video frames captured by a camera array, wherein the camera array includes a plurality of cameras with the same intrinsic parameter matrix; Determining the world coordinates of each of the feature points based on the intrinsic parameter matrix and the extrinsic parameter matrix of camera calibration in the camera array; Determine, based on a nearest neighbor matching algorithm, feature point pairs whose world coordinates match between the image to be stitched and the reference image; Based on the random sampling consensus algorithm, the matching feature point pairs are screened; Determine a first transformation matrix based on the image coordinates of each feature point in the filtered feature point pairs on the corresponding image; Determine a homography transformation matrix based on the first transformation matrix and an intrinsic parameter matrix and an extrinsic parameter matrix of a reference camera; Based on the homography transformation matrix, coordinate transformation is performed on the images to be stitched to obtain a target image to be stitched; The image stitching operation is completed based on the optimal stitching seam between the target image to be stitched and the reference image.

2. The method according to claim 1, characterized in that The reference image is acquired by a reference camera in the camera array, and the stitched image is acquired by a target camera in the camera array, where the target camera is another camera in the camera array except the reference camera; The determining the world coordinates of each of the feature points based on the intrinsic parameter matrix and the extrinsic parameter matrix of the camera calibration in the camera array includes: The world coordinates of each of the feature points are determined based on the intrinsic parameter matrix, the extrinsic parameter matrix of the reference camera calibration, and the extrinsic parameter matrix of the target camera calibration.

3. The method according to claim 1, characterized in that After determining the homography transformation matrix of the image to be stitched relative to the reference image based on the world coordinates of the feature points, the intrinsic parameter matrix and the calibrated extrinsic parameter matrix, the method includes: For the video frames subsequently acquired by the camera array, based on the homography transformation matrix, coordinate transformation is performed on subsequent images to be stitched, so as to obtain subsequent target images to be stitched; For a previous video frame captured by the camera array, obtaining a first optimal stitching seam between a previous target image to be stitched and a previous reference image; Determine whether there is a moving object at the joint seam of the subsequent reference image and the subsequent target stitching image; If so, the optimal seam update operation is performed; If not, the first optimal stitching seam is used as the optimal stitching seam to complete the image stitching operation of the subsequent reference image and the subsequent target stitching image.

4. The method according to claim 3, characterized in that The step of determining whether there is a moving object at a joint seam between a subsequent reference image and a subsequent target joint image includes: Obtaining the gradient value of the pixels at the joint seam of the previous reference image and the previous target image to be stitched, recorded as a first gradient value; Obtaining the gradient value of the pixel at the joint seam of the subsequent reference image and the subsequent target stitching image, recorded as the second gradient value; It is determined whether there is a moving object at a joint seam of a subsequent reference image and a subsequent target joint image according to the first gradient value and the second gradient value.

5. The method according to claim 4, characterized in that Judging whether there is a moving object at a joint seam of a subsequent reference image and a subsequent target joint image according to the first gradient value and the second gradient value includes: Based on the first gradient value and the second gradient value, calculating the gradient change value of each pixel at the stitching seam of the subsequent reference image and the subsequent target stitching image to obtain a gradient change set; Select two gradient change values ​​from the gradient change set as gradient change centers; Calculating the distance between the gradient change value in the gradient change set and each of the gradient change centers; Based on the distance, the gradient change values ​​in the gradient change set are divided into two groups; Update the gradient change center to the average value of each group of gradient change values; Determine whether the center of gradient change before and after the update is the same; If the gradient change centers before and after the update are the same, determining whether the number of gradient change values ​​in a group with a larger average value of the gradient change values ​​is greater than a preset number, so as to determine that a moving object exists at a stitching seam of a subsequent reference image and a subsequent target stitching image when the number of gradient change values ​​in a group with a larger average value of the gradient change values ​​is greater than the preset number; If the gradient change centers before and after the update are different, the process returns to the step of calculating the distances between the gradient change values ​​in the gradient change set and the gradient change centers.

6. The method according to claim 3, characterized in that The performing of the optimal seam updating operation comprises: Re-searching for an optimal stitching seam from stitching seams of a subsequent reference image and a subsequent target stitching image to obtain a second optimal stitching seam, and updating the optimal stitching seam to be the second optimal stitching seam; Based on the second optimal stitching seam, the subsequent reference image and the subsequent target stitching image are stitched together.

7. The method according to any one of claims 1 to 6, characterized in that: After completing the image stitching operation, the method further includes: Displays the video panorama obtained by the image stitching operation.

8. An image stitching device, characterized in that: include: A feature point extraction module, used to extract feature points of a reference image and an image to be stitched from video frames captured by a camera array, wherein the camera array includes a plurality of cameras with the same intrinsic parameter matrix; A world coordinate determination module, used to determine the world coordinates of each of the feature points based on the intrinsic parameter matrix and the extrinsic parameter matrix of the camera calibration in the camera array; A transformation matrix determination module, used for determining feature point pairs whose world coordinates match between the image to be stitched and the reference image based on a nearest neighbor matching algorithm; Based on the random sampling consensus algorithm, the matching feature point pairs are screened; Determine a first transformation matrix based on the image coordinates of each feature point in the filtered feature point pairs on the corresponding image; Determine a homography transformation matrix based on the first transformation matrix and an intrinsic parameter matrix and an extrinsic parameter matrix of a reference camera; An image transformation module, used for performing coordinate transformation on the images to be stitched based on the homography transformation matrix to obtain a target image to be stitched; The stitching module is used to complete the image stitching operation based on the optimal stitching seam between the target image to be stitched and the reference image.

9. An image stitching device, characterized in that: include: memory and at least one processor; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the image stitching method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the image stitching method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image stitching method according to any one of claims 1 to 7 is implemented.

Images