Image splicing method and device of multi-view camera and electronic equipment

Abstract

The invention is suitable for the technical field of image processing, and provides an image stitching method and device for a multi-view camera and electronic equipment, and the method comprises the steps: obtaining at least two to-be-stitched images; determining a convergence angle of any two adjacent cameras; if any convergence angle is greater than a preset included angle threshold value, determining a to-be-spliced image pair corresponding to the convergence angle greater than the preset included angle threshold value; adjusting a picture visual angle of at least one to-be-spliced image in the to-be-spliced image pair of which the convergence angle is greater than a preset included angle threshold value to obtain a corresponding to-be-spliced image pair of which the visual angle is adjusted; extracting image features of the to-be-spliced image pairs after the view angles are adjusted; carrying out feature matching on the extracted image features; and carrying out image splicing on the to-be-spliced image pair after the view angle adjustment according to a feature matching result. Through the method, the success rate of image splicing can be improved.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to image stitching methods, apparatuses, electronic devices, computer-readable storage media, and computer program products for multi-view cameras. Background Technology

[0002] Currently, people frequently need to use wide-angle images in their work and daily lives. While short focal length lenses can extend the field of view when achieving wide-angle imaging, they introduce severe distortion and reduce edge resolution. Image stitching technology, by merging overlapping images taken by multiple conventional lenses, can achieve a wide field of view while avoiding extreme distortion and retaining the high resolution advantage of the central area. Therefore, image stitching has become the preferred solution for high-quality, wide-field-of-view imaging.

[0003] Existing image stitching methods typically involve: detecting the image features of each image to be stitched, finding corresponding matching features based on the detected features, and then stitching the images together based on the found matching features. However, using this method for image stitching may result in stitching failure.

[0004] Therefore, a new splicing method is needed to solve the above-mentioned technical problems. Summary of the Invention

[0005] This application provides an image stitching method, apparatus, and electronic device for multi-view cameras, which can solve the problem that existing methods fail to extract image features, leading to stitching failure.

[0006] In a first aspect, embodiments of this application provide an image stitching method using a multi-view camera, including: At least two images to be stitched are obtained, wherein the images to be stitched are determined based on captured images, the captured images are obtained by a camera, and there is an overlapping area between two adjacent captured images obtained by the camera; Determine the convergence angle between any two adjacent cameras; If any of the convergence angles is greater than a preset angle threshold, then the image pair to be stitched corresponding to the convergence angle greater than the preset angle threshold is determined. Adjust the viewing angle of at least one of the images to be stitched in the image pair whose convergence angle is greater than the preset angle threshold to obtain the image pair to be stitched after the viewing angle adjustment, wherein the viewing angle difference of the image pair to be stitched after the viewing angle adjustment is less than the viewing angle difference of the image pair to be stitched before the viewing angle adjustment. Extract image features from the image pairs to be stitched after adjustments from each viewpoint; Feature matching is performed on the extracted image features; Based on the feature matching results, the images to be stitched together after the viewpoint adjustment are stitched together.

[0007] The beneficial effects of the embodiments in this application compared with the prior art are: In this embodiment, for a multi-view camera, if the convergence angle of any two adjacent cameras is greater than a preset angle threshold, then the image pair to be stitched corresponding to the convergence angle greater than the preset angle threshold is determined. The viewing angle of at least one image in the image pair with the convergence angle greater than the preset angle threshold is adjusted to obtain the aforementioned image pair with adjusted viewing angle. Since the viewing angle difference corresponding to the image pair with adjusted viewing angle is smaller than the viewing angle difference corresponding to the image pair before adjustment, and the smaller the viewing angle difference, the higher the probability of extracting matching image feature point pairs. Therefore, after adjusting the viewing angle of at least one image in the image pair with the convergence angle greater than the preset angle threshold, extracting the image features of each image pair with adjusted viewing angle, and performing feature matching on the extracted image features, helps improve the success rate of feature matching. Furthermore, after stitching the image pair with adjusted viewing angle based on the feature matching results, it also helps improve the accuracy of image stitching.

[0008] Secondly, embodiments of this application provide an image stitching device for a multi-view camera, comprising: The image acquisition module is used to acquire at least two images to be stitched, wherein the images to be stitched are determined based on captured images, the captured images are obtained by a camera, and there is an overlapping area between the captured images obtained by two adjacent cameras; A convergence angle determination module is used to determine the convergence angle between any two adjacent cameras; The image pair to be stitched module is used to determine the image pair to be stitched corresponding to the convergence angle that is greater than the preset angle threshold if any of the convergence angles is greater than the preset angle threshold. The image viewing angle adjustment module is used to adjust the viewing angle of at least one of the images to be stitched in the image pair to be stitched if the convergence angle is greater than the preset angle threshold, so as to obtain the image pair to be stitched after the viewing angle adjustment, wherein the viewing angle difference of the image pair to be stitched after the viewing angle adjustment is less than the viewing angle difference of the image pair to be stitched before the viewing angle adjustment. The image feature extraction module is used to extract the image features of the image pair to be stitched after adjustments from each viewpoint; The feature matching module is used to perform feature matching on the extracted image features; The image stitching module is used to stitch the image pairs to be stitched together after the viewpoint has been adjusted, based on the feature matching results.

[0009] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0010] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0011] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform the method described in the first aspect.

[0012] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0014] Figure 1 This is a flowchart illustrating an image stitching method using a multi-view camera provided in an embodiment of this application; Figure 2 This is a schematic diagram of the top view of the installation position of an eight-lens camera according to an embodiment of this application; Figure 3 This is a schematic diagram of a tilt angle provided in an embodiment of this application; Figure 4 This is a schematic diagram of a convergence angle provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating a one-to-many matching relationship provided in another embodiment of this application; Figure 6 This is a schematic diagram illustrating a one-to-one matching relationship provided in an embodiment of this application; Figure 7 This is a schematic diagram of four distortion-free images obtained by a four-eye camera according to another embodiment of this application; Figure 8 This application provides an embodiment of the method for... Figure 7 A schematic diagram of the image obtained after adjusting the horizontal perspective of four images; Figure 9 This application provides an embodiment of the method for... Figure 8 A schematic diagram of the image obtained after adjusting the viewing angle of the four images; Figure 10 This is provided by the embodiments of this application. Figure 7 A schematic diagram of the overlapping region corresponding to the image shown; Figure 11 This is a schematic diagram of a splicing result provided by an embodiment of this application; Figure 12 This is a schematic diagram of the structure of an image stitching device for a multi-view camera provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0015] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0016] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0017] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0018] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0019] Currently, people frequently need to use wide-view images in their work and daily lives. For example, wide-view landscape images are used to represent large-scale landscapes to enhance the sense of spatial depth; wide-view architectural images provide richer clues about spatial relationships, and so on. Since the field of view (or angle of view) of a single camera is limited, and the more angles there are, the more severe the distortion becomes, existing methods usually choose image stitching to obtain wide-view images.

[0020] However, when using existing image stitching methods, if the convergence angle between adjacent cameras corresponding to the images to be stitched is large, the similarity of the extracted image features may be low. Low-similarity image features have a lower probability of successful feature matching, thus reducing the success rate of image stitching. The aforementioned convergence angle refers to the angle between the optical axes of adjacent cameras.

[0021] To improve the success rate of image stitching, this application provides an image stitching method using multi-view cameras. In this method, when the convergence angle between adjacent cameras is determined to be greater than a preset angle threshold, the viewing angle of the images to be stitched corresponding to those adjacent cameras is adjusted to reduce the viewing angle difference between the images. Since a smaller viewing angle difference increases the probability of extracting highly similar image features from the corresponding images to be stitched, it helps to improve the success rate of image stitching.

[0022] The image stitching method for multi-view cameras provided in the embodiments of this application is described below with reference to the accompanying drawings.

[0023] Figure 1 A flowchart illustrating an image stitching method using a multi-view camera is shown. This image stitching method can be applied to electronic devices, and is described in detail below: S11, acquire at least two images to be stitched together, wherein the images to be stitched together are determined based on captured images, the captured images are captured by a camera, and there is an overlapping area between two adjacent captured images captured by the camera.

[0024] In this embodiment, at least two cameras can be installed according to certain rules. After installation, the images captured by adjacent cameras will have overlapping areas, enabling subsequent stitching of these images. The aforementioned rules include a top-to-bottom order, a left-to-right order, and other orders, which are not limited here. When eight cameras are installed in a left-to-right order, the top view of their installation positions is as follows: Figure 2 As shown. In Figure 2 In the image, lens 0 is the lens of the first camera, and lens 1 is the lens of the second camera. The first and second cameras are two adjacent cameras, and there is an overlapping area in the images captured by the two cameras.

[0025] Optionally, the time difference between the shooting times of the various images to be stitched is 0 or close to 0. Specifically, the clocks of each camera can be synchronized in advance, and the images to be stitched can be determined based on the images captured by each camera at the same (or nearly the same) moment. Since the time difference between shooting times is close to or equal to 0, it can be considered that each camera captured the instantaneous state of the subject almost simultaneously. Under this condition, image stitching can minimize the timing inconsistency errors caused by object movement or lighting changes, thereby significantly improving the accuracy and reliability of the subsequently stitched images.

[0026] In this embodiment of the application, after the multi-view camera captures the captured image, the electronic device determines the corresponding image to be stitched based on the captured image.

[0027] Optionally, the captured image can be directly used as the image to be stitched. For example, if the captured image has no distortion and / or the viewing angle of the captured image meets the user's requirements, the captured image can be directly used as the image to be stitched. Of course, if the captured image has distortion, but the distortion is within the user's tolerance range, the captured image can also be directly used as the image to be stitched; this is not limited here.

[0028] Optionally, the captured image can be processed before being used as the image to be stitched together. Optionally, the processing of the captured image includes: distortion correction and / or horizontal viewing angle adjustment.

[0029] In this embodiment, distortion correction refers to the digital image restoration process that performs mathematical modeling and inverse transformation on the image point position offset caused by the non-ideal refraction of the optical lens, thereby restoring the original Euclidean geometry of the scene. Distortion correction can be performed using a camera-based method or a fisheye method. Specifically, the camera's field of view can be obtained first. If the field of view is large (e.g., greater than 120°), the fisheye method is used for distortion correction; if the field of view is small (e.g., not greater than 120°), the camera-based method is used.

[0030] In this embodiment of the application, the horizontal viewing angle adjustment process refers to the process of adjusting the viewing angle of the image captured by a camera whose lens view is not horizontal to a horizontal view or close to a horizontal view.

[0031] In this application embodiment, the following scenarios may require horizontal viewing angle adjustment: (1) The multi-view camera is installed in a left-right arrangement and the lens viewing angle is not horizontal: For example, a surveillance camera installed in a public place may be installed at a height of 8 meters. People usually need to know the relevant situation on the ground. Therefore, when installing the surveillance camera, its lens is designed downward, that is, it has a downward tilt angle, in order to capture relevant information about the ground. The downward tilt angle is the angle between the lens and the vertical line, such as Figure 3 As shown, assuming the camera's tilt angle is 26°, the angle between its lens and the vertical line is 26°. For scenes where the camera has a tilt angle, adjusting the horizontal viewing angle is equivalent to adjusting the "horizon" that is not in the center of the image back to the center of the image. Specifically, the known tilt angle (e.g., 26°) and camera intrinsic parameters can be used to calculate the homography matrix (assuming it is H1) required for correction. The image can be transformed using the homography matrix H1 to achieve the purpose of changing the viewing angle. (2) Multi-view cameras are installed vertically and vertically and the lens viewing angle is not horizontal: In this case, considering that rotating the camera clockwise by a certain angle (e.g., 90°) can adjust the viewing angle of the corresponding captured image to a horizontal viewing angle, the homography matrix (assuming it is H2) required for correction can be calculated based on the clockwise rotation angle and camera intrinsic parameters. The image can be transformed using the homography matrix H2 to achieve the purpose of changing the viewing angle.

[0032] That is, obtaining at least two images to be stitched together includes: A1. Obtain images captured by at least two cameras.

[0033] Specifically, for each of at least two cameras, images captured by each camera at the same time are obtained to acquire at least two images.

[0034] A2. If the lens angle of the camera is not horizontal, then adjust the viewing angle of each of the above-mentioned captured images based on the horizontal viewing angle.

[0035] In this embodiment, when the camera lens view is not horizontal, the angle corresponding to rotating the lens view to a horizontal view is calculated, the corresponding homography matrix is ​​calculated based on the angle value, and the captured image is adjusted based on the homography matrix to obtain the captured image with the adjusted view.

[0036] A3. Based on the above-mentioned captured images after adjusting the perspective of each scene, determine at least two of the above-mentioned images to be stitched together.

[0037] In this embodiment of the application, each captured image after the viewing angle is adjusted can be used as an image to be stitched together, so as to obtain at least two images to be stitched together.

[0038] Since the horizontal viewing angle corresponds to the human's natural habit of "looking around from left to right" and is naturally aligned with the geometric constraints of display media, splicing seams, and stereo matching, adjusting the horizontal viewing angle of the captured image is beneficial to improving the visual effect of the final spliced ​​image and also helps to reduce subsequent splicing errors.

[0039] S12, determine the convergence angle of any two adjacent cameras mentioned above.

[0040] like Figure 4 As shown, the convergence angle mentioned above refers to the angle between the optical axes of the two cameras.

[0041] In this embodiment, if the installation information of two adjacent cameras includes convergence angle information, the installation information of the two adjacent cameras is obtained, and the convergence angle of the two adjacent cameras is determined based on the installation information. If the installation information of two adjacent cameras does not include convergence angle information, or if the installation information of two adjacent cameras includes convergence angle information but the installation information cannot be obtained, the convergence angle of the two adjacent cameras can be estimated.

[0042] S13, if any of the above convergence angles is greater than a preset angle threshold, then determine the image pair to be stitched corresponding to the convergence angle greater than the preset angle threshold.

[0043] The aforementioned preset angle threshold can be set according to actual needs. For example, the aforementioned preset angle threshold can be set to 45°. Of course, in actual situations, it can also be set to other values, such as 0°. In this case, as long as there is a convergence angle, it will be adjusted subsequently.

[0044] In this embodiment, the image pair to be stitched corresponding to the convergence angle is two images captured by two adjacent cameras corresponding to that convergence angle. For example, assuming camera 1 and camera 2 are two adjacent cameras, the two images to be stitched determined based on the two images captured by camera 1 and camera 2 respectively belong to the same image pair to be stitched. If the convergence angle corresponding to camera 1 and camera 2 is greater than a preset angle threshold, then the image pair to be stitched corresponding to that convergence angle is determined, that is, the image to be stitched corresponding to camera 1 and the image to be stitched corresponding to camera 2 are determined. At this time, the two determined images to be stitched belong to the above-mentioned image pair to be stitched.

[0045] S14, adjust the viewing angle of at least one of the images to be stitched in the image pair whose convergence angle is greater than the preset included angle threshold, to obtain the image pair to be stitched after the viewing angle adjustment, wherein the viewing angle difference of the image pair to be stitched after the viewing angle adjustment is less than the viewing angle difference of the image pair to be stitched before the viewing angle adjustment.

[0046] In this embodiment, the images of each pair of images to be stitched that have a convergence angle greater than a preset angle threshold are adjusted. Specifically, the viewing angle of any one of the images to be stitched in the pair with a convergence angle greater than the preset angle threshold can be kept unchanged, but the viewing angle of the other image in the pair can be adjusted; or, the viewing angles of the two images in the pair can be adjusted. The purpose of this adjustment is to reduce the difference in viewing angles between the image pairs without changing the camera's mounting structure.

[0047] Optionally, considering that the viewing angle difference is related to the baseline and convergence angle, the viewing angle difference corresponding to the image pair to be stitched can be reduced according to the convergence angle. For example, assuming the convergence angle is 45°, if the two cameras corresponding to the image pair to be stitched are Camera 1 and Camera 2, and Camera 1 is to the left of Camera 2, then the viewing angle of the image to be stitched corresponding to Camera 1 can be adjusted to rotate 45° to the right, or the viewing angle of the image to be stitched corresponding to Camera 2 can be adjusted to rotate 45° to the left. Of course, the viewing angle of the image to be stitched corresponding to Camera 1 can also be adjusted to rotate a certain angle to the right, and the viewing angle of the image to be stitched corresponding to Camera 2 can be adjusted to rotate a certain angle to the left.

[0048] Optionally, the viewing angles of the image pairs to be stitched are simultaneously rotated by the same angle. In this case, the viewing angles of at least one image in the image pair to be stitched, where the convergence angle is greater than the preset angle threshold, are adjusted to obtain the corresponding viewing angle-adjusted image pair, including: For any of the above image pairs to be stitched whose convergence angle is greater than the preset included angle threshold, the viewing angles of the two images in the image pair to be stitched are adjusted, wherein the adjusted viewing angles of the two images are the same but in opposite directions.

[0049] In this embodiment, when the convergence angle of the image pair to be stitched is greater than the aforementioned preset angle threshold and the direction of the adjusted viewing angle is opposite, the viewing angle difference of the image pair to be stitched can be reduced. Simultaneously, since the adjusted viewing angle values ​​of the image pair to be stitched are the same, the impact on the information quality of the image pair to be stitched can be reduced. For example, assuming a convergence angle of 45°, if the two cameras corresponding to the image pair to be stitched are camera 1 and camera 2, and camera 1 is to the left of camera 2, then the viewing angle of the image to be stitched corresponding to camera 1 can be rotated 22.5° to the right, and the viewing angle of the image to be stitched corresponding to camera 2 can be rotated 22.5° to the left.

[0050] S15, extract the image features of the above image pairs to be stitched together after adjusting each viewpoint.

[0051] In this embodiment, if there are image pairs to be stitched corresponding to convergence angles no greater than a preset angle threshold, then the image features of these image pairs to be stitched also need to be extracted when extracting image features. However, if there are no image pairs to be stitched corresponding to convergence angles no greater than the preset angle threshold (e.g., the preset angle threshold is set to 0°), then only the image features of image pairs to be stitched with convergence angles greater than the preset angle threshold after adjusting the viewing angle need to be extracted.

[0052] In this embodiment, considering that there are many single-lens images captured by a multi-view camera, and that different single-lens images will not all overlap, i.e., different images to be stitched will not all overlap, therefore, in order to quickly extract effective image features, the area for image feature extraction can be narrowed down, and then image features can be extracted from the narrowed area. That is, in this embodiment, extracting the image features of the above-mentioned image pairs to be stitched after adjusting each viewpoint includes: The overlapping area of ​​the image pair to be stitched after each viewpoint adjustment is estimated, wherein the size of the estimated overlapping area is smaller than the size of the image to be stitched in the image pair after the viewpoint adjustment. Extract image features from the estimated overlapping regions mentioned above.

[0053] In this embodiment, the mapped positions of some points (such as some boundary points) of the original image corresponding to the image to be stitched are obtained based on the mapping relationship between these boundary points and the homography matrix of the overlapping region. Then, the corresponding overlapping region is determined based on the mapped positions of these boundary points. The homography matrix of the overlapping region refers to the homography matrix used for calculating the overlapping region.

[0054] Optionally, the aforementioned original image refers to the image before the perspective adjustment. For example, if the image to be stitched is obtained by performing distortion correction and horizontal perspective adjustment on the captured image, then the aforementioned original image refers to the image after the distortion correction process.

[0055] Optionally, the boundary points mentioned above can be selected from the boundary points corresponding to 1 / 2 or 1 / 4 of the original image. For example, for two adjacent cameras, the boundary point corresponding to the right 1 / 2 of the original image for the left camera is selected, while the boundary point corresponding to the left 1 / 2 of the original image for the right camera is selected. Since the image captured by the left camera is closer to the right than the image captured by the right camera, the probability of overlap is greater. Similarly, the image captured by the right camera is closer to the left than the image captured by the left camera. Therefore, the boundary points selected in this way help improve the accuracy of the estimated overlap area. Optionally, when the image to be stitched is an image obtained after distortion correction and horizontal perspective adjustment of the captured image, the homography matrix of the overlapping region is determined based on the homography matrix corresponding to the horizontal perspective adjustment and the homography matrix corresponding to the image perspective adjustment. For example, assuming the homography matrix corresponding to the horizontal perspective adjustment is H3 and the homography matrix corresponding to the image perspective adjustment is H4, then the homography matrix of the overlapping region = H3*H4. If a boundary point is P(x, y, 1), then the target position coordinates (x', y') obtained by normalizing H3*H4*P are the mapped position of point P.

[0056] In this embodiment, the mapped positions of the boundary points can be determined as the corresponding overlapping regions. Optionally, considering that neural networks typically default to a rectangular shape for image feature extraction, and that rectangular shapes are easier to calculate, if the shape corresponding to the mapped positions of the boundary points is not a rectangle, the circumscribed rectangle corresponding to these mapped positions can also be determined, and the region corresponding to this circumscribed rectangle can be used as the aforementioned overlapping region.

[0057] Since the two cameras corresponding to the stitched image pairs after viewpoint adjustment are adjacent cameras, and the stitched images corresponding to adjacent cameras have overlapping areas, estimating the overlapping area of ​​the stitched images in each pair after viewpoint adjustment is beneficial to improving the accuracy of the estimated overlapping area, thereby improving the effectiveness of the image features extracted from the overlapping area. Specifically, when one extracted image feature can match another image feature, it indicates that the extracted image feature is effective.

[0058] It should be noted that when there are convergence angles of no more than the preset angle threshold corresponding to the image pairs to be stitched, the overlapping areas of these image pairs to be stitched can be estimated first, and then the corresponding image features can be extracted from these overlapping areas. This will not be elaborated here.

[0059] In this embodiment, after estimating the overlapping area of ​​the image pair to be stitched after viewpoint adjustment, image features can be extracted using the Oriented FAST and Rotated BRIEF (ORB) algorithm, or using a deep learning framework, such as a software platform or library for building and training deep neural network models, to automatically learn image features in a data-driven manner. Of course, other methods can also be used for image feature extraction, which are not limited here.

[0060] In some embodiments, the overlapping region can be treated as a whole for image feature extraction, or the overlapping region can be divided into multiple sub-regions, and image features can be extracted from each sub-region. When the overlapping region is divided into multiple sub-regions before image feature extraction, the image features of the estimated overlapping region extracted above include: The estimated overlapping area is divided into multiple sub-regions; Image features of each sub-region are extracted separately.

[0061] The sizes of the sub-regions obtained from the division can be completely equal or not completely equal; no restriction is imposed here.

[0062] In this embodiment of the application, the number of sub-regions to be divided can be determined in advance, or the number of rows and columns to be divided can be determined in advance. Then, the corresponding sub-regions are divided according to the resolution of the overlapping region and the number of sub-regions (or the number of rows and columns to be divided). Finally, for each sub-region, the image features (or image feature points) of that sub-region are extracted.

[0063] In this embodiment, considering that when extracting image feature points from the entire overlapping region or from the entire image to be stitched, the perspectives of adjacent cameras differ, so some regions with less texture may not yield any image feature points, while another perspective may yield more, leading to mismatches in image feature points. Therefore, the estimated overlapping region is divided into blocks, and a certain number of image feature points are extracted from each block separately. Furthermore, for regions with less texture, if the number of image feature points is determined to be less than expected, the confidence level can be lowered to extract as many image feature points as possible. That is, through the above method, since as many image features as possible can be extracted from each sub-region, it is beneficial to improve the uniformity of the extracted image feature points.

[0064] S16, perform feature matching on the extracted image features.

[0065] In this embodiment, image features extracted from different images to be stitched together are used for feature matching. Specifically, the descriptors corresponding to two image feature points are calculated to see if they are similar. If they are similar, the two image feature points are determined to be a matching pair.

[0066] Optionally, during feature matching, image features extracted from the images to be stitched in the image pairs corresponding to two adjacent cameras (including image pairs after viewpoint adjustment, and image pairs corresponding to convergence angles not greater than a preset angle threshold) can be used for feature matching. When performing feature matching, it is not necessary to start with either image in the image pair. For example, if the image pair includes image A and image B, image features in image A can be selected first, and it can be determined whether the selected image features match those in image B. Alternatively, image features in image B can be selected first, and it can be determined whether the selected image features match those in image A. Since there are overlapping areas in the images captured by two adjacent cameras, i.e., the two images in the image pair to be stitched have overlapping areas, performing feature matching from the image features extracted from the images in the image pairs corresponding to two adjacent cameras helps improve the efficiency of obtaining matching point pairs.

[0067] Optionally, during feature matching, if one image feature point is found to match multiple image feature points, or if multiple image feature points are found to match one image feature point, these image feature points are deleted. Since the existence of such one-to-many or many-to-one matching relationships indicates that the matching relationships of the image feature points corresponding to these matching cases are unreliable, deleting these image feature points helps improve the reliability of the final matching result. The one-to-many matching relationship can be as follows: Figure 5 As shown. In Figure 5 In the image, the four points on the left represent four image feature points belonging to the same image to be stitched together, and the one point on the right represents one image feature point belonging to another image to be stitched together.

[0068] Optionally, a bidirectional matching algorithm can be used in the feature matching process. That is, from the two images to be stitched (including image pairs after viewpoint adjustment, and also image pairs corresponding to convergence angles not greater than a preset angle threshold), image feature point pairs with one-to-one matching relationships are selected to improve the reliability of the final matching result. The one-to-one matching relationship can be as follows: Figure 6 As shown. In Figure 6 In the image, the four points on the left represent four image feature points belonging to the same image to be stitched together, and the one point on the right represents one image feature point belonging to another image to be stitched together. However, among the four image feature points on the left, only one image feature point has a one-to-one matching relationship with the image feature point on the right.

[0069] In this embodiment of the application, in order to improve the accuracy of feature matching, the corresponding method can be selected for feature matching based on the placement of the camera.

[0070] When two adjacent cameras are horizontally positioned, the above-mentioned feature matching of the extracted image features includes: When two adjacent cameras are placed horizontally, feature matching is performed on the extracted image features according to the row information of the sub-regions.

[0071] Horizontal placement refers to the fact that the vertical positions of the various cameras in a multi-camera system are relatively similar.

[0072] The row information mentioned above includes the row number or row position corresponding to the sub-region.

[0073] In this embodiment, the vertical position differences of the horizontally placed cameras are small, meaning the differences in the horizontal planes where these cameras are located are small. Furthermore, if the images to be stitched corresponding to these cameras are images obtained after horizontal viewing angle adjustment, the differences in the horizontal planes corresponding to these images to be stitched will be further reduced. Since the differences in the horizontal planes where adjacent cameras are located are small, a point above one image in a pair of images to be stitched (including pairs of images to be stitched after viewing angle adjustment, and pairs of images to be stitched corresponding to convergence angles not greater than a preset angle threshold) obtained through these cameras will necessarily correspond to a point above the other image to be stitched. Similarly, a point below one image to be stitched will necessarily correspond to a point below the other image to be stitched. Therefore, when performing image feature matching by row, it is equivalent to regrouping and combining image feature points and descriptors according to image height blocks, and then matching the points above one image to be stitched with the points above another image to be stitched, instead of matching the points above one image with the points below another image to be stitched. This reduces the error of matching the top and bottom of the image and improves the accuracy of the subsequent calculation of the pairing homography matrix, which is calculated based on the paired image feature point pairs.

[0074] When two adjacent cameras are placed vertically, the above-mentioned feature matching of the extracted image features includes: When two adjacent cameras are placed vertically, feature matching is performed on the extracted image features according to the column information of the sub-regions.

[0075] Vertical placement refers to the fact that the horizontal positions of the cameras in a multi-camera system are relatively similar, meaning that the vertical planes in which these cameras are located are relatively similar.

[0076] The column information mentioned above includes the column number or column position corresponding to the sub-region.

[0077] In this embodiment, since the difference between the vertical planes of two adjacent cameras is small, a point on the left side of one image to be stitched (including image pairs after viewpoint adjustment and image pairs corresponding to convergence angles not greater than a preset angle threshold) obtained by these cameras will always correspond to a point on the left side of the other image to be stitched. Similarly, a point on the right side of one image to be stitched will always correspond to a point on the right side of the other image to be stitched. Therefore, when performing image feature matching by column, it is equivalent to matching a point on the left side of one image to be stitched with a point on the left side of another image to be stitched, rather than matching a point on the left side of one image to be stitched with a point on the right side of another image to be stitched. This reduces the error of left-right matching and improves the accuracy of the subsequently calculated pairing homography matrix.

[0078] S17, perform image stitching on the above image pairs to be stitched after the viewpoint adjustment based on the feature matching results.

[0079] In this embodiment, when there exists a pair of images to be stitched corresponding to a convergence angle not greater than a preset angle threshold, the image stitching also includes stitching the images of the pair of images to be stitched corresponding to the convergence angle not greater than the aforementioned preset angle threshold. Specifically, RANSAC iteration can be performed on the image feature point pairs indicated by the feature matching result to calculate the paired homography matrix and its corresponding interior points. Based on the paired homography matrix and the camera's rotation matrix (R) and translation vector (T), each image to be stitched is projected onto the same plane. R and T can be obtained through checkerboard calibration.

[0080] By using spherical projection, images already projected onto the same plane are projected onto a sphere and unfolded to obtain an unfolded image. The intersection of the effective regions of each image to be stitched is calculated in the coordinate system of the unfolded image to obtain the overlapping regions. Finally, the overlapping regions are merged, for example, using pyramid fusion to merge the overlapping regions, forming the final stitched image.

[0081] In this embodiment, for a multi-view camera, if the convergence angle of any two adjacent cameras is greater than a preset angle threshold, then the image pair to be stitched corresponding to the convergence angle greater than the preset angle threshold is determined. The viewing angle of at least one image in the image pair with the convergence angle greater than the preset angle threshold is adjusted to obtain the aforementioned image pair with adjusted viewing angle. Since the viewing angle difference corresponding to the image pair with adjusted viewing angle is smaller than the viewing angle difference corresponding to the image pair before adjustment, and the smaller the viewing angle difference, the higher the probability of extracting matching image feature point pairs. Therefore, after adjusting the viewing angle of at least one image in the image pair with the convergence angle greater than the preset angle threshold, extracting the image features of each image pair with adjusted viewing angle, and performing feature matching on the extracted image features, helps improve the success rate of feature matching. Furthermore, after stitching the image pair with adjusted viewing angle based on the feature matching results, it also helps improve the accuracy of image stitching.

[0082] To more clearly describe the image stitching method for multi-view cameras provided in the embodiments of this application, the following description is based on specific examples.

[0083] Assume the multi-view camera is a 4-view camera, which is horizontally positioned and has a downward tilt angle.

[0084] If the four distortion-free images obtained by the four-lens camera are as follows: Figure 7 The four images shown are for Figure 7 After adjusting the horizontal perspective of the four images, the following was obtained: Figure 8 The four images shown.

[0085] If the convergence angle between any two adjacent cameras in the four-camera system is greater than a preset angle threshold, the viewing angle of the stitched image corresponding to these cameras will be adjusted. For example, for Figure 8 After adjusting the viewing angle of the four images shown, the result is as follows: Figure 9 The four images shown.

[0086] according to Figure 7 The image shows partial boundary points. The projection points corresponding to these boundary points after horizontal and vertical view adjustments are determined. Assuming the obtained projection points are as follows... Figure 10 The blue dots in the diagram. Based on these projection points, the bounding rectangle of the region containing these projection points is determined, resulting in... Figure 10 The red rectangle shown represents the estimated overlap area.

[0087] These overlapping regions are divided into multiple sub-regions, and image features are extracted from each sub-region. Finally, feature matching is performed on the extracted image features according to the row information of the sub-regions to obtain feature matching results. Based on the feature matching results, the image pairs to be stitched after viewpoint adjustment and the image pairs to be stitched corresponding to convergence angles not greater than the preset angle threshold are stitched together. The resulting stitching result is as follows: Figure 11 As shown.

[0088] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0089] Corresponding to the image stitching method for multi-view cameras described in the above embodiments, Figure 12 This diagram illustrates a structural block diagram of an image stitching device for a multi-view camera according to an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown.

[0090] Reference Figure 12 The multi-view camera image stitching device 12 is applied to electronic equipment and includes: a module for acquiring images to be stitched 121, a module for determining convergence angle 122, a module for determining image pairs to be stitched 123, a module for adjusting viewing angle 124, a module for extracting image features 125, a module for matching features 126, and a module for stitching images 127. Wherein: The image acquisition module 121 is used to acquire at least two images to be stitched, wherein the images to be stitched are determined based on captured images, which are captured by a camera, and there is an overlapping area between the captured images captured by two adjacent cameras.

[0091] Convergence angle determination module 122 is used to determine the convergence angle of any two adjacent cameras mentioned above.

[0092] The image pair determination module 123 is used to determine the image pair to be stitched corresponding to the convergence angle that is greater than the preset angle threshold if any of the above convergence angles is greater than the preset angle threshold.

[0093] The image viewing angle adjustment module 124 is used to adjust the viewing angle of at least one of the images to be stitched in the image pair to be stitched if the convergence angle is greater than the preset angle threshold, so as to obtain the image pair to be stitched after the viewing angle adjustment, wherein the viewing angle difference of the image pair to be stitched after the viewing angle adjustment is less than the viewing angle difference of the image pair to be stitched before the viewing angle adjustment.

[0094] The image feature extraction module 125 is used to extract the image features of the above-mentioned image pairs to be stitched together after each viewpoint adjustment.

[0095] The feature matching module 126 is used to perform feature matching on the extracted image features.

[0096] The image stitching module 127 is used to stitch the above-mentioned image pairs to be stitched after the viewpoint adjustment based on the feature matching results.

[0097] In this embodiment, for a multi-view camera, if the convergence angle of any two adjacent cameras is greater than a preset angle threshold, then the image pair to be stitched corresponding to the convergence angle greater than the preset angle threshold is determined. The viewing angle of at least one image in the image pair with the convergence angle greater than the preset angle threshold is adjusted to obtain the aforementioned image pair with adjusted viewing angle. Since the viewing angle difference corresponding to the image pair with adjusted viewing angle is smaller than the viewing angle difference corresponding to the image pair before adjustment, and the smaller the viewing angle difference, the higher the probability of extracting matching image feature point pairs. Therefore, after adjusting the viewing angle of at least one image in the image pair with the convergence angle greater than the preset angle threshold, extracting the image features of each image pair with adjusted viewing angle, and performing feature matching on the extracted image features, helps improve the success rate of feature matching. Furthermore, after stitching the image pair with adjusted viewing angle based on the feature matching results, it also helps improve the accuracy of image stitching.

[0098] Optionally, adjusting the viewing angle of at least one of the images to be stitched in the image pair whose convergence angle is greater than the preset included angle threshold to obtain the corresponding viewing angle adjusted image pair includes: For any of the above image pairs to be stitched whose convergence angle is greater than the preset included angle threshold, the viewing angles of the two images in the image pair to be stitched are adjusted, wherein the adjusted viewing angles of the two images are the same but in opposite directions.

[0099] Optionally, the image feature extraction module 125 includes: The overlapping area of ​​the image pair to be stitched after each viewpoint adjustment is estimated, wherein the size of the estimated overlapping area is smaller than the size of the image to be stitched in the image pair after the viewpoint adjustment. Extract image features from the estimated overlapping regions mentioned above.

[0100] Optionally, the extraction of image features of the estimated overlapping regions includes: The estimated overlapping area is divided into multiple sub-regions; Image features of each sub-region are extracted separately.

[0101] Optionally, the above-mentioned feature matching of the extracted image features includes: When two adjacent cameras are placed horizontally, feature matching is performed on the extracted image features according to the row information of the sub-regions.

[0102] Optionally, the above-mentioned feature matching of the extracted image features includes: When two adjacent cameras are placed vertically, feature matching is performed on the extracted image features according to the column information of the sub-regions.

[0103] Optionally, the above-mentioned image acquisition module 121 includes: The image acquisition unit is used to acquire images captured by at least two of the aforementioned cameras.

[0104] The lens angle determination unit is used to adjust the screen angle of each of the above-mentioned captured images based on the above-mentioned horizontal angle if the lens angle of the camera is not a horizontal angle.

[0105] The image to be stitched determination unit is used to determine at least two of the above-mentioned images to be stitched based on the above-mentioned captured images after the perspective of each scene has been adjusted.

[0106] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0107] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 13 As shown, the electronic device 13 of this embodiment includes: at least one processor 130 ( Figure 13 The diagram shows only one processor, a memory 131, and a computer program 132 stored in the memory 131 and executable on the at least one processor 130, wherein the processor 130 executes the computer program 132 to implement the steps in any of the above method embodiments.

[0108] The electronic device 13 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. This electronic device may include, but is not limited to, a processor 130 and a memory 131. Those skilled in the art will understand that... Figure 13 This is merely an example of electronic device 13 and does not constitute a limitation on electronic device 13. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0109] The processor 130 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0110] In some embodiments, the memory 131 may be an internal storage unit of the electronic device 13, such as a hard disk or memory of the electronic device 13. In other embodiments, the memory 131 may be an external storage device of the electronic device 13, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 13. Furthermore, the memory 131 may include both internal and external storage units of the electronic device 13. The memory 131 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 131 can also be used to temporarily store data that has been output or will be output.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0112] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0113] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0114] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0116] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0117] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0119] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] It should be noted that the information collection process (such as the facial image collection process, fingerprint information collection process, etc.) / feature extraction process involved in this application is carried out with the user's knowledge and permission. That is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

Claims

1. An image stitching method for multi-view cameras, characterized in that, include: At least two images to be stitched are obtained, wherein the images to be stitched are determined based on captured images, the captured images are obtained by a camera, and there is an overlapping area between two adjacent captured images obtained by the camera; Determine the convergence angle between any two adjacent cameras; If any of the convergence angles is greater than a preset angle threshold, then the image pair to be stitched corresponding to the convergence angle greater than the preset angle threshold is determined. Adjust the viewing angle of at least one of the images to be stitched in the image pair whose convergence angle is greater than the preset angle threshold to obtain the image pair to be stitched after the viewing angle adjustment, wherein the viewing angle difference of the image pair to be stitched after the viewing angle adjustment is less than the viewing angle difference of the image pair to be stitched before the viewing angle adjustment. Extract image features from the image pairs to be stitched after adjustments from each viewpoint; Feature matching is performed on the extracted image features; Based on the feature matching results, the images to be stitched together after the viewpoint adjustment are stitched together.

2. The image stitching method for multi-view cameras as described in claim 1, characterized in that, The step of adjusting the viewing angle of at least one image in the image pair to be stitched that has a convergence angle greater than the preset angle threshold, to obtain the corresponding viewing angle adjusted image pair, includes: For any pair of images to be stitched whose convergence angle is greater than the preset included angle threshold, the viewing angles of the two images in the pair are adjusted, wherein the adjusted viewing angles of the two images are the same but in opposite directions.

3. The image stitching method for multi-view cameras as described in claim 1, characterized in that, The extraction of image features from each viewpoint-adjusted image pair to be stitched together includes: The overlapping area of ​​the image pair to be stitched is estimated after each viewpoint adjustment, wherein the size of the estimated overlapping area is smaller than the size of the image to be stitched in the image pair after the viewpoint adjustment. Extract image features from the estimated overlapping region.

4. The image stitching method for multi-view cameras as described in claim 3, characterized in that, The extraction of image features of the estimated overlapping region includes: The estimated overlapping region is divided into multiple sub-regions; Image features of each sub-region are extracted separately.

5. The image stitching method for multi-view cameras as described in any one of claims 4, characterized in that, The step of performing feature matching on the extracted image features includes: When two adjacent cameras are placed horizontally, feature matching is performed on the extracted image features according to the row information of the sub-region.

6. The image stitching method for a multi-view camera as described in any one of claims 4, characterized in that, The step of performing feature matching on the extracted image features includes: When two adjacent cameras are placed vertically, feature matching is performed on the extracted image features according to the column information of the sub-region.

7. The image stitching method for a multi-view camera as described in any one of claims 1 to 6, characterized in that, The process of obtaining at least two images to be stitched together includes: Acquire images captured by at least two of the cameras; If the camera lens angle is not horizontal, then the viewing angle of each captured image is adjusted based on the horizontal angle. Based on the captured images after adjusting the perspective of each viewpoint, at least two images to be stitched together are determined.

8. An image stitching device for a multi-view camera, characterized in that, include: The image acquisition module is used to acquire at least two images to be stitched, wherein the images to be stitched are determined based on captured images, the captured images are obtained by a camera, and there is an overlapping area between the captured images obtained by two adjacent cameras; A convergence angle determination module is used to determine the convergence angle between any two adjacent cameras; The image pair to be stitched module is used to determine the image pair to be stitched corresponding to the convergence angle that is greater than the preset angle threshold if any of the convergence angles is greater than the preset angle threshold. The image viewing angle adjustment module is used to adjust the viewing angle of at least one of the images to be stitched in the image pair to be stitched if the convergence angle is greater than the preset angle threshold, so as to obtain the image pair to be stitched after the viewing angle adjustment, wherein the viewing angle difference of the image pair to be stitched after the viewing angle adjustment is less than the viewing angle difference of the image pair to be stitched before the viewing angle adjustment. The image feature extraction module is used to extract the image features of the image pair to be stitched after adjustments from each viewpoint; The feature matching module is used to perform feature matching on the extracted image features; The image stitching module is used to stitch the image pairs to be stitched together after the viewpoint has been adjusted, based on the feature matching results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1 to 7 to be performed.

Images