Multi-view stereoscopic image generation method and system

Abstract

The present application provides a multi-view stereoscopic image generation method and system. An image acquisition device captures an entity calibration board to obtain an image, and a processor obtains a reference image and a plurality of images to be calibrated by analyzing the image including the calibration board. The processor calculates a homography matrix of each image to be calibrated mapping to the reference image according to the reference image and the image to be calibrated. The processor obtains a correction matrix of the homography matrix by performing matrix factorization operation on each homography matrix. The processor multiplies the image to be calibrated by the corresponding correction matrix to obtain a plurality of corrected images. The processor outputs a multi-view stereoscopic image including the reference image and the corrected images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of correcting images and outputting multi-view stereoscopic images, and more particularly to a multi-view stereoscopic image generation method and system suitable for naked-eye displays. BACKGROUND

[0002] One of the existing methods for capturing and operating multi-view stereoscopic images is to set up a fixed track and a camera sliding on the track to shoot images at different times and angles. Then, a correction operation is performed to obtain multi-view stereoscopic images. However, the above-mentioned method requires the subject and the background to be stationary during shooting, so the existing method is only suitable for capturing static images and cannot capture instantaneous images. Moreover, the track must be set up first, which limits the convenience of use. SUMMARY

[0003] The present application provides a multi-view stereoscopic image generation method and a multi-view stereoscopic image generation system, which can output multi-view stereoscopic images with corrected perspective.

[0004] The multi-view stereoscopic image generation method of the present application includes the following steps: a processor receives images shot by an image acquisition device to obtain a reference image and a plurality of to-be-corrected images in the images; the processor respectively calculates a plurality of homography matrices of each to-be-corrected image mapped to the reference image according to the reference image and the to-be-corrected images; the processor respectively performs matrix factorization operation on each homography matrix according to the homography matrices to obtain a correction matrix of each homography matrix in the homography matrices; the processor respectively performs a correction operation on the to-be-corrected images based on the correction matrix to obtain a plurality of corrected images; and the processor outputs a multi-view stereoscopic image, wherein the multi-view stereoscopic image includes the reference image and the corrected images.

[0005] The multi-view stereoscopic image generation system of the present application includes (but is not limited to) an image acquisition device and a processor. The image acquisition device is used to shoot images with a physical correction plate. The processor is coupled to the image acquisition device. The processor is configured to receive images shot by the image acquisition device to obtain a reference image and a plurality of to-be-corrected images in the images, respectively calculate a plurality of homography matrices of each to-be-corrected image mapped to the reference image according to the reference image and the to-be-corrected images, respectively perform matrix factorization operation on each homography matrix according to the homography matrices to obtain a correction matrix of each homography matrix in the homography matrices, respectively perform a correction operation on the to-be-corrected images based on the correction matrix to obtain a plurality of corrected images, and output a multi-view stereoscopic image. The multi-view stereoscopic image includes the reference image and the corrected images.

[0006] Based on the above, by disassembling the homography matrix, an affine matrix and a similarity matrix are extracted as the correction matrix. Thus, the output multi-view stereoscopic image has a corrected smoothness (i.e., no deviation in position) while retaining the perspective, solving the problem of insufficient stereoscopic effect of the stereoscopic image and image alignment caused by directly multiplying the homography matrix, and achieving the correction effect between images. BRIEF DESCRIPTION OF DRAWINGS

[0007] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0008] Figure 1 is a block diagram of a multi-view stereoscopic image generation system according to an embodiment of the present application.

[0009] Figure 2 is a schematic diagram of a multi-view stereoscopic image generation system according to an embodiment of the present application.

[0010] Figure 3 is a flowchart of a multi-view stereoscopic image generation method according to an embodiment of the present application.

[0011] Figure 4 is a schematic diagram of an interpolation operation according to an embodiment of the present application.

[0012] Figure 5 is a schematic diagram of a translation process according to an embodiment of the present application.

[0013] Figure 6 is a schematic diagram of an interpolation operation and a translation process according to an embodiment of the present application.

[0014] REFERENCE NUMERALS

[0015] 1: multi-view stereoscopic image generation system

[0016] 110: image acquisition device

[0017] 120: processor

[0018] 130: storage device

[0019] CB: solid correction plate

[0020] CP: checkerboard pattern

[0021] P1, P2, Pd: image to be corrected

[0022] H1, H2, H m : homography matrix

[0023] H2 -1 : inverse matrix of the homography matrix

[0024] T: Translation matrix;

[0025] T -1 The inverse matrix of a translation matrix;

[0026] Pd, Pd': Baseline image;

[0027] S1, S2: Virtual images;

[0028] C1, C2, C3, C4, C5, Cm-1, Cm: Corrected images;

[0029] C1', C2', C3', C4', C5', Cm-1', Cm': Corrected images after translation processing;

[0030] SC1, SC3: Virtual images;

[0031] SC1', SC3': Virtual images that have undergone translation processing;

[0032] S310~S360: Steps. Detailed Implementation

[0033] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same component reference numerals are used in the drawings and description to denote the same or similar parts.

[0034] Figure 1 This is a block diagram of a multi-view stereoscopic image generation system 1 according to an embodiment of the present invention. Please refer to... Figure 1 The multi-view stereoscopic image generation system 1 includes an image acquisition device 110 and a processor 120. Here, the processor 120 is coupled to the image acquisition device 110 via a wired or wireless connection. That is, the image acquisition device 110 communicates with the processor 120 via wired or wireless transmission, enabling the processor 120 to receive data from the image acquisition device 110.

[0035] In another embodiment, the multi-view stereoscopic image generation system 1 further includes a storage device 130 and a multi-view stereoscopic image display. The processor 120 is coupled to the storage device 130 and the multi-view stereoscopic image display via a wired or wireless connection. That is, the storage device 130 and the multi-view stereoscopic image display communicate with the processor 120 via wired or wireless transmission, enabling the processor 120 to access data from the storage device 130 and transmit data to the multi-view stereoscopic image display.

[0036] The image acquisition device 110 is, for example, a camera, a video camera, or the like using a charge coupled device (CCD) lens, a complementary metal oxide semiconductor (CMOS) lens, or the like.

[0037] The processor 120 is, for example, a central processing unit (CPU), a graphic processing unit (GPU), a physics processing unit (PPU), a programmable microprocessor, an embedded controller chip, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or the like.

[0038] The storage device 130 can be any type of fixed or removable radom access memory (RAM), read only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or the like. In an embodiment, the storage device 130 is used to store program codes, software modules, configuration settings, data (e.g., images, images to be corrected, reference images, corrected images, homography matrices, correction matrices, or multi-view stereo images), or files. In the present embodiment, the storage device 130 can be coupled to a printing output device to output multi-view stereo images outputted by the processor as multi-view stereo pictures through the printing device. The printing device can be a lenticular printing device.

[0039] The multi-view stereoscopic image display can be an autostereoscopic display, a temporal multiplexed stereoscopic display, a spatial multiplexed stereoscopic display, a head mount display, a depth-fused multi-layer display, a holographic display, a volumetric display, and / or other suitable display, or an electronic device having the aforementioned display, but the present application is not limited thereto. In the present embodiment, the multi-view stereoscopic image display displays the multi-view stereoscopic image outputted by the multi-view stereoscopic image rendering processor in real time to present the corrected and / or translated multi-view stereoscopic image. The printing device can be a lenticular printing device.

[0040] Figure 2 FIG. 1 is a schematic diagram of a multi-view stereoscopic image generation system according to an embodiment of the present application. In the multi-view stereoscopic image generation system 1, a plurality of image acquisition devices 110 are used to capture a plurality of images. The plurality of images are images captured at different angles of a scene with a physical correction board CB. A processor 120 calculates homography matrices of the images with respect to a reference image, and further decomposes the homography matrices of each image into affine, perspective, and similarity matrices. Based on the characteristics of the affine and similarity matrices as correction matrices, the multi-view stereoscopic image generation system 1 and method output multi-view stereoscopic images that are visually smooth and perspective. Here, the pattern on the plane of the physical correction board CB is, for example, a regular color checkerboard pattern CP, or a wall surface with identifiable patterns, but the present application is not limited thereto.

[0041] In other embodiments, the processor 120 can be disposed in the image acquisition device 110. In other embodiments, the image acquisition device 110, the processor 120, and the storage device 130 can be integrated into the same device. For example, a personal computer, a notebook computer, a smart phone, and a tablet computer having photographing, shooting, and projecting functions, but the present application is not limited thereto.

[0042] The steps of the multi-view stereoscopic image generation method of the present application are described below in conjunction with the multi-view stereoscopic image generation system 1. Figure 3 FIG. 2 is a flowchart of a multi-view stereoscopic image generation method according to an embodiment of the present application. Please refer to FIG. 1 simultaneously. Figures 1 to 3In step S310, the plurality of image acquisition devices 110 respectively capture a plurality of images in which the physical correction plate is arranged, and the images include the image of the physical correction plate CB. Specifically, the plurality of image acquisition devices 110 are arranged adjacent to each other, and adjacent image acquisition devices among the plurality of image acquisition devices 110 have an included angle of between 0.5 degrees and 10 degrees. In this embodiment, the plurality of image acquisition devices 110 are arranged horizontally, i.e., the plurality of image acquisition devices 110 are arranged on the same horizontal plane, and each image acquisition device 110 captures / captures an image in which the correction plane (the physical correction plate CB) for correction is arranged at the same time. For example, when the number of image acquisition devices 110 is six, each image acquisition device 110 captures an image, and thus the image acquisition devices 110 output six images to the processor 120. In this embodiment, the included angle between adjacent image acquisition devices 110 is 0.9 degrees to 1.1 degrees, so that the multi-view stereoscopic image generation method and system of the present application can output multi-view stereoscopic images that are visually smoother and have a stereoscopic effect. The image acquisition devices 110 capture images including the physical correction plate CB, and transmit the images to the processor 120, i.e., step S320 is performed.

[0043] In step S320, the processor 120 receives the images captured by the image acquisition devices 110 to obtain a reference image and an image to be corrected among the images. Specifically, the processor 120 receives the plurality of images captured by the image acquisition devices 110, and the processor 120 analyzes the plurality of images to obtain the reference image and the image to be corrected. For example, when the number of image acquisition devices 110 is odd, the above-mentioned reference image is the image captured / captured by the image acquisition device 110 located at the center. On the other hand, when the number of image acquisition devices 110 is even, two images captured by the two image acquisition devices 110 closest to the center among the plurality of image acquisition devices 110, and the image obtained by interpolating the above-mentioned two images is the reference image. The interpolation operation will be described below. After the processor 120 analyzes the reference image, the remaining images are the images to be corrected. After the processor 120 obtains the reference image and the image to be corrected, i.e., step S330 is performed.

[0044] In step S330, the processor 120 calculates a plurality of homography matrices of each of the to-be-corrected images mapped to the reference image according to the reference image and the to-be-corrected images. Each image is taken with a picture including a physical correction board. Moreover, the pattern of the physical correction board CB is, for example, a regular checkerboard pattern CP, but the present application is not limited thereto. It is worth mentioning that, in the present embodiment, the physical correction board has a color pattern printed thereon, so that the processor 120 can perform position correction conversion and color correction conversion of the feature points in the to-be-corrected images and the reference image through the color pattern on the physical correction board. In the present application, the color correction is not limited to the color of the reference image, but after the color correction conversion, there is no color difference between the reference image and the plurality of to-be-corrected images, which belongs to the color correction conversion mentioned in the present application. It is worth mentioning that the present application has the effect of completing position and color correction by using the same physical correction board CB (i.e., correction board), and in actual operation, a board printed with a color pattern on the wall or placed near the photographer can be used as the physical correction board CB of the present application. Specifically, the processor 120 uses a feature detection method to find a plurality of feature points x i For the checkerboard pattern CP, the black and white alternating corner points in the checkerboard are so-called feature points.

[0045] Then, the homography matrix H is, for example, a 3x3 matrix. Here, the homography matrix is used to describe the projection conversion between the planar coordinates taken by the image acquisition device 110 and the reference image. The homography matrix H is used to describe the projection operation from a planar homogenous coordinate to another planar. As shown in equation (1), the homography matrix H is represented by a 3x3 matrix, where x i is the x' in equation (1).

[0046]

[0047] H is

[0048]

[0049] and x is

[0050]

[0051] In other words, the homography matrix H is the mapping conversion between the feature points in the to-be-corrected image and the feature points in the reference image. Then, in order to solve the homography matrix H, it is assumed that the feature point in the coordinates of the image acquisition device 110 is x i , which is mapped to the known reference image coordinates (Hx i ), and the known feature point in the reference image is x'i The outer product of the two vectors is zero vector, as shown in equation (2). It represents the expected distance minimization equation between the point coordinates of the image feature points to be corrected (i.e., Hx i ) and the reference image feature points (i.e., x' i ) after the transformation by the homography matrix, that is, the distance condition is satisfied under the premise that the outer product between the two vectors is zero.

[0052] Equation (1):

[0053]

[0054] Equation (2):

[0055] x′ i ×Hx i =[0 0 0] T

[0056] Equation (1) and equation (2) can be rewritten and simplified as equation (3) and equation (4), and finally the homography matrix H is solved by singular value decomposition (SVD). The existing singular value decomposition (SVD) method can be used to organize equation (2) to calculate the value of H. For more detailed description of the specific method of organizing equation (2), the H matrix is individually disassembled into row vectors, for example, h1 T in equation (3) is the first row (row) matrix of H. In other words, the row vectors of the homography matrix H represent h1 T , h2 T , and h3 T , respectively. Furthermore, equation (3) is organized into equation (4), and SVD operation is performed on the left matrix in equation (4) to calculate h1, h2, and h3 (i.e., H). In equation (4), x i , y i , and w i are individual scalar quantities, that is, the three components of the actual numerical values of the reference image feature points.

[0057] Equation (3):

[0058]

[0059] Equation (4):

[0060]

[0061] When the homography matrix H is calculated, it is equivalent to a regression matrix. Therefore, according to the homography matrix H, any point xi in the coordinate system of the image acquisition device 110 is multiplied by the homography matrix H to calculate the coordinate value x' i in the known coordinate system.

[0062] The processor 120 calculates the homography matrix H of each image to be corrected mapped to the reference image according to the above formula, and then performs step S340.

[0063] In step S340, the processor 120 performs a matrix decomposition operation on each homography matrix H to obtain a correction matrix of each homography matrix H according to the homography matrix. The homography matrix H includes a combination of rotation, translation, scaling, affine, and perspective projection matrices of the image. In step S340, the processor 120 decomposes each homography matrix H into a perspective matrix, a similarity matrix, and an affine matrix according to the matrix decomposition operation, and the processor 120 uses the similarity matrix and the affine matrix as the correction matrix. Further, the homography matrix H can be decomposed into a perspective transformation matrix H A , an affine transformation matrix H S , and a similarity transformation matrix H

[0064] Equation (5):

[0065]

[0066] After the processor 120 calculates the correction matrix of each image to be corrected relative to the reference image, step S350 is performed. In step S350, based on the correction matrix, the processor 120 performs a correction operation on each image to be corrected to obtain a plurality of corrected images. Specifically, the correction operation is to multiply each image to be corrected by its corresponding correction matrix to obtain a corrected image. Then step S360 is performed, in which the processor 120 outputs a multi-view stereo image. The multi-view stereo image includes the reference image and the corrected images. It is further explained that by using the affine transformation matrix H A and the similarity transformation matrix H S as the correction matrix, the converted image of the image to be corrected still has the perspective effect. In this way, the multi-view stereo image generated by the multi-view stereo image generation system 1 and the method has the perspective effect, and the correction matrix is the affine transformation matrix H A and the similarity transformation matrix H SThe technology allows multi-view stereoscopic images to present a progressive picture with the same position of feature points, so as to achieve a visually good and smooth three-dimensional effect.

[0067] In another embodiment, step S330 further includes the processor 120 performing pivot correction processing on the image to be corrected and the reference image respectively, wherein the pivot correction processing is performed based on a pivot matrix. Then, the processor 120 calculates the homography matrix H of each image to be corrected mapped to the reference image after pivot correction processing.

[0068] Next, in step S350, the processor 120 performs correction and compensation operations on the image to be corrected to obtain a corrected image; the compensation operation is performed based on the inverse matrix of the pivot matrix. Specifically, the pivot correction process first translates the image to be corrected and the reference image to the origin based on the center point of the correction plate in their image content, and then performs the subsequent correction operation using the homography matrix. Furthermore, the image after correction and the reference image are subjected to a compensation operation to translate the image (e.g., the center point of the correction plate) back to its initial position. In other words, the compensation operation compensates the pivot translation matrix of the above-mentioned pivot correction process back to its original position. In another embodiment, the compensation operation is multiplied by the inverse matrix of the reference image (e.g., T in this embodiment). src -1 matrix).

[0069] For example, the homogeneous coordinate center of the image to be corrected is (x src ,y src 1) The coordinate center can be calculated by averaging all feature points, then its matrix T src (That is, the compensation matrix). In other words, the compensation matrix T src Let T be the translation matrix of the center point of the image to be corrected. As shown in equation (6), and T src The inverse matrix T src -1 As shown in equation (7).

[0070] Equation (6):

[0071]

[0072] Equation (7):

[0073]

[0074] Next, the center point of the image to be corrected is (x dst ,y dst ,1), its matrix T dst As shown in equation (8), and its Tdst The inverse matrix T dst -1 (i.e., the pivot matrix), in other words, the pivot matrix is ​​the inverse matrix of the homogeneous coordinate center of the image to be corrected, as shown in equation (9).

[0075] Equation (8):

[0076]

[0077] Equation (9):

[0078]

[0079] As can be seen from the above, in this embodiment, the homography matrix of the correction plate between the image to be corrected and the reference image is H (i.e., from the image to be corrected to the reference image). Therefore, the homography matrix we need to perform pivot correction processing is T. dst -1 HT src That is, the actual object of matrix decomposition is T. dst -1 HT src That is, T dst -1 HT src The matrix decomposition is performed as follows (10).

[0080] Equation (10):

[0081]

[0082] In other words, H s With H A It comes from T dst -1 HT src Similarity transformation matrix and affine transformation matrix of a matrix.

[0083] Because, in this embodiment, its correction matrix (i.e., H) S With H A ) is from T dst -1 HT src Therefore, the homography matrix after pivot correction is T. dst H S H A T src -1 To compensate for the extraction correction matrix (H) s H A ) at T dst -1 HT src T indst -1 and T src translation.

[0084] For example, assume that the image to be corrected is I src matrix, the corrected image is I new matrix as shown in the following equation (11).

[0085] Equation (11):

[0086] I new = T dst H s H A T src -1 I src

[0087] The homography matrix is such that after the image is multiplied by the pivot translation matrix of the pivot correction process, the center point of the image (e.g., the feature point of the physical correction plate) will overlap the quadrant origin. Then, the pivot-corrected image is subjected to a correction operation and then multiplied by the inverse of the pivot translation matrix to translate back to the original position. In this way, the image that is subjected to the pivot correction process and then subjected to the correction operation can avoid the image from being biased during the correction process and the image being enlarged in an unequal ratio, thereby making the output multi-view stereoscopic image more stable and harmonious. It is to be noted that the center point and the coordinate center mentioned in the present disclosure can be the center of the feature point of the physical correction plate present in (i.e., imaged) the image to be corrected or the reference image.

[0088] In another embodiment, in step S360, the processor 120 further performs an interpolation operation on each two images adjacent to each other in the reference images and the corrected images to obtain a plurality of virtual images. Then, the processor 120 outputs a multi-view stereoscopic image including the reference images, the corrected images, and the virtual images. Then, the processor 120 calculates a homography matrix H of each image to be corrected mapped to the pivot-corrected reference images. Figure 4is a schematic diagram of the interpolation operation according to the present application. For example, the first image to be corrected P1, the second image to be corrected P2 and the reference image Pd. Wherein, I1 is the image data of the first image to be corrected P1, I2 is the image data of the first image to be corrected P1, Is1 and Is2 are the image data of the interpolation images (i.e. virtual images S1, S2) at the three equal interval positions between the first image to be corrected P1 and the second image to be corrected P2, and the projection conversion equation therebetween is as equation (12) and equation (13). In the present application, through the interpolation operation technology, the multi-view stereoscopic image generation system 1 and method of the present application have the advantages of setting a small number of image acquisition devices to calculate the multi-view stereoscopic image containing multiple corrected images and multiple virtual images, and outputting multi-view stereoscopic images with high stereoscopic effect while reducing the cost.

[0089] Equation (12):

[0090]

[0091] Equation (13):

[0092]

[0093] In another embodiment, in step S360, the processor 120 further performs translation processing on the corrected images respectively to obtain translated corrected images. In more detail, the corrected images and the reference images are arranged in sequence according to the relative positions of the corresponding image acquisition devices. Moreover, the translation processing is that the corrected images are multiplied by N translation matrices according to the relative N distances between the corrected images and the reference images. Specifically, N is a positive integer, if the corrected image is located on the right side of the reference image (i.e. the relative position of the corresponding image acquisition device 110), the N translation matrices are multiplied; if the corrected image is located on the left side of the reference image, the inverse matrices of the N translation matrices are multiplied. In this way, the multi-view stereoscopic image generation system and method of the present application can make the overall picture of the output multi-view stereoscopic image more expanded by performing translation processing on the corrected images, and make the visual three-dimensional effect more smooth.

[0094] Figure 5 is a schematic diagram of the translation processing according to the present application. As Figure 5As shown, for example, the total number of corrected images and reference images Pd in this embodiment is 7, and the translation matrix T is shown in equation (14). In this embodiment, the reference image is the image arranged in the fourth position in sequence, so the d of the reference image Pd is equal to 4. Next, taking the mth corrected image Cm as an example, since the number of images in this embodiment is 7, the mth corrected image Cm is the seventh image arranged in sequence. The relative distance between the mth corrected image and the reference image Pd is 3, so the mth corrected image is multiplied by 3 translation matrices T. When the ith corrected image is located on the left side of the reference image Pd, the translation matrix T' is the inverse matrix T-1 of the above translation matrix T, and the equation is shown in equation (15). That is, the first corrected image C1 is located on the left side of the reference image Pd (i.e., i is less than d), so the first corrected image C1 is multiplied by 3 translation matrices T-1.

[0095] Equation (14):

[0096]

[0097] Equation (15):

[0098]

[0099] It is additionally noted that in the above embodiment, the translation matrix T is taken as an example of 3 pixels. Therefore, the unit increment of the translation matrix can be adjusted to 1 to any number of pixels according to the user's needs. For example, if the increment unit of the translation matrix is 2, the 3 in the matrix array of equation (13) is replaced by 2, and so on.

[0100] In another embodiment, in step S360, the processor 120 further performs translation processing on the corrected images and the virtual images respectively to obtain translated corrected images and translated virtual images. Figure 6 is a schematic diagram of the interpolation operation and the translation processing according to the present application. As shown, Figure 6 In this embodiment, the number of image acquisition devices is 4, so the corrected images (C1, C2, C3, C4) after the correction processing are four, and the virtual images (SC1, Pd', SC3) obtained by the above interpolation operation are three. In this embodiment, the reference image Pd' is the reference image Pd obtained by the interpolation operation of the second corrected image and the third corrected image. As shown, Figure 6It is known that the distance between the fourth corrected image C4 and the reference image Pd' is 3, and thus the fourth corrected image C4 is multiplied by 3 translation matrices T to complete the translation process. Then, the processor 120 outputs the multi-view stereoscopic image including the translated corrected images (C1', C2', C3', C4'), the translated virtual images (SC1', SC3'), and the reference image Pd'. It is additionally noted that the distance between the image acquisition device 110 and the photographed object in the multi-view stereoscopic image generation system and method of the present application can be set to 1 to 3 meters, and more preferably, the distance between the image acquisition device 110 and the photographed object is 2.5 to 3 meters. However, the present application can also be applied to the photographing of natural scenery (distance greater than 3 meters, or further) and thus the present application should not be limited thereto.

[0101] In summary, in the multi-view stereoscopic image generation system and method of the embodiments of the present application, the homography matrix is decomposed into the perspective transformation matrix, the affine transformation matrix, and the similar transformation matrix, and the affine transformation matrix and the similar transformation matrix are used as the correction matrix of the present application. Thus, the images in the multi-view stereoscopic image output by the multi-view stereoscopic image generation system and method of the present application are corrected by the correction plate, but the perspective effect is maintained. Furthermore, the pivot correction process and the translation process can make the multi-view stereoscopic image more stable and the three-dimensional effect can be adjusted according to the user and the actual use. It is worth noting that the multi-view stereoscopic image generation system and method of the present application can output multiple corrected images by using a small number of image acquisition devices (e.g., cameras) through interpolation operation, so as to reduce the operation process and shorten the operation time, thereby achieving the effect of real-time presentation or real-time output of the stereoscopic photo after photographing.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-view stereoscopic image generating system, comprising: Comprising: a plurality of image acquisition devices for capturing a plurality of images of a physical calibration board; and a processor coupled to the plurality of image acquisition devices, the processor configured to: receive the plurality of images captured by the plurality of image acquisition devices to obtain a reference image and a plurality of images to be calibrated among the plurality of images; calculate a plurality of homography matrices for each of the images to be calibrated mapping to the reference image based on the reference image and the plurality of images to be calibrated; perform a matrix factorization operation on each of the homography matrices based on the plurality of homography matrices to obtain a calibration matrix for each of the homography matrices; perform a calibration operation on each of the images to be calibrated based on the calibration matrix to obtain a plurality of calibrated images; and output a multi-view stereoscopic image, the multi-view stereoscopic image comprising the reference image and the plurality of calibrated images, wherein the processor is further configured to: factorize each of the homography matrices into a perspective matrix, a similarity matrix, and an affine matrix based on performing the matrix factorization operation; and use the similarity matrix and the affine matrix as the calibration matrix, wherein the processor is further configured to: perform a pivot calibration on each of the images to be calibrated and the reference image based on a pivot matrix; and calculate the plurality of homography matrices for each of the images to be calibrated mapping to the pivot-calibrated reference image, wherein the processor performs a position correction and a color correction between each of the images to be calibrated and the reference image based on a color pattern on the physical calibration board.

2. The multi-view stereoscopic image generating system according to claim 1, wherein The pivot matrix is an inverse matrix of a homogeneous coordinate center of each of the images to be calibrated, and the pivot calibration further comprises a compensation matrix, which is a translation matrix of a center point of each of the images to be calibrated.

3. The multi-view stereoscopic image generating system according to claim 1, wherein The processor is further configured to: perform an interpolation operation on each of two images adjacent to each other among the reference image and the plurality of calibrated images to obtain a plurality of virtual images; and output the multi-view stereoscopic image, the multi-view stereoscopic image comprising the reference image, the plurality of calibrated images, and the plurality of virtual images.

4. The multi-view stereoscopic image generating system according to claim 1, wherein, The processor is further configured to: perform an interpolation operation on each of two images adjacent to each other among the reference image and the plurality of calibrated images to obtain a plurality of virtual images; and output the multi-view stereoscopic image, the multi-view stereoscopic image comprising the reference image, the plurality of calibrated images, and the plurality of virtual images. respectively, to obtain a plurality of translated corrected images; the plurality of corrected images and the reference image are sequentially arranged according to their relative positions of the plurality of image acquisition devices; the translation processing is that the plurality of corrected images are multiplied by N translation matrices according to their relative N distances with the reference image; and output the multi-view stereoscopic image including the plurality of translated corrected images and the reference image.

5. The multi-view stereoscopic image generating system according to claim 3, wherein, The step of outputting the multi-view stereoscopic image, the processor is further configured to: respectively, to obtain a plurality of translated corrected images and a plurality of translated virtual images; the plurality of corrected images, the reference image and the plurality of virtual images are sequentially arranged according to their relative positions of the plurality of image acquisition devices; the translation processing is that the plurality of corrected images and the plurality of virtual images are multiplied by N translation matrices according to their relative N distances with the reference image; and output the multi-view stereoscopic image including the plurality of translated corrected images, the plurality of translated virtual images and the reference image.

6. The multi-view stereoscopic image generating system according to claim 3, wherein, When the number of the plurality of image acquisition devices is odd, the reference image is an image acquired by an image acquisition device located at the middle of the plurality of image acquisition devices.

7. The multi-view stereoscopic image generating system according to claim 1, wherein, When the number of the plurality of image acquisition devices is even, the reference image is an image obtained by interpolating two images acquired by two image acquisition devices located at the middle of the plurality of image acquisition devices.

8. A multi-view stereoscopic image generating method, comprising: including: causing a plurality of image acquisition devices to respectively acquire a plurality of images of a physical correction board; a processor receiving the plurality of images acquired by the plurality of image acquisition devices to obtain a reference image and a plurality of images to be corrected from the plurality of images; causing the processor to respectively calculate a plurality of homography matrices of each of the images to be corrected mapped to the reference image according to the reference image and the plurality of images to be corrected; causing the processor to respectively perform matrix factorization operation on each of the homography matrices to obtain a correction matrix of each of the homography matrices from the plurality of homography matrices according to the plurality of homography matrices; based on the correction matrix, causing the processor to respectively perform correction operation on the plurality of images to be corrected to obtain a plurality of corrected images according to the correction matrix; and causing the processor to output a multi-view stereoscopic image including the reference image and the plurality of corrected images, wherein the step of causing the processor to respectively perform the matrix factorization operation on each of the homography matrices to obtain the correction matrix of each of the homography matrices from the plurality of homography matrices further includes: causing the processor to decompose each of the homography matrices into a perspective matrix, a similarity matrix and an affine matrix according to the matrix factorization operation; and causing the processor to take the similarity matrix and the affine matrix as the correction matrix. The processor respectively calculates the homography matrix of each of the plurality of images to be corrected mapping to the reference image according to the reference image and the plurality of images to be corrected, and the method further comprises: The processor respectively performs a pivot correction process on the plurality of images to be corrected and the reference image according to a pivot matrix; and The processor calculates the homography matrix of each of the plurality of pivot-corrected images to be corrected mapping to the pivot-corrected reference image, The processor performs position correction and color correction of feature points in the image between each of the images to be corrected and the reference image through the color pattern on the physical correction board.

9. The multi-view stereoscopic image producing method of claim 8, wherein, The pivot matrix is the inverse matrix of the homogeneous coordinate center of each of the plurality of images to be corrected, and the pivot correction process further comprises a compensation matrix, which is a translation matrix of the center point of each of the plurality of images to be corrected.

10. The multi-view stereoscopic image producing method of claim 8, wherein, The step of outputting the multi-view stereoscopic image further comprises: The processor performs interpolation operation on each two images adjacent to each other in the reference image and the plurality of corrected images to obtain a plurality of virtual images; The processor outputs the multi-view stereoscopic image, which comprises the reference image, the plurality of corrected images and the plurality of virtual images.

11. The multi-view stereoscopic image producing method of claim 8, wherein, The step of outputting the multi-view stereoscopic image further comprises: The processor respectively performs translation processing on the plurality of corrected images to obtain a plurality of translated corrected images; the plurality of corrected images and the reference image are arranged in sequence according to the relative positions of the corresponding plurality of image acquisition devices; the translation processing is that the plurality of corrected images are multiplied by N translation matrices according to the relative N distances between the plurality of corrected images and the reference image; The processor outputs the multi-view stereoscopic image comprising the translated corrected images and the reference image.

12. The multi-view stereoscopic image producing method of claim 10, wherein, The step of outputting the multi-view stereoscopic image further comprises: The processor respectively performs translation processing on the plurality of corrected images and the plurality of virtual images to obtain a plurality of translated corrected images and a plurality of translated virtual images; the plurality of corrected images, the reference image and the plurality of virtual images are arranged in sequence according to the relative positions of the corresponding plurality of image acquisition devices; the translation processing is that the plurality of corrected images and the plurality of virtual images are multiplied by N translation matrices according to the relative N distances between the plurality of corrected images and the reference image; The processor outputs the multi-view stereoscopic image comprising the plurality of translated corrected images, the plurality of translated virtual images and the reference image.

13. The multi-view stereoscopic image producing method of claim 8, wherein, When the number of the plurality of image acquisition devices is odd, the reference image is an image acquired by the image acquisition device located in the middle of the plurality of image acquisition devices.

14. The multi-view stereoscopic image producing method of claim 8, wherein, When the number of the plurality of image acquisition devices is even, the reference image is an image obtained by interpolation operation of two images acquired by the two image acquisition devices located in the middle of the plurality of image acquisition devices.

Images