A naked eye light field 3D display image fusion method and system

Abstract

The present invention provides a naked-eye light field 3D display image fusion method and system, the method comprises collecting a target image and a reference image, performing depth extraction and viewpoint image generation based on the target image and the reference image to obtain a first image and a second image; performing mapping and filtering processing on the first image and the second image to obtain a first mapping image and a second mapping image; performing preliminary fusion processing on the first mapping image and the second mapping image to obtain an initial fused image; performing iterative segmentation and image parallax correction processing on the initial fused image to obtain a corrected image; performing projection transformation on the corrected image to obtain a naked-eye light field 3D display image. The present invention can effectively and accurately extract depth information from the target image, and can also reduce the influence of noise on image fusion, so that the finally obtained naked-eye light field 3D display image is closer to the real 3D effect and more accurate.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion, and specifically relates to a naked-eye light field 3D display image fusion method and system. Background Art

[0002] Naked eye 3D is a general term for technologies that achieve stereoscopic visual effects without the help of external tools such as polarized glasses. The current mainstream naked eye 3D technical means are: slit liquid crystal grating, cylindrical lens, and directional light source. 1. Slit liquid crystal grating. The principle of this technology is that after adding a slit grating in front of the screen, when the image that should be seen by the left eye is displayed on the LCD screen, the opaque stripes will block the right eye; similarly, when the image that should be seen by the right eye is displayed on the LCD screen, the opaque stripes will block the left eye, and by separating the visible images of the left and right eyes, the viewer can see the 3D image; 2. Cylindrical lens, the principle of this technology is to project the pixels corresponding to the left and right eyes respectively into the left and right eyes through the refraction principle of the lens to achieve image separation. Compared with the slit grating technology, the biggest advantage is that the lens will not block the light, so the brightness has been greatly improved; 3. Pointing light source, in simple terms, it is to accurately control the two sets of screens to project images to the left and right eyes respectively.

[0003] For a specific naked-eye light field 3D display image, it is usually necessary to fuse multiple 2D images accordingly to present a 3D visual effect. However, the existing fusion algorithm is only a simple fusion, and its fusion effect is poor, and it cannot effectively simulate a more realistic 3D visual effect. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a naked-eye light field 3D display image fusion method and system, which are used to solve the technical problems in the prior art.

[0005] On the one hand, the present invention provides the following technical solution, a naked eye light field 3D display image fusion method, comprising:

[0006] Acquire a target image and a reference image, and perform depth extraction and viewpoint image generation based on the target image and the reference image to obtain a first image and a second image;

[0007] Performing mapping filtering processing on the first image and the second image to obtain a first mapping image and a second mapping image;

[0008] Performing a preliminary fusion process on the first mapped image and the second mapped image to obtain an initial fused image;

[0009] Iteratively segmenting and correcting the initial fused image to obtain a corrected image;

[0010] The corrected image is projected and transformed to obtain a naked-eye light field 3D display image.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention first collects a target image and a reference image, performs depth extraction and viewpoint image generation based on the target image and the reference image to obtain a first image and a second image; then performs mapping filtering processing on the first image and the second image to obtain a first mapping image and a second mapping image; then performs preliminary fusion processing on the first mapping image and the second mapping image to obtain an initial fused image; then performs iterative segmentation and image parallax correction processing on the initial fused image to obtain a corrected image; finally, performs projection transformation on the corrected image to obtain a 3D display image. The present invention can effectively and accurately extract depth information from the target image, and can also reduce the influence of noise on image fusion, so that the final naked-eye light field 3D display image is closer to the real 3D effect and more accurate.

[0012] Preferably, the step of acquiring a target image and a reference image, and performing depth extraction and viewpoint image generation based on the target image and the reference image to obtain a first image and a second image includes:

[0013] Collecting initial target images captured by several cameras with equally spaced viewpoints, taking the target image corresponding to the middle viewpoint as the reference image and taking the remaining initial target images as the target images;

[0014] Calculate any pixel in the reference image Equivalent pixels in the target image :

[0015] ; ;

[0016] In the formula, is the proportionality coefficient, , Respectively represent the reference image and the target image in , Coordinate differences on the axes;

[0017] Determine the image difference between the target image and the reference image based on the equivalent pixel points :

[0018] ;

[0019] In the formula, represents the reference image, represents the target image;

[0020] Determine the difference range Based on the difference range Determine the optimal difference :

[0021] ;

[0022] Based on the optimal difference Determine target depth , and based on the target depth A first image and a second image are determined.

[0023] Preferably, the optimal difference Determine target depth , and based on the target depth The step of determining the first image and the second image includes:

[0024] Based on the optimal difference Determine target depth , and based on the target depth Several depth images are extracted:

[0025] ;

[0026] In the formula, represents the focal length of the camera, Indicates the distance between two adjacent cameras;

[0027] For several depth images Perform filtering to obtain a filtered image :

[0028] ;

[0029] In the formula, Indicates the preset optimal depth. Indicates the depth of the current depth image, Indicates the preset optimal filtering strength;

[0030] The filtered image corresponding to the left viewpoint camera is used as the first image and the filtered image corresponding to the right viewpoint camera is used as the second image.

[0031] Preferably, the step of performing mapping filtering on the first image and the second image to obtain the first mapped image and the second mapped image comprises:

[0032] Two virtual cameras are respectively determined in world coordinates based on the first image and the second image, and the projection matrices of the virtual cameras are transformed based on a transformation matrix to obtain a transformed projection matrix, wherein the transformation matrix for:

[0033] ;

[0034] In the formula, represents the distance between the viewpoints corresponding to the first image and the second image and the imaging plane of the virtual camera in the X direction, represents the distance between the imaging plane of the virtual camera and the first image and the second image in the Z direction;

[0035] Projecting the first image and the second image to a world coordinate system and continuing to project them onto an imaging plane of the virtual camera based on the changed projection matrix;

[0036] Calculate the corrected depth of the first image and the second image , and based on the modified depth Correcting the first image and the second image to obtain a first corrected image and a second corrected image:

[0037] ;

[0038] In the formula, represents the cutoff function, represents the number of pixels of the first image and the second image projected on the imaging plane of the virtual camera, Represents the horizontal inclination angle of the virtual camera, represents the best difference, Indicates the target depth corresponding to the first image and the second image;

[0039] A first mapped image and a second mapped image are determined based on the first corrected image and the second corrected image.

[0040] Preferably, the step of determining the first mapping image and the second mapping image based on the first corrected image and the second corrected image comprises:

[0041] Performing several mean filtering processes on the first corrected image and the second corrected image, identifying the hole pixel points of the first corrected image and the second corrected image after the mean filtering, and mapping the hole pixel points to the imaging planes of the two virtual cameras to obtain the first mapping point and the second mapping point;

[0042] Determine whether the first mapping point and the second mapping point are empty. If the first mapping point is empty, use the second mapping point to fill the hole pixel point. If the second mapping point is empty, use the first mapping point to fill the hole pixel point. If both the first mapping point and the second mapping point are not empty, compare the size between the first mapping point and the second mapping point. If the first mapping point is larger, use the second mapping point to fill the hole pixel point. If the second mapping point is larger, use the first mapping point to fill the hole pixel point, so as to obtain the first mapping image and the second mapping image.

[0043] Preferably, the step of performing preliminary fusion processing on the first mapped image and the second mapped image to obtain an initial fused image comprises:

[0044] Calculate the fusion weight between the first mapped image and the second mapped image :

[0045] ;

[0046] In the formula, represents the baseline distance between the virtual camera on the left and the viewpoint corresponding to the first mapping image, represents the baseline distance between the virtual camera on the right and the corresponding viewpoint of the second mapping image;

[0047] Based on the fusion weight The first mapping image With the second mapping image Perform preliminary fusion processing to obtain the initial fused image :

[0048] .

[0049] Preferably, the step of performing iterative segmentation and image parallax correction processing on the initial fused image to obtain a corrected image includes:

[0050] Performing guided filtering on the initial fused image to obtain a guided image, and determining an iteration value based on the guided image and the initial fused image :

[0051] ;

[0052] In the formula, Indicates The iteration value after iterations, represents the kernel function, , They represent the spatial domain bandwidth and color domain bandwidth of the initial fused image respectively. , Respectively represent the pixel coordinates of the initial fusion image and the guide image;

[0053] Calculate the iteration difference between the current iteration value and the previous iteration value, and repeat the iteration until the iteration difference is less than the preset difference;

[0054] During the iteration process, the pixel points closest to the spatial domain bandwidth and color domain bandwidth of the initial fusion image are stored in the iteration point set. And assign a value to each pixel ,in, , Respectively represent the color channel, the first pixels;

[0055] will belong to the same The pixel points are updated, and all the pixel points are iteratively updated to obtain several segmented images;

[0056] A rectified image is determined based on the plurality of segmented images.

[0057] Preferably, the step of determining the corrected image based on the plurality of segmented images comprises:

[0058] Randomly selecting one of the segmented images as a standard image, determining the disparity values ​​between the pixels in the remaining segmented images and the standard image, and judging whether the disparity value is less than a disparity threshold;

[0059] If the disparity value is not less than the disparity threshold, the corresponding pixel is regarded as an invalid pixel; if the disparity value is less than the disparity threshold, the corresponding pixel is regarded as a valid pixel;

[0060] A search is performed on the pixel points in the same column of the invalid pixel to search for a valid pixel with the smallest disparity value as a basic pixel, and the disparity value of the invalid pixel is replaced with the disparity value of the basic pixel to obtain a plurality of initial corrected images, and the plurality of initial corrected images are fused to obtain a corrected image.

[0061] Preferably, the step of performing a projection transformation on the corrected image to obtain a naked-eye light field 3D display image comprises:

[0062] Based on the human eyes, respectively set virtual viewpoints in a virtual coordinate system and set a virtual camera at the virtual viewpoints;

[0063] Project the corrected image onto the imaging plane where the virtual camera is located to obtain projection parameters ;

[0064] Based on the projection parameters Determine the naked eye light field 3D display image:

[0065] ;

[0066] In the formula, represents the pixel coordinates of the rectified image, Represents the depth of each target in the rectified image, Represents the internal parameters of the virtual camera.

[0067] In a second aspect, the present invention provides the following technical solution, a naked eye light field 3D display image fusion system, the system comprising:

[0068] An extraction module, used to collect a target image and a reference image, and perform depth extraction and viewpoint image generation based on the target image and the reference image to obtain a first image and a second image;

[0069] A mapping module, configured to perform mapping filtering processing on the first image and the second image to obtain a first mapping image and a second mapping image;

[0070] a fusion module, configured to perform preliminary fusion processing on the first mapping image and the second mapping image to obtain an initial fused image;

[0071] A correction module, used for iteratively segmenting and performing image parallax correction processing on the initial fused image to obtain a corrected image;

[0072] The transformation module is used to perform a projection transformation on the rectified image to obtain a 3D display image.

[0073] In a third aspect, the present invention provides the following technical solution: a computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned naked-eye light field 3D display image fusion method when executing the computer program.

[0074] In a fourth aspect, the present invention provides the following technical solution: a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned naked-eye light field 3D display image fusion method. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1 A flowchart of a naked-eye light field 3D display image fusion method provided in Embodiment 1 of the present invention;

[0077] Figure 2 A structural block diagram of a naked-eye light field 3D display image fusion system provided in Embodiment 2 of the present invention;

[0078] Figure 3 A schematic diagram of the hardware structure of a computer provided in another embodiment of the present invention.

[0079] The embodiments of the present invention will be further described below with reference to the accompanying drawings. DETAILED DESCRIPTION

[0080] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the embodiments of the present invention, and should not be construed as limiting the present invention.

[0081] Embodiment 1

[0082] In the first embodiment of the present invention, Figure 1 As shown, a naked eye light field 3D display image fusion method includes:

[0083] S1, collecting a target image and a reference image, and performing depth extraction and viewpoint image generation based on the target image and the reference image to obtain a first image and a second image;

[0084] Wherein, the step S1 comprises:

[0085] S11, collecting initial target images captured by a number of cameras with equally spaced viewpoints, taking the target image corresponding to the middle viewpoint as a reference image and taking the remaining initial target images as target images;

[0086] Specifically, for a three-dimensional target, a straight line is set within a certain distance in front of it, and several cameras are set on the straight line. The image of the three-dimensional target taken by the several cameras is the initial target image, the image taken by the viewpoint located in the middle is used as the reference image, and the images taken by the other viewpoints are used as the target image.

[0087] S12: Calculate any pixel point in the reference image Equivalent pixels in the target image :

[0088] ; ;

[0089] In the formula, is the proportionality coefficient, , Respectively represent the reference image and the target image in , Coordinate differences on the axes;

[0090] Specifically, for equivalent pixel points, the row coordinates of the equivalent pixel points on the reference image and the target image form a proportional function, and the coefficient of the proportional function is the proportional coefficient in this application.

[0091] S13: Determine the image difference between the target image and the reference image based on the equivalent pixel points :

[0092] ;

[0093] In the formula, represents the reference image, represents the target image;

[0094] Specifically, for identical pixels, the colors between identical pixels are highly similar, so the color difference is minimal. Therefore, the corresponding image difference can be obtained by calculating the corresponding Euclidean distance, which is also expressed as similarity.

[0095] S14. Determine the difference range Based on the difference range Determine the optimal difference :

[0096] ;

[0097] Specifically, the difference range is a parallax range, and the target depth can be obtained by searching for the corresponding depth within the difference range and minimizing the optimal difference.

[0098] S15, based on the optimal difference Determine target depth , and based on the target depth determining a first image and a second image;

[0099] Wherein, the step S15 comprises:

[0100] S151, based on the optimal difference Determine target depth , and based on the target depth Several depth images are extracted:

[0101] ;

[0102] In the formula, represents the focal length of the camera, Indicates the distance between two adjacent cameras.

[0103] S152: performing a plurality of depth images Perform filtering to obtain a filtered image :

[0104] ;

[0105] In the formula, Indicates the preset optimal depth. Indicates the depth of the current depth image, Indicates the preset optimal filtering strength;

[0106] Specifically, the above matrix is ​​specifically a Laplacian operator, and the matrix after the plus sign is specifically the pixel value of the original image. The texture information in the image is extracted by the Laplacian operator, thereby obtaining a filtered image after corresponding filtering intensity.

[0107] S153 . Use the filtered image corresponding to the left viewpoint camera as the first image and use the filtered image corresponding to the right viewpoint camera as the second image.

[0108] S2. performing mapping filtering processing on the first image and the second image to obtain a first mapping image and a second mapping image;

[0109] Wherein, the step S2 comprises:

[0110] S21, determining two virtual cameras in world coordinates based on the first image and the second image, respectively, and transforming the projection matrices of the virtual cameras based on a transformation matrix to obtain a transformed projection matrix, wherein the transformation matrix for:

[0111] ;

[0112] In the formula, represents the distance between the viewpoints corresponding to the first image and the second image and the imaging plane of the virtual camera in the X direction, represents the distance between the imaging plane of the virtual camera and the first image and the second image in the Z direction;

[0113] Specifically, the above transformation process is a shear transformation process.

[0114] S22: Project the first image and the second image to a world coordinate system and continue to project them onto an imaging plane of the virtual camera based on the changed projection matrix.

[0115] S23, calculating the corrected depth of the first image and the second image , and based on the modified depth Correcting the first image and the second image to obtain a first corrected image and a second corrected image:

[0116] ;

[0117] In the formula, represents the cutoff function, represents the number of pixels of the first image and the second image projected on the imaging plane of the virtual camera, Represents the horizontal inclination angle of the virtual camera, represents the best difference, Indicates the target depth corresponding to the first image and the second image.

[0118] S24, determining a first mapping image and a second mapping image based on the first corrected image and the second corrected image;

[0119] Wherein, the step S24 comprises:

[0120] S241, performing mean filtering processing on the first corrected image and the second corrected image for several times, identifying the hole pixel points of the first corrected image and the second corrected image after the mean filtering, and mapping the hole pixel points to the imaging planes of the two virtual cameras to obtain a first mapping point and a second mapping point;

[0121] Specifically, the purpose of multiple mean filtering is to make the filter window small enough until an image with smaller holes is obtained, which can not only ensure that subsequent holes can be filled, but also avoid over-smoothing and retain more original image information.

[0122] S242, determining whether the first mapping point and the second mapping point are empty, if the first mapping point is empty, using the second mapping point to fill the hole pixel point, if the second mapping point is empty, using the first mapping point to fill the hole pixel point, if both the first mapping point and the second mapping point are not empty, comparing the sizes of the first mapping point and the second mapping point, if the first mapping point is larger, using the second mapping point to fill the hole pixel point, if the second mapping point is larger, using the first mapping point to fill the hole pixel point, so as to obtain a first mapping image and a second mapping image;

[0123] Specifically, if the first mapping point and the second mapping point are both empty, correction can be performed in the subsequent step S4.

[0124] S3, performing preliminary fusion processing on the first mapped image and the second mapped image to obtain an initial fused image;

[0125] Wherein, the step S3 comprises:

[0126] S31: Calculate the fusion weight between the first mapped image and the second mapped image :

[0127] ;

[0128] In the formula, represents the baseline distance between the virtual camera on the left and the viewpoint corresponding to the first mapping image, Represents the baseline distance between the virtual camera on the right and the corresponding viewpoint of the second mapping image.

[0129] S32, based on the fusion weight The first mapping image With the second mapping image Perform preliminary fusion processing to obtain the initial fused image :

[0130] .

[0131] S4, performing iterative segmentation and image parallax correction processing on the initial fused image to obtain a corrected image;

[0132] Wherein, the step S4 comprises:

[0133] S41, performing guided filtering processing on the initial fused image to obtain a guided image, and determining an iteration value based on the guided image and the initial fused image :

[0134] ;

[0135] In the formula, Indicates The iteration value after iterations, represents the kernel function, , They represent the spatial domain bandwidth and color domain bandwidth of the initial fused image respectively. , Represent the pixel coordinates of the initial fused image and the guide image respectively.

[0136] S42, calculating the iteration difference between the current iteration value and the previous iteration value, and repeating the iteration until the iteration difference is less than a preset difference.

[0137] S43, storing the pixel points closest to the spatial domain bandwidth and color domain bandwidth of the initial fused image in the iteration process into the iteration point set And assign a value to each pixel ,in, , Respectively represent the color channel, the first Pixels.

[0138] S44, will belong to the same The pixel points are updated, and all the pixel points are iteratively updated to obtain several segmented images;

[0139] Specifically, during the iteration process, segmented images with a pixel number less than a preset pixel number may be eliminated. After the iteration is completed, the image is segmented according to the corresponding assignment to obtain a number of segmented images.

[0140] S45, determining a corrected image based on the plurality of segmented images;

[0141] Wherein, the step S45 comprises:

[0142] S451, randomly selecting one of the segmented images as a standard image, determining disparity values ​​between pixels in the remaining segmented images and the standard image, and judging whether the disparity value is less than a disparity threshold;

[0143] Specifically, when determining the standard image, an image with the largest number of pixels may be selected as the standard image.

[0144] S452: If the disparity value is not less than the disparity threshold, the corresponding pixel is regarded as an invalid pixel; if the disparity value is less than the disparity threshold, the corresponding pixel is regarded as a valid pixel.

[0145] S453, searching for valid pixels with the smallest disparity value in the same column of the invalid pixels as basic pixels, and using the basic pixel disparity value to replace the disparity value of the invalid pixels to obtain a plurality of initial corrected images, and fusing the plurality of initial corrected images to obtain a corrected image;

[0146] Specifically, several initial corrected images may be fused by image registration.

[0147] S5. Performing projection transformation on the corrected image to obtain a naked-eye light field 3D display image.

[0148] Wherein, the step S5 comprises:

[0149] S51, respectively setting virtual viewpoints in a virtual coordinate system based on human eyes and setting a virtual camera at the virtual viewpoints.

[0150] S52, projecting the corrected image onto the imaging plane where the virtual camera is located to obtain projection parameters ;

[0151] Specifically, the projection parameters here can be determined by projecting the pixel points on the rectified image onto the transformation relationship between the pixel points on the imaging plane where the virtual camera is located.

[0152] S53, based on the projection parameters Determine the naked eye light field 3D display image:

[0153] ;

[0154] In the formula, represents the pixel coordinates of the rectified image, Represents the depth of each target in the rectified image, Represents the internal parameters of the virtual camera.

[0155] The naked-eye light field 3D display image fusion method provided in the first embodiment of the present invention first collects a target image and a reference image, performs depth extraction and viewpoint image generation based on the target image and the reference image to obtain a first image and a second image; then performs mapping and filtering processing on the first image and the second image to obtain a first mapping image and a second mapping image; then performs preliminary fusion processing on the first mapping image and the second mapping image to obtain an initial fused image; then performs iterative segmentation and image parallax correction processing on the initial fused image to obtain a corrected image; finally, performs projection transformation on the corrected image to obtain a 3D display image. The present invention can effectively and accurately extract depth information from the target image, and can also reduce the influence of noise on image fusion, so that the final naked-eye light field 3D display image is closer to the real 3D effect and more accurate.

[0156] Embodiment 2

[0157] like Figure 2 As shown, in the second embodiment of the present invention, a naked eye light field 3D display image fusion system is provided, and the system includes:

[0158] Extraction module 1, used to collect a target image and a reference image, and perform depth extraction and viewpoint image generation based on the target image and the reference image to obtain a first image and a second image;

[0159] A mapping module 2, configured to perform mapping filtering processing on the first image and the second image to obtain a first mapping image and a second mapping image;

[0160] A fusion module 3, configured to perform preliminary fusion processing on the first mapping image and the second mapping image to obtain an initial fused image;

[0161] A correction module 4 is used for iteratively segmenting and performing image parallax correction processing on the initial fused image to obtain a corrected image;

[0162] A transformation module 5, used for performing a projection transformation on the corrected image to obtain a 3D display image;

[0163] The extraction module 1 comprises:

[0164] The acquisition submodule is used to acquire initial target images captured by a number of cameras with equidistant viewpoints, and use the target image corresponding to the middle viewpoint as the reference image and the remaining initial target images as the target images;

[0165] The equivalent pixel submodule is used to calculate any pixel in the reference image Equivalent pixels in the target image :

[0166] ; ;

[0167] In the formula, is the proportionality coefficient, , Respectively represent the reference image and the target image in , Coordinate differences on the axes;

[0168] An image difference submodule is used to determine the image difference between the target image and the reference image based on the equivalent pixels. :

[0169] ;

[0170] In the formula, represents the reference image, represents the target image;

[0171] The best difference submodule is used to determine the difference range Based on the difference range Determine the optimal difference :

[0172] ;

[0173] An image determination submodule is used to determine the optimal difference based on the image Determine target depth , and based on the target depth A first image and a second image are determined.

[0174] The image determination submodule comprises:

[0175] Depth unit for the optimal difference based on the Determine target depth , and based on the target depth Several depth images are extracted:

[0176] ;

[0177] In the formula, represents the focal length of the camera, Indicates the distance between two adjacent cameras;

[0178] A filtering unit is used to filter the depth images Perform filtering to obtain a filtered image :

[0179] ;

[0180] In the formula, Indicates the preset optimal depth. Indicates the depth of the current depth image, Indicates the preset optimal filtering strength;

[0181] The image determination unit is used to use the filtered image corresponding to the left viewpoint camera as the first image and the filtered image corresponding to the right viewpoint camera as the second image.

[0182] The mapping module 2 comprises:

[0183] A transformation submodule is used to determine two virtual cameras in world coordinates based on the first image and the second image, and transform the projection matrix of the virtual camera based on the transformation matrix to obtain a transformed projection matrix, wherein the transformation matrix for:

[0184] ;

[0185] In the formula, represents the distance between the viewpoints corresponding to the first image and the second image and the imaging plane of the virtual camera in the X direction, represents the distance between the imaging plane of the virtual camera and the first image and the second image in the Z direction;

[0186] A projection submodule, used for projecting the first image and the second image to a world coordinate system and continuing to project them to an imaging plane of the virtual camera based on a changed projection matrix;

[0187] A correction submodule, used to calculate the corrected depth of the first image and the second image , and based on the modified depth Correcting the first image and the second image to obtain a first corrected image and a second corrected image:

[0188] ;

[0189] In the formula, represents the cutoff function, represents the number of pixels of the first image and the second image projected on the imaging plane of the virtual camera, Represents the horizontal inclination angle of the virtual camera, represents the best difference, Indicates the target depth corresponding to the first image and the second image;

[0190] The mapping image submodule is used to determine a first mapping image and a second mapping image based on the first corrected image and the second corrected image.

[0191] The mapping image submodule includes:

[0192] A hole unit, used to perform mean filtering on the first corrected image and the second corrected image for several times, identify hole pixel points of the first corrected image and the second corrected image after mean filtering, and map the hole pixel points to the imaging planes of the two virtual cameras to obtain first mapping points and second mapping points;

[0193] A filling unit is used to determine whether the first mapping point and the second mapping point are empty. If the first mapping point is empty, the second mapping point is used to fill the hole pixel point. If the second mapping point is empty, the first mapping point is used to fill the hole pixel point. If both the first mapping point and the second mapping point are not empty, the size between the first mapping point and the second mapping point is compared. If the first mapping point is larger, the second mapping point is used to fill the hole pixel point. If the second mapping point is larger, the first mapping point is used to fill the hole pixel point, so as to obtain a first mapping image and a second mapping image.

[0194] The fusion module 3 includes:

[0195] A weight submodule, used to calculate the fusion weight between the first mapped image and the second mapped image :

[0196] ;

[0197] In the formula, represents the baseline distance between the virtual camera on the left and the viewpoint corresponding to the first mapping image, represents the baseline distance between the virtual camera on the right and the corresponding viewpoint of the second mapping image;

[0198] A preliminary fusion submodule is used to The first mapping image With the second mapping image Perform preliminary fusion processing to obtain the initial fused image :

[0199] .

[0200] The correction module 4 comprises:

[0201] The iteration value calculation submodule is used to perform guided filtering on the initial fused image to obtain a guided image, and determine the iteration value based on the guided image and the initial fused image. :

[0202] ;

[0203] In the formula, Indicates The iteration value after iterations, represents the kernel function, , They represent the spatial domain bandwidth and color domain bandwidth of the initial fused image respectively. , Respectively represent the pixel coordinates of the initial fusion image and the guide image;

[0204] An iteration difference calculation submodule is used to calculate the iteration difference between the current iteration value and the previous iteration value, and repeat the iteration until the iteration difference is less than a preset difference;

[0205] The assignment submodule is used to store the pixel points closest to the spatial domain bandwidth and color domain bandwidth of the initial fusion image in the iteration point set during the iteration process. And assign a value to each pixel ,in, , Respectively represent the color channel, the first pixels;

[0206] Update submodules to include modules belonging to the same The pixel points are updated, and all the pixel points are iteratively updated to obtain several segmented images;

[0207] A correction submodule is used to determine a corrected image based on the plurality of segmented images.

[0208] The correction submodule comprises:

[0209] A disparity value calculation submodule, used to select any one of the segmented images as a standard image, determine the disparity values ​​between the pixels in the remaining segmented images and the standard image, and determine whether the disparity value is less than a disparity threshold;

[0210] A pixel judgment submodule, configured to treat the corresponding pixel as an invalid pixel if the disparity value is not less than the disparity threshold, and to treat the corresponding pixel as a valid pixel if the disparity value is less than the disparity threshold;

[0211] A search submodule is used to search the pixel points in the same column of the invalid pixel, and use the valid pixel with the smallest disparity value as the basic pixel, and use the basic pixel disparity value to replace the disparity value of the invalid pixel to obtain a number of initial corrected images, and fuse the several initial corrected images to obtain a corrected image.

[0212] The transformation module 5 comprises:

[0213] A virtual submodule, used to set virtual viewpoints in a virtual coordinate system based on the eyes of a person and to set a virtual camera at the virtual viewpoints;

[0214] A parameter submodule is used to project the corrected image onto the imaging plane where the virtual camera is located to obtain projection parameters ;

[0215] Output submodule for Determine the naked eye light field 3D display image:

[0216] ;

[0217] In the formula, represents the pixel coordinates of the rectified image, Represents the depth of each target in the rectified image, Represents the internal parameters of the virtual camera.

[0218] In some other embodiments of the present invention, the embodiments of the present invention provide the following technical solution: a computer, comprising a memory 102, a processor 101, and a computer program stored in the memory 102 and executable on the processor 101, wherein the processor 101 implements the naked-eye light field 3D display image fusion method as described above when executing the computer program.

[0219] Specifically, the processor 101 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0220] Among them, the memory 102 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 102 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 102 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 102 may be inside or outside the data processing device. In a specific embodiment, the memory 102 is a non-volatile memory. In a specific embodiment, the memory 102 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0221] The memory 102 may be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 101 .

[0222] The processor 101 implements the above-mentioned naked-eye light field 3D display image fusion method by reading and executing the computer program instructions stored in the memory 102 .

[0223] In some embodiments, the computer may further include a communication interface 103 and a bus 100. Figure 3 As shown, the processor 101, the memory 102, and the communication interface 103 are connected via a bus 100 and communicate with each other.

[0224] The communication interface 103 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present invention. The communication interface 103 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0225] The bus 100 includes hardware, software or both, and couples the components of the computer device to each other. The bus 100 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 100 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 100 may include one or more buses. Although embodiments of the present invention describe and illustrate a particular bus, the present invention contemplates any suitable bus or interconnect.

[0226] The computer can execute the naked-eye light field 3D display image fusion method of the present invention based on the acquired naked-eye light field 3D display image fusion system, thereby realizing naked-eye light field 3D display image fusion.

[0227] In some further embodiments of the present invention, in combination with the above-mentioned naked-eye light field 3D display image fusion method, the embodiments of the present invention provide the following technical solutions: a storage medium having a computer program stored thereon, and the computer program implements the above-mentioned naked-eye light field 3D display image fusion method when executed by a processor.

[0228] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0229] More specific examples of readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0230] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0231] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0232] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A naked eye light field 3D display image fusion method, characterized in that: include: Acquire a target image and a reference image, and perform depth extraction and viewpoint image generation based on the target image and the reference image to obtain a first image and a second image; Performing mapping filtering processing on the first image and the second image to obtain a first mapping image and a second mapping image; Performing a preliminary fusion process on the first mapped image and the second mapped image to obtain an initial fused image; Iteratively segmenting and correcting the initial fused image to obtain a corrected image; Performing a projection transformation on the corrected image to obtain a naked-eye light field 3D display image; The step of acquiring a target image and a reference image, and performing depth extraction and viewpoint image generation based on the target image and the reference image to obtain a first image and a second image includes: Collecting initial target images captured by several cameras with equally spaced viewpoints, taking the target image corresponding to the middle viewpoint as the reference image and taking the remaining initial target images as the target images; Calculate any pixel in the reference image Equivalent pixels in the target image : ； ； In the formula, is the proportionality coefficient, , Respectively represent the reference image and the target image in , Coordinate differences on the axes; Determine the image difference between the target image and the reference image based on the equivalent pixel points : ； In the formula, represents the reference image, represents the target image; Determine the difference range Based on the difference range Determine the optimal difference : ； Based on the optimal difference Determine target depth , and based on the target depth A first image and a second image are determined.

2. The naked eye light field 3D display image fusion method according to claim 1, characterized in that: The optimal difference Determine target depth , and based on the target depth The step of determining the first image and the second image includes: Based on the optimal difference Determine target depth , and based on the target depth Several depth images are extracted: ； In the formula, represents the focal length of the camera, Indicates the distance between two adjacent cameras; For several depth images Perform filtering to obtain a filtered image : ； In the formula, Indicates the preset optimal depth. Indicates the depth of the current depth image, Indicates the preset optimal filtering strength; The filtered image corresponding to the left viewpoint camera is used as the first image and the filtered image corresponding to the right viewpoint camera is used as the second image.

3. The naked eye light field 3D display image fusion method according to claim 1, characterized in that: The step of performing mapping filtering on the first image and the second image to obtain a first mapping image and a second mapping image comprises: Two virtual cameras are respectively determined in world coordinates based on the first image and the second image, and the projection matrices of the virtual cameras are transformed based on a transformation matrix to obtain a transformed projection matrix, wherein the transformation matrix for: ； In the formula, represents the distance between the viewpoints corresponding to the first image and the second image and the imaging plane of the virtual camera in the X direction, Indicates the distance between the imaging plane of the virtual camera and the first image and the second image in the Z direction; Projecting the first image and the second image to a world coordinate system and continuing to project them onto an imaging plane of the virtual camera based on the changed projection matrix; Calculate the corrected depth of the first image and the second image , and based on the modified depth Correcting the first image and the second image to obtain a first corrected image and a second corrected image: ； In the formula, represents the cutoff function, represents the number of pixels of the first image and the second image projected on the imaging plane of the virtual camera, Represents the horizontal inclination of the virtual camera, represents the best difference, Indicates the target depth corresponding to the first image and the second image; A first mapped image and a second mapped image are determined based on the first corrected image and the second corrected image.

4. The naked eye light field 3D display image fusion method according to claim 3, characterized in that: The step of determining the first mapping image and the second mapping image based on the first corrected image and the second corrected image comprises: Performing several mean filtering processes on the first corrected image and the second corrected image, identifying the hole pixel points of the first corrected image and the second corrected image after the mean filtering, and mapping the hole pixel points to the imaging planes of the two virtual cameras to obtain the first mapping point and the second mapping point; Determine whether the first mapping point and the second mapping point are empty. If the first mapping point is empty, use the second mapping point to fill the hole pixel point. If the second mapping point is empty, use the first mapping point to fill the hole pixel point. If both the first mapping point and the second mapping point are not empty, compare the size between the first mapping point and the second mapping point. If the first mapping point is larger, use the second mapping point to fill the hole pixel point. If the second mapping point is larger, use the first mapping point to fill the hole pixel point, so as to obtain the first mapping image and the second mapping image.

5. The naked eye light field 3D display image fusion method according to claim 1, characterized in that: The step of performing preliminary fusion processing on the first mapped image and the second mapped image to obtain an initial fused image comprises: Calculate the fusion weight between the first mapped image and the second mapped image : ； In the formula, represents the baseline distance between the virtual camera on the left and the viewpoint corresponding to the first mapping image, represents the baseline distance between the virtual camera on the right and the corresponding viewpoint of the second mapping image; Based on the fusion weight The first mapping image With the second mapping image Perform preliminary fusion processing to obtain the initial fused image : 。 6. The naked eye light field 3D display image fusion method according to claim 1, characterized in that: The step of performing iterative segmentation and image parallax correction processing on the initial fused image to obtain a corrected image includes: Performing guided filtering on the initial fused image to obtain a guided image, and determining an iteration value based on the guided image and the initial fused image : ； In the formula, Indicates The iteration value after iterations, represents the kernel function, , They represent the spatial domain bandwidth and color domain bandwidth of the initial fused image respectively. , Respectively represent the pixel coordinates of the initial fusion image and the guide image; Calculate the iteration difference between the current iteration value and the previous iteration value, and repeat the iteration until the iteration difference is less than the preset difference; During the iteration process, the pixel points closest to the spatial domain bandwidth and color domain bandwidth of the initial fusion image are stored in the iteration point set. And assign a value to each pixel ,in, , Respectively represent the color channel, the first pixels; will belong to the same The pixel points are updated, and all the pixel points are iteratively updated to obtain several segmented images; A rectified image is determined based on the plurality of segmented images.

7. The naked eye light field 3D display image fusion method according to claim 6, characterized in that: The step of determining a rectified image based on the plurality of segmented images comprises: Randomly selecting one of the segmented images as a standard image, determining the disparity values ​​between the pixels in the remaining segmented images and the standard image, and judging whether the disparity value is less than a disparity threshold; If the disparity value is not less than the disparity threshold, the corresponding pixel is regarded as an invalid pixel; if the disparity value is less than the disparity threshold, the corresponding pixel is regarded as a valid pixel; A search is performed on the pixel points in the same column of the invalid pixel to search for a valid pixel with the smallest disparity value as a basic pixel, and the disparity value of the invalid pixel is replaced with the disparity value of the basic pixel to obtain a plurality of initial corrected images, and the plurality of initial corrected images are fused to obtain a corrected image.

8. The naked eye light field 3D display image fusion method according to claim 1, characterized in that: The step of performing a projection transformation on the corrected image to obtain a naked-eye light field 3D display image comprises: Based on the human eyes, respectively set virtual viewpoints in a virtual coordinate system and set a virtual camera at the virtual viewpoints; Project the corrected image onto the imaging plane where the virtual camera is located to obtain projection parameters ; Based on the projection parameters Determine the naked eye light field 3D display image: ； In the formula, represents the pixel coordinates of the rectified image, Represents the depth of each target in the rectified image, Represents the internal parameters of the virtual camera.

9. A naked-eye light field 3D display image fusion system, the system adopts the naked-eye light field 3D display image fusion method according to claim 1, characterized in that: The system comprises: An extraction module, used to collect a target image and a reference image, and perform depth extraction and viewpoint image generation based on the target image and the reference image to obtain a first image and a second image; A mapping module, configured to perform mapping filtering processing on the first image and the second image to obtain a first mapping image and a second mapping image; a fusion module, configured to perform preliminary fusion processing on the first mapping image and the second mapping image to obtain an initial fused image; A correction module, used for iteratively segmenting and performing image parallax correction processing on the initial fused image to obtain a corrected image; The transformation module is used to perform a projection transformation on the rectified image to obtain a 3D display image.

Images