A high-precision multi-view image rendering method for light field cameras

Abstract

The present invention provides a high-precision multi-view image rendering method for a light field camera, which ensures the efficiency and accuracy of multi-view image rendering through the following processing steps. First, in step S1, the center of each microlens image MI is accurately located. Then, in step S2, based on the center positioning result, the MI edge brightness attenuation is compensated to improve the spatial resolution. Step S3 dynamically estimates and adaptively adjusts the image block patch size to ensure continuity and consistency during stitching. Finally, step S4 extracts and accurately stitches the image blocks according to the estimated size to generate a multi-view image with rich details and no visual artifacts. The whole process optimizes brightness uniformity, size adaptability and stitching accuracy, providing an innovative solution for multi-view image rendering.

Description

Technical Field

[0001] The present invention relates to light field imaging technology, and in particular to a high-precision multi-view image rendering method for a light field camera. Background Art

[0002] Light field imaging is an emerging imaging technology that can freely adjust the depth of field and focus during image reconstruction by recording the direction and position information of the light field. The micro-lens array (MLA) structure in the light field camera can capture light field data and generate focused light field images (Lenslet Image). This technology has brought broad application prospects in 3D reconstruction, virtual reality, computational photography and other fields. However, how to efficiently and high-quality render multi-view images (Multi-view Images) from focused light field images remains a technical challenge.

[0003] The existing light field image rendering technology has the following main problems in the processing process:

[0004] 1. Inaccurate positioning of the MI center: When determining the center of a micro-image (MI), existing methods are easily affected by factors such as image noise and lens distortion, resulting in inaccurate positioning. This will directly affect the subsequent cropping and stitching of image patches, resulting in overlaps, gaps, or blurs in the final generated multi-view images.

[0005] 2. Edge brightness attenuation: Due to the optical properties of microlenses, the edge areas of MI often have obvious brightness attenuation. This brightness attenuation will reduce the spatial resolution of the image and affect the quality of the final multi-view image. Existing compensation methods usually use simple linear fitting or global brightness adjustment, which cannot accurately restore the brightness of the edge area.

[0006] 3. Inappropriate patch size: When stitching multi-view images, each MI needs to be patched. However, due to the difference in details in different areas caused by focal length changes, it is difficult for existing methods to adaptively select a suitable patch size, resulting in artifacts or loss of details after stitching. At the same time, for areas with different depths of field, existing methods cannot flexibly adjust the patch size, thus failing to ensure image clarity and details.

[0007] 4. Improper handling of overlap during patch stitching: In the patch stitching process, existing methods usually use a simple average blending method to handle overlapping areas, which can easily lead to blurred image details and affect the visual effect.

[0008] In summary, the existing technology has obvious deficiencies in MI center positioning, edge brightness compensation, patch size adaptive selection and stitching processing, and a more efficient and accurate solution is urgently needed.

[0009] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0010] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and to provide a high-precision multi-view image rendering method for a light field camera.

[0011] To achieve the above object, the present invention adopts the following technical solutions:

[0012] A high-precision multi-view image rendering method for a light field camera comprises the following steps:

[0013] S1: Positioning the center of the microlens image MI generated by each microlens of the light field camera;

[0014] S2: Based on the center positioning result of the microlens image MI in step S1, the attenuation of the edge brightness of the microlens image MI caused by the optical characteristics of the microlens is compensated to achieve uniformity between the edge and the center of the microlens image MI, so as to improve the spatial resolution of the image;

[0015] S3: dynamically estimating the size of the image block patch used for rendering from the microlens image MI that has completed edge brightness compensation in step S2; wherein, by analyzing the matching areas and different depth of field areas of adjacent MIs, the size of the image block patch used for rendering is adaptively adjusted to achieve non-overlapping and non-gap stitching;

[0016] S4: according to the size of the image block patch estimated in step S3, extract the corresponding image block from each microlens image MI, and stitch the extracted image blocks together according to their corresponding positions in the light field image to generate a final multi-view image.

[0017] In some optional implementations, in step S1, the center of the microlens image MI is located using an edge MI interpolation method, specifically including:

[0018] Select and calibrate several microlens images MI at the edge positions around the light field image, where these edge MIs are easy to identify and calibrate due to their clear positions;

[0019] Using computer vision technology or manual annotation methods, determine the center position of the above edge MI to establish reference points at the four corners of the image;

[0020] The bilinear interpolation method is used to infer the center positions of all MIs in the middle area of ​​the image based on the known center positions of the edge MIs. The center positions are determined based on the weighted average of the adjacent known edge MI positions.

[0021] In some optional implementations, in step S1, using computer vision technology to identify the center of the microlens image MI specifically includes:

[0022] Take a light field image with a pure white background, and use the bright spot or circular area formed by each MI on the pure white background to provide image features for subsequent feature extraction and recognition;

[0023] Perform image preprocessing on the pure white background image, including grayscale and binarization, to enhance the bright spots or boundaries of each MI in the image;

[0024] Using computer vision algorithms for feature extraction and recognition, including: using edge detection algorithms to identify circular boundaries, performing circular fitting on the detected boundaries, determining the center coordinates of each MI, and applying feature matching algorithms to extract and match feature points of the MIs;

[0025] Preferably, it also includes: performing verification and calibration, comparing the recognition results of computer vision technology with the manually calibrated edge MI positions, ensuring the accuracy of the recognition results, and making necessary adjustments to the deviation areas.

[0026] In some optional implementations, in step S1, the center of the microlens image MI is located using an AI pairing method based on deep learning, specifically including:

[0027] Prepare the dataset and annotate the light field image matrix, establish the true position label of the center point of each MI, and construct a dataset containing all MIs and their center positions;

[0028] Implement data enhancement to generate diverse training samples to increase the robustness of the model, and adjust the center point position corresponding to each enhanced sample according to the type of data enhancement operation;

[0029] The convolutional neural network (CNN) is used as the model architecture. The network structure includes input layer, convolution layer, pooling layer and fully connected layer to automatically extract light field image features and predict the center point coordinates of MI.

[0030] The loss function uses the Euclidean distance to minimize the error between the predicted center position and the true center position, thereby guiding the optimization of network parameters;

[0031] Perform model training, using the back-propagation algorithm and gradient descent optimizer to iteratively update network parameters to minimize the loss function;

[0032] Perform model validation and adjustment, evaluate model performance and adjust hyperparameters to optimize the model's prediction accuracy;

[0033] Apply the trained model to the new light field image and obtain the predicted value of the center position of each MI through forward propagation;

[0034] Perform a post-processing step to fine-tune the prediction results based on the physical characteristics of the light field camera to ensure the accuracy of the center positioning;

[0035] By calculating the mean Euclidean distance error (MED), the prediction accuracy of the model is quantitatively analyzed, and the error distribution is analyzed by visualization. If there are systematic errors that exceed the standard, the model is adjusted or a new calibration method is introduced.

[0036] In some optional implementations, in step S1, the center of the microlens image MI is located using an AI pairing method based on a combination of deep learning and a physical model, specifically including:

[0037] Prepare the dataset and annotate the light field image matrix, establish the true position label of the center point of each MI, and construct a dataset containing all MIs and their center positions;

[0038] Implement data enhancement to generate multiple different training samples to increase the robustness of the model, and adjust the center point position corresponding to each enhanced sample according to the type of data enhancement operation;

[0039] The convolutional neural network (CNN) is used as the model architecture. The network structure includes input layer, convolution layer, pooling layer and fully connected layer. It automatically extracts light field image features and predicts the center point coordinates of MI.

[0040] Define the loss function and use the Euclidean distance to minimize the error between the predicted center position and the true center position to guide the optimization of network parameters;

[0041] Perform model training, using the back-propagation algorithm and gradient descent optimizer to iteratively update network parameters to minimize the loss function;

[0042] Perform model validation and adjustment, evaluate model performance and adjust hyperparameters to optimize the model’s prediction accuracy;

[0043] Apply the trained model to the new light field image and obtain the predicted value of the center position of each MI through forward propagation;

[0044] Perform corrections based on the physical model to calibrate the AI ​​prediction results according to the physical characteristics of the light field camera to ensure the accuracy of the center positioning. In particular, the weight factor is applied to adjust the center position predicted by AI in combination with the ideal center position calculated based on the physical model to optimize the prediction accuracy.

[0045] Perform error correction, further correct the center position according to imaging conditions or known deviations, and obtain the final predicted value;

[0046] The prediction accuracy of the model is quantitatively analyzed by calculating the mean Euclidean distance error (MED), and the error distribution is analyzed using visualization methods. If there are systematic errors that exceed the standard, the model is further adjusted or a new calibration method is introduced.

[0047] In some optional implementations, in step S2, brightness compensation is performed by shooting a pure white background image, specifically including:

[0048] Take a pure white background image and obtain the bright spot or circular area of ​​each microlens image MI under the pure white background to provide a reference image for the calculation of the brightness attenuation coefficient;

[0049] Calculate the brightness attenuation coefficient of each MI in the pure white background image, and determine the brightness attenuation degree of each pixel by analyzing the image brightness distribution;

[0050] According to the calculated brightness attenuation coefficient, brightness compensation is performed on each MI in the actual light field image to correct the edge brightness attenuation.

[0051] In some optional implementations, in step S2, brightness compensation is performed by artificially fitting the edge brightness attenuation trend, specifically including:

[0052] A brightness attenuation model is established, assuming that the brightness attenuation is a linear relationship proportional to the distance from the pixel to the MI center, and this model is used to describe the brightness attenuation within each MI;

[0053] The least squares method is used to estimate the model parameters. By fitting the brightness distribution of MI in the pure white background image, the coefficient describing the brightness attenuation is determined to minimize the error between the model prediction and the actual brightness.

[0054] According to the fitted brightness attenuation model, the brightness of each MI in the actual light field image is adjusted to correct the edge brightness attenuation.

[0055] In some optional implementations, in step S2, an adaptive brightness compensation method is used to perform brightness compensation, specifically including:

[0056] Collect and annotate light field images and their corresponding brightness attenuation characteristic data, and construct a dataset containing images and brightness attenuation coefficients for training brightness compensation models;

[0057] Using a machine learning algorithm, such as linear regression or a neural network, to train a brightness compensation model, so that the model can adaptively predict the brightness attenuation coefficient according to the characteristics of the input image;

[0058] Define the loss function, use the mean square error between the predicted value and the true brightness attenuation coefficient as the optimization target, and minimize the loss function during the training process to improve the prediction accuracy of the model;

[0059] The trained brightness compensation model is used to predict the brightness attenuation coefficient of each pixel, and then the image is brightness compensated according to the brightness attenuation coefficient to correct the edge brightness attenuation.

[0060] In some optional implementations, in step S2, brightness compensation is performed using a brightness compensation method based on the frequency domain, specifically including:

[0061] Perform Fourier transform on the light field image to convert the spatial domain image into the frequency domain, so as to facilitate the analysis and compensation of brightness attenuation in the frequency domain;

[0062] Constructing a frequency domain brightness compensation filter, which is based on the spectrum characteristics of brightness attenuation and is used to compensate for the brightness attenuation of different frequency components;

[0063] Apply the compensation filter to the frequency domain image, and compensate for the brightness attenuation by point-by-point multiplication of the filter and the frequency domain image;

[0064] An inverse Fourier transform is performed to convert the compensated frequency domain image back to the spatial domain to obtain a compensated light field image with uniform brightness.

[0065] In some optional implementations, in step S2, brightness compensation is performed using a brightness compensation method based on multiple reference images, specifically including:

[0066] Take multiple reference images with different brightness backgrounds, such as black and white checkerboard or grayscale gradient images, to obtain diverse brightness attenuation data;

[0067] Analyze the brightness attenuation characteristics of each reference image and calculate the brightness attenuation coefficient corresponding to each reference image, which reflects the proportional relationship between the image brightness and the average brightness;

[0068] Establish a multi-reference brightness compensation model, integrate the brightness attenuation coefficients of all reference images, and form a unified brightness compensation model through a merging function such as weighted average;

[0069] The multi-reference brightness compensation model is used to perform brightness compensation on the actual light field image, and the image brightness is adjusted by the brightness attenuation coefficient predicted by the multi-reference brightness compensation model to correct the edge brightness attenuation.

[0070] In some optional implementations, in step S2, brightness compensation is performed using a brightness compensation method based on image enhancement technology, specifically including:

[0071] Preprocess the light field image and apply image enhancement algorithms such as histogram equalization or adaptive histogram equalization to improve the contrast and brightness distribution of the image;

[0072] Perform brightness attenuation analysis on the enhanced image and establish a brightness compensation model based on the ratio of the brightness of the enhanced image to the average brightness;

[0073] The brightness compensation model is applied to adjust the brightness of the actual light field image, and the image brightness is compensated by the brightness attenuation coefficient predicted by the model to achieve brightness uniformity.

[0074] In some optional implementations, in step S3, a matching region estimation method based on adjacent microlens images MI is used to determine the size of the image block patch, specifically including:

[0075] Selecting multiple matching points distributed in a hexagon in each microlens image MI to form a set of matching template points for reference;

[0076] Searching for an area matching the reference matching point in the adjacent microlens image MI, and calculating an offset distance of the matching area relative to the reference point;

[0077] The image patch size of the current microlens image is estimated to be the average of the offset distances of all matching points, and this average is used as the standard size for cropping each microlens image.

[0078] In some optional implementations, in step S3, the size of the image patch is determined by using an estimation method for applying different image patch sizes to different depth of field areas, specifically including:

[0079] Based on the matching distance obtained by the matching area estimation method of adjacent microlens images (MI), when the variance of the matching distance exceeds a preset threshold, the automatic division process of the depth of field area is triggered;

[0080] Use image gradient information to perform region division, divide the image into foreground and background areas, and use image gradient algorithms such as the Sobel operator to calculate the gradient value of each pixel as the basis for region division;

[0081] The image block sizes are calculated for the foreground area and the background area respectively, and the image block sizes are determined according to the number of matching points and the offset distance in each area.

[0082] In some optional implementations, in step S3, the size of the image patch is determined using the same area offset method, which specifically includes:

[0083] Finding regions in the current microlens image MI that are highly similar to adjacent MIs, and determining the corresponding relationship between the regions;

[0084] For each feature point in the current MI, identify the matching feature points in the adjacent MI and calculate the offset distance between the two;

[0085] The calculated offset distance is used as the image block size of the current MI.

[0086] In some optional implementations, in step S3, a multi-scale feature matching method is used to determine the size of the image patch, specifically including:

[0087] Extracting feature points of the microlens image MI at multiple different scales to form a set of feature points at each scale;

[0088] For each feature point at each scale, find the corresponding feature point in the adjacent MI and calculate the offset distance between the two;

[0089] The offset distances at all scales are comprehensively considered and the final image block size is calculated by weighted average, where the offset distance at a higher resolution scale is given a greater weight.

[0090] In some optional implementations, in step S3, a feature matching method based on edge enhancement is used to determine the size of the image patch, specifically including:

[0091] Perform edge enhancement processing on the microlens image MI, use edge detection algorithm to highlight the edge features of the image, and obtain an enhanced image;

[0092] Extract feature points on the enhanced image and find corresponding matching feature points in adjacent MIs;

[0093] Calculate the offset distance between the matching feature points and use this offset distance as the image block size of the current MI.

[0094] In some optional implementations, in step S3, an adaptive brightness compensation method is used to determine the size of the image block patch, specifically including:

[0095] Before feature point extraction, adaptive brightness compensation is performed on each microlens image MI, and the brightness of the original image is adjusted using the brightness compensation coefficient to obtain a compensated image;

[0096] Extract feature points using the compensated image and search for matching feature points in adjacent MIs;

[0097] Calculate the offset distance between the matching feature points and use this offset distance as the basis for adjusting the size of the image block;

[0098] The image block size of each MI is dynamically determined according to the calculated offset distance.

[0099] In some optional implementations, in step S3, a texture analysis and enhancement method is used to determine the size of the image patch, specifically including:

[0100] Performing texture analysis on each microlens image MI, identifying texture features in the image, and generating a texture feature map;

[0101] Performing texture enhancement processing on the area with less edge texture identified in the texture feature map, improving the texture features of the area through high-frequency filtering or texture synthesis method, and obtaining an enhanced image;

[0102] Extract feature points on the texture-enhanced image and find corresponding matching feature points in adjacent MIs;

[0103] Calculate the offset distance between the matching feature points and use this offset distance as the image block size of the current MI.

[0104] In some optional implementations, in step S3, a machine learning-based method is used to determine the size of the image patch, specifically including:

[0105] Collect and annotate microlens image MI and its matching results under different brightness and texture conditions, and construct a dataset containing images and actual image block sizes;

[0106] Using machine learning algorithms, such as regression analysis or deep learning, to train an image patch size estimation model that can automatically predict the size of image patches based on the features of the input image;

[0107] Define the loss function as the error measure between the predicted image block size and the actual size, and train the model by minimizing the loss function;

[0108] In practical applications, a new microlens image is input into the trained model to predict the size of the image patch.

[0109] In some optional implementations, in step S4, a simple stitching method is used to stitch together the multi-view images, specifically including:

[0110] According to the image patch size determined in step S3, extracting an image patch of corresponding size from each microlens image MI;

[0111] The extracted image blocks are arranged and stitched according to their corresponding positions in the light field image, and a continuous multi-view image is formed by direct stitching.

[0112] In some optional implementations, in step S4, a gradient gradient processing method is used to stitch the multi-view images, specifically including:

[0113] According to the image patch size determined in step S3, extracting an image patch of corresponding size from each microlens image MI;

[0114] For the overlapping area between image blocks, a weight function is set, which varies with the distance from the overlapping boundary;

[0115] In the overlapping area, the pixel values ​​are linearly mixed according to the weight function;

[0116] The image blocks that have been processed by gradient fading are pieced together according to their corresponding positions in the light field image to generate the final multi-view image.

[0117] In some optional implementations, in step S4, a multi-scale stitching method is used to generate a multi-view image, specifically including:

[0118] At multiple different scales, multiple sets of image blocks of different sizes are extracted, and the image block sets at each scale are processed separately;

[0119] Perform gradient processing on the image blocks at each scale, and stitch the image blocks together according to the predetermined positions to form the preliminary result of the multi-view image at that scale;

[0120] Multi-scale fusion technology is used to weightedly fuse the preliminary results at different scales, with the high-resolution scale having a greater weight, to generate the final multi-view image.

[0121] In some optional implementations, in step S4, a feature point-based stitching method is used to generate a multi-view image, specifically including:

[0122] Extracting feature points from each microlens image MI and assigning an index identifier to each feature point;

[0123] Perform feature point matching in adjacent microlens images and find matching feature point pairs based on the positions and descriptors of the feature points;

[0124] According to the result of feature point matching, the position and size of the image patch of each microlens image are adjusted so that the image patches can be accurately aligned during the stitching process;

[0125] The overlapping areas of the image blocks are subjected to gradient processing, and after smooth transition, the image blocks are stitched together to generate the final multi-view image.

[0126] In some optional implementations, in step S4, a stitching method based on image fusion is used to generate a multi-view image, specifically including:

[0127] Extract the image patch in each microlens image MI;

[0128] Use Laplacian pyramid fusion technology to fuse image blocks, decompose each image block into sub-image levels of different resolutions, and generate Laplacian pyramid representation;

[0129] On each layer of the Laplacian pyramid, the weights of the overlapping areas are adjusted, and the image information of different image blocks is fused;

[0130] Reconstruct the fused results from each layer of the Laplacian pyramid back to the original resolution to obtain the fused image;

[0131] The fused image blocks are stitched together according to their corresponding positions in the light field image to generate the final multi-view image.

[0132] In some optional implementations, in step S4, a deep learning-based stitching method is used to generate a multi-view image, specifically including:

[0133] Collect the microlens image MI and the corresponding multi-view image I MV The paired data is used to construct a training dataset, and a deep learning model such as CNN is used to train a stitching network, which can predict and generate corresponding multi-view images based on the input MI image;

[0134] Define the loss function, use the mean square error between the predicted result and the real multi-view image as the loss function, and optimize the network parameters by minimizing the loss function;

[0135] In practical applications, the new microlens image is input into the trained stitching network, and the network automatically outputs the predicted multi-view image;

[0136] The prediction results output by the network are post-processed to obtain the final multi-view image.

[0137] The present invention also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the high-precision multi-view image rendering method for a light field camera.

[0138] The present invention has the following beneficial effects:

[0139] The present invention proposes an improved method for rendering multi-view images from focused light field images, which effectively solves the above problems in the prior art. The embodiments of the present invention specifically include the following main innovations:

[0140] 1. MI center positioning method based on coordinate interpolation: The coordinate system is modeled by calibrating the positions of several edge MIs, and then the center position of each MI is accurately determined by the coordinate interpolation method. This method can effectively reduce the influence of image noise and lens distortion, and ensure the accuracy of subsequent patch cutting.

[0141] 2. Multi-reference MI brightness compensation technology: When performing MI edge brightness compensation, the present invention utilizes the information of multiple surrounding MIs and comprehensively analyzes the brightness change trends of different MIs to perform more accurate compensation and improve the spatial resolution of the image.

[0142] 3. Adaptive patch size estimation method: The present invention proposes an adaptive patch size estimation method based on adjacent MI matching. By finding highly similar areas in adjacent MIs, the patch size is dynamically adjusted to ensure the details and resolution of different focal length areas. In particular, for MIs spanning different depth of field areas, the present invention can automatically divide the depth of field area and flexibly adjust the patch size for different areas to adapt to different depth of field requirements.

[0143] 4. Patch stitching technology with gradient gradual change processing: In the patch stitching process, the present invention adopts gradient gradual change to process the overlapping area, replacing the traditional average mixing method, which significantly improves the clarity and detail performance of the image.

[0144] 5. Improvement of the multi-reference estimation method: By using the information of all six surrounding MIs to estimate the patch size, the present invention further improves the accuracy of patch size estimation, ensuring seamless connection and high-quality performance of the stitched image.

[0145] Through the above innovations, the present invention effectively solves the problems of inaccurate MI center positioning, edge brightness attenuation, patch size mismatch and improper stitching in the prior art, and realizes efficient and high-quality multi-view image rendering. These technical innovations provide new methods for light field image processing, which have important application value and broad market prospects.

[0146] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0147] Figure 1 The present invention is a flowchart of rendering a multi-view image from a focused light field image according to an embodiment of the present invention.

[0148] Figure 2 Schematic diagram of the same area offset method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0149] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0150] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0151] See also Figure 1 The embodiment of the present invention provides a high-precision multi-view image rendering method for a light field camera, comprising the following steps:

[0152] S1: Positioning the center of the microlens image MI generated by each microlens of the light field camera;

[0153] S2: Based on the center positioning result of the microlens image MI in step S1, the attenuation of the edge brightness of the microlens image MI caused by the optical characteristics of the microlens is compensated to achieve uniformity between the edge and the center of the microlens image MI, so as to improve the spatial resolution of the image;

[0154] S3: dynamically estimating the size of the image block patch for rendering from the microlens image MI that has completed edge brightness compensation in step S2; wherein, by analyzing the matching areas and different depth of field areas of adjacent MIs, the size of the image block patch for rendering is adaptively adjusted to ensure non-overlapping and gap-free stitching;

[0155] S4: according to the size of the image block patch estimated in step S3, extract the corresponding image block from each microlens image MI, and stitch the extracted image blocks together according to their corresponding positions in the light field image to generate a final multi-view image.

[0156] The high-precision multi-view image rendering method for a light field camera of the present invention effectively improves the accuracy and quality of image rendering through a series of innovative steps. First, by accurately locating the center of each microlens image MI, a solid benchmark is provided for subsequent processing, ensuring the accuracy of image processing. Secondly, in view of the edge brightness attenuation problem caused by the optical characteristics of the microlens, the present invention adopts brightness compensation technology to achieve brightness uniformity between the edge and the center of the image, significantly improving the spatial resolution of the image. In addition, the present invention ensures that the image details in different regions and different depths of field are properly processed by dynamically estimating and adaptively adjusting the size of the image block patch, avoiding overlap and gap problems in the stitching process. Finally, the above steps are comprehensively utilized to accurately extract and stitch image blocks to generate high-quality multi-view images with high detail expression and visual effects. Overall, the technical solution of the present invention shows comprehensive advantages from accurate center positioning to brightness compensation, and then to dynamic size estimation and accurate stitching when processing light field images, providing an efficient, accurate and practical solution for the field of multi-view image rendering.

[0157] Compared with traditional methods, the high-precision multi-view image rendering method for a light field camera of the present invention has significant advantages in terms of data processing volume and positioning accuracy.

[0158] The specific embodiments of the present invention are further described below.

[0159] Step S1: Determine MI center

[0160] The center position of each micro-image (MI) is identified through edge MI interpolation or computer vision technology to ensure accurate positioning.

[0161] In a light field camera, each micro-lens (Micro-Lens) generates a micro-image (MI). Accurately locating the center of these MIs is crucial for subsequent patch extraction and stitching. If the center is not accurately located, it may cause image overlap, gaps, blur, and other problems. The following is a detailed process for determining the MI center, including the initial solution and the improved solution:

[0162] Solution 1: Edge MI interpolation method.

[0163] step:

[0164] Calibration edge MI:

[0165] In the light field image, several MIs are manually or through a simple algorithm to calibrate the edge positions around the image. These edge MIs are easy to identify because of their clear positions.

[0166] Calculate the center position of the edge MI:

[0167] Use computer vision technology or manual annotation to determine the center positions of these edge MIs. Suppose we mark the center positions of four edge MIs at the four corners of the image.

[0168] Interpolate to calculate the intermediate MI center:

[0169] Based on the calibrated edge MI center position, the center positions of all MIs in the middle area are calculated by bilinear interpolation. The bilinear interpolation method can use the known center position of the edge MI to infer the center position of each MI in the image.

[0170] The specific interpolation method is as follows:

[0171] For an MI at any position, its center position can be determined by the weighted average of the adjacent known edge MI positions.

[0172] Assume that the center coordinates of the upper left, upper right, lower left, and lower right corner MIs are known to be (Ax, Ay), (Bx, By), (Cx, Cy), and (Dx, Dy). For any MI at (x, y) in the image, its center position can be calculated by bilinear interpolation.

[0173] advantage:

[0174] The calculation is simple and can quickly estimate the center position of the middle area MI.

[0175] Solution 2: Computer Vision Technology Identification

[0176] In order to improve the accuracy of MI center positioning, we introduce computer vision technology to automatically identify the center position of each MI.

[0177] step:

[0178] To capture an image with a pure white background:

[0179] Use a light field camera to take an image with a pure white background. Under the pure white background, each MI will form a bright spot or a circular area with a clear boundary at its center due to the optical properties of the lens.

[0180] Image preprocessing:

[0181] The captured pure white background image is preprocessed, including grayscale, binarization and other operations, to highlight the bright spots or boundaries of each MI.

[0182] Feature extraction and recognition:

[0183] Use computer vision algorithms (such as Hough circle transform, morphological operations, etc.) to identify the boundary or center position of each MI. The specific steps are as follows:

[0184] Boundary Detection: Identify circular boundaries in an image using edge detection algorithms such as Canny edge detection.

[0185] Circular fitting: Use Hough circle transform or least squares method to perform circular fitting on the detected boundaries and determine the center coordinates of each MI.

[0186] Feature matching: For images with complex backgrounds, feature matching algorithms (such as SIFT and SURF) can be used to extract the feature points of each MI and identify its center position through matching.

[0187] Verification and calibration:

[0188] To ensure the accuracy of the recognition results, the computer vision recognition results can be compared and calibrated with the manually calibrated edge MI positions. For areas with obvious deviations, secondary recognition or manual adjustment can be performed.

[0189] advantage:

[0190] With a high degree of automation, it can accurately locate the center of each MI.

[0191] The cumulative error that may be introduced by the edge MI interpolation method is avoided.

[0192] Solution 3: AI-based pairing method

[0193] Problem Definition and Notation

[0194] In light field imaging, each micro-lens (Micro-Lens) generates a micro-lens image (Micro-Image, MI). Assume that the light field image of the light field camera is composed of Our goal is to accurately locate each Central location ,in and Represent the row and column indices in the image respectively.

[0195] 1. Data Preparation

[0196] Annotated dataset:

[0197] Use symbols represents the input light field image matrix, Representative Row, No. MI of the column.

[0198] Indicates each The center point true location label.

[0199] Dataset Contains annotations of all MIs and their center positions in the light field image. The size of the training set is ,in is the total number of MIs.

[0200] Data Augmentation:

[0201] By rotating, scaling, adding noise, etc. Enhance and generate multiple different training samples , increasing the robustness of the model.

[0202] The center point position corresponding to each enhanced sample is adjusted accordingly according to the transformation type.

[0203] 2. Model selection and training

[0204] Select the model architecture:

[0205] Convolutional neural network (CNN) is used as the core architecture to automatically extract image features and locate the center point. Assume that the network parameters are .

[0206] The network input is a light field image , the output is the predicted center point coordinates .

[0207] The specific network structure can be divided into the following layers:

[0208] Input layer: accepts MI images .

[0209] Convolution layer: Extract local features of MI image, the convolution kernel size is , the convolution operation can be expressed as:

[0210]

[0211] in It is The output feature map of the layer, It is The convolution kernel weight matrix of the layer, is the bias, represents the convolution operation, is the activation function (such as ReLU).

[0212] Pooling layer: Downsampling is performed to reduce the dimension of the feature map. The pooling operation is usually maximum pooling or average pooling, which is defined as:

[0213]

[0214] in It is the feature map after pooling.

[0215] Fully connected layer: Flatten the pooled feature map and map it to the output layer, outputting the predicted center coordinates , the calculation formula is:

[0216]

[0217] in is the weight matrix of the fully connected layer, is the bias term.

[0218] Loss function:

[0219] Use Euclidean distance as the loss function to minimize the predicted center position With the real center location The loss function Defined as:

[0220]

[0221] in Represents all learnable parameters of the model.

[0222] Model training:

[0223] Use backpropagation and a gradient descent optimizer (such as Adam) to update network parameters , to minimize the loss function .

[0224] The number of model training iterations is , the learning rate is The formula for updating parameters at each iteration is:

[0225]

[0226] Verification and adjustment:

[0227] Evaluate model performance on the validation set and optimize the model by adjusting model hyperparameters (such as learning rate, convolution kernel size, network depth, etc.).

[0228] 3. Reasoning and Positioning

[0229] Center point prediction:

[0230] For the new light field image , input the trained model to get the predicted value of the center position of each MI .

[0231] The prediction process is forward propagation, and the calculation formula is:

[0232]

[0233] in represents the trained CNN model, are model parameters.

[0234] Post-processing:

[0235] Based on the physical properties of the light field camera (such as the known geometry of the lens array), the AI-predicted results are fine-tuned appropriately.

[0236] The fine-tuning process can use correction formulas based on the physical model, such as applying a fixed offset calibration in a specific direction.

[0237] 4. Performance Evaluation

[0238] Evaluation Metrics:

[0239] Calculate the mean Euclidean distance error (MED) between the predicted center position and the true center position using the formula:

[0240]

[0241] in is the number of MIs in the validation set.

[0242] Visualization and Analysis:

[0243] By visualizing the center position of each MI and comparing the predicted value with the true value, the accuracy and error distribution of the model can be analyzed.

[0244] If there are obvious systematic errors, the model can be further adjusted or new calibration methods can be introduced.

[0245] The AI ​​pairing-based method automatically extracts features from light field images through a deep learning model and efficiently and accurately predicts the center position of each microlens image. Compared with traditional computer vision methods, this method is more robust, especially when faced with complex optical distortion and noise. In addition, the positioning accuracy is further improved through reasonable post-processing and physical model calibration.

[0246] Solution 4: Method based on AI+physical matching

[0247] Problem Definition and Notation

[0248] In light field imaging, each micro-lens (Micro-Lens) generates a micro-lens image (Micro-Image, MI). Assume that the light field image of the light field camera is composed of Our goal is to accurately locate each Central location ,in and Represent the row and column indices in the image respectively.

[0249] 1. Data Preparation

[0250] Annotated dataset:

[0251] Use symbols represents the input light field image matrix, Representative Row, No. MI of the column.

[0252] Indicates each The center point true location label.

[0253] Dataset Contains annotations of all MIs and their center positions in the light field image. The size of the training set is ,in is the total number of MIs.

[0254] Data Augmentation:

[0255] By rotating, scaling, adding noise, etc. Enhance and generate multiple different training samples , increasing the robustness of the model.

[0256] The center point position corresponding to each enhanced sample is adjusted accordingly according to the transformation type.

[0257] 2. Model selection and training

[0258] Select the model architecture:

[0259] Convolutional neural network (CNN) is used as the core architecture to automatically extract image features and locate the center point. Assume that the network parameters are .

[0260] The network input is a light field image , the output is the predicted center point coordinates .

[0261] The specific network structure can be divided into the following layers:

[0262] Input layer: accepts MI images .

[0263] Convolution layer: Extract local features of MI image, the convolution kernel size is , the convolution operation can be expressed as:

[0264]

[0265] in It is The output feature map of the layer, It is The convolution kernel weight matrix of the layer, is the bias, represents the convolution operation, is the activation function (such as ReLU).

[0266] Pooling layer: Downsampling is performed to reduce the dimension of the feature map. The pooling operation is usually maximum pooling or average pooling, which is defined as:

[0267]

[0268] in It is the feature map after pooling.

[0269] Fully connected layer: Flatten the pooled feature map and map it to the output layer, outputting the predicted center coordinates , the calculation formula is:

[0270]

[0271] in is the weight matrix of the fully connected layer, is the bias term.

[0272] Loss function:

[0273] Use Euclidean distance as the loss function to minimize the predicted center position With the real center location The loss function Defined as:

[0274]

[0275] in Represents all learnable parameters of the model.

[0276] Model training:

[0277] Use backpropagation and a gradient descent optimizer (such as Adam) to update network parameters , to minimize the loss function .

[0278] The number of model training iterations is , the learning rate is The formula for updating parameters at each iteration is:

[0279]

[0280] Verification and adjustment:

[0281] Evaluate model performance on the validation set and optimize the model by adjusting model hyperparameters (such as learning rate, convolution kernel size, network depth, etc.).

[0282] 3. Physical model calibration

[0283] Physical model of light field camera:

[0284] Each microlens of a light field camera follows specific geometric and optical laws during imaging. These laws can be described by mathematical models. For example, assuming that the lens array is regular, the center position of the microlens should satisfy a certain geometric distribution.

[0285] Assume that ideally, the center position of each MI satisfies the following relationship:

[0286]

[0287] in and is the fixed distance between adjacent microlenses.

[0288] Corrections based on the physical model:

[0289] Central place for AI predictions Fine-tune it to fit the physical model of the light field camera. Corrected center position Obtained by the following formula:

[0290]

[0291] in It is the ideal central location calculated based on the physical model. It is a weighting factor used to balance the difference between AI prediction results and physical models.

[0292] Error correction:

[0293] By analyzing the results of the calibration, the model can be further modified. For example, if it is found that the deviation in some directions is large, additional correction factors or offsets can be introduced:

[0294]

[0295] in It is a correction introduced according to specific circumstances.

[0296] 4. Reasoning and Positioning

[0297] Center point prediction:

[0298] For the new light field image , input the trained model to get the predicted value of the center position of each MI .

[0299] Physical model calibration:

[0300] According to the physical model of the light field camera, the AI ​​prediction results are fine-tuned to obtain the calibrated center position. .

[0301] Error correction:

[0302] According to specific imaging conditions or known deviations, the center position is further corrected to obtain the final predicted value .

[0303] 5. Performance Evaluation

[0304] Evaluation Metrics:

[0305] Calculate the mean Euclidean distance error (MED) between the predicted center position and the true center position using the formula:

[0306]

[0307] in is the number of MIs in the validation set.

[0308] Visualization and Analysis:

[0309] By visualizing the center position of each MI and comparing the predicted value with the true value, the accuracy and error distribution of the model can be analyzed.

[0310] If there are obvious systematic errors, the model can be further adjusted or new calibration methods can be introduced.

[0311] The AI+physical matching-based method combines the powerful feature extraction capabilities of the deep learning model with the physical model of the light field camera to achieve more accurate MI center positioning. The AI ​​model provides preliminary predictions, and the physical model performs calibration and correction, so that the final positioning result has both the flexibility of AI and conformity to the laws of physics. This method has strong robustness and accuracy in complex imaging environments.

[0312] Step 2: MI edge brightness compensation

[0313] The brightness of the MI edge is compensated by shooting a pure white background image or manually fitting the edge brightness attenuation trend to improve the spatial resolution of the image.

[0314] Due to the optical properties of microlenses, the edge brightness of each micro-image (MI) usually decays, and this brightness decay affects the spatial resolution of the image. In order to improve image quality, it is necessary to compensate the brightness of the MI edge. The following are the currently used solutions and some extended solutions:

[0315] Solution 1: Take a photo with a pure white background.

[0316] To capture an image with a pure white background:

[0317] The input pure white background image is ,in Represents pixel coordinates.

[0318] For each , in the image Extract the corresponding area .

[0319] Calculate the brightness attenuation coefficient:

[0320] Calculate each The average brightness value of the pixels in :

[0321]

[0322] in It should The number of pixels in .

[0323] For each pixel , calculate the brightness attenuation coefficient :

[0324]

[0325] To apply brightness compensation:

[0326] For actual light field images , for each Each pixel To perform brightness compensation:

[0327]

[0328] Advantages: The compensation effect is more accurate because it is directly based on the actual captured data.

[0329] Solution 2: Artificially fitting the edge brightness attenuation trend

[0330] Implementation details:

[0331] Establish brightness attenuation model:

[0332] Assuming that the brightness decay can be described by a linear function model, for each Pixels in , the model is:

[0333]

[0334] in Is pixel to The distance from the center, and are the fitting parameters.

[0335] Fit parameter estimates:

[0336] Fitting model parameters using least squares method and , so that it minimizes the error in the following formula:

[0337]

[0338] Brightness compensation:

[0339] For actual light field images , for each Each pixel in To perform brightness compensation:

[0340]

[0341] Advantages: No need to take additional images, processing can be performed directly based on existing images.

[0342] Solution 3: Adaptive brightness compensation

[0343] Implementation details:

[0344] Data collection and annotation:

[0345] Collect a large number of light field images and their corresponding brightness attenuation characteristic data to build a training set ,in is the brightness attenuation coefficient matrix for each pixel.

[0346] Model training:

[0347] Use machine learning algorithms such as linear regression or neural networks to train a brightness compensation model , so that the model can adaptively predict the brightness attenuation coefficient according to the characteristics of the input image:

[0348]

[0349] The loss function can be defined as the mean square error between the predicted value and the true brightness attenuation coefficient:

[0350]

[0351] Brightness compensation:

[0352] For new images , use the trained model to predict the brightness attenuation coefficient , and then compensate:

[0353]

[0354] Advantages: The compensation effect is more accurate and the adaptive model can handle different shooting conditions and optical characteristics.

[0355] Solution 4: Brightness compensation based on frequency domain

[0356] Implementation details:

[0357] Fourier Transform:

[0358] Light field image Convert to frequency domain :

[0359]

[0360] Frequency domain compensation filter:

[0361] Design of frequency domain brightness compensation filter , which is used to compensate for the brightness attenuation of different frequency components. The filter design can be based on the spectral characteristics of brightness attenuation.

[0362] Apply the compensation filter:

[0363] Perform compensation operations in the frequency domain:

[0364]

[0365] Inverse Fourier transform:

[0366] Convert the compensated frequency domain image back to the spatial domain:

[0367]

[0368] Advantages: Can effectively handle a wide range of brightness attenuation, and the compensation effect is uniform.

[0369] Solution 5: Brightness compensation based on multiple reference images

[0370] Implementation details:

[0371] Take multiple reference images:

[0372] Take multiple reference images with different brightness backgrounds, such as black and white chessboard or grayscale gradient images, and record them as ,in Indicates different reference images.

[0373] Analyze the brightness attenuation characteristics:

[0374] For each reference image, calculate its brightness attenuation coefficient :

[0375]

[0376] in is the average brightness of the reference image.

[0377] Establish a multi-reference brightness compensation model:

[0378] Comprehensively consider the brightness attenuation characteristics of all reference images and establish a multi-reference brightness compensation model :

[0379]

[0380] in is a combining function, which can be a weighted average or other suitable combining method.

[0381] Brightness compensation:

[0382] For actual light field images To perform brightness compensation:

[0383]

[0384] Advantages: Using information from multiple reference images, the compensation model is more comprehensive and accurate.

[0385] Solution 6: Implementation details of brightness compensation based on image enhancement technology:

[0386] Image preprocessing:

[0387] Light field images Use image enhancement algorithms such as Histogram Equalization (HE) or Adaptive Histogram Equalization (AHE):

[0388]

[0389] Brightness decay analysis:

[0390] The enhanced image Conduct brightness attenuation analysis and establish a brightness compensation model :

[0391]

[0392] Brightness compensation:

[0393] For actual light field images Apply the compensation model:

[0394]

[0395] Advantages: Image enhancement technology can effectively improve the brightness and contrast of MI, and the compensation effect is significant.

[0396] Step 3: Estimate the image patch size

[0397] Based on the matching area of ​​adjacent MIs, the patch size is dynamically adjusted to ensure no overlap and gaps when stitching. For MIs spanning different depths of field, different patch sizes are adaptively adjusted.

[0398] In the process of rendering multi-view images, estimating the size of the patch is a key step, especially when dealing with high LED brightness and less edge texture. Ensuring that patches are seamless and gap-free when stitching, and adapting to the needs of different depth of field areas, is the key to achieving high-quality images. The following is a detailed solution:

[0399] In order to render MV (Multi-View), we need to take patches of specific sizes from each MI (Micro-Image, the small round image captured by the Micro-Lens) and stitch them together.

[0400] In order to ensure that there is no overlap and gap when the patches are stitched together, we need to estimate the size of the patch.

[0401] In addition, an adaptive patchsize is required. The In Focus area has a moderate focal length and rich details, so a larger patchsize can be selected; the Out of Focus area has objects that are out of focus and blurred details, so a smaller patchsize needs to be selected to reduce artifacts at the expense of resolution. In order to adapt to this dynamic focus state, an adaptive patchsize is required.

[0402] At the same time, visual errors will occur when the patchsize is incorrect, and the perspective can be switched by adjusting the position of the patch (by adding a uniform fixed offset).

[0403] Solution 1: Matching area based on adjacent MI

[0404] Implementation details:

[0405] Select matching points:

[0406] In each Select 6 matching template points distributed in a hexagon and record them as ,in Represents the index of 6 points.

[0407] The coordinates of these matching points are ,in Indicates Matching points in The pixel coordinates in .

[0408] Calculate the offset distance:

[0409] Search for areas matching these template points in adjacent MIs, denoted as ,in is the index of the adjacent MI, and the corresponding matching point coordinates are .

[0410] Calculate the offset distance of the matching area :

[0411]

[0412] in, Indicates Matching points in and adjacent The distance between.

[0413] Dynamically adjust patchsize:

[0414] Estimate the current MI patch size is the average of all offset distances:

[0415]

[0416] in, yes The patch size, It is The offset distance of the matching points.

[0417] This estimated patch size is used to crop each MI patch and stitch it into the final multi-view image.

[0418] Advantages: Able to adaptively adjust the patch size to meet different image details and resolution requirements.

[0419] Solution 2: Apply different patch sizes to different depth of field areas

[0420] Implementation details:

[0421] Reuse matching results:

[0422] When the variance of the matching distances of the six matching templates in scheme 1 is large, depth of field region division is started.

[0423] Calculate the variance of the matching distance :

[0424]

[0425] in, is the variance of the matching distance, is the average offset distance, calculated as:

[0426]

[0427] Depth of field area division:

[0428] Using the region partitioning algorithm based on image gradient, the image is divided into two parts: foreground (In Focus) and background (Outof Focus), which are respectively denoted as and .

[0429] Use Sobel operator to calculate image gradient :

[0430]

[0431] in, is a light field image, and The images are and Direction gradient.

[0432] MI is divided into regions according to the gradient value, with higher gradient values ​​corresponding to the foreground area and lower gradient values ​​corresponding to the background area.

[0433] Calculate the patch size:

[0434] Calculate the patch size for the foreground and background areas separately:

[0435]

[0436]

[0437] in, and are the patch sizes of the foreground and background areas, respectively. and is the number of matching points in the foreground and background regions.

[0438] Use different patch sizes for cropping to accommodate areas with different depths of field.

[0439] Advantages: Able to adaptively adjust the patch size according to the depth of field to avoid image tearing.

[0440] Solution 3: Same Area Offset Method

[0441] Since the patch is required to have no overlap and no gap after stitching, we can find a highly similar area in the adjacent MI of the current MI and use the offset distance of the area as the patchsize of the current MI, such as Figure 2 shown.

[0442] In the next MI, the content originally located at position 1 has moved to position 2. In other words, the content at position 2 of the next MI is the same as the content at position 1 of the current MI. We use the offset of position 2 relative to position 1 as the patchsize of the current MI, which can just make sure that the patches from the two MIs do not overlap. Then, the patches are cut out according to the patchsize and then pieced together to form a multi-view image.

[0443] Implementation details:

[0444] Find the same area:

[0445] For each , find the area in the adjacent MI that is highly similar to it. Assume that the position of the feature point in the current MI is , find matching feature points in adjacent MI .

[0446] Calculate the offset distance:

[0447] Calculate the offset distance between matching feature points :

[0448]

[0449] in, Represents the distance between the matching points in the current MI and the adjacent MI.

[0450] Dynamically adjust patchsize:

[0451] The offset distance As the patch size of the current MI :

[0452]

[0453] in, It is the patch size of the current MI.

[0454] This estimated patch size is used to crop each MI patch and stitch it into the final multi-view image.

[0455] Advantages: Ensure that patches have no overlap or gaps when they are joined.

[0456] Solution 4: Multi-scale feature matching

[0457] Implementation details:

[0458] Multi-scale feature extraction:

[0459] Extract feature points of MI at different scales (multi-resolution). Set multiple scales , the feature points at each scale are ,in Indicates the index of the scale.

[0460] Matching feature points:

[0461] For each feature point at each scale, calculate the offset distance from the corresponding feature point in the adjacent MI :

[0462]

[0463] in, is in scale The offset distance below, yes In scale The coordinates of the feature points below.

[0464] Comprehensive multi-scale matching results:

[0465] Calculate the weighted average of the offset distances at all scales to get the final patch size :

[0466]

[0467] in, is the weight of each scale, and usually the higher resolution scale has a larger weight.

[0468] Advantages: Multi-scale feature matching can adapt to different levels of detail and improve matching accuracy.

[0469] Solution 5: Feature matching based on edge enhancement

[0470] Implementation details:

[0471] Edge enhancement processing:

[0472] Perform edge enhancement on the MI image, for example, obtain an enhanced image through Canny edge detection :

[0473]

[0474] in, is the enhanced image, Indicates that the image is processed using the Canny edge detection algorithm.

[0475] Extract feature points:

[0476] Extract feature points on the enhanced image , and calculate its matching area in the adjacent MI .

[0477] Calculate the offset distance and adjust the patchsize:

[0478] Calculate the offset distance of the matching area and as the patch size :

[0479]

[0480] in, is the offset distance calculated after enhancement processing, is the estimated patch size.

[0481] Advantages: Edge enhancement processing improves the prominence of feature points and enhances matching accuracy.

[0482] Solution 6: Adaptive brightness compensation

[0483] Implementation details:

[0484] Brightness compensation:

[0485] Before feature point extraction, the brightness of each MI is adaptively compensated. Let the compensated image be :

[0486]

[0487] in, is the brightness compensation coefficient, is the compensated image.

[0488] Extract feature points and match them:

[0489] The image after compensation Extract feature points , and calculate the matching area in the adjacent MI .

[0490] Calculate the offset distance and adjust the patchsize:

[0491] Calculate the offset distance of the matching area and as the patch size :

[0492]

[0493] in, is the offset distance calculated after compensation, is the estimated patch size.

[0494] Advantages: Adaptive brightness compensation can balance MI brightness differences and improve the accuracy of feature point matching.

[0495] Solution 7: Texture analysis and enhancement

[0496] Implementation details:

[0497] Texture Analysis:

[0498] Perform texture analysis on each MI to identify areas with less edge texture. Use Local Binary Pattern (LBP) or Gabor filter to perform texture analysis and obtain texture feature map. :

[0499]

[0500] in, yes Texture feature map.

[0501] Texture enhancement:

[0502] Texture enhancement is performed on areas with less edge texture, such as by enhancing texture features through high-frequency filtering or texture synthesis. The enhanced image is :

[0503]

[0504] in, is a texture enhancement operation, is the enhanced image.

[0505] Extract feature points and match them:

[0506] After the enhancement Extract feature points , and calculate the matching area in the adjacent MI .

[0507] Calculate the offset distance and adjust the patchsize:

[0508] Calculate the offset distance of the matching area and as the patch size :

[0509]

[0510] in, is the offset distance calculated after enhancement processing, is the estimated patch size.

[0511] Advantages: Texture enhancement processing can improve the accuracy of feature point matching and reduce blocking effects.

[0512] Solution 8: Patchsize estimation based on machine learning

[0513] Implementation details:

[0514] Data collection and annotation:

[0515] Collect a large number of MI images under different brightness and texture conditions and their matching results to build a training set ,in is the actual patch size of the MI image.

[0516] Model training:

[0517] Use machine learning algorithms (e.g. regression analysis, deep learning) to train a patchsize estimation model , so that the model can predict the patch size based on the characteristics of the input image:

[0518]

[0519] in, is the patch size predicted by the model, is the trained model, are the parameters of the model.

[0520] The loss function can be defined as the mean squared error between the predicted value and the true patch size:

[0521]

[0522] in, is the loss function of the model, is the number of samples in the training set.

[0523] Patchsize estimation in practical applications:

[0524] In practical applications, input a new MI image , estimate the patch size through the trained model :

[0525]

[0526] Using estimated patch size To crop each MI patch and stitch it into the final multi-view image.

[0527] Advantages: The machine learning algorithm is able to handle complex lighting and texture conditions and provide accurate patchsize estimates.

[0528] Step 4: Take image patches from each microlens image MI and stitch the multi-view images together

[0529] According to the estimated patchsize, patches are taken from each MI and these patches are stitched together by gradient fading processing to generate high-quality multi-view images.

[0530] According to the estimated patch size, patches are extracted from each micro-image (MIs), and a high-quality multi-view image is generated through a reasonable stitching strategy. This process needs to ensure that there is no overlap or gap between the stitched patches, and handle the overlapping areas well to reduce visual artifacts and block effects.

[0531] Solution 1: Simple splicing

[0532] Implementation details:

[0533] Extract the patch:

[0534] Based on the estimated patch size , from each Extract a fixed-size patch from . Assume for The pixel position in , the extracted patch is recorded as :

[0535]

[0536] in, and For relative The offset of the center point.

[0537] To combine patches:

[0538] The extracted patches are stitched together according to their positions in the light field image to generate a multi-view image :

[0539]

[0540] in, The final multi-view image is generated.

[0541] Advantages: Simple implementation and small amount of calculation.

[0542] Solution 2: Gradient processing

[0543] Implementation details:

[0544] Extract the patch:

[0545] Also based on the estimated patch size , from each Extract the patch .

[0546] Gradient processing:

[0547] For overlapping areas, set a weight function , so that Gradually changes in the overlapping area. Assume that the width of the overlapping area is , then the weight function can be defined as:

[0548]

[0549] in, Indicates the distance from the current point to the overlap boundary.

[0550] Linearly blend the pixel values ​​in the overlapping area:

[0551]

[0552] in, and are the pixel values ​​of adjacent patches respectively.

[0553] To combine patches:

[0554] The processed patches are stitched together according to their positions in the light field image to generate the final multi-view image .

[0555] Advantages: Gradient processing can effectively smooth the edges and reduce visual artifacts.

[0556] Solution 3: Multi-scale stitching

[0557] Implementation details:

[0558] Multi-scale patch extraction:

[0559] At different scales Extract multiple patch sets of different sizes ,in For scale The following patch.

[0560] Multi-scale stitching:

[0561] At every scale On the left, gradient processing is performed separately, and the patches are stitched together to generate stitching results of multiple scales. .

[0562] Multi-scale fusion:

[0563] The stitching results at different scales are weighted fused to obtain the final multi-view image. :

[0564]

[0565] in, is the weight of each scale, and usually the scale with higher resolution has a larger weight.

[0566] Advantages: Multi-scale stitching can better handle different levels of detail and improve image quality.

[0567] Solution 4: Feature point-based stitching

[0568] Implementation details:

[0569] Extract feature points:

[0570] For each Extract feature points ,in Indicates the index of the feature point.

[0571] Feature point matching:

[0572] According to the location and descriptor of the feature points, in the adjacent Find matching feature point pairs in .

[0573] Adjust patch position and size:

[0574] According to the result of feature point matching, adjust the current The patch position and size are adjusted so that the patches can be accurately aligned when stitching.

[0575] Gradient processing:

[0576] Perform gradient processing on the overlapping areas, and after smooth transition, stitch the patches together to generate the final multi-view image. .

[0577] Advantages: Feature point matching can improve the accuracy of stitching and reduce blocking effects.

[0578] Solution 5: Image fusion-based stitching

[0579] Implementation details:

[0580] Extract the patch:

[0581] For each Extracting patches .

[0582] Image Fusion:

[0583] Use Laplacian pyramid fusion technology to fuse patches:

[0584] Decompose each patch into sub-image levels of different resolutions to generate a Laplacian pyramid representation ,in Represents a layer index.

[0585] In each layer The weights of the overlapping areas are adjusted, and the image information of different patches is fused:

[0586]

[0587] The fused results are reconstructed back to the original resolution from each layer of the Laplacian pyramid, and we get .

[0588] Seamless stitching:

[0589] The fused patches are stitched together according to their positions in the light field image to generate the final multi-view image .

[0590] Advantages: Image fusion technology can effectively process overlapping areas, improve image quality and reduce stitching traces.

[0591] Solution 6: Deep learning-based stitching

[0592] Implementation details:

[0593] Train the combined network:

[0594] Collecting training dataset ,in is the input MI image, is the corresponding multi-view image.

[0595] Train a stitching network using a convolutional neural network (CNN) or other suitable deep learning model :

[0596]

[0597] in, is the combined result of network prediction, are the parameters of the network.

[0598] The loss function can be defined as the mean square error between the predicted result and the true multi-view image:

[0599]

[0600] Practical application of stitching:

[0601] In practical applications, the new MI image Input into the trained stitching network, the network will automatically generate stitched multi-view images .

[0602] Generate multi-view images:

[0603] After post-processing, the final multi-view image is obtained , directly used in applications.

[0604] Advantages: Deep learning models can handle complex stitching processes, provide high-quality stitching results, and handle a variety of complex situations.

[0605] The above schemes of the embodiments of the present invention provide multiple ways to extract patches from each MI and stitch them into multi-view images. Different schemes have different focuses on processing overlapping areas, stitching accuracy, and computational complexity. According to specific application requirements and computing resources, the most appropriate stitching strategy can be selected to ensure that the generated multi-view images are of high quality, with few artifacts and good visual effects. The variables and symbols in all formulas are clearly stated to ensure the reproducibility of the scheme.

[0606] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0607] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0608] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0609] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0610] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0611] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0612] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0613] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0614] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0615] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0616] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0617] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0618] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A high-precision multi-view image rendering method for a light field camera, characterized in that: The steps include: S1: Positioning the center of the microlens image MI generated by each microlens of the light field camera; S2: Based on the center positioning result of the microlens image MI in step S1, the attenuation of the edge brightness of the microlens image MI caused by the optical characteristics of the microlens is compensated to achieve uniformity between the edge and the center of the microlens image MI, so as to improve the spatial resolution of the image; S3: dynamically estimating the size of the image block patch used for rendering from the microlens image MI that has completed edge brightness compensation in step S2; wherein, by analyzing the matching areas and different depth of field areas of adjacent MIs, the size of the image block patch used for rendering is adaptively adjusted to achieve non-overlapping and non-gap stitching; S4: according to the size of the image block patch estimated in step S3, extract the corresponding image block from each microlens image MI, and stitch the extracted image blocks together according to their corresponding positions in the light field image to generate a final multi-view image.

2. The high-precision multi-view image rendering method for a light field camera according to claim 1, characterized in that: In step S1, the center of the microlens image MI is located using the edge MI interpolation method, which specifically includes: Select and calibrate a number of microlens images MI at the edge positions around the light field image, where these edge MIs are easy to identify and calibrate due to their clear positions; Using computer vision technology or manual annotation methods, determine the center position of the above edge MI to establish reference points at the four corners of the image; The bilinear interpolation method is used to infer the center positions of all MIs in the middle area of ​​the image based on the known center positions of the edge MIs. The center positions are determined based on the weighted average of the adjacent known edge MI positions.

3. The high-precision multi-view image rendering method for a light field camera according to claim 1, characterized in that: In step S1, the center of the microlens image MI is identified using computer vision technology, specifically including: Take a light field image with a pure white background, and use the bright spot or circular area formed by each MI on the pure white background to provide image features for subsequent feature extraction and recognition; Perform image preprocessing on the pure white background image, including grayscale and binarization, to enhance the bright spots or boundaries of each MI in the image; Using computer vision algorithms for feature extraction and recognition, including: using edge detection algorithms to identify circular boundaries, performing circular fitting on the detected boundaries, determining the center coordinates of each MI, and applying feature matching algorithms to extract and match feature points of the MIs; It also includes: performing verification and calibration, comparing the recognition results of computer vision technology with the manually calibrated edge MI positions, ensuring the accuracy of the recognition results, and adjusting the deviation areas; or, In step S1, the center of the microlens image MI is located using an AI pairing method based on deep learning, specifically including: Prepare the dataset and annotate the light field image matrix, establish the true position label of the center point of each MI, and construct a dataset containing all MIs and their center positions; Implement data enhancement to generate diverse training samples to increase the robustness of the model, and adjust the center point position corresponding to each enhanced sample according to the type of data enhancement operation; The convolutional neural network (CNN) is used as the model architecture. The network structure includes input layer, convolution layer, pooling layer and fully connected layer to automatically extract light field image features and predict the center point coordinates of MI. The loss function uses the Euclidean distance to minimize the error between the predicted center position and the true center position, thereby guiding the optimization of network parameters; Perform model training, using the back-propagation algorithm and gradient descent optimizer to iteratively update network parameters to minimize the loss function; Perform model validation and adjustment, evaluate model performance and adjust hyperparameters to optimize the model's prediction accuracy; Apply the trained model to the new light field image and obtain the predicted value of the center position of each MI through forward propagation; Perform a post-processing step to fine-tune the prediction results based on the physical characteristics of the light field camera to ensure the accuracy of the center positioning; By calculating the mean Euclidean distance error (MED), the prediction accuracy of the model is quantitatively analyzed, and the error distribution is analyzed by visualization. If there are systematic errors that exceed the standard, the model is adjusted or a new calibration method is introduced. or, In step S1, the center of the microlens image MI is located using an AI pairing method based on a combination of deep learning and a physical model, specifically including: Prepare the dataset and annotate the light field image matrix, establish the true position label of the center point of each MI, and construct a dataset containing all MIs and their center positions; Implement data enhancement to generate multiple different training samples to increase the robustness of the model, and adjust the center point position corresponding to each enhanced sample according to the type of data enhancement operation; The convolutional neural network (CNN) is used as the model architecture. The network structure includes input layer, convolution layer, pooling layer and fully connected layer. It automatically extracts light field image features and predicts the center point coordinates of MI. Define the loss function and use the Euclidean distance to minimize the error between the predicted center position and the true center position to guide the optimization of network parameters; Perform model training, using the back-propagation algorithm and gradient descent optimizer to iteratively update network parameters to minimize the loss function; Perform model validation and adjustment, evaluate model performance and adjust hyperparameters to optimize the model’s prediction accuracy; Apply the trained model to the new light field image and obtain the predicted value of the center position of each MI through forward propagation; Perform corrections based on the physical model to calibrate the AI ​​prediction results according to the physical characteristics of the light field camera to ensure the accuracy of the center positioning. In particular, the weight factor is applied to adjust the center position predicted by the AI ​​in combination with the ideal center position calculated based on the physical model to optimize the prediction accuracy. Perform error correction, further correct the center position according to imaging conditions or known deviations, and obtain the final predicted value; The prediction accuracy of the model is quantitatively analyzed by calculating the mean Euclidean distance error (MED), and the error distribution is analyzed using visualization methods. If there are systematic errors that exceed the standard, the model is further adjusted or a new calibration method is introduced.

4. The high-precision multi-view image rendering method for a light field camera according to any one of claims 1 to 2, characterized in that: In step S2, brightness compensation is performed by shooting a pure white background image, which specifically includes: Take a pure white background image and obtain the bright spot or circular area of ​​each microlens image MI under the pure white background to provide a reference image for the calculation of the brightness attenuation coefficient; Calculate the brightness attenuation coefficient of each MI in the pure white background image, and determine the brightness attenuation degree of each pixel by analyzing the image brightness distribution; According to the calculated brightness attenuation coefficient, brightness compensation is performed on each MI in the actual light field image to correct the edge brightness attenuation.

5. The high-precision multi-view image rendering method for a light field camera according to any one of claims 1 to 2, characterized in that: In step S2, brightness compensation is performed by artificially fitting the edge brightness attenuation trend, which specifically includes: A brightness attenuation model is established, assuming that the brightness attenuation is a linear relationship proportional to the distance from the pixel to the MI center, and this model is used to describe the brightness attenuation within each MI; The least squares method is used to estimate the model parameters. By fitting the brightness distribution of MI in the pure white background image, the coefficient describing the brightness attenuation is determined to minimize the error between the model prediction and the actual brightness. According to the fitted brightness attenuation model, the brightness of each MI in the actual light field image is adjusted to correct the edge brightness attenuation; or, In step S2, an adaptive brightness compensation method is used to perform brightness compensation, which specifically includes: Collect and annotate light field images and their corresponding brightness attenuation characteristic data, and construct a dataset containing images and brightness attenuation coefficients for training brightness compensation models; Using a machine learning algorithm, such as linear regression or a neural network, to train a brightness compensation model, so that the model can adaptively predict the brightness attenuation coefficient according to the characteristics of the input image; Define the loss function, use the mean square error between the predicted value and the true brightness attenuation coefficient as the optimization target, and minimize the loss function during the training process to improve the prediction accuracy of the model; Use the trained brightness compensation model to predict the brightness attenuation coefficient of each pixel, and then perform brightness compensation on the image according to the brightness attenuation coefficient to correct the edge brightness attenuation; or, In step S2, brightness compensation is performed using a brightness compensation method based on frequency domain, specifically including: Perform Fourier transform on the light field image to convert the spatial domain image into the frequency domain, so as to facilitate the analysis and compensation of brightness attenuation in the frequency domain; Constructing a frequency domain brightness compensation filter, which is based on the spectrum characteristics of brightness attenuation and is used to compensate for the brightness attenuation of different frequency components; Apply the compensation filter to the frequency domain image, and compensate for the brightness attenuation by point-by-point multiplication of the filter and the frequency domain image; Performing inverse Fourier transform to convert the compensated frequency domain image back to the spatial domain to obtain a compensated light field image with uniform brightness; or, In step S2, brightness compensation is performed using a brightness compensation method based on multiple reference images, specifically including: Take multiple reference images with different brightness backgrounds, such as black and white checkerboard or grayscale gradient images, to obtain diverse brightness attenuation data; Analyze the brightness attenuation characteristics of each reference image and calculate the brightness attenuation coefficient corresponding to each reference image, which reflects the proportional relationship between the image brightness and the average brightness; Establish a multi-reference brightness compensation model, integrate the brightness attenuation coefficients of all reference images, and form a unified brightness compensation model through a merging function such as weighted average; The multi-reference brightness compensation model is used to perform brightness compensation on the actual light field image, and the image brightness is adjusted by the brightness attenuation coefficient predicted by the multi-reference brightness compensation model to correct the edge brightness attenuation; or, In step S2, brightness compensation is performed using a brightness compensation method based on image enhancement technology, specifically including: Preprocess the light field image and apply image enhancement algorithms such as histogram equalization or adaptive histogram equalization to improve the contrast and brightness distribution of the image; Perform brightness attenuation analysis on the enhanced image and establish a brightness compensation model based on the ratio of the brightness of the enhanced image to the average brightness; The brightness compensation model is applied to adjust the brightness of the actual light field image, and the image brightness is compensated by the brightness attenuation coefficient predicted by the model to achieve brightness uniformity.

6. The high-precision multi-view image rendering method for a light field camera according to any one of claims 1 to 2, characterized in that: In step S3, a matching region estimation method based on adjacent microlens images MI is used to determine the size of the image block patch, which specifically includes: Selecting multiple matching points distributed in a hexagon in each microlens image MI to form a set of matching template points for reference; Searching for an area matching the reference matching point in the adjacent microlens image MI, and calculating an offset distance of the matching area relative to the reference point; The image patch size of the current microlens image is estimated to be the average of the offset distances of all matching points, and this average is used as the standard size for cropping each microlens image.

7. The high-precision multi-view image rendering method for a light field camera according to any one of claims 1 to 2, characterized in that: In step S3, the size of the image patch is determined by using an estimation method that applies different image patch sizes to different depth of field areas, specifically including: Based on the matching distance obtained by the matching area estimation method of adjacent microlens images (MI), when the variance of the matching distance exceeds a preset threshold, the automatic division process of the depth of field area is triggered; Use image gradient information to perform region division, divide the image into foreground and background areas, and use image gradient algorithms such as the Sobel operator to calculate the gradient value of each pixel as the basis for region division; The image block sizes are calculated for the foreground area and the background area respectively, and the image block sizes are determined according to the number of matching points and the offset distance in each area; or, In step S3, the size of the image patch is determined by using the same area offset method, which specifically includes: Finding regions in the current microlens image MI that are highly similar to adjacent MIs, and determining the corresponding relationship between the regions; For each feature point in the current MI, identify the matching feature points in the adjacent MI and calculate the offset distance between the two; The calculated offset distance is used as the image block size of the current MI; or, In step S3, a multi-scale feature matching method is used to determine the size of the image patch, which specifically includes: Extracting feature points of the microlens image MI at multiple different scales to form a set of feature points at each scale; For each feature point at each scale, find the corresponding feature point in the adjacent MI and calculate the offset distance between the two; The final image block size is calculated by taking into account the offset distances at all scales through weighted average, where the offset distance at a higher resolution scale is given a greater weight. or, In step S3, a feature matching method based on edge enhancement is used to determine the size of the image patch, which specifically includes: Perform edge enhancement processing on the microlens image MI, use edge detection algorithm to highlight the edge features of the image, and obtain an enhanced image; Extract feature points on the enhanced image and find corresponding matching feature points in adjacent MIs; Calculate the offset distance between the matching feature points and use this offset distance as the image block size of the current MI; or, In step S3, an adaptive brightness compensation method is used to determine the size of the image block patch, which specifically includes: Before feature point extraction, adaptive brightness compensation is performed on each microlens image MI, and the brightness of the original image is adjusted using the brightness compensation coefficient to obtain a compensated image; Extract feature points using the compensated image and search for matching feature points in adjacent MIs; Calculate the offset distance between the matching feature points and use this offset distance as the basis for adjusting the size of the image block; According to the calculated offset distance, the image block size of each MI is dynamically determined; or, In step S3, texture analysis and enhancement methods are used to determine the size of the image patch, specifically including: Performing texture analysis on each microlens image MI, identifying texture features in the image, and generating a texture feature map; Performing texture enhancement processing on the area with less edge texture identified in the texture feature map, improving the texture features of the area through high-frequency filtering or texture synthesis method, and obtaining an enhanced image; Extract feature points on the texture-enhanced image and find corresponding matching feature points in adjacent MIs; Calculate the offset distance between the matching feature points and use this offset distance as the image block size of the current MI; or, In step S3, a machine learning-based method is used to determine the size of the image patch, specifically including: Collect and annotate microlens image MI and its matching results under different brightness and texture conditions, and construct a dataset containing images and actual image block sizes; Using machine learning algorithms, such as regression analysis or deep learning, to train an image patch size estimation model that can automatically predict the size of image patches based on the features of the input image; Define the loss function as the error measure between the predicted image block size and the actual size, and train the model by minimizing the loss function; In practical applications, a new microlens image is input into the trained model to predict the size of the image patch.

8. The high-precision multi-view image rendering method for a light field camera according to any one of claims 1 to 2, characterized in that: In step S4, a simple stitching method is used to stitch together the multi-view images, which specifically includes: According to the image patch size determined in step S3, extracting an image patch of corresponding size from each microlens image MI; The extracted image blocks are arranged and stitched according to their corresponding positions in the light field image, and a continuous multi-view image is formed by direct stitching.

9. The high-precision multi-view image rendering method for a light field camera according to any one of claims 1 to 2, characterized in that: In step S4, a gradient processing method is used to stitch the multi-view images, which specifically includes: According to the image patch size determined in step S3, extracting an image patch of corresponding size from each microlens image MI; For the overlapping area between image blocks, a weight function is set, which varies with the distance from the overlapping boundary; In the overlapping area, the pixel values ​​are linearly mixed according to the weight function; The image blocks that have been processed by gradient fading are stitched together according to their corresponding positions in the light field image to generate the final multi-view image; or, In step S4, a multi-scale stitching method is used to generate a multi-view image, which specifically includes: At multiple different scales, multiple sets of image blocks of different sizes are extracted, and the image block sets at each scale are processed separately; Perform gradient processing on the image blocks at each scale, and stitch the image blocks together according to the predetermined positions to form the preliminary result of the multi-view image at that scale; Using multi-scale fusion technology, the preliminary results at different scales are weighted and fused, with the high-resolution scale having a greater weight, to generate the final multi-view image; or, In step S4, a feature point-based stitching method is used to generate a multi-view image, which specifically includes: Extracting feature points from each microlens image MI and assigning an index identifier to each feature point; Perform feature point matching in adjacent microlens images and find matching feature point pairs based on the positions and descriptors of the feature points; According to the result of feature point matching, the position and size of the image patch of each microlens image are adjusted so that the image patches can be accurately aligned during the stitching process; Perform gradient processing on the overlapping areas of the image blocks, and after smooth transition, stitch the image blocks together to generate the final multi-view image; or, In step S4, a stitching method based on image fusion is used to generate a multi-view image, which specifically includes: Extract the image patch in each microlens image MI; Use Laplacian pyramid fusion technology to fuse image blocks, decompose each image block into sub-image levels of different resolutions, and generate Laplacian pyramid representation; On each layer of the Laplacian pyramid, the weights of the overlapping areas are adjusted, and the image information of different image blocks is fused; Reconstruct the fused results from each layer of the Laplacian pyramid back to the original resolution to obtain the fused image; The fused image blocks are stitched together according to their corresponding positions in the light field image to generate the final multi-view image; or, In step S4, a deep learning-based stitching method is used to generate a multi-view image, which specifically includes: Collect the microlens image MI and the corresponding multi-view image I MV The paired data is used to construct a training dataset, and a deep learning model such as CNN is used to train a stitching network, which can predict and generate corresponding multi-view images based on the input MI image; Define the loss function, use the mean square error between the predicted result and the real multi-view image as the loss function, and optimize the network parameters by minimizing the loss function; In practical applications, the new microlens image is input into the trained stitching network, and the network automatically outputs the predicted multi-view image; The prediction results output by the network are post-processed to obtain the final multi-view image.

10. A non-transitory computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the high-precision multi-view image rendering method for a light field camera according to any one of claims 1 to 9 is implemented.

Images