Depth estimation method and apparatus based on light field imaging, and terminal device

Abstract

The application relates to the technical field of image processing, and provides a depth estimation method and device based on light field imaging and a terminal device, the method comprising the following steps: generating epipolar plane images along multiple directions based on four-dimensional light field data of a target seed; performing multi-feature texture saliency evaluation on a space region corresponding to each macro-pixel based on the four-dimensional light field data, and screening out texture effective macro-pixels; performing focal plane diversity analysis on each texture effective macro-pixel, and calculating a depth reliability weight; determining a dominant texture direction of the texture effective macro-pixel; obtaining a weighting coefficient of the epipolar plane image in each direction according to the depth reliability weight and the dominant texture direction; performing depth estimation on the epipolar plane image in each direction respectively, and obtaining a depth estimation result in each direction; and performing weighted fusion on the depth estimation result by using the weighting coefficient, and outputting a final depth map of the target seed. In this way, the robustness of depth estimation is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a depth estimation method, apparatus, and terminal device based on light field imaging. Background Technology

[0002] Depth recovery based on light field is usually achieved by epipolar plane image (EPI) analysis. This analysis method mainly generates EPI slices by fixing a set of spatial coordinates and angular coordinates. The projection of object points in the scene at different viewpoints forms a linear structure in the EPI domain. The slope of the linear structure is inversely proportional to the depth of the object point. That is, the depth estimation problem can be transformed into the problem of detecting the direction of the linear structure in the EPI domain.

[0003] Existing optical field depth estimation methods generally suffer from the fundamental flaw of collinearity of texture directions. This is especially true when the dominant texture direction of the target surface is parallel to the EPI sampling direction, causing the parallax gradient to vanish, resulting in severe artifacts or reconstruction holes in the depth estimation. Moreover, traditional methods apply uniform weights to all angle views, making it difficult to distinguish between reliable and unreliable directions. This problem is particularly prominent on seed samples with significant texture features, such as rice and bitter melon.

[0004] It is evident that the optical field depth estimation methods in related technologies suffer from poor robustness. Summary of the Invention

[0005] This application aims to address the problem of poor robustness in light field depth estimation methods in related technologies, and provides a depth estimation method, apparatus, and terminal device based on light field imaging.

[0006] To solve the above problems, this application is implemented as follows:

[0007] In a first aspect, this application provides a depth estimation method based on light field imaging, including: The original light field image of the target seed is acquired, and the original light field image is decoded into four-dimensional light field data. Based on the four-dimensional light field data, an epipolar plane image along multiple directions is generated. Based on the four-dimensional light field data, a multi-feature texture saliency evaluation is performed on the spatial region corresponding to each macro-pixel to screen out effective texture macro-pixels. By performing focal plane diversity analysis on each effective macropixel of the texture through digital refocusing, the depth reliability weight of each effective macropixel of the texture is calculated. By using gradient direction exponential analysis, a gradient magnitude-weighted direction histogram is constructed within the spatial region corresponding to the effective macro-pixel of the texture, and the dominant texture direction of the effective macro-pixel is determined based on the peak direction of the direction histogram. Based on the depth reliability weight and the dominant texture direction, the voting results of the epipolar plane images in each direction are generated by soft voting weighting. The voting results are then normalized to obtain the weighting coefficients of the epipolar plane images in each direction. Depth estimation is performed on the epipolar plane image in each direction to obtain the depth estimation results for each direction; The depth estimation results are weighted and fused using the weighting coefficients to output the final depth map of the target seed, which is used to characterize the three-dimensional surface structure of the target seed.

[0008] Secondly, this application provides a depth estimation device based on light field imaging, comprising: The acquisition module is used to acquire the original light field image of the target seed, decode the original light field image into four-dimensional light field data, and generate an epipolar plane image along multiple directions based on the four-dimensional light field data. The evaluation and screening module is used to evaluate the saliency of multi-feature textures in the spatial region corresponding to each macro-pixel based on the four-dimensional light field data, and to screen out the effective macro-pixels of texture. The first analysis module is used to perform focal plane diversity analysis on each effective macro pixel of the texture through digital refocusing and calculate the depth reliability weight of each effective macro pixel of the texture. The second analysis module is used to construct a gradient magnitude-weighted orientation histogram in the spatial region corresponding to the effective macro-pixel of the texture through gradient orientation exponential analysis, and to determine the dominant texture orientation of the effective macro-pixel of the texture based on the peak orientation of the orientation histogram. The first processing module is used to generate voting results for the epipolar plane images of each direction by soft voting weighting based on the depth reliability weight and the dominant texture direction, and to normalize the voting results to obtain the weighting coefficients of the epipolar plane images of each direction. The second processing module is used to perform depth estimation on the epipolar plane image in each direction to obtain the depth estimation results in each direction. The output module is used to perform weighted fusion of the depth estimation results using the weighting coefficients, and output the final depth map of the target seed, which is used to characterize the three-dimensional surface structure of the target seed.

[0009] Thirdly, this application provides a terminal device including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0010] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0011] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the method described in the first aspect.

[0012] In a sixth aspect, this application provides a computer program product stored in a storage medium, which is executed by at least one processor to perform the steps of the method described in the first aspect.

[0013] Compared with the prior art, this application has the following beneficial effects: by performing multi-directional EPI analysis on the light field data of the target seed, and combining focal plane diversity depth confidence assessment and gradient direction index anisotropy feature decoding, the texture direction collinearity problem is effectively suppressed, and the robustness of depth estimation is significantly improved. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic flowchart of a depth estimation method based on light field imaging provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a depth estimation device based on light field imaging provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] The terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: including A alone, B alone, C alone, and the presence of both A and B, both B and C, both A and C, and the presence of A, B, and C.

[0018] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0019] The depth estimation method based on light field imaging provided in this application will be described below.

[0020] See Figure 1 , Figure 1 This is a schematic flowchart of a depth estimation method based on light field imaging provided in an embodiment of this application. Figure 1 The depth estimation method based on light field imaging shown can be executed by terminal devices such as mobile phones, computers, and agricultural equipment.

[0021] like Figure 1 As shown, the depth estimation method based on light field imaging provided in this application may include the following steps: Step 101: Obtain the original light field image of the target seed, decode the original light field image into four-dimensional light field data, and generate an epipolar plane image along multiple directions based on the four-dimensional light field data.

[0022] The aforementioned target seeds can be understood as biological samples with significant surface texture features, such as bitter gourd seeds and rice seeds.

[0023] In some embodiments, the original light field image of the target seed can be acquired based on a tunable light field imaging architecture.

[0024] The tunable light field imaging architecture includes an objective lens, an electrically tunable lens, a microlens array, and an image sensor arranged sequentially along the optical path.

[0025] The focal length of the aforementioned primary objective lens can be set to 25.8 mm, which is used for preliminary imaging and to converge the reflected light from the surface of the target seed to subsequent optical elements.

[0026] The focal length of the aforementioned electrically tunable lens can be set to 76.8 mm, and its aperture value can be set to 2. The electrically tunable lens is positioned behind the main objective lens and supports dynamic focal length adjustment. It can automatically or manually adjust optical parameters according to the seed morphology.

[0027] In some embodiments, the electrically tunable lens supports dynamic switching between high-resolution mode and extended depth-of-field mode.

[0028] For example, when the target seed is flat and has a fine texture, such as the net-like structure of bitter gourd seeds, the electrically tunable lens can be switched to high-resolution mode to ensure lateral resolution and thus capture fine surface texture features.

[0029] For example, when the target seed has high curvature and significant depth variation, such as the longitudinal keel structure of rice seeds, the electrically tuned lens can be switched to extended depth-of-field mode to ensure the depth range and thus obtain complete seed contour depth information.

[0030] The microlens array described above is positioned after the electrically tunable lens. The focal length of a single microlens can be set to 100 µm, and the center-to-center distance between adjacent microlenses can be set to 20 µm.

[0031] The aforementioned image sensor can be a complementary metal-oxide-semiconductor (CMOS) sensor, which is positioned after the microlens array, and the pixel pitch of the image sensor can be set to 1.5 µm, and it is used to acquire the original image of the light field of the target seed.

[0032] In some embodiments, the tunable light field imaging architecture further includes a computational processing unit connected to an image sensor for performing computational operations such as decoding, depth estimation, and 3D reconstruction on the raw light field image.

[0033] In some embodiments, the target seed can be placed on the sample stage of a tunable light field imaging architecture, and the seed surface can be uniformly illuminated by adjusting the illumination conditions. After being reflected by the seed surface, the light passes sequentially through the main objective lens, an electrically tunable lens, and a microlens array before being projected onto the image sensor. Due to the function of the microlens array, each microlens can refract the light received from different directions to different pixel positions on the image sensor, thereby forming macropixel units. The raw light field image output by the image sensor can be a single mosaic-like raw light field image composed of tens of thousands of macropixels, such as outputting a raw light field image in RAW format, and the spatial and angular information of the target seed is interwoven and encoded in this raw light field image.

[0034] In some embodiments, the acquired raw light field image can be decoded based on the computing processing unit and decoded into four-dimensional light field data.

[0035] In some embodiments, the four-dimensional light field data obtained after decoding can be represented as follows: ,in: These are spatial coordinates, corresponding to the position indices of macropixels on the sensor plane, used to describe the distribution of light in space; These are angular coordinates, corresponding to the position index of a sub-pixel within a macro-pixel, used to describe the direction of light propagation.

[0036] In this way, the interwoven spatial and angular information in the original light field image is separated into four independent dimensions, providing a structured data foundation for subsequent epipolar plane image generation and depth estimation.

[0037] In some embodiments, from four-dimensional light field data In, through fixed angular coordinates It can extract sub-aperture views from different observation directions, and each sub-aperture view can be understood as a two-dimensional image taken by a virtual camera from a specific perspective.

[0038] For example, when the angular resolution is This allows for the extraction of 49 sub-aperture views from different observation directions. Among these, parallax exists between the extracted sub-aperture views, and the magnitude of the parallax is inversely proportional to the depth of the object point.

[0039] Understandably, angular resolution middle The value of can be determined based on the actual application scenario, and is not limited here.

[0040] In some embodiments, a fixed spatial coordinate can be used. and an angular coordinate From four-dimensional light field data Two-dimensional slices are extracted to obtain an epipolar plane image.

[0041] Specifically, the epipolar plane image can be obtained using the following formula:

[0042] In the formula, spatial coordinates are fixed in the spatial domain. In the angular domain, fixed angular coordinates Let spatial coordinates and angular coordinates Free variation to extract the epipolar plane image from the corresponding viewpoint. .

[0043] In the epipolar plane image, the projections of the same object point under different viewpoints form a straight line, and the slope of this line is inversely proportional to the depth of the object point. Therefore, the depth estimation problem is transformed into the problem of detecting the orientation of linear structures within the epipolar plane image domain.

[0044] In some embodiments, two-dimensional slices can be extracted from four-dimensional light field data along 0° horizontally, 90° vertically, 45° main diagonal, and 135° secondary diagonal to obtain epipolar plane images along multiple directions.

[0045] Extracting the epipolar plane image in the horizontal direction can be understood as fixing spatial coordinates. and angular coordinates ,exist Extract the epipolar plane image from the plane.

[0046] Extracting the epipolar plane image in the vertical direction can be understood as fixing spatial coordinates. and angular coordinates ,exist Extract the epipolar plane image from the plane.

[0047] Extracting the epipolar plane image along the main diagonal can be understood as a synchronous change. coordinates and Coordinates are used to extract the epipolar plane image in the corresponding direction.

[0048] Extracting the epipolar plane image along the secondary diagonal can be understood as an inverse transformation. coordinates and Coordinates are used to extract the epipolar plane image in the corresponding direction.

[0049] In this embodiment, by acquiring epipolar plane images along multiple directions, the problem of parallax gradient vanishing when an epipolar plane image in a single direction is parallel to the seed surface texture direction can be avoided, and a multi-directional, structured data foundation can be provided for subsequent texture adaptive depth estimation.

[0050] Step 102: Based on the four-dimensional light field data, perform multi-feature texture saliency evaluation on the spatial region corresponding to each macro-pixel, and screen out the effective macro-pixels of texture.

[0051] In this embodiment, by performing multi-feature texture saliency evaluation on the spatial region corresponding to each macro pixel, effective macro pixels with texture are selected, which can ensure that the subsequent calculation process can focus on macro pixels containing reliable geometric information, providing a stable basis for adaptive EPI direction weighting.

[0052] In some embodiments, the step of performing multi-feature texture saliency evaluation on the spatial region corresponding to each macro-pixel based on the four-dimensional light field data, and filtering out effective texture macro-pixels, includes: Based on the four-dimensional light field data, local image patches are extracted from the spatial region corresponding to each macro pixel; Calculate the gradient magnitude, orientation entropy, local intensity variance, and structural tensor coherence of the local image patch; The gradient magnitude, the orientation entropy, the local intensity variance, and the structural tensor coherence are weighted and fused to obtain the texture saliency score for each macro-pixel; Based on the texture saliency score of each macropixel, effective macropixels of texture are selected.

[0053] In this embodiment, the above process ensures that subsequent calculations can focus on macro-pixels containing reliable geometric information, providing a stable basis for adaptive EPI direction weighting.

[0054] The gradient magnitudes described above are used to characterize the edge strength of local regions, reflecting the sharpness of the texture. A larger gradient magnitude indicates more pronounced edges and richer texture information in that region.

[0055] The aforementioned directional entropy is used to evaluate the consistency of texture orientation within a local region. A smaller directional entropy indicates a more concentrated texture orientation, while a larger directional entropy indicates a more dispersed texture orientation.

[0056] The aforementioned local intensity variance measures the dispersion of pixel intensity within a local region, reflecting texture contrast. A larger local intensity variance indicates stronger texture contrast and more prominent texture.

[0057] The aforementioned structural tensor coherence is used to characterize the anisotropic robustness of texture within a local region, reflecting the strength of the principal direction of the texture. Specifically, the closer the coherence is to 1, the more significant the principal direction of the texture is, i.e., the stronger the anisotropy; the closer the coherence is to 0, the more dispersed or isotropic the texture direction is.

[0058] In some embodiments, the texture saliency score of a macro-pixel can be calculated based on the following formula:

[0059] In the formula, The texture saliency score represents the macro-pixel. Indicates the gradient magnitude. Represents directional entropy. Indicates the local strength variance. This represents the coherence of the structure tensor.

[0060] Among them, the weighting coefficient It can be determined through experimental data, and is used to balance edge saliency, orientation consistency, contrast, and anisotropic features.

[0061] In some embodiments, the calculated texture saliency score can be used as a reference. Compare with the preset rating: If the texture saliency score of macro pixels If the score is greater than or equal to the preset score, the macro pixel is determined as a valid macro pixel of the texture and used for subsequent depth estimation calculations. If the texture saliency score of macro pixels If the score is less than the preset score, the macro pixel is identified as an invalid texture macro pixel and excluded from subsequent processing.

[0062] In some embodiments, the preset score can be set to 0.8.

[0063] Step 103: Perform focal plane diversity analysis on each effective macropixel of the texture through digital refocusing, and calculate the depth reliability weight of each effective macropixel of the texture.

[0064] In this embodiment, the depth reliability weight of each effective macro-pixel of the texture is calculated to distinguish between reliable and unreliable depth estimation signals, providing a quantitative basis for subsequent adaptive weighting of directions.

[0065] In some embodiments, the step of performing focal plane diversity analysis on each effective macropixel of the texture through digital refocusing to calculate the depth reliability weight of each effective macropixel of the texture includes: For each effective macro pixel of the texture, sub-aperture views of the effective macro pixel of the texture in different observation directions are obtained. The sub-aperture views in different observation directions are aggregated at different depth levels to obtain the focal plane diversity response curve. Sharpness, variability, and significance parameters are extracted from the focal plane diversity response curves. The sharpness parameter, the variability parameter, and the saliency parameter are weighted and fused to obtain the depth reliability weight of each effective macropixel of the texture.

[0066] In some embodiments, the refocusing parameters can be changed. To simulate different depth levels, with step size It is discretized into 20 depth levels.

[0067] For each depth level, sub-aperture views from different observation directions can be aggregated based on the following formula:

[0068] In the formula, This represents the focal plane diversity response curve obtained by aggregation. This represents the number of sampling points for each macropixel in the angular dimension, and its value can be 7. The sub-aperture pixel index encodes different observation directions, with a value range of [value range missing]. ; The distance between the centers of the microlenses; These are refocusing parameters used to control the refocusing depth; The center coordinates of the current macro pixel; This is a bilinear interpolation function used to obtain pixel values ​​at non-integer coordinate positions.

[0069] In some embodiments, when the refocusing parameter When the corresponding refocusing depth matches the true physical depth of the macropixel, constructive interference occurs between the sub-aperture views after parallax correction. Light rays from different viewpoints superimpose in phase, resulting in enhanced intensity after convergence, and thus a focal plane diversity response curve. It exhibits a significant peak. The location of the peak is... This reflects the true depth information of the macropixel.

[0070] In some embodiments, when the refocusing depth does not match the true depth, there is a phase difference between the sub-aperture views, and destructive interference occurs, resulting in a weakened intensity after convergence, and thus a focal plane diversity response curve. The situation is gradually stabilizing.

[0071] In this embodiment, the peak sharpness and significance of the focal plane diversity response curve can directly reflect the depth reliability of the macropixel; the sharper and more significant the peak, the more reliable the depth estimation of the macropixel; the flatter the curve and the absence of obvious peaks, the more reliable the depth estimation is, indicating that the macropixel is located in a flat region or a region with blurred texture.

[0072] The sharpness parameter mentioned above reflects the sharpness of the peak and can be obtained by calculating the second derivative at the peak or the standard deviation of the fitted Gaussian curve. A larger sharpness parameter value indicates a sharper peak and more reliable depth.

[0073] The aforementioned variability parameters, used to reflect the curve's sensitivity to depth changes, can be obtained by calculating the variance of the curve's first derivative or the full width at half maximum (FWHM) of the peak value. The more drastic the change and the smaller the FWHM, the higher the discriminative power.

[0074] The aforementioned salience parameter reflects the prominence of the peak relative to the surrounding area, and can be obtained by calculating the difference ratio between the peak and the amplitude of the surrounding area. The more prominent the peak, the more defined the depth.

[0075] In some embodiments, the depth reliability weights of effective macropixels of the texture can be calculated based on the following formula:

[0076] In the formula, The depth reliability weights represent the effective macropixels of the texture. This represents the sharpness parameter. Indicates the variability parameter, Indicates the significance parameter.

[0077] Among them, the weighting coefficient It can be determined through experimental or empirical data, aiming to enhance peak sharpness as the main indicator of depth reliability, while integrating variability and saliency features to maintain sensitivity to texture contrast and focus discrimination.

[0078] Step 104: Construct a gradient magnitude-weighted orientation histogram within the spatial region corresponding to the effective macro-pixel of the texture through gradient orientation index analysis, and determine the dominant texture orientation of the effective macro-pixel of the texture based on the peak orientation of the orientation histogram.

[0079] In this embodiment, by constructing a gradient magnitude-weighted orientation histogram, the anisotropic features of the seed surface can be effectively captured. The extracted dominant texture orientation provides key orientation constraints for subsequent suppression of texture orientation collinearity artifacts and realization of EPI orientation adaptive weighting, thereby improving the robustness of depth estimation on anisotropic surfaces.

[0080] In some embodiments, constructing a gradient magnitude-weighted direction histogram within the spatial region corresponding to the effective macro-pixels of the texture through gradient direction exponential analysis includes: The gradient magnitude and gradient direction of each pixel within the spatial region corresponding to the effective macro-pixel of the texture are calculated through gradient direction exponential analysis. The gradient direction is quantized into multiple preset angle ranges; The gradient magnitude of each pixel is accumulated into the angle range corresponding to its gradient direction to obtain the orientation histogram of the effective macropixels of the texture.

[0081] In this embodiment, each pixel within the spatial region corresponding to the effective macro-pixel of the texture is considered. Its horizontal gradient can be calculated using the Sobel operator or other gradient operators. and vertical gradient Then, the gradient magnitude of each pixel can be calculated using the following formula. and gradient direction :

[0082]

[0083] The gradient magnitudes described above reflect the intensity of grayscale changes at that pixel. A larger magnitude indicates that the pixel is located in an edge or textured region, and its contribution to orientation statistics should be greater; a smaller magnitude indicates that the pixel is located in a flat region, and its contribution to orientation statistics should be smaller.

[0084] The gradient direction mentioned above is used to point to the direction of the fastest grayscale change at that pixel.

[0085] It should be noted that the gradient direction is mathematically perpendicular to the local contour lines of equal intensity, meaning that the distribution of gradient directions indirectly reflects the distribution of texture directions. Specifically, the principal direction of the texture differs from the peak direction of the gradient direction by 90°.

[0086] In some embodiments, continuous gradient direction values ​​can be discretized into a finite number of angle intervals to achieve the process of quantizing gradient directions into multiple preset angle intervals.

[0087] For example, the 360° directional range can be divided into 18 uniform angular intervals, each interval corresponding to an angular range of 20°; for the gradient direction of each pixel Map it to the corresponding range index .

[0088] After quantizing the gradient direction, a histogram of directions with initial values ​​of zero is constructed. This histogram has 18 intervals. For each pixel within a spatial region... According to the interval to which its gradient direction belongs The gradient magnitude of the pixel Accumulate to this interval:

[0089] After traversing all pixels within the region, This results in a weighted directional histogram with the horizontal axis representing the directional interval and the vertical axis representing the cumulative gradient magnitude. This weighted directional histogram reflects the distribution of texture directions within a local region; that is, the larger the value on the vertical axis, the higher the gradient intensity in that direction, and the more significant the texture features in that direction.

[0090] After constructing the orientation histogram, the peak value in the histogram can be detected, that is, the orientation interval with the largest cumulative gradient magnitude can be found, and the angle corresponding to the peak interval, usually the middle value of the interval, can be determined as the dominant texture orientation of the effective macro pixels of the texture.

[0091] In this embodiment, by adopting a gradient magnitude weighting strategy, the interference of noise in flat regions on orientation statistics can be effectively suppressed, thereby improving the robustness of depth estimation on anisotropic surfaces.

[0092] Step 105: Based on the depth reliability weight and the dominant texture direction, generate voting results for the epipolar plane images in each direction through soft voting weighting, and normalize the voting results to obtain the weighting coefficients for the epipolar plane images in each direction.

[0093] In this embodiment, weights can be adaptively assigned to EPIs in different directions, prioritizing geometrically reliable viewpoints. This effectively suppresses artifacts caused by collinear textures, thereby improving the robustness of depth estimation on anisotropic surfaces.

[0094] In some embodiments, for each texture valid macro pixel , The total number of valid macropixels in a texture can be determined based on the depth reliability weight of those valid macropixels. and dominant texture direction The effective macropixel pairs of the texture for each standard EPI direction are calculated according to the following formula. Voting intensity :

[0095] in, For the standard EPI direction, the value can be 0°, 45°, 90° or 135°; This is the orientation alignment factor, with a value range of [value range missing]. The value is 1 when the dominant texture direction is perfectly aligned with the EPI direction; and 0 when the dominant texture direction is perpendicular to the EPI direction.

[0096] After calculating the voting intensity of all valid macropixels of the texture, each macropixel is paired with the same direction. The voting strengths are summed to obtain the cumulative voting result in that direction:

[0097] In the formula, Represents all valid macro pixel pairs of textures The sum of votes for a direction, which reflects the overall importance of the EPI in that direction among all macro pixels.

[0098] In some embodiments, the voting results can be normalized so that the sum of the weighting coefficients for all directions is 1:

[0099] In the formula, And satisfy ; This represents the set of standard directions; this set of weighted coefficients constitutes the probability distribution of different standard directions, reflecting the relative importance of EPI in depth estimation for each direction.

[0100] In this embodiment, when the dominant texture direction is parallel to a certain EPI direction, that direction receives a lower voting intensity, thereby effectively suppressing depth artifacts caused by parallax gradient vanishing. The entire process does not require manual weight setting and is entirely data-driven, automatically calculating based on the physical characteristics of each macropixel, i.e., automatically calculating based on depth reliability and texture direction. Moreover, the soft voting mechanism avoids the quantization error caused by hard allocation, allowing the direction weight to transition smoothly between different texture regions. This allows the depth reliability weight to be integrated into the voting process, ensuring that only macropixels with high confidence have a high voting contribution.

[0101] Step 106: Perform depth estimation on the epipolar plane image in each direction to obtain the depth estimation results for each direction.

[0102] In this embodiment, for each direction of the epipolar plane image, linear structures formed by the projection of scene object points can be detected. In EPI, the lines connecting the projection points of the same object point under different viewpoints form a straight line, and the slope of this straight line... Parallax of object point The following relationship exists between them:

[0103] In the formula, For pixel displacement in the spatial direction, This represents the change in viewing angle in the angular direction.

[0104] In some embodiments, the disparity information estimated in the EPI can be converted into a physical depth value by combining calibration parameters, thereby obtaining the depth estimation result corresponding to the EPI in that direction.

[0105] For example, a data-driven hyperbolic calibration model can be used:

[0106] In the formula, For physical depth, and These are the system parameters obtained through calibration experiments. For parallax.

[0107] For each pixel position in EPI According to its parallax value The physical depth value at this location is calculated using the hyperbolic model described above. This generates a depth map corresponding to the EPI in that direction. .

[0108] In this application, the depth estimation results in each direction are independent of each other, reflecting the depth information under different EPI sampling directions. Due to the anisotropic nature of the seed surface texture, the depth estimation quality in different directions may vary significantly. Specifically, the depth estimation quality is poor when the texture direction is parallel to the EPI direction, and high when the texture direction is perpendicular to the EPI direction.

[0109] Step 107: Use the weighting coefficients to perform weighted fusion on the depth estimation results and output the final depth map of the target seed. The final depth map is used to characterize the three-dimensional surface structure of the target seed.

[0110] In some embodiments, the final depth map of the target seed can be calculated based on the following formula:

[0111] In the formula, This represents the final depth map of the target seed, where each pixel value represents the depth information of the corresponding spatial point. Represents the set of standard directions.

[0112] In this embodiment, when the dominant texture direction of a macropixel is perpendicular to a certain EPI direction, the depth estimation quality of the EPI in that direction is poor, and step 105 assigns a lower weighting coefficient to that direction; conversely, when the direction alignment is high, the weighting coefficient is high, and the corresponding depth estimation result dominates in the fusion; moreover, even if the depth estimation fails in a certain direction due to texture collinearity, the depth estimation results in other directions can still make up for it; through the soft voting mechanism, the weight of unreliable directions is automatically suppressed, and the weight of reliable directions is enhanced, thereby achieving adaptive information selection; in addition, since the weighting coefficient is a global value obtained by normalizing the voting results of all valid macropixels, the fusion process remains consistent across the entire image, avoiding stitching marks caused by sudden changes in weight in local areas.

[0113] In some embodiments, the spatial pixel coordinates of the macropixel center on the sensor are set. Furthermore, it can recover three-dimensional spatial coordinates based on the pinhole back projection method:

[0114] In the formula, Let these be the coordinates of the camera's principal point. The effective focal length of the camera. Represents the pixel coordinates on the image plane.

[0115] In this application, by performing multi-directional EPI analysis on the light field data of the target seed, and combining focal plane diversity depth confidence assessment and gradient direction index anisotropy feature decoding, the problem of collinearity of texture directions is effectively suppressed, and the robustness of depth estimation is significantly improved.

[0116] See Figure 2 , Figure 2 This is a schematic diagram of the structure of a depth estimation device based on light field imaging provided in an embodiment of this application. Figure 2 As shown, the device 200 includes: The acquisition module 201 is used to acquire the original light field image of the target seed, decode the original light field image into four-dimensional light field data, and generate an epipolar plane image along multiple directions based on the four-dimensional light field data. The evaluation and screening module 202 is used to evaluate the saliency of multi-feature textures in the spatial region corresponding to each macro-pixel based on the four-dimensional light field data, and screen out the effective macro-pixels of texture. The first analysis module 203 is used to perform focal plane diversity analysis on each effective macro pixel of the texture through digital refocusing and calculate the depth reliability weight of each effective macro pixel of the texture. The second analysis module 204 is used to construct a gradient magnitude-weighted orientation histogram in the spatial region corresponding to the effective macro-pixel of the texture through gradient orientation exponential analysis, and to determine the dominant texture orientation of the effective macro-pixel of the texture based on the peak orientation of the orientation histogram. The first processing module 205 is used to generate voting results of the epipolar plane images in each direction by soft voting weighting based on the depth reliability weight and the dominant texture direction, and to normalize the voting results to obtain the weighting coefficients of the epipolar plane images in each direction. The second processing module 206 is used to perform depth estimation on the epipolar plane image in each direction to obtain the depth estimation results in each direction. The output module 207 is used to perform weighted fusion of the depth estimation results using the weighting coefficients, and output the final depth map of the target seed, which is used to characterize the three-dimensional surface structure of the target seed.

[0117] Optionally, the evaluation and screening module is specifically used for: Based on the four-dimensional light field data, local image patches are extracted from the spatial region corresponding to each macro pixel; Calculate the gradient magnitude, orientation entropy, local intensity variance, and structural tensor coherence of the local image patch; The gradient magnitude, the orientation entropy, the local intensity variance, and the structural tensor coherence are weighted and fused to obtain the texture saliency score for each macro-pixel; Based on the texture saliency score of each macropixel, effective macropixels of texture are selected.

[0118] Optionally, the first analysis module is specifically used to include: For each effective macro pixel of the texture, sub-aperture views of the effective macro pixel of the texture in different observation directions are obtained. The sub-aperture views in different observation directions are aggregated at different depth levels to obtain the focal plane diversity response curve. Sharpness, variability, and significance parameters are extracted from the focal plane diversity response curves. The sharpness parameter, the variability parameter, and the saliency parameter are weighted and fused to obtain the depth reliability weight of each effective macropixel of the texture.

[0119] Optionally, the second analysis module is specifically used for: The gradient magnitude and gradient direction of each pixel within the spatial region corresponding to the effective macro-pixel of the texture are calculated through gradient direction exponential analysis. The gradient direction is quantized into multiple preset angle ranges; The gradient magnitude of each pixel is accumulated into the angle range corresponding to its gradient direction to obtain the orientation histogram of the effective macropixels of the texture.

[0120] The aforementioned device 200 is capable of achieving the functions described in this application. Figure 1 The various processes in the method embodiments, and the ways to achieve the same beneficial effects, will not be repeated here to avoid repetition.

[0121] like Figure 3 As shown, this application also provides a terminal device, including a processor 301 and a memory 302. The memory 302 stores a program or instructions that can run on the processor 301. When the program or instructions are executed by the processor 301, they implement the various steps of the above-described embodiment of the depth estimation method based on light field imaging and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0122] It should be noted that the terminal device in this application can be a terminal or other devices besides a terminal. For example, the terminal device can be a mobile phone, tablet computer, laptop computer, agricultural equipment, etc., and this application does not make any specific limitation.

[0123] This application also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the various processes of the above-described depth estimation method embodiments based on light field imaging and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0124] The processor is the processor in the terminal device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (Read-Only Memory). Only memory (ROM), random access memory (RAM), magnetic disks or optical disks, etc.

[0125] This application also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiments of the depth estimation method based on light field imaging, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0126] It should be understood that the chip mentioned in this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0127] This application provides a computer program product stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-described embodiments of the depth estimation method based on light field imaging, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0128] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as a read-only memory). The device includes a number of instructions in a ROM (random access memory), RAM (magnetic disk), or optical disk to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0130] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A depth estimation method based on light field imaging, characterized in that, include: The original light field image of the target seed is acquired, and the original light field image is decoded into four-dimensional light field data. Based on the four-dimensional light field data, an epipolar plane image along multiple directions is generated. Based on the four-dimensional light field data, a multi-feature texture saliency evaluation is performed on the spatial region corresponding to each macro-pixel to screen out effective texture macro-pixels. By performing focal plane diversity analysis on each effective macropixel of the texture through digital refocusing, the depth reliability weight of each effective macropixel of the texture is calculated. By using gradient direction exponential analysis, a gradient magnitude-weighted direction histogram is constructed within the spatial region corresponding to the effective macro-pixel of the texture, and the dominant texture direction of the effective macro-pixel is determined based on the peak direction of the direction histogram. Based on the depth reliability weight and the dominant texture direction, the voting results of the epipolar plane images in each direction are generated by soft voting weighting. The voting results are then normalized to obtain the weighting coefficients of the epipolar plane images in each direction. Depth estimation is performed on the epipolar plane image in each direction to obtain the depth estimation results for each direction; The depth estimation results are weighted and fused using the weighting coefficients to output the final depth map of the target seed, which is used to characterize the three-dimensional surface structure of the target seed.

2. The method according to claim 1, characterized in that, Based on the four-dimensional light field data, a multi-feature texture saliency evaluation is performed on the spatial region corresponding to each macro-pixel to filter out effective texture macro-pixels, including: Based on the four-dimensional light field data, local image patches are extracted from the spatial region corresponding to each macro pixel; Calculate the gradient magnitude, orientation entropy, local intensity variance, and structural tensor coherence of the local image patch; The gradient magnitude, the orientation entropy, the local intensity variance, and the structural tensor coherence are weighted and fused to obtain the texture saliency score for each macro-pixel; Based on the texture saliency score of each macropixel, effective macropixels of texture are selected.

3. The method according to claim 2, characterized in that, The step of performing focal plane diversity analysis on each effective macro-pixel of the texture through digital refocusing to calculate the depth reliability weight of each effective macro-pixel of the texture includes: For each effective macro pixel of the texture, sub-aperture views of the effective macro pixel of the texture in different observation directions are obtained. The sub-aperture views in different observation directions are aggregated at different depth levels to obtain the focal plane diversity response curve. Sharpness, variability, and significance parameters are extracted from the focal plane diversity response curves. The sharpness parameter, the variability parameter, and the saliency parameter are weighted and fused to obtain the depth reliability weight of each effective macropixel of the texture.

4. The method according to claim 3, characterized in that, The step of constructing a gradient magnitude-weighted direction histogram within the spatial region corresponding to the effective macro-pixels of the texture through gradient direction exponential analysis includes: The gradient magnitude and gradient direction of each pixel within the spatial region corresponding to the effective macro-pixel of the texture are calculated through gradient direction exponential analysis. The gradient direction is quantized into multiple preset angle ranges; The gradient magnitude of each pixel is accumulated into the angle range corresponding to its gradient direction to obtain the orientation histogram of the effective macropixels of the texture.

5. A depth estimation device based on light field imaging, characterized in that, include: The acquisition module is used to acquire the original light field image of the target seed, decode the original light field image into four-dimensional light field data, and generate an epipolar plane image along multiple directions based on the four-dimensional light field data. The evaluation and screening module is used to evaluate the saliency of multi-feature textures in the spatial region corresponding to each macro-pixel based on the four-dimensional light field data, and to screen out the effective macro-pixels of texture. The first analysis module is used to perform focal plane diversity analysis on each effective macro pixel of the texture through digital refocusing and calculate the depth reliability weight of each effective macro pixel of the texture. The second analysis module is used to construct a gradient magnitude-weighted orientation histogram in the spatial region corresponding to the effective macro-pixel of the texture through gradient orientation exponential analysis, and to determine the dominant texture orientation of the effective macro-pixel of the texture based on the peak orientation of the orientation histogram. The first processing module is used to generate voting results for the epipolar plane images of each direction by soft voting weighting based on the depth reliability weight and the dominant texture direction, and to normalize the voting results to obtain the weighting coefficients of the epipolar plane images of each direction. The second processing module is used to perform depth estimation on the epipolar plane image in each direction to obtain the depth estimation results in each direction. The output module is used to perform weighted fusion of the depth estimation results using the weighting coefficients, and output the final depth map of the target seed, which is used to characterize the three-dimensional surface structure of the target seed.

6. The apparatus according to claim 5, characterized in that, The evaluation and screening module is specifically used for: Based on the four-dimensional light field data, local image patches are extracted from the spatial region corresponding to each macro pixel; Calculate the gradient magnitude, orientation entropy, local intensity variance, and structural tensor coherence of the local image patch; The gradient magnitude, the orientation entropy, the local intensity variance, and the structural tensor coherence are weighted and fused to obtain the texture saliency score for each macro-pixel; Based on the texture saliency score of each macropixel, effective macropixels of texture are selected.

7. The apparatus according to claim 6, characterized in that, The first analysis module is specifically used for: For each effective macro pixel of the texture, sub-aperture views of the effective macro pixel of the texture in different observation directions are obtained. The sub-aperture views in different observation directions are aggregated at different depth levels to obtain the focal plane diversity response curve. Sharpness, variability, and significance parameters are extracted from the focal plane diversity response curves. The sharpness parameter, the variability parameter, and the saliency parameter are weighted and fused to obtain the depth reliability weight of each effective macropixel of the texture.

8. The apparatus according to claim 7, characterized in that, The second analysis module is specifically used for: The gradient magnitude and gradient direction of each pixel within the spatial region corresponding to the effective macro-pixel of the texture are calculated through gradient direction exponential analysis. The gradient direction is quantized into multiple preset angle ranges; The gradient magnitude of each pixel is accumulated into the angle range corresponding to its gradient direction to obtain the orientation histogram of the effective macropixels of the texture.

9. A terminal device, characterized in that, It includes a processor and a memory, wherein the memory stores a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in any one of claims 1 to 4.

Images