Light field depth estimation method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of image processing, and relates to a light field depth estimation method and device, computer equipment and a storage medium, and the method comprises the steps: extracting an epipolar plane image of an original light field image; respectively calculating a line consistency cost body and a region consistency cost body of each epipolar plane image; performing cost body fusion operation and weighted fusion operation on the line consistency cost body and the region consistency cost body at the same preset angle to obtain a weighted fusion cost body; performing multi-view fusion operation on all the weighted fusion cost bodies to obtain a multi-view fusion cost body; and performing depth estimation operation on the multi-view fusion cost body according to a winner communication algorithm to obtain a target depth value. According to the method, the imaging characteristics of the polar plane images at different visual angles and different angles and different consistency characteristics are effectively fused by using the fusion algorithm, so that the algorithm can better solve the problems of shielding and noise, and meanwhile, the overall robustness and accuracy of the algorithm are also improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a light field depth estimation method, device, computer equipment and storage medium. Background Art

[0002] Depth estimation is the core issue of 3D reconstruction technology. Common depth estimation methods mainly include monocular depth estimation, binocular depth estimation and multi-camera depth estimation. The light field depth estimation method is the inheritance and development of the traditional image depth estimation method.

[0003] In recent years, light field cameras have made great progress. They can now be made in a similar size to traditional industrial cameras and are increasingly used. Compared with traditional cameras, light field cameras not only capture the cumulative intensity of light in all directions at a spatial point, but also record the intensity of light in all directions reaching a certain spatial point. Therefore, information from multiple perspectives can be obtained through a single shot. Using this multi-perspective information, digital refocusing, new perspective synthesis, image depth estimation, etc. can be performed, providing a new and reliable data source for 3D reconstruction.

[0004] However, depth estimation technology based on light field images still faces a series of challenges. Current light field cameras are built by integrating microlens arrays on the basis of traditional cameras, which may introduce noise, reduce angular resolution, produce artifacts, aliasing, and camera optical distortion in actual use. In addition, in terms of depth estimation, it is also necessary to face the challenges of traditional problems such as occlusion and weak texture. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a light field depth estimation method, apparatus, computer equipment and storage medium to solve the problems of current light field cameras introducing noise, reducing angular resolution, generating artifacts, aliasing and camera optical distortion.

[0006] In order to solve the above technical problems, the present application provides a method for light field depth estimation, which adopts the following technical solutions:

[0007] Receiving a depth estimation request carrying an original light field image sent by a user terminal;

[0008] Extracting an epipolar plane image of the original light field image according to a plurality of preset angles;

[0009] respectively calculating the line consistency cost volume of each of the epipolar plane images;

[0010] Calculating the regional consistency cost volume of each epipolar plane image respectively;

[0011] Performing cost volume fusion operations on the line consistency cost volume and the region consistency cost volume of the same preset angle respectively to obtain a fused cost volume;

[0012] Perform weighted fusion operation on the fusion cost volume of each preset angle respectively to obtain a weighted fusion cost volume;

[0013] Performing a multi-view fusion operation on all the weighted fusion cost volumes to obtain a multi-view fusion cost volume;

[0014] A depth estimation operation is performed on the multi-view fusion cost volume according to a winner-takes-all algorithm to obtain a target depth value.

[0015] Further, after the step of receiving the depth estimation request carrying the original light field image sent by the user terminal and before the step of extracting the epipolar plane image of the original light field image according to a plurality of preset angles, the following step is also included:

[0016] If the original light field image is a photographed image, performing a triangular weight interpolation operation on the original light field image to obtain a standard light field image;

[0017] If the original light field image is a synthetic image, preprocessing the original light field image to obtain a standard light field image;

[0018] The step of extracting the epipolar plane image of the original light field image according to a plurality of preset angles specifically comprises the following steps:

[0019] The epipolar plane image of the standard light field image is extracted according to a plurality of preset angles.

[0020] Furthermore, the line consistency cost line (s, t, d) is expressed as:

[0021]

[0022] in, u represents the width of the angular resolution, represents pixel point, P′ d Represents the neighborhood pixels on the center line.

[0023] Furthermore, the regional consistency cost area(s,t,d) It is expressed as:

[0024]

[0025] Among them, p left Indicates that the pixel point where each central viewpoint is located corresponds to the pixel set to the left of the parallax diagonal line in the window area at the angular resolution, p rightIndicates that the pixel point where each central viewpoint is located corresponds to the pixel set to the right of the parallax diagonal line in the window area at the angular resolution.

[0026] Furthermore, the step of performing a cost volume fusion operation on the line consistency cost volume and the region consistency cost volume of the same preset angle to obtain a fused cost volume specifically includes the following steps:

[0027] Calculate the PKR peak ratio of the line consistency cost volume and the regional consistency cost volume at the same preset angle and the same viewing angle respectively;

[0028] Calculate the mean maximum confidence of the line consistency cost volume and the region consistency cost volume at the same preset angle and the same viewing angle respectively;

[0029] A cost volume fusion operation is performed on each PKR peak ratio and mean maximum confidence at the same preset angle and the same viewing angle to obtain the fused cost volume.

[0030] Furthermore, the fusion cost Cost merge (s, t, d) is expressed as:

[0031] Cost merge (s,t,d)

[0032] =NW linePKR (s,t)×NW lineconf (s,t)×Ncost line(s,t,d)

[0033] +NW areaPKR (s,t)×NW areaconf (s,t)×Ncost area(s,t,d)

[0034] Among them, Cost merge (s, t) represents the fused cost volume, NW linePKR (s,t),NW lineconf (s,t),NW areaPKR (s,t),NW areaconf (s, t) represent the normalized PKR weight and mean maximum weight of the line and region, respectively, Ncost line(s,t,d) and Ncost area(s,t,d) They represent the normalized line consistency cost volume and region consistency cost volume respectively.

[0035] Furthermore, the weighted fusion cost Cost mmerge (s, t, d) is expressed as:

[0036]

[0037] Where n represents the number of multi-view angles, represents the cost volume of the fusion line and region cost at the i-th perspective, Represents the PKR peak ratio weight calculated by the fused cost volume at the i-th perspective.

[0038] In order to solve the above technical problems, the embodiment of the present application further provides a light field depth estimation device, which adopts the following technical solution:

[0039] A depth estimation request module is used to receive a depth estimation request carrying an original light field map sent by a user terminal;

[0040] An image extraction module, used for extracting an epipolar plane image of the original light field image according to a plurality of preset angles;

[0041] A line consistency cost volume calculation module, used to calculate the line consistency cost volume of each of the epipolar plane images respectively;

[0042] A regional consistency cost volume calculation module, used to calculate the regional consistency cost volume of each epipolar plane image respectively;

[0043] A cost volume fusion module is used to perform cost volume fusion operations on the line consistency cost volume and the region consistency cost volume of the same preset angle to obtain a fused cost volume;

[0044] A weighted fusion module is used to perform a weighted fusion operation on the fusion cost volume of each preset angle to obtain a weighted fusion cost volume;

[0045] A multi-view fusion module, used for performing a multi-view fusion operation on all the weighted fusion cost volumes to obtain a multi-view fusion cost volume;

[0046] The depth estimation module is used to perform a depth estimation operation on the multi-view fusion cost volume according to a winner-takes-all algorithm to obtain a target depth value.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0048] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the light field depth estimation method as described above when executing the computer-readable instructions.

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0050] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the light field depth estimation method described above are implemented.

[0051] The present application provides a light field depth estimation method, comprising: receiving a depth estimation request carrying an original light field image sent by a user terminal; extracting an epipolar plane image of the original light field image according to a number of preset angles; respectively calculating a line consistency cost volume of each of the epipolar plane images; respectively calculating a region consistency cost volume of each of the epipolar plane images; respectively performing a cost volume fusion operation on the line consistency cost volume and the region consistency cost volume of the same preset angle to obtain a fused cost volume; respectively performing a weighted fusion operation on the fused cost volume of each preset angle to obtain a weighted fused cost volume; performing a multi-perspective fusion operation on all the weighted fused cost volumes to obtain a multi-perspective fused cost volume; performing a depth estimation operation on the multi-perspective fused cost volume according to a winner-takes-all algorithm to obtain a target depth value. Compared with the prior art, the present application introduces multi-perspective EPI (epipolar plane image) images and the multi-perspective characteristics of the light field to further improve the accuracy and robustness of the algorithm in occlusion. In addition, the present application uses an improved JS divergence and line consistency algorithm to better fuse the two cost volumes, and the fused algorithm is more accurate and robust in noise and texture-free areas. The improved PKR peak confidence weight is combined with the mean maximum confidence to fuse cost volumes of different perspectives, directions, and consistencies, which has better effects in occlusion, noise, and weak texture. It has good depth estimation effects in both synthetic scenes and scenes shot by real light field cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0054] Figure 2 is a flowchart of the implementation of the light field depth estimation method provided in Example 1 of the present application;

[0055] Figure 3 is a schematic diagram of the structure of a light field depth estimation device provided in Embodiment 2 of the present application;

[0056] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0058] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0060] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0061] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0062] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0063] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0064] It should be noted that the light field depth estimation method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the light field depth estimation device is generally arranged in the server / terminal device.

[0065] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0066] Embodiment 1

[0067] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for estimating light field depth according to the present application. The method for estimating light field depth includes: step S201, step S202, step S203, step S204, step S205, step S206, step S207 and step S208.

[0068] In step S201, a depth estimation request carrying an original light field map sent by a user terminal is received.

[0069] In the embodiments of the present application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided by the present application. The user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for the convenience of understanding and are not used to limit the present application.

[0070] In an embodiment of the present application, the standard light field image of the nth scene is recorded as, wherein t and s respectively represent the height and width of images of different viewing angles in the spatial dimension, u and v respectively represent the number of viewing angles in the horizontal and vertical directions, in the microlens array light field camera, s*t is the total number of microlenses, u*v is the angular resolution under each microlens, and is also the angular domain under each microlens, and N represents the total number of light field data.

[0071] In step S202, epipolar plane images of the original light field image are extracted according to a number of preset angles.

[0072] In the embodiment of the present application, at a suitable angular resolution, four straight lines can be drawn with different viewpoints as the central subpoint p, and the angles between the four straight lines and the horizontal coordinate x-axis are α, α∈{0°, 45°, 90°, 135°}. For each standard light field image, the pixels intersected by the straight lines at different angles are stacked in a certain dimension to generate EPI images in different directions at different viewing angles. k 、EPI45 k 、EPI90 k 、EPI135 k Respectively represent the EPI images of the four angle directions centered at k viewpoints, where EPI0 k 、EPI45 k The size is (t×v, s), EPI90 k 、EPI135 k The size of is (t×v,s).

[0073] In step S203 , the line consistency cost volume of each epipolar plane image is calculated respectively.

[0074] In the embodiment of the present application, the calculation of the line consistency cost volume can be to calculate the average sum of the absolute differences between the grayscale values ​​of the multiple pixels on the center line of the disparity slope and the center pixel itself in the window area at the angular resolution corresponding to the pixel point where each central viewpoint mid is located under each depth label d in the extracted EPI image, and obtain the cost volume cost calculated from the middle line of the quadrilateral of different EPI images line (s,t,d).

[0075] In step S204, the regional consistency cost volume of each epipolar plane image is calculated respectively.

[0076] In the embodiment of the present application, similar to the line consistency calculation form, the pixel points of each central viewpoint mid under each depth label d in the extracted EPI image are calculated respectively. The cost of regional consistency is obtained by the JS divergence variant between the pixel set ppleft on the left side of the parallax diagonal line and the pixel set pright on the right side of the window area at the angular resolution. area(s,t,d) .

[0077] In step S205 , a cost volume fusion operation is performed on the line consistency cost volume and the region consistency cost volume of the same preset angle to obtain a fused cost volume.

[0078] In step S206, a weighted fusion operation is performed on the fusion cost volume of each preset angle to obtain a weighted fusion cost volume.

[0079] In step S207, a multi-view fusion operation is performed on all weighted fusion cost volumes to obtain a multi-view fusion cost volume.

[0080] In step S208, a depth estimation operation is performed on the multi-view fusion cost volume according to a winner-takes-all algorithm to obtain a target depth value.

[0081] In an embodiment of the present application, a light field depth estimation method is provided, including: receiving a depth estimation request carrying an original light field image sent by a user terminal; extracting an epipolar plane image of the original light field image according to a number of preset angles; calculating a line consistency cost volume of each epipolar plane image respectively; calculating a region consistency cost volume of each epipolar plane image respectively; performing a cost volume fusion operation on the line consistency cost volume and the region consistency cost volume of the same preset angle respectively to obtain a fused cost volume; performing a weighted fusion operation on the fused cost volume of each preset angle respectively to obtain a weighted fused cost volume; performing a multi-perspective fusion operation on all weighted fused cost volumes to obtain a multi-perspective fused cost volume; performing a depth estimation operation on the multi-perspective fused cost volume according to a winner-takes-all algorithm to obtain a target depth value. Compared with the prior art, the present application introduces multi-perspective EPI (epipolar plane image) images and the multi-perspective characteristics of the light field to further improve the accuracy and robustness of the algorithm in occlusion. In addition, the present application uses an improved JS divergence and line consistency algorithm to better fuse the two cost volumes, and the fused algorithm is more accurate and robust in noise and texture-free areas. The improved PKR peak confidence weight is combined with the mean maximum confidence to fuse cost volumes of different perspectives, directions, and consistencies, which has better effects in occlusion, noise, and weak texture. It has good depth estimation effects in both synthetic scenes and scenes shot by real light field cameras.

[0082] In some optional implementations of the embodiments of the present application, after step S201 and before step S202, the following steps are further included:

[0083] If the original light field image is a photographed image, a triangular weight interpolation operation is performed on the original light field image to obtain a standard light field image;

[0084] If the original light field image is a synthetic image, a preprocessing operation is performed on the original light field image to obtain a standard light field image;

[0085] The above step S202 includes the following steps:

[0086] The epipolar plane images of the standard light field map are extracted according to several preset angles.

[0087] In an embodiment of the present application, if the input is a real light field image taken by a light field camera, a standard light field image is obtained by triangular weight interpolation of uniformly sampled points; if the input is a synthetic image, a standard light field image is obtained by preprocessing.

[0088] In some optional implementations of the embodiments of the present application, the above line consistency costline (s, t, d) is expressed as:

[0089]

[0090] in, u represents the width of the angular resolution, represents pixel point, P′ d Represents the neighborhood pixels on the center line.

[0091] In this application, the line consistency cost is calculated line In the embodiment of (s, t, d), each pixel point used for calculation and the neighboring pixel point P′ on the center line d are obtained by Gaussian weighting of the pixel itself and its n neighboring points in the same row. and P′ d The extractions in different EPI maps are represented as follows:

[0092]

[0093] Among them, k∈[1,nD] represents the unit depth label, σ is the Gaussian coefficient, and i represents the point index adjacent to the center point.

[0094] Finally, the calculation formula of the line consistency costline(s,t,d) is as follows:

[0095]

[0096] Here, u represents the width of the angular resolution.

[0097] In some optional implementations of the embodiments of the present application, the above-mentioned regional consistency cost body costarea(s,t,d) It is expressed as:

[0098]

[0099] Among them, p left Indicates that the pixel point where each central viewpoint is located corresponds to the pixel set to the left of the parallax diagonal line in the window area at the angular resolution, p right Indicates that the pixel point where each central viewpoint is located corresponds to the pixel set to the right of the parallax diagonal line in the window area at the angular resolution.

[0100] In some optional implementations of the embodiments of the present application, the above step S206 specifically includes the following steps:

[0101] Calculate the PKR peak ratio of the line consistency cost volume and the regional consistency cost volume at the same preset angle and the same viewing angle respectively;

[0102] Calculate the mean maximum confidence of the line consistency cost volume and the region consistency cost volume at the same preset angle and the same viewing angle respectively;

[0103] The cost volume fusion operation is performed on the PKR peak ratio and the mean maximum confidence at each preset angle and the same viewing angle to obtain a fused cost volume.

[0104] In the embodiment of the present application, the PKR peak ratios of the regional cost volume and the line cost volume are calculated respectively, wherein the weight calculation formula of the peak ratio is as follows:

[0105]

[0106] Among them, max1 and max2 represent the maximum cost value and the second largest local maximum cost value in the cost volume corresponding to the pixel point (s, t), respectively, and σ is the Gaussian standard deviation. The greater the difference between the first largest cost value and the second largest local maximum cost value, the greater the peak ratio weight.

[0107] In the embodiment of the present application, the mean maximum confidence of the region cost volume and the line cost volume are calculated respectively, and the calculation method is as follows:

[0108]

[0109] Among them, mean(cost) represents the mean cost of the corresponding cost body, max(cost) represents the maximum cost value of the corresponding cost body, and cost refers to the regional consistency cost body cost area(s,t,d) Or the line consistency costline(s,t,d).

[0110] In some optional implementations of the embodiments of the present application, the above fusion cost Cost merge (s, t, d) is expressed as:

[0111] Cost merge (s,t,d)

[0112] =NW linrPKR (s,t)×NW lineconf (s,t)×Ncost line(s,t,d)

[0113] +NW areaPKR (s,t)×NW areaconf (s,t)×Ncost area(s,t,d)

[0114] Among them, Cost merge (s, t) represents the fused cost volume, NW linePKR (s,t),NW lineconf (s,t),NW areaPKR (s,t),NW areaconf (s, t) represent the normalized PKR weight and mean maximum weight of the line and region, respectively, Ncost line(s,t,d) and Ncost area(s,t,d) They represent the normalized line consistency cost volume and region consistency cost volume respectively.

[0115] In some optional implementations of the embodiments of the present application, the weighted fusion cost Cost mmerge (s, t, d) is expressed as:

[0116]

[0117] Where n represents the number of multi-view angles, represents the cost volume of the fusion line and region cost under the i-th perspective, Represents the PKR peak ratio weight calculated by the fused cost volume at the i-th perspective.

[0118] In some optional implementations of the embodiments of the present application, the multi-view fusion cost volume calculated from four angles: Cost0 mmerge (s,t,d),Cost45 mmerge (s,t,d),Cost90 mmerge (s,t,d),Cost135 mmerge (s, t, d) are fused and the calculated PKR weight is used to obtain the multi-view fusion cost body Cost 4mmerge (s,t,d), expressed as:

[0119]

[0120] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0121] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0122] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0123] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0124] Embodiment 2

[0125] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a light field depth estimation device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0126] like Figure 3 As shown, the light field depth estimation device 200 of the embodiment of the present application includes:

[0127] A depth estimation request module 210 is configured to receive a depth estimation request carrying an original light field map sent by a user terminal;

[0128] An image extraction module 220, for extracting an epipolar plane image of the original light field image according to a number of preset angles;

[0129] A line consistency cost volume calculation module 230, used to calculate the line consistency cost volume of each epipolar plane image respectively;

[0130] A regional consistency cost volume calculation module 240, used to calculate the regional consistency cost volume of each epipolar plane image respectively;

[0131] The cost volume fusion module 250 is used to perform cost volume fusion operations on the line consistency cost volume and the region consistency cost volume of the same preset angle to obtain a fused cost volume;

[0132] The weighted fusion module 260 is used to perform a weighted fusion operation on the fusion cost volume of each preset angle to obtain a weighted fusion cost volume;

[0133] A multi-view fusion module 270 is used to perform a multi-view fusion operation on all weighted fusion cost volumes to obtain a multi-view fusion cost volume;

[0134] The depth estimation module 280 is used to perform a depth estimation operation on the multi-view fusion cost volume according to the winner-takes-all algorithm to obtain a target depth value.

[0135] In an embodiment of the present application, a light field depth estimation device 200 is provided, including: a depth estimation request module 210, which is used to receive a depth estimation request carrying an original light field image sent by a user terminal; an image extraction module 220, which is used to extract an epipolar plane image of the original light field image according to a number of preset angles; a line consistency cost volume calculation module 230, which is used to respectively calculate the line consistency cost volume of each epipolar plane image; a region consistency cost volume calculation module 240, which is used to respectively calculate the region consistency cost volume of each epipolar plane image; a cost volume fusion module 250, which is used to perform a cost volume fusion operation on the line consistency cost volume and the region consistency cost volume of the same preset angle respectively to obtain a fused cost volume; a weighted fusion module 260, which is used to perform a weighted fusion operation on the fused cost volume of each preset angle respectively to obtain a weighted fused cost volume; a multi-view fusion module 270, which is used to perform a multi-view fusion operation on all weighted fused cost volumes to obtain a multi-view fused cost volume; a depth estimation module 280, which is used to perform a depth estimation operation on the multi-view fused cost volume according to a winner-takes-all algorithm to obtain a target depth value. Compared with the prior art, the present application introduces multi-perspective EPI (epipolar plane image) images and the multi-perspective characteristics of the light field to further improve the accuracy and robustness of the algorithm in occlusion. In addition, the present application uses an improved JS divergence and line consistency algorithm to better fuse the two cost volumes, and the fused algorithm is more accurate and robust in noise and texture-free areas. The improved PKR peak confidence weight is combined with the mean maximum confidence to fuse cost volumes of different perspectives, directions, and consistencies, which has better effects in occlusion, noise, and weak texture. It has good depth estimation effects in both synthetic scenes and scenes shot by real light field cameras.

[0136] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0137] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0138] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0139] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store an operating system and various application software installed on the computer device 300, such as computer-readable instructions of a light field depth estimation method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0140] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as computer-readable instructions for running the light field depth estimation method.

[0141] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0142] The computer device provided by the present application further improves the accuracy and robustness of the algorithm in occlusion by introducing multi-perspective EPI images and the multi-perspective characteristics of the light field; in addition, the present application uses an improved JS divergence and line consistency algorithm to better fuse the two cost volumes, and the fused algorithm has higher accuracy and is more robust in noise and texture-free areas; the improved PKR peak confidence weight is combined with the mean maximum confidence to fuse cost volumes of different perspectives, different directions, and different consistencies, which has good effects in occlusion, noise, and weak textures; and good depth estimation effects are achieved in both synthetic scenes and scenes shot by real light field cameras.

[0143] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the light field depth estimation method as described above.

[0144] The computer-readable storage medium provided by the present application further improves the accuracy and robustness of the algorithm in occlusion by introducing multi-perspective EPI images and the multi-perspective characteristics of the light field; in addition, the present application uses an improved JS divergence and line consistency algorithm to better fuse the two cost bodies, and the fused algorithm has higher accuracy and is more robust in noise and texture-free areas; the improved PKR peak confidence weight is combined with the mean maximum confidence to fuse cost bodies of different perspectives, different directions, and different consistencies, which has good effects in occlusion, noise, and weak textures; and good depth estimation effects are achieved in both synthetic scenes and scenes shot by real light field cameras.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0146] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A light field depth estimation method, characterized in that: The steps include: Receiving a depth estimation request carrying an original light field image sent by a user terminal; Extracting an epipolar plane image of the original light field image according to a plurality of preset angles; respectively calculating the line consistency cost volume of each of the epipolar plane images; Calculating the regional consistency cost volume of each epipolar plane image respectively; Performing cost volume fusion operations on the line consistency cost volume and the region consistency cost volume of the same preset angle respectively to obtain a fused cost volume; Perform weighted fusion operation on the fusion cost volume of each preset angle respectively to obtain a weighted fusion cost volume; Performing a multi-view fusion operation on all the weighted fusion cost volumes to obtain a multi-view fusion cost volume; A depth estimation operation is performed on the multi-view fusion cost volume according to a winner-takes-all algorithm to obtain a target depth value.

2. The light field depth estimation method according to claim 1, characterized in that: After the step of receiving a depth estimation request carrying an original light field image sent by a user terminal and before the step of extracting an epipolar plane image of the original light field image according to a plurality of preset angles, the method further includes the following steps: If the original light field image is a photographed image, performing a triangular weight interpolation operation on the original light field image to obtain a standard light field image; If the original light field image is a synthetic image, preprocessing the original light field image to obtain a standard light field image; The step of extracting the epipolar plane image of the original light field image according to a plurality of preset angles specifically comprises the following steps: The epipolar plane image of the standard light field image is extracted according to a plurality of preset angles.

3. The light field depth estimation method according to claim 1, characterized in that: The line consistency cost line (s, t, d) is expressed as: in, u represents the width of the angular resolution, represents pixel point, P′ d Represents the neighborhood pixels on the center line.

4. The light field depth estimation method according to claim 1, characterized in that: The regional consistency cost area(s,t,d) It is expressed as: Among them, p left Indicates that the pixel point where each central viewpoint is located corresponds to the pixel set to the left of the parallax diagonal line in the window area at the angular resolution, p right Indicates that the pixel point where each central viewpoint is located corresponds to the pixel set to the right of the parallax diagonal line in the window area at the angular resolution.

5. The light field depth estimation method according to claim 1, characterized in that: The step of performing a cost volume fusion operation on the line consistency cost volume and the region consistency cost volume of the same preset angle to obtain a fused cost volume specifically includes the following steps: Calculate the PKR peak ratio of the line consistency cost volume and the regional consistency cost volume at the same preset angle and the same viewing angle respectively; Calculate the mean maximum confidence of the line consistency cost volume and the region consistency cost volume at the same preset angle and the same viewing angle respectively; A cost volume fusion operation is performed on each PKR peak ratio and mean maximum confidence at the same preset angle and the same viewing angle to obtain the fused cost volume.

6. The method for light field depth estimation according to claim 5, characterized in that: The fusion cost Cost merge (s, t, d) is expressed as: Cost merge (s,t,d) =NW linePKR (s,t)×NW lineconf (s,t)×Ncost line(s,t,d) +NW areaPKR (s,t)×NW areaconf (s,t)×Ncost area(s,t,d) Among them, Cost merge (s, t) represents the fused cost volume, NW linePKR (s,t),NW lineconf (s,t),NW areaPKR (s,t),NW areaconf (s, t) represent the normalized PKR weight and mean maximum weight of the line and region, respectively, Ncost line(s,t,d) and Ncost area(s,t,d) They represent the normalized line consistency cost volume and region consistency cost volume respectively.

7. The light field depth estimation method according to claim 1, characterized in that: The weighted fusion cost Cost mmerge (s, t, d) is expressed as: Where n represents the number of multi-view angles, represents the cost volume of the fusion line and region cost at the i-th perspective, Represents the PKR peak ratio weight calculated by the fused cost volume at the i-th perspective.

8. A light field depth estimation device, characterized in that: include: A depth estimation request module is used to receive a depth estimation request carrying an original light field map sent by a user terminal; An image extraction module, used for extracting an epipolar plane image of the original light field image according to a plurality of preset angles; A line consistency cost volume calculation module, used to calculate the line consistency cost volume of each of the epipolar plane images respectively; A regional consistency cost volume calculation module, used to calculate the regional consistency cost volume of each epipolar plane image respectively; A cost volume fusion module is used to perform cost volume fusion operations on the line consistency cost volume and the region consistency cost volume of the same preset angle to obtain a fused cost volume; A weighted fusion module is used to perform a weighted fusion operation on the fusion cost volume of each preset angle to obtain a weighted fusion cost volume; A multi-view fusion module, used for performing a multi-view fusion operation on all the weighted fusion cost volumes to obtain a multi-view fusion cost volume; The depth estimation module is used to perform a depth estimation operation on the multi-view fusion cost volume according to a winner-takes-all algorithm to obtain a target depth value.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the light field depth estimation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the light field depth estimation method according to any one of claims 1 to 7 are implemented.

Images