Sparse light field structure, light field camera and sparse light field image resolution improving method

Abstract

The invention relates to the technical field of light field cameras, in particular to a sparse light field structure, a light field camera and a sparse light field image resolution improving method, which are used for improving the resolution of an image shot by the light field camera. According to the main scheme, the sparse light field structure is applied to a light field camera and is arranged between a main lens and a sensor in the light field camera; the sparse light field structure comprises a lens layer and a phase delay layer which are attached together or separated from each other; the lens layer is provided with a plurality of spaced micro-lens units to form a sparse micro-lens array.

Description

Technical Field

[0001] The present application relates to the technical field of light field cameras, and in particular to a sparse light field structure, a light field camera, and a method for improving the resolution of sparse light field images. Background Art

[0002] With the continuous advancement of computational optics, the performance of light field cameras has been significantly improved. By using a special microlens array structure, the position and direction of light can be recorded simultaneously, expanding the traditional two-dimensional image to four dimensions. The early light field structure was a non-focused light field, such as Figure 1a As shown, each tiny microlens unit is densely packed in a certain way to form a microlens array; the microlens array is placed on the image plane of the main lens; and the sensor is placed on the focal plane of the microlens array. The biggest problem with this light field recording method is the severe loss of resolution. To address this low resolution issue, researchers are focusing on two directions: hardware improvements and subsequent algorithm refinements.

[0003] In terms of hardware changes, a focusing light field structure was proposed, such as Figure 1b As shown, each tiny microlens unit is densely laid in a certain way to form a microlens array; the microlens array performs secondary imaging on the primary imaging of the main lens; the sensor is placed on the image plane of the microlens array, and the focusing light field regards the microlens array as a secondary imaging unit. The sensor is placed on the image plane of the microlens, and higher resolution is achieved by adjusting the distance between the microlens array and the sensor, but this solution requires strict and complex calibration; the solution of improving resolution by scanning the light field requires an additional integrated scanning platform, and the accuracy and stability of the scanning platform will also affect the effect of resolution recovery; the solution of collecting mixed data by multiple cameras can also improve resolution, but it requires additional camera equipment, and also requires calibration processing of multiple cameras, and has not yet met the requirements of existing application scenarios for image data acquisition resolution.

[0004] On the other hand, with the development of deep learning technology, neural network-based super-resolution technology has been applied to the field of light field resolution restoration, such as using a residual network to combine the features of the input view and adjacent views to super-resolve the central view of the light field to improve the resolution of the light field image, but there is a problem of distortion of the reconstruction results.

[0005] In existing technologies, hardware iterative optimization often requires high manpower and material costs, and software algorithm optimization solutions are based on early basic light field models. The resolution restoration effect and authenticity have not yet fully met the requirements. Summary of the Invention

[0006] In view of this, the present application provides a sparse light field structure, a light field camera, and a sparse light field image resolution enhancement method, which are used to improve the resolution of images captured by a light field camera.

[0007] In a first aspect, an embodiment of the present application provides a sparse light field structure, which is applied to a light field camera and is disposed between a main lens and a sensor in the light field camera; The sparse light field structure includes: a lens layer and a phase delay layer that are bonded together or separated; the lens layer is provided with a plurality of spaced microlens units to form a sparse microlens array.

[0008] In an optional embodiment provided by the present invention, the microlens units and phase delay units are arranged according to different combinations of two-dimensional coordinates to form a sparse microlens array; the microlens units include one or more morphological specifications, and the morphological specifications are the number of pixels covered by the microlens units and the shape of the cross-section of the microlens units.

[0009] In an optional embodiment provided by the present invention, the shape of the cross section of the microlens unit includes but is not limited to circle, square, and rectangle.

[0010] In an optional embodiment provided by the present invention, the microlens units and phase delay units are arranged in a regular non-orthogonal, regular orthogonal, irregular orthogonal or irregular non-orthogonal manner using one form specification or multiple form specifications to form a sparse microlens array.

[0011] In an optional embodiment provided by the present invention, when the image plane of the main lens is placed on the lens layer of the sparse light field structure, the sparse light field image data collected by the sparse light field image data is unfocused sparse light field image data; when the image plane of the main lens is placed between the sparse light field structure and the sensor, the sparse light field image data collected by the sparse light field image data is focused sparse light field image data.

[0012] In a second aspect, an embodiment of the present application further provides a light field camera, comprising: the sparse light field structure provided in the first aspect, a main lens and a sensor, and a processor, wherein the processor is configured to convert the sparse light field image data collected by the sparse light field structure into a high-resolution image.

[0013] In a third aspect, embodiments of the present application further provide a method for improving the resolution of a sparse light field image. The method is applied to a light field camera provided in the second aspect, wherein the light field camera includes a sparse light field structure, a main lens, a sensor, and a processor; the sparse light field structure is disposed between the main lens and the sensor in the light field camera, and the processor is configured to perform the following method: Acquiring sparse light field image data through the sparse light field structure; extracting light field image data and two-dimensional image data from the sparse light field image data; Extracting perspective image data corresponding to a plurality of angles from the light field image data; A high-resolution image corresponding to the sparse light field image data is obtained according to the perspective image data corresponding to the multiple angles and the two-dimensional image data.

[0014] In an optional embodiment provided by the present invention, the sparse light field structure includes: a lens layer and a phase delay layer that are bonded together or separated; the lens layer is provided with a plurality of spaced microlens units to form a sparse microlens array, and extracting light field image data and two-dimensional image data from the sparse light field image data includes: Light field image data and two-dimensional image data are extracted from the sparse light field image data according to the distribution and position of the microlens units in the sparse light field structure, or according to data features of the light field data and two-dimensional data determined by a preset algorithm.

[0015] In an optional embodiment provided by the present invention, obtaining a high-resolution image corresponding to the sparse light field image data according to the perspective image data corresponding to the multiple angles and the two-dimensional image data includes: The perspective image data corresponding to the multiple angles and the two-dimensional image data are input into an image restoration model to obtain a high-resolution image corresponding to the sparse light field image data.

[0016] In an optional embodiment provided by the present invention, obtaining a high-resolution image corresponding to the sparse light field image data according to the perspective image data corresponding to the multiple angles and the two-dimensional image data includes: Dividing the perspective image data and the two-dimensional image data into N perspective region image data and two-dimensional region image data respectively according to preset data regions; Combining the visual area image data and the two-dimensional area image data of the same labeled area into a joint data block; Obtaining data quality weights corresponding to the viewing area image data and the two-dimensional area image data in the N joint data blocks according to a preset image quality evaluation method, and dividing the joint data blocks into a first joint data block or a second joint data block according to the data quality weights; Inputting the first joint data block into the first model to obtain a first high-resolution image data block, and inputting the second joint data block into the second model to obtain a second high-resolution image data block; Inputting all of the first high-resolution image data blocks and the second high-resolution image data blocks into a fusion model to obtain a high-resolution image corresponding to the sparse light field image data; The first high-resolution image data block is an image data block obtained by restoring the resolution of the two-dimensional area image data guided by the viewing area image data, and the second high-resolution image data block is an image data block obtained by restoring the resolution of the viewing area image data guided by the two-dimensional area image data.

[0017] The embodiments of the present application provide a sparse light field structure, a light field camera, and a method for improving the resolution of a sparse light field image. The sparse light field structure is applied to a light field camera and is arranged between the main lens and the sensor in the light field camera. The sparse light field structure includes: a lens layer and a phase delay layer that are bonded together or separated. The lens layer is provided with a plurality of spaced microlens units to form a sparse microlens array. This application is different from the traditional light field that uses densely packed microlenses or camera arrays for dense sampling to acquire data. The sparse microlens array composed of the sparse light field structure in the front-end hardware collects sparse light field image data, extracts light field image data and two-dimensional image data from the sparse light field image data, and the back-end algorithm fuses the two different data, making the algorithm model more lightweight while maintaining the authenticity of the reconstruction results, and improving the resolution of images taken by the light field camera.

[0018] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0020] Figure 1a A non-focusing light field structure diagram provided in an embodiment of the present application is shown; Figure 1b A focusing light field structure diagram provided by an embodiment of the present application is shown; Figure 2 A structural diagram of a light field camera provided in an embodiment of the present application is shown; Figure 3a A sparse light field structure diagram provided by an embodiment of the present application is shown; Figure 3b Another sparse light field structure diagram provided by an embodiment of the present application is shown; Figure 4 A top view of an array composed of microlens units and phase delay units provided in an embodiment of the present application is shown; Figure 5 A side view of an array composed of microlens units and phase delay units provided in an embodiment of the present application is shown; Figure 6 The following diagram shows the arrangement structure of micro-lens units of two shapes and specifications provided in the embodiment of the present application; Figure 7 A graph showing a trend of the resolution provided by an embodiment of the present application as a function of the number of pixels included in a light field image and the number of pixels included in a two-dimensional image is shown; Figure 8 A top view of a microlens array with sparsely arranged circular microlens units provided in an embodiment of the present application is shown; Figure 9 A top view of a microlens array with non-uniform and sparse arrangement of rectangular microlens units provided in an embodiment of the present application is shown; Figure 10 A flow chart of a method for improving the resolution of a sparse light field image provided by an embodiment of the present application is shown; Figure 11 A schematic diagram of sparse light field image data provided by an embodiment of the present application is shown; Figure 12 The left side is a light field image data diagram and the right side is a two-dimensional image data diagram provided by an embodiment of the present application; Figure 13 A schematic diagram of extracting objects observed at different angles provided by an embodiment of the present application is shown; Figure 14 A high-resolution image determination flow chart provided in an embodiment of the present application is shown; Figure 15 A flowchart of another method for improving the resolution of a sparse light field image provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] The terms "first", "second" and "third" in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects rather than to limit a specific order.

[0022] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner to facilitate understanding.

[0023] In the description of this application, unless otherwise specified, " / " indicates that the objects associated with each other are in an "or" relationship. For example, A / B can mean A or B. "And / or" in this application is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, in the description of this application, unless otherwise specified, "multiple" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0024] In the embodiments of the present application, at least one can also be described as one or more, and multiple can be two, three, four or more, which is not limited in this application.

[0025] like Figure 2 As shown, an embodiment of the present application provides a light field camera, which includes: a sparse light field structure, a main lens, a sensor, and a processor. The sparse light field structure is disposed between the main lens and the sensor, and the processor is configured to convert sparse light field image data collected by the sparse light field structure into a high-resolution image.

[0026] An embodiment of the present application provides a sparse light field structure, which is applied to a light field camera and is arranged between the main lens and the sensor in the light field camera. The sparse light field structure in this embodiment includes: a lens layer and a phase delay layer that are bonded together or separated; the lens layer is on the left or right side of the phase delay layer, and the microlens unit can have a convex surface facing the main lens or a convex surface facing the sensor. The lens layer is provided with a plurality of spaced microlens units to form a sparse microlens array. Figure 3a As shown, when the image plane of the main lens is located between the lens layer and the phase delay layer or on the surface of the lens layer, the lens layer is on the left side of the phase delay layer (applicable to non-focusing light field, or applicable to focusing light field); Figure 3b As shown, when the image plane of the main lens is located between the sensor and the phase delay layer, the lens layer is on the right side of the phase delay layer (applicable to focusing light fields).

[0027] Among them, a phase delay layer is formed by a phase delay unit with a specific thickness, and the focus position of the light field data path / two-dimensional data path is moved, so that the sensor can collect clear two-dimensional image data while collecting light field data.

[0028] In this embodiment, the thickness of the phase retardation layer can be controlled and adjusted by optimizing and iteratively the parameters of the phase retardation layer, the microlens, and the medium between the microlens unit and the sensor.

[0029] Specifically, the sparse light field structure in this embodiment is used to collect sparse light field image data, which includes light field image data and two-dimensional image data. Light field image data is the signal received from sensor pixels covered by the microlens units in the sparse light field structure; two-dimensional image data is the signal received from sensor pixels not covered by the microlens units in the sparse light field structure.

[0030] In this embodiment, in a sparse light field structure, the microlens units are not densely packed on the sensor, but rather are arranged at certain periodic intervals, non-densely packed, to form a sparse microlens array for collecting sparse light field image data; or the microlens units and phase delay units are arranged according to different combinations of two-dimensional coordinates to form a sparse microlens array. The microlens units include one or more morphological specifications, including the number of pixels covered by the microlens units and the shape of the microlens unit cross-section.

[0031] By setting different interval periods T and microlens unit morphology specifications, light field data with different proportions can be obtained. After reconstruction with the subsequent algorithm, different degrees of target resolution recovery can be achieved. This condition can be adjusted according to actual needs; Figure 4 and Figure 5 As shown, the period can be represented by (Tx, Ty), where Tx represents the period in the x-direction and Ty represents the period in the y-direction. The period includes microlens units and phase delay units. There are certain pixel units between each microlens unit that are not covered by the lens, which constitutes a sparse microlens array, thereby collecting sparse light field image data with spatial information and angle information on the microlens unit. In this embodiment, each microlens unit is arranged with a period interval of T, that is, the microlens units are not densely spread on the sensor, forming a modulation of the light by the microlens units to form a sparse microlens array. The morphological specifications of the microlens units can be different. For example, three period intervals can be set, and the morphological specifications corresponding to each period interval are different, that is, the number of pixels covered by the microlens units corresponding to the period T1, period T2, and period T3 or the shape of the cross section of the microlens units are different.

[0032] The aforementioned period T can be regarded as a description of one dimension. In this embodiment, it can be expanded to two dimensions Tx, Ty; the microlens units are asymmetrically arranged according to different combinations of Tx, Ty to form a new sparse microlens array. Figure 9The microlens units and phase delay units in the array shown demonstrate a non-uniform modulation array with different horizontal and vertical resolutions. In certain scenarios (such as industrial inspection), the required horizontal and vertical resolutions may vary, so the shape and proportion of the microlens units can be tailored to meet specific needs.

[0033] The arrangement can be adjusted according to the target resolution requirements to adapt to different scenarios, including but not limited to: arranging microlens units of one shape and specification sparsely on the sensor with a period T; arranging microlens units of several different shapes and specifications with a period T Sparsely arranged on the sensor, such as Figure 6 The following diagram shows the arrangement structure of two types of microlens units. As can be seen from the center to the edge, a phase delay unit, microlens unit 2, a phase delay unit, and microlens unit 1 are arranged respectively. This arrangement scheme is based on the situation where the center aberration of a real lens is small, while the edge aberration is large. Based on this, this embodiment can set the full field of view to 0, 0.25, 0.5, 0.75, and 1 zones, according to the situation where the center resolution of the actual lens is high and the edge aberration leads to low resolution. No microlens unit is placed in the 0-0.25 and 0.5-0.75 fields of view (phase delay unit is placed), a small microlens unit is placed in the 0.25-0.5 field of view (lens area: microlens unit 2 is placed), and a large microlens unit is placed in the 0.75-1 field of view (lens area: microlens unit 1 is placed). Combining the characteristics of the lens, the advantages of the sparse arrangement of microlenses are brought into play.

[0034] Furthermore, this embodiment can also adjust the ratio of the number of pixels occupied by the light field image and the number of pixels occupied by the two-dimensional image according to the resolution requirement, such as Figure 7 As shown, the recoverable target resolution (percentage) is calculated based on the duty cycle of the lens unit. The appropriate number of pixels occupied by the microlens unit is selected in different dimensions to form different microlens morphology specifications. The duty cycle of the lens unit is the ratio of the number of pixels occupied by all microlens units to the total number of pixels, that is, the ratio of the number of pixels contained in the light field image to the number of pixels contained in the two-dimensional image. The resolution recovery capabilities of lens units under different specifications vary, so when selecting the lens morphology specifications and ratios, they can be adjusted according to the design target resolution.

[0035] In an optional embodiment provided by the present invention, the microlens units and phase delay units are arranged in a regular non-orthogonal, regular orthogonal, irregular orthogonal or irregular non-orthogonal manner using one form specification or multiple form specifications to form a sparse microlens array.

[0036] The cross-sectional shape of the microlens unit includes, but is not limited to, circular, square, and rectangular. The specifications of the microlens unit are defined as: the number of pixels covered by the microlens unit and the pixel shape. For example, if the microlens unit covers an inscribed circle of N x N pixels, the shape is circular; if the microlens unit covers a circumscribed circle of N x N pixels, the shape is square; and if the microlens unit covers M x N pixels, the shape is rectangular.

[0037] Each microlens unit is sparsely arranged in a certain shape (including but not limited to circle, square, rectangle) and with different periods T to form a sparse microlens array; Figure 8 Shown is a circular, uniformly distributed sparse microlens array; Figure 9 Shown is a rectangular, non-uniform, sparse microlens array. To address the low resolution and insufficient resolving power of light field cameras, the present invention provides a sparse light field structure, which differs from the prior art densely packed microlens array, where microlenses are densely packed tangentially with adjacent microlenses. The present invention's sparse light field structure is applicable to both unfocused and focused light fields, and can achieve stable output without requiring extensive and complex calibration processes. It is highly time-efficient and has a wide range of applications. Furthermore, the sparse light field structure sampling allows the algorithm to simultaneously obtain both light field data and two-dimensional data. The back-end algorithm then fuses the two different data types to maintain the authenticity of the reconstruction results. This embodiment provides a sparse light field structure and a light field camera. The sparse light field structure is applied to a light field camera and is arranged between a main lens and a sensor in the light field camera. The sparse light field structure includes: a lens layer and a phase delay layer that are bonded together or separated. The lens layer is provided with a plurality of spaced microlens units to form a sparse microlens array. This application is different from the traditional light field that uses densely packed microlenses or camera arrays for dense sampling to acquire data. The sparse microlens array composed of the sparse light field structure in the front-end hardware collects sparse light field image data, extracts light field image data and two-dimensional image data from the sparse light field image data, and the back-end algorithm fuses the two different data to maintain the authenticity of the reconstruction result and improve the resolution of the image captured by the light field camera.

[0038] like Figure 10 As shown, an embodiment of the present application provides a method for improving the resolution of a sparse light field image. The method for improving the resolution of a sparse light field image provided by the present application may include: S101 : Acquire sparse light field image data through the sparse light field structure.

[0039] The sparse light field structure in the light field camera of this embodiment is used to collect data to obtain sparse light field image data. Since the sparse light field structure in this embodiment is composed of lens layers and phase delay layers that are bonded together or separated, and the lens layer is provided with a plurality of spaced microlens units to form a sparse microlens array. Therefore, the sparse light field image data obtained by the sparse light field structure includes light field image data obtained by the microlens unit and two-dimensional image data not obtained by the microlens unit. In order to improve the resolution of the sparse light field image data, it is necessary to extract light field image data and two-dimensional image data from the sparse light field image data in subsequent steps.

[0040] S102: Extract light field image data and two-dimensional image data from the sparse light field image data.

[0041] In this embodiment, the sparse light field structure includes: a lens layer and a phase delay layer that are bonded together or separated; the lens layer is provided with a plurality of spaced microlens units to form a sparse microlens array, and the extracting of light field image data and two-dimensional image data from the sparse light field image data includes: extracting light field image data and two-dimensional image data from the sparse light field image data according to the distribution and position of the microlens units in the sparse light field structure, or according to data features of the light field data and two-dimensional data determined by a preset algorithm.

[0042] Specifically, this embodiment extracts light field image data and two-dimensional image data from the sparse light field image data based on the distribution and position of the microlens units in the sparse light field structure. That is, the sparse light field image data ( Figure 11 As shown, the sparse light field image data collected by the sparse light field structure is extracted according to the pixels covered by the microlens unit and the pixels not covered by the microlens, so as to obtain light field image data, two-dimensional image data ( Figure 12 As shown, split light field image data and two-dimensional image data).

[0043] In this embodiment, if the distribution and position of the microlens units within the sparse light field structure are unknown, the data features of the light field data and the two-dimensional data can be determined according to a preset algorithm, and then the light field image data and the two-dimensional image data can be extracted from the sparse light field image data. Specifically, this embodiment can use operators including Haris, SIFT, ORB, SURF, etc. to extract feature points of the light field data and the two-dimensional data, and use brute force matching algorithms, correlation-based matching algorithms, shape-based matching algorithms, and feature point-based matching algorithms for matching, thereby determining the distribution of the light field data and the two-dimensional data, and further extracting the light field image data and the two-dimensional image data from the sparse light field image data.

[0044] S103 : Extracting perspective image data corresponding to a plurality of angles from the light field image data.

[0045] Due to the characteristics of the sensor, the two-dimensional image data only records the intensity distribution of light, while the light field data in the sparse light field image data records the intensity information and phase (angle) information of light. Therefore, this embodiment can perform spatial and angular splitting on the sparse light field image data, that is, the data obtained within a microlens unit is information from different angles of the same object point. Figure 13 In the multi-view image shown, in the light field image data, the pixels covered by the microlens can be regarded as collecting different angle information, and the object points represented by the light of the same angle in different microlens units are extracted to form a new image. Therefore, through this embodiment, the perspective image data corresponding to the low-resolution spatial information and angle information can be obtained.

[0046] It should be noted that this embodiment determines multiple perspective image data based on the morphological specifications of the microlens units and the sparse light field structure. Specifically, the corresponding perspective image data is determined based on the number of angles decomposed by the microlens units. If the sparse light field structure in this embodiment includes microlens units of multiple morphological specifications, corresponding light field image data can be obtained for each microlens unit morphological specification, and then perspective image data corresponding to multiple angles can be extracted from the light field image data for each morphological specification.

[0047] S104 : Obtain a high-resolution image corresponding to the sparse light field image data according to the perspective image data corresponding to the multiple angles and the two-dimensional image data.

[0048] In an optional embodiment provided by the present invention, obtaining a high-resolution image corresponding to the sparse light field image data based on the perspective image data and the two-dimensional image data corresponding to the multiple angles includes: inputting the perspective image data and the two-dimensional image data corresponding to the multiple angles into an image restoration model to obtain the high-resolution image corresponding to the sparse light field image data. The image restoration model is a pre-trained network model trained based on the perspective sample image data and the two-dimensional sample image data corresponding to the multiple perspectives, and their corresponding high-resolution images.

[0049] like Figure 14 As shown, in another optional embodiment provided by the present invention, obtaining a high-resolution image corresponding to the sparse light field image data according to the perspective image data and the two-dimensional image data corresponding to the multiple angles includes: S1041 , dividing the viewing angle image data and the two-dimensional image data into N viewing angle region image data and two-dimensional region image data respectively according to preset data regions.

[0050] Specifically, in this embodiment, the perspective image data and the two-dimensional image data can be divided into N perspective region image data and two-dimensional region image data. The size of N can be set according to actual needs. For example, N can be 6, 8, 10, etc., which is not specifically limited in this embodiment.

[0051] S1042: Combining the viewing area image data and the two-dimensional area image data of the same labeled area into a joint data block.

[0052] The labels may be specifically represented by numbers. For example, if the area is divided into 10 areas, the labels are 1-10.

[0053] For example, if three perspective image data and one two-dimensional image data are obtained through the previous steps, it is necessary to perform regional division on the three perspective image data and the two-dimensional image data respectively to obtain regional image data of four regions corresponding to the distribution of each image data, that is, to obtain regional image data of 16 regions (including 12 perspective regional image data and 4 two-dimensional regional image data). Then, the three perspective regional image data labeled 1 and the two-dimensional regional image data labeled 1 can be combined into a joint data block 1, the three perspective regional image data labeled 2 and the two-dimensional regional image data labeled 2 can be combined into a joint data block 2, the three perspective regional image data labeled 3 and the two-dimensional regional image data labeled 3 can be combined into a joint data block 3, and the three perspective regional image data labeled 4 and the two-dimensional regional image data labeled 4 can be combined into a joint data block 4.

[0054] S1043. Obtain data quality weights corresponding to the viewing area image data and the two-dimensional area image data in the N joint data blocks according to a preset image quality evaluation method, and divide the joint data blocks into a first joint data block or a second joint data block according to the data quality weights.

[0055] The preset image quality evaluation method is used to evaluate the image quality of the image data, and the data quality weight can be obtained using an existing image quality evaluation method. After obtaining the data quality weight, this embodiment divides the joint data block into the first joint data block or the second joint data block based on the size of the data quality weight.

[0056] Specifically, if the data quality weight of the multi-view area image data in the joint data block is higher than the data quality weight of the two-dimensional area image data, the joint data block is divided into a first joint data block; if the data quality weight of the multi-view area image data in the joint data block is lower than the data quality weight of the two-dimensional area image data, the joint data block is divided into a second joint data block; if the data quality weight of the multi-view area image data in the joint data block is equal to the data quality weight of the two-dimensional area image data, the data block is divided into both the first joint data block and the second joint data block.

[0057] S1044: Input the first combined data block into the first model to obtain a first high-resolution image data block, and input the second combined data block into the second model to obtain a second high-resolution image data block.

[0058] The first high-resolution image data block is an image data block obtained by restoring the resolution of the two-dimensional area image data guided by the viewing area image data, and the second high-resolution image data block is an image data block obtained by restoring the resolution of the viewing area image data guided by the two-dimensional area image data.

[0059] It should be noted that the first model, the second model, and the fusion model are pre-trained network models. The first model and the second model are trained based on sample data and sample labels. The image quality of the perspective image data in the sample data used to train the first model is higher than that of the two-dimensional image data; while the image quality of the perspective image data in the sample data used to train the second model is lower than that of the two-dimensional image data.

[0060] The first model is constructed and used to leverage the additional angular information provided by light field image data to compensate for the missing details in the sparse 2D image data. Given that light field image data records phase information, focusing light from the same direction to form an angular image is equivalent to observing the same object from different angles, enabling the observation of details missing from the 2D image data. This detail from the light field image data can be used to guide the reconstruction of the 2D image data.

[0061] A second model is constructed and used to leverage the spatial information contained in the sparse two-dimensional image data to compensate for the loss of spatial resolution in the light field. Considering that the resolution of light field image data decreases when the angular and spatial information are separated (for example, in multi-view images, the image resolution is relatively low compared to the two-dimensional image data, resulting in blurred edges when upsampled to the same size and number of pixels), the sparse reconstruction of the details of the same local object point, which records more edge details in the two-dimensional image data, can be utilized. By inputting the second joint data block into the second model to obtain a second high-resolution image data block, the resolution of the view image data is improved.

[0062] S1045 . Input all first high-resolution image data blocks and second high-resolution image data blocks into a fusion model to obtain a high-resolution image corresponding to the sparse light field image data.

[0063] like Figure 15 As shown in the flowchart of another method for improving the resolution of a sparse light field image provided by this embodiment, first, the sparse light field structure of this embodiment is used to capture the image. Figure 15 The sparse light field image data (1) obtained by “A” in the image is then extracted from the sparse light field image data (1), light field image data (2), two-dimensional image data (3), and perspective image data (4) corresponding to multiple angles are extracted from the light field image data (2), wherein Figure 15 The labels "1, 2, 3, 4" in the light field image data (2) represent various angles. The view angle image data (4) corresponding to the multiple angles are extracted from the light field image data (2), and then the view angle image data (4) and the two-dimensional image data (3) are divided into regions to obtain N view angle region image data and two-dimensional region image data, and the joint data block is divided into a first joint data block or a second joint data block according to the data quality weight; then the first joint data block is input into the first model (5) to obtain a first high-resolution image data block, and the second joint data block is input into the second model (6) to obtain a second high-resolution image data block; finally, all the results obtained by the first model (5) and the second model (6) are input into the fusion model (7), that is, all the first high-resolution image data blocks and the second high-resolution image data blocks are input into the fusion model (7), and a high-resolution image (8) corresponding to the sparse light field image data (1) is obtained.

[0064] Construct and use a fusion module, the main function of which is to receive the output of the first model and the second model as the input of the fusion model, so as to fuse them to obtain a high-resolution image corresponding to the sparse light field image data.

[0065] An embodiment of the present application provides a method for improving the resolution of a sparse light field image. The sparse light field structure is applied to a light field camera and is disposed between the main lens and the sensor in the light field camera. The sparse light field structure includes: a lens layer and a phase delay layer that are bonded together or separated. The lens layer is provided with a plurality of spaced microlens units to form a sparse microlens array. This application differs from the traditional light field method that uses densely packed microlenses or camera arrays for dense sampling to acquire data. The sparse microlens array composed of the sparse light field structure in the front-end hardware collects sparse light field image data, extracts light field image data and two-dimensional image data from the sparse light field image data, and the back-end algorithm fuses the two different data to maintain the authenticity of the reconstruction result and improve the resolution of the image captured by the light field camera.

[0066] In the case of dividing the functional modules into corresponding functional modules, a possible composition diagram of the processor involved in the above and embodiments may include: an acquisition module, configured to acquire sparse light field image data through the sparse light field structure; an extraction module, configured to extract light field image data and two-dimensional image data from the sparse light field image data; The extraction module is further configured to extract perspective image data corresponding to a plurality of angles from the light field image data; A determination module is configured to obtain a high-resolution image corresponding to the sparse light field image data based on the perspective image data corresponding to the multiple angles and the two-dimensional image data.

[0067] In an optional embodiment provided by the present invention, the sparse light field structure includes: a lens layer and a phase delay layer that are bonded together or separated; an extraction module, specifically used to extract light field image data and two-dimensional image data from the sparse light field image data based on the distribution and position of the microlens units in the sparse light field structure, or based on data features of the light field data and two-dimensional data determined by a preset algorithm.

[0068] In an optional embodiment provided by the present invention, the determination module is specifically configured to: The perspective image data corresponding to the multiple angles and the two-dimensional image data are input into an image restoration model to obtain a high-resolution image corresponding to the sparse light field image data.

[0069] In an optional embodiment provided by the present invention, obtaining a high-resolution image corresponding to the sparse light field image data according to the perspective image data corresponding to the multiple angles and the two-dimensional image data includes: Dividing the perspective image data and the two-dimensional image data into N perspective region image data and two-dimensional region image data respectively according to preset data regions; Combining the visual area image data and the two-dimensional area image data of the same labeled area into a joint data block; Obtaining data quality weights corresponding to the viewing area image data and the two-dimensional area image data in the N joint data blocks according to a preset image quality evaluation method, and dividing the joint data blocks into a first joint data block or a second joint data block according to the data quality weights; Inputting the first joint data block into the first model to obtain a first high-resolution image data block, and inputting the second joint data block into the second model to obtain a second high-resolution image data block; Inputting all of the first high-resolution image data blocks and the second high-resolution image data blocks into a fusion model to obtain a high-resolution image corresponding to the sparse light field image data; The first high-resolution image data block is an image data block obtained by restoring the resolution of the two-dimensional area image data guided by the viewing area image data, and the second high-resolution image data block is an image data block obtained by restoring the resolution of the viewing area image data guided by the two-dimensional area image data.

[0070] For specific definitions of the processor, please refer to the definitions of the sparse light field image resolution enhancement method above and will not be repeated here. Each module in the above-mentioned device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0071] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external terminal via a network connection. When executed by the processor, the computer program implements a method for improving the resolution of sparse light field images.

[0072] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed: Acquiring sparse light field image data through the sparse light field structure; extracting light field image data and two-dimensional image data from the sparse light field image data; Extracting perspective image data corresponding to a plurality of angles from the light field image data; A high-resolution image corresponding to the sparse light field image data is obtained according to the perspective image data corresponding to the multiple angles and the two-dimensional image data.

[0073] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: Acquiring sparse light field image data through the sparse light field structure; extracting light field image data and two-dimensional image data from the sparse light field image data; Extracting perspective image data corresponding to a plurality of angles from the light field image data; A high-resolution image corresponding to the sparse light field image data is obtained according to the perspective image data corresponding to the multiple angles and the two-dimensional image data.

[0074] In one embodiment, a computer program product is provided, the computer program product comprising a computer program, the computer program being executed by a processor to implement the following steps: Acquiring sparse light field image data through the sparse light field structure; extracting light field image data and two-dimensional image data from the sparse light field image data; Extracting perspective image data corresponding to a plurality of angles from the light field image data; A high-resolution image corresponding to the sparse light field image data is obtained according to the perspective image data corresponding to the multiple angles and the two-dimensional image data.

[0075] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0076] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A sparse light field structure, characterized in that: The sparse light field structure is applied to a light field camera and is arranged between a main lens and a sensor in the light field camera; The sparse light field structure includes: a lens layer and a phase delay layer that are bonded together or separated; the lens layer is provided with a plurality of spaced microlens units to form a sparse microlens array.

2. The sparse light field structure according to claim 1, characterized in that The microlens units and phase delay units are arranged according to different combinations of two-dimensional coordinates to form a sparse microlens array; the microlens units include one or more morphological specifications, which are the number of pixels covered by the microlens units and the shape of the cross section of the microlens units.

3. The sparse light field structure according to claim 2, characterized in that The cross-sectional shape of the microlens unit includes but is not limited to circle, square, and rectangle.

4. The sparse light field structure according to claim 2, wherein: The microlens units and phase delay units are arranged in a regular non-orthogonal, regular orthogonal, irregular orthogonal or irregular non-orthogonal manner using one form specification or multiple form specifications to form a sparse microlens array.

5. The sparse light field structure according to claim 1, wherein: The lens layer is on the left or right side of the phase retardation layer; When the image plane of the main lens is located between the lens layer and the phase delay layer or on the surface of the lens layer, the lens layer is on the left side of the phase delay layer; When the image plane of the main lens is located between the sensor and the phase delay layer, the lens layer is on the right side of the phase delay layer.

6. A light field camera, characterized in that The light field camera comprises: the sparse light field structure provided by any one of claims 1 to 5, a main lens and a sensor, and a processor, wherein the processor is configured to convert the sparse light field image data collected by the sparse light field structure into a high-resolution image.

7. A method for improving the resolution of a sparse light field image, characterized in that: The method is applied to a light field camera provided in claim 6, wherein the light field camera comprises a sparse light field structure, a main lens, a sensor, and a processor; the sparse light field structure is disposed between the main lens and the sensor in the light field camera, and the processor is configured to perform the following method: Acquiring sparse light field image data through the sparse light field structure; extracting light field image data and two-dimensional image data from the sparse light field image data; Extracting perspective image data corresponding to a plurality of angles from the light field image data; A high-resolution image corresponding to the sparse light field image data is obtained according to the perspective image data corresponding to the multiple angles and the two-dimensional image data.

8. The method according to claim 7, characterized in that The sparse light field structure includes: a lens layer and a phase delay layer that are bonded together or separated; the lens layer is provided with a plurality of spaced microlens units to form a sparse microlens array, and extracting light field image data and two-dimensional image data from the sparse light field image data includes: Light field image data and two-dimensional image data are extracted from the sparse light field image data according to the distribution and position of the microlens units in the sparse light field structure, or according to data features of the light field data and two-dimensional data determined by a preset algorithm.

9. The method according to claim 7, characterized in that The obtaining, according to the perspective image data corresponding to the multiple angles and the two-dimensional image data, a high-resolution image corresponding to the sparse light field image data, comprises: The perspective image data corresponding to the multiple angles and the two-dimensional image data are input into an image restoration model to obtain a high-resolution image corresponding to the sparse light field image data.

10. The method according to claim 7, characterized in that The obtaining, according to the perspective image data corresponding to the multiple angles and the two-dimensional image data, a high-resolution image corresponding to the sparse light field image data, comprises: Dividing the perspective image data and the two-dimensional image data into N perspective region image data and two-dimensional region image data respectively according to preset data regions; Combining the visual area image data and the two-dimensional area image data of the same labeled area into a joint data block; Obtaining data quality weights corresponding to the viewing area image data and the two-dimensional area image data in the N joint data blocks according to a preset image quality evaluation method, and dividing the joint data blocks into a first joint data block or a second joint data block according to the data quality weights; Inputting the first joint data block into the first model to obtain a first high-resolution image data block, and inputting the second joint data block into the second model to obtain a second high-resolution image data block; Inputting all of the first high-resolution image data blocks and the second high-resolution image data blocks into a fusion model to obtain a high-resolution image corresponding to the sparse light field image data; The first high-resolution image data block is an image data block obtained by restoring the resolution of the two-dimensional area image data guided by the viewing area image data, and the second high-resolution image data block is an image data block obtained by restoring the resolution of the viewing area image data guided by the two-dimensional area image data.

Images