Method for acquiring light field through multifocus image analysis and optical system for acquiring light field

Abstract

The present invention relates to an algorithm for acquiring light field information by analyzing images of multiple focal points and a measurement structure thereof. A method for acquiring a light field using an optical system according to an embodiment of the present invention may include a step in which, when it is assumed that each of images measured at multiple focal points measures light rays or light fields starting from a specific position, predicted images are generated using the images of multiple focal points, only components that are always at the same position and of the same brightness are compared and extracted from the predicted images generated using the images of the multiple focal points, and a light field including only light rays starting from the specific position is acquired using only the components that are always at the same position and of the same brightness, and obtained as a result of the extraction.

Description

Method for acquiring a light field through multifocal image analysis and an optical system for acquiring a light field

[0001] The present invention relates to an algorithm for obtaining light field information by analyzing images of multiple focal points and a measurement structure thereof.

[0002] With the recent Fourth Industrial Revolution, including autonomous vehicles, GPS, lidar, Nuriho, and advanced city design, the acquisition and utilization of 3D spatial information is on the rise. Furthermore, 3D visual information measurement and expression systems, such as VR, AR, and virtual reality, are also attracting attention.

[0003] Among these, measurement systems for acquiring 3D information, despite their importance, are limited and fragmented. Although systems for acquiring 3D positional or depth information, such as radar, GPS, depth cameras, and motion sensors, have been systematically developed and commercialized, they only acquire limited characteristics of objects and light sources. Human visual perception includes not only depth information but also angle and color information. Just as diamonds and soap bubbles shine in various colors depending on the angle, objects appear differently depending on the light source and observation angle. Even objects with the same light source and shape are observed differently depending on the object's reflection, transmission, and absorption characteristics. In other words, practical observation requires not only 3D positional information but also luminescence information at each location or luminescence information that varies depending on the viewpoint.

[0004] A field that measures or expresses light information including not only position but also position and angle information is called a light field, and the present invention is also a type of light field camera structure and light field information acquisition algorithm. A light field generally expresses the information of light rays as 5-dimensional information of position (3 dimensions) and angle (2 dimensions), excluding color, polarization, brightness information, etc., and if two sensor coordinates (x, y) and (u, v) whose positions are known are used, the information of light rays can be expressed in 4 dimensions. If 4D light field information acquisition is possible, it becomes possible to freely control visual information, such as changing the focus and viewpoint of the image and creating an image that is in focus at all locations.

[0005] A typical 2D camera uses a lens to focus and capture images. Light rays from multiple angles originating from a single point at the focal point converge to a single point on the sensor, losing angular information and leaving only positional information. Light rays originating from multiple angles originating from a point outside the focal point are measured at multiple locations on the sensor because they are out of focus. If there is a single light source, and its location and camera information are known, it is possible to decompose the angles from each of the blurry measurements on the sensor. However, if the origin of the light is unknown, the angles from each blurry measurement are completely unknown. Furthermore, if there are multiple light sources, the light rays originating from various locations are all mixed together, and even if all the light sources are known, it is impossible to distinguish the location from which the light was incident when measured at a specific location on the sensor.

[0006] A light field camera has a position (x, y, z) and an angle ( ) to obtain brightness (color) information according to the subject. In the case of the multi-camera method, multiple cameras are precisely arranged, and multi-view images are stitched together to obtain position and angle information. It is possible to obtain an image similar to a photograph taken with a large aperture camera with a short exposure time, and it is possible to restore an image obscured by an object. However, it is difficult for an individual to use because dozens of cameras must be precisely arranged. The MLA (micro lens array) type light field camera obtains position and angle information by using the MLA placed between the lens and sensor of a conventional camera. Among the MLA methods, Plenoptics 1.0 places the MLA on the image plane of the main lens and records the light rays focused on the ML (micro lens) by separating them by angle. Therefore, it has information of a relatively large number of viewpoints, but has a spatial resolution equal to the number of MLAs at one viewpoint. Plenoptics 2.0 is a method to solve the low spatial resolution of Plenoptics 1.0 by arranging MLAs in front or behind the main lens focus position, so that each ML acts like a small camera and acquires an image formed on the image plane of the main lens.

[0007] Plenoptics 2.0 can utilize MLAs with relatively large MLs, achieving very high spatial resolution at a single point in time, but at the expense of reduced angular resolution. Both MLA methods follow the equation (sensor resolution) = (number of viewpoints) x (spatial resolution per viewpoint), so even when using the same sensor, they inevitably exhibit lower spatial resolution than 2D cameras, and limitations exist, such as the use of MLA.

[0008] Existing light field cameras have a lower resolution than general cameras because they have a trade-off relationship between angular information (number of viewpoints) and spatial information (spatial resolution per viewpoint), and require multiple cameras or additional optical elements such as MLA.

[0009] The purpose of the present invention is to provide an algorithm for obtaining light field information by analyzing images of multiple focal points and a measurement structure thereof.

[0010] The present invention aims to provide a technology capable of obtaining light field information by using existing 2D (or 3D) optical systems such as cameras and microscopes without additional optical elements.

[0011] The present invention aims to provide an optical system capable of simultaneously measuring multiple focus images in a short period of time.

[0012] The purpose of the present invention is to enable acquisition of light field information without prior information on the luminescence pattern, and to leave only the PSF or only the position information.

[0013] The purpose of the present invention is to enable calculation in a non-iteration manner even without knowing the location of a light source or information on a light emission pattern in addition to the structure of a measurement system.

[0014] The present invention aims to enable the acquisition of 4D light field information using two or more focus images (2D information), and thus to enable the use of the same as a compression method for storing 4D light field information.

[0015] A method for obtaining a light field using an optical system according to an embodiment may include, assuming that each image measured at multiple foci all measures a ray or light field originating from a specific location, a step of generating prediction images using the images of the multiple foci, a step of comparing and extracting only components of brightness that are always the same location and common from the prediction images using the images of the multiple foci, and a step of obtaining a light field that includes only light rays originating from the specific location by using only the components of brightness that are always the same location and common, which are the extracted results.

[0016] The step of always using the predicted images using the images of the various focal points according to one embodiment to generate the predicted images may include a step of predicting the light rays starting from the specific location as a common component, which is a common brightness, at the same location even when using different focal points images.

[0017] The step of comparing and extracting only the components of common brightness at the same location from the predicted images using images of multiple focal points according to an embodiment may include a step of representing the components of light rays originating from different locations, which are not the common components at the specific location, as dynamic components, by other light sources.

[0018] The step of comparing and extracting only the components of common brightness at the same location from the predicted images using the images of the above-described multiple focuses according to an embodiment may include a step of confirming the dynamic components by continuously moving the components by light rays originating from different locations when the predicted images are generated using images measured continuously while changing the focus in one direction.

[0019] The step of extracting and comparing only the same location, common brightness components according to one embodiment may include a step of extracting and comparing only the same location, common brightness components by considering at least one of the cases where a light source exists only at one specific location or the cases where a light source exists at multiple locations.

[0020] The step of extracting and comparing only the same location and common brightness components according to one embodiment may include a step of extracting common components by comparing at least two or more images including prediction using at least two or more measured images or prediction using one measured image and actual values ​​at the predicted location, and a step of distinguishing the common components from dynamic components based on the extracted common components.

[0021] The step of extracting and comparing only the components of common brightness at the same location according to one embodiment may include a step of extracting only the common components based on a minimum method that obtains only the common components with the minimum value among several sets of components of common brightness at the same location.

[0022] According to one embodiment, the same location and common brightness may mean that the measured brightness occupies a common brightness.

[0023] The optical system according to one embodiment includes a light splitter that splits light transmitted through a lens system, and can be implemented with a structure in which sensors are placed at different distances or focal points using at least one light splitter in the form of a surface.

[0024] The optical system according to one embodiment can be implemented with a structure in which sensors are arranged at multiple locations and measured simultaneously using a light sensor having a transmittance higher than a reference value for light transmitted through a lens system.

[0025] An optical system according to one embodiment comprises a light splitter for splitting light transmitted through a lens system, and sensors, wherein the optical system is structured to place sensors at different distances or foci using at least one light splitter in the form of a plane, and when it is assumed that each image measured at multiple foci all measures a ray or light field originating from a specific location, the optical system is characterized in that prediction images are generated using the images at multiple foci, and only the light rays originating from the specific location are compared with the prediction images at multiple foci to extract only the components of the same location and common brightness.

[0026] In an embodiment, an optical system is configured so that even when different focus images are used, light rays departing from a specific location are predicted to have a common component, that is, a common brightness, at the same location, and light rays departing from different locations are predicted to have different dynamic components depending on the image used. In addition, when prediction is made using images continuously measured while changing the focus in one direction, components by light rays departing from different locations can also be predicted to move continuously or change shape continuously.

[0027] An optical system according to an embodiment can extract and compare only the same location, common brightness components by considering at least one of the cases where a light source exists only at one specific location or the cases where light sources exist at multiple locations.

[0028] An optical system according to an embodiment can extract common components by comparing predictions using at least two or more measured images or predictions using one measured image and at least two or more images including actual values ​​at the predicted location, in order to compare and extract only the components of common brightness at the same location.

[0029] An optical system according to an embodiment is a structure in which sensors are arranged at multiple locations and measured simultaneously using a light sensor having a transmittance higher than a reference value for light transmitted through a lens system, thereby obtaining a light field.

[0030] In addition, the optical system according to one embodiment can predict how measurements will be made at any location when each image measured at multiple focal points is assumed to measure a ray or light field originating from a specific location, and can extract common components by predicting based on a specific location in order to obtain only light rays originating from a specific location.

[0031] According to one embodiment, an algorithm and a measurement structure for obtaining light field information by analyzing images of multiple focal points can be provided.

[0032] According to one embodiment, a technology can be provided that can acquire light field information by using existing 2D (or 3D) optical systems such as cameras and microscopes without additional optical elements.

[0033] According to one embodiment, an optical system capable of simultaneously measuring multiple focus images in a short period of time can be provided.

[0034] According to one embodiment, it is possible to obtain light field information without prior information on the luminescence pattern, leaving only the PSF, or leaving only the position information.

[0035] According to one embodiment, calculations can be performed in a non-iteration manner even without knowing the location of the light source or the light emission pattern information in addition to the structure of the measurement system.

[0036] According to one embodiment, since it is possible to obtain 4D light field information using two or more focus images (2D information), it can be used conversely as a compression method for storing 4D light field information.

[0037] FIG. 1 is a drawing illustrating a method for obtaining a light field according to an embodiment.

[0038] Figure 2 is a diagram showing common components and dynamic components that vary from set to set in relation to the minimum method.

[0039] Fig. 3 shows a virtual sensor (S) using the measurement values ​​of each sensor, assuming that the location of the light source existing at location L is at location L'. ref ) is a drawing showing an embodiment of predicting a value to be measured at a location.

[0040] Figure 4 is a diagram showing the path of light rays when one light source and two or more sensors exist on different planes.

[0041] Figure 5 shows a virtual sensor (S) using the measured values ​​of sensors when light sources are present at positions L1 and L2. ref ) is a diagram showing the prediction process at the location.

[0042] Figure 6 is a drawing showing the results of a simulation in which light sources of four different light-emitting forms L1 to L4 are measured at a total of seven locations from D1 to D7.

[0043] FIG. 7A is a diagram illustrating an optical system including a light splitter structure for simultaneously acquiring multiple focus images according to one embodiment.

[0044] FIG. 7b is a diagram illustrating an optical system including a transparent optical sensor structure for simultaneously acquiring multiple focus images according to one embodiment.

[0045] Figure 8 is a diagram showing a case where there are images composed of multiple patterns that are additively mixed.

[0046] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

[0047] Embodiments according to the concept of the present invention may have various modifications and take various forms, and thus, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit embodiments according to the concept of the present invention to specific disclosed forms, but rather includes modifications, equivalents, or alternatives that fall within the spirit and technical scope of the present invention.

[0048] While terms such as "first" or "second" may be used to describe various components, these components should not be limited by these terms. These terms are intended solely to distinguish one component from another. For example, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component," without departing from the scope of the invention.

[0049] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions that describe relationships between components, such as "between," "immediately between," or "directly adjacent to," should be interpreted similarly.

[0050] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present invention. The singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0051] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0052]

[0053] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these embodiments. The same reference numerals in each drawing represent the same components.

[0054]

[0055] FIG. 1 is a drawing illustrating a method for obtaining a light field according to an embodiment.

[0056] A method for acquiring a light field according to an embodiment first assumes that the measured images are measured only by rays originating from a light source (location) of interest, and when each image measured at multiple foci is assumed to measure rays or light fields originating from a specific location, prediction images can be generated using the images at the multiple foci (step 101).

[0057] In order to distinguish only the light rays emitted from a specific location (point), multiple focus images (two or more) are required.

[0058] In the present invention, we need to use these images to predict how they will be measured at an "arbitrary focus (plane)".

[0059] This process works differently depending on how the A focus image will look at B focus, and whether it is similar to the image taken at the actual B focus image.

[0060]

[0061] In the present invention, the basic operation unit is a point unit, and although parallel operation or surface unit operation is possible, the basis is a point.

[0062] Let's say there is an image of focus a, an image of focus b, etc., and prediction using the image of focus a can be made by assuming that all pixel information constituting the image of focus a or the measurement values ​​of each pixel of the sensor are all due to a point light source that exists only at a "specific location", and based on this, it can be predicted how it will look at an "arbitrary focus".

[0063] The process of predicting an image can be interpreted as creating multiple predicted images using the image of focus b, the image of focus c, etc.

[0064] When comparing predicted images, if there is a component that always appears at the same location with the same brightness, it corresponds to a component that is identical or estimated to be identical to the light rays that started at a "specific location" when measured at an "arbitrary focus" location.

[0065] The present invention obtains a light field through a measurement-prediction-comparison-extraction process.

[0066] Among these, the prediction step is to create prediction images of what multiple measurement images would look like at a specific light source location and a common specific location (plane).

[0067] For example, if a particular light source is at position 5,5,5,

[0068] Assuming that a light ray passing through the locations x = 0~10, y = 0~10, z = 0 is measured, depending on the characteristics of the light source, it may be measured as a spherical wave, in the shape of a cone, or in various other ways.

[0069] And the process of predicting what it will look like at position z = -5 is simply to take the measured image and connect lines one by one from 5,5,5 to calculate the position and brightness to reach z = -5.

[0070] More specifically, if the brightness was measured to be 10 at 1,1,0, then by connecting the line passing through 5,5,5 and 1,1,0, we determine where 1,1,0 will reach at z=-5, and since the distance from the position of the light source to the measurement plane is twice the distance from the position of the light source to z=-5, assuming conservation of energy or diffusion, we can predict that the approximate brightness would have been measured to be 2.5.

[0071] The same thing can be done to predict what it would look like at z = -5 using images measured at x = 0~10, y = 0~10, z = 0,1,2,3.

[0072] Since the brightness change is proportional to the line connection and distance, if we look at the images measured at z = -5 using z = 0, 1, 2, and 3, they are all the same as the images actually measured at z = -5.

[0073] Meanwhile, assuming that the light source is not only at 5,5,5 but also at 7,7,7, but that the values ​​or images measured at z = 0,1,2,3 are only at 5,5,5, and that the measured values ​​at 5,5,5 are the same as before, and considering that one more ray of light at 7,7,7 is measured here, if we assume that the ray of 7,7,7 is measured as a circle with a radius of 1 centered at position 5,5 at z = 0, then it is measured at different positions and sizes at Z = 0,1,2,3 as an oblique, thin cone-shaped light.

[0074] Afterwards, using the images measured at z = 0, 1, 2, 3, if you connect the lines from the positions 5, 5, 5 and change the brightness in proportion to the distance, you can see the common component and something small moving in the image using z = 0 and the predicted images using z = 1, z = 2, z = 3, so the common component is the component caused by the actual 5, 5, 5 light source, and the component that moves while changing size is the component caused by 7, 7, 7, but it is different from the actual 7, 7, 7 measured at z = -5.

[0075] That is, if the 7,7,7 light source was measured at z = -5, it should be still, but it is predicted to move.

[0076] To generalize this a bit more, even if there are hundreds of light sources, when obtaining information about what the light source looks like at position 5,5,5, we simply measure at z = 0,1,2,3, extend a line from 5,5,5 to the measured value, change the brightness, and then compare the image measured at z = 0,1,2,3 with the image predicted at z = -5, and the common component at the common location becomes the information about the light source and the emission shape at 5,5,5.

[0077] This prediction process is the same regardless of the number of light sources.

[0078] In addition, a method for obtaining a light field according to an embodiment of the present invention compares and extracts only components of common brightness at the same location in prediction images using images of multiple focuses (step 102), and obtains a light field that includes only light rays originating from the specific location using only components of common brightness at the same location as the extracted result (step 103).

[0079] The predicted image corresponds to a prediction assuming a specific location using focus a.

[0080] According to the present invention, a process for obtaining information on one point (specific location) is possible.

[0081] In addition, similarly, it is possible to perform surface-level operations or various application methods by assuming that all measurements originate from the surface and making predictions and comparisons.

[0082] In order to compare and extract only the components of the same location and common brightness, at least one of the cases where a light source exists only at one specific location or the cases where a light source exists at multiple locations can be considered and only the components of the same location and common brightness can be extracted and compared.

[0083] In addition, in order to compare and extract only the components of common brightness at the same location, prediction using at least two or more measured images or prediction using one measured image and at least two or more images including the actual value at the predicted location are compared to extract the common components, and based on the extracted common components, the common components and the dynamic components can be distinguished.

[0084] In addition, in order to compare and extract only the components of common brightness at the same location, only the common components can be extracted based on the minimum method that obtains only the common components with the minimum value among several sets of components of common brightness at the same location.

[0085] A "focus image" generally refers to an image measured by focusing on a specific location. Conventional optical systems, such as cameras and microscopes, utilize lenses to focus and measure, and thus use terms like "specific focus image" and "focus position." However, the present invention is applicable regardless of the presence of other optical elements, such as lenses, and even with a sensor. In other words, the same principles can be applied to most optical systems, including those without lenses.

[0086] If only a sensor is present, an in-focus image could mean that the sensor is positioned at focus.

[0087] Additionally, the expression “rays originating from or emitted from a specific light source location” used in this specification may be reflected, transmitted, scattered, or emitted from the light source location.

[0088] In other words, a light field means a ray of light, and more specifically, when expressing light, it can be expressed as a wave, but it can also be expressed as moving from a specific location at a specific angle, passing through two locations, or spreading out to some extent at a specific angle from a specific location.

[0089] Similarly to the above, the emission pattern can be a pattern of reflection, transmission, scattering, or emission, and the light source and light source location can mean a point light source.

[0090] Multi-focus image(s) or multiple focus image(s) may mean image(s) captured (measured) with focus at different locations or image(s) captured with sensors placed at different locations, unless otherwise specified.

[0091] Unlike these methods, the present invention is an algorithm and a measurement structure for obtaining light field information by analyzing images of multiple focal points. Unlike existing methods, light field information can be obtained by using existing 2D (or 3D) optical systems such as cameras and microscopes without additional optical elements, and light field information can be obtained efficiently through the additional structure of the invention. As a more direct example, when the algorithm of the present invention is applied to an existing, relatively inexpensive, general 2D microscope to obtain light field information, the measured image can be reconstructed into 3D, such as by observing cells while rotating them or analyzing surface characteristics. It is expected that high-dimensional information can be obtained through relatively inexpensive equipment, and thus, it will have significantly higher accessibility in education and industry.

[0092] However, when using a general optical system to take images of multiple focuses, the shooting time may be long depending on the number of focus images, and it may not be suitable for moving objects. Therefore, a structure capable of measuring multiple focus images simultaneously in a short period of time was invented.

[0093] The algorithm and structure can be applied to microscopes, astronomical telescopes, etc., and it is also possible to remove PSF (point spread function) information and leave only the position, like deconvolution microscopy and CLEAN algorithms. In general, these have restrictions such as assuming a shift-invariant PSF or specifying a certain level of PSF conditions in advance, but the present invention can obtain light field information without prior information on the luminescence pattern, and can leave only the PSF or only the position information.

[0094] The present invention enables calculation in a non-iteration manner even without knowing the location of the light source or the information on the light emission pattern in addition to the structure of the measurement system.

[0095] Note that additional features such as additional functions or improved image quality may be possible through iteration.

[0096] Since it is possible to obtain 4D light field information using two or more focus images (2D information), it can be used conversely as a compression method for storing 4D light field information.

[0097] The algorithm for implementing the light field acquisition method according to the present invention consists of two processes: prediction and comparison, utilizing images measured at multiple focal points. While area- and surface-level operations are possible, the basic algorithm is point-level operations. For ease of computation, the algorithm requires knowledge of camera information, or the ability to obtain it through testing.

[0098] Camera information means that when a particular point in three-dimensional space emits light in various forms, the relationship between images focused at various locations must be known, and it must be possible to accurately (or similarly) predict how measurements will be made at other focal or measurement locations from one focused image, excluding areas outside the sensor area or where information is not available due to the aperture.

[0099] This refers to how the focal lengths and arrangements of lenses vary at each focus point, and approximate information on matrix optics and the entire lens and measurement system is sufficient. The degree of defocusing at different focal points can also be used to obtain a rough estimate of the relationship between multiple focused images.

[0100] In order to selectively extract only the information of a light source located at a specific point in a three-dimensional space, the algorithm of the present invention can assume that all values ​​or images measured at multiple focal points were measured due to light rays originating from the point.

[0101] Although it is expressed as all values ​​or images measured at multiple focal points, depending on the purpose, only a few images can be selectively used, or only a specific area from each image can be used.

[0102] Based on the above assumptions, in the prediction step of step 101, we can use each image to predict how it will be measured in any (focus) plane.

[0103] Even if the images used for prediction are different (different focus images are used), light rays that actually originate from that location will be predicted with a common brightness at the same location (stationary component), and light rays that originate from different locations will be predicted differently (dynamic component) depending on the images used.

[0104] When making predictions using images measured continuously while changing focus in one direction, such as from near focus to far focus or from far focus to near focus, the components due to light rays originating from different locations also move continuously.

[0105] To extract only the light rays originating from a specific location, as shown above, we must always extract only the components with the same location and common brightness from the predictions using each (multiple focus) image. This step is called comparison. The definition of "same location and common brightness" is accurate if the light source is located at only one specific location. However, if the light source is located at multiple locations, an extended definition must be used.

[0106] Except for some exceptional cases such as special materials and very strong light rays, the image measured by the sensor is measured in the form of a linear sum (additive mixing) of each light beam, and the prediction using images measured at multiple focal points is also predicted in the form of a linear sum of light rays originating from multiple locations. The same location, common brightness does not mean that the common brightness is measured at the same location, but that it occupies a common brightness in the measured brightness. For example, in prediction using multiple images at pixel A, even though the common component always occupies a brightness of 1, a value greater than 1 may be predicted at pixel A due to the dynamic component.

[0107] However, in prediction using a single focus image, common components and dynamic components cannot be distinguished, and extraction of common components is possible only by comparing two or more images, such as prediction using at least two predictions or one measurement image and the actual value at the prediction location.

[0108] There may be various algorithms for extracting only common components in an additive mixed system, but the present invention can use the minimum method.

[0109] Figure 2 is a diagram showing common components and dynamic components that vary from set to set in relation to the minimum method.

[0110] As shown in Figure 2, the sum component is displayed as the sum of the common component and the dynamic component.

[0111] Because it is an additive mixed system, the sum component is the sum of the common component and the dynamic component. In the system of the present invention, only the sum component can be measured, and an algorithm that extracts only the common component is required. The common component and the dynamic component may appear in multiple locations, and the dynamic component in particular refers to dynamic components that change in various directions or shape.

[0112] The minimum method of the present invention obtains a common component as a minimum value at the same position for each set. For example, when the position is 4.5, the value is 1 for Sets A, C, and D, and 3 for Set B, so the minimum value in all sets at position 4.5 is 1. At position 2, the minimum value is 0 because Set A has 2, and Sets B, C, and D have 0. In this way, the minimum method is an algorithm that obtains only the common component as a minimum value among multiple sets at the same position. When the minimum method algorithm is applied to the light field acquisition algorithm of the present invention, additive noise can be reduced or some image quality can be improved depending on the predicted position setting.

[0113] Even in the case of area units and surface units, rather than point units, prediction and comparison algorithms can be used, but there may be limitations in accuracy depending on the application method. However, depending on the conditions, it may be possible to obtain information on light sources with an accuracy of 90% or more.

[0114] The resolution of the predicted / extracted common components may vary depending on the focus positions, prediction plane settings, and sensor resolution. In this case, since the same object was observed at multiple resolutions, high-resolution reconstruction is expected.

[0115] FIG. 3 is a drawing (300) showing the path of light rays when one light source and two or more sensors exist on different planes.

[0116] In order to extract light field information, the present invention uses out-of-focus images. Out-of-focus images contain light source emission information. For convenience of explanation, one light source (L) whose location is known is (x L , y L , z L ) exists, and it can be assumed that the light source emits multiple rays at different angles, and that the wavelength and optical power may vary depending on the angle.

[0117] Assume that a light sensor (S1) of known position, consisting of several pixels outside the light source position, is located in the plane z = d1. The position (x p , y p , d) Since the position of the light source is known, brightness information for each angle of the light source can be obtained by using the optical power measurement value of each pixel, and using this information, the measured value can be predicted even if the sensor is located in a different position.

[0118] In the preceding examples, the invention was described using a single light source for convenience of explanation. However, the light source can generally be interpreted as two or more.

[0119] In the present invention, predictions using at least two or more measured images, or predictions using a single measured image, can be compared with at least two or more images containing actual values ​​at the predicted location. Furthermore, only common components can be extracted based on a minimum method that obtains only common components by using the minimum value among multiple sets of components of the same location and common brightness measured at the same location.

[0120] When there is more than one light source, even if the locations of the light sources and the sensor are precisely known, it is difficult to distinguish which light source the light measured at a specific pixel originates from. To address this issue, the present invention utilizes a method that measures at multiple locations or focal points and selectively extracts only the components of the light emitted at a specific location using each measurement structure and location.

[0121] To achieve this, the present invention utilizes correlation. As before, assuming a single light source L and a single light sensor S1, the measured information for each pixel is independent of each other. Even assuming another, non-overlapping sensor (S2) located at z = d1, the measured values ​​for each pixel are independent of each other.

[0122] Fig. 3 shows a virtual sensor (S) using the measurement values ​​of each sensor, assuming that the location of the light source existing at location L is at location L'. ref ) is a drawing (300) showing an embodiment of predicting a value to be measured at a location.

[0123]

[0124] Figure 3 shows the case S where one light source is assumed to exist at a location different from the actual light source location. ref Describes the prediction of the values ​​to be measured.

[0125] The arrows in drawings 301 and 302 represent light beams, which may have different wavelengths and powers.

[0126] Drawing symbol 300 is a virtual sensor position (S) using the measurement values ​​of each sensor, assuming that the position of the light source existing at position L is at position L'. ref ) is a schematic diagram that predicts the values ​​to be measured at the location, S ref is S1~S n It can be one of them.

[0127] Drawing reference numeral 400 of FIG. 4 is a schematic diagram showing the path of light rays when one light source and two or more sensors exist on different planes. An optical sensor, including a lens, may exist between the light source and the sensors, but for convenience of explanation, only one light source and sensors are present. The arrow indicated by the dotted line indicates the path of light when passing through position C in S1.

[0128] In contrast, two sensors, S1 and S2, have z = d1, z = d2 (d2>d1>z L) When placed on a plane, the measured value of one sensor can be used to obtain the measured value of another sensor, and they are interdependent. Since general sensors do not transmit light, they cannot obtain the trajectory of light rays. For the structure above, there are methods to use a sensor with high transmittance, move the sensor to measure from two locations, or use an optical element such as a BS (Beam splitter) to divide and measure the light.

[0129] The same explanation can be given even if multiple optical elements, such as lenses and polarizing plates, are added, but for convenience of explanation, drawing reference numeral 400 indicates a structure consisting of one light source and multiple sensors.

[0130] The arrows in drawing symbol 400 represent light rays, which can have different wavelengths and powers. Depending on the structure of the optical system, light from a single light source can converge or diverge as it passes through a nearby sensor. However, in this case, due to the absence of a lens, the light source is likely to be divergent, as dictated by the law of conservation of energy.

[0131] If we describe the light ray in the drawing symbol 400, it passes through position a in S1, and if it is measured, if we know exactly the position of the light source, the sensor, and the optical system information, we can accurately predict that it will pass through position a' in S2, as shown in Fig. 3.

[0132] The relationship between pixels a and a' is determined when the positions of the light source, sensor, and optical system are fixed, so the relationship between pixels c and c' remains the same even if no light ray exists or is not measured at any pixel c. In this case, if no measurement is made at c, no measurement will be made at c'.

[0133] This is the relationship between pixel coordinates, whereby transforming the measurements from one sensor can predict the values ​​measured by another sensor. It's similar to a coordinate system transformation, and, excluding diffraction, scattering, etc., is determined by the positions of the light source and sensors, as well as other optical elements. When the combination of sensor and optical system is fixed, the relationship between the pixel coordinates of the two sensors varies depending on the light source's position, and is different for each light source.

[0134] Figure 5 shows a virtual sensor (S) using the measured values ​​of sensors when light sources are present at positions L1 and L2. ref ) is a diagram showing the prediction process at the location.

[0135] Figure 5 shows a virtual sensor position (S) using the measured values ​​of the sensors when the light source is at positions L1 and L2. ref ) is a schematic diagram showing the prediction process. Figure 5 assumes only two light sources existing in two locations, but the explanation is the same even if there are more than two light sources, and each light source can be composed of countless light rays with their own characteristics. S ref can be virtual sensor locations, S1~S n It could be one of them.

[0136] The present invention relates to a specific light source (L s ) to extract only the components of the light emitted from a specific light source (L s ) is assumed to start from a specific light source (L). As will be explained later, the components starting from each sensor are determined by a specific light source (L s ) and a specific light source (L s ) can be divided into components that do not originate from a specific light source (L s ) is a component that starts from 'S', assuming that one light source exists at a location different from the actual light source location. ref can be interpreted similarly to the light rays originating from the actual light source position (L) in the 'prediction of the value to be measured at', and a specific light source (Ls ) is a component due to light rays that do not originate from S, assuming that one light source exists at a location different from the actual light source location. ref The actual light source location in the prediction of the value to be measured at can be interpreted similarly to the other location (L').

[0137] In Fig. 5, a specific light source is set to L1 (assuming that all sensor measurements originate from L1), and S ref The values ​​to be measured can be predicted for each sensor. The values ​​measured from each sensor are composed of the sum of the components by L1 and L2, and since they are generally linear sums, the components by L1 and L2 can be considered separately.

[0138] S using the component by L1 in each sensor ref When the value to be measured at a location is predicted, it is predicted exactly as in the actual measurement, and the predicted values ​​of each sensor are the same. Using the component by L2, S ref The predicted values ​​that will be measured at a location are different from the actual measurements, and the predicted values ​​for each sensor are all different.

[0139] That is, when there are light sources L1 and L2, assuming that all measurements of the sensors originate from L1, S ref When predicting the values ​​to be measured in , the predicted values ​​of each sensor that are always predicted to be in the same location and brightness are components by L1, and the predicted values ​​that are always predicted differently from all predicted values ​​are components by L2.

[0140] Even if there are three or more light sources instead of two, it is assumed that all measurements originate from L1, and S ref When predicting the values ​​to be measured in , among the predicted values ​​of each sensor, the ones that are always predicted with the same location and brightness are components by L1, and the others are incorrect prediction values ​​that are different from the actual values ​​due to light sources other than L1. If only the components that are always predicted with the same location and brightness are extracted, only the L1 component can be selectively extracted.

[0141] The above structures are examples for general explanation. Sensors may be two or more, and may be arranged at angles or curved rather than planar. The inclusion of two or more sensors does not necessarily imply that they are connected as a single circuit or system. However, as described in the invention, if a single ray of light passes through two or more pixels, the two pixels belong to different sensors.

[0142] Additionally, as previously mentioned, the structure may be configured to measure (or utilize multiple focused images) from multiple locations while moving the sensor or lens. In addition to the optical sensor, additional optical elements such as lenses, beam splitters, and polarizing elements may be present.

[0143] Figure 6 is a drawing showing the results of a simulation in which light sources of four different light-emitting forms L1 to L4 are measured at a total of seven locations from D1 to D7.

[0144] Drawing number 610 corresponds to a structure in which light sources of four different light emission types from L1 to L4 are measured at a total of seven locations from D1 to D7, and assuming that all of the measured values ​​(620) at locations D1 to D7 were emitted from L4, the values ​​to be measured at the Ref. Plane are as shown in drawing number 630. In addition, the extracted pattern of L4 corresponds to drawing number 640.

[0145] Although the simulation in Figure 6 assumes seven sensors, information on the light source can be obtained with two to three measurements.

[0146] It is also possible to obtain the shape of the light source of L1, L2, and L3, assuming that all measurement values ​​are values ​​from light rays originating from L1, L2, and L3, and since the position of the light source and the position of the reference plane (640) are known and the shape of each light source visible on the reference plane (640) is obtained, it is the same as obtaining the position of the light source and brightness information by angle, and the image can be freely transformed, such as by refocusing and changing the viewpoint.

[0147] If the measurements assume a location other than L1 to L4 (L5), there is no component (L5) that is always predicted with the same location, brightness, and shape, because the predictions on the reference plane (640) are all different. That is, there is no component whose brightness is 0 at all angles or starts from L5.

[0148] What is needed for actual measurement is the structure of the measuring instrument or the structures at the time of measurement, and it is not necessary to know in advance the location of the light source or the emission pattern of each light source.

[0149] FIG. 7A is a diagram illustrating an optical system including a light splitter structure for simultaneously acquiring multiple focus images according to one embodiment.

[0150] Drawing reference numeral 710 represents a general optical system capable of changing focus, and drawing reference numeral 720 represents an optical system including a beam splitter for simultaneously acquiring multiple focused images.

[0151] The present invention obtains and analyzes images at multiple focal points by moving a sensor or modifying a lens system or optical system. Figure 5a illustrates the structure of a typical lens or optical system capable of changing focus. When utilizing such a structure, physical movement or modification is required to measure multiple focal points, such as moving the system to perform measurements at multiple locations or changing the sensor or lens at the same location. Therefore, while it is possible to apply the algorithm of the present invention by directly applying an existing lens or optical system, it may not be suitable for fast-moving objects.

[0152] Therefore, as shown in drawing symbol 720, a method of arranging sensors at different distances or focal points using a beam splitter can be considered. In this case, image and video measurements can be made at the same frame rate as existing systems, and when one of the sensors is focused on the desired object, 2D and 3D images can be acquired simultaneously, similar to existing 2D optical systems, cameras, and microscopes. The beam splitter can be in the form of a plane, and multiple beam splitters can be connected in series instead of just one.

[0153] FIG. 7b is a diagram illustrating an optical system including a transparent optical sensor structure for simultaneously acquiring multiple focus images according to one embodiment.

[0154] The optical system utilizing the structure of drawing number 730 of FIG. 7b arranges sensors at multiple locations and simultaneously measures using a method that utilizes high-transmittance optical sensors. The structure of drawing number 730 has the same advantages and characteristics as drawing number 720, but there is a limitation on the amount of light depending on the transmittance.

[0155] The sensors of drawing symbols 720 and 730 can be any number of two or more.

[0156] In addition, the algorithm of the present invention can be applied in the same way when measuring by changing the focus or position of the optical system using most 2D and 3D optical systems.

[0157] Figure 8 is a diagram showing a case where there are images composed of multiple patterns that are additively mixed.

[0158] Drawing numbers 810 to 850 of FIG. 8 are images composed of several additively mixed patterns. Among these, only the blue circle indicated in drawing number 850 is always the same, and the other patterns have different shapes, colors, etc. For analysis, the images are divided into sections as shown in drawing number 810.

[0159] The present invention requires an algorithm for extracting components that are always the same in location, brightness, and pattern, and components that move or change differently from each other. While many algorithms have been developed for situations where objects in front obscure objects behind, such as cars and roads, the present invention requires extracting components that are always the same in location, brightness, and pattern, among patterns that are added together (additive mixing), such as light.

[0160] When there are images composed of multiple additively mixed patterns, such as those shown in drawings 810 to 840, they are divided into sections and analyzed, as shown in drawing 610. Drawings 810 to 840 can all be divided into sections and analyzed in the same way as drawing 810.

[0161] Drawing number 810 is divided into 6 rows and 10 columns, but any number of sections may be divided, and the smaller the size of each section, the better. This can be the same as the resolution of the image or sensor.

[0162] For monochrome additive mixed images, the brightness can be compared, and for multi-chrome additive mixed images, the same can be applied even if separated by color channel.

[0163] Drawing numbers 810 to 840 are multi-chrome images, but when separated by color channel and explained as one color channel, they are the same as monochrome, so they can be explained as monochrome brightness.

[0164] The algorithm is very simple. The minimum intensity information in the same region in each image always corresponds to the same location, brightness, and pattern. If the regions are sufficiently small, stationary components, such as reference number 850, can be accurately extracted. Furthermore, if additive noise is present in reference numbers 810 to 840, an image with less noise can be extracted.

[0165] However, in this case, if the added, transformed, or unwanted components hardly move, the components of the same position, brightness, and pattern can be perfectly extracted, but some of the added, transformed, or unwanted components may be additionally mixed in.

[0166] When applied to the present invention, if the approximate positions of light sources are predicted and the added, deformed, or unwanted components are sufficiently moved or deformed, information on the light field can be effectively obtained.

[0167] Ultimately, by utilizing the present invention, an algorithm and a measurement structure for obtaining light field information by analyzing images of multiple focal points can be provided.

[0168] In addition, a technology can be provided that can acquire light field information by using existing 2D (or 3D) optical systems such as cameras and microscopes without additional optical elements, and an optical system that can simultaneously measure multiple focus images in a short period of time can be provided.

[0169] In addition, it is possible to obtain light field information without prior information on the light emission pattern, leaving only the PSF or only the position information, and calculation is possible in a non-iteration manner even if the location of the light source or the information on the light emission pattern is not known in addition to the structure of the measurement system.

[0170] Note that additional features such as additional functions or improved image quality may be possible through iteration.

[0171] In addition, since the present invention enables acquisition of 4D light field information using two or more focus images (2D information), it can be used conversely as a compression method for storing 4D light field information.

[0172]

[0173] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0174] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0175] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0176] Although the embodiments described above have been described with limited drawings, those skilled in the art will recognize that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0177] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A method for obtaining a light field using an optical system, A step of generating predicted images using images of multiple foci, assuming that each image measured at multiple foci measures a ray or light field originating from a specific location; A step of extracting only the components of common brightness at the same location by comparing them in the predicted images using the images of the above multiple focuses; and A step of obtaining a light field that includes only light rays originating from the specific location by using only the components of common brightness at the same location as the above extracted result. A method for obtaining a light field including:

2. In paragraph 1, The step of extracting only the components of common brightness at the same location from the predicted images using the images of the above multiple focuses is as follows. Even when using different focus images, the light rays starting from the specific location are predicted to have a common component, which is a common brightness, at the same location. A method for obtaining a light field, characterized by including:

3. In paragraph 2, The step of extracting only the components of common brightness at the same location from the predicted images using the images of the above multiple focuses is as follows. About the rays of light that started from different locations, The components that are not common at the specific location above are components from other light sources, and are displayed as dynamic components. A method for obtaining a light field, characterized by including:

4. In paragraph 1, The step of extracting only the components of common brightness at the same location from the predicted images using the images of the above multiple focuses is as follows. When generating predicted images using images measured continuously while changing the focus in one direction, a step to confirm dynamic components by continuously moving components caused by light rays originating from different locations. A method for obtaining a light field including:

5. In paragraph 1, The step of extracting and comparing only the components of the same location and common brightness is as follows: A step of extracting and comparing only the components of the same location and common brightness by considering at least one of the cases where a light source exists only at one specific location or where a light source exists at multiple locations. A method for obtaining a light field including:

6. In paragraph 1, The step of extracting and comparing only the components of the same location and common brightness is as follows: A step of extracting common components by comparing at least two images including predictions using at least two measurement images or predictions using one measurement image and actual values ​​at the predicted location; A step of distinguishing between the common components and dynamic components based on the common components extracted above. A method for obtaining a light field including:

7. In paragraph 6, The step of extracting and comparing only the components of the same location and common brightness is as follows: A step of extracting only common components based on the minimum method that obtains only common components with the minimum value among several sets of common brightness components at the same location. A method for obtaining a light field including:

8. In paragraph 1, A light field acquisition method characterized in that the above same location and common brightness mean that the measured brightness occupies a common brightness.

9. In paragraph 1, The above optical system Including a light splitter that splits light transmitted through the lens system, A light field acquisition method characterized in that it is implemented with a structure in which sensors are placed at different distances or focal points using at least one light splitter in the form of a surface.

10. In paragraph 1, The above optical system A light field acquisition method characterized by being implemented with a structure in which sensors are arranged at multiple locations and measured simultaneously using a light sensor having a transmittance higher than a reference value for light transmitted through a lens system.

11. In the optical system, A light splitter that splits light transmitted through a lens system, and sensors, A structure that places sensors at different distances or focal points using at least one light splitter in the form of a surface, and obtains a light field, The above optical system, An optical system for obtaining a light field, characterized in that, assuming that each image measured at multiple foci all measures a ray or light field originating from a specific location, prediction images are generated using the images of the multiple foci, and only the components of common brightness that are always the same location are compared and extracted from the prediction images using the images of the multiple foci, and a light field that includes only the light rays originating from the specific location is obtained using only the components of common brightness that are always the same location as the extracted result.

12. In paragraph 11, The above optical system, Even when using different focus images, light rays originating from the specific location are predicted to have a common component, which is a common brightness, at the same location. About the rays of light that started from different locations, Any components that are not common at the specific location above are components from other light sources and appear as dynamic components. An optical system for light field acquisition characterized in that when predictive images are generated using images measured continuously while changing the focus in one direction, a dynamic component is confirmed by continuously moving components caused by light rays originating from different locations.

13. In paragraph 11, The above optical system, In order to compare and extract only the components of the same location and common brightness, In the case where a light source exists only at one specific location, or in the case where a light source exists at multiple locations, at least one of the same location and common brightness components is extracted and compared, An optical system for light field acquisition, characterized in that it extracts common components by comparing at least two images including predictions using at least two or more measured images or predictions using one measured image and actual values ​​at the predicted location, and distinguishes the common components from dynamic components based on the extracted common components.

14. In the optical system, It is characterized by being implemented as a structure that arranges sensors in multiple locations and measures simultaneously using a light sensor having a transmittance higher than a reference value for light transmitted through a lens system. The above optical system, An optical system for generating prediction images using the images of the multiple foci, assuming that each image measured at multiple foci measures a ray or light field originating from a specific location, comparing and extracting only the components of common brightness at the same location from the prediction images using the images of the multiple foci, and obtaining a light field that includes only the light rays originating from the specific location using only the components of common brightness at the same location as the extracted result.

Images