Image generation device for generating a depth map with phase detection pixels

Abstract

Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) comprising the following: an image sensor (1331, 5230) which has the following features: a plurality of image sensor pixels configured to generate image signals corresponding to an object (1100), and a plurality of phase detection pixels configured to generate first and second phase signals (PS1, PS2) which are used to calculate a phase difference (PD1, PD2) between images; a lens driver (2332) which is configured to adjust the position of a lens (1310, 2310, 3310, 4310) in order to adjust the distance between the lens (1310, 2310, 3310, 4310) and the object (1100); a phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373) which is configured to to calculate first phase differences (PD1) based on the first phase signals (PS1), wherein the first phase signals (PS1) are generated when the lens (1310, 2310, 3310, 4310) is in a first position, and to calculate second phase differences (PD2) based on the second phase signals (PS2), wherein the second phase signals (PS1) are generated when the lens (1310, 2310, 3310, 4310) is in a second position that differs from the first position; and a depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374) which is configured to to generate first and second depth data (DD1, DD2) based on the first and second phase differences (PD1, PD2) respectively, wherein each of the first and second depth data (DD1, DD2) is assigned a distance between the majority of phase detection pixels and the object (1100), and to generate a depth map (DM) based on the first and second depth data (DD1, DD2).

Description

BACKGROUND 1. Technical field

[0001] At least some exemplary embodiments of the inventive concepts relate to imaging technologies and more precisely to a device for generating a depth map. 2. Description of the state of the art

[0002] Recently, various types of imaging technologies have been used. A wide variety of electronic devices have been widely deployed. Most of these electronic devices can perform image processing to display or generate images. For this purpose, most electronic devices may include a display device, an image acquisition device, an image generation device, an image processing device, and the like.

[0003] In particular, a "depth map" is one of the imaging technologies used in modern industry. The depth map contains information related to the distance between an object and an image-generating device (for example, a camera with an image sensor). The depth map can be used to create a three-dimensional (3D) image. This 3D image can be used to make films, video games, and similar media more engaging.

[0004] To generate a depth map, various types of devices and methods are used. For example, the depth map can be obtained through different methods, such as observing the waveform of light reflected after light with a specific waveform is emitted to an object, measuring the time it takes for light emitted to an object to return, and obtaining stereo information using two or more cameras. However, most depth map methods require an additional sensor or device, or additional image processing, such as image registration.

[0005] Therefore, although the need for electronic devices with high processing power increases as the size of the electronic device is gradually reduced, it is difficult to apply most of the depth mapping methods used in modern industry to "small" electronic devices. In other words, it is necessary to provide a depth mapping method using a device or circuit that occupies a small area or size.

[0006] Document US 2014 / 0036134A1 relates to a focus adjustment device comprising an image sensor with a plurality of phase difference detection pixels, each of which performs a pupil division of the recording lens, and a correction value unit for detecting a correction value for the focus position as a function of the spatial frequency of an object determined by the image sensor, wherein the focus position of the recording lens is adjusted based on the determined correction value.

[0007] Document US 2011 / 0 274 ​​420 A1 relates to an imaging device and a control method in which a phase difference is detected by means of a focus detection unit, a focus adjustment signal is generated from the image signal, and a focus lens is automatically controlled based on a determined focus deviation and distance information derived from it, so that improved automatic focusing is enabled even at shallow depths of field.

[0008] Publication US 2012 / 0 327 291 A1 concerns an automatic focusing system for image recording devices that combines contrast AF with phase difference AF to enable precise focus detection even in the case of perspective conflicts and shallow depth of field; for this purpose, in the event of unreliable measurement results, image areas are dynamically analyzed, divided into sub-areas and selectively re-evaluated. SUMMARY

[0009] At least some exemplary embodiments can provide an image-generating device that occupies a small area or size and can efficiently generate a depth map. According to some exemplary embodiments, a depth map can be generated without any additional sensor or device. According to at least some exemplary embodiments of the inventive concepts, multiple depth data points can be generated by a lens that can move to different positions.

[0010] According to at least some exemplary embodiments, an image-generating device comprises an image sensor having a plurality of image sensor pixels configured to generate image signals corresponding to an object, and a plurality of phase-detection pixels configured to generate a first and a second phase signal, which are used to calculate a phase difference between images; a lens driver configured to adjust the position of a lens in order to adjust the distance between the lens and the object;a phase difference calculator configured to calculate first phase differences based on the first phase signals, wherein the first phase signals are generated when the lens is in a first position, and to calculate second phase differences based on the second phase signals, wherein the second phase signals are generated when the lens is in a second position different from the first position; and a depth map generator configured to generate first and second depth data based on the first and second phase differences respectively, wherein each of the first and second depth data is associated with a distance between the plurality of phase detection pixels and the object, and to generate a depth map based on the first and second depth data.

[0011] The image generation device may include a phase difference predictor configured to predict values ​​of the second phase differences based on the first phase differences.

[0012] The phase difference calculator can be configured to calculate the first phase differences before calculating the second phase differences when the lens moves from the first position to the second position under control of the lens driver; the phase difference predictor can be configured to predict the values ​​of the second phase differences before or while the lens moves to the second position.

[0013] The depth map generator can be configured such that if a difference between the values ​​predicted by the phase difference predictor and the second phase differences calculated by the phase difference calculator is greater than a reference value, the depth map generator creates the depth map with reference to the difference.

[0014] Each of the majority of phase detection pixels can correspond to two of the majority of image sensor pixels.

[0015] The majority of phase detection pixels can be arranged in different positions from the majority of image sensor pixels such that the majority of phase detection pixels do not overlap with the majority of image sensor pixels.

[0016] The image generation device may also have a spatial frequency calculator which is configured to generate information about a spatial frequency associated with an image where the object was recorded by processing the image signals.

[0017] The spatial frequency calculator can be configured to generate first spatial frequency information when the lens is placed in the first position, and to generate second spatial frequency information when the lens is placed in the second position.

[0018] The spatial frequency calculator can further be configured to obtain at least one direction and one magnitude of a change in a spatial frequency value when the lens moves from the first position to the second position, based on the first and second spatial frequency information, and the depth map generator can be configured to generate the depth map with reference to at least one direction and one magnitude of the change in the spatial frequency value.

[0019] The image generation device may further include an image sensor chip which comprises the image sensor, the lens driver, the phase difference calculator and the depth map generator.

[0020] According to at least one exemplary embodiment of the inventive concepts, an image-generating device comprises a phase difference calculator configured to receive a first and a second phase signal generated by a plurality of phase-detection pixels contained in an image sensor, to calculate first phase differences based on the first phase signals, wherein the first phase signals are generated when a lens is in a first position, the lens being configured to move in a direction in which a distance from an object increases or decreases, and to calculate second phase differences based on the second phase signals, wherein the second phase signals are generated when the lens is in a second position different from the first position; a lens position controller, or aa lens position controller configured to calculate an in-focus position of the lens for focusing on the object based on at least one of the first and second phase differences, and to generate a lens driver signal to move the lens into the in-focus position; and a depth map generator configured to generate first and second depth data based on the first and second phase differences respectively, each of the first and second depth data being associated with a distance between the plurality of phase difference pixels and the object, and to generate a depth map based on the first and second depth data.

[0021] One of the first positions and the second position can correspond to the in-focus position.

[0022] The image generation device may further include a phase difference predictor configured to predict second phase difference values ​​based on first phase differences; and a spatial frequency calculator configured to generate information about a spatial frequency associated with an image where the object was recorded by processing image signals generated by a plurality of image sensor pixels contained in the image sensor.

[0023] The image-generating device may further include a reliability level calculator configured to calculate a first reliability level associated with the first phase differences and a second reliability level associated with the second phase differences, based on at least one of the values ​​predicted by the phase difference predictor, the second phase differences calculated by the phase difference calculator, and a direction of change in a spatial frequency value as the lens moves from the first position to the second position.

[0024] The depth map generator can be configured to generate the depth map by applying weighting values ​​to the first and second depth data based on the first and second reliability levels.

[0025] The image generation device may further include a depth map post-processor configured to change the resolution of the depth map by performing image registration on an object image and the depth map, wherein the object image is generated based on image signals produced by a plurality of image sensor pixels contained in the image sensor.

[0026] The image generation device may further include an operational processing device which includes an application processor, wherein the operational processing device is configured to implement the phase difference calculator, the lens position controller and the depth map generator.

[0027] According to at least some exemplary embodiments of the inventive concepts, an image-generating device is configured to generate a depth map, wherein the image-generating device comprises a phase difference calculator configured to calculate first and second phase differences based on the first and second phase signals respectively, wherein the first phase signals are generated by a plurality of phase-sensing pixels when a lens is in a first position, wherein the lens is configured to move in a direction in which a distance from an object increases or decreases, and wherein the second phase signals are generated by the plurality of phase-sensing pixels when the lens is in a second position which is different from the first position;and a depth map generator configured to generate first and second depth data based on the first and second phase differences respectively, with each of the first and second depth data being assigned a distance between the plurality of phase detection pixels and the object, and to generate the depth map based on the first and second depth data.

[0028] The image-generating device may further include a phase difference predictor configured to predict values ​​of the second phase differences based on the first phase differences, wherein the phase difference calculator and the depth map generator are configured such that, if the values ​​predicted by the phase difference predictor differ from the second phase differences calculated by the phase difference calculator, the phase difference calculator computes third phase differences based on third phase signals, wherein the third phase signals are generated by the plurality of phase detection pixels when the lens is in a third position that is different from the first and second positions;and the depth map generator produces third depth data, which is assigned to a distance between the majority of phase detection pixels and the object, based on the third phase differences, and generates the depth map based on the first to third depth data.

[0029] The image generation device may further include a spatial frequency calculator configured to generate first spatial frequency information associated with a first image where the object is captured by processing first image signals generated by a plurality of image sensor pixels when the lens is placed in the first position, to generate second spatial frequency information associated with a second image where the object is captured by processing second image signals generated by the plurality of image sensor pixels when the lens is placed in the second position, and to obtain a direction of change in a spatial frequency value based on the first and second spatial frequency information;and a reliability level calculator configured to calculate a first reliability level associated with the first phase differences and a second reliability level associated with the second phase differences, based on the direction in which the spatial frequency value is changed, wherein the depth map generator is configured to generate the depth map by applying weighting values ​​to the first and second depth data based on the first and second reliability levels.

[0030] According to at least one exemplary embodiment of the inventive concepts, an image-generating device comprises a lens; an image sensor comprising a plurality of image sensor pixels configured to generate image signals corresponding to an object, and a plurality of phase-detection pixels configured to generate first and second phase signals, wherein a position of the lens is movable relative to a position of the image sensor; and a memory which stores computer-readable instructions.and one or more processors configured to execute the instructions to determine first phase differences based on the first phase signals, wherein the first phase signals are generated based on the lens being placed in a first position relative to the image sensor, and to determine second phase differences based on the second phase signals, wherein the second phase signals are generated based on the lens being placed in a second position relative to the image sensor, to generate first and second depth data based on the first and second phase differences respectively, wherein each of the first and second depth data indicates a distance between the plurality of phase detection pixels and the object, and to generate a depth map based on the first and second depth data.

[0031] The image generation device may further include a lens driver configured to selectively change the position of the lens relative to the image sensor.

[0032] The one or more processors are further configured to predict the values ​​of the second phase differences based on the first phase differences.

[0033] One or more processors can be configured to calculate the first phase differences before calculating the second phase differences when the lens is moved from the first position to the second position under control of the lens driver, and to predict the values ​​of the second phase differences before or while the lens is moved to the second position under control of the lens driver. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other features and advantages of exemplary embodiments of the inventive concepts will become clearer through a detailed description of exemplary embodiments of the inventive concepts with reference to the attached drawings. The attached drawings are intended to illustrate exemplary embodiments of the inventive concepts and should not be interpreted to limit the intended scope of the claims. The attached drawings are not to be considered to be drawn to scale unless expressly stated otherwise. Fig. Figure 1 is a block diagram illustrating an image generation system which includes an image generation device according to at least one exemplary embodiment of the inventive concepts; Fig. 2, Fig. 3 to Fig. Figure 4 are concept diagrams illustrating an image sensor which has a phase detection pixel according to at least one exemplary embodiment of the inventive concepts; Fig. 5 and Fig. 6 are concept diagrams illustrating the process of using a phase detection pixel; Fig. Figure 7 is a block diagram illustrating an image generation device according to at least one exemplary embodiment of the inventive concepts; Fig. 8 is a flowchart which illustrates the operation of an image generation device of the Fig. 7 describes; Fig. 9 is a concept diagram illustrating the operation of an image generation device. Fig. 7 illustrates; Fig. 10, Fig. 11, Fig. 12, Fig. 13, Fig. 14, Fig. 15, Fig. 16 to Fig. Figure 17 are block diagrams illustrating image-generating devices according to at least some exemplary embodiments of the inventive concepts; and Fig. Figure 18 is a block diagram illustrating an electronic device which includes an image generation device according to at least one exemplary embodiment of the inventive concepts and interfaces thereof. DETAILED DESCRIPTION OF EXECUTION FORMS

[0035] Detailed exemplary embodiments of the inventive concepts are disclosed herein. However, certain structural and functional details disclosed herein are merely representative for the purpose of describing exemplary embodiments of the inventive concepts. Exemplary embodiments of the inventive concepts can, however, be implemented in many alternative forms and should not be considered limited to the embodiments described herein.

[0036] Accordingly, while exemplary embodiments of the inventive concepts are capable of various modifications and alternative forms, embodiments of these are shown by way of example in the drawings and described in detail. It should be understood, however, that there is no intention to limit exemplary embodiments of the inventive concepts to the specific forms that are disclosed, but rather, on the contrary, exemplary embodiments of the inventive concepts must encompass all modifications, equivalents, and alternatives that fall within the scope of exemplary embodiments of the inventive concepts. The same numbers refer to the same elements throughout the description of the figures.

[0037] It will be understood that, although the terms "first / first / first", "second / second / second", etc., may be used herein to describe different elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element could be referred to as a second element, and similarly, a second element could be referred to as a first element, without departing from the scope of exemplary embodiments of the inventive concepts. When used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0038] It will be understood that when an element is referred to as "connected" or "coupled" with another element, it may be directly connected or coupled to the other element, or intervening elements may be present. In contrast, when an element is referred to as "directly connected" or "directly coupled" with another element, no intervening elements are present. Other words used to describe a relationship between elements should be interpreted similarly (for example, "between" versus "directly between," "adjacent" versus "directly adjacent," etc.).

[0039] The terminology used herein is solely for the purpose of describing certain embodiments and is not intended to be limited to exemplary embodiments of the inventive concepts. When used herein, the singular forms "one" and "the" are intended to indicate the plural forms as well, unless the context clearly indicates otherwise. Furthermore, it shall be understood that the terms "includes," "including," "contains," and / or "including," when used herein, specify the presence of said features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0040] It should also be noted that in some alternative implementations, the functions / actions depicted may occur out of order, as indicated in the figures. For example, two figures shown sequentially may actually be executed essentially simultaneously, or sometimes in reverse order, depending on the functionality / actions involved.

[0041] Exemplary embodiments of the inventive concepts are described herein with reference to schematic illustrations of idealized embodiments (and intermediate structures) of the inventive concepts. As such, variations in the forms of the illustrations are to be expected as a result of, for example, manufacturing techniques and / or tolerances. Accordingly, exemplary embodiments of the inventive concepts should not be considered as limited to the specific forms of areas illustrated herein, but must include deviations in the forms that result, for example, from the manufacturing process.

[0042] Although corresponding top and / or perspective views of a (some) cross-sectional view(s) may not be shown, the cross-sectional view(s) of device structures illustrated herein may support a plurality of device structures extending along two different directions, as would be explained in a top view, and / or in three different directions, as would be illustrated in a perspective view. The two different directions may or may not be orthogonal to each other. The three different directions may include a third direction, which may be orthogonal to the two different directions. The plurality of device structures may be integrated into the same electronic device.For example, if a device structure (such as a memory cell structure or a transistor structure) is illustrated in a cross-sectional view, an electronic device may have a plurality of device structures (such as memory cell structures or transistor structures), as would be illustrated by a top view of the electronic device. The plurality of device structures may be arranged in an array or matrix and / or in a two-dimensional pattern.

[0043] Fig. Figure 1 is a block diagram illustrating an image generation system comprising an image generation device according to at least one exemplary embodiment of the inventive concepts. Referring to Fig. 1. An image generation system 1000 can have an object 1100 and an image generation device 1300. The object 1100 is a target to be captured. An image IMG, which is assigned to the object 1100, can be generated by operating the image generation device 1300.

[0044] As at least one exemplary embodiment of the inventive concepts, the image generating device 1300 can comprise a lens 1310, an image sensor chip 1330, and an image signal processor 1350. The image generating device 1300 can further comprise other components which are Fig. 1 not shown, exhibit. The image generating device 1300, which is in Fig. Figure 1 is only one example to facilitate an understanding of at least some exemplary embodiments of the inventive concepts. The image generation device 1300 can generate the image IMG, which is assigned to the object 1100.

[0045] The term "processor," as used herein, may refer to, for example, a hardware-implemented data processing device that has circuits physically structured to perform desired operations, including, for example, operations represented as code and / or instructions contained in a program. Examples of the hardware-implementing data processing device referred to above include, but are not limited to, a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), and a field-programmable gate array (FPGA).Processors that execute program code are programmed processors and therefore special-purpose computers.

[0046] The lens 1310 can receive light reflected by the object 1100 after it has been emitted by one or more light sources. As at least one exemplary embodiment of the inventive concepts, the image-generating device 1300 can comprise one or more lenses. Light passing through the lens 1310 can be directed to the image sensor chip 1330.

[0047] The image sensor chip 1330 can generate one or more image signals based on the light provided by the lens 1310. These image signals can contain information used to generate the image IMG associated with the object 1100. The image IMG can be generated based on these image signals. The image signals can be intended for the image signal processor 1350.

[0048] For example, the image sensor chip 1330 can have an image sensor pixel. The image sensor pixel can have one or more light transmission filters and one or more light-sensitive sensors. For example, each of the light transmission filters can allow red light, green light, and blue light to pass through; however, at least some exemplary embodiments of the inventive concepts are not limited to this example. Each of the light-sensitive sensors can generate an electrical signal (i.e., an image signal) which has an electrical characteristic (for example, a voltage) that corresponds to a characteristic (for example, intensity) of the light passing through a respective light transmission filter. For example, the one or more light transmission filters and the one or more light-sensitive sensors can be arranged in a pixel unit.For example, an image signal can be generated in accordance with each pixel.

[0049] Herein, at least some exemplary embodiments of the inventive concepts are not limited to the examples mentioned above. A configuration of a light transmission filter, the arrangement of a light transmission filter and a light-sensitive sensor, and the generation of an image signal can be implemented in various ways. Additionally, for example, the image sensor chip 1330 can further comprise various components such as an infrared light transmission filter and an infrared light sensor.

[0050] Furthermore, according to at least some exemplary embodiments of the inventive concepts, the image sensor chip 1330 can also include a phase detection pixel. The phase detection pixel can be used to perform "phase-difference autofocus." The phase detection pixel can generate a phase signal. The phase signal can be used to calculate a phase difference between image signals. According to at least some exemplary embodiments of the inventive concepts, the phase difference can be used to focus on an object and to measure a distance between an object and an image sensor. The phase detection pixel is further described with reference to the Fig. 2, Fig. 3, Fig. 4, Fig. 5 to Fig. 6 will be described.

[0051] The 1350 image signal processor can receive image and phase signals generated by the 1330 image sensor chip. The 1350 can perform operations to process these image signals. The image IMG, assigned to object 1100, can be generated based on these image signals. However, the image signals themselves are not necessarily suitable for generating the IMG image. To generate the appropriate IMG image, the 1350 can perform image signal processing.

[0052] For example, the image signal processor 1350 can perform image signal processing such as bad pixel correction, demosaicing, noise reduction, lens shading correction, gamma correction, and edge enhancement. However, at least some exemplary embodiments of the inventive concepts are not limited to the examples above. The image signal processor 1350 can also perform other types of image signal processing.

[0053] The image signal processor 1350 can focus on the object 1100 by processing the phase signals. Additionally, according to at least some exemplary embodiments of the inventive concepts, the image signal processor 1350 can generate a depth map DM by processing the phase signals. The depth map DM can be an image showing information associated with a distance between the object 1100 and an image sensor. It has been described that the image signal processor 1350 performs phase signal processing, focusing, and generating the depth map DM. However, as will be described below, at least one of the phase signal processing, focusing, depth map DM generation, and any combination thereof can be performed by the image sensor chip 1330.

[0054] The image signal processor 1350 can be implemented in hardware. For example, the image signal processor 1350 can include analog circuits or logic circuits for performing image signal processing. Alternatively, the image signal processor 1350 can be implemented in an operational processing unit. For example, the image signal processor 1350 can be implemented in an operational processing device which includes an application processor. The operational processing device can perform image signal processing by executing instruction code stored in a read-only memory (ROM) or program code loaded into a random-access memory (RAM). However, at least some exemplary embodiments of the inventive concepts are not limited to these examples.

[0055] According to at least one exemplary embodiment of the inventive concepts, as in Fig. As shown in Figure 1, the image signal processor 1350 is contained together with the image sensor chip 1330 in the same device. In at least the example shown in Figure 1, the image signal processor 1350 is contained together with the image sensor chip 1330. Fig. As shown in Figure 1, the image generating device 1300 can be implemented in a portable electronic device such as a digital camera, a smartphone, a tablet and a portable device including the image sensor chip 1330 and the image signal processor 1350.

[0056] According to at least one other exemplary embodiment of the inventive concepts, unlike in Fig. 1. The image signal processor 1350 and the image sensor chip 1330 are provided in separate devices. In at least the present exemplary embodiment, for example, the device comprising the image sensor chip 1330 can be an image acquisition device, and the device comprising the image signal processor 1350 can be a computing device comprising one or more processors. In other words, at least one exemplary embodiment of the inventive concepts can be implemented in various ways, and at least some exemplary embodiments of the inventive concepts are not limited to the configuration described in Fig. 1 is shown, limited.

[0057] According to at least one exemplary embodiment of the inventive concepts, as in Fig. As shown in Figure 1, the image signal processor 1350 is an image signal processing circuit, an image signal processing chip, or an image signal processing device that is provided separately from the image sensor chip 1330. In at least the example shown in Figure 1, the image signal processor 1350 is a separate image signal processing circuit, an image signal processing chip, or an image signal processing device. Fig. As shown in Figure 1, if the image generating device 1300 is a portable electronic device, the image sensor chip 1330 can be provided separately from the image signal processor 1350, and the image signal processor 1350 can be contained in an application processor.

[0058] According to at least one other exemplary embodiment of the inventive concepts, it is possible, according to at least one other exemplary embodiment of the inventive concepts, to proceed differently than in Fig. 1. The image signal processor 1350 may be partially or completely contained within the image sensor chip 1330. In this exemplary embodiment, the image sensor chip 1330 can generate an image signal and perform image signal processing. In other words, at least one exemplary embodiment of the inventive concepts can be implemented with different configurations, and at least some exemplary embodiments of the inventive concepts are not limited to the configuration described in Fig. 1 is shown, limited. Fig. Figure 1 merely illustrates an exemplary configuration to facilitate an understanding of at least some exemplary embodiments of the inventive concepts.

[0059] Components of an "image generation device" according to any exemplary embodiment of, or alternatively at least some exemplary embodiments of, the inventive concepts described below, can be implemented within one of the image sensor chip 1330 and the image signal processor 1350, or can be implemented to be subdivided into the image sensor chip 1330 and the image signal processor 1350. Alternatively, the components of the "image generation device" can be provided separately from both the image sensor chip 1330 and the image signal processor 1350. At least some exemplary embodiments of the inventive concepts can be implemented with different configurations. Configurations according to at least some exemplary embodiments of the inventive concepts are described with reference to the Fig. 7, Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12, Fig. 13, Fig. 14, Fig. 15, Fig. 16 to Fig. 17 will be described.

[0060] The Fig. 2, Fig. 3 to Fig. Figure 4 are concept diagrams illustrating an image sensor comprising a phase detection pixel according to at least one exemplary embodiment of the inventive concepts. An image sensor 1331 comprising a phase detection pixel according to at least one exemplary embodiment of the inventive concepts can be integrated into an image sensor chip 1330 of the Fig. It should contain 1.

[0061] Referring to Fig. 2. The image sensor 1331 can have a pixel array PA. The pixel array PA can be formed by a pixel unit PX. The image sensor 1331 can have a plurality of image sensor pixels and a plurality of phase detection pixels. The plurality of image sensor pixels can generate image signals corresponding to an object. The plurality of phase detection pixels can generate phase signals used to calculate a phase difference between images. The plurality of image sensor pixels and the plurality of phase detection pixels can be arranged by the pixel unit PX. The arrangement of the plurality of image sensor pixels and the plurality of phase detection pixels is described in greater detail below with reference to Fig. 3 and Fig. 4 will be described.

[0062] According to at least one exemplary embodiment of the inventive concepts, each of the plurality of phase-detection pixels can be configured to correspond to two of the plurality of image sensor pixels. Referring to the Fig. 2 and Fig. 3. Each of the multiple image sensor pixels can correspond to a pixel unit PX. Additionally, each of the multiple phase detection pixels PPX1 to PPXn can correspond to two pixel units. In other words, for example, two image sensor pixels can be used as one phase detection pixel PPX1. In this exemplary embodiment, a pixel unit PX can be used as both a phase detection pixel and an image sensor pixel.

[0063] In the exemplary embodiment above, all pairs of image sensor pixels can be used as individual phase-detection pixels. Alternatively, some of the majority of image sensor pixels cannot be used as any phase-detection pixel. For example, if the number of the majority of image sensor pixels is "p", the number of the majority of phase-detection pixels can be equal to or less than (p / 2) (i.e., n ≤ / (p / 2)).

[0064] According to at least one other exemplary embodiment of the inventive concepts, the plurality of phase-detection pixels can be arranged such that the plurality of phase-detection pixels do not overlap with the plurality of image sensor pixels. Referring to Fig. 4 represents a white rectangular shape, an image sensor pixel IPX, and a shaded rectangular shape represents a phase detection pixel PPX. In other words, the majority of phase detection pixels can be arranged in different positions from the majority of image sensor pixels. In at least the example shown in Fig. As shown in Figure 4, the phase detection pixels cannot be used as image sensor pixels. For example, each of the phase detection pixels can have a white light sensor; however, at least some exemplary embodiments of the inventive concepts are not limited to this.

[0065] In the exemplary embodiment above, some of the pixels contained in the pixel array PA can be used as image sensor pixels. Additionally, pixels not used as image sensor pixels can be used as phase detection pixels. According to at least some exemplary embodiments of the inventive concepts, all the pixels contained in the pixel array PA can be used as phase detection pixels.

[0066] In one exemplary embodiment, various modifications or corrections can be made to the configurations of phase detection pixels, the number of phase detection pixels, the arrangement of the phase detection pixels, and / or the positions of the phase detection pixels. Fig. 2, Fig. 3 to Fig. Figure 4 merely illustrates some of the possible configurations of the image sensor 1331, which has the majority of phase-detection pixels, and at least some exemplary embodiments of the inventive concepts are not limited thereto. The image sensor 1331 can be implemented to differ from configurations described in the Fig. 2, Fig. 3 to Fig. 4 are shown.

[0067] The Fig. 5 and Fig. Figure 6 are concept diagrams illustrating the process of using a phase-detection pixel. For example, a lens 1310, which is in an image-generating device 1300 (refer to Figure 6), can be used to... Fig. 1) is included, configured to be movable. More specifically, the lens 1310 can move in a direction in which the distance to an object 1100 increases or decreases. Accordingly, the distance between the lens 1310 and the object 1100 can be adjusted. The object 1100 can be focused or defocused according to the position of the lens 1310.

[0068] First, the following will be applied: Fig. 5. Referring to the first case CASE1, the distance between lens 1310 and object 1100 is relatively small. In the first case CASE1, lens 1310 moves out of focus. The in-focus position is the position of lens 1310 when focusing on object 1100. Because lens 1310 moves out of focus, a phase difference D1 can occur between the images formed on the image sensor 1331, which is contained in the image sensor chip 1330. Consequently, in the first case CASE1, object 1100 can be out of focus.

[0069] Referring to a second case, CASE2, lens 1310 is positioned in the in-focus position. When lens 1310 is in the in-focus position, the phase difference between images formed on image sensor 1331 can be zero (0). Therefore, in the second case, CASE2, object 110 can be focused.

[0070] Referring to a third case, CASE3, the distance between lens 1310 and object 1100 is relatively large. In this case, lens 1310 moves out of focus. Because lens 1310 moves out of focus, a phase difference D3 can occur between the images formed on image sensor 1331. Therefore, in case CASE3, object 1100 can be out of focus.

[0071] A plurality of phase-detection pixels contained in the image sensor 1331 can be used to focus on an object. As described above, the plurality of phase-detection pixels can generate phase signals. These phase signals can contain information about the positions of images formed on the image sensor 1331. Consequently, the phase signals can be used to calculate phase differences between images. The in-focus position of the lens 1310 can be calculated based on these phase differences. For example, a position of the lens 1310 where the phase difference is zero can be the in-focus position.

[0072] According to at least some exemplary embodiments of the inventive concepts, the plurality of phase-detection pixels can be used to focus on the object 1100 and can also be used to measure a distance between the object 1100 and the image sensor 1331. For example, to measure a distance between the object 1100 and the image sensor 1331, additional information such as phase differences between images formed on the image sensor 1331, a distance between the lens 1310 and the image sensor 1331, the size of the lens 1310, and the in-focus position of the lens 1310 can be used.

[0073] For example, information associated with a distance between the object 1100 and the image sensor 1331 can be pre-provided according to specific conditions, such as a specific in-focus position and a specific phase difference. The image-generating device 1300 can store information about specific conditions and the provided information, for example, in the form of a lookup table. According to at least one exemplary embodiment of the inventive concepts, the lookup table can be stored in the memory contained in the image-generating device. For example, the lookup table can display the relationship between the specific conditions and the provided information.The image generation device 1300 can calculate conditions such as an in-focus position and a phase difference and can then obtain a distance between the object 1100 and the image sensor 1331 according to the calculated conditions with reference to the stored information.

[0074] As another example, the image generation device 1300 can calculate conditions such as a phase difference and a distance between the lens 1310 and the image sensor 1331. The image generation device 1300 can perform a mathematical calculation (for example, calculating a trigonometric function using lengths, sides, and angles formed by sides, and the like) with respect to the calculated conditions. Based on the mathematical calculation, the image generation device 1300 can calculate a distance between the object 1100 and the image sensor 1331.

[0075] As described in the examples above, an absolute distance between object 1100 and image sensor 1331 can be calculated by the image generation device 1300 (for example, using image sensor chip 1330 and / or image signal processor 1350). Alternatively, if insufficient additional information is available, a relative distance between object 1100 and image sensor 1331 can be calculated based on phase difference information. A depth map showing information associated with the distance between object 1100 and image sensor 1331 can be generated based on either the absolute or the relative distance between object 1100 and image sensor 1331.

[0076] The examples mentioned above are provided here only to facilitate the understanding of at least some exemplary embodiments of the inventive concepts, and are not limited to these. A distance between the object 1100 and the image sensor 1331 can be calculated using various processing methods. In particular, if additional information such as phase differences between images formed on the image sensor 1331, the distance between the lens 1310 and the image sensor 1331, the size of the lens 1310, and the in-focus position of the lens 1310 are used, the more accurate absolute distance can be calculated.

[0077] Reference is now made to Fig. 6. When a distance between the object 1100 and the image sensor 1331 is calculated, the image generation device 1300 can generate a depth map. As described above, a pixel array PA of the image sensor 1331 can have a plurality of phase detection pixels PPX. According to at least some exemplary embodiments of the inventive concepts, a depth map of the entire section of the object 1100 can be generated by the plurality of phase detection pixels PPX. For example, according to at least some exemplary embodiments of the inventive concepts, a distance between the entire section of the object 1100 and the image sensor 1331 can be calculated instead of merely calculating a distance between a point or a relatively small section of the object 1100 and the image sensor 1331. Thus, the image generation device 1300 can generate the depth map of the entire section of the object 1100.

[0078] This is Fig. 6. An exemplary concept diagram is provided to facilitate the understanding of at least some exemplary embodiments of the inventive concepts and is not intended to limit at least some exemplary embodiments of the inventive concepts. As described above, the phase detection pixels can be implemented with different shapes. Various modifications or corrections can be made to the configurations of the phase detection pixels, the number of phase detection pixels, the arrangement of the phase detection pixels, and the positions of the phase detection pixels.

[0079] Fig. Figure 7 is a block diagram illustrating an image generation device according to at least one exemplary embodiment of the inventive concepts. Fig. 8 is a flowchart which illustrates the operation of an image generation device of the Fig. 7 describes. Fig. 9 is a concept diagram illustrating the operation of an image generation device. Fig. 7 illustrates.

[0080] Referring to Fig. 7. An image generation device 100 can comprise a phase difference calculator 103 and a depth map generator 104. The image generation device 100 can generate a depth map DM according to at least one exemplary embodiment of the inventive concepts.

[0081] According to at least one exemplary embodiment of the inventive concepts, the image-generating device 100 may comprise or be implemented by one or more circuits (for example, hardware) specifically structured to perform some or all of the operations described herein as being performed by the image-generating device 100 (or an element thereof). According to at least one exemplary embodiment of the inventive concepts, the image-generating device 100 may comprise or be implemented by a memory and one or more processors executing computer-readable code (for example, software) stored in the memory, which includes instructions corresponding to some or all of the operations described herein as being performed by the image-generating device 100 (or an element thereof).According to at least one exemplary embodiment of the inventive concepts, the image generating device 100 can be implemented by, for example, a combination of the hardware and processors described above which execute computer-readable code.

[0082] For example, the image generation device 100 can be equipped with the image sensor chip 1130 and / or the image signal processor 1350. Fig. 1 must be implemented.

[0083] In order to describe at least one exemplary embodiment of the inventive concepts, reference is now made to the following: Fig. 7, Fig. 8 to Fig. 9.

[0084] In Operation S110 of the Fig. 8 can be a lens 1310 of the Fig. 9 to a first position. As described above, the lens 1310 can move in a direction in which a distance from an object 1100 of the Fig. 9 increases or decreases. Accordingly, a distance between the lens 1310 and the object 1100 can be adjusted. At time “t1” of the Fig. 9. The lens 1310 can move to the first position in response to a lens position control signal. For example, the first position can be a fixed position. Alternatively, the first position can be an adjustable position. The first position can be selected or determined in various ways.

[0085] In Operation S120 of the Fig. 8. First phase signals PS1 can be generated. When the lens 1310 is placed in the first position, the object 1100 can be recorded. An image sensor 1331 of the Fig. Light reflected from object 1100 can be received by lens 1310. A plurality of phase detection pixels contained in image sensor 1331 can generate the first phase signals PS1, for example, based on the reflected light. The first phase signals PS1 can contain information about the positions of images that are formed on image sensor 1331 when lens 1310 is in the first position.

[0086] In Operation S130 of the Fig. 8. The first phase differences PD1 can be calculated. The phase difference calculator 103 can receive the first phase signals PS1 from the image sensor 1331. The phase difference calculator 103 can calculate the first phase differences PD1 based on the first phase signals PS1. The first phase differences PD1 can be phase differences that correspond to the images formed on the image sensor 1331 when the lens 1310 is placed in the first position.

[0087] In Operation S140 of the Fig. 8. Initial depth data DD1 can be generated. The initial depth data DD1 can be data that corresponds to a distance between the majority of phase detection pixels of the image sensors 1331 and the object 1100. The depth map generator 104 can receive the initial phase differences PD1 from the phase difference calculator 103. The depth map generator 104 can generate the initial depth data DD1 based on the initial phase differences PD1. For example, the depth data generator 104 can generate the initial depth data DD1 with reference to additional information such as a distance between the lens 1310 and the image sensor 1331, a size of the lens 1310, and an in-focus position of the lens 1310, as well as the initial phase differences PD1.

[0088] For example, a depth map DM can be generated based on only the initial depth data DD1. However, it is possible that the initial depth data DD1 contains inaccurate distance data, depending on the environment in which the object 1100 is recorded. If the initial depth data DD1 contains inaccurate distance data, the depth map DM cannot be generated accurately. Therefore, according to at least some exemplary embodiments of the inventive concepts, other depth data can still be generated to produce the depth map DM, which has a higher degree of reliability.

[0089] In Operation S150 of the Fig. At time 8, lens 1310 can move to a second position. The second position is different from the first position. At time “t2” of the Fig. 9. The lens 1310 can move to the second position in response to the lens position control signal. For example, the second position can be a fixed position or an adjustable position. The second position can be selected or determined in various ways.

[0090] According to at least one exemplary embodiment of the inventive concepts, one of the first and second positions can correspond to the in-focus position. According to at least the present exemplary embodiment of the inventive concepts, information associated with the in-focus position of lens 1310 can be obtained along with the generation of the depth map. However, at least some exemplary embodiments of the inventive concepts are not limited to the exemplary embodiment above. Each of the first or second positions can be any position other than the in-focus position.

[0091] Referring to Fig. 9. As the lens 1310 moves from the first position to the second position, the distance between the lens 1310 and the image sensor 1331 can decrease. However, at least some exemplary embodiments of the inventive concepts are not based on Fig. 9 limited. In some other exemplary embodiments, when the lens 1310 moves from the first position to the second position, a distance can be removed between the lens 1310 and the image sensor 1331. Fig. Figure 9 illustrates an example to facilitate an understanding of at least some exemplary embodiments of the inventive concepts.

[0092] In Operation S160 of the Fig. 8. Second phase signals PS2 can be generated. When lens 1310 is placed in the second position, object 1100 can be captured. The image sensor 1331 can receive light reflected from object 1100 through lens 1310. The majority of phase detection pixels contained in the image sensor 1331 can generate the second phase signals PS2. The second phase signals PS2 can contain information about the positions of images generated on the image sensor 1331 when lens 1310 is placed in the second position.

[0093] In Operation S170 of the Fig. 8. Second phase differences PD2 can be calculated. The phase difference calculator 103 can receive the second phase signals PS2 from the image sensor 1331. The phase difference calculator 103 can calculate the second phase differences PD2 based on the second phase signals PS2. The second phase differences PD2 can be phase differences that are associated with the images formed on the image sensor 1331 when the lens 1310 is placed in the second position.

[0094] In Operation S180 of the Fig. Second depth data DD2 can be generated. The second depth data DD2 can be data assigned to a distance between the majority of phase detection pixels of the image sensor 1331 and the object 1100. The depth map generator 104 can receive the second phase differences PD2 from the phase difference calculator 103. The depth map generator 104 can generate the second depth data DD2 based on the second phase differences PD2. For example, the depth map generator 104 can calculate the second depth data DD2 with reference to additional information such as a distance between the lens 1310 and the image sensor 1331, the size of the lens 1310, the in-focus position of the lens 1310, and the second phase differences PD2.

[0095] In Operation S190 of the Fig. Depth map DM can be generated using function 8. Depth map DM can be generated based on the first depth data DD1 and the second depth data DD2. Depth map generator 104 can generate depth map DM based on the first depth data DD1 and the second depth data DD2.

[0096] As described above, the first depth data DD1 may contain inaccurate distance data. Therefore, according to at least some exemplary embodiments of the inventive concepts, the depth map generator 104 can also generate the second depth data DD2. By referencing multiple depth data DD1 and DD2, inaccurate distance data (i.e., an error) contained in the first depth data DD1 or the second depth data DD2 can be corrected. The image generation device 100, according to at least one exemplary embodiment of the inventive concepts, can generate the depth map DM, which has higher reliability, with reference to the multiple depth data DD1 and DD2. The process of generating the depth map DM1 with reference to the first depth data DD1 and the second depth data DD2 is further described by reference to the Fig. 13, Fig. 14, Fig. 15, Fig. 16 to Fig. 17 will be described.

[0097] Referring to the Fig. 7, Fig. 8 to Fig. Section 9 describes how the lens 1310 moves to the first position and the second position, generating the two depth data points DD1 and DD2. However, to generate the depth map DM more accurately, three or more lens positions and three or more depth data points can be used. Fig. 7, Fig. 8 to Fig. The 9 examples are merely intended to facilitate an understanding of at least some exemplary embodiments of the inventive concepts, and at least some exemplary embodiments of the inventive concepts are not limited to these.

[0098] Additionally, with reference to the Fig. 7, Fig. 8 to Fig. Figure 9 describes how the lens 1310 moves to the second position after the operations performed when the lens 1310 is in the first position have been completed. However, the sequence of performing the operations according to at least one exemplary embodiment of the inventive concepts can be changed or modified according to a embodiment of the image-generating device 100.

[0099] For example, the lens 1310 can move to the second position during or before the first phase signals PS1, the first phase differences PD1, or the first depth data DD1 are generated. For example, the phase difference calculator 103 can calculate the first phase differences PD1 and the second phase differences PD2 simultaneously, or it can calculate the second phase differences PD2 earlier than the first phase differences PD1. For example, the depth map generator 104 can generate the first depth data DD1 and the second depth data simultaneously, or it can generate the second depth data DD2 earlier than the first depth data DD1.

[0100] According to at least some exemplary embodiments of the inventive concepts, it is sufficient that the image-generating device 100 generates the depth map DM based on the depth data DD1 and DD2. The sequence of carrying out the operations, which are described with reference to the Fig. 7, Fig. 8 to Fig. The embodiments described in section 9 can be changed or modified in various ways. At least some exemplary embodiments of the inventive concepts are not limited to the descriptions referred to in section 9. Fig. 7, Fig. 8 to Fig. 9 limited.

[0101] The image generation device 100 according to at least one exemplary embodiment of the inventive concepts does not require additional devices such as a time-of-flight (ToF) sensor, an infrared sensor, and a stereo camera. Consequently, the image generation device 100 according to at least one exemplary embodiment of the inventive concepts can generate the depth map DM while occupying a small area or size. Additionally, according to at least some exemplary embodiments of the inventive concepts, the image generation device 100 can generate the depth map DM, which has higher reliability, by correcting an error with reference to the multiple depth data DD1 and DD2.

[0102] Fig. Figure 10 is a block diagram illustrating an image-generating device according to at least one exemplary embodiment of the inventive concepts. Referring to Fig. 10. An image generating device 2300 can comprise a lens 2310, an image sensor chip 2330, and an image signal processor 2350. Additionally, the image sensor chip 2330 can comprise an image sensor 2331, a lens driver 2332, a phase difference calculator 2333, and a depth map generator 2334.

[0103] The configurations and functions of lens 2130 may be the same as those of lens 1310, which are described in the following sections: Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, Fig. 7, Fig. 8 to Fig. 9 have been described. Redundant descriptions relating to lens 2310 are omitted below for brevity. Light passing through lens 2310 may be directed to the image sensor 2331 of the image sensor chip 2330.

[0104] The 2331 image sensor can have multiple image sensor pixels and multiple phase detection pixels. The multiple image sensor pixels can generate image signals corresponding to an object. The multiple phase detection pixels can generate first phase signals PS1 and second phase signals PS2, which are used to calculate phase differences between images. The first phase signals PS1 can be generated when the 2310 lens is in a first position, and the second phase signals PS2 can be generated when the 2310 lens is in a second position.

[0105] Configurations and functions of the 2331 image sensor may be those of the 1331 image sensor, which are described in the Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, Fig. 7, Fig. 8 to Fig. 9 were described. Redundant descriptions, which are assigned to the image sensor 2331, are omitted below for the sake of brevity.

[0106] The lens driver 2332 can generate a lens position control signal LN. The lens position control signal LN can be a signal used to adjust the position of the lens 2310. In response to the lens position control signal LN, the lens 2310 can move in a direction that increases or decreases its distance from the object. Accordingly, the distance between the lens 2310 and the object can be adjusted. For example, the lens driver 2332 can generate the lens position control signal LN based on a lens driver signal LD ​​provided by the image signal processor 2350. Unlike in Fig. 10 However, in some other exemplary embodiments, the lens driver 2332 can generate the lens position control signal LN by performing an operation to control a position of the lens 2310 without the lens driver signal LD.

[0107] The phase difference calculator 2333 can calculate first phase differences PD1 and second phase differences PD2 based on the first phase signals PS1 and the second phase signals PS2, respectively. The depth map generator 2334 can generate a depth map DM based on the first phase differences PD1 and the second phase differences PD2.

[0108] The configurations and functions of the phase difference calculator 2333 and the depth map generator 2334 may have those of the phase difference calculator 103 and the depth map generator 104, which, with reference to the Fig. 7, Fig. 8 to Fig. 9 (for example, with reference to the image generation device 100). Redundant descriptions, which are assigned to the phase difference calculator 2333 and the depth map generator 2334, are omitted below for the sake of brevity.

[0109] In an exemplary embodiment, the image signal processor 2350 can perform image signal processing on the depth map DM, which is generated by the depth map generator 2334. Consequently, the image signal processor 2350 can generate a depth map DM' that is processed more accurately. Unlike in Fig. 10 However, in at least one other exemplary embodiment of the inventive concepts, the depth map DM, which is generated by the depth map generator 2334, can be output directly from the image generation device 2300 without image signal processing.

[0110] In an exemplary embodiment, the image signal processor 2350 can generate the lens driver signal LD ​​to control the lens driver 2332. Unlike in Fig. 10 However, in at least one other exemplary embodiment of the inventive concepts, the image signal processor 2350 can directly control a position of the lens 2310 without the lens driver 2332. In other words, it illustrates Fig. 10. An exemplary configuration is provided to facilitate an understanding of at least some exemplary embodiments of the inventive concepts and is not intended to limit at least some exemplary embodiments of the inventive concepts. At least one exemplary embodiment of the inventive concepts may be implemented with a different configuration from the configuration described in Fig. As shown in 10, it will be implemented.

[0111] The configurations and functions of the 2350 image signal processor may differ from those of the 1350 image signal processor. Fig. 1. Redundant descriptions, which are assigned to the 2350 image signal processor, are omitted below for the sake of brevity.

[0112] According to an exemplary embodiment of the Fig. 10. The phase difference calculator 2333 and the depth map generator 2334 can be implemented in the image sensor chip 2330. According to at least one exemplary embodiment of the inventive concepts, the image sensor 2331, the lens driver 2332, the phase difference calculator 2333 and the depth map generator 2334 can be implemented in an image sensor chip 2330.

[0113] However, at least some exemplary embodiments of the inventive concepts are not limited to the configurations which are in Fig. The image-generating device 2300, as shown in 10, is limited. It can also incorporate other components, which are shown in Fig. 10 not shown, exhibit, or may contain one or more components which are in Fig. 10 are shown, do not exhibit any. Fig. Figure 10 merely illustrates an exemplary configuration of the image generation device 2300.

[0114] Fig. Figure 11 is a block diagram illustrating an image-generating device according to at least one exemplary embodiment of the inventive concepts. Referring to Fig. 11. An image generating device 3300 can comprise a lens 3310, an image sensor chip 3330, and an image signal processor 3350. Additionally, the image signal processor 3350 can comprise a phase difference calculator 3353, a depth map generator 3354, and a lens position controller 3355.

[0115] The configurations and functions of lens 3310 may be the same as those of lens 1310, which are described in the following sections: Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, Fig. 7, Fig. 8 to Fig. 9 have been described. Redundant descriptions relating to lens 3310 are omitted below for brevity. Light passing through lens 3310 may be directed to image sensor chip 3330.

[0116] The 3330 image sensor chip can replace an 1331 image sensor (refer to the Fig. 2, Fig. 3, Fig. 4, Fig. 5 to Fig. 6) exhibit. The image sensor 1331 can have a plurality of image sensor pixels and a plurality of phase detection pixels. The plurality of image sensor pixels can generate image signals corresponding to an object. The plurality of phase detection pixels can generate first phase signals PS1 and second phase signals PS2, which are used to calculate phase differences between images. The first phase signals PS1 can be generated when the lens 2310 is placed in a first position, and the second phase signals PS2 can be generated when the lens 2310 is placed in a second position. The configurations and functions of the image sensor 1331 are described with reference to the Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, Fig. 7, Fig. 8 to Fig. 9 have been described.

[0117] The phase difference calculator 3353 can receive the first phase signals PS1 and the second phase signals PS2 from the image sensor chip 3330. The phase difference calculator 3353 can calculate first phase differences PD1 and second phase differences PD2 based on these first and second phase signals, respectively. The depth map generator 3354 can generate a depth map DM based on the first phase differences PD1 and the second phase differences PD2.

[0118] The configurations and functions of the phase difference calculator 3353 and the depth map generator 3354 may have those of the phase difference calculator 103 and the depth map generator 104, which, with reference to the Fig. 7, Fig. 8 to Fig. 9 were described (for example, with reference to the image generation device 100). Redundant descriptions, which are assigned to the phase difference calculator 3353 and the depth map generator 3354, are omitted below for the sake of brevity.

[0119] The lens position controller 3355 can generate a lens driver signal LD. This lens driver signal LD ​​can be used to move the lens 3310. In response to the lens driver signal LD, the lens 3310 can move in a direction that increases or decreases its distance from the object. Accordingly, the distance between the lens 3310 and the object can be adjusted. Unlike in Fig. However, the lens position controller 3355 cannot directly control the position of the lens 3310. For example, the lens position controller 3355 can control the position of the lens 3310 by controlling another component (for example, the lens driver 2332 of the Fig. 10) control indirectly.

[0120] In an exemplary embodiment, the lens position controller 3355 can calculate an in-focus position of the lens 3310. As described above, the in-focus position can be the position of the lens 3310 for focusing an object. The lens position controller 3355 can calculate the in-focus position based on the first phase differences PD1 and / or the second phase differences PD2. For example, the lens position controller 3355 can calculate and determine a position of the lens 3310 in which one phase difference is zero as the in-focus position. When the lens 3310 needs to move to the in-focus position, the lens position controller 3355 can generate the lens driver signal LD ​​to move the lens 3310 to the in-focus position.

[0121] However, unlike the descriptions above, in at least one other exemplary embodiment of the inventive concepts, the in-focus position can be calculated by the image sensor chip 3330. If the image sensor chip 3330 has a separate lens position controller, the image sensor chip 3330 can generate a lens driver signal to move the lens 3310 to the calculated in-focus position. At least one exemplary embodiment of the inventive concepts can be changed to or modified to a different configuration from that described in Fig. 11 is shown.

[0122] According to at least some exemplary embodiments of the inventive concepts, the image signal processor 3350 can perform various types of image signal processing on a depth map DM generated by the depth map generator 3354. Accordingly, a more accurately processed depth map DM can be generated. In other words, it illustrates Fig. 11 merely provides an exemplary configuration to facilitate an understanding of at least some exemplary embodiments of the inventive concepts and is not intended to limit at least some exemplary embodiments of the inventive concepts. At least one exemplary embodiment of the inventive concepts may be implemented with a different configuration from that shown in Fig. As shown in 11, it should be implemented.

[0123] According to an exemplary embodiment, which is described in Fig. As shown in Figure 11, the phase difference calculator 3353 and the depth map generator 3354 can be implemented in the image signal processor 3350. For example, the image signal processor 3350 can be implemented in a surgical processing device which includes an application processor. According to at least one exemplary embodiment of the inventive concepts, the phase difference calculator 3353, the depth map generator 3354, and the lens position controller 3355 can be implemented in a surgical processing device which includes an application processor.

[0124] However, at least some exemplary embodiments of the inventive concepts are not limited to the configurations described in Fig. The image-generating device 3300 can also include other components shown in 11. Fig. 11 not shown, exhibit, or may contain one or more components which are in Fig. 11 are shown, do not exhibit. Fig. Figure 11 merely illustrates an exemplary configuration of the image generation device 3300.

[0125] Fig. Figure 12 is a block diagram illustrating an image-generating device according to at least one exemplary embodiment of the inventive concepts. Referring to Fig. 12 An image generating device 4300 can include a lens 4310, an image sensor chip 4330, an image signal processor 4350, a phase difference calculator 4373 and a depth map generator 4374.

[0126] The configurations and functions of lens 4310 may differ from those of lens 1310, as described in the following. Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, Fig. 7, Fig. 8 to Fig. 9. Redundant descriptions assigned to lens 4310 are omitted below for brevity. Light passing through lens 3310 may be directed to image sensor chip 4330.

[0127] The 4330 image sensor chip can replace an 1331 image sensor (refer to the Fig. 2, Fig. 3, Fig. 4, Fig. 5 to Fig. 6) exhibit. The image sensor 1331 can have a plurality of image sensor pixels and a plurality of phase detection pixels. The plurality of image sensor pixels can generate image signals corresponding to an object. The plurality of phase detection pixels can generate first phase signals PS1 and second phase signals PS2, which are used to calculate phase differences between images. The first phase signals PS1 can be generated when the lens 2310 is placed in a first position, and the second phase signals PS2 can be generated when the lens 2310 is placed in a second position.

[0128] The configurations and functions of the 1331 image sensor were developed with reference to the Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, Fig. 7, Fig. 8 to Fig. 9 described.

[0129] The 4350 image signal processor can perform image signal processing to generate a suitable image of an object. The configurations and functions of the 4350 image signal processor may differ from those of the 1350 image signal processor. Fig. 1. Redundant descriptions assigned to the 4350 image signal processor may be omitted below for brevity.

[0130] The phase difference calculator 4373 can receive the first phase signals PS1 and the second phase signals PS2 from the image sensor chip 4330. The phase difference calculator 4373 can calculate first phase differences PD1 and second phase differences PD2 based on these first and second phase signals, respectively. The depth map generator 4374 can generate a depth map DM based on the first phase differences PD1 and the second phase difference PD2.

[0131] The configurations and functions of the phase difference calculator 4373 and the depth map generator 4374 may have those of the phase difference calculator 103 and the depth map generator 104, which, with reference to the Fig. 7, Fig. 8 to Fig. 9 (for example, with reference to the image generation device 100). Redundant descriptions, which are assigned to the phase difference calculator 4373 and the depth map generator 4374, are omitted below for the sake of brevity.

[0132] In one exemplary embodiment, the image signal processor 4350 can perform image signal processing on the depth map DM, which is generated by the depth map generator 4374. Consequently, the image signal processor 4350 can generate a depth map DM' that is processed more accurately. In at least one other exemplary embodiment of the inventive concepts, however, the depth map DM, which is generated by the depth map generator 4374, can be output directly by the image generating device 4300 without image signal processing.

[0133] According to an exemplary embodiment, which is described in Fig. As shown in Figure 12, the phase difference calculator 4373 and the depth map generator 4374 can be provided separately from the image sensor chip 4330 and the image signal processor 4350. Alternatively, unlike in the Fig. 10 and Fig. The phase difference calculator 103 and the depth map generator 104 are implemented in such a way as to be distributed across the image sensor chip 1330 and the image signal processor 1350. According to at least some exemplary embodiments of the inventive concepts, the phase difference calculator 103 and the depth map generator 104 can be implemented in one of several configurations.

[0134] However, at least some exemplary embodiments of the inventive concepts are not limited to the configurations described in Fig. The image-generating device 4300 can also incorporate other components, which are described in 12. Fig. 12 not shown, may contain or may contain one or more components which are in Fig. 12 are shown, do not exhibit any of them. Fig. Figure 12 illustrates one of the possible exemplary configurations of the image generation device 4300.

[0135] As described above, the image generation device according to at least one exemplary embodiment of the inventive concepts can generate a depth map simultaneously with the acquisition of an object. An image generation device according to at least another exemplary embodiment of the inventive concepts can directly access a memory device or a storage device via a direct memory access (DMA) operation. The memory device or storage device can store pre-generated phase information. In this exemplary embodiment, the image generation device can generate a depth map based on the phase information stored in the memory device or storage device.In this exemplary embodiment, the image-generating device may lack a lens and some of the functions of an image sensor chip. At least one exemplary embodiment of the inventive concepts can be changed or modified to one of several different configurations.

[0136] Fig. Figure 13 is a block diagram illustrating an image-generating device according to at least one exemplary embodiment of the inventive concepts. Referring to the Fig. 13 An image generating device 200 can include a phase difference calculator 203, a depth map generator 204 and a phase difference predictor 206.

[0137] According to at least one exemplary embodiment of the inventive concepts, the image-generating device 200 may comprise or be implemented by one or more circuits (for example, hardware) specifically structured to perform some or all of the operations described herein as being performed by the image-generating device 200 (or an element thereof). According to at least one exemplary embodiment of the inventive concepts, the image-generating device 200 may comprise or be implemented by a memory and one or more processors executing computer-readable code (for example, software) stored in the memory, which includes instructions corresponding to some or all of the operations described herein as being performed by the image-generating device 200 (or an element thereof).According to at least one exemplary embodiment of the inventive concepts, the image generating device 200 can be implemented by, for example, a combination of the hardware and processors described above which execute computer-readable code.

[0138] The phase difference calculator 203, the depth map generator 204 and the phase difference predictor 206 can be integrated into an image sensor chip 1330 (reference is made to Fig. 1) and an image signal processor 1350 (reference is made to Fig. 1) be implemented, or may be implemented, to be distributed across the image sensor chip 1330 and the image signal processor 1350. Alternatively, the phase difference calculator 203, the depth map generator 204, and the phase difference predictor 206 may be provided separately from the image sensor chip 1330 and the image signal processor 1350.

[0139] The phase difference calculator 203 can receive first phase signals PS1 and second phase signals PS2. Based on these signals, the phase difference calculator 203 can calculate first phase differences PD1 and second phase differences PD2. The depth map generator 204 can generate a depth map DM based on the first phase differences PD1 and the second phase differences PD2.

[0140] The configurations and functions of the phase difference calculator 203 and the depth map generator 204 may have those of the phase difference calculator 103 and the depth map generator 104, which are based on the Fig. 7, Fig. 8 to Fig. 9 (for example, with reference to the image generation device 100). Redundant descriptions, which are assigned to the phase difference calculator 203 and the depth map generator 204, are omitted below for the sake of brevity.

[0141] The Phase Difference Predictor 206 can predict values, which are to be calculated as the second phase differences PD2, based on the first phase differences PD1. The Phase Difference Predictor 206 can provide the predicted values ​​PV to the Depth Map Generator 204. The Depth Map Generator 204 can generate a depth map DM with reference to the predicted values ​​PV together with the first phase differences PD1 and the second phase differences PD2.

[0142] The first phase differences PD1 can be calculated if a lens 1310 (refer to) Fig. 1) is placed in a first position. Based on the first phase differences PD1, recording environments such as an in-focus position of the lens 1310, a distance between an object and an image sensor 1331 (refer to the Fig. 2, Fig. 3, Fig. 4, Fig. 5 to Fig. 6) and a distance between the lens 1310 and the image sensor 1331 is obtained. Once the shooting conditions are established, the values ​​to be calculated as the second phase differences PD2 when the lens 1310 moves to a second position can be predicted.

[0143] For example, when lens 1310 moves from the first position to the second position, the first phase differences PD1 can be calculated earlier than the second phase differences PD2. The phase difference predictor 206 can predict the values ​​to be calculated as the second phase differences PD2 before lens 1310 moves to the second position. Alternatively, the phase difference predictor 206 can predict the values ​​to be calculated as the second phase differences PD2 while lens 1310 is moving to the second position. This example is included here only to facilitate the understanding of at least some exemplary embodiments of the inventive concepts and is not intended to limit at least some exemplary embodiments of the inventive concepts.

[0144] In some cases, the PV values ​​predicted by the Phase Difference Predictor 206 may differ from the second phase differences PD2 currently being calculated by the Phase Difference Calculator 203. For example, if at least one of the first phase differences PD1 and the second phase difference PD2 contains an error, the predicted PV values ​​may differ from the second phase difference PD2. Therefore, if the predicted PV values ​​differ from the second phase difference PD2, it is necessary to correct the error.

[0145] According to at least one exemplary embodiment of the inventive concepts, if the difference between the predicted values ​​PV and the second phase differences PD2 is greater than a reference value, processing to correct an error can be performed. The reference value can be a fixed value or an adjustable value. The reference value can be selected differently in each exemplary embodiment. According to at least another exemplary embodiment of the inventive concepts, if the predicted values ​​PV differ from the second phase differences PD2, processing to correct an error can be performed.

[0146] A process for introducing an error can be carried out in various ways. According to at least one exemplary embodiment of the inventive concepts, the depth map generator 204 can generate a depth map DM with reference to a difference between the predicted values ​​PV and the second phase differences PD2 in order to correct an error. For example, the depth map generator 204 can generate the depth map DM based on an average of the predicted values ​​PV and the second phase differences PD2.

[0147] As another example, the depth map generator 204 can collect information associated with a position of lens 1310, which enables more accurate calculation of phase differences. For instance, the depth map generator 204 can collect this information periodically during repeated exposures or whenever a specific condition is met. Based on the collected information, the depth map generator 204 can determine the reliability of each of the first phase differences PD1 and the second phase differences PD2. The depth map generator 204 can assign a higher weight to phase differences that have greater reliability among the first phase difference PD1 and the second phase differences PD2, and can calculate a weighted average of these two values. The depth map generator 204 can then generate the depth map DM based on this calculated weighted average.

[0148] According to at least one other exemplary embodiment of the inventive concepts, in order to correct an error, the phase difference calculator 203 can further calculate other phase differences. For example, the lens 1310 can move to a third position, which is different from the first and second positions. When the lens 1310 is in the third position, a plurality of phase detection pixels of the image sensor 1331 can generate third phase signals. The phase difference calculator 203 can calculate third phase differences based on these third phase signals. The depth map generator 204 can generate the depth map DM based on the first phase differences PD1, the second phase differences PD2, and the third phase differences.

[0149] Furthermore, if the third phase differences are calculated, it can be determined which of the first phase differences PD1 and the second phase differences PD2 have higher reliability. For example, if the third phase differences are more similar to the first phase differences PD1 than the second phase differences PD2, it can be considered that the first phase differences PD1 have higher reliability than the second phase differences PD2. In this example, the depth map generator 204 can assign higher reliability to the first phase differences PD1 and the third phase differences and can calculate a weighted average of the first phase differences PD1, the second phase differences PD2, and the third phase differences. The depth map generator 204 can then generate the depth map DM based on the calculated weighted average.In some other exemplary embodiments, the phase difference calculator 203 can also calculate phase differences other than the third phase differences.

[0150] The exemplary embodiments described above are provided here only to facilitate an understanding of at least some exemplary embodiments of the inventive concepts. Conditions and processing for correcting a defect can be changed or modified in various ways.

[0151] Fig. Figure 14 is a block diagram illustrating an image-generating device according to at least one exemplary embodiment of the inventive concepts. Referring to Fig. 14 An image generating device 300 can include a phase difference calculator 303, a depth map generator 304 and a spatial frequency calculator 307.

[0152] According to at least one exemplary embodiment of the inventive concepts, the image-generating device 300 may comprise or be implemented by one or more circuits (for example, hardware) specifically structured to perform some or all of the operations described herein as being performed by the image-generating device 300 (or an element thereof). According to at least one exemplary embodiment of the inventive concepts, the image-generating device 300 may comprise or be implemented by a memory and one or more processors executing computer-readable code (for example, software) stored in the memory, which includes instructions corresponding to some or all of the operations described herein as being performed by the image-generating device 300 (or an element thereof).According to at least one exemplary embodiment of the inventive concepts, the image generating device 300 can be implemented by, for example, a combination of the hardware and processors described above which execute computer-readable code.

[0153] The phase difference calculator 303, the depth map generator 304 and the spatial frequency calculator 307 can be integrated into an image sensor chip 1330 (reference is made to Fig. 1) and an image signal processor 1350 (reference is made to Fig. 1) be implemented, or can be implemented, to be distributed across the image sensor chip 1330 and the image signal processor 1350. Alternatively, the phase difference calculator 303, the depth map generator 304, and the spatial frequency calculator 307 can be provided separately from the image sensor chip 1330 and the image signal processor 1350.

[0154] The phase difference calculator 303 can receive first phase signals PS1 and second phase signals PS2. Based on these signals, the phase difference calculator 303 can calculate first phase differences PD1 and second phase differences PD2. The depth map generator 304 can generate a depth map DM based on the first phase differences PD1 and the second phase differences PD2.

[0155] The configurations and functions of the phase difference calculator 303 and the depth map generator 304 may have those of the phase difference calculator 103 and the depth map generator 104, which are based on the Fig. 7, Fig. 8 to Fig. 9 (for example, with reference to the image generation device 100). Redundant descriptions, which are assigned to the phase difference calculator 303 and the depth map generator 304, are omitted below for the sake of brevity.

[0156] According to at least one exemplary embodiment of the inventive concepts, the spatial frequency calculator 307 can receive an image signal IS. The image signal IS can be generated by a plurality of image sensor pixels located in an image sensor 1331 (reference is made to the Fig. 2, Fig. 3, Fig. 4, Fig. 5 to Fig. 6) is included. As described above, the image signals IS can be used to generate an image of an object. The spatial frequency calculator 307 can generate information about a spatial frequency associated with an image where an object was recorded by processing the image signal IS.

[0157] The spatial frequency of an image can be associated with whether the object being captured to create the image is in focus. Spatial frequency components of an image captured when an object is out of focus may be concentrated in a low-frequency range. Conversely, spatial frequency components of an image captured when the object is in focus may be distributed evenly from a low-frequency range to a high-frequency range.

[0158] According to at least one exemplary embodiment of the inventive concepts, the spatial frequency calculator 307 can generate the first spatial frequency information when a lens 1310 (reference is made to Fig. 1) is placed in a first position. The spatial frequency calculator 307 can generate second spatial frequency information when the lens 1310 is placed in a second position. If the position of the lens 1310 is changed from the first position to the second position, the first spatial frequency information may differ from the second spatial frequency information.

[0159] The spatial frequency calculator 307 can determine the direction in which a spatial frequency value changes (i.e., an increase or decrease in the amount of spatial frequency components of a high-frequency range) when the lens 1310 moves from the first position to the second position, based on the first spatial frequency information and the second spatial frequency information. For example, a spatial frequency value can be considered to change in a positive direction when the amount of spatial frequency components of a high-frequency range increases.

[0160] The spatial frequency calculator 307 can determine the amount by which a spatial frequency value changes when the lens 1310 moves from the first position to the second position, based on the first and second spatial frequency information. The spatial frequency calculator 307 can supply the depth map generator 304 with spatial frequency information SF, including the first spatial frequency information, the second spatial frequency information, the direction in which the spatial frequency changes, and the amount by which the spatial frequency changes, etc.

[0161] The depth map generator 304 can generate the depth map DM with reference to the spatial frequency information SF, along with the first phase differences PD1 and the second phase differences PD2. Specifically, the depth map generator 304 can specify the direction in which the spatial frequency value is changed and / or the extent to which the spatial frequency value is changed. The spatial frequency can be used to determine the reliability of each of the first phase differences PD1 and the second phase differences PD2. The use of the spatial frequency will be further described with reference to the Fig. 15 and Fig. 16.

[0162] Fig. Figure 15 is a block diagram illustrating an image-generating device according to at least one exemplary embodiment of the inventive concepts. Referring to Fig. 15 An image generating device 400 can include a phase difference calculator 403, a depth map generator 404, a spatial frequency calculator 407 and a reliability level calculator 408.

[0163] According to at least one exemplary embodiment of the inventive concepts, the image-generating device 400 may comprise or be implemented by one or more circuits (for example, hardware) specifically structured to perform some or all of the operations described herein as being performed by the image-generating device 400 (or an element thereof). According to at least one exemplary embodiment of the inventive concepts, the image-generating device 400 may comprise or be implemented by a memory and one or more processors executing computer-readable code (for example, software) stored in the memory, which includes instructions corresponding to some or all of the operations described herein as being performed by the image-generating device 400 (or an element thereof).According to at least one exemplary embodiment of the inventive concepts, the image generating device 400 can be implemented by, for example, a combination of the hardware and processors described above which execute computer-readable code.

[0164] The phase difference calculator 403, the depth map generator 404, the spatial frequency calculator 407 and the reliability level calculator 408 can be integrated into an image sensor chip 1330 (reference is made to Fig. 1) and an image signal processor 1350 (reference is made to Fig. 1) may be implemented, or may be implemented, to be distributed across the image sensor chip 1330 and the image signal processor 1350. Alternatively, the phase difference calculator 403, the depth map generator 404, the spatial frequency calculator 407, and the reliability level calculator 408 may be provided separately from the image sensor chip 1330 and the image signal processor 1350.

[0165] The phase difference calculator 403 can receive first phase signals PS1 and second phase signals PS2. Based on these signals, the phase difference calculator 403 can calculate first phase differences PD1 and second phase differences PD2. The depth map generator 404 can generate a depth map DM based on the first phase differences PD1 and the second phase differences PD2.

[0166] The configurations and functions of the phase difference calculator 403 and the depth map generator 404 may have those of the phase difference calculator 103 and the depth map generator 104, which are based on the Fig. 7, Fig. 8 to Fig. 9 (for example, with reference to the image generation device 100). Redundant descriptions relating to the phase difference calculator 403 and the depth map generator 404 are omitted below for the sake of brevity.

[0167] The spatial frequency calculator 407 can receive an image signal IS, which is generated by a plurality of image sensor pixels located in an image sensor 1331 (reference is made to the Fig. 2, Fig. 3, Fig. 4, Fig. 5 to Fig. 6) are contained, is generated. The spatial frequency calculator 407 can generate spatial frequency information SF by processing the image signal IS. The spatial frequency calculator 407 can provide the spatial frequency information SF for the depth map generator 404.

[0168] For example, the spatial frequency calculator 407 can process initial spatial frequency information associated with a first image where an object is recorded by processing initial image signals generated when a lens 1310 (refer to) Fig. 1) is placed in a first position. The spatial frequency calculator 407 can generate second spatial frequency information, which is associated with a second image on which an object is recorded, by processing second image signals that are generated when the lens 1310 is placed in a second position. The spatial frequency calculator 407 can obtain a direction in which a spatial frequency is changed and / or a quantity where a spatial frequency is changed, based on the first spatial frequency information and the second spatial frequency information.

[0169] Configurations and functions of the spatial frequency calculator 407 may differ from those of the spatial frequency calculator 307, which are based on the Fig. 14 have been described. Redundant descriptions, which are assigned to the spatial frequency calculator 40, are omitted below for the sake of brevity.

[0170] The Reliability Level Calculator 408 can calculate a first reliability level, which is assigned to the first phase difference (PD1). The Reliability Level Calculator 408 can calculate a second reliability level, which is assigned to the second phase difference (PD2). As described above, one or both of the first phase difference (PD1) and the second phase difference (PD2) can contain a fault. The Reliability Level Calculator 408 can calculate reliability levels (RL), which are assigned to faults contained in the first phase difference (PD1) and the second phase difference (PD2). For example, the lower the level of a fault, the higher the reliability level.

[0171] According to at least one exemplary embodiment of the inventive concepts, the reliability level calculator 408 can calculate reliability levels RL based on the direction in which a spatial frequency value is changed. For example, if a position of the lens 1310 moves closer to an in-focus position in accordance with a movement of the lens 1310 from the first position to the second position, the amount of spatial frequency components of a high-frequency range may increase. However, if one or both of the first phase differences PD1 and the second phase differences PD2 have an error, it may appear that the amount of spatial frequency components of a high-frequency range decreases, even though a position of the lens 1310 moves closer to the in-focus position. In this case, the first phase differences PD1 or the second phase differences PD2 may have a low reliability level.

[0172] According to at least one exemplary embodiment of the inventive concepts, the reliability level calculator 408 can collect information about the position of the lens 1310, which causes phase differences to be calculated more accurately. For example, the reliability level calculator 408 can collect information periodically during a repeated recording, or whenever a certain condition is met. The reliability level calculator 408 can select phase differences from among the first phase differences PD1 and the second phase differences PD2 to achieve a low or high reliability level.

[0173] As another example, if the lens 1310 is placed in a third position, which differs from the first and second positions, third phase differences can still be generated. For example, the reliability level calculator 408 can select phase differences to achieve a low reliability level by comparing the first phase difference PD1, the second phase difference PD2, and the third phase difference with each other. Additionally, the reliability level calculator 408 can determine a high reliability level for phase differences that are not selected.

[0174] According to at least one exemplary embodiment of the inventive concepts, the depth map generator 404 can generate the depth map DM by reflecting the first reliability level and the second reliability level onto first depth data generated based on the first phase differences PD1 and second depth data generated based on the second phase differences PD2. For example, the depth map generator 404 can assign a higher weight to phase differences with a higher reliability level and can calculate a weighted average. The depth map generator 404 can then generate a depth map DM based on the calculated weighted average.

[0175] Fig. Figure 16 is a block diagram illustrating an image-generating device according to at least one exemplary embodiment of the inventive concepts. Referring to Fig. 16 An image generating device 500 can have a phase difference calculator 503, a depth map generator 504, a phase difference predictor 506, a spatial frequency calculator 507 and a reliability level calculator 508.

[0176] According to at least one exemplary embodiment of the inventive concepts, the image-generating device 500 may comprise or be implemented by one or more circuits (for example, hardware) specifically structured to perform some or all of the operations described herein as being performed by the image-generating device 500 (or an element thereof). According to at least one exemplary embodiment of the inventive concepts, the image-generating device 500 may comprise or be implemented by a memory and one or more processors executing computer-readable code (for example, software) stored in the memory, which includes instructions corresponding to some or all of the operations described herein as being performed by the image-generating device 500 (or an element thereof).According to at least one exemplary embodiment of the inventive concepts, the image generating device 500 can be implemented by, for example, a combination of the hardware and processors described above which execute computer-readable code.

[0177] The phase difference calculator 503, the depth map generator 504, the phase difference predictor 506, the spatial frequency calculator 507 and the reliability level calculator 508 can be integrated into an image sensor chip 1330 (reference is made to Fig. 1) and an image signal processor 1350 (reference is made to Fig. 1) may be implemented, or be implemented, to be distributed across the image sensor chip 1330 and the image signal processor 1350. Alternatively, the phase difference calculator 503, the depth map generator 504, the phase difference predictor 506, the spatial frequency calculator 507, and the reliability level calculator 508 may be provided separately from the image sensor chip 1330 and the image signal processor 1350.

[0178] The Phase Difference Calculator 503 can receive first phase signals PS1 and second phase signals PS2. Based on these signals, the Phase Difference Calculator 503 can calculate first phase differences PD1 and second phase differences PD2. The Depth Map Generator 504 can generate a depth map DM based on the first phase differences PD1 and the second phase differences PD2.

[0179] The configurations and functions of the phase difference calculator 503 and the depth map generator 504 may have those of the phase difference calculator 103 and the depth map generator 104, which are based on the Fig. 7, Fig. 8 to Fig. 9 (for example, with reference to the image generation device 100). Redundant descriptions relating to the phase difference calculator 503 and the depth map generator 504 are omitted below for the sake of brevity.

[0180] The Phase Difference Predictor 506 can predict values, which are to be calculated as the second phase differences PD2, based on the first phase differences PD1. The Phase Difference Predictor 506 can provide the predicted values ​​PV for the Depth Map Generator 504. Configurations and functions of the Phase Difference Predictor 506 may differ from those of a Phase Difference Predictor 206. Fig. 13. Redundant descriptions, which are assigned to the phase difference predictor 506, are omitted below for the sake of brevity.

[0181] The spatial frequency calculator 507 can receive an image signal IS. The image signal IS can be generated by a plurality of image sensor pixels, which are located in an image sensor 1331 (reference is made to the Fig. 2, Fig. 3, Fig. 4, Fig. 5 to Fig. 6) are contained within. The Spatial Frequency Calculator 507 can generate spatial frequency information (SF) associated with an image containing an object by processing the image signal (IS). Configurations and functions of the Spatial Frequency Calculator 507 may differ from those of a Spatial Frequency Calculator 307. Fig. 14 and / or those of a spatial frequency calculator 407 of the Fig. 15. Redundant descriptions, which are assigned to the spatial frequency calculator 507, are omitted below for the sake of brevity.

[0182] The reliability level calculator 508 can calculate a first reliability level associated with the first phase differences PD1. The reliability level calculator 508 can calculate a second reliability level associated with the second phase differences PD2. By means of at least one exemplary embodiment of the inventive concepts, the reliability level calculator 508 can calculate reliability levels RL based on at least one of the values ​​PV predicted by the phase difference predictor 506, the first phase differences PD1, the second phase differences PD2, the spatial frequency information SF, and any combination thereof. Configurations and functions of the reliability level calculator 508 can be those of a reliability level calculator 408. Fig. 15. Redundant descriptions, which are assigned to the reliability level calculator 508, are omitted below for the sake of brevity.

[0183] There must be consistency between a direction of movement of a lens 1310 (referring to Fig. 1) and a change in the spatial frequency. Additionally, it is desirable that the first phase differences PD1 and the second phase differences PD2 contain no error, and that the values ​​PV predicted by the phase difference predictor 506 are identical to the second phase differences PD2 currently calculated by the phase difference calculator 503. If there is no consistency between the direction of movement of the lens 1310 and the change in the spatial frequency, or if the predicted values ​​PV differ from the second phase differences PD2, the reliability level calculator 508 can calculate reliability level RL with reference to the inconsistency and the error.

[0184] The depth map generator 504 can generate the depth map DM based on the reliability levels RL together with the first phase differences PD1 and the second phase differences PD2. In an exemplary embodiment, the depth map generator 504 can generate the depth map DM by reflecting the first reliability level and the second reliability level onto first depth data generated based on the first phase differences PD1 and second depth data generated based on the second phase differences PD2. This exemplary embodiment was developed with reference to Fig. 15 described.

[0185] Each of the phase difference calculator 503, depth map generator 504, phase difference predictor 506, spatial frequency calculator 507, and reliability level calculator 508 can be implemented using hardware such as an analog circuit and a logic circuit. Alternatively, functions of the phase difference calculator 503, depth map generator 504, phase difference predictor 506, spatial frequency calculator 507, and reliability level calculator 508 can be implemented using software.

[0186] The image generation device according to any exemplary embodiment of, or alternatively at least some exemplary embodiments of, the inventive concepts can occupy a narrow area or size. Additionally, according to any exemplary embodiment of, or alternatively at least some exemplary embodiments of, the image generation device can generate a depth map with higher reliability by correcting an error with respect to multiple depth data points.

[0187] Fig. Figure 17 is a block diagram illustrating an image-generating device according to at least one exemplary embodiment of the inventive concepts. Referring to Fig. 17 An image generating device 600 can include a phase difference calculator 603, a depth map generator 604 and a depth map post-processor 609.

[0188] According to at least one exemplary embodiment of the inventive concepts, the image-generating device 600 may comprise or be implemented by one or more circuits (for example, hardware) specifically structured to perform some or all of the operations described herein as being performed by the image-generating device 600 (or an element thereof). According to at least one exemplary embodiment of the inventive concepts, the image-generating device 600 may comprise or be implemented by a memory and one or more processors executing computer-readable code (for example, software) stored in the memory, which includes instructions corresponding to some or all of the operations described herein as being performed by the image-generating device 600 (or an element thereof).According to at least one exemplary embodiment of the inventive concepts, the image generating device 600 can be implemented by, for example, a combination of the hardware and processors described above which execute computer-readable code.

[0189] The phase difference calculator 603, the depth map generator 604 and the depth map post-processor 609 can be integrated into an image sensor chip 1330 (reference is made to Fig. 1) and an image signal processor 1350 (reference is made to Fig. 1) may be implemented, or be implemented, to be distributed across the image sensor chip 1330 and the image signal processor 1350. Alternatively, the phase difference calculator 603, the depth map generator 604, and the depth map post-processor 609 may be provided separately from the image sensor chip 1330 and the image signal processor 1350.

[0190] The Phase Difference Calculator 603 can receive first phase signals PS1 and second phase signals PS2. Based on these signals, the Phase Difference Calculator 603 can calculate first phase differences PD1 and second phase differences PD2. The Depth Map Generator 604 can generate a depth map DM based on the first phase differences PD1 and the second phase differences PD2.

[0191] The configurations and functions of the phase difference calculator 603 and the depth map generator 604 may have those of the phase difference calculator 103 and the depth map generator 104, which are described in the Fig. 7, Fig. 8 to Fig. 9 (for example, with reference to the image generation device 100). Redundant descriptions relating to the phase difference calculator 603 and the depth map generator 604 are omitted below for the sake of brevity.

[0192] The depth map post-processor 609 can change the resolution of the depth map DM. According to at least one exemplary embodiment of the inventive concepts, the depth map post-processor 609 can perform image registration on an object image and the depth map DM. The object image can be an image generated based on image signals produced by a plurality of image sensor pixels located in an image sensor 1331 (reference is made to the Fig. 2, Fig. 3, Fig. 4, Fig. 5 to Fig. 6) are included. The Depth Map Post Processor 609 can generate a depth map DM', which has a modified resolution, by performing an image registration.

[0193] For example, as in Fig. As shown in Figure 4, when multiple phase-detection pixels are configured to avoid overlap with multiple image sensor chips, the depth map (DM) will have a lower resolution than the object image. The Depth Map Post-Processor 609 can estimate depth data corresponding to an image region where depth data is not obtained, with reference to the object image. For example, the Depth Map Post-Processor 609 can estimate the depth data corresponding to the image region where depth data is not obtained by extracting edge components contained in the object image and performing image registration on or at the edge components and the Depth Map DM.

[0194] In the example described above, the depth map post-processor 609 can generate the depth map DM', which has a modified resolution, based on the depth map DM and the estimated depth data. For example, the depth map DM' can have the same resolution as that of the object image by increasing the resolution of the depth map DM.

[0195] However, at least some exemplary embodiments of the inventive concepts are not limited to the example above. The depth map post-processor 609 can change the resolution of the depth map DM in various ways. The depth map DM' can have a different resolution than that of the object image. The resolution of the depth map DM can be increased or decreased. Additionally, as shown in Fig. Figure 3 shows that when a pixel unit is used as a phase detection pixel as well as an image sensor pixel, the depth map post-processor 609 can be used to change the resolution of the depth map DM.

[0196] For example, the depth map post-processor 609 can be configured to operate when a specific condition is met. In this example, the image generator 600 can further include determination logic or circuitry used to determine whether the depth map post-processor 609 will operate. By another example, the image generator 600 can further include an additional control signal line or an additional user interface. The image generator 600 can receive a command to operate the depth map post-processor 609 via the additional control line or the additional user interface.

[0197] Fig. Figure 18 is a block diagram illustrating an electronic device comprising an image-generating device according to at least one exemplary embodiment of the inventive concepts and interfaces thereof. An electronic system 5000 can be implemented with a data processing device that can employ or support an interface proposed by a Mobile Industrial Processor Interface (MIPI) Alliance. For example, the electronic system 5000 can be implemented with an electronic device such as a portable communications data terminal, a personal digital assistant (PDA), a portable media player (PMP), a smartphone, or a portable device.

[0198] The electronic system 5000 can include an application processor 5100, a display 5220, and an image sensor 5230. The application processor 5100 can include a DigRF maser 5110, a serial display interface (DSI) host 5120, a serial camera interface (CSI) host 5130, a physical layer (PHY) 5140, and an image signal processor (ISP) 5150.

[0199] The DSI host 5120 can communicate with a DSI device 5225 of the display 5220 in accordance with DSI. For example, an optical serializer (SER) can be implemented in the DSI host 5120. For example, an optical deserializer (DES) can be implemented in the DSI device 5225.

[0200] The CSI host 5130 can communicate with a CSI device 5235 of the image sensor 5230 in accordance with CSI. For example, an optical DES can be implemented in the CSI host 5130. For example, an optical SER can be implemented in the CSI device 5230. The ISP 5150 can communicate with the CSI host 5130 via a memory (for example, a working memory 5250 or embedded memory of the application processor 5110) and a bus.

[0201] At least one of the ISP 5150, the image sensor 5230, and any combination thereof can be configured according to at least one of at least some exemplary embodiments of the inventive concepts, which are described with reference to the Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, Fig. 7, Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12, Fig. 13, Fig. 14, Fig. 15, Fig. 16 to Fig. 17 were described. For example, as in Fig. As described in section 10, the 5230 image sensor can move a lens to each of two or more positions, generate multiple depth data points using a plurality of phase-detection pixels, and create a depth map based on these multiple depth data points. For example, as described in Fig. As described in Figure 11, the ISP 5150 generates a lens driver signal to move a lens to each of two or more positions and can generate a depth map based on multiple depth data points generated using a plurality of phase-detection pixels. Redundant descriptions of at least some exemplary embodiments of the inventive concepts are omitted below.

[0202] The electronic system 5000 can further include a radio frequency (RF) chip 5240, which communicates with the application processor 5100. The RF chip 5240 can include a PHY 5242, a DigRF slave 5244, and an antenna 5246. For example, the PHY 5242 of the RF chip 5240 and the PHY 5140 of the application processor 5100 can exchange data with each other via a DigRF interface proposed by the MIPI Alliance.

[0203] The electronic system 5000 may also include the working memory 5250 and an embedded / card memory 5255. The working memory 5250 and the embedded / card memory 5255 can store data provided by the application processor 5100. Additionally, the working memory 5250 and the embedded / card memory 5255 can provide the application processor 5100 with the data stored therein.

[0204] The 5250 main memory can temporarily store data that is being processed or is to be processed by the 5100 application processor. The 5250 main memory can be volatile memory, such as static random access memory (SRAM), dynamic RAM (DRAM), and synchronous DRAM (SDRAM), and / or non-volatile memory, such as flash memory, phase-change RAM (PRAM), magnetoresistive RAM (MRAM), resistive RAM (ReRAM), and ferroelectric RAM (FRAM). The 5255 embedded / card memory can store data independently of the power supply.

[0205] The electronic system 5000 can communicate with an external system via a communication module such as at least one Worldwide Interoperability for Microwave Access (WiMAX) 5260, one Wireless Local Area Network (WLAN) 5262, one Ultra-Wideband (UWB), and any combination thereof. The electronic system 5000 can also include a loudspeaker 5270 and a microphone 5275 for processing voice information. Additionally, the electronic system 5000 can include a global positioning system (GPS) device 5280 for processing position information. The electronic system 5000 can also include a bridge chip 5290 for managing a connection(s) with peripheral devices.

[0206] According to at least some exemplary embodiments of the inventive concepts, the area or size occupied by the image-generating device can be reduced. Additionally, according to at least some exemplary embodiments of the inventive concepts, the image-generating device can produce a depth map with higher reliability by correcting an error with reference to multiple depth data points.

[0207] A configuration illustrated in each concept diagram should be understood solely from a conceptual point of view. The shape, structure, and size of each component illustrated in each concept diagram are exaggerated or reduced to illustrate exemplary embodiments of the inventive concepts. A currently implemented configuration may have a physical shape different from any configuration shown in each concept diagram. At least some exemplary embodiments of the inventive concepts are not limited to a physical shape or size illustrated in each concept diagram.

[0208] A device configuration, illustrated in each block diagram, is provided to help understand at least some exemplary embodiments of the inventive concepts. Each block can be composed of smaller blocks according to functions. Alternatively, a plurality of blocks can form a larger block according to a function. This means that at least some exemplary embodiments of the inventive concepts are not limited to components illustrated in a block diagram.

Claims

[1] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) comprising the following: an image sensor (1331, 5230) which has the following features: a plurality of image sensor pixels configured to generate image signals corresponding to an object (1100), and a plurality of phase detection pixels configured to generate first and second phase signals (PS1, PS2) which are used to calculate a phase difference (PD1, PD2) between images; a lens driver (2332) which is configured to adjust the position of a lens (1310, 2310, 3310, 4310) in order to adjust the distance between the lens (1310, 2310, 3310, 4310) and the object (1100); a phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373) which is configured to to calculate first phase differences (PD1) based on the first phase signals (PS1), wherein the first phase signals (PS1) are generated when the lens (1310, 2310, 3310, 4310) is in a first position, and to calculate second phase differences (PD2) based on the second phase signals (PS2), wherein the second phase signals (PS1) are generated when the lens (1310, 2310, 3310, 4310) is in a second position that differs from the first position; and a depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374) which is configured to to generate first and second depth data (DD1, DD2) based on the first and second phase differences (PD1, PD2) respectively, wherein each of the first and second depth data (DD1, DD2) is assigned a distance between the majority of phase detection pixels and the object (1100), and to generate a depth map (DM) based on the first and second depth data (DD1, DD2). [2] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 1, further comprising: a phase difference predictor (206, 506) which is configured to predict values ​​of the second phase difference (PD2) based on the first phase difference (PD1). [3] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 2, wherein the phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373) is configured to calculate the first phase differences (PD1) before calculating the second phase differences (PD2) when the lens (1310, 2310, 3310, 4310) moves from the first position to the second position under control of the lens driver (2332), and wherein the phase difference predictor (206, 506) is configured to predict the values ​​of the second phase differences (PD2) before or while the lens (1310, 2310, 3310, 4310) moved to the second position. [4] Image generation device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 2, wherein the depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374) is configured such that when a difference between the values ​​predicted by the phase difference predictor (206, 506) and the second phase differences (PD2) calculated by the phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373) is greater than a reference value, the depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374) the depth map (DM) is generated with reference to the difference. [5] Image generation device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 1, wherein each of the plurality of phase detection pixels corresponds to two of the plurality of image sensor pixels. [6] Image generation device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 1, wherein the plurality of phase detection pixels are arranged in different positions from the plurality of image sensor pixels such that the plurality of phase detection pixels do not overlap with the plurality of image sensor pixels. [7] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 1, further comprising: a spatial frequency calculator (307, 407, 507) which is configured to generate information about a spatial frequency which is associated with an image in which the object (1100) is recorded by processing the image signals. [8] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 7, wherein the spatial frequency calculator (307, 407, 507) is configured to: to generate initial spatial frequency information when the lens (1310, 2310, 3310, 4310) is placed in the first position, and to generate second spatial frequency information when the lens (1310, 2310, 3310, 4310) is placed in the second position. [9] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 8, wherein the spatial frequency calculator (307, 407, 507) is further configured to obtain a direction and / or magnitude of a change in a spatial frequency value when the lens (1310, 2310, 3310, 4310) moves from the first position to the second position, based on the first and second spatial frequency information, and wherein the depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374) is configured to generate the depth map (DM) with reference to the direction and / or magnitude of the change in the spatial frequency value. [10] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 1, further comprising: an image sensor chip which includes the image sensor (1331, 5230), the lens driver (2332), the phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373) and the depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374). [11] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) comprising the following: a phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373) which is configured to to receive first and second phase signals (PS1, PS2) which are generated by a plurality of phase detection pixels contained in an image sensor (1331, 5230), to calculate first phase differences (PD1) based on the first phase signals (PS1), wherein the first phase signals (PS1) are generated when a lens (1310, 2310, 3310, 4310) is in a first position, wherein the lens (1310, 2310, 3310, 4310) is configured to move in a direction in which a distance from an object (1100) increases or decreases, and to calculate second phase differences (PD2) based on the second phase signals (PS2), wherein the second phase signals (PS2) are generated when the lens (1310, 2310, 3310, 4310) is in a second position which is different from the first position; a lens position controller (3355) which is configured to to calculate an in-focus position of the lens (1310, 2310, 3310, 4310) for focusing on the object (1100) based on the first and / or the second phase differences (PD1, PD2), and to generate a lens driver signal (LD) to move the lens (1310, 2310, 3310, 4310) to the in-focus position; and a depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374) which is configured to to generate first and second depth data (DD1, DD2) based on the first and second phase differences (PD1, PD2), wherein each of the first and second depth data (DD1, DD2) is assigned a distance between the majority of phase detection pixels and the object (1100), and to generate a depth map (DM) based on the first and second depth data (DD1, DD2). [12] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 11, wherein the first position or the second position corresponds to the in-focus position. [13] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 11, further comprising: a phase difference predictor (206, 506) configured to predict second phase difference (PD2) values ​​based on first phase differences (PD1); and a spatial frequency calculator (307, 407, 507) which is configured to generate information about a spatial frequency which is associated with an image in which the object (1100) is recorded by processing image signals which are generated by a plurality of image sensor pixels which are contained in the image sensor (1331, 5230). [14] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 13, further comprising: a reliability level calculator (408, 508) configured to calculate a first reliability level associated with the first phase differences (PD1) and a second reliability level associated with the second phase differences (PD2) based on the values ​​predicted by the phase difference predictor (206, 506) and / or the second phase differences (PD2) calculated by the phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373) and / or a direction of change in a spatial frequency value as the lens (1310, 2310, 3310, 4310) moves from the first position to the second position. [15] Image generation device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 14, wherein the depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374) is configured to generate the depth map (DM) by applying weighted values ​​to the first and second depth data (DD1, DD2) based on the first and second reliability levels. [16] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 11, further comprising: a depth map post-processor (609) configured to change the resolution of the depth map (DM) by performing image registration on an object image and the depth map (DM), wherein the object image is generated based on image signals produced by a plurality of image sensor pixels contained in the image sensor (1331, 5230). [17] Image generation device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 11, further comprising an operation processing device comprising an application processor, wherein the operation processing device is configured to implement the phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373), the lens position controller (3355) and the depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374). [18] Image generation device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) configured to generate a depth map (DM), wherein the image generation device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) comprises: a phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373) configured to calculate first and second phase differences (PD1, PD2) based on first and second phase signals (PS1, PS2), respectively, wherein the first phase signals (PS1) are generated by a plurality of phase detection pixels when a lens (1310, 2310, 3310, 4310) is in a first position, wherein the lens (1310, 2310, 3310, 4310) is configured to move in a direction in which a distance from an object (1100) increases or decreases, wherein the second phase signals (PS2) are generated by the plurality of phase detection pixels when the lens (1310, 2310, 3310, 4310) is in a second position which is different from the first position; and a depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374) which is configured to to generate first and second depth data (DD1, DD2) based on the first and second phase differences (PD1, PD2) respectively, wherein each of the first and second depth data (DD1, DD2) is assigned a distance between the majority of phase detection pixels and the object (1100), and to generate the depth map (DM) based on the first and second depth data (DD1, DD2). [19] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 18, further comprising: a phase difference predictor (206, 506) which is configured to predict values ​​of the second phase differences (PD2) based on the first phase differences (PD1), wherein the phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373) and the depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374) are configured such that if the values ​​predicted by the phase difference predictor (206, 506) are different from the second phase differences (PD2) calculated by the phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373), The phase difference calculator (103, 203, 303, 403, 503, 603, 2333, 3353, 4373) calculates third phase differences based on third phase signals, wherein the third phase signals are generated by the majority of phase detection pixels when the lens (1310, 2310, 3310, 4310) is in a third position that is different from the first and second positions; and The depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374) generates third depth data, which is assigned to a distance between the majority of phase detection pixels and the object (1100) based on the third phase differences, and generates the depth map (DM) based on the first to third depth data. [20] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 18, further comprising: a spatial frequency calculator (307, 407, 507) which is configured to to generate first spatial frequency information, which is assigned to a first image in which the object (1100) is recorded, by processing first image signals which are generated by a plurality of image sensor pixels when the lens (1310, 2310, 3310, 4310) is placed in the first position, and to generate second spatial frequency information, which is assigned to a second image in which the object (1100) is recorded, by processing second image signals which are generated by the majority of image sensor pixels when the lens (1310, 2310, 3310, 4310) is placed in the second position, and to obtain a direction of change in a spatial frequency value based on the first and second spatial frequency information; and a reliability level calculator (408, 508) which is configured to calculate a first reliability level associated with the first phase differences (PD1) and a second reliability level associated with the second phase differences (PD1) based on the direction in which the spatial frequency value is changed. wherein the depth map generator (104, 204, 304, 404, 504, 604, 2334, 3354, 4374) is configured to generate the depth map (DM) by applying weighted values ​​to the first and second depth data (DD1, DD2) based on the first and second reliability levels. [21] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) comprising the following: a lens; an image sensor (1331, 5230) which has the following features a plurality of image sensor pixels configured to generate image signals corresponding to an object (1100), and a plurality of phase detection pixels configured to generate first and second phase signals (PS1, PS2), wherein a position of the lens (1310, 2310, 3310, 4310) is movable with respect to a position of the image sensor (1331, 5230); a memory that stores computer-readable instructions; and one or more processors, configured to execute the instructions to to determine first phase differences (PD1) based on the first phase signals (PS1), wherein the first phase signals (PS1) are generated based on the lens (1310, 2310, 3310, 4310) being placed in a first position relative to the image sensor (1331, 5230), and to determine second phase differences (PD2) based on the second phase signals (PS2), wherein the second phase signals (PS2) are generated based on the lens (1310, 2310, 3310, 4310) being placed in a second position relative to the image sensor (1331, 5230), first and second depth data (DD1, DD2) are calculated based on the first and second phase differences (PD1, PD2) respectively, with each of the first and second depth data (DD1, DD2) indicating a distance between the majority of phase detection pixels and the object (1100), and to generate a depth map (DM) based on the first and second depth data (DD1, DD2). [22] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 21, further comprising: a lens driver (2332) which is configured to selectively change the position of the lens (1310, 2310, 3310, 4310) relative to the image sensor (1331, 5230). [23] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 22, wherein one or more processors are further configured to predict values ​​of the second phase differences (PD2) based on the first phase differences (PD1). [24] Image generating device (100, 200, 300, 400, 500, 600, 1300, 2300, 3300, 4300) according to claim 23, where one or more processors are configured to to calculate the first phase differences (PD1) before calculating the second phase differences (PD2) when the lens (1310, 2310, 3310, 4310) is moved from the first position to the second position under control of the lens driver (2332), and to predict the values ​​of the second phase difference (PD2) before or while the lens (1310, 2310, 3310, 4310) is moved to the second position under the control of the lens driver (2332).

Images