Imaging device and electronic device including the same

Abstract

An imaging device comprises: a light source that operates in response to an optical control signal having a predetermined frequency; a sensor comprising a plurality of pixels configured to generate an electrical signal in response to a light receiving signal output by the light source and reflected from an object, and a logic circuit configured to generate first raw data in a spatial domain based on the electrical signal; and an image signal processor configured to convert the first raw data into second raw data in a frequency domain, select low-frequency data of the second raw data within a frequency band lower than a predetermined reference frequency, apply predetermined weighting to the low-frequency data to generate weighted data, and generate a composite image based on the first raw data and the weighted data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of priority from Korean Patent Application No. 10-2020-0027643 filed on March 5, 2020 in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] Example embodiments of the present disclosure relate to an imaging device and an electronic device including the imaging device. Background Art

[0004] An imaging device converts an optical signal into an electrical signal to generate an image. An imaging device can be installed in an electronic device to provide a camera function. A depth map is an image containing distance information related to the distance of the surface of an object from a viewpoint. An imaging device can generate a depth map using an optical signal. However, when the optical signal includes noise, the depth map may include errors. Therefore, when the optical signal includes excessive noise, the quality of the image generated by the imaging device may be reduced. Summary of the invention

[0005] At least one exemplary embodiment of the inventive concept provides an imaging device that is not easily affected by optical noise and an electronic device including the imaging device, and an imaging device that can improve noise characteristics and improve the accuracy of a synthesized image.

[0006] According to an example embodiment of the present invention, an imaging device includes: a light source that operates in response to an optical control signal having a predetermined frequency; a sensor including a plurality of pixels configured to generate an electrical signal in response to a light receiving signal output by the light source and reflected from an object, and a logic circuit configured to generate first raw data in a spatial domain based on the electrical signal; and an image signal processor configured to convert the first raw data into second raw data in a frequency domain, select low-frequency data of the second raw data within a frequency band lower than a predetermined reference frequency, apply predetermined weighting to the low-frequency data to generate weighted data, and generate a composite image based on the first raw data and the weighted data.

[0007] According to an example embodiment of the inventive concept, an imaging device includes: a light source that operates in response to an optical control signal having a predetermined frequency; a sensor configured to generate raw data using a light receiving signal output by the light source and reflected from an object, the raw data including a first region having a depth less than a predetermined reference depth and a second region having a depth greater than the predetermined reference depth; and an image signal processor configured to convert the raw data of the second region into a frequency domain, select low-frequency data within the frequency domain, and apply weighting to the low-frequency data to generate a synthetic image.

[0008] According to an example embodiment of the present invention, an imaging device includes: a pixel array including a plurality of pixels, each pixel operating in response to a first photoelectric (photo) control signal and a second photoelectric control signal having different phases to output a first electrical signal and a second electrical signal; a logic circuit configured to generate raw data for generating a depth image using the first electrical signal and the second electrical signal; and an image signal processor configured to convert the raw data from a spatial domain to a frequency domain and reduce the size of low-frequency data in the frequency domain included in a low-frequency band below a predetermined reference frequency to generate a synthetic image.

[0009] According to an exemplary embodiment of the inventive concept, an electronic device includes: a sensor having a plurality of pixels for generating an electrical signal in response to an externally incident optical signal, and a logic circuit configured to generate raw data for generating a depth image based on the electrical signal; and a processor communicatively connected to the sensor through a predetermined interface. The processor converts the raw data into a frequency domain, selects low-frequency data in the frequency domain included in a low-frequency band lower than a predetermined reference frequency, and reduces the size of the low-frequency data to generate a depth image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present disclosure will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0011] Figure 1 and Figure 2 is a schematic block diagram of an imaging device according to an example embodiment of the inventive concept.

[0012] Figure 3 is a schematic block diagram of an imaging device according to an example embodiment of the inventive concept.

[0013] FIG. 4A to FIG. 4C The operation of the image forming apparatus according to an exemplary embodiment of the inventive concept is illustrated.

[0014] Figure 5A and Figure 5B The operation of the image forming apparatus according to an exemplary embodiment of the inventive concept is illustrated.

[0015] Figure 6 and Figure 7 The operation of the image forming apparatus according to an exemplary embodiment of the inventive concept is illustrated.

[0016] Figure 8 A method of generating depth information in an imaging device according to an example embodiment of the inventive concept is illustrated.

[0017] Fig. 9 is a schematic block diagram of an imaging device according to an example embodiment of the inventive concept.

[0018] Fig.10 is a flowchart illustrating an operation of an image forming apparatus according to an example embodiment of the inventive concept.

[0019] Fig.11A and Fig. 11B are exemplary images illustrating operations of an imaging apparatus according to an example embodiment of the inventive concept.

[0020] Fig. 12A and Fig. 12B is based on Fig. 11B An enlarged view of a portion of an image of an example embodiment is shown.

[0021] Fig.13 and Fig.14 Data in a frequency domain generated in an imaging device according to an example embodiment of the inventive concept is shown.

[0022] Fig.15 and Fig.16 The operation of the image forming apparatus according to an exemplary embodiment of the inventive concept is illustrated.

[0023] Fig.17 is a flowchart illustrating an operation of an image forming apparatus according to an example embodiment of the inventive concept.

[0024] Fig.18 and Fig.19 is a resultant image generated in an imaging device according to an exemplary embodiment of the inventive concept.

[0025] Fig. 20 is a schematic block diagram of an imaging device according to an example embodiment of the inventive concept.

[0026] Fig.21 is a schematic block diagram of an electronic device including a memory device according to an example embodiment of the inventive concept. DETAILED DESCRIPTION

[0027] Hereinafter, example embodiments of the inventive concept will be described with reference to the accompanying drawings.

[0028] Figure 1 and Figure 2 is a schematic block diagram of an imaging device according to an example embodiment of the inventive concept.

[0029] First reference Figure 1 , an imaging device 1 according to an exemplary embodiment of the inventive concept includes a sensor module 2 and an image signal processor 5. The sensor module 2 includes a light source 3 and a sensor 4. Figure 1In the illustrated embodiment, the sensor module 2 and the image signal processor 5 may be included in different semiconductor devices. As an example, the sensor module 2 is implemented in the form of a semiconductor package including a light source 3 and a sensor 4, and the image signal processor 5 is included in the semiconductor device. The semiconductor device may be connected to communicate with the sensor module 2 through a predetermined interface, such as an application processor, a central processing unit, a system on chip, etc.

[0030] The light source 3 may include at least one light emitting element that outputs an optical signal in a predetermined wavelength band. The light emitting element may be implemented as a vertical cavity surface emitting laser (VCSEL) or a light emitting diode (LED). For example, when the light source 3 includes a plurality of light emitting elements, the light emitting elements may be arranged on the substrate in an array form. The light source 3 may also include an optical element, etc., which is arranged on the propagation path of the optical signal to improve the characteristics of the optical signal. For example, the optical element may include a lens. As an example, the optical signal output from the light source 3 may be an optical signal in an infrared wavelength band.

[0031] The light source 3 may be operated with two or more optical control signals having different operating frequencies. For example, the light source 3 may be operated with a first optical control signal having a first frequency during a first light emission time, and with a second optical control signal having a second frequency different from the first frequency during a second light emission time. Alternatively, the light source 3 may include a first light source operated with the first optical control signal and a second light source operated with the second optical control signal. In this case, the first light source and the second light source may be operated in turn.

[0032] The optical signal output from the light source 3 may be reflected by the subject 6, and the light receiving signal reflected by the subject 6 may be incident into the sensor 4. The sensor 4 may include pixels that generate electrical signals in response to the light receiving signals and a logic circuit that generates raw data (e.g., raw image data) required to generate a synthetic image using the electrical signals. The raw data may include depth information indicating the distance between the subject 6 and the background around or behind the subject 6. The image signal processor 5 may generate a synthetic image using the raw data, and the synthetic image may be a depth image indicating the distance between the subject 6 and the background around or behind the subject 6.

[0033] refer to Figure 2 According to an exemplary embodiment of the inventive concept, a sensor module 2A and an image signal processor 5A of an imaging device 1A are implemented in a single package. Figure 2 , the light source 3, the sensor 4A and the image signal processor 5A may be included in a single semiconductor package. The image signal processor 5A may be as follows Figure 2 It is shown to be included in the sensor 4A, or it can be implemented separately from the sensor 4A.

[0034] exist Figure 1 and Figure 2 In the example embodiment shown, the image signal processors 5 and 5A can be implemented as software or hardware. Figure 1 In the example embodiment shown, when the image signal processor 5 is implemented separately from the sensor module 2, the image signal processor 5 can be implemented as software inside the application processor. For example, the functions performed by the image signal processor can be included in the software as instructions to be executed by the application processor. Figure 2 In the illustrated example embodiment, when the image signal processor 5A is integrated into the sensor module 2A, the image signal processor 5A may be implemented as hardware.

[0035] Figure 3 is a schematic block diagram of an imaging device according to an example embodiment of the inventive concept.

[0036] refer to Figure 3 , the imaging device 10 includes a logic circuit 20, a pixel array 30, a light source driver 40 (e.g., a driver circuit), and a light source 50. The pixel array 30 may include a plurality of pixels PX arranged in an array along a plurality of rows and a plurality of columns. Each of the plurality of pixels PX may include a photodiode that generates a charge in response to a light reception signal incident from an object 60, and a pixel circuit that generates an electrical signal corresponding to the charge generated by the photodiode.

[0037] As an example, the pixel circuit may include a floating diffusion region, a transfer transistor, a reset transistor, a drive transistor, and a selection transistor. The configuration of the pixel PX may vary according to the example embodiment. As an example, unlike a silicon photodiode, each of the pixels PX may include an organic photodiode containing an organic material, or may be implemented as a digital pixel. When the pixel PX is implemented as a digital pixel, each of the pixels PX may include a comparator and a counter that converts the output of the comparator into a digital signal and outputs the digital signal.

[0038] The logic circuit 20 may include a plurality of circuits for controlling the pixel array 30. As an example, the logic circuit 20 includes a clock driver 21 (e.g., a driver circuit), a readout circuit 22, an operation circuit 23, and a control logic 24 (e.g., a logic circuit). The clock driver 21 may drive the pixel array 30 in a first direction or a second direction. For example, the clock driver 21 may generate a transfer control signal input to a transfer gate of the pixel array 30, a reset control signal input to a reset gate of the pixel circuit 30, a selection control signal input to a selection gate of the pixel circuit 30, or a photoelectric control signal input to a photoelectric gate of the pixel array 30. The first direction and the second direction may be defined in various ways. As an example, the first direction may correspond to a row direction, and the second direction may correspond to a column direction.

[0039] The readout circuit 22 may include at least one of a correlated double sampler (CDS) and an analog-to-digital converter (ADC). The correlated double sampler may be connected to the pixel PX selected by the clock signal provided by the clock driver 21 through a column line, and may perform correlated double sampling to detect a reset voltage and a pixel voltage. The analog-to-digital converter may convert the reset voltage and the pixel voltage detected by the correlated double sampler into digital signals, and may send the digital signals to the operation circuit 23.

[0040] The operation circuit 23 may include a latch or buffer circuit that can temporarily store a digital signal, and an amplifier circuit that amplifies the digital signal. The operation circuit 23 may process the digital signal received from the readout circuit 22. The clock driver 21, the readout circuit 22, and the operation circuit 23 may be controlled by the control logic 24. The control logic 24 may include a timing controller for controlling the operation timing of the readout circuit 22 and the operation circuit 23, and an image signal processor for processing image data. In an exemplary embodiment of the inventive concept, the operation circuit 23 is included in the control logic 24.

[0041] The control logic 24 may perform signal processing on the data output by the readout circuit 22 and the operation circuit 23 to generate raw data for generating a synthetic image. As an example, the synthetic image generated by the raw data may be a depth image. When an image signal processor is included in the control logic 24, the control logic 24 may generate a synthetic image using the raw data.

[0042] According to an example embodiment of the inventive concept, the control logic 24 calculates the distance between the object 60 and the imaging device 10, or determines whether the object 60 is close to the imaging device 10, based on the operation mode of the imaging device 10, using the data output by the readout circuit 22 and the operation circuit 23. Alternatively, the operation circuit 23 may generate a depth image, and the control logic 24 may post-process the depth image to improve the quality of the synthesized image.

[0043] The imaging device 10 may include a light source 50 for outputting an optical signal to the object 60. The light source 50 may include at least one light emitting element. As an example, the light source 50 may include a semiconductor chip in which a plurality of light emitting elements are arranged in an array. The light source 50 may operate under the control of a light source driver 40. The light source driver 40 may be controlled by a logic circuit 20.

[0044] In an exemplary embodiment of the present invention, the light source driver 40 generates an optical control signal having a pulse signal characteristic to drive the light source 50. The light source driver 40 may generate the optical control signal as a pulse width modulation (PWM) signal in response to a control command of the logic circuit 20. The light source driver 40 may use the control command to determine the frequency, duty cycle, and duration of the optical control signal. As an example, the logic circuit 20 may synchronize at least one of the plurality of clock signals input to the pixel array 30 by the clock driver 21 with the optical control signal input to the light source 50. In an exemplary embodiment of the present invention, the signal synchronized with the optical control signal input to the light source 50 may be a photoelectric control signal input to the pixel PX by the clock driver 21. The photoelectric control signal may be a signal that controls the on / off switching of a transistor connected between a photodiode and a floating diffusion region of each of the pixels PX.

[0045] According to an example embodiment of the present inventive concept, an optical element such as a lens and a filter may be disposed on a propagation path through which a light receiving signal reflected by the object 60 is incident into the pixel array 30. For example, a microlens may be formed on a light incident surface of the pixel array 30 to which the light receiving signal is incident. When a portion of the light receiving signal is reflected from the light incident surface of the pixel array 30, the reflected light receiving signal may be scattered between the optical element and the pixel array 30, and may re-enter the pixel PX as an optical noise signal. Therefore, in addition to the light receiving signal, the influence of the optical noise signal may also be reflected in the pixel PX, thereby causing an error in the depth information included in the synthesized image.

[0046] In an exemplary embodiment of the present invention, a method for reducing an error of a synthetic image caused by an optical noise signal is proposed. In an exemplary embodiment of the present invention, raw data for generating a synthetic image is converted into a frequency domain, data in the frequency domain having a predetermined frequency band is selected, and weighting is applied to the selected data to reduce the influence caused by the optical noise signal. As an example, since the optical noise signal generated by scattering has an influence on a large area of ​​the pixel array 30, data of a frequency band having a low frequency band (e.g., less than a predetermined frequency) is selected, and weighting is applied to the selected data to reduce the error of the synthetic image.

[0047] FIG. 4A to FIG. 4C The operation of the image forming apparatus according to an exemplary embodiment of the inventive concept is illustrated.

[0048] First reference Figure 4A , a pixel PX of an imaging device (e.g., 10) according to an exemplary embodiment of the inventive concept includes a photodiode PD generating a charge in response to an optical signal, and a pixel circuit outputting an electrical signal corresponding to the charge generated by the photodiode PD. The pixel circuit may include a first pixel circuit PC1 and a second pixel circuit PC2. The first pixel circuit PC1 may be connected to a first sampling circuit SA1 and a first analog-to-digital converter ADC1 through a first column line COL1, and the second pixel circuit PC2 may be connected to a second sampling circuit SA2 and a second analog-to-digital converter ADC2 through a second column line COL2.

[0049] The first pixel circuit PC1 may include a first phototransistor PX1 connected to the photodiode PD, a first transfer transistor TX1, a first floating diffusion area FD1 in which the charge generated by the photodiode PD is accumulated, and a plurality of first circuit elements RX1, DX1, and SX1. The plurality of first circuit elements RX1, DX1, and SX1 may include a first reset transistor RX1, a first drive transistor DX1, and a first selection transistor SX1. The second pixel circuit PC2 may have a structure similar to that of the first pixel circuit PC1. The control signals TG1, RG1, and SEL1 for controlling the first transfer transistor TX1, the first reset transistor RX1, and the first selection transistor SX1 may be input by a clock driver (e.g., 21) of an imaging device. For example, the second pixel circuit PC2 may include a second phototransistor PX2 connected to the photodiode PD, a second transfer transistor TX2, a second floating diffusion area FD2 in which the charge generated by the photodiode PD is accumulated, and a plurality of second circuit elements RX2, DX2, and SX2. The control signals TG2, RG2, and SEL2 for controlling the second transfer transistor TX2, the second reset transistor RX2, and the second selection transistor SX2 may be input by a clock driver (e.g., 21) of an imaging device.

[0050] When the first reset transistor RX1 is turned on, the voltage of the first floating diffusion area FD1 can be reset to the power supply voltage VDD, and the selection transistor SX1 can be turned on to enable the first sampling circuit SA1 to detect the first pixel voltage. During the first exposure time before the first reset transistor RX1 is turned off and the first transfer transistor TX1 is turned on, the photodiode PD is exposed to light to generate charge.

[0051] When the first transfer transistor TX1 is turned on, the charge generated by the photodiode PD and accumulated in the first photo transistor PX1 may move to the first floating diffusion area FD1. The first sampling circuit SA1 may detect the first pixel voltage in response to the turning on of the first selection transistor SX1. The first analog-to-digital converter ADC1 may convert the difference between the first reset voltage and the first pixel voltage into the first original data DATA1 having a digital format.

[0052] The operation of the second pixel circuit PC2 may be similar to the operation of the first pixel circuit PC1. For example, the second sampling circuit SA2 may detect the second pixel voltage in response to the turning on of the second selection transistor SX2, and the second analog-to-digital converter ADC2 may convert the difference between the second reset voltage and the second pixel voltage into the second original data DATA2 in a digital format. However, the second phototransistor PX2 may be turned on at a time point different from the time point of the first phototransistor PX1. Therefore, the second pixel voltage output by the second pixel circuit PC2 through the second column line COL2 may correspond to the charge obtained by exposing the photodiode PD to light during a second exposure time different from the first exposure time. The second analog-to-digital converter ADC2 may convert the difference between the second reset voltage and the second pixel voltage into the second original data DATA2.

[0053] Figure 4B and Figure 4C is a diagram showing the operation of a pixel PX depending on a first photoelectric control signal PG1 and a second photoelectric control signal PG2. As an example, Figure 4B shows a first photoelectric control signal PG1 and a second photoelectric control signal PG2 input to the pixel PX during a first time, and Figure 4C The first photo control signal PG1 and the second photo control signal PG2 input to the pixel PX during the second time are shown. In example embodiments of the inventive concept, each of the first time and the second time may be a frame period of the image sensor.

[0054] refer to Figure 4B During the first time period, the first photoelectric control signal PG1 has the same phase as the optical control signal, and the second photoelectric control signal PG2 has a phase difference of 180 degrees with the optical control signal. Figure 4C During the second time period, the first photoelectric control signal PG1 has a phase difference of 90 degrees with the optical control signal, and the second transmission control signal TG2 has a phase difference of 270 degrees with the optical control signal.

[0055] The imaging device may generate a composite image including depth information of an object using first raw data DATA1 and second raw data DATA2 obtained during a first time and first raw data DATA1 and second raw data DATA2 obtained during a second time. The first raw data DATA1 obtained by the imaging device at each of the first time and the second time may be determined by a first exposure time ex1 in which the first photoelectric control signal PG1 and the light receiving signal overlap each other. Similarly, the second raw data DATA2 may be determined by a second exposure time ex2 in which the second photoelectric control signal PG2 and the light receiving signal overlap each other.

[0056] For example, when the first original data DATA1 and the second original data DATA2 obtained by the imaging device during the first time are defined as A0 and A1, respectively, and the first original data DATA1 and the second original data DATA2 obtained by the imaging device during the second time are defined as A2 and A3, respectively, the phase difference φ between the optical control signal and the light receiving signal can be calculated based on Equation 1.

[0057]

[0058] Furthermore, information of the depth of the pixel PX and the object generating the light reception signal may be calculated based on Equation 2.

[0059]

[0060] In Equation 2, f m denotes the frequency of the optical control signal, and c denotes the speed of light.

[0061] Figure 5A and Figure 5B The operation of the image forming apparatus according to an exemplary embodiment of the inventive concept is illustrated.

[0062] refer to Figure 5A In the pixel PX of the image sensor according to the example embodiment, a plurality of pixel circuits PC1, PC2, PC3, and PC4 are connected to a single photodiode PD. Figure 5AIn the illustrated example embodiment, the first pixel circuit to the fourth pixel circuit PC1, PC2, PC3 and PC4 are connected to the photodiode PD, and the first pixel circuit to the fourth pixel circuit PC1, PC2, PC3 and PC4 may have substantially the same structure. The third pixel circuit PC3 may include a third phototransistor PX3 connected to the photodiode PD, a third transfer transistor TX3, a third floating diffusion region FD3 in which the charge generated by the photodiode PD is accumulated, and a plurality of third circuit elements RX3, DX3 and SX3. The control signals TG3, RG3 and SEL3 for controlling the third transfer transistor TX3, the third reset transistor RX3 and the third selection transistor SX3 may be input by a clock driver (e.g., 21) of the imaging device. The fourth pixel circuit PC4 may include a fourth phototransistor PX4 connected to the photodiode PD, a fourth transfer transistor TX4, a fourth floating diffusion region FD4 in which the charge generated by the photodiode PD is accumulated, and a plurality of fourth circuit elements RX4, DX4 and SX4. Control signals TG4 , RG4 , and SEL4 for controlling the fourth transfer transistor TX4 , the fourth reset transistor RX4 , and the fourth selection transistor SX4 may be input by a clock driver (eg, 21 ) of the imaging device.

[0063] The operations of the first to fourth pixel circuits PC1, PC2, PC3, and PC4 may be similar to those of reference numerals. FIG. 4A to FIG. 4C In example embodiments, first to fourth phototransistors PX1, PX2, PX3, and PX4 included in first to fourth pixel circuits PC1, PC2, PC3, and PC4 operate in different phases from each other when photodiodes are exposed to light.

[0064] refer to Figure 5B , the first photoelectric control signal PG1 input to the first phototransistor PX1 and the second photoelectric control signal PG2 input to the second phototransistor PX2 have a phase difference of 180 degrees from each other. The third photoelectric control signal PG3 input to the third phototransistor PX3 and the fourth photoelectric control signal PG4 input to the fourth phototransistor PX4 also have a phase difference of 180 degrees from each other, and the third photoelectric control signal PG3 has a phase difference of 90 degrees from the first photoelectric control signal PG1. In an example embodiment, the first to fourth photoelectric control signals PG1, PG2, PG3, and PG4 have the same period as the optical control signal, and may have a duty cycle similar to the duty cycle of the optical control signal. For example, the duty cycle of the first to fourth photoelectric control signals PG1, PG2, PG3, and PG4 may be half the duty cycle of the optical control signal.

[0065] The image sensor can generate a depth map using a pixel voltage obtained from the charges stored in the first to fourth phototransistors PX1, PX2, PX3, and PX4 through the above-mentioned phase difference operation. In the readout operation, the first raw data corresponding to the charge stored in the first phototransistor PX1 is output through the first column line COL1, and the second raw data corresponding to the charge stored in the second phototransistor PX2 is output through the second column line COL2. In addition, the third raw data corresponding to the charge of the third phototransistor PX3 is output through the third column line COL3, and the fourth raw data corresponding to the charge of the fourth phototransistor PX4 is output through the fourth column line COL4. The method of calculating the depth information using the first raw data to the fourth raw data can be the same as described with reference to Equations 1 and 2.

[0066] According to an exemplary embodiment of the present invention, the first pixel circuit PC1 and the third pixel circuit PC3 are connected to a single column line, and the second pixel circuit PC2 and the fourth pixel circuit PC4 are connected to a single column line. For example, the column lines COL1 and COL3 can be connected together to form one of the single column lines, and the column lines COL2 and COL4 can be connected together to form another of the single column lines. Similar to the above, the pixels PX disposed at the same position in the row direction and adjacent in the column direction can share the first to fourth photoelectric control signals PG1, PG2, PG3, and PG4 with each other.

[0067] Figure 6 and Figure 7 The operation of the image forming apparatus according to an exemplary embodiment of the inventive concept is illustrated.

[0068] refer to Figure 6 , an imaging device according to an example embodiment includes a pixel array 100 and a lens 110 disposed in a propagation path of a light receiving signal 140 incident into the pixel array 100. Figure 6 In the illustrated example embodiment, the objects included in the synthesized image may be the object 120 and the background 130 around or behind the object 120. As an example, the distance from the imaging device to the object 120 and the distance from the imaging device to the background 130 may be different from each other. In addition, according to the shapes of the object 120 and the background 130, various types of depth information may be generated for each of the object 120 and the background 130.

[0069] The light receiving signal 140 incident into the pixel array 100 through the lens 110 may be an optical signal output from a light source included in the imaging device and reflected from the object 120 or the background 130. The light receiving signal 140 reflected from the object 120 and the light receiving signal 140 reflected from the background 130 may have different phases depending on the distance between the object 120 and the imaging device and the distance between the background 130 and the imaging device.

[0070] In example embodiments, a light receiving signal 140 incident through the lens 110 may be reflected and / or scattered between the pixel array 100 and the lens 110 to generate an optical noise signal 150. When reflected and / or scattered, the optical noise signal 150 may be modulated with a phase different from that of the light receiving signal 140 and may be introduced into the pixel array 100. Therefore, in addition to the light receiving signal, the influence of the optical noise signal 150 may be reflected in raw data output from at least a portion of pixels included in the pixel array 100.

[0071] As described above, the imaging device can generate raw data from electrical signals (the electrical signals are respectively output by pixels in response to photoelectric control signals with different phases), and can generate a synthetic image from the raw data. When the optical noise signal 150 exists, the size of the multiple pieces of raw data output by the light receiving signal 140 through the pixel may change. Therefore, in the synthetic image generated using the multiple pieces of raw data, the depth information of the object 120 and / or the background 130 may be different from the actual depth data. Now, reference will be made to Figure 7 This is described in more detail.

[0072] refer to Figure 7 , the pixel array 100 includes a plurality of pixels PX. Among the plurality of pixels PX, a first pixel PX1 and a second pixel PX2 are pixels disposed at different positions. In an example embodiment, an optical noise signal generated by reflection and / or scattering of a light receiving signal incident into the second pixel PX2 may be incident into the first pixel PX1.

[0073] The optical noise signal may diffuse from the first point of the second pixel PX2 where reflection and / or scattering occurs to the pixel PX while having a specific shape. As an example, the optical noise signal may diffuse to the pixel PX while having a Gaussian signal shape. Figure 7 As shown, a plurality of optical noise areas 160 (161 to 165) may be defined based on the second pixel PX2. The influence of the optical noise signal on the pixel PX included in each of the optical noise areas 161 to 165 may be substantially the same.

[0074] refer to Figure 7, the imaging device may obtain raw data 101 from the first pixel PX1. The raw data 101 of the first pixel PX1 may include first to fourth raw data A0 to A3. The first to fourth raw data A0 to A3 may be a plurality of pieces of data generated by an optical control signal of a light source included in the imaging device and a photoelectric control signal having phase differences of 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively.

[0075] As an example, the raw data 101 obtained by the imaging device from the first pixel PX1 may include signal data 102 corresponding to a light receiving signal incident into the first pixel PX1, and noise data 103 corresponding to an optical noise signal generated by reflection and / or scattering in the second pixel PX2 to be incident into the first pixel PX1. The signal data 102 may include first to fourth signal data B0 to B3, and the noise data 103 may include first to fourth noise data C0 to C3 generated by photoelectric control signals having phase differences of 0, 90, 180, and 270 degrees.

[0076] exist Figure 7 In the illustrated embodiment, each of the first to fourth original data A0 to A3 is described as including signal data 102 and noise data 103. However, this is merely an example and is not necessarily limited thereto. For example, in at least one of the first to fourth original data A0 to A3, only signal data 102 or noise data 103 may exist.

[0077] refer to Figure 7 , the amplitudes of the first to fourth signal data B0 to B3 may have a distribution different from the distribution of the amplitudes of the first to fourth noise data C0 to C3. As a result, the first to fourth noise data C0 to C3 may cause an error in the depth value calculated by the imaging device from the first to fourth raw data A0 to A3 obtained by the first pixel PX1.

[0078] The ratio of the amplitude of the signal data 102 to the amplitude of the noise data 103 in the first to fourth original data A0 to A3 may vary depending on the position of the first pixel PX1. As an example, when the first pixel PX1 is included in the first area of ​​the pixel array 100 that receives the light reception signal reflected from the object 120, the first to fourth signal data B0 to B3 may tend to have a larger amplitude than the first to fourth noise data C0 to C3. As an example, when the first pixel PX1 is included in the first area, the sum of the amplitudes of the first to fourth signal data B0 to B3 may be greater than the sum of the amplitudes of the first to fourth noise data C0 to C3.

[0079] On the other hand, when the first pixel PX1 is included in the second region of the pixel array 100 that receives the light receiving signal reflected from the background 130, the first to fourth signal data B0 to B3 may tend to have smaller amplitudes than the first to fourth noise data C0 to C3. For example, when the first pixel PX1 is included in the second region, the sum of the amplitudes of the first to fourth signal data B0 to B3 may be smaller than the sum of the amplitudes of the first to fourth noise data C0 to C3.

[0080] Figure 7 The illustrated example embodiment may correspond to a case where the first pixel PX1 receives a light reception signal reflected from the background 130. Therefore, in most of the first to fourth original data A0 to A3 obtained by the imaging device from the first pixel PX1, the first to fourth noise data C0 to C3 may be greater than the first to fourth signal data B0 to B3.

[0081] In an exemplary embodiment of the present inventive concept, the raw data 101 is converted into the frequency domain, and weighting is applied to the low-frequency data of the frequency domain in a low-frequency band having a frequency lower than a predetermined reference frequency to reduce the influence of the noise data 103. For example, the data of the frequency domain outside the low-frequency band can be multiplied by 1, and the low-frequency data can be multiplied by a number lower than 1. The optical noise signal generated by the reflection and / or scattering of the light receiving signal can have an influence on the entire pixel array 100 uniformly, which can be reflected in the raw data 101 and the synthesized image as a low-frequency element. Therefore, the raw data 101 can be converted into the frequency band, and only the low-frequency data can be extracted to adjust its size, thereby allowing the influence of the optical noise signal to be reduced and allowing the quality of the synthesized image to be improved.

[0082] Figure 8 A method of generating depth information in an imaging device according to an example embodiment of the inventive concept is illustrated.

[0083] refer to Figure 8 , showing a first distance and a second distance. The first distance is depth information obtained by the imaging device when the light source is operated with a first optical control signal having a first frequency, and the second distance is depth information obtained by the imaging device when the light source is operated with an optical control signal having a second frequency. For example, the first frequency may be lower than the second frequency.

[0084] The maximum depth measurable during a single cycle of the first optical control signal may be a first depth D1, and the maximum depth measurable during a single cycle of the second optical control signal may be a second depth D2. The maximum depth measurable by the predetermined optical control signal may be inversely proportional to the frequency of the optical control signal. Figure 8The diagram shown may correspond to an example embodiment in which the first frequency of the first optical control signal is 0.8 times the second frequency of the second optical control signal. Thus, the second distance D2 may be 0.8 times the first distance D1.

[0085] exist Figure 8 In the diagram shown, point A may correspond to an object spaced a third depth from the imaging device. As an example, the third depth may be 0.5 meters. In an example embodiment, the round-trip flight time required for the optical signal output from the light source to be reflected to return from point A may be 1 / 2 times the period T1 of the first optical control signal.

[0086] On the other hand, the round-trip flight time required for the optical signal output from the light source to be reflected to return from point B may be the period T1 of the first optical control signal. In addition, the round-trip flight time in point B may be 0.8 times the period T2 of the second optical control signal. For example, when the depth of point B is defined as a fourth depth, the fourth depth may be the same as the second depth D2, which is the maximum depth that can be measured during a single period of the first optical control signal by operating the light source with the first optical control signal.

[0087] The round-trip flight time required for the optical signal output from the light source to be reflected to return from point C may be 4.5 times the period T1 of the first optical control signal, and may be 3.6 times the period T2 of the second optical control signal. As described above, the light source may be driven by the first optical control signal and the second optical control signal having different frequencies from each other to measure various depths.

[0088] The round-trip flight time required for the optical signal to be reflected to return from each of points A, B, and C can be calculated from the phase difference between the optical control signal and the light receiving signal. When an optical noise signal generated by reflection and / or scattering in an imaging device is introduced into a pixel, an error may occur due to the phase difference between the optical control signal and the light receiving signal generated by the optical noise signal. In this case, an error may occur at the depth of each of points A, B, and C.

[0089] As an example, when the phase difference between the light receiving signal and the optical control signal reflected to return from point A is increased by the optical noise signal, the depth of point A can be measured as greater than the actual depth. At the same time, when the phase difference between the light receiving signal and the optical control signal reflected to return from point A is reduced by the optical noise signal, the depth of point A can be measured as less than the actual depth. Therefore, in at least one example embodiment of the present inventive concept, the influence of the optical noise signal can be eliminated or significantly reduced to improve the accuracy of the depth information obtained by the imaging device.

[0090] Fig. 9is a schematic block diagram of an imaging device according to an example embodiment of the inventive concept.

[0091] refer to Fig. 9 , the imaging device 200 according to an exemplary embodiment of the inventive concept includes a frequency conversion (transformation) module 210, a low-pass filter 220, a weighted operation module 230, and an image generation module 240. In an exemplary embodiment, the frequency conversion module 210, the low-pass filter 220, the weighted operation module 230, and the image generation module 240 may be implemented in software or hardware and may be installed in an image signal processor of the imaging device 200.

[0092] The frequency conversion module 210 can receive input data 201 and convert the received input data into the frequency domain. The input data 201 can be raw data generated by a sensor of the imaging device 200. As an example, the imaging device 200 includes a light source (e.g., 3 or 50) and a sensor (e.g., 4 or 4A) operating synchronously with the light source, and the sensor can include a pixel array (e.g., 30) including pixels and a logic circuit (e.g., 20) controlling the pixel array. The logic circuit can input an optical control signal and a photoelectric control signal of a light source having a predetermined phase difference to the pixels of the sensor to obtain raw data.

[0093] In an exemplary embodiment, the logic circuit transmits the first original data A0 to the fourth original data A3 as input data 201 to the image signal processor, as described above with reference to FIG. Figure 7 Alternatively, the logic circuit may calculate the difference between the first original data A0 and the third original data A2 and the difference between the second original data A1 and the fourth original data A3 and transmit the calculated differences as input data 201 to the image signal processor.

[0094] The frequency conversion module 210 may convert the input data 201 into data in the frequency domain using a method such as Fourier transform. The data converted into the frequency domain may be filtered by the low-pass filter 220. The low-pass filter 220 may generate low-frequency data corresponding to or lower than a predetermined reference frequency, and may output the low-frequency data to the weighted operation module 230.

[0095] The weighted operation module 230 can determine the weighting to be applied to the low-frequency data with reference to the size of the low-frequency data appearing in the low-frequency band that is lower than or equal to the reference frequency. According to an exemplary embodiment of the present invention, the weighted operation module 230 performs an inverse Fourier transform on the low-frequency data output from the low-pass filter 220 to generate a result, and applies weighting to the result to generate weighted low-frequency data. As an example, the weighting can be applied uniformly in the low-frequency band, or the weighting can be applied differently according to a plurality of sub-bands formed by dividing the low-frequency band. The low-frequency band can be divided into sub-bands based on the size of the low-frequency data. In an exemplary embodiment, the weighting is selectively applied only to the portion of the low-frequency data having a size greater than a predetermined threshold.

[0096] The image generation module 240 can synthesize the weighted low frequency data with the input data 201 to generate a resultant image 202. Figure 6 and Figure 7 As described above, the influence of the optical noise signal generated due to reflection and / or scattering in the imaging device 200 can be uniformly reflected on the pixels included in the imaging device 200. Therefore, when the input data 201 is converted into the frequency domain, the influence of the optical noise signal may mainly appear in the low frequency band. As a result, by applying a weighting less than 1 to the low frequency data, the size of the low frequency data corresponding to the low frequency band can be reduced to significantly reduce the error of the depth information appearing in the synthesized image 202 due to the optical noise signal.

[0097] In an exemplary embodiment of the inventive concept, the reference frequency of the low-pass filter 220 and the weighting applied to the low-frequency data by the weighted operation module 230 may be determined relative to each other. For example, when the reference frequency is set to high, weighting may be applied to the raw data included in a relatively high frequency band to reduce the size of the raw data. Meanwhile, when the reference frequency is set to low, weighting may be applied only to the raw data included in a significantly narrow or low frequency band.

[0098] Therefore, the reference frequency and the weighting can be determined in relation to each other to significantly reduce the quality degradation of the synthesized image while effectively eliminating the influence of the optical noise signal caused by reflection and / or scattering. As an example, when the reference frequency is set to high, the weighting can be reduced. Therefore, a significantly large weighting is applied to an area where the influence of the optical noise signal hardly occurs to prevent the quality of the synthesized image from deteriorating. On the other hand, when the reference frequency is set to low, the weighting can be increased. Therefore, the influence of the optical noise signal can be eliminated more reliably.

[0099] Alternatively, according to an example embodiment, different kernels may be applied in the process of low-pass filtering the input data 201 converted into the frequency domain. As an example, the input data 201 converted into the frequency domain may be low-pass filtered using a first kernel and a second kernel that are different from each other. The low-pass filtering using the first kernel and the second kernel may include a convolution operation.

[0100] In an exemplary embodiment of the inventive concept, in the low-pass filtering through the first kernel and the second kernel, different reference frequencies and different weights may be applied to the input data 201. The image generation module 240 may generate a synthetic image 202 using the low-frequency data filtered through the first kernel and the second kernel and the input data 201. According to an exemplary embodiment of the inventive concept, the weighted operation module 230 applies weights to the low-frequency data filtered through the first kernel and the second kernel. The weighted operation module 230 may apply the same weight or different weights to the first low-frequency data filtered through the first kernel and the second low-frequency data filtered through the second kernel.

[0101] Fig.10 is a flowchart illustrating an operation of an image forming apparatus according to an example embodiment of the inventive concept.

[0102] refer to Fig.10 , raw data is generated according to the operation of the imaging device of the exemplary embodiment of the present inventive concept (S10). When the light source operates under the control of the optical control signal having a predetermined frequency and a predetermined duty cycle, the pixel of the sensor may generate an electrical signal in response to the light reception signal reflected from the object and the background. The electrical signal may be generated by a plurality of photoelectric control signals input to the pixel, the plurality of photoelectric control signals having a predetermined phase difference with the optical control signal. A logic circuit connected to the pixel may process the electrical signal to generate raw data.

[0103] The raw data is sent from the sensor to the image signal processor, and the image signal processor converts the raw data into the frequency domain (S11). As an example, the image signal processor may convert the raw data into the frequency domain using Fourier transform.

[0104] The image signal processor selects low-frequency data of the frequency domain having a reference frequency or a frequency lower than the reference frequency (S12). When the raw data for generating the depth image is converted into the frequency domain, the high-frequency band corresponds to an area where a large change occurs (such as an edge of an object and a background), and the low-frequency band may correspond to an area where a gradual change occurs. Therefore, the influence of the optical noise signal generated by reflection and / or scattering of the imaging device may be mainly included in the low-frequency data.

[0105] The image signal processor gives (e.g., applies) a predetermined weight to the low-frequency data (S13). The weight may be selected to be a value less than 1 to reduce the influence of the optical noise signal. For example, the weight may be applied to the low-frequency data to generate weighted low-frequency data. The image signal processor generates a composite image using the weighted low-frequency data and the original data (S14).

[0106] Hereinafter, the operation of the imaging device will be described in detail.

[0107] Fig.11A and Fig. 11B are exemplary images illustrating operations of an imaging apparatus according to an example embodiment of the inventive concept.

[0108] Fig.11A and Fig. 11B 1 and 2 show a real image RI and a depth image DI capturing the same scene. The real image RI includes an object SUB and a background BG. In the depth image DI, the object SUB and the background BG can be represented as depth information corresponding to the distance from the imaging device. Fig. 11B , the depth image DI includes a first region 310 corresponding to the object SUB and a second region 320 corresponding to the background BG. The first region 310 and the second region 320 may be represented by different colors and / or brightness.

[0109] According to the depth information, the first area 310 corresponding to the object SUB may be divided into a first portion 311, a second portion 312, and a third portion 313. The first portion 311, the second portion 312, and the third portion 313 may be represented by different colors or brightness. Fig. 11B In the illustrated exemplary embodiment, a first portion 311 corresponding to a hand of a person as the object SUB is closest to the imaging device and is therefore represented by a first brightness (e.g., the highest brightness). A second portion 312 corresponding to the face of the person is darker than the first portion 311 and brighter than a third portion 313 and is represented by a second brightness. A third portion 313 corresponding to the body of the person farthest from the imaging device is represented by a third brightness that is the darkest in the object SUB. For example, the second brightness is brighter than the third brightness and darker than the first brightness.

[0110] refer to Fig. 11B , in the depth image DI, the fourth part corresponding to the background BG can be represented by one brightness. This may be because there is no significant difference in the depth information of the background BG, or the imaging device does not divide the depth information included in the background BG. According to other embodiments, the background BG may also be represented by various colors or brightness depending on the depth information.

[0111] In an ideal case, the depth image DI is generated by a light receiving signal reflected from the object SUB and the background BG and then incident on the pixels of the imaging device. However, in the actual operation of the imaging device, in addition to the light receiving signal, an optical noise signal generated by reflection and / or scattering may be incident on the pixels of the imaging device. Therefore, an error may occur in the depth information included in the depth image DI. Hereinafter, reference will be made to Fig. 12A and Fig. 12B Describe it.

[0112] Fig. 12A and Fig. 12B According to an example embodiment Fig. 11B Magnified view of a portion of the image shown.

[0113] Fig. 12A and Fig. 12B yes Fig. 11B A magnified view of a specific area 301 of the depth image DI is shown. Fig. 12A corresponds to the ideal case where there is no error caused by the optical noise signal, while Fig. 12B This corresponds to a situation where an optical noise signal is introduced into a pixel to cause an error in depth information.

[0114] Reference Fig. 11B , Fig. 12A and 12B The specific area 301 enlarged in FIG. 1 includes a first portion 311 corresponding to the hand of the subject SUB and a second area 320 corresponding to the background BG. Fig. 12A , the boundary between the first portion 311 and the second region 320 is obvious, and the depth information of the first portion 311 and the second region 320 is also clearly expressed.

[0115] Referring to the case where an optical noise signal is introduced into a pixel Fig. 12B , due to the influence of optical noise signals, the depth image may be generated to be different from the actual depth information. Fig. 12B , the second area 320 corresponding to the background BG is divided into a plurality of areas 321 to 324 indicated by different colors and / or shades. For example, the second area (which should be indicated as one depth information because the depth information is incorrectly calculated due to the influence of the optical noise signal) can be indicated as a plurality of areas 321 to 324 having different depth information.

[0116] In an exemplary embodiment of the present invention, in order to solve the above problem, the raw data for generating a depth image is converted into the frequency domain, and only the low-frequency data in the low-frequency band is selected, and the selected data is weighted. As described above, the influence of the optical noise signal is reflected on the entire pixel, and therefore, most of the influence can be included in the low-frequency data. The error of the depth information can be significantly reduced by selecting the low-frequency data and applying weighting (e.g., less than 1) to exclude the influence of the optical noise signal.

[0117] Fig.13 and Fig.14 Data in a frequency domain generated in an imaging device according to an example embodiment of the inventive concept is shown.

[0118] Fig.13 is a diagram showing the result of converting raw data generated by a sensor of an imaging device and transmitted to an image signal processor into a frequency domain. The raw data used to generate a depth image may be data defined in a spatial domain, and Fig.13 The result of converting raw data defined in the spatial domain into the frequency domain is shown. As an example, the imaging device can convert raw data from the spatial domain to the frequency domain using Fourier transform. In the frequency domain generated using Fourier transform, the raw data can be represented by sine functions and / or cosine functions with various amplitudes and phases.

[0119] The image signal processor may perform low pass filtering on the raw data converted into the frequency domain. As an example, a Gaussian kernel may be used to perform the low pass filtering. Fig.14 is a graph showing low-frequency data passed through a low-pass filter.

[0120] Image signal processors can process Fig.14 The generated low-frequency data is weighted. The low-frequency data represents a gradual change in the depth image, and the influence of the optical noise signal on multiple pixels can be reflected in the low-frequency data. By weighting the low-frequency data to generate weighted data, and merging (or combining) the weighted data with the original data, a depth image can be generated to significantly reduce the influence of the optical noise signal and improve the quality of the depth image.

[0121] Fig.15 and Fig.16 The operation of the image forming apparatus according to an exemplary embodiment of the inventive concept is illustrated.

[0122] Fig.15 and Fig.16It is shown that a first area 410 corresponding to an object and a second area 420 corresponding to a background are separated in raw data for generating a depth image. As described above, the raw data may be divided into data having information of a depth less than a predetermined reference depth and data having information of a depth greater than the reference depth to distinguish the first area 410 and the second area 420 from each other. Fig.15 and Fig.16 As shown, the first area 410 and the second area 420 can be separated from each other, and only the second area 420 is converted into the frequency domain, or part of the first area 410 and at least part of the second area 420 are selectively converted into the frequency domain, and therefore, the amount of calculation of the imaging device can be reduced. When the second area 420 is converted into the frequency domain, or when part of the first area 410 and at least part of the second area 420 are selectively converted into the frequency domain, low-pass filtering performed on the data converted into the frequency domain can be omitted.

[0123] As an example, the influence of the optical noise signal in the frequency domain is mainly included in the low-frequency data, and a significant part of the low-frequency data may be included in the second area 420, rather than the first area 410. The depth information of the object may be less than the depth information of the background. Therefore, the depth information may be generated using the original data defined in the spatial domain, and the depth information may be compared with a predetermined reference depth to divide the first area 410 corresponding to the object and the second area 420 corresponding to the background.

[0124] After converting the raw data to the frequency domain and selecting the low-frequency data of the frequency domain, the imaging device can determine the weight to be given to the low-frequency data. In this case, the weight of the low-frequency data is only applied to the second area 420, not to the first area 410. Therefore, the amount of calculation of the imaging device can be reduced, and the speed of generating a synthetic image by the imaging device can be improved.

[0125] In addition, in an exemplary embodiment of the present inventive concept, the imaging device divides the raw data in the spatial domain into a first region 410 and a second region 420 using a predetermined reference depth, and converts only at least a portion of the second region corresponding to the background into the frequency domain. Alternatively, considering that there may be data of a low frequency band in the object, at least a portion of the first region 410 and at least a portion of the second region 420 may be selected and converted into the frequency domain. The reference depth may be changed to appropriately select an object to be converted into the frequency domain in the raw data defined in the spatial domain.

[0126] Since the portion of the original data converted into the frequency domain by the above process mainly corresponds to the low frequency band, the low-pass filtering can be omitted. The imaging device can determine the weighting based on the size of the low frequency data generated by converting the portion of the original data into the frequency domain, and the weighting can be applied only to the second area 420 to generate a composite image.

[0127] Fig.17 is a flowchart illustrating an operation of an image forming apparatus according to an example embodiment of the inventive concept.

[0128] refer to Fig.17 , the operation of the imaging device according to the exemplary embodiment of the present invention is to divide the raw data into a first area and a second area (S20). The raw data may be data defined in a spatial domain and may include the coordinates of each pixel. Each of the pixels may output a portion of the raw data in response to a photoelectric control signal having a different phase. As an example, the depth information may be calculated from the data included in the raw data and may then be compared with a reference depth to be divided into a first area and a second area. In an exemplary embodiment of the present invention, the first area is an area where the depth information is less than a reference depth, and the second area is an area where the depth information is greater than the reference depth.

[0129] The imaging device converts the raw data into the frequency domain (S21). As described above, the raw data can be converted from the spatial domain to the frequency domain using Fourier transform. Low-frequency data belonging to a reference frequency band or a frequency band lower than the reference frequency band is selected (S22).

[0130] The imaging device determines the weighting of the low-frequency data (S23). The imaging device gives (or applies) the weighting determined in operation S23 to the raw data included in the second region (S24). The second region includes large depth information, for example, a region corresponding to a background in a depth image, and the influence of the optical noise signal may be greater in the second region than in the first region. The imaging device generates a composite image using the weighted raw data of the second region and the raw data of the first region (S25).

[0131] As described above, according to example embodiments, the operation of the imaging device may be modified in various ways. For example, in operation S21, the second region or a portion of the first region and a portion of the second region may be selected, rather than the entire first and second regions, to be converted to the frequency domain. In this case, operation S22 corresponding to low-pass filtering may be omitted. In addition, in the operation of step S24 of giving weighting, weighting may also be given to a portion of the first region in addition to the second region.

[0132] Fig.18 and Fig.19 is a composite image generated in an imaging device according to an exemplary embodiment of the inventive concept.

[0133] Fig.18 and Fig.19 is a diagram illustrating a synthesized image 500 generated by converting original data into a frequency domain and selectively applying weighting only to low-frequency data using a method according to example embodiments.

[0134] refer to Fig.18 , the synthesized image 500 includes a first area 510 corresponding to the object and a second area 520 corresponding to the background, and depending on the difference in depth information, the first area 510 includes a first portion 511, a second portion 512, and a third portion 513. As an example, in the first area 510, the first portion 511 may be closest to the imaging device, and the third portion 513 may be farthest from the imaging device.

[0135] refer to Fig.19 , Fig.18 The enlarged view of the region 501 includes a first portion 511 corresponding to the subject's hand and a second region 520 corresponding to the background. Errors caused by the influence of an optical noise signal or the like may mainly appear in low-frequency data. Therefore, errors in depth information may mainly appear in the second region 520. When the error in depth information is not corrected, a background having almost uniform depth information may be displayed as being divided into a plurality of regions 321 to 324 having different depth information.

[0136] refer to Fig.19 , it is possible to reduce errors occurring in the second region 520 due to optical noise signals, etc. Fig. 12B Compared to the illustrated example embodiment, since weighting is applied to low frequency data, the second region 520 may be divided into a first section 521 and a second section 522 to reduce an error in depth information.

[0137] Fig. 20 is a schematic block diagram of an imaging device according to an example embodiment of the inventive concept.

[0138] refer to Fig. 20 , an imaging device 600 according to an exemplary embodiment of the inventive concept includes a first scaler 610, a frequency conversion module 620, a low-pass filter 630, a second scaler 640, a weighted operation module 650, and an image generation module 660. In an exemplary embodiment, the first scaler 610, the frequency conversion module 620, the low-pass filter 630, the second scaler 640, the weighted operation module 650, and the image generation module 660 may be implemented in software or hardware and may be installed in an image signal processor of the imaging device 600.

[0139] The operations of the frequency conversion module 620, the low pass filter 630, the weighted operation module 650 and the image generation module 660 may be similar to those described above with reference to Fig. 9 Those described. Fig. 20 In the illustrated example embodiment, the imaging device further includes a first scaler 610 and a second scaler 640. The first scaler 610 can downscale the input data 601 by reducing the resolution of the input data 601, and the second scaler 640 can upscale the low-frequency data output from the low-pass filter 630 by increasing or restoring the resolution of the low-frequency data. The amount of computation of the imaging device 600 can be reduced by downscaling the input data 601 using the first scaler 610 and performing frequency conversion on the downscaled data to generate data in the frequency domain and low-pass filtering the data in the frequency domain to generate low-pass filtered low-frequency data. In addition, the quality of the synthesized image 602 can be prevented from deteriorating by amplifying the low-pass filtered low-frequency data and calculating weighting.

[0140] Fig.21 is a schematic block diagram of an electronic device including a memory device according to an example embodiment of the inventive concept.

[0141] refer to Fig.21 , the mobile system 1000 includes a camera 1100, a display 1200, an audio processing unit 1300, a modem 1400, DRAMs 1500a and 1500b, flash memory devices 1600a and 1600b, input / output (I / O) devices 1700a and 1700b, and an application processor (hereinafter referred to as “AP”) 1800.

[0142] The mobile system 1000 may be implemented as a laptop computer, a portable terminal, a smart phone, a tablet personal computer (PC), a wearable device, a healthcare device, or an Internet of Things (IoT) device. In addition, the mobile system 1000 may be implemented as a server or a PC.

[0143] The camera 1100 can capture still images or videos under the control of the user. The mobile system 1000 can use the still images / videos captured by the camera 1100 to obtain specific information, or can convert and store the still images / videos into other types of data, such as text. Alternatively, the mobile system 1000 can recognize a string included in the still image / video captured by the camera 1100, and can provide a text or audio translation corresponding to the string. As described above, the camera 1100 in the mobile system 1000 tends to be used in various application fields. In an exemplary embodiment, the camera 1100 can send data, such as still images / videos, to the AP 1800 according to the D-PHY or C-PHY interface in the MIPI standard.

[0144] The camera 1100 may include a plurality of cameras with different viewing angles or aperture values. In addition to a camera that captures an object and generates a real image, the camera 1100 may also include a camera that generates a depth image using depth information of an object and / or background. The raw data generated by the camera for generating a depth image may be converted to a frequency domain by an image signal processor included in the camera 1100 or the AP 1800, and the image signal processor may selectively extract the raw data in the frequency domain in a low frequency band to give weighting. Due to the above operation, in addition to the light receiving signal reflected from the object and / or background, the influence caused by the optical noise signal reflected and / or scattered in the camera 1100, etc., can be significantly reduced to improve the accuracy of the depth image.

[0145] The display 1200 may be implemented in various forms, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode (AMOLED) display, a plasma display panel (PDP), a field emission display (FED), electronic paper, etc. In an exemplary embodiment of the inventive concept, the display 1200 provides a touch screen function, which is also used as an input device of the mobile system 1000. In addition, the display 1200 may be integrated with a fingerprint sensor to provide a security function of the mobile system 1000. In an exemplary embodiment of the inventive concept, the AP 1800 may transmit image data to the display 1200 to be displayed on the display 1200 according to a D-PHY or C-PHY interface in the MIPI standard.

[0146] The audio processing unit 1300 may process audio data stored in the flash memory devices 1600a and 1600b, or audio data included in content received externally through the modem 1400 or the I / O devices 1700a and 1700b. For example, the audio processing unit 1300 may perform various processes such as encoding / decoding, amplification, and noise filtering on the audio data.

[0147] Modem 1400 can modulate a signal and send the modulated signal to send and receive wired / wireless data, and can demodulate a signal received externally to restore the original signal. I / O devices 1700a and 1700b can provide digital input and output, and can include input devices such as ports that can be connected to external recording media, touch screens or mechanical keys, and output devices that can output vibrations in a tactile manner. In some examples, I / O devices 1700a and 1700b can be connected to external recording media through ports such as USB, lightning cable, SD card, micro SD card, DVD, or network adapter.

[0148] The AP 1800 may control the overall operation of the mobile system 1000. Specifically, the AP 1800 may control the display 1200 to display a portion of the content stored in the flash memory devices 1600a and 1600b on the screen. When a user input is received through the I / O devices 1700a and 1700b, the AP 1800 may perform a control operation corresponding to the user input.

[0149] The AP 1800 may be provided as a system on chip (SoC) driving an application or an operating system (OS). In addition, the AP 1800 may be included in a single semiconductor package together with other devices included in the mobile system 1000, for example, the DRAM 1500a, the flash memory 1620, and / or the memory controller 1610. For example, the AP 1800 and at least one device may be provided in a package form, such as a package on package (PoP), a ball grid array (BGA), a chip scale package (CSP), a system level package (SIP), a multi-chip package (MCP), a wafer level manufacturing package (WFP), or a wafer provided in a package form, such as a horizontal process stack package (WSP). The operating system kernel driven on the AP 1800 may include an input / output scheduler and a device driver for controlling the flash memory devices 1600a and 1600b. The device driver may control access performance of the flash memory devices 1600a and 1600b with reference to the number of synchronization queues managed by the input / output scheduler, or may control a CPU mode and a dynamic voltage and frequency scaling (DVFS) level inside the SoC.

[0150] In an example embodiment, the AP 1800 includes a processor block that runs an operation or drives an application and / or an operating system, and various other peripheral components connected through a system block and a system bus. The peripheral components may include a memory controller, an internal memory, a power management block, an error detection block, and a monitoring block. The processor block may include one or more cores. When a plurality of cores are included in the processor block, each of the cores may include a cache memory and a public cache shared by the cores.

[0151] In an example embodiment of the inventive concept, the AP 1800 includes an accelerator block 1820. The accelerator block 1820 may be a dedicated circuit for artificial intelligence (AI) data operations. Alternatively, according to an example embodiment, a discrete accelerator chip separated from the AP 1800 may be provided, and the DRAM 1500b may be additionally connected to the accelerator block 1820 or the accelerator chip. The accelerator block 1820 may be a functional block dedicated to performing a specific function of the AP 1800, and may include a graphics processing unit (GPU) used as a functional block dedicated to processing graphic data, a neural processing unit (NPU) used as a functional block dedicated to performing AI calculations and reasoning, or a data processing unit (DPU) used as a functional block dedicated to sending data.

[0152] According to an exemplary embodiment of the inventive concept, the mobile system 1000 may include a plurality of DRAMs 1500a and 1500b. In an exemplary embodiment of the inventive concept, the AP 1800 includes a controller 1810 for controlling the DRAMs 1500a and 1500b, and the DRAM 1500a may be directly connected to the AP 1800.

[0153] The AP 1800 may control the DRAM according to the JEDEC standard setting command and the mode register setting (MRS) command, or may set the specifications and functions required by the mobile system 1000, such as low voltage, high speed, and reliability, and a DRAM interface protocol for CRC / ECC execution communication. For example, the AP 1800 may set a new DRAM interface protocol to control the DRAM 1500b for an accelerator, wherein an accelerator chip or accelerator block 1820 provided independently of the AP 1800 has a higher bandwidth than the DRAM 1500a to perform communication.

[0154] Although in Fig.21DRAM 1500a and 1500b are shown in the figure, but the configuration of the mobile system 1000 is not necessarily limited thereto. According to the bandwidth and reaction speed of the AP 1800 and the accelerator block 1820 and the voltage condition, memories other than DRAM 1500a and 1500b may be included in the mobile system 1000. As an example, the controller 1810 and / or the accelerator block 1820 may control various memories, such as PRAM, SRAM, MRAM, RRAM, FRAM or hybrid RAM. Compared with the input / output devices 1700a and 1700b or the flash memory devices 1600a and 1600b, the DRAM 1500a and 1500b have relatively low latency and high bandwidth. The DRAM 1500a and 1500b may be initialized at the power-on time point of the mobile system 1000. When an operating system and application data are loaded, DRAMs 1500a and 1500b may be used as a location where the operating system and application data are temporarily stored, or as a space for running various software codes.

[0155] Four basic arithmetic operations such as addition, subtraction, multiplication and division, vector operations, address operations or FFT operation data can be stored in DRAM 1500a and 1500b. In an example embodiment, DRAM 1500a and 1500b can be provided as a memory processing (PIM) with an operation function. For example, a function for performing reasoning in DRAM 1500a and 1500b can be performed. In this case, an artificial neural network can be used to perform reasoning in a deep learning algorithm. The deep learning algorithm may include a training step in which a model is learned using various training data, and an inference step in which the trained model is used to identify (or classify) data. For example, the function used in the reasoning may include a hyperbolic tangent function, a sigmoid function, or a rectified linear unit (ReLU) function.

[0156] According to an example embodiment of the inventive concept, an image captured by a user through the camera 1100 is signal-processed and stored in the DRAM 1500 b , and the accelerator block 1820 or accelerator chip performs AI data operations using the data stored in the DRAM 1500 b and a function for recognizing data in reasoning.

[0157] According to an example embodiment, the mobile system 1000 may include a plurality of storage devices or a plurality of flash memory devices 1600a and 1600b having a capacity higher than that of the DRAMs 1500a and 1500b. The flash memory devices 1600a and 1600b may include a controller 1610 and a flash memory 1620. The controller 1610 may receive a control command and data from the AP 1800, and may write data to the flash memory 1620 in response to a control command, or may read data stored in the flash memory 1620 and transmit the read data to the AP 1800.

[0158] According to an example embodiment of the inventive concept, the accelerator block 1820 or the accelerator chip may use the flash memory devices 1600a and 1600b to perform training steps and AI data operations. In an exemplary embodiment, the operation logic capable of performing predetermined operations in the flash memory devices 1600a and 1600b may be implemented in the controller 1610. Instead of the AP 1800 and / or the accelerator block 1820, the operation logic may use the data stored in the flash memory 1620 to perform at least part of the training steps and reasoning steps performed by the AP 1800 and / or the accelerator block 1820.

[0159] In an exemplary embodiment of the inventive concept, the AP 1800 includes an interface 1830. Therefore, the flash memory devices 1600a and 1600b can be directly connected to the AP 1800. For example, the AP 1800 can be implemented as an SoC, the flash memory device 1600a can be implemented as a chip independent of the AP 1800, and the AP 1800 and the flash memory device 1600a can be installed in a single package. However, the exemplary embodiments of the inventive concept are not limited thereto, and a plurality of flash memory devices 1600a and 1600b can be electrically connected to the mobile system 1000 through a connection.

[0160] The flash memory devices 1600a and 1600b may store data captured by the camera 1100, such as still images / videos, or data received via a communication network and / or ports included in the input and output devices 1700a and 1700b. For example, the flash memory devices 1600a and 1600b may store augmented reality / virtual reality, high definition (HD), or ultra high definition (UHD) content.

[0161] According to an example embodiment, at least one of the DRAMs 1500a and 1500b and the flash memory devices 1600a and 1600b may be implemented as a memory device. As an example, according to an example embodiment, at least one of the DRAMs 1500a and 1500b and the flash memory devices 1600a and 1600b may perform a verification operation every predetermined period, and may run a refresh operation and / or a repair operation as needed. Therefore, the operating performance and reliability of the mobile system 1000 may be improved.

[0162] As described above, according to at least one example embodiment of the inventive concept, an imaging device can convert raw data into a frequency domain, and apply weighting to low-frequency data of a frequency domain having a predetermined reference frequency or a frequency lower than the reference frequency, so as to significantly reduce the error of depth information appearing in a synthesized image due to scattering. Therefore, the performance of the imaging device can be improved.

[0163] While example embodiments of the inventive concept have been shown and described above, it will be apparent to those skilled in the art that various modifications may be made to these embodiments without departing from the spirit and scope of the inventive concept.

Claims

1. An imaging device, comprising: a light source that operates in response to an optical control signal having a predetermined frequency; a sensor including a plurality of pixels configured to generate electrical signals in response to light reception signals output by a light source and reflected from an object, and a logic circuit configured to generate first raw data in a spatial domain based on the electrical signals; and An image signal processor is configured to convert the first original data into second original data in a frequency domain, select low-frequency data of the second original data within a frequency band lower than a predetermined reference frequency, apply predetermined weighting to the low-frequency data to generate weighted data, and generate a composite image based on the first original data and the weighted data.

2. The imaging device according to claim 1, wherein The image signal processor comprises: a frequency conversion module configured to apply a Fourier transform to the first raw data; a low-pass filter configured to filter the output of the frequency conversion module to obtain low-frequency data; and The weighted operation module is configured to apply a predetermined weight to the low-frequency data.

3. The imaging device according to claim 2, wherein: The frequency conversion module, the low-pass filter, the weighted operation module and the sensor are all located in a single semiconductor device.

4. The imaging device according to claim 2, wherein: The frequency conversion module, the low-pass filter and the weighted operation module are located in a semiconductor device, and the sensor is located outside the semiconductor device.

5. The imaging device according to claim 1, wherein The light source operates in response to a first optical control signal having a first frequency during a first light emission time, and operates in response to a second optical control signal having a second frequency different from the first frequency during a second light emission time different from the first light emission time.

6. The imaging device according to claim 1, wherein The light source includes a first light source that operates in response to a first optical control signal having a first frequency, and a second light source that operates in response to a second optical control signal having a second frequency different from the first frequency.

7. The imaging device according to claim 1, wherein: The synthesized image is a depth image representing the distance to the object and the background.

8. The imaging device according to claim 1, wherein The image signal processor comprises: a first scaler configured to downscale the first raw data before converting the first raw data into second raw data in a frequency domain; as well as A second sealer is configured to amplify the low frequency data before applying weighting to the low frequency data.

9. The imaging device according to claim 1, wherein: The sensor includes a lens disposed on a propagation path of a light reception signal, and at least a portion of the light reception signal is scattered by the lens to be re-incident into the pixel.

10. The imaging device according to claim 1, wherein The image signal processor applies a Gaussian filter to the second original data in the frequency domain to select low-frequency data.

11. An imaging device comprising: a light source that operates in response to an optical control signal having a predetermined frequency; a sensor configured to generate raw data using a light reception signal output by a light source and reflected from an object, the raw data including a first area having a depth less than a predetermined reference depth and a second area having a depth greater than the predetermined reference depth; and An image signal processor is configured to convert the raw data into a frequency domain, select low-frequency data in the frequency domain, and apply weighting to the low-frequency data included in the second region to generate a composite image.

12. The imaging device according to claim 11, wherein The first area includes the object, and the second area includes the background.

13. The imaging device according to claim 11, wherein The image signal processor converts at least a portion of the first region and at least a portion of the second region of the original data into a frequency domain to select low frequency data.

14. The imaging device according to claim 11, wherein The image signal processor converts only at least a portion of the second region of the original data into the frequency domain to select low frequency data.

15. The imaging device according to claim 11, wherein The sensor includes: a plurality of pixels, each pixel generating an electrical signal in response to a light reception signal; and a logic circuit configured to control the pixels, and The logic circuit inputs a plurality of control signals having a phase difference with the optical control signal to the pixels respectively, and generates raw data using an electric signal generated by each pixel.

16. The imaging device according to claim 15, wherein Among the pixels, a first pixel generates a first electric signal in response to a light reception signal and an optical noise signal incident into the first pixel by scattering the light reception signal incident into at least one second pixel different from the first pixel, and The logic circuit generates a plurality of pieces of original data of the first pixel using the first electrical signal generated by the first pixel.

17. The imaging device according to claim 16, wherein: At least one of the plurality of pieces of raw data of the first pixel includes signal data corresponding to a light receiving signal incident into the first pixel and noise data corresponding to an optical noise signal incident into the first pixel, and In at least a portion of the plurality of pieces of raw data of the first pixel, ratios of the amplitude of the signal data to the amplitude of the noise data are different from each other.

18. The imaging device according to claim 17, wherein: When the first pixel is included in the first region, the amplitude of the signal data is greater than the amplitude of the noise data, and when the first pixel is included in the second region, the amplitude of the signal data is less than the amplitude of the noise data.

19. An imaging device comprising: a pixel array including a plurality of pixels, each pixel operating in response to a first photoelectric control signal and a second photoelectric control signal having different phases to output a first electrical signal and a second electrical signal; a logic circuit configured to generate raw data for generating a depth image using the first electrical signal and the second electrical signal; as well as An image signal processor is configured to convert raw data from a spatial domain into a frequency domain and reduce the size of low-frequency data of the frequency domain in a low-frequency band lower than a predetermined reference frequency to generate a synthesized image.

20. The imaging device according to claim 19, wherein The image signal processor divides a low frequency band into a plurality of sub-bands based on a size of the low frequency data, and differently reduces the size of the low frequency data in at least parts of the sub-bands.

Images