Electronic device and method for performing auto-focusing

Abstract

An electronic device according to an embodiment disclosed in this document includes: an image sensor including a plurality of pixels; an AF processing unit; a memory; and a control unit. Each pixel included in the plurality of pixels includes a plurality of photodiodes and a microlens. The control unit provides at least one of a first piece of AF data and a second piece of AF data to the AF processing unit, and the AF processing unit may perform a first phase auto focus (PAF) operation based on the first piece of AF data or a second PAF operation based on the second piece of AF data.

Description

Technical Field

[0001] Various embodiments of the present disclosure relate to an electronic device for capturing an image and a method for performing auto focus (AF). Background Art

[0002] Various types of electronic devices capable of capturing images, such as smart phones, tablet PCs, and camera devices, have entered the market. The electronic devices may recognize an object to perform auto focus (AF).

[0003] A phase auto focus (PAF) scheme for detecting a phase difference between two pixels adjacent to each other and determining a focus is used as a method for performing auto focus (AF). The PAF scheme can be compared with a continuous AF scheme for measuring a focus according to a change in a planar frequency characteristic of an image to obtain a more accurate and rapid result when determining the directionality of the focus.

[0004] In the PAF scheme, two-dimensional image pixels arranged continuously and periodically at adjacent positions have multiple photodiodes (hereinafter referred to as PDs) sharing one microlens. When PDs that are electrically separated from each other and optically have the same characteristics are not accurately focused, they may indicate different detection characteristics. The PAF scheme can analyze multiple pieces of information between multiple PDs detected when the PDs are not focused, and can determine the consistency of AF. Summary of the invention

[0005] Technical issues

[0006] An image sensor of an electronic device according to the prior art can perform AF by detecting the phase difference between two PDs arranged in the horizontal direction. The two PDs separated horizontally can share a microlens and can be separated by a pixel wall. Light incident through the microlens can be refracted differently along the path to separate from each other. Because the image is formed horizontally by the same lens, the characteristics and level values ​​of the horizontally positioned pixels may change when the focus matches and there is a subject in the arrangement direction of the pixels. In this case, the edges separated in the vertical direction cause the problem that the ideal value cannot be detected regardless of whether the focus matches.

[0007] When using a 4PD image sensor including 4 (2x2) PDs separated in the horizontal and vertical directions, the AF detection circuit performs calculations in the horizontal direction and calculations in the vertical direction at the same time, so relatively more buffers are required therein to check the correlation between pixels adjacent to each other. In addition, the calculation process becomes complicated and problems may arise due to an increase in power consumption or cost.

[0008] Technical Solutions

[0009] Various embodiments of the present disclosure may process AF data in multiple directions simultaneously or sequentially using data obtained by a 4PD image sensor.

[0010] According to one aspect of the present disclosure, an electronic device is provided. The electronic device may include: an image sensor including a plurality of pixels; an AF processing unit that performs calculations to perform autofocus based on a phase difference; a memory; and a controller that delivers data obtained by the image sensor to the AF processing unit or the memory. Each pixel included in the plurality of pixels may include a plurality of photodiodes and a microlens covering the plurality of photodiodes. The controller may provide the AF processing unit with at least one of first AF data of a photodiode arranged along a first direction among the plurality of photodiodes or second AF data of a photodiode arranged along a second direction among the plurality of photodiodes. The AF processing unit may perform a first phase autofocus (PAF) calculation based on the first AF data, or may perform a second PAF calculation based on the second AF data.

[0011] Beneficial effects

[0012] The electronic device according to the embodiment disclosed in the present disclosure may process AF data in multiple directions simultaneously or sequentially using data obtained by the image sensor.

[0013] The electronic device according to the embodiment disclosed in the present disclosure can detect the directionality of an edge and can reorder AF data in the edge direction, thereby simplifying the calculation of the AF detection circuit.

[0014] The electronic device according to the embodiments disclosed in the present disclosure may reduce calculation time and may improve AF accuracy by rotating or reordering AF data of an image sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 An electronic device capable of capturing an image according to an embodiment is shown;

[0016] Figure 2 shows a configuration of an image sensor according to an embodiment;

[0017] Figure 3 is a diagram illustrating an edge direction of an object and AF detection according to various embodiments;

[0018] Figure 4 illustrates a camera module according to various embodiments;

[0019] Figure 5 is an output timing diagram of image data and AF data according to various embodiments;

[0020] Figure 6 shows AF processing using two 2PDPAF circuits in Bayer mode according to various embodiments;

[0021] Figure 7a shows AF processing using a 2PD PAF circuit in Bayer mode according to various embodiments;

[0022] Figure 7b is a timing diagram of output of processed image data and AF data according to various embodiments;

[0023] Figure 8a illustrates AF processing using a directional filter in Bayer mode according to various embodiments;

[0024] Figure 8b is an output timing diagram of a selective AF process using a directional filter according to various embodiments;

[0025] Fig. 9 shows AF processing using two 2PDPAF circuits in processing mode according to various embodiments;

[0026] Fig.10 shows AF processing using a 2PD PAF circuit in processing mode according to various embodiments;

[0027] Fig.11 AF processing using a directional filter in a processing mode is shown according to various embodiments;

[0028] Fig.12a and Figure 12b is a diagram showing detecting edges in multiple directions through a directional filter and performing PAF calculation according to various embodiments;

[0029] Fig.13 is a block diagram illustrating an electronic device in a network environment according to various embodiments; and

[0030] Fig.14 is a block diagram illustrating a camera module according to various embodiments.

[0031] In describing the drawings, the same or similar components may be labeled with the same or similar reference numerals. DETAILED DESCRIPTION

[0032] In the following, various embodiments of the present disclosure are described with reference to the accompanying drawings. Therefore, those skilled in the art will recognize that various modifications, equivalents and / or substitutes may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. With respect to the description of the accompanying drawings, similar components may use similar reference numerals.

[0033] Figure 1 An electronic device capable of capturing an image according to an embodiment is shown.

[0034] refer to Figure 1 , the electronic device 100 according to the embodiment may generate a preview image based on image data obtained using the camera module (or camera device) 110 , and may output the generated preview image to the display 120 .

[0035] The camera module 110 may include, for example, a lens assembly including one or more lenses and an image sensor. Each pixel of the image sensor may include a plurality of photodiodes.

[0036] On the first screen 101 , the electronic device 100 may generate phase difference data (or depth data) through an optical path difference generated by a plurality of photodiodes (hereinafter referred to as PDs) sharing a microlens.

[0037] On the second screen 105 , the electronic device 100 may adjust the focus on the external object by moving one or more lenses included in the camera module 110 based on the generated phase difference data. Therefore, the electronic device 100 may display a preview image focused on the external object on the display 120 .

[0038] exist Figure 1 illustratively shows that the camera module 110 is a front camera of the electronic device 100, but is not limited thereto. For example, the camera module 110 may be provided on at least one of a rear surface or a side surface of the electronic device 100.

[0039] Figure 2 A configuration of an image sensor according to an embodiment is shown.

[0040] refer to Figure 2 , including in camera modules (e.g. Figure 1 The image sensor 200 in the camera module 110 may include a plurality of pixels. Figure 2 illustratively shows that the image sensor 200 outputs an image in a Bayer pattern based on a signal generated by a 4PD image sensor, but is not limited thereto.

[0041] According to an embodiment, one of the plurality of pixels 210 may include a microlens 205, a color filter 206, a first PD (or first sub-pixel) (PD1) 211, a second PD (or second sub-pixel) (PD2) 212, a third PD (or third sub-pixel) (PD3) 213 and a fourth PD (or fourth sub-pixel) (PD4) 214.

[0042] The microlens 205 may cover the first to fourth PDs 211, 212, 213, and 214. The microlens 205 may adjust a path of incident light so that light incident from the outside may reach the first to fourth PDs 211, 212, 213, and 214.

[0043] The color filter 206 may be disposed between the microlens 205 and the first to fourth PDs 211, 212, 213, and 214 to pass light of a specified wavelength range (e.g., a wavelength range corresponding to green light). The color filter 206 may allow only light of a specified wavelength range among the light to pass through the microlens 205, and may limit light other than the specified wavelength range.

[0044] Each of the first to fourth PDs 211 , 212 , 213 , and 214 may convert light passing through the microlens 205 and the color filter 206 into an electrical signal.

[0045] Light introduced from the outside may be refracted to change the advancing path by the microlens 205. Light passing through the microlens 205 may be directly introduced into the PD, or may be reflected from the pixel wall W between the PDs to be introduced into the PD.

[0046] Figure 3 2 is a diagram illustrating an edge direction of a description object and AF detection according to various embodiments.

[0047] refer to Figure 3 , the image sensor 200 may include a plurality of pixels. One 210 of the plurality of pixels may be covered by one microlens 205. For example, the one pixel 210 may include first to fourth PDs 211, 212, 213, and 214.

[0048] The first object 310 arranged in the vertical direction may form an edge 311 in the horizontal direction. Two pixels separated horizontally (e.g., the first PD 211 and the second PD 212) may be separated by a pixel wall 312. Light incident through the microlens 205 may be refracted differently along the path to be separated from each other. Since the image is formed by the same microlens 205, for the first object 310 having the same focus and existing in the pixel arrangement direction, the characteristics and level values ​​of the horizontally positioned pixels (e.g., the first PD 211 and the second PD 212) may vary.

[0049] The second object 320 arranged in the horizontal direction may form an edge 321 in the vertical direction. Two pixels separated vertically (e.g., the first PD 211 and the third PD 213) may be separated by a pixel wall 322. Light incident through the microlens 205 may be refracted differently along the path to separate from each other. Since the image is formed by the same microlens, for the second object 320 having the same focus and existing in the pixel arrangement direction, the characteristics and level values ​​of the vertically positioned pixels (e.g., the first PD 211 and the third PD 213) may vary.

[0050] Figure 4 A camera module according to various embodiments is shown.

[0051] refer to Figure 4 , the camera module 400 may include an image sensor 410 , a controller 420 , a memory 430 , an image processing unit 440 , and an AF processing unit 450 . Figure 4 For example, the image sensor 410 and the controller 420 may be integrated into one, or the controller 420 and the AF processing unit 450 may be integrated into one.

[0052] The image sensor 410 may convert light incident from the outside into an electrical signal. The image sensor 410 may be composed of a plurality of pixels. Each pixel may include a plurality of PDs. Hereinafter, a case where each pixel includes four PDs will be described, but is not limited thereto.

[0053] According to an embodiment, the image sensor 410 may generate image data for taking a photo or a moving image or image data for displaying a preview screen. In addition, the image sensor 410 may generate data (or a signal) for performing AF (hereinafter referred to as AF data).

[0054] According to an embodiment, when the image sensor 410 is composed of a plurality of PDs sharing the same microlens, data of the plurality of PDs may be output in an arranged order (e.g., a Bayer pattern). According to another embodiment, the image sensor 410 may generate and output a signal in a format processed by an output mode such as merging to reduce output data. The image sensor 410 may output a signal in a format processed by performing image processing in a horizontal / vertical direction or a diagonal direction according to the arrangement of pixels and when necessary.

[0055] The controller (or pre-processing unit or data conversion unit) 420 may process the image data or AF data received from the image sensor 410. The controller 420 may provide the image data to the image processing unit (e.g., ISP) 440. The controller 420 may temporarily store the AF data in the memory 430, or may deliver the AF data to the AF processing unit 450.

[0056] The controller 420 may detect directivity from the AF data of the image sensor 410 to perform accurate and simple AF calculation for horizontal and vertical or more directions, and may reorder the data in a dominant direction so that the AF processing unit 450 performs AF calculation.

[0057] For example, the controller 420 may maintain the AF data in a first direction (e.g., horizontal direction), may rotate or reorder the AF data in a second direction (e.g., vertical direction), and may deliver the AF data to the AF processing unit 450. Thus, the AF processing unit 450 may consistently perform PAF calculation regardless of directionality.

[0058] According to an embodiment, the controller 420 may convert and deliver image data or AF data received from the image sensor 410 to the memory 430, the image processing unit 440, or the AF processing unit 450, or may deliver the image data or AF data to the memory 430, the image processing unit 440, or the AF processing unit 450 without a separate conversion task.

[0059] According to various embodiments, the controller 420 may include a direction filter 421 or a rotator 422. The direction filter 421 may be used to determine a dominant direction among edge directions of an object. The rotator 422 may change the direction of the AF data to change the AF data into a form of data that can be easily processed by the AF processing unit 450.

[0060] The memory 430 may store AF data (or converted AF data). For example, the memory 430 may be a dynamic random access memory (DRAM). The stored AF data may be delivered to the AF processing unit 450 according to the directionality of the edge.

[0061] The image processing unit (eg, ISP) 440 may process the image data. For example, the received image data may be processed to be displayed as a preview image.

[0062] The AF processing unit 450 may perform AF (PAF) using a phase difference based on the received AF data. The AF processing unit 450 may include at least one or more PAF circuits 451. Figure 4 The embodiment illustratively shows that a PAF circuit is included, but is not limited thereto.

[0063] The PAF circuit 451 may perform PAF calculation in a designated direction. For example, the PAF circuit 451 may process AF data of some (eg, two) photodiodes among a plurality of photodiodes (eg, first to fourth PDs) included in one pixel of the image sensor 410 .

[0064] According to an embodiment, the PAF circuit 451 may perform PAF calculation using AF data extracted from horizontally separated pixels (e.g., the first PD and the second PD or the third PD and the fourth PD), or may perform PAF calculation using AF data extracted from vertically separated pixels (e.g., the first PD and the third PD or the second PD and the fourth PD). Hereinafter, a description of a 2PD PAF circuit will be given, but is not limited thereto.

[0065] According to an embodiment, the camera module 400 may include a plurality of PAF circuits respectively corresponding to a plurality of edge directions that can be detected by the image sensor 410. For example, in the case of edge detection in the vertical / horizontal direction, two PAF circuits may be included. For another example, when edges in eight directions can be detected, eight PAF circuits may be included.

[0066] According to another embodiment, the camera module 400 may use one PAF circuit to perform PAF calculations on multiple edge directions. For example, in the case of edge detection in the vertical / horizontal direction, the PAF calculation of the edge in the horizontal direction may be performed first, and the PAF calculation of the edge in the vertical direction may be performed secondly.

[0067] According to another embodiment, the camera module 400 may determine a dominant direction among edge directions using one PAF circuit, perform PAF for one dominant direction, and may not or assistively proceed with PAF for other directions.

[0068] According to various embodiments, at least some of the functions performed by the controller 420, the image processing unit 440, or the AF processing unit 450 may be performed by an electronic device (eg, Figure 1 The process is executed by a processor (e.g., AP) of the electronic device 100.

[0069] According to various embodiments, at least some of the controller 420, the image processing unit 440, or the AF processing unit 450 may be separate components (e.g., chips) in hardware. Alternatively, at least some of the controller 420, the image processing unit 440, or the AF processing unit 450 may be separate operations in software in the same computing element.

[0070] Figure 5 is an output timing diagram of image data and AF data according to various embodiments.

[0071] refer to Figure 5When the state of the enable signal 501 of the image sensor 410 changes, the image sensor 410 may output the image data (e.g., Bayer signal) 510. The image sensor 410 may output the AF data 520 output together with the image data (e.g., Bayer signal) 510 at a longer time interval than the image data (e.g., Bayer signal) 510.

[0072] The image sensor 410 may process the AF data in one direction to obtain AF from the image data (eg, Bayer signal) 510 , and may output AF data 520 having a smaller resolution than the image data (eg, Bayer signal) 510 by merging or the like.

[0073] exist Figure 5 5. The AF data AF_LR 521 in the horizontal direction and the AF data AF_TB 522 in the vertical direction are illustratively shown in FIG. 5 as being output at the same timing, but the present invention is not limited thereto. For example, the AF data AF_LR 521 in the horizontal direction and the AF data AF_TB 522 in the vertical direction are output at different timings.

[0074] Figure 6 FIG. 4 shows an AF process using two 2PDPAF circuits in Bayer mode according to various embodiments. Figure 6 Two 2PDPAF circuits are illustratively shown in FIG. 1 , but are not limited thereto.

[0075] refer to Figure 6 , the 4PD image sensor 610 may include four PDs in one pixel. The 4PD image sensor 610 may sequentially read out the PDs included in each pixel to generate a multi-PD signal (or raw data) 601.

[0076] The image generator 620 may integrate the values ​​of four PDs included in one pixel to generate the image data 602. The image generator 620 may transmit the image data 602 to an image processing unit (eg, ISP) 625.

[0077] The first AF data generator 630 may deliver the AF data 603 in a first direction (eg, horizontal direction) in the multi PD signal 601 to the first PAF circuit 635 .

[0078] The second AF data generator 640 may deliver the AF data 604 in a second direction (eg, a vertical direction) in the multi-PD signal 601 to the second PAF circuit 645 .

[0079] Each of the first PAF circuit 635 and the second PAF circuit 645 can perform PAF calculation for different directions. The first PAF circuit 635 and the second PAF circuit 645 can perform PAF calculation at the same time. Therefore, fast and accurate AF can be performed.

[0080] exist Figure 6 2 , but the present invention is not limited thereto. For example, eight PAF circuits may be used to process AF data in eight directions simultaneously and respectively.

[0081] Figure 7a AF processing using a 2PD PAF circuit in Bayer mode is shown according to various embodiments.

[0082] refer to Figure 7a , the 4PD image sensor 710 may include four PDs in one pixel. The 4PD image sensor 710 may sequentially read out the PDs included in each pixel to generate a multi-PD signal (or raw data) 701.

[0083] The image generator 720 may integrate the values ​​of four PDs included in one pixel to generate the main image data 702. The image generator 720 may transmit the image data 702 to an image processing unit (eg, ISP) 725.

[0084] The AF data generator 730 may convert the multi-PD signal 701 into AF data 703 that can be processed by the PAF circuit 735, and may store the AF data 703 in a memory (eg, DRAM) 750. For example, vertical data 703a and horizontal data 703b may be stored.

[0085] The memory 750 may store the AF data 703. The memory 750 may be a memory buffer included in the camera module or a memory area formed independently of the camera module.

[0086] The PAF circuit 735 may process the AF data 703 stored for each direction from the memory 750. According to an embodiment, the PAF circuit 735 may sequentially process the AF data 703 according to the direction. For example, the PAF circuit 735 may first process the horizontal data 703b and may secondly process the vertical data 703a.

[0087] According to an embodiment, the rotator 760 may rotate a signal in a direction other than the first direction (eg, the horizontal direction) and deliver it to the PAF circuit 735 .

[0088] For example, the vertical data 703a may be rotated 90 degrees to be provided to the PAF circuit 735 (703a-1). Therefore, the horizontal data 703b and the converted vertical data 703a-1 may have similar data structures to each other, and the calculation process of the PAF circuit 735 may be simplified.

[0089] Figure 7b is a timing diagram of output of processed image data and AF data according to various embodiments.

[0090] refer to Figure 7b In response to the timing signal SENSOR_VSYNC of the image sensor 710, the image sensor 710 may store the AF data SAVE_LR and SAVE_TB at designated time intervals.

[0091] The PAF circuit 735 may start PAF using the horizontal data SAVE_LR when the storage of the horizontal data SAVE_LR is completed. The horizontal PAF (AF_LR) may be completed at the same time as the storage of the horizontal data SAVE_LR ends (or as a portion of the storage of the horizontal data SAVE_LR is delayed).

[0092] After completing the PAF in the horizontal direction, the PAF circuit 735 may start PAF (AF_TB) using the stored vertical data SAVE_TB. The PAF using the vertical data SAVE_TB may continue in the state where the vertical data SAVE_TB is stored (the state where the timing signal SENSOR_VSYNC is low).

[0093] AF data can be converted into a size smaller than the image data to be output. PAF (AF_TB) can be processed in a relatively short time using PAF (AF_LR) in the horizontal direction or vertical data SAVE_TB. Due to different readout directions, it is possible to continue PAF (AF_TB) using vertical data SAVE_TB after the readout of all frames is completed.

[0094] According to various embodiments, the PAF circuit 735 can perform PAF calculations for multiple directions when the timing signal SENSOR_VSYNC is low. For example, the PAF circuit 735 can perform PAF calculations for multiple directions in the vertical direction, the 45 degree direction, and the 125 degree direction when the timing signal SENSOR_VSYNC is low.

[0095] Figure 8a An AF process using a directional filter in Bayer pattern is shown according to various embodiments.

[0096] refer to Figure 8a, the 4PD image sensor 810 may include four PDs in one pixel. The 4PD image sensor 810 may sequentially read out the PDs included in each pixel to generate a multi-PD signal (or raw data) 801.

[0097] The image generator 820 may integrate the values ​​of four PDs included in one pixel to generate the main image data 802. The image generator 820 may transmit the image data 802 to an image processing unit (eg, ISP) 825.

[0098] The AF data generator 830 may convert the multi-PD signal 801 into AF data 803 that can be processed by the PAF circuit 835, and may store the AF data 803 in a memory (eg, DRAM) 850. For example, vertical data 803a and horizontal data 803b may be stored.

[0099] The memory 850 may store the AF data 803. The memory 850 may be a memory buffer included in the camera module or a memory area formed independently of the camera module.

[0100] The direction filter 870 may detect an edge direction using the AF data 803. The PAF circuit 835 may process AF data for one direction among the AF data 803 stored for each direction from the memory 850. The PAF circuit 835 may perform one PAF calculation for one direction determined by the direction filter 870.

[0101] For example, vertical data 803a as one direction according to the arrangement direction of edges (or the arrangement direction of objects) may be selected between horizontal data 803a and vertical data 803b. The PAF circuit 835 may rotate the vertical data 803a (803a-1) through a rotator 860 to perform a PAF calculation.

[0102] According to an embodiment, the rotator 860 may rotate a signal in a direction other than the first direction (e.g., the horizontal direction) and deliver it to the PAF circuit 835. For example, the vertical data 803a may be rotated 90 degrees to be delivered to the PAF circuit 835 (803a-1). Therefore, the horizontal data 803b and the converted vertical data 803a-1 may have data structures similar to each other, and the calculation process of the PAF circuit 835 may be simplified.

[0103] Figure 8b is an output timing diagram of a selective AF process using a directional filter according to various embodiments.

[0104] refer to Figure 8bIn response to the timing signal SENSOR_VSYNC of the image sensor 810, the image sensor 810 may store the AF data SAVE_LR and SAVE_TB at designated time intervals.

[0105] While the storage of the horizontal data SAVE_LR and the vertical data SAVE_TB continues, the PAF circuit 835 may perform edge detection (or directionality determination of an image) using the directional filter 870 .

[0106] After the storage of the horizontal data SAVE_LR and the vertical data SAVE_TB is completed, the PAF circuit 835 may continue to perform PAF for one direction selected by the direction filter 870. In a state where the timing signal SENSOR_VSYNC is low, PAF may be performed once.

[0107] Fig. 9 FIG. 4 shows AF processing using two 2PDPAF circuits in processing mode according to various embodiments. Fig. 9 Two 2PDPAF circuits are illustratively shown in FIG. 1 , but are not limited thereto.

[0108] refer to Fig. 9 , the 4PD image sensor 910 may include four PDs in one pixel. The 4PD image sensor 910 may integrate the values ​​of the four PDs included in one pixel to generate image data 902 . The 4PD image sensor 910 may send the image data 902 to an image processing unit (eg, ISP) 925 .

[0109] The 4PD image sensor 910 may deliver first processed data 903 obtained by processing AF data in a first direction (eg, a horizontal direction) to the first PAF circuit 935 .

[0110] The 4PD image sensor 910 may deliver second processed data 904 obtained by processing AF data in a second direction (eg, a vertical direction) to the second PAF circuit 945 .

[0111] Each of the first PAF circuit 935 and the second PAF circuit 945 can perform PAF calculation for different directions. The first PAF circuit 935 and the second PAF circuit 945 can perform PAF calculation at the same time. Therefore, fast and accurate AF can be performed.

[0112] exist Fig. 9 2 , but the present invention is not limited thereto. For example, eight PAF circuits may be used to process AF data in eight directions simultaneously and respectively.

[0113] Fig.10AF processing using a 2PDPAF circuit in processing mode is shown according to various embodiments.

[0114] refer to Fig.10 , the 4PD image sensor 1010 may include four PDs in one pixel. The 4PD image sensor 1010 may integrate the values ​​of the four PDs included in one pixel to generate image data 1002 . The 4PD image sensor 1010 may send the image data 1002 to an image processing unit (eg, ISP) 1025 .

[0115] The 4PD image sensor 1010 may store first processed data 903 obtained by processing AF data in a first direction (eg, horizontal direction) in a memory (eg, DRAM) 1050. For example, vertical data 1003a and horizontal data 1003b may be stored.

[0116] The memory 1050 may store the AF data 1003. The memory 1050 may be a memory buffer included in the camera module or a memory area formed independently of the camera module.

[0117] The PAF circuit 1035 may process the AF data 1003 stored for each direction from the memory 1050. According to an embodiment, the PAF circuit 1035 may sequentially process the AF data 1003 according to the direction. For example, the PAF circuit 1035 may first process the horizontal data 1003b and may secondly process the vertical data 1003a.

[0118] According to an embodiment, the rotator 1060 may rotate a signal in a direction other than the first direction (e.g., the horizontal direction) and deliver it to the PAF circuit 1035. For example, the vertical data 1003a may be rotated 90 degrees to be provided to the PAF circuit 1035 (1003a-1). Therefore, the horizontal data 1003b and the converted vertical data 1003a-1 may have data structures similar to each other, and the calculation process of the PAF circuit 1035 may be simplified.

[0119] Fig.11 AF processing using a directional filter in processing mode is shown according to various embodiments.

[0120] refer to Fig.11 , the 4PD image sensor 1110 may include 4 PDs in one pixel. The 4PD image sensor 1110 may integrate the values ​​of the four PDs included in one pixel to generate image data 1102 . The 4PD image sensor 1110 may send the image data 1102 to an image processing unit (eg, ISP) 1125 .

[0121] The 4PD image sensor 1110 may store first processed data 1103 obtained by processing AF data in a first direction (eg, horizontal direction) in a memory (eg, DRAM) 1150. For example, vertical data 1103a and horizontal data 1103b may be stored.

[0122] The memory 1150 may store the AF data 1103. The memory 1150 may be a memory buffer included in the camera module or a memory area formed independently of the camera module.

[0123] The direction filter 1160 may detect an edge direction using the AF data 1103. The PAF circuit 1135 may process AF data for one direction among the AF data 1103 stored for each direction from the memory 1150. The PAF circuit 1135 may perform one PAF calculation for one direction determined by the direction filter 1170.

[0124] For example, vertical data 1103a as one direction according to the arrangement direction of edges (or the arrangement direction of objects) may be selected between horizontal data 1103a and vertical data 1103b. The PAF circuit 1135 may rotate the vertical data 1103a (1103a-1) through a rotator 1160 to perform a PAF calculation.

[0125] According to an embodiment, the rotator 1160 may rotate a signal in a direction other than the first direction (e.g., the horizontal direction) and deliver it to the PAF circuit 1135. For example, the vertical data 1103a may be rotated 90 degrees to be provided to the PAF circuit 1135 (1103a-1). Therefore, the horizontal data 1103b and the converted vertical data 1103a-1 may have data structures similar to each other, and the calculation process of the PAF circuit 1135 may be simplified.

[0126] Fig.12a and Figure 12b is a diagram illustrating detecting edges in multiple directions through a directional filter and performing PAF calculation according to various embodiments.

[0127] refer to Fig.12a , the 4PD image sensor 1210 may include four PDs in one pixel. The 4PD image sensor 1210 may sequentially read out the PDs included in each pixel to generate a multi-PD signal (or raw data) 1201.

[0128] The direction filter 1270 can compare the detection values ​​of multiple directions in the AF data using the multi-PD signal (or raw data) to detect the directionality (edge ​​direction) of the image. For example, when the data of the first part and the second part among all pixels of the AF data are changed to a specified value or more by an external object, it can be determined that the boundary (edge) of the first part and the second part is formed.

[0129] For example, in Figure 12b , data of the first portion 1201a1 and the second portion 1201a2 of the 4PD image sensor 1210 may be changed to a specified value or more by the first object 1251. The direction filter 1270 may compare the values ​​of the image data to detect an edge direction 1251a perpendicular to the boundary of the first portion 1201a1 and the second portion 1201a2 as a fifth direction (0 degrees).

[0130] For another example, data of the first portion 1201b1 and the second portion 1201b2 of the 4PD image sensor 1210 may be changed to a specified value or more by the second object 1252. The direction filter 1270 may compare the values ​​of the image data to detect an edge direction 1252a perpendicular to the boundary of the first portion 1201b1 and the second portion 1201b2 as a third direction (45 degrees).

[0131] For example, data of the first portion 1201c1 and the second portion 1201c2 of the 4PD image sensor 1210 may be changed to a specified value or more by the third object 1253. The direction filter 1270 may compare the values ​​of the image data to detect an edge direction 1253a perpendicular to the boundary of the first portion 1201c1 and the second portion 1201c2 as a first direction (90 degrees).

[0132] For another example, data of the first portion 1201d1 and the second portion 1201d2 of the 4PD image sensor 1210 may be changed to a specified value or more by the fourth object 1254. The direction filter 1270 may compare the values ​​of the image data to detect an edge direction 1254a perpendicular to the boundary of the first portion 1201d1 and the second portion 1201d2 as a seventh direction (135 degrees).

[0133] The reordering unit 1280 may rotate or reorder the AF data for each direction (or the direction in which the edge is mainly set). The reordering unit 1280 may rotate or reorder the AF data in an edge direction 1251a, 1252a, 1253a, or 1254a perpendicular to the boundary of the first part 1201a1, 1201b1, 1201c1, or 1201d1 and the second part 1201a2, 1201b2, 1201c2, or 1201d2. Therefore, the PAF circuit 1235 may perform PAF calculation according to the directionality of the edge.

[0134] Although the PAF circuit 1235 performs a determined calculation without considering information about the directionality, it can perform PAF calculation according to the directionality of the edge to obtain a PAF result with high reliability. The PAF circuit 1235 can extract a statistical value for each direction and can perform AF determination.

[0135] Fig.13 1301 in a network environment 1300 according to various embodiments of the present disclosure. The electronic device according to various embodiments disclosed in this document may be various types of devices. The electronic device includes, for example, a portable communication device (e.g., a smart phone), a computer device (e.g., a personal digital assistant; PDA), a tablet PC, a laptop PC (desktop PC, workstation or server), a portable multimedia device (e.g., an e-book reader or an MP3 player), a portable medical device (e.g., a heartbeat measurement device, a blood sugar monitoring device, a blood pressure measurement device, and a body temperature measurement device), a camera, or at least one of a wearable device. The wearable device may include at least one of an accessory type (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD)), a fabric or clothing integration type (e.g., an electronic garment), a body attachment type (e.g., a skin pad or a tattoo), or a biological implant type (e.g., an implanted circuit). According to various embodiments, the electronic device may include, for example, at least one of a television (TV), a digital versatile disc (DVD) player, audio, an audio accessory device (such as a speaker, earphones, or headphones), a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a game console, an electronic dictionary, an electronic key, a camera, or an electronic photo frame.

[0136] In another embodiment, the electronic device may include at least one of a navigation device, a satellite navigation system (e.g., a global navigation satellite system (GNSS)), an event data recorder (EDR) (e.g., a black box of a car, ship, or airplane), a vehicle infotainment device (e.g., a head-up display of a vehicle), an industrial or domestic robot, a drone, an automated teller machine (ATM), a point of sale (POS), a measuring instrument (e.g., a water meter, an electric meter, or a gas meter), or an Internet of Things (e.g., a light bulb, a sprinkler, a fire alarm, a thermostat, or a street light). The electronic device according to an embodiment of the present disclosure may not be limited to the above-mentioned devices, and may provide the functions of multiple devices, such as a smartphone with a personal biometric information (e.g., heart rate or blood sugar) measurement function. In the present disclosure, the term "user" may refer to a person using an electronic device, or may refer to a device (e.g., an artificial intelligence electronic device) using the electronic device.

[0137] Fig.13 FIG. 1 is a diagram showing an electronic device 1301 (eg, Figure 1 A block diagram of a foldable electronic device 101 is shown. Fig.13 , the electronic device 1301 in the network environment 1300 may communicate with the electronic device 1302 via the first network 1398 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 1304 or the server 1308 via the second network 1399 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 1301 may communicate with the electronic device 1304 via the server 1308. According to an embodiment, the electronic device 1301 may include a processor 1320, a memory 1330, an input device 1350, a sound output device 1355, a display device 1360, an audio module 1370, a sensor module 1376, an interface 1377, a connection terminal 1378, a haptic module 1379, a camera module 1380, a power management module 1388, a battery 1389, a communication module 1390, a user identification module (SIM) 1396, or an antenna module 1397. In some embodiments, at least one of the components (e.g., the connection terminal 1378) may be omitted from the electronic device 1301, or one or more other components may be added to the electronic device 1301. In some embodiments, some components (e.g., the sensor module 1376, the camera module 1380, or the antenna module 1397) may be implemented as a single component (e.g., the display device 1360).

[0138] The processor 1320 may run, for example, software (e.g., program 1340) to control at least one other component (e.g., hardware component or software component) of the electronic device 1301 connected to the processor 1320, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 1320 may load a command or data received from another component (e.g., sensor module 1376 or communication module 1390) into the volatile memory 1332, process the command or data stored in the volatile memory 1332, and store the resultant data in the non-volatile memory 1334. According to an embodiment, the processor 1320 may include a main processor 1321 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 1323 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 1321 in operation. For example, when the electronic device 1301 includes a main processor 1321 and an auxiliary processor 1323, the auxiliary processor 1323 may be adapted to consume less power than the main processor 1321, or may be adapted to be specifically used for a specified function. The auxiliary processor 1323 may be implemented as a separate part from the main processor 1321, or as part of the main processor 1321.

[0139] When the main processor 1321 is in an inactive (e.g., sleep) state, the auxiliary processor 1323 (rather than the main processor 1321) may control at least some of the functions or states related to at least one component among the components of the electronic device 1301 (e.g., the display device 1360, the sensor module 1376, or the communication module 1390), or when the main processor 1321 is in an active state (e.g., running an application), the auxiliary processor 1323 may control at least some of the functions or states related to at least one component among the components of the electronic device 1301 (e.g., the display device 1360, the sensor module 1376, or the communication module 1390) together with the main processor 1321. According to an embodiment, the auxiliary processor 1323 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., a camera module 1380 or a communication module 1390) that is functionally related to the auxiliary processor 1323. According to an embodiment, the auxiliary processor 1323 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model can be generated by machine learning. For example, such learning can be performed by the electronic device 1301 where the artificial intelligence is executed or via a separate server (e.g., server 1308). The learning algorithm may include, but is not limited to, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN) or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or alternatively, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0140] The memory 1330 may store various data used by at least one component of the electronic device 1301 (e.g., the processor 1320 or the sensor module 1376). The various data may include, for example, software (e.g., program 1340) and input data or output data for commands related thereto. The memory 1330 may include a volatile memory 1332 or a non-volatile memory 1334.

[0141] The program 1340 may be stored as software in the memory 1330 , and the program 1340 may include, for example, an operating system (OS) 1342 , middleware 1344 , or an application 1346 .

[0142] The input device 1350 may receive commands or data from outside the electronic device 1301 (e.g., a user) to be used by other components of the electronic device 1301 (e.g., the processor 1320). The input device 1350 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0143] The sound output device 1355 can output a sound signal to the outside of the electronic device 1301. The sound output device 1355 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records, and the receiver can be used for incoming calls. Depending on the embodiment, the receiver can be implemented as a separate part from the speaker, or as a part of the speaker.

[0144] The display device 1360 may visually provide information to the outside of the electronic device 1301 (e.g., a user). The display device 1360 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display device 1360 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0145] The audio module 1370 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 1370 may obtain sound via the input device 1350, or output sound via the sound output device 1355 or an earphone of an external electronic device (e.g., electronic device 1302) directly (e.g., wired) connected to the electronic device 1301 or wirelessly connected thereto.

[0146] The sensor module 1376 may detect an operating state (e.g., power or temperature) of the electronic device 1301 or an environmental state (e.g., a state of a user) outside the electronic device 1301, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 1376 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0147] The interface 1377 may support one or more specific protocols to be used to connect the electronic device 1301 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 1302). According to an embodiment, the interface 1377 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0148] The connection end 1378 may include a connector, wherein the electronic device 1301 can be physically connected to an external electronic device (e.g., the electronic device 1302) via the connector. According to an embodiment, the connection end 1378 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0149] The haptic module 1379 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 1379 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0150] The camera module 1380 may capture a still image or a moving image. According to an embodiment, the camera module 1380 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0151] The power management module 1388 may manage power supply to the electronic device 1301. According to an embodiment, the power management module 1388 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0152] The battery 1389 may power at least one component of the electronic device 1301. According to an embodiment, the battery 1389 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0153] The communication module 1390 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 1301 and an external electronic device (e.g., electronic device 1302, electronic device 1304, or server 1308), and perform communication via the established communication channel. The communication module 1390 may include one or more communication processors capable of operating independently from the processor 1320 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 1390 may include a wireless communication module 1392 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1394 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 1398 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 1399 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 1392 can identify and authenticate the electronic device 1301 in a communication network (such as the first network 1398 or the second network 1399) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 1396.

[0154] The antenna module 1397 may transmit a signal or power to the outside of the electronic device 1301 (e.g., an external electronic device) or receive a signal or power from the outside of the electronic device 1301 (e.g., an external electronic device). According to an embodiment, the antenna module 1397 may include an antenna including a radiating element, the radiating element being composed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a PCB). According to an embodiment, the antenna module 1397 may include a plurality of antennas. In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 1398 or the second network 1399) may be selected from the plurality of antennas by, for example, the communication module 1390 (e.g., the wireless communication module 1392). A signal or power may then be transmitted or received between the communication module 1390 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as part of the antenna module 1397.

[0155] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0156] According to an embodiment, a command or data may be sent or received between the electronic device 1301 and the external electronic device 1304 via the server 1308 connected to the second network 1399. Each of the electronic device 1302 and the electronic device 1304 may be a device of the same type as the electronic device 1301, or a device of a different type from the electronic device 1301. According to an embodiment, all or some operations to be executed in the electronic device 1301 may be executed in one or more of the external electronic device 1302, the external electronic device 1304, or the server 1308. For example, if the electronic device 1301 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 1301 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 1301 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute the requested at least part of the function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 1301. The electronic device 1301 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, for example, cloud computing technology, distributed computing technology, or client-server computing technology may be used.

[0157] Fig.14 is a block diagram 1400 illustrating a camera module 1380 according to various embodiments.

[0158] Reference Fig.14, the camera module 1380 may include a lens assembly 1410, a flash 1420, an image sensor 1430, an image stabilizer 1440, a memory 1450 (e.g., a buffer memory), or an image signal processor 1460. The lens assembly 1410 may collect light emitted or reflected from an object whose image is to be captured. The lens assembly 1410 may include one or more lenses. According to an embodiment, the camera module 1380 may include a plurality of lens assemblies 1410. In this case, the camera module 1380 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies 1410 may have the same lens properties (e.g., viewing angle, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from those of another lens assembly. The lens assembly 1410 may include, for example, a wide-angle lens or a telephoto lens.

[0159] The flash 1420 may emit light, wherein the emitted light is used to enhance the light reflected from the object. According to an embodiment, the flash 1420 may include one or more light emitting diodes (LEDs) (e.g., red, green, blue (RGB) LEDs, white LEDs, infrared (IR) LEDs, or ultraviolet (UV) LEDs) or xenon lamps. The image sensor 1430 may acquire an image corresponding to the object by converting light emitted or reflected from the object and transmitted through the lens assembly 1410 into an electrical signal. According to an embodiment, the image sensor 1430 may include one image sensor selected from a plurality of image sensors having different properties (e.g., an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor), a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor 1430 may be implemented using, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0160] The image stabilizer 1440 may move the image sensor 1430 or at least one lens included in the lens assembly 1410 in a specific direction, or control an operational property of the image sensor 1430 (e.g., adjust the readout timing) in response to the movement of the camera module 1380 or the electronic device 1301 including the camera module 1380. In this way, at least a portion of the negative effects (e.g., image blur) generated due to the movement of the image being captured is allowed to be compensated. According to an embodiment, the image stabilizer 1440 may sense such movement of the camera module 1380 or the electronic device 1301 using a gyroscope sensor (not shown) or an acceleration sensor (not shown) arranged inside or outside the camera module 1380. According to an embodiment, the image stabilizer 1440 may be implemented as, for example, an optical image stabilizer. The memory 1450 may at least temporarily store at least a portion of the image acquired via the image sensor 1430 for subsequent image processing tasks. For example, if a plurality of images are captured quickly or image capture is delayed due to shutter lag, the acquired original image (e.g., Bayer pattern image, high-resolution image) may be stored in the memory 1450, and its corresponding copy image (e.g., low-resolution image) may be previewed via the display device 1360. Then, if a specified condition is satisfied (e.g., by a user's input or a system command), at least a portion of the original image stored in the memory 1450 may be acquired and processed by, for example, the image signal processor 1460. According to an embodiment, the memory 1450 may be configured as at least a portion of the memory 1330, or the memory 1450 may be configured as a separate memory that operates independently of the memory 1330.

[0161] The image signal processor 1460 may perform one or more image processing on an image acquired via the image sensor 1430 or an image stored in the memory 1450. The one or more image processing may include, for example, depth map generation, three-dimensional (3D) modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor 1460 may perform control (e.g., exposure time control or readout timing control) on at least one of the components included in the camera module 1380 (e.g., image sensor 1430). The image processed by the image signal processor 1460 may be stored back to the memory 1450 for further processing, or the image may be provided to an external component outside the camera module 1380 (e.g., memory 1330, display device 1360, electronic device 1302, electronic device 1304, or server 1308). According to an embodiment, the image signal processor 1460 may be configured as at least a part of the processor 1320, or may be configured as a separate processor that operates independently of the processor 1320. If the image signal processor 1460 is configured as a separate processor from the processor 1320, at least one image processed by the image signal processor 1460 may be displayed as it is by the processor 1320 via the display device 1360, or the at least one image may be displayed after being further processed.

[0162] According to an embodiment, the electronic device 1301 may include a plurality of camera modules 1380 having different properties or functions. In this case, at least one camera module 1380 of the plurality of camera modules 1380 may form, for example, a wide-angle camera, and at least another camera module 1380 of the plurality of camera modules 1380 may form a telephoto camera. Similarly, at least one camera module 1380 of the plurality of camera modules 1380 may form, for example, a front camera, and at least another camera module 1380 of the plurality of camera modules 1380 may form a rear camera.

[0163] According to various embodiments, an electronic device (eg, Figure 1 The electronic device 100 or Fig.13 The electronic device 1301 may include an image sensor including a plurality of pixels (eg, Figure 2 The image sensor 200 or Fig.14 sensor 1430), an AF processing unit for performing calculations for auto focusing based on phase differences (e.g., Figure 4 AF processing unit 450), memory (e.g., Figure 4 Memory 430, Fig.13 Memory 1330 or Fig.14 1450), and for transmitting the image data generated by the image sensor (e.g., Figure 2 Image sensor or Fig.14 The image sensor 1430) sends the data obtained by the AF processing unit or the memory (for example, Figure 4 Memory 430, Fig.13 Memory 1330 or Fig.14 1450) of the memory controller (e.g., Figure 4 Each pixel included in the plurality of pixels may include a plurality of photodiodes and a microlens for covering the plurality of photodiodes. The controller (eg, Figure 4 The controller 420 of the embodiment may provide a signal to the AF processing unit (eg, Figure 4 The AF processing unit (e.g., AF processing unit 450) provides at least one of first AF data of photodiodes arranged in a first direction among the plurality of photodiodes or second AF data of photodiodes arranged in a second direction among the plurality of photodiodes. Figure 4 The AF processing unit 450 may perform a first phase auto focus (PAF) calculation based on the first AF data, or may perform a second PAF calculation based on the second AF data.

[0164] According to various embodiments, an AF processing unit (e.g., Figure 4 The AF processing unit 450 may include a first AF circuit and a second AF circuit. The second AF circuit may perform a second phase auto focus (PAF) calculation, while the first AF circuit performs a first phase auto focus (PAF) calculation.

[0165] According to various embodiments, an AF processing unit (e.g., Figure 4 The AF processing unit 450 may include an AF circuit. The AF circuit may perform a first phase auto focus (PAF) calculation and may perform a second PAF when the first phase auto focus (PAF) calculation is completed.

[0166] According to various embodiments, a controller (e.g., Figure 4 The controller 420 of the embodiment may rotate the second AF data to correspond to the first direction, and may provide the second AF data to the AF processing unit (eg, Figure 4 AF processing unit 450).

[0167] According to various embodiments, an AF processing unit (e.g., Figure 4 The AF processing unit 450) may select one of the first direction or the second direction through a directional filter, and may perform PAF calculation on AF data corresponding to the selected direction.

[0168] According to various embodiments, a controller (e.g., Figure 4 The controller 420 of the embodiment may store the first AF data or the second AF data in a memory (eg, Figure 4 Memory 430, Fig.13 Memory 1330 or Fig.14 1450). A controller (e.g., Figure 4 The controller 420 of the embodiment may rotate or reorder and store at least one of the first AF data or the second AF data in a memory (eg, Figure 4 Memory 430, Fig.13 Memory 1330 or Fig.14 in memory 1450).

[0169] According to various embodiments, a controller (e.g., Figure 4 The controller 420 of the embodiment may provide a signal to the AF processing unit (eg, Figure 4 The AF processing unit 450) provides the first AF data or the second AF data having the Bayer pattern or the processing pattern.

[0170] According to various embodiments, an AF processing unit (e.g., Figure 4 The AF processing unit 450 may store the first AF data and the second AF data in a memory (eg, Figure 4 Memory 430, Fig.13 Memory 1330 or Fig.14 A first PAF calculation is performed during a process in memory 1450 of the processor, and a second PAF calculation may be calculated after the process ends.

[0171] According to various embodiments, when an image sensor (e.g., Figure 2 The image sensor 200 or Fig.14 When the image sensor 1430 of FIG. 1 reads out the first AF data and the second AF data, the AF processing unit (eg, Figure 4 The AF processing unit 450) may perform a first PAF calculation and may perform a second PAF calculation after the readout is completed.

[0172] According to various embodiments, an image sensor (e.g., Figure 2 The image sensor 200 or Fig.14 The image sensor 1430 of FIG. 1430 may be formed such that one microlens covers four photodiodes. Figure 4 The AF processing unit 450) can compare the measurements of the two photodiodes to perform a PAF calculation.

[0173] According to various embodiments, an electronic device (eg, Figure 1 The electronic device 100 or Fig.13 The electronic device 1301 may include an image sensor including a plurality of pixels (eg, Figure 2 The image sensor 200 or Fig.14 The image sensor 1430), wherein each pixel included in the plurality of pixels includes a plurality of photodiodes and a microlens for covering the plurality of photodiodes, an AF processing unit (e.g., Figure 4 AF processing unit 450) for performing calculations for performing auto-focus based on the phase difference; a memory (e.g., Figure 4 Memory 430, Fig.13 Memory 1330 or Fig.14 Memory 1450); and a controller (e.g., Figure 4 Controller 420) for transmitting the image signal received by the image sensor (eg, Figure 2 The image sensor 200 or Fig.14 The image sensor 1430) sends the data obtained by the AF processing unit or the memory (for example, Figure 4 Memory 430, Fig.13 Memory 1330 or Fig.14 1450). A controller (e.g., Figure 4 The controller 420 of the AF processing unit (eg, Figure 4 The AF processing unit 450) may perform PAF calculation based on priority.

[0174] According to various embodiments, an AF processing unit (e.g., Figure 4 The AF processing unit 450 may include a plurality of PAF circuits corresponding to a plurality of directions, respectively. Each of the plurality of PAF circuits may start PAF calculation at the same time.

[0175] According to various embodiments, an AF processing unit (e.g., Figure 4 The AF processing unit 450 may include an AF circuit. The AF circuit may perform PAF calculations in a priority order.

[0176] According to various embodiments, in an electronic device (eg, Figure 1 The electronic device 100 or Fig.13 The method for performing auto focus (AF) performed in the electronic device 1301 of the embodiment may include: Figure 2 The image sensor 200 or Fig.14The method comprises the steps of: obtaining image data using an image sensor 1430, obtaining first AF data of a photodiode arranged along a first direction among a plurality of photodiodes included in each pixel of the plurality of pixels based on the image data, obtaining second AF data of a photodiode arranged along a second direction among the plurality of photodiodes based on the image data, and performing a first phase autofocus (PAF) calculation based on the first AF data or performing a second PAF calculation based on the second AF data.

[0177] According to various embodiments, an electronic device (e.g., Figure 1 The electronic device 100 or Fig.13 The electronic device 1301 may include a first AF circuit and a second AF circuit. Performing the first PAF calculation or the second PAF calculation may include performing a first phase auto focus (PAF) by the first AF circuit while performing a second phase auto focus (PAF) by the second AF circuit.

[0178] According to various embodiments, an electronic device (e.g., Figure 1 The electronic device 100 or Fig.13 The electronic device 1301 may include an AF circuit. Performing the first PAF calculation or the second PAF calculation may include performing a first phase autofocus (PAF) using the AF circuit, and performing the second PAF calculation when the first phase autofocus (PAF) calculation is completed.

[0179] According to various embodiments, obtaining the second AF data may include rotating the second AF data to correspond to the first direction.

[0180] According to various embodiments, performing the first PAF calculation or the second PAF calculation may include, by an electronic device (eg, Figure 1 The electronic device 100 or Fig.13 The directional filter of the electronic device 1301) selects one of the first direction or the second direction, and performs PAF calculation on AF data corresponding to the selected direction.

[0181] According to various embodiments, the method of performing AF may further include storing the first AF data or the second AF data in an electronic device (eg, Figure 1 The electronic device 100 or Fig.13 of the electronic device 1301) (e.g., Figure 4 Memory 430, Fig.13 Memory 1330 or Fig.14 in memory 1450).

[0182] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0183] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms for corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “combined with another element (e.g., the second element)”, “combined to another element (e.g., the second element)”, “connected with another element (e.g., the second element)” or “connected to another element (e.g., the second element)” with or without the terms “operably” or “communicatively” being used, it means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0184] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0185] The various embodiments described herein may be implemented as software (e.g., program 1340) including one or more instructions stored in a storage medium (e.g., internal memory 1336 or external memory 1338) that can be read by a machine (e.g., electronic device 1301). For example, under the control of a processor, a processor (e.g., processor 1320) of the machine (e.g., electronic device 1301) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), but the term does not distinguish between data being semi-permanently stored in a storage medium and data being temporarily stored in a storage medium.

[0186] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., PlayStore). TM ) The computer program product may be published (e.g., downloaded or uploaded) online, or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a forwarding server).

[0187] According to various embodiments, each component (e.g., module or program) in the above-mentioned components may include a single entity or multiple entities, and some entities in the multiple entities may be separately arranged in different components. According to various embodiments, one or more components in the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each component in the multiple components in the same or similar manner as the corresponding one component in the multiple components before integration. According to various embodiments, the operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operations may be run or omitted in different orders, or one or more other operations may be added.

Claims

1. An electronic device, comprising: An image sensor, the image sensor comprising a plurality of pixels; an auto-focus AF processing unit configured to perform calculations for performing auto-focusing based on the phase difference; Memory; as well as at least one processor configured to deliver data obtained by the image sensor to the AF processing unit or the memory, Each of the plurality of pixels includes a plurality of photodiodes and a microlens configured to cover the plurality of photodiodes, wherein the at least one processor is configured to provide the AF processing unit with first AF data of the photodiodes arranged along the first direction among the plurality of photodiodes, wherein the at least one processor is configured to provide the AF processing unit with second AF data of the photodiodes arranged in the second direction among the plurality of photodiodes, wherein the second AF data is rotated or reordered to have a data structure in the first direction, and The AF processing unit performs a first phase auto focus (PAF) calculation based on the first AF data, and performs a second PAF calculation based on the second AF data.

2. The electronic device according to claim 1, wherein the AF processing unit comprises a first AF circuit and a second AF circuit, and in, While the first AF circuit performs a first phase auto focus (PAF) calculation, the second AF circuit performs a second phase auto focus (PAF) calculation.

3. The electronic device as claimed in claim 1, wherein the AF processing unit comprises an AF circuit, and The AF circuit performs a first phase auto focus (PAF) calculation, and performs a second PAF when the first phase auto focus (PAF) calculation is completed. 4 . The electronic device of claim 1 , wherein the AF processing unit selects the first direction through a direction filter and performs PAF calculation on AF data corresponding to the selected first direction. 5 . The electronic device of claim 1 , wherein the at least one processor is configured to store the first AF data or the second AF data in the memory. 6 . The electronic device of claim 1 , wherein the at least one processor is configured to provide the first AF data or the second AF data having a Bayer pattern or a processing pattern to the AF processing unit. 7 . The electronic device of claim 1 , wherein the AF processing unit performs the first PAF calculation in a process of storing the first AF data and the second AF data in the memory, and calculates the second PAF calculation after the process ends. 8 . The electronic device of claim 1 , wherein the AF processing unit performs a first PAF calculation while the image sensor reads out the first AF data and the second AF data, and performs a second PAF calculation after the readout ends.

9. The electronic device of claim 1, wherein the image sensor is formed such that one microlens covers four photodiodes, and The AF processing unit compares the measurement values ​​of the two photodiodes to perform PAF calculation.

10. The electronic device of claim 1, wherein the at least one processor is configured to set priorities based on measurement values ​​of photodiodes corresponding to a plurality of directions, and The AF processing unit performs PAF calculation based on the priority.

11. The electronic device of claim 10, wherein the AF processing unit comprises a plurality of PAF circuits corresponding to the plurality of directions, respectively, and Each of the plurality of PAF circuits starts the PAF calculation simultaneously.

12. The electronic device of claim 10, wherein the AF processing unit comprises an AF circuit, and The AF circuit performs PAF calculation in the order of priority.

13. A method for performing auto focus (AF) performed in an electronic device, the method comprising: obtaining image data using an image sensor comprising a plurality of pixels; obtaining first AF data of a photodiode arranged in a first direction among a plurality of photodiodes included in each of the plurality of pixels based on the image data; obtaining second AF data of photodiodes arranged in a second direction among the plurality of photodiodes based on the image data, wherein the second AF data is rotated or reordered to have a data structure in a first direction; as well as A first phase auto focus (PAF) calculation is performed based on the first AF data and a second PAF calculation is performed based on the second AF data.

Images