Electronic device

Abstract

An electronic device suppresses image quality degradation caused by artifacts. The electronic device includes a display unit, a camera unit disposed on the side opposite to the display surface of the display unit, and a signal processing unit. The camera unit includes a plurality of on-chip lenses and a plurality of pixels. The on-chip lenses include a first on-chip lens, and the plurality of pixels include a first pixel, which is disposed overlapping with the first on-chip lens. The first pixel has a plurality of photoelectric conversion units. The signal processing unit processes signals output from the plurality of pixels.

Description

Technical Field

[0001] This disclosure relates to electronic devices. Background Technology

[0002] In recent electronic devices such as smartphones, mobile phones, and PCs (Personal Computers), cameras are integrated into the bezel of the display to facilitate video calls and video recording. Since smartphones and mobile phones are often carried in pockets or bags, their form factor needs to be as compact as possible. On the other hand, if the display screen is small, even with a higher display resolution, the displayed characters become smaller and harder to read. Therefore, research is underway to maximize the display screen size without increasing the overall size of the electronic device by reducing the width of the bezel around the display.

[0003] However, since cameras and other devices are mostly mounted on the bezel of electronic devices, it's impossible to make the bezel width smaller than the camera's outer diameter. Furthermore, when a camera is mounted on the bezel, for example during a video call, the viewer's gaze is mostly focused near the center of the display screen, causing the gaze to deviate from the camera's optical axis, resulting in an unnatural and unnatural image. To avoid these problems, the following solution is proposed: Position the camera module on the side opposite the display surface of the device, and use the camera to capture the light passing through the display.

[0004] Existing technical documents

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2013-211413 Summary of the Invention

[0007] The technical problem that the invention aims to solve

[0008] However, since some of the light passing through the display unit will be reflected and diffracted and then incident on the camera, it will be affected by flare and diffraction caused by reflection, or by wiring arranged on the top of the camera (the display side of the monitor), etc., resulting in a problem of reduced image quality due to artifacts.

[0009] In one aspect of the invention, an electronic device is provided that can suppress image quality degradation caused by artifacts.

[0010] Solutions for solving technical problems

[0011] According to one embodiment, an electronic device includes a display unit; a camera unit disposed on the side opposite to the display surface of the display unit; and a signal processing unit. The camera unit includes multiple on-chip lenses and multiple pixels. The on-chip lenses include a first on-chip lens, and the multiple pixels each have a first pixel. The first pixel and the first on-chip lens are overlapped, and the first pixel has multiple photoelectric conversion units. The signal processing unit processes signals output from the multiple pixels.

[0012] Alternatively, the first pixel may obtain information about a predetermined color. For example, each pixel may obtain specified color information based on a color arrangement such as the Bayer arrangement.

[0013] Alternatively, the first pixel may have a filter.

[0014] Alternatively, the first pixel may have an organic photoelectric conversion film, which enables each photoelectric conversion unit belonging to the same first pixel to receive light of the same color. Therefore, specific color information can also be extracted using filters and organic photoelectric conversion films.

[0015] Alternatively, each pixel may have m×n (where m and n are integers greater than or equal to 2) photoelectric conversion units. For example, each pixel may also have 2×2 photoelectric conversion units, 3×3 photoelectric conversion units, 4×4 photoelectric conversion units, or 2×3 photoelectric conversion units.

[0016] The photoelectric conversion section is equipped with photoelectric conversion elements.

[0017] Alternatively, the photoelectric conversion element can be a photodiode.

[0018] Alternatively, the camera unit may have multiple first pixels colored by a Bayer array.

[0019] Alternatively, the lens could be an on-chip lens. Furthermore, it could also include microlenses on the chip.

[0020] Alternatively, an optical system different from a lens may be provided between the display unit and the camera unit.

[0021] Alternatively, the optical system could be a microlens array.

[0022] Alternatively, the signal processing unit may add the output values ​​of the photoelectric conversion units belonging to the same first pixel and use that sum as the pixel's output value.

[0023] Alternatively, in photoelectric conversion units formed in the same first pixel, if the output values ​​from each photoelectric conversion unit exceed a predetermined difference or a predetermined ratio, the signal processing unit corrects the output value of the first pixel.

[0024] Alternatively, the predetermined difference or predetermined ratio can be determined based on at least one of the following: individual differences of the photoelectric conversion unit, phase difference caused by the position of the photoelectric conversion unit, and noise generated in the photoelectric conversion unit.

[0025] Alternatively, the signal processing unit may use the pixel with the lowest output value in the photoelectric conversion unit of the first pixel to calculate the output value of the first pixel.

[0026] Alternatively, the signal processing unit may use the output value of the first pixel obtained by acquiring output values ​​of the same color around the first pixel to correct the output value of the first pixel.

[0027] Alternatively, when the subject includes a brightness level above a predetermined value, the signal processing unit corrects the first pixel.

[0028] Alternatively, the signal processing unit can acquire image data after flare correction based on the acquired signal.

[0029] Alternatively, the signal processing unit may determine whether flares are present in the captured image.

[0030] Alternatively, multiple camera units may exist at different locations on the display surface, and the signal processing unit may correct pixels belonging to areas determined to produce flares based on the output of the pixel areas corresponding to the camera units at different locations.

[0031] Alternatively, the signal processing unit can correct pixels belonging to regions identified as producing flares based on a learned model.

[0032] Alternatively, the signal processing unit may perform correction based on at least one of the following: the average value of the outputs of a plurality of photoelectric conversion units belonging to the first pixel in the acquired first pixel, the low sensitivity value among the outputs of the photoelectric conversion units belonging to the first pixel, and the value obtained by inputting the outputs of the plurality of photoelectric conversion units belonging to the first pixel into the learned model.

[0033] Alternatively, the signal processing unit may perform correction on the first pixel using a photoelectric conversion unit based on the circuitry or optical system of the display unit.

[0034] Alternatively, the signal processing unit may remove other noise points that are different from the first pixel after correcting the first pixel based on the circuitry or optical system of the display unit.

[0035] Alternatively, the display unit can be located on both sides of the device. Attached Figure Description

[0036] Figure 1 This is a schematic cross-sectional view of an electronic device according to one embodiment.

[0037] Figure 2 This is a schematic external view and cross-sectional view of an electronic device according to one embodiment.

[0038] Figure 3A This is a schematic cross-sectional view of an electronic device according to one embodiment.

[0039] Figure 3B This is a schematic cross-sectional view of an electronic device according to one embodiment.

[0040] Figure 4 This is a block diagram illustrating an outline of an electronic device according to one embodiment.

[0041] Figure 5 This is a schematic diagram illustrating a camera unit according to one embodiment.

[0042] Figure 6 This is a diagram schematically illustrating the camera pixels involved in one embodiment.

[0043] Figure 7 This is a diagram schematically illustrating the camera pixels involved in one embodiment.

[0044] Figure 8 This is a diagram showing light incident from a bright subject onto a camera unit according to one embodiment.

[0045] Figure 9 This is a diagram illustrating a camera pixel that receives light from a bright subject, according to one embodiment.

[0046] Figure 10 This is a flowchart illustrating the correction process involved in one embodiment.

[0047] Figure 11 This is a diagram illustrating noise correction according to one embodiment.

[0048] Figure 12 This is a flowchart illustrating the correction process involved in one embodiment.

[0049] Figure 13 This is a flowchart illustrating the correction process involved in one embodiment.

[0050] Figure 14 This is a cross-sectional view showing an example of the configuration of a camera unit according to one embodiment.

[0051] Figure 15 This diagram illustrates the image degradation of the camera unit caused by the wiring of the display unit.

[0052] Figure 16 This diagram illustrates the image degradation of the camera unit caused by the wiring of the display unit.

[0053] Figure 17 This is a schematic cross-sectional view of a camera module according to one embodiment.

[0054] Figure 18 This is a plan view of an electronic device according to one embodiment applied to a capsule endoscope.

[0055] Figure 19 This is a rear view of an electronic device implemented in one embodiment applied to a digital SLR camera.

[0056] Figure 20 This is a rear view of an electronic device according to one embodiment applied to a head-mounted display.

[0057] Figure 21 This is a rear view of an electronic device according to one embodiment applied to a head-mounted display.

[0058] Explanation of reference numerals in the attached figures

[0059] 1: Electronic device; 1a: Display screen; 1b: Bezel; 2: Display unit; 3: Camera module; 4: Display panel; 4a: Substrate; 40: Display pixel; 41: Wiring; 5: Circular polarizing plate; 6: Touch panel; 7: Protective glass; 8: Camera unit; 80: Camera pixel; 81: Lens; 82: Segmented pixel; 83: Shading; 9: Optical system; 10: Signal acquisition unit; 12: Signal processing unit; 14: Post-processing unit; 16: Output unit; 18: Control unit; 20: Storage unit. Detailed Implementation

[0060] The implementation of the electronic device will now be described with reference to the accompanying drawings. The main components of the electronic device will be described below, but it may also include components and functions not shown or described. The following description does not exclude components and functions not shown or described. Furthermore, dimensions, shapes, aspect ratios, etc., may be changed for the sake of explanation, but these will be appropriate in actual installation. Also, in the following description, the acquired signals are recorded as image information or video information, but this image information or video information is a broad concept and includes still images, moving images, or one frame of an image. Additionally, "larger" and "smaller" can be replaced with "above" and "below," respectively.

[0061] (First embodiment)

[0062] Figure 1 This is a schematic cross-sectional view of the electronic device according to the first embodiment. Figure 1The electronic device 1 is, for example, any device with display and shooting functions, such as a smartphone, mobile phone, tablet, or personal computer. The electronic device 1 includes a display unit 2 and a camera module 3. The camera module 3 is disposed inside the display surface of the display unit 2. That is, the camera module 3 passes through the display unit 2 to take pictures. In the following description of the embodiments, the camera module 3 may sometimes be described as being located below the display.

[0063] Figure 2 yes Figure 1 A schematic external view and cross-sectional view of electronic device 1. Figure 2 In the example, the display screen 1a is expanded to nearly the size of the electronic device 1, and the width of the bezel 1b surrounding the display screen 1a can also be set to, for example, less than a few millimeters. Typically, the front-facing camera is mostly mounted on the bezel 1b, but... Figure 2 As shown by the dashed line, the camera module 3, which serves as the front-facing camera, is positioned on the rear side within the display screen 1a. Therefore, by having the front-facing camera on the rear side of the display screen 1a, it is not necessary to position the front-facing camera on the bezel 1b, and the width of the bezel 1b can be reduced.

[0064] In addition, Figure 2 In this configuration, the camera module 3 is positioned on the rear side near approximately the center of the display screen 1a, but any location on the rear side of the display screen 1a is acceptable. For example, the camera module 3 can be positioned on the rear side near the periphery of the display screen 1a, or it can be positioned for applications such as fingerprint authentication. Figure 2 The rear side is located below the center of the display screen 1a. As described above, in this embodiment, the camera module 3 can be configured at any position overlapping with the display screen 1a. Furthermore, unless otherwise mentioned, "overlapping" in this disclosure means, for example, in... Figure 1 In this context, the areas that are common in the horizontal direction or the areas that exist in the horizontal direction are common, and the location is offset in the vertical direction. As an example, when the display surface of the display unit 2 is set to the top and the camera unit 8 is set to the bottom, this refers to a state where there is an offset in the vertical direction but no offset in the horizontal direction.

[0065] exist Figure 2 The device has a display unit 2 and a camera module 3 on one side, but is not limited to this. For example, the display unit 2 and the camera module 3 may also be provided on both sides of the device.

[0066] like Figure 1 As shown, the display unit 2, as a display optical system, is a structure formed by stacking the display panel 4, the circular polarizing plate 5, the touch panel 6, and the protective glass 7 in sequence. Furthermore, their arrangement is not limited to the above; they can be appropriately replaced, or two or more identical structures may exist.

[0067] The display panel 4 may include, for example, OLED (Organic Light Emitting Diode), liquid crystal, MicroLED, and other display principle-based components. OLED and other display panels 4 are composed of multiple layers. Display panels 4 often include components with low transmittance, such as filter layers. As described later, depending on the placement of the camera module 3, through-holes may be formed in the low transmittance components of the display panel 4. If light from the subject passing through the through-hole is incident on the camera module 3, the image quality captured by the camera module 3 can be improved.

[0068] A circular polarizing plate 5 is provided to reduce glare or improve the visibility of the display screen 1a even in bright environments. A touch sensor is installed in the touch panel 6. Touch sensors can be of various types, including capacitive, resistive, and pressure-sensitive types; any of these can be used. Alternatively, the touch panel 6 can be integrated with the display panel 4. A protective glass 7 is provided to protect the display panel 4 and other components.

[0069] The camera module 3 includes an image-capturing unit 8 and an optical system 9. The optical system 9 is disposed on the light-incident surface side of the image-capturing unit 8, that is, on the side closer to the display unit 2, so that the light passing through the display unit 2 is focused onto the image-capturing unit 8. The optical system 9 may include one or more lenses.

[0070] The imaging unit 8 includes multiple photoelectric conversion units. Each photoelectric conversion unit is equipped with a lens. These lenses allow the photoelectric conversion units constituting each pixel to receive light appropriately emitted from the optical system 9 and delivered to the imaging unit 8. The photoelectric conversion units perform photoelectric conversion on the light incident via the display unit 2. The photoelectric conversion units can be CMOS (Complementary Metal Oxide Semiconductor) sensors or CCD (Charge Coupled Device) sensors. Furthermore, the photoelectric conversion units can include photodiodes or organic photoelectric conversion films. The multiple photoelectric conversion units can be arranged in any manner. Examples of arrangements include Bayer arrangement, interlaced arrangement, patterned arrangement, striped arrangement, and other arrangements.

[0071] In this disclosure, the output value of the photoelectric conversion unit or the value obtained by a specified conversion based on the output value is referred to as the pixel value.

[0072] Figure 3A To explain in more detail Figure 1The diagram shows the relationship between the camera module 3 and the display panel 4. It is an example diagram showing the camera module 3. The camera module 3 includes, for example, an image-capturing unit 8 and an optical system 9. The optical system 9 is disposed on the light-incident surface side of the image-capturing unit 8, that is, on the side closer to the display unit 2. Light passing through the display surface of the display unit 2 is transmitted to the image-capturing unit 8 through the optical system 9.

[0073] The imaging unit 8 includes, for example, light-receiving elements such as photodiodes and photoelectric conversion elements. Light propagated by the optical system 9 through focusing, refraction, and diffusion is received by the photoelectric conversion unit of the imaging unit 8 and an analog signal is output. The photoelectric conversion unit may also include color filters, such as those arranged in a Bayer array, on the incident surface side of each imaging element, and may also include stacked color filters. Alternatively, an organic photoelectric conversion film may be used instead of a color filter. Furthermore, an item that can replace the filter used to acquire a color image may be included. In addition, although not shown, other components and circuits required for light reception and analog signal output may also be included. Furthermore, polarization elements may also be included.

[0074] The optical system 9 may also include an opening in the display panel 4, which is a through-hole in a low-transmittance component as described above. For example, the optical system 9 may include an opening in the display panel 4 and a lens positioned closer to the camera unit 8 than the opening. This opening may be located on a low-transmittance substrate 4a, and may also include a lens that allows light passing through the opening to propagate to the camera unit 8. For example, optical characteristics such as the numerical aperture (Na) and F-number of each camera module 3 are defined based on the lens and the opening. Furthermore, according to this optical system 9, the camera module 3 may further possess other optical characteristics, such as having different Abbe numbers.

[0075] Furthermore, the opening and lens are shown as examples, and the structure of optical system 9 is not necessarily limited to these combinations. Additionally, in the figures, one or more lenses are provided for each opening, but this is not a limitation. For example, as... Figure 3B As shown, multiple openings can be provided for a lens in the optical system 9. In areas where no openings exist, for example, light-emitting elements of the display panel 4 are provided, and openings can also be provided by stitching them between these light-emitting elements. By configuring the camera module 3 in this way, the camera module 3 can be provided without compromising the display.

[0076] As described above, multiple camera modules 3 can also be formed with different optical characteristics depending on the shape of the opening, the performance of the lens, etc. When there are two or more camera modules 3, each corresponding optical system 9 can have different optical characteristics. As another example, the camera modules 3 can be divided into multiple groups, each group having different optical characteristics. For example, the optical system 9 can have camera modules with the following characteristics by changing the shape and orientation of its opening or the material of the lens: two camera modules 3 with common optical characteristics and one camera module 3 with different optical characteristics.

[0077] like Figure 3A As indicated by the arrow, light incident from the display surface side of the display unit 2 is refracted by the optical system 9 and received by the camera unit 8. Where the optical system 9 is not provided, reflections can be appropriately suppressed, similar to a conventional display, and the display on the display unit 2 can be adjusted for viewing. For example, an opening is provided between the light-emitting pixels of the display panel 4, and a lens is provided in this opening on the side opposite to the display surface, through which light incident from the display surface side is emitted to the camera unit 8. Furthermore, openings can also be provided between consecutive light-emitting pixels. In other words, light-emitting pixels can also be configured to be positioned between openings.

[0078] Here, an example of the camera function of electronic device 1 is explained.

[0079] Figure 4 This is an example of a block diagram illustrating the structure related to the camera operation of the electronic device 1 according to this embodiment. The electronic device 1 includes a display unit 2, multiple camera modules 3, a signal acquisition unit 10, a signal processing unit 12, a post-processing unit 14, an output unit 16, a control unit 18, and a storage unit 20.

[0080] Similar to the figures described above, a camera module 3 is provided on the side opposite to the display surface of the display unit 2. Multiple camera modules 3 may also be provided for the display surface of one display unit 2. Each camera module 3 includes an image-capturing unit 8 and an optical system 9.

[0081] The signal acquisition unit 10 is a circuit that processes the analog signal output by the camera unit 8. The signal acquisition unit 10 includes, for example, an ADC (Analog to Digital Converter) and converts the input analog signal into digital image data.

[0082] The signal processing unit 12 acquires the captured image based on the digital image data converted by the signal acquisition unit 10. The imaging result is acquired based on the digital image data acquired from the camera module 3. More specifically, the signal processing unit 12 acquires, for example, the imaging result obtained by using the acquired image data to suppress artifacts such as flares generated in the camera module 3 through signal processing.

[0083] The post-processing unit 14 performs appropriate processing on the imaging results output by the signal processing unit 12 before outputting them. Appropriate processing may include image processing or signal processing such as pixel defect correction, edge enhancement, noise removal, brightness adjustment, color correction, white balance adjustment, distortion correction, and autofocus processing. Alternatively, the appropriate processing may be user-specified. Furthermore, a separate post-processing unit 14 is not required; for example, the signal processing unit 12 can also perform the aforementioned post-processing.

[0084] The output unit 16 outputs information to the outside of the electronic device 1. The output unit 16 may include an output interface, for example. This output interface may be an interface that outputs digital signals, such as USB (Universal Serial Bus), or a user interface such as a display. Furthermore, the output interface of the output unit 16 can also function as an input interface. Additionally, the output unit 16 may store data in its internal storage unit 20, thus the output can have a broader meaning.

[0085] The control unit 18 controls the processing in the electronic device 1. The control unit 18 may include, for example, a CPU (Central Processing Unit), and may also control the processing of the signal acquisition unit 10, signal processing unit 12, post-processing unit 14, and output unit 16. Furthermore, it may execute control of the camera module 3 to capture images based on the camera timing indicated from the user interface.

[0086] Storage unit 20 stores data in electronic device 1. Storage unit 20 may be, for example, a memory such as DRAM (Dynamic Random Access Memory) or a storage device such as SSD (Solid State Drive). Storage unit 20 may be built-in memory or a removable memory card. Furthermore, storage unit 20 is not necessarily located inside electronic device 1; it may be an external storage device connected via an input / output interface. Information is appropriately input to / output from storage unit 20 at the timing required by electronic device 1.

[0087] Some or all of the components described above can be formed on the same substrate. For example, the camera module 3, signal acquisition unit 10, signal processing unit 12, post-processing unit 14, output unit 16, control unit 18, and storage unit 20 can be formed on a single chip, or a portion of them can be formed on another chip. Furthermore, a portion of the configuration formed on the same substrate of a single chip can be stacked with a portion of the configuration formed on other substrates during the manufacturing process using technologies such as CoC (Chip on Chip), CoW (Chip on Wafer), and WoW (Wafer on Wafer).

[0088] Next, the structure of the camera unit 8 will be described. As an example, a Bayer arrangement will be used, but as mentioned above, the color configuration is not limited to a Bayer arrangement; other arrangements are possible as long as appropriate color information can be acquired. For example, it can be a Bayer arrangement based on the RGB series, or a Bayer arrangement based on the CMY complementary color system. It can also be a mixture of RGB and CMY. Furthermore, the pixel shape will be described as a square, but it is not limited to this. For example, it can be a rectangle other than a square, or a hexagonal honeycomb structure.

[0089] Figure 5 This is a schematic diagram illustrating the color information acquired by each imaging element of the imaging unit 8. As an example, Figure 5 The state of the 8×8 camera element provided by the camera unit 8 is shown. Of course, the camera element is not necessarily 8×8, but has an appropriate number based on the number of pixels to be acquired, the area that can be configured, etc.

[0090] The camera unit 8 includes camera pixels 80. As shown in the figure, the camera pixels 80 are arranged in an array in the camera unit 8. Each camera pixel 80 has a photoelectric conversion element on the side opposite to the display surface of the display unit 2, and a lens is provided on its upper part. The RGB values ​​stored within the camera pixels 80 represent colors, specifically R: red, G: green, and B: blue. As described above, this color differentiation method can be a filter provided on the display surface side of the photoelectric conversion element, or it can be a photoelectric conversion element formed of an organic photoelectric conversion film.

[0091] Each camera pixel 80 includes a lens 81. The lens 81 is, for example, an on-chip lens, and more particularly, an on-chip microlens. In this way, by equipping each pixel with a lens, the accuracy of light intensity, color reproduction, etc., can be improved. The lens 81 can be formed, for example, from Si, but can also be formed from other suitable materials.

[0092] One camera pixel 80 has multiple segmented pixels 82. For example, Figure 5Each camera pixel 80 has 2×2 segmented pixels 82. Thus, by having multiple segmented pixels in one pixel, as an example of the effect, the dynamic range between pixels can be expanded. This disclosure further utilizes the segmented pixels 82 to remove artifacts. As shown in one example, the camera pixel 80 has multiple segmented pixels 82, and each camera pixel 80 is arranged overlapping the lens 81 in a one-to-one correspondence.

[0093] Figure 6 This is an image extracted from one group of cameras with 80 pixels arranged in the Bayer array. For example... Figure 6 As shown, for example, the camera pixel 80 is configured to have four 2×2 segmented pixels 82. Each segmented pixel 82 is configured to have a photoelectric conversion element. This photoelectric conversion element may be a photodiode. Figure 1 The signal acquisition unit 10 acquires the analog signal output from the segmented pixel 82 equipped with a photoelectric conversion element. Then, the signal processing unit 12 performs signal processing on the output from the signal acquisition unit 10.

[0094] For example, the pixel value of the camera pixel 80 is calculated by adding the output values ​​obtained by the signal acquisition unit 10 from the segmented pixels 82 included in each camera pixel 80. This addition can be performed in the state of an analog signal or after being converted into a digital signal.

[0095] As an example, the signal acquisition unit 10 acquires analog signals from each segmented pixel 82 and converts the analog signals into digital signals. The signal processing unit 12 corrects the digital signals of each segmented pixel 82 output by the signal acquisition unit 10 as needed, and then adds them together to obtain the pixel value as the camera pixel 80.

[0096] Such pixel segmentation, as described above, is used in general digital cameras and the like to expand the dynamic range of pixels, i.e., each color, or to improve color reproduction. Furthermore, the signal processing unit can obtain the phase difference within the pixels by segmenting them. The imaging device can also use this result for autofocus, etc. In this disclosure, by employing this segmented pixel 82 in the camera module 3 below the display, artifacts caused by the display unit 2, etc., are also removed.

[0097] Figure 7 This is a diagram showing another example of segmented pixels 82 in camera pixels 80.

[0098] The camera pixel 80 may have, for example, 3×3 segmented pixels 82 as shown at the top, and may have a lens 81 above it.

[0099] The camera pixel 80 may, for example, have a 4×4 segmented pixel 82 as shown second from the top, and may have a lens 81 above it.

[0100] In addition, the camera pixel 80 may have, for example, a 3×4 segmented pixel 82 as shown at the bottom, and may have a lens 81 above it.

[0101] Therefore, integers m and n greater than 2 can be used to form an array of m×n segmented pixels 82. These settings can be appropriately configured according to the size and number of camera pixels 80, or the allowable forming area and volume on the chip, and the intended use. In the following description, the camera pixels 80 are set to have 2×2 segmented pixels 82, but this is shown as an example and does not exclude segmentation based on other values ​​as described above.

[0102] The following explains how to remove flare caused by a bright subject in the photograph.

[0103] like Figure 8 As shown, consider the case where light from a bright subject is incident on the imaging unit 8. In this situation, flare may occur in some or all of the pixels of the imaging pixels 80 of the imaging unit 8. In this embodiment, the generation of flare is suppressed by using segmented pixels 82.

[0104] Figure 9 This shows that the incident light comes from... Figure 8 The diagram shows a camera pixel 80 representing the light from a bright subject. However, although only one camera pixel 80 is shown, it is not limited to this, and the same processing is performed on multiple camera pixels 80 that may produce flare. Furthermore, a bright subject is defined as, for example, a light source with high brightness or illuminance, or an object that reflects light strongly after being illuminated by strong light. In addition, a bright subject includes, for example, objects with higher brightness or illuminance than their surroundings, which may cause flare.

[0105] The camera pixel 80 is composed of segmented pixels 82A, 82B, 82C, and 82D equipped with photoelectric conversion elements, and a lens 81 formed in a common manner with these photoelectric conversion elements. It is assumed that light from a bright subject is incident from the upper right direction in the figure.

[0106] In this situation, light is not uniformly emitted from lens 81 to all segmented pixels 82, but rather emitted with higher intensity in a direction relative to the direction of light propagation. This is due to the characteristics between lens 81 and the angle of incidence of light. For example, in Figure 9 In the case shown, the incident light on segmentation pixel 82C is stronger than that on other segmentation pixels 82. As a result, regarding the pixel values ​​of each segmentation pixel 82 acquired by the signal acquisition unit 10, segmentation pixel 82C is higher than that of other segmentation pixels 82A, 82B, and 82D.

[0107] Based on this phenomenon, the signal processing unit 12 suppresses the effect of flare that may occur in the camera pixel 80.

[0108] As described above, the phase difference in the camera pixel 80 can be obtained by using the segmented pixels 82. For example, the phase difference is determined based on whether there is an intensity difference greater than a predetermined threshold among the multiple segmented pixels 82 in a camera pixel 80. For example, if a pixel value higher than other pixels is obtained in segmented pixel 82C, and the difference between the pixel value of segmented pixel 82C and other segmented pixels 82A, 82B, and 82D is greater than a predetermined value, it is determined that a phase difference exists.

[0109] In cases where flare occurs, for example, if the pixel value of segmentation pixel 82C is higher than the pixel values ​​of other segmentation pixels 82A, 82B, and 82D, the same judgment is made. However, the threshold th related to the phase difference... p and the threshold th associated with flares f It is th p <th f The relationship is as follows. Compared to the threshold related to phase difference, the threshold related to flare is intentionally made higher; these thresholds are clearly distinguishable and can be preset. For example, the width of the sensitivity difference when focusing is inconsistent can be obtained in advance. This width can be set to the threshold value. p to th f To determine a predetermined threshold. Therefore, when correcting flare, the signal processing unit 12 can use the threshold th. f To make a judgment.

[0110] The pixel values ​​of the segmented images 82A, 82B, 82C, and 82D are xA, xB, xC, and xD, respectively. As an example, it is assumed that a relationship exists between them: xC > xD > xA > xB. These are merely illustrative examples, and the techniques disclosed herein are not limited to them.

[0111] Formula 1

[0112] xC-min{xA, xB, xD}=xC-xB>th f (1)

[0113] Formula 2

[0114]

[0115] For example, as shown in formula (1), the difference between the pixel value of segmentation pixel 82C with the highest pixel value and the lowest pixel value is compared with a threshold. If the difference is greater than the threshold, it can be determined that a flare may occur. In some cases, it can also be compared with the maximum value max{xA, xB, xD} instead of the minimum value min{xA, xB, xD}. Alternatively, as shown in formula (2), a ratio can also be used for judgment.

[0116] As another example, the average value can be considered.

[0117] Formula 3

[0118] xC-avg(xA, xB, xD)>th f (3)

[0119] Formula 4

[0120]

[0121] Here, avg() is a function that calculates the average value of the parameters. Even when considering the average value, either the difference or the ratio can be used for judgment. Alternatively, when taking the average value, considering the case where light is incident perpendicularly to the edge of the camera pixel 80, the average value of the two lower pixel values ​​among the segmented pixels 82A, 82B, and 82D can be taken as (xA+xB) / 2, for example, the average value of the two pixel values ​​of segmented pixels 82A and 82B.

[0122] In this way, by using a predetermined threshold as a predetermined difference and a predetermined ratio, the segmented pixel 82 with the largest pixel value in the camera pixel 80 can be subjected to formulas (1) to (4) or similar operations to determine whether there is a possibility of flare generation in the camera pixel 80.

[0123] In the above, the threshold for generating flares (predetermined difference, predetermined ratio) is determined based on a threshold related to the phase difference, but it is not limited to this. For example, a threshold for the individual differences of segmented pixels 82 and the influence of noise generated in segmented pixels 82 can also be set. When observing individual differences, for example, the imaging unit 8 can photograph a subject that does not generate flares, and the signal processing unit 12 can perform basic processing based on the image obtained from photographing the subject. That is, the signal processing unit 12 can also pre-determine the threshold of each segmented pixel 82 in each imaging pixel 80. The threshold for noise can also be determined in the same way, and further, the threshold after reducing the influence of various noises can be determined by multiple measurements.

[0124] In cases where flare may occur as described above, the signal processing unit 12 may also use segmentation pixels 82A, 82B, and 82D (excluding segmentation pixel 82C, which has a higher pixel value) to calculate the pixel value of the imaging pixel 80 and acquire image data corrected for flare. For example, the signal processing unit 12 may use the pixel value xB of the segmentation pixel 82B with the smallest pixel value and set the pixel value of the imaging pixel 80 to 4×xB. As another example, the signal processing unit 12 may use the pixel values ​​of the two segmentation pixels 82A and 82B with lower pixel values ​​and set them to 2×(xA+xB). The signal processing unit 12 may further use pixel values ​​other than the largest pixel value xC and set them to (4 / 3)×(xA+xB+xD). In this way, the signal processing unit 12 uses the pixel value of the segmentation pixel 82, which is the output of the imaging pixel 80 with the lowest output value, to calculate the pixel value of the imaging pixel 80.

[0125] Furthermore, the signal processing unit 12 can also perform flare correction using a trained model. For example, the signal processing unit 12 can also use a learned model to perform correction, which is a model trained by machine learning based on the generation and correction state of flares under various conditions. In this case, the signal processing unit 12 can input the individual values ​​of the segmented pixel 82 into the learned model to obtain the pixel value of the camera pixel 80. Moreover, as another example, the signal processing unit 12 can also use a trained model that further inputs the pixel values ​​of the surrounding camera pixels 80 of the same color.

[0126] For example, the model can also be a statistical model. The model is generated by statistically calculating which operation should be used to synthesize the various camera modules 3, and the signal processing unit 12 can also obtain images less affected by flares by inputting information obtained from multiple camera modules 3 into the model.

[0127] For example, the model can also be a neural network model trained through deep learning. The neural network model can be formed by MLP (Multi-Layer Perceptron), CNN (Convolutional Neural Network), etc. In this case, parameters trained in advance using multiple teaching data can be stored in the storage unit 20 or the signal processing unit 12, and the signal processing unit 12 forms a neural network model based on these stored parameters. By using the formed trained model, the signal processing unit 12 can acquire an image with suppressed flares by using data output from multiple camera modules 3.

[0128] Furthermore, when using a trained model, electronic device 1 can further improve the training accuracy by using the captured images. For example, training can be performed in the control unit 18 of electronic device 1. As another example, multiple electronic devices 1 can send data to a storage device such as the cloud, perform training on a server, and reflect the retrained parameters on electronic device 1. In this case, only information about the flares can be sent so as not to include privacy information containing the user's facial information. In addition, the sending / receiving of data from electronic device 1 can be configured, for example, to a state in which the user can select via input or output.

[0129] Therefore, the signal processing unit 12 can obtain flare-corrected images not only through linear processing but also through nonlinear processing, particularly by using operations of various models, including trained models.

[0130] Figure 10 This is a flowchart illustrating the processing flow involved in this embodiment. In this flowchart, the flare correction process is described, while processes such as image capture and data output are omitted.

[0131] First, the signal processing unit 12 acquires the output value of the segmented pixel 82 in each camera pixel 80 via the signal acquisition unit 10 (S100).

[0132] Next, the signal processing unit 12 determines whether the sensitivity difference among the output values ​​of the segmented pixels 82 belonging to the same camera pixel 80 is greater than the sensitivity difference in the phase difference (S102). That is, based on formulas (1) and (3), etc., the threshold th is used to determine whether the sensitivity difference is greater than the sensitivity difference in the phase difference. f This determines whether the sensitivity difference between the segmented pixel with the highest sensitivity and other segmented pixels in segmented pixels 82 is within a range that is not a phase difference. Alternatively, this process, such as the above formulas (2) and (4), can also be performed based on the ratio instead of the difference.

[0133] When the sensitivity difference is greater than th f At this time (S102: Yes), the signal processing unit 12 performs flare correction using the segmented pixel 82. Thereafter, the signal processing unit 12 acquires the flare-corrected value as the pixel value of the imaging pixel 80 (S106).

[0134] When the sensitivity difference is in th f In the following case (S102: No), the signal processing unit 12 performs normal pixel value acquisition processing, for example, acquiring the sum of the output values ​​of each segmented pixel 82 as the pixel value of the camera pixel 80 (S106).

[0135] As described above, according to this embodiment, when strong light, which is a cause of flare, is incident, pixel flare caused by the incident strong light can be suppressed by segmenting the pixels. In other words, the incident strong light, which is a cause of flare, can be concentrated in a portion of the segmented pixels, and the pixel values ​​can be corrected by using other segmented pixels, thereby obtaining pixel values ​​after suppressing the effect of flare.

[0136] (Second embodiment)

[0137] In the first embodiment described above, as an example, a segmented pixel 82 with the smallest pixel value among the camera pixels 80 was used. However, when using a segmented pixel 82 in this way, there is a strong possibility that it will be affected by noise. In this embodiment, a method to mitigate the impact of this noise will be described.

[0138] Figure 11 This diagram illustrates the noise correction method described in this embodiment. For example, in the camera pixel 80 that acquires the intensity of the green color located in the center, the case where flare correction is performed using a segmented pixel 82B (a segmented pixel with a right-diagonal line) located in the upper right corner will be described.

[0139] For example, when comparing pixel values ​​between a segmented pixel using formulas (1), (2), etc., the condition for satisfying these formulas is the influence of noise when noise is generated in segmented pixel 82B. For example, in the case of near-black noise in segmented pixel 82B, xC-xB may sometimes become a large value, which may be judged as a flare even if no flare is generated normally.

[0140] To suppress the impact of this noise, noise can be determined based on the following formula.

[0141] Formula 5

[0142] avg(xA, xC, xD) - xB > th n (5)

[0143] Here, th n This is a threshold used to determine noise. The signal processing unit 12 determines whether the difference between the average pixel value of segmented pixels 82A, 82C, and 82D and the pixel value of segmented pixel 82B used for flare correction is greater than this predetermined threshold. The threshold used to determine noise... n For example, it satisfies th f >th n >th p The threshold. In addition, noise determination is not limited to formula (5), as long as noise can be detected appropriately. For example, it can also be detected based on ratio as in formula (2).

[0144] When formula (5) is true, it is determined that noise has been generated in segmented pixel 82B, and noise correction processing can be performed.

[0145] For example, the signal processing unit 12 may also perform noise correction on the pixel values ​​after flare correction based on the pixel values ​​of the surrounding camera pixels 80 of the same color. The signal processing unit 12 may also use, for example, [the following text is incomplete and requires further context: "for example, it may use..."] Figure 11 The pixel value represented by the left diagonal line is corrected using methods such as bilinear interpolation and bicubic interpolation. For example, correction can be performed using one pixel value of the same color from the front, back, left, and right, but it is not limited to this. Information from the camera pixel 80 in the tilt direction can also be used, or information from two or more camera pixels 80 in the same direction can be used.

[0146] In addition, other information about the segmented pixels 82 is used in formula (5), but it is not limited to this. For example, xB can also be used. t Set it to the output value of the segmented pixel 82B of the current frame and consider it on the time axis.

[0147] Formula 6

[0148] avg(xB t-2 xB t-1 xB t+1 xB t+2 )-xB t >th n (6)

[0149] As shown in formula (6), noise detection can also be performed based on the difference between the average value and the average value between frames. This method shown in formula (6) is particularly effective when shooting still images. Alternatively, a ratio can be used for judgment instead of formula (6).

[0150] When using inter-frame information, it can also be used to perform noise correction. For example, the pixel value of camera pixel 80 can also be corrected using the following formula.

[0151] Formula 7

[0152] xB t =avg(xB t-2 xB t-1 xB t+1 xB t+2 (7)

[0153] In the above, information from two frames before and after was used in both noise determination and correction, but it is not limited to this. Information from one or more frames before and after can also be used.

[0154] In addition, the signal processing unit 12 can also perform noise correction by using the trained model described above.

[0155] Figure 12 This is a flowchart illustrating the process involved in this embodiment.

[0156] Processing from S200 to S204 Figure 10 The processing from S100 to S104 is the same, so its description is omitted.

[0157] After performing flare correction, or without performing flare correction, the next step is to determine whether the flare correction amount is greater than a threshold (S206). That is, based on formulas (5), (6), etc., the threshold value is used to determine whether the flare correction amount is greater than a threshold value (S206). n To determine whether the sensitivity difference between the segmented pixel used for flare correction and other segmented pixels in segmented pixel 82 is noise.

[0158] When the flare correction amount is greater than the threshold (S206: Yes), noise correction is performed (S208). The signal processing unit 12 performs noise correction, for example, using information in the spatial or temporal directions as described above. Furthermore, the correction method is not limited to the methods listed above, and appropriate methods may also be used.

[0159] After noise correction, or when the flare correction amount is in the th n In the following case (S206: No), the signal processing unit 12 acquires image data (S210).

[0160] As described above, according to this embodiment, the determination of whether the output value of the segmented pixel used for flare correction is noise is performed together with flare correction, and if it is determined to be noise, it can be obtained by performing correction in space and time. In particular, this method is effective when using one of the segmented pixels to perform flare correction, but it is also applicable in other cases. For example, xB in the above formulas (6) and (7) can also be used as the pixel value after flare correction.

[0161] Additionally, due to the noise threshold th n The threshold th varies depending on the amount of light received as a whole on the subject or the amount of signal received as a whole on the camera unit 8, and therefore can also be adaptively varied. That is, the signal processing unit 12 can also appropriately change the threshold th based on the surrounding conditions. n To perform the noise detection processing described above.

[0162] (Third Implementation)

[0163] In the first or second embodiment, flare correction is performed on all subjects, but it is not limited to this. For example, the signal processing unit 12 may also detect subjects that have generated flares and perform flare correction.

[0164] The signal processing unit 12 may, for example, detect subjects included in a subject with a brightness of a specified value or higher, and perform the aforementioned flare correction. Furthermore, as another example, the signal processing unit 12 may also detect the generation of flares based on the acquired pixel values ​​and perform the aforementioned flare correction.

[0165] Figure 13 This is a flowchart illustrating the flare correction process involved in this embodiment.

[0166] First, the signal processing unit 12 acquires information about the surrounding environment and state or image data for which no corrections such as flare correction have been performed (S300).

[0167] Then, the signal processing unit 12 determines, based on this information, whether there is a possibility of a flare occurring, or whether a flare is currently occurring (S302). For example, when an object with brightness and illuminance exceeding a predetermined threshold exists as the subject, the signal processing unit 12 determines that there is a possibility of a flare occurring. Additionally, it determines whether a flare is occurring based on factors such as whether there are areas of blank pixels in the acquired pixel values. These are merely examples; the determination can also be performed appropriately based on the probability of a flare occurring or a criterion that allows for the determination of flare occurrence.

[0168] When it is determined that there is a possibility of a flare occurring or that a flare is actually occurring (S302: Yes), the signal processing unit 12 performs flare correction and noise correction (S304). This subroutine is, for example, related to... Figure 10 The processes shown in S100 to S104, or Figure 12 The processing of S200 to S208 shown is the same as that of S208.

[0169] After performing flare correction and noise correction, or in the case where no flare is generated (S302: No), the signal processing unit 12 acquires the pixel value (S306).

[0170] As described above, according to this embodiment, flare correction is not always performed, and it can be omitted as needed. While there is a possibility of degradation compared to the original pixel values ​​after correction processing when flare correction is not required, by avoiding this processing, a more reproducible image can be obtained if there is no possibility of flare generation or if flare generation does occur. Furthermore, the operation of the signal processing unit 12 can be reduced.

[0171] (Fourth Implementation)

[0172] In this embodiment, the electronic device 1 is configured to weaken the generation of flares in any camera module 3, even when flares are generated.

[0173] Figure 14 This is a cross-sectional view showing the electronic device 1 according to this embodiment. Figure 14 As shown, the electronic device 1 includes multiple camera modules 3 on the display panel 4, that is, it has multiple optical systems 9 and multiple imaging units 8. For example, in the camera module 3 used to acquire images, when a flare is generated in a certain imaging pixel 80, attention should be paid to capturing imaging pixels 80 at the positions of the same subject that are located in different positions of the camera module 3. Due to the positional shift, the position of the flare in the captured image will also shift. Therefore, among the imaging pixels 80 representing the positions of the same subject, it is more likely that even if a flare is generated in one of them, no flare is generated in the other.

[0174] Based on this, flare correction can also be performed on the camera pixel 80 that is determined to produce flare by referring to the camera pixel 80 of another camera unit 8 at the same position of the subject. More specifically, flare correction of the camera pixel 80 can also be performed using data of the segmentation pixel 82 of another camera pixel 80 that corresponds to the segmentation pixel 82 of one camera pixel 80.

[0175] As described above, according to this embodiment, by setting multiple camera units 8 and using the data of the corresponding segmented pixels 82 of the corresponding camera pixels 80, the accuracy of flare correction can be improved.

[0176] Furthermore, a light-shielding part 30 may be provided between the camera modules 3 to prevent flare from being transmitted through the display panel 4, etc. The light-shielding part 30 may be, for example, a light-shielding film formed of a material with high light-shielding properties, or an absorption film formed of a material with high light absorption rate.

[0177] (Fifth Implementation)

[0178] Although the application of flare correction has been described in the above embodiments, the technology disclosed herein is not limited to this application. Above the camera module 3, as... Figure 1 As shown, various modules such as a display and a touch panel are installed. Therefore, a portion of the light incident on the camera unit 8 is sometimes blocked by these wiring components. The electronic device according to this embodiment is configured to correct pixel values ​​by segmenting pixels and signal processing when a portion of the light is blocked.

[0179] Figure 15 This diagram shows the shadow of the wiring formed on the camera unit 8 by shining light onto the display unit 2.

[0180] The diagram above schematically illustrates the display unit 2. For example, the display unit 2 includes multiple display pixels 40 and multiple wirings 41 for driving these display pixels 40. These wirings may include, for example, control lines and signal lines that cross or traverse the display. In the camera module 3 located below the display, the shadows cast by these wirings 41 can sometimes be problematic.

[0181] The diagram below schematically illustrates the camera unit 8. Solid lines represent the boundaries of the camera pixel 80, and dashed lines represent the boundaries of the segmented pixels 82. For example, when light is incident, as shown in the diagram below, the shadow 83 of the wiring 41 falls on the camera pixel 80. If such a shadow 83 exists, the amount of light incident in the shadowed area is reduced when the camera pixel 80 has individual photoelectric elements instead of using the segmented pixels 82, thus resulting in a decrease in the overall brightness that can be obtained.

[0182] Therefore, the reduction in brightness is suppressed by using segmented pixels 82. Even when using segmented pixels 82, if a shadow 83 is present, a reduction in brightness occurs because the sum of the values ​​of the segmented pixels 82 is used as the pixel value of the camera pixel 80. For example, in the top left camera pixel 80 in the diagram below, the output value from the segmented pixel 82 in the bottom right decreases, resulting in a reduction in the overall brightness of camera pixel 80. Additionally, in the camera pixel 80 adjacent to it on the right, the output value from the segmented pixel 8 in the lower half decreases, resulting in a reduction in the overall brightness of camera pixel 80.

[0183] Figure 16 This diagram illustrates the segmented pixels 82 whose brightness, i.e., the intensity of incident light, is reduced due to shadow 83. The segmented pixels 82 indicated by the diagonal lines are those affected by shadow 83; the value of the output signal in these segmented pixels 82 decreases, which is the cause of the reduced brightness of the imaging pixel 80. To avoid this effect, the reduction in the pixel value of the imaging pixel 80 caused by the output value from the segmented pixels 82 shown by the diagonal lines is calculated using other segmented pixels 82.

[0184] For example, when the signal processing unit 12 does not consider the lower right segmented pixel 82D in the segmented pixels 82 belonging to the camera pixel 80A, it calculates the brightness value x as x = (4 / 3) × (xA + xB + xC).

[0185] For example, when the signal processing unit 12 does not consider the lower half of the segmented pixels 82C and 82D in the segmented pixels 82 belonging to the camera pixel 80B, it calculates the brightness value x as x = 2 × (xA + xb).

[0186] For example, the signal processing unit 12 uses the lower right segmentation pixel 82D in the segmentation pixel 82 belonging to the camera pixel 80C to calculate the brightness value x as x = 4 × xD.

[0187] In summary, the signal processing unit 12 calculates the brightness value x by summing the gain set for each camera pixel 80 with the output values ​​from the usable segmented pixels 82. The gain is determined, for example, by the number of usable segmented pixels within the camera pixel 80. Through this calculation, the signal processing unit 12 can obtain the pixel value of the camera pixel 80 that has suppressed the influence of the shadow 83.

[0188] Furthermore, in this case, the impact of noise may become greater. To avoid this, the same noise correction as in the second embodiment described above can be performed. Alternatively, noise correction different from that in the second embodiment can also be performed. For example, in the camera pixel 80C, since the output is always based on the pixel value from the output of a segmented pixel 82D, the output value may be unstable. Therefore, for example, in the camera pixel 80C, unless otherwise specified, the signal processing unit 12 may also use surrounding camera pixels 80 of the same color to perform interpolation processing to reduce the impact of noise.

[0189] As described above, according to this embodiment, by using the region where the artifact is generated, the output value of the segmented pixel 82 that can be utilized by each camera pixel 80, and the gain, the influence of artifacts formed in the camera unit 8 by wiring and the like in the display unit 2 can be reduced.

[0190] In addition, along with this processing, flare correction and noise correction as described in the above embodiments can also be performed.

[0191] Thus, in the camera module 3 located below the display, by forming segmented pixels 82 for the imaging pixels 80 in the imaging unit 8, it is possible to correct artifacts caused by flares, shadows on the display, etc.

[0192] (Sixth Implementation Method)

[0193] In this embodiment, the electronic device 1 includes a microlens array as the optical system 9 of the camera module 3.

[0194] Figure 17 This diagram illustrates the camera module 3 according to this embodiment. The optical system 9 of the camera module 3 includes a microlens array 90. Light passing through the microlens array 90 is appropriately incident on the imaging unit 8 and converted into a signal and output in the imaging unit 8.

[0195] The signal acquisition unit 10 can also reconstruct the image based on the signal output from the imaging unit 8. The signal processing unit 12 acquires pixel values ​​that suppress the effects of artifacts in the above embodiments based on the reconstructed image.

[0196] As described above, the optical system 9 of the camera module 3, in addition to being configured to cover the entire structure, may also include a microlens array. Furthermore, Fresnel lenses, zone plates, etc., may also be used.

[0197] Here are a few application examples.

[0198] (Seventh Implementation)

[0199] Various electronic devices can be considered as specific candidates for the electronic device 1 having the configuration described in the above embodiments. For example, Figure 18 This is a plan view of the electronic device 1 of each embodiment applied to the capsule endoscope 50. Figure 18 The capsule endoscope 50, for example, includes within a housing 51 that is hemispherical at both ends and cylindrical in the center: a camera (miniature camera) 52 for capturing images inside the body cavity; a memory 53 for recording image data captured by the camera 52; and a wireless transmitter 55 for transmitting the recorded image data to the outside via an antenna 54 after the capsule endoscope 50 is ejected from the patient's body.

[0200] Additionally, a CPU 56 and a coil (magnetic force / current conversion coil) 57 are disposed within the housing 51. The CPU 56 controls the shooting operations performed by the camera 52 and the data storage operations to the memory 53, and controls the transmission of data from the memory 53 to a data receiving device (not shown) outside the housing 51 via the wireless transmitter 55. The coil 57 provides power to the camera 52, memory 53, wireless transmitter 55, antenna 54, and light source 52b (described later).

[0201] Furthermore, a magnetic (wire) switch 58 is provided in the housing 51 for detecting when the capsule endoscope 50 is placed on the data receiving device. The CPU 56 detects the placement of the wire switch 58 on the data receiving device and supplies power from the coil 57 to the wireless transmitter 55 when data can be transmitted.

[0202] The camera 52 includes, for example, an image sensor 52a and multiple light sources 52b. The image sensor 52a includes an optical system 9 for capturing images of the interior of the body cavity, and the multiple light sources 52b are used to illuminate the interior of the body cavity. Specifically, the light sources 52b of the camera 52 are, for example, composed of a CMOS sensor or CCD equipped with LEDs.

[0203] The display unit 2 of the electronic device 1 in the above embodiment includes, as follows: Figure 18 The concept of a light source like the light source 52b in the text. Figure 18The capsule endoscope 50 may have two light sources 52b, but these light sources 52b can be constructed using a display panel 4 with multiple light source units and an LED module with multiple LEDs. In this case, by arranging the imaging unit 8 of the camera 52 below the display panel 4 and the LED module, the constraints related to the layout and configuration of the camera 52 are reduced, and a smaller capsule endoscope 50 can be realized.

[0204] (Eighth Implementation Method)

[0205] in addition, Figure 19 This is a rear view of an electronic device 1 according to the above embodiments applied to a digital SLR camera 60. The digital SLR camera 60, a compact camera, has a display unit 2 on its rear side opposite to the lens for displaying a preview image. It is possible to configure the camera module 3 on the side opposite to the display surface of the display unit 2, thereby displaying an image of the photographer's face on the display screen 1a of the display unit 2. In the electronic device 1 of the above embodiments, since the camera module 3 can be configured in the area overlapping with the display unit 2, it is not necessary to place the camera module 3 in the outer frame of the display unit 2, allowing the size of the display unit 2 to be maximized.

[0206] (Ninth Implementation)

[0207] Figure 20 This is a plan view showing an example of applying the electronic device 1 of the aforementioned embodiment to a head-mounted display (hereinafter referred to as HMD) 61. Figure 20 The HMD61 is used for VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), or SR (Substitutional Reality), etc. For example... Figure 21 As shown, current HMDs are equipped with cameras 62 on their exterior, allowing HMD wearers to visually confirm the surrounding images. However, there is a problem that people around them cannot see the HMD wearer's eyes or facial expressions.

[0208] Therefore, in Figure 20 In this design, a display surface of the display unit 2 is provided on the outer surface of the HMD61, and a camera module 3 is provided on the side opposite to the display surface of the display unit 2. This allows the display surface of the display unit 2 to display the facial expressions of the wearer captured by the camera module 3, enabling people around the wearer to monitor the wearer's facial expressions and eye movements in real time.

[0209] exist Figure 20In the case shown, the camera module 3 is provided on the back side of the display unit 2, so the placement of the camera module 3 is not restricted, which can increase the freedom of the HMD61's appearance design. In addition, the camera can be positioned in an optimal position, thus preventing defects such as inconsistent viewing angles for the wearer on the display surface.

[0210] Thus, in this embodiment, the electronic device 1 of the above embodiment can be used for various purposes, and its utilization value can be improved.

[0211] Furthermore, this technology can be configured as described below.

[0212] (1) An electronic device, characterized in that it comprises: a display unit; a camera unit disposed on a side opposite to the display surface of the display unit; and a signal processing unit, the camera unit comprising: a plurality of on-chip lenses; and a plurality of pixels, the on-chip lenses having a first on-chip lens, the plurality of pixels having a first pixel, the first pixel being disposed overlapping with the first on-chip lens, the first pixel having a plurality of photoelectric conversion units, and the signal processing unit processing signals output from the plurality of pixels.

[0213] (2) The electronic device according to (1) is characterized in that the first pixel acquires information of a predetermined color.

[0214] (3) The electronic device according to (2) is characterized in that the first pixel has a filter.

[0215] (4) The electronic device according to (2) is characterized in that the first pixel has an organic photoelectric conversion film, wherein the organic photoelectric conversion film enables each photoelectric conversion unit belonging to the same first pixel to receive light of the same color.

[0216] (5) The electronic device according to any one of (1) to (4), characterized in that the first pixel has m×n photoelectric conversion units, wherein m and n are integers of 2 or more.

[0217] (6) The electronic device according to any one of (1) to (5), characterized in that the photoelectric conversion unit includes a photoelectric conversion element.

[0218] (7) The electronic device according to (6) is characterized in that the photoelectric conversion element is a photodiode.

[0219] (8) The electronic device according to (2) or (3) is characterized in that the camera unit has a plurality of first pixels colored by a Bayer arrangement.

[0220] (9) An electronic device according to any one of (1) to (8), characterized in that the lens is an on-chip lens.

[0221] (10) An electronic device according to any one of (1) to (9), characterized in that an optical system different from the lens is provided between the display unit and the camera unit.

[0222] (11) The electronic device according to (10) is characterized in that the optical system is a microlens array.

[0223] (12) The electronic device according to any one of (1) to (11) is characterized in that the signal processing unit adds the output values ​​of the photoelectric conversion units belonging to the same first pixel and uses them as the output value of the first pixel.

[0224] (13) An electronic device according to any one of (1) to (12), characterized in that, in the photoelectric conversion unit formed in the same first pixel, when the output value from each of the photoelectric conversion units exceeds a predetermined difference or a predetermined ratio, the signal processing unit corrects the output value of the first pixel.

[0225] (14) The electronic device according to (13) is characterized in that the predetermined difference or the predetermined ratio is determined based on at least one of the individual differences of the photoelectric conversion unit, the phase difference caused by the position of the photoelectric conversion unit, and the noise generated in the photoelectric conversion unit.

[0226] (15) The electronic device according to (13) or (14) is characterized in that the signal processing unit uses the pixel with the lower output value of the photoelectric conversion unit in the first pixel to calculate the output value of the first pixel.

[0227] (16) The electronic device according to any one of (13) to (15), characterized in that the signal processing unit corrects the output value of the first pixel by using the output value of the first pixel obtained by obtaining the same color output value around the first pixel.

[0228] (17) The electronic device according to any one of (13) to (16) is characterized in that, when a subject with a predetermined brightness or higher is included, the signal processing unit corrects the first pixel.

[0229] (18) The electronic device according to any one of (13) to (17), characterized in that the signal processing unit acquires image data after flare correction based on the acquired signal.

[0230] (19) The electronic device according to any one of (13) to (18) is characterized in that the signal processing unit determines that a flare is generated in the captured image.

[0231] (20) The electronic device according to (19) is characterized in that there are multiple camera units at different positions in the display surface, and the signal processing unit corrects pixels belonging to the region determined to produce flare based on the output of the pixel region corresponding to the camera units at different positions.

[0232] (21) The electronic device according to (19) or (20) is characterized in that the signal processing unit corrects pixels belonging to the region determined to produce flare based on a learned model.

[0233] (22) The electronic device according to any one of (13) to (21), characterized in that the signal processing unit performs correction based on at least one of the average value of the outputs of a plurality of photoelectric conversion units belonging to the first pixel in the acquired first pixel, a value with low sensitivity among the outputs of the photoelectric conversion units belonging to the first pixel, and a value obtained by inputting the outputs of the plurality of photoelectric conversion units belonging to the first pixel into a learned model.

[0234] (23) The electronic device according to (10) is characterized in that the signal processing unit performs correction on the first pixel using the photoelectric conversion unit based on the circuit of the display unit or the optical system.

[0235] (24) The electronic device according to (23) is characterized in that, when the signal processing unit corrects the first pixel based on the circuit of the display unit or the optical system, the signal processing unit removes other noise points that are different from the first pixel.

[0236] (25) The electronic device according to any one of (1) to (24), characterized in that the display unit is disposed on both sides of the device.

[0237] The methods disclosed herein are not limited to the various embodiments described above, but also include various modifications that can be conceived by those skilled in the art, and the effects of this disclosure are not limited to the above content. That is, various additions, changes, and partial deletions can be made without departing from the conceptual idea and spirit of this disclosure derived from the content specified in the claims and their equivalents.

Claims

1. An electronic device, characterized in that, have: Display section; A camera unit is disposed on the side opposite to the display surface of the display unit; and Signal processing unit The camera unit has: Multiple on-chip lenses; and Multiple pixels, The plurality of on-chip lenses includes a first on-chip lens. The plurality of pixels have a first pixel. The first pixel is configured to overlap with the lens on the first chip. The first pixel has multiple photoelectric conversion units. The signal processing unit processes the signals output from the plurality of pixels. In the photoelectric conversion units formed in the same first pixel, if the output values ​​from each of the photoelectric conversion units exceed a predetermined difference or a predetermined ratio, the signal processing unit corrects the output value of the first pixel. The signal processing unit performs correction based on at least one of the following: the average value of the outputs of the plurality of photoelectric conversion units belonging to the first pixel in the acquired first pixel, the lowest sensitivity value among the outputs of the photoelectric conversion units belonging to the first pixel, and a value obtained by inputting the outputs of the plurality of photoelectric conversion units belonging to the first pixel into a learned model. The predetermined difference or the predetermined ratio is defined based on the phase difference caused by the position of the photoelectric conversion unit in the same first pixel.

2. The electronic device according to claim 1, characterized in that, The first pixel acquires information about a predetermined color.

3. The electronic device according to claim 2, characterized in that, The first pixel has a filter or an organic photoelectric conversion film, wherein the organic photoelectric conversion film enables each photoelectric conversion unit belonging to the same first pixel to receive light of the same color.

4. The electronic device according to claim 1, characterized in that, The first pixel has m×n photoelectric conversion units, where m and n are integers greater than or equal to 2.

5. The electronic device according to claim 1, characterized in that, The photoelectric conversion unit includes a photodiode.

6. The electronic device according to claim 2, characterized in that, The camera unit has a plurality of first pixels colored by a Bayer arrangement.

7. The electronic device according to claim 1, characterized in that, The signal processing unit adds the output values ​​of the photoelectric conversion units belonging to the same first pixel, and uses this sum as the output value of the first pixel.

8. The electronic device according to claim 1, characterized in that, The predetermined difference or the predetermined ratio is determined based on at least one of the individual differences of the photoelectric conversion unit, the phase difference caused by the position of the photoelectric conversion unit, and the noise generated in the photoelectric conversion unit.

9. The electronic device according to claim 1, characterized in that, The signal processing unit uses the pixel with the lowest output value in the photoelectric conversion unit of the first pixel to calculate the output value of the first pixel.

10. The electronic device according to claim 1, characterized in that, The signal processing unit uses the output value of the first pixel obtained by acquiring the same color output values ​​around the first pixel to correct the output value of the first pixel.

11. The electronic device according to claim 1, characterized in that, When a subject with a predetermined brightness or higher is included, the signal processing unit corrects the first pixel.

12. The electronic device according to claim 1, characterized in that, The signal processing unit acquires image data after flare correction based on the acquired signal.

13. The electronic device according to claim 1, characterized in that, The signal processing unit determines whether flares are generated in the captured image and performs flare correction when flares are determined to be generated.

14. The electronic device according to claim 1, characterized in that, The signal processing unit performs correction on the first pixel using the photoelectric conversion unit based on the circuit of the display unit or the optical system provided between the display unit and the camera unit.

15. The electronic device according to claim 14, characterized in that, When the signal processing unit corrects the first pixel based on the circuit of the display unit or the optical system, the signal processing unit removes other noise points that are different from the first pixel.

16. The electronic device according to claim 1, characterized in that, The display unit is located on both sides of the device.

Images