Devices with cameras covered by the display

Abstract

A device includes a display stack, at least one camera sensor, one or more light emitting elements, a processor, and a memory. The display stack has display pixels. The at least one camera sensor is located near the display stack. The one or more light emitting elements are near the at least one camera sensor. The memory stores program instructions accessible by the processor. In response to execution of the program instructions, the processor is used to obtain display information related to the display pixels and modify the one or more light emitting elements based on the display information. The device avoids some difficulties in implementing a camera sensor located behind a display stack of an electronic device, such as difficulties caused by degradation of light capture of the camera sensor due to camera display pixels of the display stack located between the camera sensor and the scene.

Description

Technical Field

[0001] The present disclosure relates to a device for installing a camera behind a display. Background Art

[0002] Common smart phones, tablet computers, laptops and other mobile electronic devices include a display surface and a front camera on the front of the device. The display technology used in these mobile electronic devices includes a liquid crystal display (LCD) or an active-matrix organic light-emitting diode (AMOLED). Typically, the display surface does not occupy the entire front of the mobile electronic device because a frame, border or recess is required to accommodate the lens of the front camera. As a result, the size of the display surface is reduced. Given the limited area of ​​the front surface of the mobile electronic device, it is desirable to eliminate any frame, border or recess to increase the display area.

[0003] Front-facing cameras in mobile electronic devices can be used for selfies and video calls, among other applications. As the resolution of front-facing cameras continues to increase, the quality of images and videos captured by the front-facing cameras continues to improve. Such systems are considered camera-under-display (CUD) systems because the camera sensor is located below the display.

[0004] In a CUD system, there may be display artifacts from the camera area of ​​the display. Basically, the display includes openings in the display stackup surrounding the display pixels to let light pass through to the camera sensor located underneath. When the display is in normal use, without the camera in use, these openings are just black, making the appearance unsightly.

[0005] Currently, to address this issue, display pixels may be reduced to minimize any artifacts in the display from the camera display pixels in the CUD area. While reducing the camera display pixels in the CUD area improves artifacts, the reduced camera display pixels also typically reduce the life of the display due to wear and density reduction of the camera display pixels in the CUD area. Specifically, by changing the wear mechanism of the camera display pixels, different wear patterns will occur in the CUD area and the rest of the display. Summary of the invention

[0006] According to an embodiment of the present disclosure, a device is provided, which includes a display stack having display pixels. The device also includes a substrate. The substrate includes at least one camera sensor located near the display stack, and one or more light-emitting elements located near the at least one camera sensor. The device also includes a processor and a memory storing program instructions accessible to the processor. In response to the execution of the program instructions, the processor is used to obtain display information associated with the display pixel and modify one or more of the light-emitting elements based on the display information.

[0007] Optionally, the display stack includes an upper layer having the display pixels and a bottom cathode layer. The substrate is below the bottom cathode layer. In one aspect, the display pixels include camera display pixels located near the at least one camera sensor. In another aspect, the density of the camera display pixels is less than the density of non-camera display pixels. In one example, the camera display pixels define an opening used by the at least one camera sensor to obtain image data. Alternatively, the one or more light-emitting elements are used to emit auxiliary light output through the opening defined by the camera display pixels. In one embodiment, the one or more light-emitting elements are light-emitting diodes.

[0008] Optionally, the display information includes at least one of a color of a display pixel or a brightness of a display pixel. On the other hand, modifying the one or more light-emitting elements includes activating the one or more light-emitting elements, changing a color of an auxiliary light output from the one or more light-emitting elements, changing an intensity of an auxiliary light output from the one or more light-emitting elements, or deactivating at least one of the one or more light-emitting elements. In one example, modifying the one or more light-emitting elements includes matching a color of a light-emitting element in the one or more light-emitting elements with a color of a display pixel in the display pixel.

[0009] According to an embodiment of the present disclosure, a method is provided, wherein under the control of one or more processors including program instructions, the method includes emitting display light output using display pixels of a display stack. The method also includes sensing image data through an opening defined by a camera display pixel of the display pixel using at least one camera sensor located near the substrate of the display stack. The camera display pixel is located near the at least one camera sensor. The method also includes obtaining display information associated with the display light output, and modifying the one or more light elements located near the at least one camera sensor based on the display information to emit auxiliary light output through the opening defined by the camera display pixel.

[0010] Optionally, modifying the one or more light-emitting elements includes activating at least one of the one or more light-emitting elements. Alternatively, modifying the one or more light-emitting elements includes changing at least one of a color or a brightness of an auxiliary light output of the one or more light-emitting elements. In one aspect, modifying the one or more light-emitting elements includes deactivating at least one of the one or more light-emitting elements. In another aspect, obtaining the display information includes at least determining a brightness or a color of a display pixel in a display pixel. In one example, obtaining the display information includes obtaining the display information only from the camera display pixel.

[0011] According to an embodiment of the present disclosure, a device is provided, which includes a substrate. The substrate includes at least one camera sensor used to obtain image data, and a first light-emitting element located near the at least one camera sensor. The device also includes a processor and a memory storing program instructions accessible to the processor. In response to the execution of the program instructions, the processor is used to obtain display information associated with a camera display pixel of a display substrate near the at least one camera sensor, and modify the first light element based on the display information.

[0012] Optionally, the first light emitting element is a light emitting diode. In one aspect, the device includes a second light emitting element located near the at least one camera sensor, wherein the processor is further configured to modify the second light emitting element to be different from one of a color or a brightness of the first light emitting element based on the display information. In another aspect, the device includes a second light emitting element located near the at least one camera sensor, wherein the processor is further configured to activate the first light emitting element and deactivate the second light emitting element based on the display information. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a simplified schematic block diagram of an electronic device provided by an embodiment of the present disclosure.

[0014] Figure 2A It is a schematic diagram of an exemplary display stack provided by an embodiment of the present disclosure.

[0015] Figure 2B is another schematic block diagram illustrating an exemplary display stack provided by an embodiment of the present disclosure.

[0016] Figure 2C It is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure.

[0017] Figure 2D It is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure.

[0018] Figure 2EIt is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure.

[0019] Figure 2F It is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure.

[0020] Figure 3 It is a schematic diagram of a system for implementing a camera under a display screen provided by an embodiment of the present disclosure.

[0021] Figure 4 It is a schematic flowchart of a method for displaying an image on a camera display provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] It is readily understood that the components of the embodiments as generally described and illustrated in the drawings of the present disclosure may be arranged and designed in a variety of different configurations in addition to the described example embodiments. Therefore, the following more detailed description of the example embodiments as represented in the drawings is not intended to limit the scope of the embodiments as claimed, but is merely representative of example embodiments.

[0023] References throughout the specification to "one embodiment" or "an embodiment" (or the like) mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the phrases "in one embodiment" or "in an embodiment" etc. appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

[0024] In addition, the described features, structures or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to give a comprehensive understanding of the embodiments. However, those skilled in the relevant art will recognize that various embodiments can be practiced without one or more specific details, or using other methods, components, materials, etc. In other cases, well-known structures, materials or operations are not shown or described in detail in order to avoid confusion. The following description is intended to be used as an example only, and some example embodiments are simply described.

[0025] The present disclosure provides a system and method for improving the appearance of pictures on a display screen of an electronic device. The electronic device uses one or more light-emitting elements of a camera-under-display (CUD), such as a light-emitting diode (liquid-crystal display, LCD), which are positioned close to an opening defined by camera display pixels of a display stack in a CUD area. Each light-emitting element emits through the opening defined by the camera display pixels to fill the space created by the reduced density of the camera display pixels in the CUD area. The system then obtains display information to determine the characteristics of the image on the display, such as color, brightness, etc. Based on the display information, the auxiliary light output of the light-emitting element can be changed to match the characteristics of the display. In this way, the light-emitting element fills the opening defined by the camera display pixels in the CUD area to provide an aesthetically pleasing appearance that is maintained throughout the life cycle of the display.

[0026] The term "display pixel" shall mean any lit area on a display screen that is combined to form an image on the display screen. One lit area may be one picture element. The image formed by the combination of display pixels may be a solid color, a bar, an object, a person, a number, a letter, a background, a border, etc.

[0027] The term "camera display pixel" shall mean any and all display pixels that are proximate to a camera sensor. In this manner, the camera display pixel can block, partially block, or define an opening that allows light to pass through so that the camera sensor can obtain image data. Specifically, the camera display pixel includes the display pixel that can affect the degree of light capture by the camera sensor. The camera display pixel is only located in the vicinity of the camera sensor. Therefore, the camera display pixel only forms a portion of the display screen.

[0028] The term "camera sensor" shall refer to an area or element that obtains image data. The camera sensor may be on the substrate and obtain image data by receiving light reflected from an item being imaged. This includes receiving part of the image data as a result of the camera display pixels partially blocking light from being received by the camera sensor.

[0029] The term "image data" shall mean data associated with one or more of a two-dimensional (2-D) image, a three-dimensional (3-D) image, a panoramic image, a still image, and / or a video image.

[0030] The term "degree of light capture" shall mean the amount, level or degree of light captured by a camera sensor through one or more of a display, display layers, display pixels or sub-pixel color elements, as compared to light capture without any obstruction.

[0031] The term "display information" shall refer to any and all information, data, etc. related to or associated with the display and the display pixels. The display information may include the color of one or more display pixels, the brightness of one or more display pixels, the arrangement of display pixels, whether one or more display pixels are activated, whether one or more display pixels are deactivated, etc.

[0032] The term "auxiliary light output" shall mean any light provided at the display level by a light emitting element that is not a display pixel. The light emitting element may be on a substrate near the display screen, on a substrate including a camera sensor, on a display layer or coupled to a display layer, etc. The light emitting element may be a light emitting diode, a pixel, a current control, etc. The light emitting element may change color, change intensity or brightness, activate, deactivate, etc.

[0033] The terms "obtain" and "acquire" shall mean when used in conjunction with image data, other data, signals, information, etc., including at least one of the following: i) accessing a memory of an external device or remote server storing the image data, other data, signals, information, etc., ii) receiving the image data, other data, signals, information, etc. via a wireless communication link between the ICM and a local external device, and / or iii) receiving the image data, other data, signals, information, etc. on a remote server via a network connection. From the perspective of the ICM, the acquisition operation may include sensing new signals in real time, and / or accessing a memory to read stored data, signals, information, etc. from a memory within the ICM. From the perspective of the local external device, the acquisition operation includes receiving the data, signals, information, etc. at a transceiver of the local external device, wherein the data, signals, information, etc. are transmitted from the IMD and / or a remote server. The acquisition operation may be from the perspective of the remote server. For example, a network interface receives the image data, other data, signals, information, etc. from a local external device and / or directly from the IMD. The remote server may also obtain the image data, other data, signals, information, etc. from local memory and / or from other memory, such as in a cloud storage environment and / or from the memory of a workstation or an external programmer.

[0034] The terms "processor", "a processor", "one or more processors", and "the processor" shall refer to one or more processors. The one or more processors may be implemented by one or a combination of one or more implantable medical devices, wearable devices, local devices, remote devices, server computing devices, a network of server computing devices, and the like. The one or more processors may be implemented in a common location or a distributed location. The one or more processors may implement various operations described in the present disclosure in a serial or parallel manner, in a shared resource configuration, and the like.

[0035] Figure 1 An electronic device 100 is shown. The electronic device 100 may be implemented in conjunction with a device having a camera sensor located behind a display surface or display stack of the electronic device 100. For example, an electronic device having a borderless and / or notchless display surface may include a front-facing camera and / or a camera sensor placed behind a display stack of the electronic device. Embodiments in the present disclosure avoid some of the difficulties in implementing a camera sensor located behind a display stack of an electronic device, such as difficulties caused by degradation of light capture of the camera sensor due to camera display pixels of the display stack located between the camera sensor and the scene. The electronic device 100 may be fixed or portable / handheld.

[0036] The electronic device 100 includes a display screen 102 on which an image 103 is displayed. The image may include colors, patterns, pictures, letters, numbers, symbols, etc. Specifically, as will be described in further detail in the present disclosure, in one example, the display screen 102 is a display stack including a plurality of display pixels. Each display pixel individually provides a different display light output, such as color, brightness, etc. The display pixels are combined to provide a display light output. The display light output is provided to form the image on the display 102.

[0037] The display 102 includes a CUD area 104. At the CUD area 104, a plurality of camera sensors 106 are disposed below the display 102 on a substrate 108. The camera sensors 106 are used to obtain and capture image data through openings defined by display pixels. To this end, the display pixels close to the camera sensors 106 in the CUD area 104 are camera display pixels. In particular, the density of the camera display pixels is less than the density of non-camera display pixels to define more openings and open spaces for the camera sensors to obtain the image data.

[0038] A plurality of light-emitting elements 110 are also provided on the substrate 108, and are located near the opening defined by the camera display pixel. In one example, the light-emitting element 110 is a light-emitting diode. The light-emitting element 110 is used to be operated by one or more processors of the electronic device 100 to emit auxiliary light output to supplement the display light output. Specifically, the one or more processors in the example embodiment monitor the display to determine the display information generated by the display light output. The display information may include the color of the display pixel, the brightness of the display pixel, etc. To this end, in one example, the one or more processors monitor the color and brightness of the display pixel.

[0039] In another embodiment, the one or more processors monitor only the camera display pixels. And, the one or more processors modify the light emitting elements based on the display information to provide the auxiliary light output that matches the camera display pixel near each light emitting element 110. Such matching may include matching color, brightness, etc. By matching the camera display pixel with the auxiliary light output, the light emitting elements provide a more aesthetically pleasing presentation effect by alleviating the typical dark background produced at the CUD area 104.

[0040] like Figure 2A As shown, in one example, the display 102 includes a display stack 200. The display stack 200 includes an upper layer 201, and the upper layer 201 includes display pixels 202A arranged in a display matrix 204A. In one embodiment, the display matrix 204A is a first resolution. The display pixels 202A are separated by openings 208A defined by the display pixels 202A. In one example, the display pixels are separated by openings 208A arranged in rows 210A and columns 212A. A bottom cathode layer 214 is provided as the last layer of the display stack 200 and operates to supply power to the display pixels 202A.

[0041] Additionally or alternatively, such as Figure 2B As shown, the display stack 200 includes display sub-pixel color elements 202B arranged in the upper layer 201 in a display matrix 204B. In one example, the display matrix 204B has a first resolution. The display sub-pixel color elements 202B are separated by openings 208B. In one example, the display sub-pixel color elements are separated by openings 208B arranged in rows 210B and columns 212B. Each display sub-pixel color element 202B may include red, green, blue, yellow, white, or a similar primary color. Multiple sub-pixel color elements 202B cooperate to form a pixel, for example Figure 2AIn one example, display matrix 204B may be the sub-pixel color element equivalent to display matrix 204A.

[0042] For clarity, unless the context clearly indicates otherwise, pixel 202 refers to display pixel 202A and display sub-pixel color element 202B. Display matrix 204 refers to display matrix 204A and display matrix 204B. Opening 208 refers to opening 208A and opening 208B. Row 210 refers to row 210A and row 210B. Column 212 refers to column 212A and column 212B.

[0043] At least a portion of the display stack 200 covers the CUD area 104, such that a first subset of the camera sensor pixels of the sensor matrix are aligned with and covered by the camera display pixels 215, and a second subset of the camera sensor pixels of the sensor matrix are aligned with and exposed through the openings 208 defined by the camera display pixels 215. Specifically, the density of the camera display pixels 215 is less than the density of the non-camera display pixels, so as to provide more space for the camera sensor and the camera sensor pixels to acquire image data. In an example, the openings 208 may include covering and exposing at least one row of at least two rows of the camera sensor pixels, and may include covering and exposing one column of at least two columns of the camera sensor pixels. Optionally, a third subset of the camera sensor pixels of the sensor matrix may be partially covered by the camera display pixels 215 and partially exposed through the openings 208 between the camera display pixels 215. Each of the first, second, and third subsets of the camera sensor pixels has a different degree of light capture, depending on the amount by which each camera sensor pixel is covered or not covered by the corresponding adjacent camera display pixel 215.

[0044] In one example, when the camera display pixels 215 define openings 208 arranged in rows 210 and columns 212, the image data represents the vertical and horizontal strips of the scene corresponding to rows 210 and columns 212. The rows 210 and / or columns 212 opposite the openings 208 represent first and second data segments.

[0045] Figure 2C and 2DA schematic block diagram of an alternative embodiment of the electronic device 100 is shown. In this embodiment, the display 102 includes the display stack 200 at the CUD area 104. The display stack 200 in the example includes an upper layer having camera display pixels, and an opening defined by the camera display pixels. Similarly, the display stack may include a bottom cathode layer as described above. As shown, a camera sensor 106 is placed on a substrate 108 below the display 102. In one example, the camera sensor 106 is a camera sensor pixel. Alternatively, the camera sensor 106 includes a plurality of camera sensor pixels. In yet another example, a plurality of camera sensors 106 are spaced apart from each other. This spacing is equidistant in one example, and in another example, the spacing is not equidistant. In all examples, the opening defined by the camera display pixels allows light captured by the camera sensor 106 to pass therethrough.

[0046] In the embodiment of FIG. 2C , the substrate 108 includes a first light emitting element 110A and a second light emitting element 110B on either side and adjacent to the camera. In one example, the substrate 108 is a printed circuit board that provides current to the first light emitting element 110A and the second light emitting element 110B to control the operation of the first light emitting element 110A and the second light emitting element 110B. In one example, the light output of the first light emitting element 110A and the second light emitting element 110B are different. Specifically, the brightness depends on the current provided by the substrate 108. In this way, the intensity of the display light output is controlled and thus modified by controlling the current provided to the first light emitting element 110A and the second light emitting element 110B.

[0047] In another embodiment, the intensity of each of the first light emitting element 110A and the second light emitting element 110B does not change. Instead, the current provided only activates the first light emitting element 110A and the second light emitting element 110B. In such an embodiment, when the current flows to the first light emitting element 110A and the second light emitting element 110B, the first light emitting element 110A and the second light emitting element 110B are activated. When the current stops flowing to the first light emitting element 110A and the second light emitting element 110B, the first light emitting element 110A and the second light emitting element 110B are deactivated. In this way, the first light emitting element 110A and the second light emitting element 110B are modified by activating and deactivating each of the first and second light emitting elements 110A, 110B.

[0048] In yet another example, the first light emitting element 110A can be modified by activating the first light emitting element 110A, deactivating the first light emitting element 110A, increasing the intensity or light output of the first light emitting element 110A, decreasing the intensity or light output of the first light emitting element 110A, etc. Meanwhile, in this embodiment, the second light emitting element can be modified by activating the second light emitting element 110B and deactivating the second light emitting element 110B.

[0049] In one embodiment, the first light emitting element 110A is a first color or has a first color characteristic, and the second light emitting element 110B is a second color or has a second color characteristic. In one example, the first light emitting element 110A emits light of a red wavelength, and the second light emitting element 110B emits light of a blue wavelength. In such an example, based on the display information, if the camera display pixel near the camera sensor 106 in the display stack is red, only the first light emitting element 110A emits light. Meanwhile, the second light emitting element 110B does not emit light. Then, if the camera display pixel near the camera sensor 106 turns blue according to the display information, the first light emitting element 110A turns off, and the second light emitting element 110B turns on to emit blue light. Meanwhile, if the display information indicates that the camera display pixel near the camera sensor 106 turns black, if any one of the first light emitting element 110A or the second light emitting element 110B is activated to emit light output, the activated light emitting element is modified to be deactivated. In yet another example, if the display information indicates that the camera display pixel proximate to the camera sensor 106 is providing purple, both the first light emitting element 110A and the second light emitting element 110B are modified to be activated so that the combination of their auxiliary light outputs provides purple. In one such embodiment, the first light emitting element 110A can be a directional light emitting element, and the second light emitting element 110B can also be a directional light emitting element. Each of the first light emitting element 110A and the second light emitting element 110B is then directed to meet at an opening defined by the camera display pixel. In this way, the auxiliary light output of the first light emitting element 110A and the second light emitting element 110B is a combined light output of purple, provided by the adjacent camera display pixel defining the opening.

[0050] exist Figure 2DIn the exemplary embodiment of the present invention, the first light emitting element 110A and the second light emitting element 110B are not on the substrate 108. However, they are still located on either side of the camera sensor 106 and adjacent to the camera sensor 106. In this way, the first light emitting element 110A and the second light emitting element 110B emit light into the display stack 200 as needed to provide the auxiliary light output to the display 102. Nevertheless, it is still possible to Figure 2C Modifications to the first light emitting element 110A and the second light emitting element 110B are provided as described. In particular, the first light emitting element 110A and the second light emitting element 110B do not have to be located on the substrate. Instead, the first light emitting element 110A and the second light emitting element 110B only need to be placed in proximity to the camera display pixel so that light emitted by the corresponding light emitting element 110A and / or 110B is provided through the opening defined by the camera display pixel. To this end, the first light emitting element 110A and / or the second light emitting element 110 may be a directional light element, an angled light element, etc.

[0051] Figure 2E Another schematic block diagram of an example embodiment of the electronic device is shown. In this embodiment, the electronic device 100 also includes the display 102 having the display stack 200 in the CUD area 104. The display stack 200 in the example includes an upper layer. The upper layer includes the camera display pixels and the openings defined by the camera display pixels. Similarly, the display stack may include a bottom cathode layer as also described above. In addition, Figure 2E The electronic device 100 of the exemplary embodiment of comprises a camera sensor 106 placed on the substrate 108. In this embodiment, instead of the first light emitting element and the second light emitting element, a first subset 112A of light emitting elements and a second subset 112B of light emitting elements are provided.

[0052] The first subset 112A of light emitting elements includes a first light emitting element 114A, a second light emitting element 114B, and a third light emitting element 114C. Similarly, the second subset 112B of light emitting elements includes a first light emitting element 116A, a second light emitting element 116B, and a third light emitting element 116C. In an exemplary embodiment, the first light emitting element 114A of the first subset 112A of light emitting elements emits light of a red wavelength, the second light emitting element 114B of the first subset 112A of light emitting elements emits light of a green wavelength, and the third light emitting element 114C of the first subset 112A of light emitting elements emits light of a blue wavelength. Although the applicant will describe the characteristics of the first subset 112A of light emitting elements in the present disclosure, it is understood that the second subset 112B of light emitting elements can be composed and operated in the same manner as described with respect to the first subset 112A of light emitting elements.

[0053] In one example, when the first light emitting element 114A of the first subset 112A of the light emitting elements emits light of a red wavelength. The second light emitting element 114B of the first subset 112A of the light emitting elements emits light of a green wavelength, and the second light element 114B emits light of a green wavelength. The third light emitting element 114C of the first subset 112A of the light emitting elements emits light of a blue wavelength. Each light emitting element 114A-C is directional so that the emitted light output converges at an opening defined by the camera display pixel. In this way, based on the display information, in particular the color of a nearby camera display pixel, each light emitting element 114A-C can be modified to provide the color. Such modification may include deactivating one or more light emitting elements while activating other light emitting elements, activating one or more light emitting elements while activating other light emitting elements, activating and / or deactivating one or more light emitting elements and increasing the intensity of one or more light emitting elements, activating and / or deactivating one or more light emitting elements and decreasing the intensity of one or more light emitting elements, and the like. Specifically, by modifying the red, green, and blue light emitting elements in combination with one another, the light emitting elements 114A-C may more conveniently match the color and intensity of the nearby camera display pixels.

[0054] Figure 2F Another schematic block diagram of an example embodiment of the electronic device is shown. The schematic block diagram presents a top view of the electronic device 100. In this embodiment, the electronic device 100 also includes the display 102 having a display stack 200 in the CUD area 104. The display stack 200 in the example includes an upper layer with camera display pixels, and an opening defined by the camera display pixels. Similarly, the display stack may include a bottom cathode layer as described above.

[0055] also, Figure 2FThe electronic device 100 of the example embodiment includes a camera sensor 106 placed on the substrate. In this embodiment, a first light-emitting element 120A, a second light-emitting element 120B, a third light-emitting element 120C, and a fourth light-emitting element 120D are provided. Each of the light-emitting elements 120A-D is equidistant from each other, forming square dots around the opening defined by the camera display pixels. In other embodiments, the light-emitting elements 120A-D are not equidistant from each other. For example, the light-emitting elements 120A-D can form a rectangle around the camera sensor 106. Alternatively, additional light-emitting elements can be placed around the camera sensor 106. In one example, a circular pattern can be formed around the camera sensor 106.

[0056] In one example, each of the light elements 120A-D is directional and converges at the opening defined by the camera display pixel. In this example, the first light emitting element 120A emits a light output at a red wavelength, the second light emitting element 120B emits a light output at a green wavelength, the third light emitting element 120C emits a light output at a blue wavelength, and the fourth light emitting element 120D emits a white light output. In this manner, the first, second, and third light elements 120A-C may be configured as described with respect to Figure 2E The described combinations are activated and deactivated to provide auxiliary light output that matches the display light output of the camera display pixels proximate to the light emitting elements 120A-D.

[0057] Furthermore, when the display information indicates that the camera display pixel is emitting a white display light output, the fourth light emitting element 120D may be activated while the first, second, and third light emitting elements 120A-C are deactivated to provide white light through the opening defined by the camera display pixel. Furthermore, the white light of the fourth light emitting element may be used in combination with any and all of the first, second, and third light emitting elements 120A-C to clear or brighten any color provided by the first, second, and third light emitting elements 120A-C. In this manner, a variety of colors, hues, etc. may be provided by a combination of the light emitting elements 120A-D to match the camera display pixel proximate to the light emitting elements 120A-D.

[0058] Figure 3A system 300 is shown for implementing a camera below a display screen of an electronic device 302 according to an embodiment of the present disclosure. The system 300 includes an electronic device 302 that may be fixed or portable / handheld. The electronic device 302 includes, among other things, a processor 304, a local storage medium 306, and a graphical user interface (including a display) 308. The electronic device 302 also includes a CUD 310. In one example, the CUD 310 is a digital camera unit.

[0059] The electronic device 302 includes a housing 312 for housing a processor 304, a local storage medium 306, a user interface 308, a display stack 316 (eg, Figure 2A 301 and a CUD 310. The housing 312 includes a display surface 301 and a rear surface 303. A display stack 316 is located in the housing 312 near the display surface 301. In one embodiment, the display stack 316 includes an upper surface having a plurality of camera display pixels and a bottom cathode layer for powering the camera display pixels. The CUD 310 also includes camera sensors 314 and 315. The CUD 310 is oriented in the housing so that the camera sensors 314 and 315 of the CUD 310 face the display surface 301. The display stack 316 is located between the display surface 301 and the camera sensors 314 and 315 of the CUD 310.

[0060] Each of the camera sensors 314 and 315 has a respective field of view 322 and 323 and therefore a respective light capture degree. The CUD 310 operates under the control of the processor 304 to capture image data of a scene 326 in the field of view 322 and 323 of the corresponding camera sensors 314 and 315. The camera sensors 314 and 315 include camera sensor pixels to capture image data of the scene 326 in the field of view 322 and 323 of the camera sensors 314 and 315 of the CUD. The camera sensor pixels are arranged in a sensor matrix having a sensor resolution.

[0061] In one example, the camera sensors 314 and 315 may be identical to each other but spatially offset from each other to produce different views of the scene 326. In another example, the camera sensor 314 may determine the viewpoint of the CUD and the camera sensor 315 may have a lower resolution. The camera sensor 314 may measure image data of the scene, such as color data, depth data, infrared data, etc. Additionally or alternatively, the camera sensor 315 may be located above the display stack 316 and the camera sensor 314 may be located below the display stack 316.

[0062] As described in more detail above, the light emitting element 328 is disposed below the display stack 316. In particular, the light emitting element is placed near the opening defined by the camera display pixels in the CUD area. Specifically, the CUD area is the area where the CUD 310 is located in the electronic display. The CUD area includes all parts of the electronic device 302 that are perpendicular to the CUD 310. In this way, the portion of the display stack located above the CUD 310 is considered to be the CUD area of ​​the display. In addition, as described above, the display pixels in the CUD area of ​​the display are the camera display pixels, which define an opening through which light that allows the camera sensors 314 and 315 to obtain or capture image data passes.

[0063] In one example, each of the light emitting elements 328 is located below the CUD area of ​​the display to provide auxiliary light output through the opening defined by the camera display pixels. Additionally, the light emitting elements 328 in other embodiments are not below the CUD area of ​​the display. Instead, at least some, if not all, of the light emitting elements are not below the CUD area of ​​the display. Instead, the light emitting elements are directional, angled, etc., to emit light through the opening defined by the camera display pixels.

[0064] In yet another example, a portion of the light emitting elements 328 are located below the CUD area of ​​the display and another portion are not located below the CUD area of ​​the display. The light emitting elements below the CUD area of ​​the display and the light emitting elements not below the CUD area of ​​the display can both be directional, angled, etc. to emit wavelengths of light output through the opening defined by the camera display pixels. In an example embodiment, more than one light emitting element emits light through the same opening defined by the camera display pixels, including light that converges with each other. In this way, the light emitting elements can be controlled to provide multiple colors and intensities at a single opening defined by the camera display pixels.

[0065] Figure 4 A flow chart of a method 400 for displaying an image on a display screen of an electronic device is shown. In one example, Figure 1-3 The electronic device and system shown in are used to implement the method.

[0066] At 402, one or more processors operate display pixels of the display stack to emit display light output. The display light output can be any type of image, including pictures, letters, numbers, background colors, patterns, etc. The display pixels include display pixels near the CUD area. As described above, such display pixels are considered camera display pixels.

[0067] At 404, the one or more processors operate a camera sensor of a substrate located near the display stack to sense image data through an opening defined by the camera display pixels of the display pixels. In particular, while typically not in use, when an individual desires to utilize a camera function of the electronic device, the electronic device can activate a camera application and acquire image data, and specifically captured by the camera sensor. In one example, all of the camera sensors are aligned to be proximate to the opening defined by the camera display pixels. In other examples, at least some of the camera sensors are partially blocked by the camera display pixels. In other examples, at least some of the camera sensors are completely blocked by some of the camera display pixels.

[0068] At 406, the one or more processors obtain display information associated with the display light output. In one example, the display pixels are monitored to determine the color, brightness, etc. of each display pixel. In one example, the amount of current from a bottom cathode layer within the display stack to the display pixels is monitored. In another example, commands sent by the one or more processors to provide the display light output are monitored. In one embodiment, only the camera display pixels are monitored to determine a portion of the display light output provided in the CUD area.

[0069] At 408, it is determined whether one or more light emitting elements should be modified to provide auxiliary light output. Specifically, the display usually only displays a similar background, or a completely white or completely black background, etc. However, when an individual accesses an application such as a specific web page, image, meme, etc., the display light output may change. At 408, if it is determined that there is no change when monitoring the display information, the one or more processors continue to obtain display information associated with the display light output to determine whether there is such a change.

[0070] If it is determined at 408 that a change has occurred in the display light output that requires auxiliary light output, the one or more processors modify one or more light emitting elements located near the camera sensor to emit auxiliary light output through the opening defined by the camera display pixels based on the display information at 410. Modifying the light emitting elements may include activating the one or more light emitting elements, changing a color of the auxiliary light output emitted from the one or more light emitting elements, changing an intensity of the auxiliary light output emitted from the one or more light emitting elements, deactivating the one or more light emitting elements, and the like.

[0071] For example, when the CUD area has a black display light output, and a new display light output is to provide white in the CUD area, all light emitting elements near the camera display pixel are activated. Alternatively, in one embodiment, where multiple light emitting elements can emit light of different colors through a certain opening defined by the camera display pixel, only light emitting elements that produce white light are activated, while light emitting elements that produce light of other colors are deactivated. In another alternative embodiment, light emitting elements that can combine to form white light can be activated, while light emitting elements that do not combine to form white light are deactivated.

[0072] In another example, in the case where the camera display pixel is providing white light and all the light emitting elements are activated, when a change in the display light output is provided and a completely black picture is provided in the CUD area, all the light emitting elements are deactivated. Similarly, only a portion or one of the light emitting elements is deactivated based on the received display information.

[0073] In another embodiment, the light emitting elements are modified by changing the characteristics of one or more light emitting elements. For example, the one or more light emitting elements can increase or decrease the brightness or intensity. In another example, the auxiliary light output can be changed. In this way, if in the CUD area, the picture of the camera display pixels shows the orange and red parts of the picture, the light emitting elements close to the red camera display pixels provide red light output, and the light emitting elements close to the orange camera display pixels provide orange light output. The orange light output can be provided directly by the light emitting element emitting the orange light output, or by two or more light emitting elements emitting light outputs that converge to form the orange light output. As a result, the auxiliary light output is modified accordingly.

[0074] In yet another example, the light emitting element emits only white light, and therefore, when the display information indicates that the light color is provided by the camera display pixel near the one or more light emitting elements, the one or more light emitting elements emit the white light. Therefore, even if the colors do not match, a more aesthetically pleasing display is still provided by providing white light rather than no light. Similarly, when the display information indicates that a darker color is provided, the light emitting element may be deactivated. Although not completely matching the display, the appearance of the display is more aesthetically pleasing than when the white light is emitted. In particular, not emitting light more closely matches the camera display pixel near the light emitting element. In this way, a user is always provided with an aesthetically pleasing image, thereby enhancing the enjoyment of the electronic device. In addition, the chance of the CUD area becoming a darkened area is reduced over time, even if it is not eliminated.

[0075] Before summarizing, it should be understood that although, for example, software applications for implementing embodiments of the present disclosure may be sold together with devices such as system 100, embodiments of the present disclosure are applicable to situations where such applications are downloaded to the device from a server via a network (e.g., the Internet), for example. In addition, embodiments of the present disclosure are applicable to situations where, for example, such applications are included on a computer-readable storage medium that is being sold and / or provided, where the computer-readable storage medium itself is not a carrier wave or signal.

[0076] As will be appreciated by those skilled in the art, various aspects may be embodied as systems, methods or computer (device) program products. Therefore, various aspects may take the form of a complete hardware embodiment or an embodiment comprising hardware and software. These hardware and software may be collectively referred to as "circuits", "modules" or "systems" in the present disclosure. In addition, various aspects may take the form of a computer (device) program product embodied in one or more computer (device) readable storage media having a computer (device) readable program code.

[0077] Any combination of one or more non-signal computer (device) readable media may be used. The non-signal medium may be a storage medium. The storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any suitable combination of the foregoing. More specific examples of storage media include: a portable computer floppy disk, a hard disk, a random access memory (RAM), a dynamic random access memory (DRAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0078] The program code for performing the operation can be written with any combination of one or more programming languages. The program code can be performed completely on a single device, partially on a single device, as an independent software package, partially on a single device and partially on another device, or completely on another device. In some cases, the device can be connected by any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected (for example, using an Internet service provider) or by hard-wire connection, such as by USB connection, by other devices. For example, a server with a first processor, a network interface and a storage device for storing code can store a program code for performing the operation, and the code is provided to a second device with a second processor via a network by its network interface for executing code on a second device.

[0079] The units / modules / applications in the present disclosure may include any processor-based or microprocessor-based system, including the use of microcontrollers, reduced instruction set computers (RISC), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), logic circuits, and any other circuits or processors capable of performing the functions described in the present disclosure. Additionally or alternatively, the units / modules / controllers herein may represent circuit modules that can be implemented as hardware with associated instructions (e.g., software stored on a tangible and non-temporary computer-readable storage medium, such as a computer hard drive, ROM, RAM, etc.) to perform the operations described herein. The above examples are merely exemplary and are therefore not intended to limit the definition and / or meaning of the term "controller" in any way. The units / modules / applications herein may execute a set of instructions stored in one or more storage elements in order to process data. The storage elements may also store data or other information as needed or required. The storage elements may be in the form of element information sources or physical memory elements in the modules / controllers of the present disclosure. The set of instructions may include various commands that instruct the units / modules / applications herein to perform specific operations, such as the methods and processes of various embodiments of the subject matter described herein. The set of instructions may be in the form of a software program. The software may be in various forms such as system software or application software. In addition, the software may be in the form of a collection of separate programs or modules, a program module within a larger program, or a portion of a program module. The software may also include modular programming in the form of object-oriented programming. The processing of input data by the processing machine may be a response to a user command, or a response to a result of a previous process, or a response to a request made by another processing machine.

[0080] It should be understood that the subject matter described in the present disclosure is not limited to the details of the construction and arrangement of the components set forth in the description of the present disclosure or shown in the drawings of the present disclosure. The subject matter described in the present disclosure can have other embodiments and can be practiced or executed in various ways. In addition, it should be understood that the words and terms used in the present disclosure are for descriptive purposes and should not be considered restrictive. "Including," "including," or "having" and their variations used herein are intended to cover the items listed thereafter and their equivalents and additional items. In addition, in the following claims, the phrases "at least A or B," "A and / or B," and "one or more of A and B" (where "A" and "B" represent claim elements) are used to cover i) A, ii) B, and / or iii) A and B.

[0081] It should be understood that the above description is illustrative, not restrictive. For example, the above embodiments (and / or aspects thereof) may be used in combination with each other. In addition, many modifications may be made to adapt specific situations or materials to the teachings of the present disclosure without departing from its scope. Although the dimensions, material types, and coatings described in the present disclosure are intended to define various parameters, they are by no means restrictive and are illustrative in nature. After reading the above description, many other embodiments will be apparent to those skilled in the art. Therefore, the scope of the embodiments should be determined with reference to the appended claims and the full scope of equivalents to which these claims are entitled. In the appended claims, the terms "including" and "wherein" are used as simplified descriptions of the corresponding terms "comprising" and "characterized by". In addition, in the following claims, the terms "first", "second", and "third", etc., are used only as labels and are not intended to impose numerical requirements on their objects or to impose an execution order on their actions.

Claims

1. A device comprising: a display stack having display pixels; at least one camera sensor located proximate to the display stack; one or more light emitting elements proximate to the at least one camera sensor; processor; as well as a memory for storing program instructions accessible by the processor, wherein in response to execution of the program instructions, the processor is configured to: Acquiring display information related to the display pixel; and modifying the one or more light emitting elements based on the display information, characterised in that modifying the one or more light emitting elements comprises matching a colour of one of the one or more light emitting elements with a colour of one of the display pixels, and Characterized in that the display stack includes an upper layer having the display pixels and a bottom cathode layer above a substrate.

2. The device according to claim 1, characterized in that The at least one camera sensor is coupled to the substrate.

3. The device according to claim 1, characterized in that The display pixels include camera display pixels located proximate to the at least one camera sensor.

4. The device according to claim 3, characterized in that The camera display pixels define openings used by the at least one camera sensor to acquire image data.

5. The device according to claim 4, characterized in that The one or more light emitting elements are configured to emit auxiliary light output through the opening defined by the camera display pixels.

6. The device according to claim 1, characterized in that The one or more light emitting elements are light emitting diodes.

7. The device according to claim 1, characterized in that The display information includes at least one of a color of a display pixel and a brightness of a display pixel.

8. The device according to claim 1, characterized in that Modifying the one or more light emitting elements includes at least one of activating the one or more light emitting elements, changing an intensity of an auxiliary light output emitted from the one or more light emitting elements, or deactivating the one or more light emitting elements.

9. A method comprising: emitting a display light output using display pixels of a display stack, wherein the display stack includes an upper layer having the display pixels and a bottom cathode layer above a substrate; sensing, with at least one camera sensor located proximate to the display stack, image data through openings defined by camera display pixels of the display pixels, wherein the camera display pixels are located proximate to the at least one camera sensor; acquiring display information related to the display light output; and modifying one or more light emitting elements located proximate to the at least one camera sensor to emit auxiliary light output through the opening defined by the camera display pixels based on the display information, Characterized in that modifying the one or more light-emitting elements includes deactivating at least one of the one or more light-emitting elements.

10. The method according to claim 9, characterized in that Modifying the one or more light emitting elements includes activating at least one of the one or more light emitting elements.

11. The method according to claim 9, characterized in that Modifying the one or more light emitting elements includes changing at least one of a color or a brightness of the auxiliary light output of the one or more light emitting elements.

12. The method according to claim 9, characterized in that Obtaining the display information includes determining at least a brightness or a color of one of the display pixels.

13. The method according to claim 9, characterized in that Obtaining the display information includes obtaining the display information only from the camera display pixels.

14. A device comprising: a substrate including at least one camera sensor for acquiring image data; a first light emitting element located near the at least one camera sensor; a second light emitting element located proximate to the at least one camera sensor; A display stack including an upper layer having display pixels and a bottom cathode layer above the substrate processor; as well as A memory storing program instructions accessible by the processor, wherein in response to execution of the program instructions, the processor is configured to: obtaining display information associated with camera display pixels of the substrate proximate the at least one camera sensor; modifying the first light emitting element based on the display information, and The second light emitting element is modified to have a different color and / or a different brightness than the first light emitting element based on the display information.

15. The device according to claim 14, characterized in that The first light emitting element is a light emitting diode.

16. The device of claim 14, wherein the processor is further configured to activate the first light emitting element while deactivating the second light emitting element based on the display information.

Images