Lid Controller Hub Architecture for an Enhanced Touch Experience

Abstract

To provide a method, apparatus system and mobile computing device in which a lid controller hub processes sensor data generated by input sensors and realizes improved and enhanced experiences over existing devices.SOLUTION: A method comprises receiving touch sensor data from a touch display controller. In response to receiving a display refresh notification, the method comprises: generating conditioned touch sensor data based on the touch sensor data; and sending the conditioned touch sensor data to an operating system. The steps of receiving the touch sensor data, generating the conditioned touch sensor data and sending the conditioned touch sensor data are performed in a lid of a mobile computing device. The operating system is executed on one or more processing units located in a base of the mobile computing device.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application is a continuation of U.S. application Ser. No. 17 / 247,836 (entitled "Lid Controller Hub"), filed on December 24, 2020, which claims the benefit of priority under 35 U.S.C. § 120 to U.S. Provisional Patent Application Ser. No. 63 / 067,071 (entitled "Lid Controller Hub Architecture for Enhanced Touch Experience"), filed on August 18, 2020. The disclosures of the prior applications are considered part of the disclosure of this application and are incorporated by reference in their entirety into the disclosure of this application. [Background technology]

[0002] Existing laptops include various input sensors in the lid, such as a microphone, camera, and touchscreen. The sensor data generated by these lid sensors is sent over wires that extend over the hinge to the base of the laptop, where it is processed by the laptop's computing resources and made accessible to the operating system and applications. [Brief explanation of the drawings]

[0003] [Figure 1A] 1 illustrates a block diagram of a first exemplary computing device including a lid controller hub.

[0004] [Figure 1B] 10 shows a perspective view of a second exemplary mobile computing device including a lid controller hub.

[0005] [Figure 2] FIG. 10 shows a block diagram of a third exemplary mobile computing device including a lid controller hub.

[0006] [Figure 3]FIG. 10 shows a block diagram of a fourth exemplary mobile computing device including a lid controller hub.

[0007] [Figure 4] FIG. 4 shows a block diagram of the security module of the lid controller hub of FIG. 3.

[0008] [Figure 5] FIG. 4 is a block diagram of a host module of the lid controller hub of FIG. 3.

[0009] [Figure 6] FIG. 4 shows a block diagram of the vision / imaging module of the lid controller hub of FIG. 3.

[0010] [Figure 7] FIG. 4 shows a block diagram of the audio module of the lid controller hub of FIG. 3.

[0011] [Figure 8] 4 shows a block diagram of a timing controller, an integrated display panel, and additional electronics used with the lid controller hub of FIG. 3.

[0012] [Figure 9] FIG. 1 shows a block diagram illustrating an exemplary physical arrangement of components in a mobile computing device including a lid controller hub.

[0013] [Figure 10A] FIG. 1 shows a block diagram of an exemplary physical layout of a timing controller and a lid controller hub within the lid. [Figure 10B] FIG. 1 shows a block diagram of an exemplary physical layout of a timing controller and a lid controller hub within the lid. [Figure 10C] FIG. 1 shows a block diagram of an exemplary physical layout of a timing controller and a lid controller hub within the lid. [Figure 10D] FIG. 1 shows a block diagram of an exemplary physical layout of a timing controller and a lid controller hub within the lid. [Figure 10E] FIG. 1 shows a block diagram of an exemplary physical layout of a timing controller and a lid controller hub within the lid.

[0014] [Figure 11A] 10 shows a table of hinge wire count breakdowns for various lid controller hub embodiments. [Figure 11B] 10 shows a table of hinge wire count breakdowns for various lid controller hub embodiments. [Figure 11C] 10 shows a table of hinge wire count breakdowns for various lid controller hub embodiments.

[0015] [Figure 12A] 10 shows an example placement of an in-display microphone and camera on the lid. [Figure 12B] 10 shows an example placement of an in-display microphone and camera on the lid. [Figure 12C] 10 shows an example placement of an in-display microphone and camera on the lid.

[0016] [Figure 13A] 1 shows a simplified cross section of a pixel in an exemplary emissive display. [Figure 13B] 1 shows a simplified cross section of a pixel in an exemplary emissive display.

[0017] [Figure 14A] 1 shows an exemplary set of pixels with integrated microphones.

[0018] [Figure 14B] 14B shows a cross section of the example pixel of FIG. 14A taken along line AA′.

[0019] [Figure 14C]1 illustrates an exemplary microphone spanning multiple pixels. [Figure 14D] 1 illustrates an exemplary microphone spanning multiple pixels.

[0020] [Figure 15A] An exemplary set of pixels is shown along with an in-display camera.

[0021] [Figure 15B] 15B shows a cross section of the example pixel of FIG. 15A taken along line AA′.

[0022] [Figure 15C] 1 illustrates an exemplary camera spanning multiple pixels. [Figure 15D] 1 illustrates an exemplary camera spanning multiple pixels.

[0023] [Figure 16] 1 illustrates an exemplary camera that can be incorporated into a built-in display.

[0024] [Figure 17] 1 shows a block diagram illustrating an exemplary software / firmware environment for a mobile computing device including a lid controller hub.

[0025] [Figure 18A] 1 illustrates a top view of a mobile computing device in an open configuration, where a first exemplary foldable display includes a portion that can operate as an always-on display. [Figure 18B] 1 illustrates a top view of a mobile computing device in a closed configuration, where the first exemplary foldable display includes a portion that can operate as an always-on display.

[0026] [Figure 19A] 1 illustrates a top view of a mobile computing device in an open configuration having a second exemplary foldable display that includes a portion that can operate as an always-on display.

[0027] [Figure 19B] 19B shows a cross-sectional view of the mobile computing device of FIG. 19A in a closed configuration. [Figure 19C] 19B illustrates a top view of the mobile computing device of FIG. 19A in a closed configuration.

[0028] [Figure 20A] 1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. [Figure 20B] 1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. [Figure 20C] 1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. [Figure 20D] 1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. [Figure 20E] 1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. [Figure 20F] 1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. [Figure 20G] 1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. [Figure 20H] 1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. [Figure 20I]1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. [Figure 20J] 1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. [Figure 20K] 1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. [Figure 20L] 1 shows a diagram of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display.

[0029] [Figure 21] FIG. 1 shows a block diagram of an example timing controller and additional display pipeline components associated with a foldable display having an always-on display portion.

[0030] [Figure 22] 1 illustrates an exemplary method for operating a foldable display of a mobile computing device that can operate as an always-on display.

[0031] [Figure 23A] FIG. 1 is a diagram illustrating a user's attention to different display devices in a typical dual-display computing system. [Figure 23B] FIG. 1 is a diagram illustrating a user's attention to different display devices in a typical dual-display computing system.

[0032] [Figure 24A] FIG. 1 is a diagram illustrating a user's attention to different display devices in a typical multiple display computing system. [Figure 24B]FIG. 1 is a diagram illustrating a user's attention to different display devices in a typical multiple display computing system. [Figure 24C] FIG. 1 is a diagram illustrating a user's attention to different display devices in a typical multiple display computing system.

[0033] [Figure 25] FIG. 1 is a simplified block diagram of a multiple display computing system configured to manage display devices based on user presence and attention according to at least one embodiment.

[0034] [Figure 26] FIG. 1 is a top plan view illustrating possible fields of view of cameras in a multiple display computing system.

[0035] [Figure 27A] FIG. 1 is a top plan view showing possible user head / face orientations relative to the display device. [Figure 27B] FIG. 1 is a top plan view showing possible user head / face orientations relative to the display device. [Figure 27C] FIG. 1 is a top plan view showing possible user head / face orientations relative to the display device.

[0036] [Figure 28A] FIG. 2 is a diagram illustrating a user's attention to different display devices of a dual-display computing system configured to perform user presence-based display management for multiple displays according to at least one embodiment. [Figure 28B] FIG. 2 is a diagram illustrating a user's attention to different display devices of a dual-display computing system configured to perform user presence-based display management for multiple displays according to at least one embodiment.

[0037] [Figure 29A] FIG. 2 is a diagram illustrating a user's attention to different display devices of a multiple display computing system configured to perform user presence-based display management for multiple displays according to at least one embodiment. [Figure 29B] FIG. 2 is a diagram illustrating a user's attention to different display devices of a multiple display computing system configured to perform user presence-based display management for multiple displays according to at least one embodiment. [Figure 29C] FIG. 2 is a diagram illustrating a user's attention to different display devices of a multiple display computing system configured to perform user presence-based display management for multiple displays according to at least one embodiment.

[0038] [Figure 30] FIG. 47 is a schematic diagram of additional details of the computing system of FIG. 46 according to at least one embodiment.

[0039] [Figure 31] FIG. 31 is a simplified block diagram of additional details of the components of FIG. 30 according to at least one embodiment.

[0040] [Figure 32] 1 is a high-level flowchart of an example process that may be associated with a lid controller hub in accordance with at least one embodiment.

[0041] [Figure 33] 1 is a simplified flowchart of an example process that may be associated with detecting the presence of a user in accordance with at least one embodiment.

[0042] [Figure 34]1 is a simplified flowchart of an example process that may be associated with triggering an authentication mechanism according to at least one embodiment.

[0043] [Figure 35] 1 is a simplified flowchart of an example process that may be associated with adaptively dimming a display panel in accordance with at least one embodiment.

[0044] [Figure 36] 1 is a simplified flowchart of an example process that may be associated with adaptively dimming a display panel in accordance with at least one embodiment.

[0045] [Figure 37] 1 is a simplified flowchart of an example process that may be associated with an inactivity timeout for a display device according to at least one embodiment.

[0046] [Figure 38] FIG. 1 is a simplified block diagram of an exemplary computing device including a lid controller hub capable of providing an enhanced touch user experience.

[0047] [Figure 39] 39 illustrates an exemplary flow of information between components in the computing device illustrated in FIG. 38.

[0048] [Figure 40] 1 illustrates an exemplary method for synchronizing the sending of touch sensor data to an operating system with a display refresh rate.

[0049] [Figure 41] FIG. 1 is a simplified block diagram of an exemplary computing device capable of supporting combination and multi-plane gestures.

[0050] [Figure 42] 1 illustrates an exemplary combination gesture performed on a computing device.

[0051] [Figure 43A] 1 shows a first example of an associated multi-plane gesture and a reference touch gesture. [Figure 43B] 1 shows a first example of an associated multi-plane gesture and a reference touch gesture. [Figure 43C] 1 shows a first example of an associated multi-plane gesture and a reference touch gesture.

[0052] [Figure 44A] 10 shows a second example of a related multi-plane multi-user gesture. [Figure 44B] 10 shows a second example of a related multi-plane multi-user gesture. [Figure 44C] 10 shows a second example of a related multi-plane multi-user gesture.

[0053] [Figure 45A] 10 shows a third example of a multi-plane gesture. [Figure 45B] 10 shows a third example of a multi-plane gesture.

[0054] [Figure 46A] 10 shows a fourth example of a related multi-plane gesture. [Figure 46B] 10 shows a fourth example of a related multi-plane gesture.

[0055] [Figure 47A] 10 shows a fifth example of a related multi-plane gesture. [Figure 47B] 10 shows a fifth example of a related multi-plane gesture.

[0056] [Figure 48A] 10 shows a sixth example of a related multi-plane gesture. [Figure 48B] 10 shows a sixth example of a related multi-plane gesture.

[0057] [Figure 49] 1 is an exemplary method for determining a combination gesture applied to a mobile computing device and performing an action based on the combination gesture.

[0058] [Figure 50] 1 is an exemplary method for determining the presence of a series of multi-plane gestures and performing an action based on the multi-plane gestures.

[0059] [Figure 51] 1 illustrates an exemplary method for adjusting touch operations based on the presence of a user.

[0060] [Figure 52] FIG. 1 is a simplified block diagram of an exemplary computing device capable of controlling touch operations based on user presence, engagement, and interaction.

[0061] [Figure 53] 1 is an exemplary method for controlling touch operations based on user presence, engagement, and interaction.

[0062] [Figure 54] FIG. 1 is a block diagram of computing device components within the base of an exemplary mobile computing device including a lid controller hub.

[0063] [Figure 55] FIG. 1 is a block diagram of an exemplary processing unit capable of executing instructions as part of implementing the techniques described herein. DETAILED DESCRIPTION OF THE INVENTION

[0064] The lid controller hub disclosed herein performs various computational tasks within the lid of a laptop or computing device having a similar form factor. The lid controller hub can process sensor data generated by microphones, touchscreens, cameras, and other sensors located within the lid. The lid controller hub enables laptops with an improved and enhanced user experience, increased privacy and security, lower power consumption, and improved industrial design over existing devices. For example, the lid controller hub can synchronize the sampling and processing of touch sensor data with the display refresh rate, resulting in a smooth and responsive touch experience. Continuous monitoring and processing of image and audio sensor data captured by the camera and microphone in the lid allows the laptop to wake up when the voice or face of an authorized user is detected. The lid controller hub provides improved security by operating in a trusted execution environment. Only properly authenticated firmware is allowed to run in the lid controller hub. This means that unwanted applications cannot access the lid-based microphone and camera, and that image and audio sensor data processed by the lid controller hub to support its functionality is kept local to the lid controller hub.

[0065] The lid controller hub's computational resources enable an enhanced and improved experience. For example, a neural network accelerator within the lid controller hub can blur displays or faces in the background of a video call or filter out the sound of a barking dog in the background of a voice call. Furthermore, power savings are achieved through the use of various techniques, such as enabling sensors only when they are likely to be used, sampling touch display input at a typical sampling rate when a touch interaction is detected, and so on. Furthermore, latency is improved by processing sensor data locally in the lid, instead of sending it over the hinge and having the operating system process it. The lid controller hub also enables laptop designs with fewer wires threaded over the hinge. This not only allows for reduced hinge costs, but can also result in a simpler, more aesthetically pleasing industrial design. These and other lid controller hub features and advantages are described in more detail below.

[0066] In the following description, specific details are set forth; however, embodiments of the technology described herein may be practiced without these specific details. Known circuits, structures, and techniques have not been shown in detail to avoid obscuring an understanding of this specification. "An embodiment," "various embodiments," "some embodiments," etc. may include a feature, structure, or characteristic, but not all embodiments necessarily include the particular feature, structure, or characteristic.

[0067] Some embodiments may have some, all, or none of the features of other embodiments. "First," "second," "third," etc., describe a common object and indicate that different instances of the same object are being referred to. Such adjectives do not imply that the objects so described need be in a given order, sequence, or any other manner, either temporally or spatially. "Connected" may indicate that elements are in direct physical or electrical contact with each other, and "coupled" may indicate that elements cooperate or interact with each other, but may or may not be in direct physical or electrical contact. Terms modified by the word "approximately" include placement, orientation, spacing, or position that is slightly altered from the meaning of the unmodified term. For example, a description of a lid on a mobile computing device that is rotatable approximately 360 degrees relative to the mobile computing base includes a lid that is rotatable within a few degrees of 360 degrees relative to the device base.

[0068] In this description, the phrases "in one embodiment," "in an embodiment," "in some embodiments," and / or "in various embodiments" may be used, each of which may refer to one or more of the same or different embodiments. Furthermore, terms such as "comprising," "including," and "having," when used with respect to embodiments of the present disclosure, are synonymous.

[0069] Reference will now be made to the drawings, which are not necessarily drawn to scale, and in which similar or identical reference numerals may be used to designate similar or identical parts in different views. The use of similar or identical reference numerals in different views does not imply that all views containing similar or identical reference numerals constitute a single or the same embodiment. Similar reference numerals with different letter suffixes may represent different examples of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed herein.

[0070] In the following description, for purposes of explanation, numerical specific details are set forth to provide an understanding thereof. However, it may be apparent that the novel embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to facilitate the description. The present invention covers all modifications, equivalents, and alternatives falling within the scope of the claims.

[0071] FIG. 1A shows a block diagram of a first exemplary mobile computing device including a lid controller hub. The computing device 100 includes a base 110 connected to a lid 120 by a hinge 130. The mobile computing device (also referred to herein as a "user device") 100 may be a laptop or a mobile computing device having a similar form factor. The base 110 includes a host system-on-chip (SoC) 140 that includes one or more processing units integrated with one or more additional components, such as a memory controller, a graphics processing unit (GPU), a cache, an image processing module, and other components described herein. The base 110 may also include a physical keyboard, a touchpad, a battery, memory, storage, and external ports. The lid 120 includes an integrated display panel 145, a timing controller (TCON) 150, a lid controller hub (LCH) 155, a microphone 158, one or more cameras 160, and a touch display controller (touch controller) 165. The TCON 150 converts the video data 190 received from the SoC 140 into signals that drive the display panel 145 .

[0072] Display panel 145 may be any type of built-in display in which a display element responsible for generating or allowing light transmission is disposed at each pixel. Such displays may include TFT LCD (thin film transistor liquid crystal display), micro LED (micro light emitting diode (LED)), OLED (organic LED), and QLED (quantum dot LED) displays. Touch controller 165 drives the touchscreen technology utilized in display panel 145 and collects touch sensor data provided by the applied touchscreen technology. Display panel 145 may be a touch display and include a touchscreen, which may include one or more dedicated layers that implement touch functionality, or "in-cell" or "on-cell" touchscreen technology that does not require a dedicated touchscreen layer.

[0073] The microphone 158 may include a microphone located in the bezel of the lid or an in-display microphone located in the display area, i.e., the area of ​​the panel that displays the content. The one or more cameras 160 may similarly include a camera located in the bezel or an in-display camera located in the display area.

[0074] The LCH 155 includes an audio module 170, a vision / imaging module 172, a security module 174, and a host module 176. The audio module 170, the vision / imaging module 172, and the host module 176 interact with the lid sensors and process the sensor data generated by the sensors. The audio module 170 interacts with the microphone 158 and processes the audio sensor data generated by the microphone 158. The vision / imaging module 172 interacts with the one or more cameras 160 and processes the image sensor data generated by the one or more cameras 160. The host module 176 interacts with the touch controller 165 and processes the touch sensor data generated by the touch controller 165. A synchronization signal 180 is shared between the timing controller 150 and the lid controller hub 155. Synchronization signal 180 can be used to synchronize the sampling of touch sensor data and the transmission of touch sensor data to SoC 140 with the refresh rate of display panel 145, enabling a smooth and responsive touch experience at the system level.

[0075] As used herein, the term "sensor data" may refer to sensor data generated or provided by a sensor, as well as sensor data that has undergone subsequent processing. For example, image sensor data may refer to sensor data received at a frame router in a vision / imaging module, as well as processed sensor data output by a frame router processing stack in a vision / imaging module. The term "sensor data" may also refer to individual sensor data (e.g., one or more images captured by a camera) or a stream of sensor data (e.g., a video stream generated by a camera, an audio stream generated by a microphone). The term "sensor data" may also refer to metadata generated from sensor data, such as gestures determined from touch sensor data or head orientation or facial landmark information generated from image sensor data.

[0076] Audio module 170 processes audio sensor data generated by microphone 158 and, in some embodiments, enables features such as voice activation (causing device 100 to exit a low power state when a voice is detected in the audio sensor data), speaker ID (causing device 100 to exit a low power state when an authenticated user's voice is detected in the audio sensor data), acoustic context awareness (e.g., filtering undesirable background noise), speech and voice pre-processing to condition the audio sensor data for further processing by a neural network accelerator, dynamic noise reduction, and audio-based adaptive thermal solutions.

[0077] Vision / imaging module 172 processes image sensor data generated by one or more cameras 160 and, in various embodiments, can enable features such as Wake on Face (which causes device 100 to come out of a low power state when a face is detected in the image sensor data) and Face ID (which causes device 100 to come out of a low power state when an authenticated user's face is detected in the image sensor data). In some embodiments, vision / imaging module 172 can enable one or more of the following features: head orientation detection, determining the location of facial landmarks (e.g., eyes, mouth, nose, eyebrows, cheeks) in an image, and multi-face detection.

[0078] The host module 176 processes touch sensor data provided by the touch controller 165. The host module 176 can synchronize touch-related operations with the refresh rate of the built-in display panel 145, thereby enabling synchronization of touch and display operations at a system level, thereby providing an improved touch experience for any application running on the mobile computing device.

[0079] Thus, the LCH 155 can be thought of as a companion die to the SoC 140 in that it handles some of the sensor data-related processing tasks that the SoC performs in existing mobile computing devices. The proximity of the LCH 155 to the lid sensor enables experiences and features that may not be possible if sensor data were sent across the hinge 130 for processing by the SoC 140. The proximity of the LCH 155 to the lid sensor reduces latency, thereby providing more time for sensor data processing. For example, as described in more detail below, the LCH 155 includes a neural network accelerator, a digital signal processor, and image and audio sensor data processing modules to enable features such as voice activation, face activation, and contextual understanding. Placing the LCH computational resources close to the lid sensor further enables power savings because the lid sensor data can travel a shorter distance to the LCH rather than across the hinge to the base.

[0080] The lid controller hub enables additional power savings. For example, the LCH allows the SoC and other components in the base to enter a low-power state while the LCH monitors incoming sensor data to determine whether the device should transition to an active state. By allowing the device to wake up only when the presence of an authenticated user is detected (e.g., via Speaker ID or Face ID), the device can remain in a low-power state longer than if the device were woken up in response to detecting the presence of any person. The lid controller hub can also reduce (or disable) the sampling of touch input on the built-in display panel to a lower rate in certain contexts. Additional power savings enabled by the lid controller hub are described in more detail below.

[0081] As used herein, the term "active state" with respect to the system-level state of a mobile computing device refers to a state in which the device is fully operational. That is, all functionality of the host processing unit and lid controller hub is available, one or more applications can be executed, the device can provide an interactive and responsive user experience, and a user can watch videos, participate in video calls, surf the web, operate computer-aided design tools, or use the device in one of a variety of other ways. When the device is in the active state, one or more modules or other components of the device, including the lid controller hub or its constituent modules or other components, can be placed into a low-power state to conserve power. While the device is in the active state, the host processing unit can be temporarily placed into a high-performance mode to accommodate a heavy workload. Thus, the mobile computing device can operate within a range of power levels in the active state.

[0082] As used herein, the term "low power state" with respect to system-level states of a mobile computing device refers to a state in which the device is operating at a lower power consumption level than when the device is operating in an active state. Typically, the host processing unit is operating at a lower power consumption level than when the device is in an active state, and more device modules or other components are collectively operating in a lower power state than when the device is in an active state. A device can operate in one or more low power states. One distinction between low power states is characterized by the device-level power consumption level. In some embodiments, another distinction between low power states is characterized by the length of time the device takes to wake up in response to user input (e.g., keyboard, mouse, touch, voice, user presence detection in image sensor data, user opening or moving the device), a network event, or input from an attached device (e.g., a USB device). Such low power states may be characterized as "standby," "idle," "sleep," or "hibernation" states.

[0083] In a first type of device-level low power state, such as characterized as an "idle" or "standby" low power state, the device can rapidly transition from the low power state to an active state in response to user input, hardware, or network events. In a second type of device-level low power state, such as characterized as a "sleep" state, the device consumes less power than the first type of low power state and maintains volatile memory refresh to maintain the device state. In a third type of device-level low power state, such as characterized as a "hibernate" low power state, the device consumes less power than the second type of low power state. Nonvolatile memory refresh is not maintained and the device state is stored in nonvolatile memory. Waking up the device from the third type of low power state takes longer than the first or second type of low power state because the system state must be restored from nonvolatile memory. In a fourth type of low power state, the device is off and not consuming power. Waking up the device from the off state requires a full reboot of the device. As used herein, waking up the device refers to the device transitioning from a low power state to an active state.

[0084] With respect to the lid controller hub, the term “active state” refers to a lid controller hub state in which all of the lid controller hub's resources are available. That is, the LCH processes sensor data as it is generated, sends the sensor data and any data generated by the LCH based on the sensor data to the host SoC, and may display images based on video data received from the host SoC. One or more components of the LCH may be individually placed in a lower power state when the LCH is in the active state. For example, if the LCH detects that an authenticated user is not detected in the image sensor data, the LCH may disable the lid display. In another example, if a privacy mode is enabled, an LCH component that sends sensor data to the host SoC may be disabled. The term “low power” state with respect to the lid controller hub may refer to a power state in which the LCH operates at a lower power consumption level than when it is in the active state, and is typically characterized by one or more LCH modules or other components being in a lower power state than when the LCH is in the active state. For example, when the lid of a computing device is closed, the lid display can be disabled, the LCH vision / imaging module can be put into a low power state, and the LCH audio module can continue to operate to support voice activation functionality so that the device can continue to respond to voice queries.

[0085] A module or any other component of a mobile computing device can be placed into a low power state in a variety of ways, such as by reducing its operating voltage, providing a clock signal at a lower frequency, or receiving a control signal that causes the component to operate with lower power consumption (such as placing a module in an image display pipeline into a low power state that performs image processing on only a portion of an image).

[0086] In some embodiments, the power savings enabled by the LCH allows a mobile computing device to operate for a day under typical usage conditions without needing to be recharged. Furthermore, being able to power a day's use with less power allows for the use of smaller batteries in the mobile computing device. By allowing for smaller batteries and fewer wires over the hinge connecting the device to the lid, laptops including the LCH can be thinner and lighter, and therefore have an improved industrial design than existing devices.

[0087] In some embodiments, the lid controller hub technology disclosed herein enables laptops with intelligent collaboration and personal assistant capabilities. For example, the LCH may provide near-field and far-field audio capabilities, which enable improved audio reception by detecting the location of a distant sound source and improving the detection of audio coming from the location of the distant sound source. When combined with voice activation and speaker ID capabilities, near-field and far-field audio capabilities enable mobile computing devices to operate similarly to the "smart speakers" prevalent on the market today. For example, consider a scenario in which a user stops working, walks away from their laptop, and asks the laptop from across the room, "What's the weather going to be like tomorrow?" The laptop, which entered a low-power state after not detecting the authenticated user's face in the image sensor data provided by the user-facing camera, continues to monitor incoming audio sensor data to detect speech from the authenticated user. The laptop then exits its low-power state, captures the requested information, and responds to the user's question.

[0088] Hinge 130 may be any physical hinge that allows base 110 and lid 120 to be rotatably connected. Wires across hinge 130 include wires for transmitting video data 190 from SoC 140 to TCON 150, wires for transmitting audio data 192 between SoC 140 and audio module 170, wires for providing image data 194 from vision / imaging module 172 to SoC 140, wires for providing touch data 196 from LCH 155 to SoC 140, and wires for providing data determined from image sensor data and other information generated by LCH 155 from host module 176 to SoC 140. In some embodiments, data shown as being transmitted over different sets of wires between the SoC and LCH is communicated over the same set of wires. For example, in some embodiments, touch data, sensing data, and other information generated by the LCH may be transmitted over a single USB bus.

[0089] In some embodiments, the lid 120 is removably attached to the base 110. In some embodiments, the hinge can allow the base 110 and the lid 120 to rotate approximately 360 degrees relative to one another. In some embodiments, the hinge 130 carries fewer wires to communicatively couple the lid 120 to the base 110 than existing computing devices without an LCH. This reduction in wires over the hinge 130 can result in lower device costs, not only due to reduced wires, but also due to simpler electromagnetic and radio frequency interference (EMI / RFI) solutions.

[0090] Components shown in FIG. 1A as being located in the base of the mobile computing device may be located in the base housing, and components shown in FIG. 1A as being located in the lid of the mobile computing device may be located in the lid housing.

[0091] FIG. 1B shows a perspective view of a secondary exemplary mobile computing device including a lid controller hub. Mobile computing device 122 can be a laptop or other mobile computing device with a similar form factor, such as a foldable tablet or smartphone. Lid 123 includes an "A-cover" 139, which is the world-facing surface of lid 123 when mobile computing device 122 is in a closed configuration, and a "B-cover" 140, which includes the user-facing display when lid 123 is open. Base 129 includes a "C-cover" 126, which includes a keyboard facing up when device 122 is in an open configuration, and a "D-cover" 127, which is the bottom of base 129. In some embodiments, base 129 includes the device's 122 primary computational resources (e.g., host processing unit(s), GPU), as well as a battery, memory, and storage, and communicates with lid 123 via wires passing through hinge 128. Thus, in embodiments in which the mobile computing device is a dual display device, such as a dual display laptop, tablet, or smartphone, the base may be considered the portion of the device that includes the host processing unit, and the lid may be considered the portion of the device that includes the LCH. A Wi-Fi® antenna may be located in the base or lid of any of the computing devices described herein.

[0092] In another embodiment, computing device 122 may be a dual display device in which the second display includes a portion of C-cover 126. For example, in some embodiments, an "always on" display (AOD) may occupy an area of ​​the C-cover below the keyboard that is visible even when lid 123 is closed. In another embodiment, the second display covers most of the surface of the C-cover, and a detachable keyboard may be located on the second display, or the second display may present a virtual keyboard to allow keyboard input.

[0093] The lid controller hub is not limited to implementation within a laptop or another mobile computing device having a form factor similar to that shown in FIG. 1B. The lid controller hub technology disclosed herein is applicable to mobile computing devices that include one or more portions beyond a base and a single lid. The additional portion or portions include a display and / or one or more sensors. For example, a mobile computing device including an LCH may include a base, a primary display portion including a first touch display, a camera, and a microphone, and a secondary display portion including a second touch display. A first hinge rotatably couples the base to the secondary display portion, and a second hinge rotatably couples the primary display portion to the secondary display portion. An LCH located within either display portion can process sensor data generated by a lid sensor located within the same display portion in which the LCH is located or by lid sensors generated within both display portions. In this example, the lid controller hub may be located within either or both of the primary and secondary display portions. For example, a first LCH may be disposed within a secondary display that communicates with the base via a wire passing through the first hinge, and a second LCH may be disposed within a primary display that communicates with the base via a wire passing through the first and second hinges.

[0094] 2 shows a block diagram of a third exemplary mobile computing device including a lid controller hub. The device 200 includes a base 210 connected to a lid 220 by a hinge 230. The base 210 includes an SoC 390. The lid 220 includes a timing controller (TCON) 400, a lid controller hub (LCH) 260, a user-facing camera 270, an integrated display panel 280, and one or more microphones 290.

[0095] The SoC 390 includes a display module 391, an integrated sensor hub 392, an audio capture module 393, a universal serial bus (USB) module 394, an image processing module 395, and multiple processor cores 235. The display module 391 communicates with an integrated display port (eDP) module in the TCON 400 via an 8-wire eDP connection 233. In some embodiments, the integrated display panel 280 is a "3K2K" display (a display having 3Kx2K resolution) with a refresh rate of up to 120Hz. The connection 233 includes two eDP high bit rate 2 (HBR2 (17.28 Gb / s)) connections. The integrated sensor hub 392 communicates with the LCH 260's vision / imaging module 263 via a 2-wire Mobile Industry Processor Interface (MIPI) I3C (sense wire) connection 221. The audio capture module 393 communicates with the audio module 264 of the LCH 260 via a 4-wire MIPI SoundWire® connection 222. The USB module 394 communicates with the security / host module 261 of the LCH 260 via a USB connection 223. The image processing module 395 receives image data from the MIPI D-PHY transmit port 265 of the frame router 267 of the LCH 260 via a 4-lane MIPI D-PHY connection 239 comprising 10 wires. The built-in sensor hub 392 can be an Intel® built-in sensor hub or any other sensor hub capable of processing sensor data from one or more sensors.

[0096] As used herein, the term "port" may refer to any interface, module, connector, etc. that transmits and / or receives information or signals from one module, circuit, component, etc. to another. A port may be implemented in hardware, software, firmware, or a component thereof.

[0097] The TCON 400 includes an eDP port 402 and a Peripheral Component Interface Express (PCIe) port 404 that drives an internal display panel 280 using PCIe's peer-to-peer (P2P) communication capabilities over a 48-wire connection 240.

[0098] The LCH 260 includes a security / host module 261, a vision / imaging module 263, an audio module 264, and a frame router 267. The security / host module 261 includes a digital signal processing (DSP) processor 271, a security processor 272, a vault and one-time password generator (OTP) 273, and memory 274. In some embodiments, the DSP 271 is a Synopsis® DesignWare® ARC® EM7D or EM11D DSP processor, and the security processor is a Synopsis® DesignWare® ARC® SEM security processor. In addition to communicating with the USB module 394 in the SoC 390, the security / host module 261 communicates with the TCON 400 via an inter-integrated circuit (I2C) connection 226 to achieve synchronization between the LCH and TCON operations. The memory 274 stores instructions executed by the components of the LCH 260.

[0099] The vision / imaging module 263 includes a DSP 275, a neural network accelerator (NNA) 276, an image preprocessor 278, and memory 277. In some embodiments, the DSP 275 is a DesignWare® ARC® EM11D processor. The vision / imaging module 263 communicates with a frame router 267 via an Intelligent Peripheral Interface (IPI) connection 227. The vision / imaging module 263 can perform face detection, head orientation detection, and enable device access based on detecting a human face (face wake) or the face of an authenticated user (face ID) in the image sensor data. In some embodiments, the vision / imaging module 263 can implement one or more artificial intelligence (AI) models via the neural network accelerator 276 to enable these functions. For example, the neural network accelerator 276 may implement a model trained to recognize the face of an authenticated user in the image sensor data to enable the face wake function. Vision / imaging module 263 communicates with camera 270 via connection 228, which includes a pair of I2C or I3C wires and a 5-wire general-purpose I / O (GPIO) connection. Frame router 267 includes a D-to-PHY transmit port 265 and a D-to-PHY receiver 266, which receives image sensor data provided by user-facing camera 270 via connection 231, which includes a 4-wire MIPI Camera Serial Interface 2 (CSI2) connection. LCH 260 communicates with touch display controller (touch controller) 285 via connection 232, which may include an 8-wire Serial Peripheral Interface (SPI) or a 4-wire I2C connection.

[0100] Audio module 264 includes one or more DSPs 281, a neural network accelerator 282, an audio preprocessor 284, and memory 283. In some embodiments, lid 220 includes four microphones 290, and audio module 264 includes four DSPs 281, one for each microphone. In some embodiments, each DSP 281 is a Cadence® Tensilica® HiFi DSP. Audio module 264 communicates with one or more microphones 290 via connection 229, which may include a MIPI SoundWire® connection or signals transmitted via pulse density modulation (PDM). In another embodiment, connection 229 includes a four-wire digital microphone (DMIC) interface, a two-wire integrated inter-IC sound bus (I2S) connection, and one or more GPIO wires. The audio module 264 can wake the device from a low-power state in response to detecting a human voice (voice activation) or the voice of an authenticated user (speaker ID), activate near- and far-field audio (input and output), and perform additional voice recognition tasks. In some embodiments, the NNA 282 is an artificial neural network accelerator that implements one or more artificial intelligence (AI) models to enable various LCH functions. For example, the NNA 282 can implement an AI model trained to detect activation words or expressions in audio sensor data generated by one or more microphones 290 to enable voice activation functions.

[0101] In some embodiments, security / host module memory 274, vision / imaging module memory 277, and audio module memory 283 are portions of shared memory accessible to security / host module 261, vision / imaging module 263, and audio module 264. At boot-up of device 200, a section of shared memory is allocated to each of security / host module 261, vision / imaging module 263, and audio module 264. After boot-up, each section of shared memory allocated to a module is firewalled from other allocated sections. In some embodiments, the shared memory may be a 12 MB memory partitioned as follows: security / host memory (1 MB), vision / imaging memory (3 MB), and audio memory (8 MB).

[0102] Any connections described herein connecting two or more components may utilize different interfaces, protocols, or connection technologies, and / or utilize a different number of wires than described for a particular connection. While display module 391, integrated sensor hub 392, audio capture module 393, USB module 394, and image processing module 395 are shown as being integrated into SoC 390, in other embodiments, one or more of these components may be located outside the SoC. For example, one or more of these components may be located on the die, in a package, or on a substrate separate from the die, package, or substrate that includes the host processing unit (e.g., core 235).

[0103] 3 shows a block diagram of a fourth exemplary mobile computing device including a lid controller hub. Mobile computing device 300 includes a lid 301 connected via a base 315 by a hinge 330. Lid 301 includes a lid controller hub (LCH) 305, a timing controller 355, a user-facing camera 346, a microphone 390, an integrated display panel 380, a touch display controller (display controller) 385, and memory 353. LCH 305 includes a security module 361, a host module 362, a vision / imaging module 363, and an audio module 364. Security module 361 provides a secure processing environment for LCH 305 and includes a vault 320, a security processor 321, a fabric 310, I / O 332, an always-on (AON) block 316, and memory 323. Security module 361 is responsible for loading and authenticating firmware stored in memory 353 and executed by various components of LCH 305 (e.g., DSP, neural network accelerator). Security module 361 authenticates the firmware by running a cryptographic hash function on the firmware and using key information stored in security module 361 to verify that the resulting hash is correct and the firmware has the appropriate signature. The cryptographic hash function is performed by vault 320. In some embodiments, vault 320 includes a cryptographic accelerator. In some embodiments, security module 361 may present a Product Root of Trust (PRoT) interface that allows another component of device 200 to query LCH 305 about the results of firmware authentication. In some embodiments, the PRoT interface may be provided via an I2C / I3C interface (e.g., I2C / I3C interface 470).

[0104] As used herein with respect to software or firmware in connection with a lid controller hub, a lid controller hub component, a host processing unit, a SoC, or other computing device component, the terms "operating," "executing," or "running" are used interchangeably and may refer to software or firmware stored on one or more computer-readable storage media accessible by a computing device component, even if the instructions contained within the software or firmware are not actively being executed by the component.

[0105] The security module 361 also stores privacy information and handles privacy tasks. In some embodiments, when a microphone picks up an authenticated user's voice or a camera captures an authenticated user's face, information that the LCH 305 uses to perform face ID or speaker ID for activating the computing device is stored in the security module 361. The security module 361 also enables a privacy mode for the LCH or computing device. For example, when user input indicates that the user wants to enable privacy mode, the security module 361 may disable access by LCH resources to sensor data generated by one or more of the lid input devices (e.g., touchscreen, microphone, camera). In some embodiments, a user may set privacy settings to transition the device into privacy mode. Privacy settings include, for example, disabling video and / or audio input in a videoconferencing application or enabling operating system-level privacy settings that prevent any application or operating system from receiving and / or processing sensor data. Setting an application or operating system privacy setting may cause information to be sent to the lid controller hub to transition the LCH into privacy mode. In privacy mode, the lid controller hub may transition input sensors into a low power state, prevent LCH resources from processing sensor data, or prevent raw or processed sensor data from being sent to the host processing unit.

[0106] In some embodiments, the LCH 305 can enable face activation or face ID functionality while keeping the image sensor data private to the rest of the system (e.g., the operating system and any applications running on the operating system). In some embodiments, the vision / imaging module 363 continues to process the image sensor data to allow face activation or face ID functionality to remain active while the device is in privacy mode. In some embodiments, to improve privacy and reduce power consumption, image sensor data is sent through the vision / imaging module 363 to the image processing module 345 in the SoC 340 only when a face (or the face of an authenticated user) is detected, regardless of whether privacy mode is enabled. In some embodiments, the mobile computing device 300 can include one or more world-facing cameras and one or more world-facing microphones (e.g., microphones integrated into the “A-cover” of a laptop) in addition to the user-facing camera 346.

[0107] In some embodiments, the lid controller hub 305 transitions to privacy mode in response to a user pressing a privacy button, twisting or flipping a privacy switch, or sliding a slider on an input sensor in the lid. In some embodiments, a privacy indicator may be provided to the user to indicate that the LCH is in privacy mode. The privacy indicator may be, for example, an LED located on the base or display bezel, or a privacy icon displayed on the display. In some embodiments, a privacy mode configured at the hardware or system level is enabled when the user activates an external privacy button, switch, slider, hotkey, etc. That is, the privacy mode applies to all applications and operating systems running on the mobile computing device. For example, if a user presses a privacy switch located on the lid bezel, the LCH may responsively disable all audio sensor data and all image sensor data made available to the SoC. While audio and image sensor data remains available to the LCH to perform tasks such as voice activation and speaker ID, audio and image sensor data accessible by the lid controller hub is inaccessible to other processing components.

[0108] The host module 362 includes a security processor 324, a DSP 325, a memory 326, a fabric 311, an always-on block 317, and an I / O 333. In some embodiments, the host module 362 can initiate the LCH, send LCH telemetry and data interrupts to the SoC, manage interactions with the touch controller 385, and send touch sensor data to the SoC 340. The host module 362 sends lid sensor data from multiple lid sensors to a USB module 344 in the SoC 340 over a USB connection. Sending sensor data for multiple lid sensors over a single connection helps reduce the number of wires running through the hinge 330 compared to existing laptop designs. The DSP 325 processes the touch sensor data received from the touch controller 385. The host module 362 can synchronize the transmission of touch sensor data to the SoC 340 with the display panel refresh rate by utilizing a synchronization signal 370 shared between the TCON 355 and the host module 362.

[0109] The host module 362 can dynamically adjust the refresh rate of the display panel 380 based on factors such as the presence of a user and the amount of user touch interaction with the panel 380. For example, the host module 362 can reduce the refresh rate of the panel 380 if no user is detected or if no authenticated user is detected in front of the camera 346. In another example, the refresh rate can increase in response to detecting a touch interaction on the panel 380 based on touch sensor data. In some embodiments, and depending on the refresh rate capabilities of the display panel 380, the host module 362 can increase the refresh rate of the panel 380 up to 120 Hz or reduce it to 20 Hz or less.

[0110] The host module 362 can also adjust the refresh rate based on the application with which the user is interacting. For example, if the user is interacting with an illustration application, the host module 362 can increase the refresh rate to 120 Hz to provide a smoother touch experience for the user (which can also increase the rate at which touch data is transmitted to the SoC 340 if the display panel refresh rate and the processing of the touch sensor data are synchronized). Similarly, if the host module 362 detects that the user is currently interacting with an application with relatively static content or one that involves low or simple user touch interaction (e.g., selecting an icon or typing a message), the host module 362 can reduce the refresh rate to a lower frequency. In some embodiments, the host module 362 can adjust the refresh rate and touch sampling frequency by monitoring the frequency of touch interaction. For example, the refresh rate can be adjusted upward when the host module 362 detects that there is a high level of user interaction or that the user is utilizing a particular touch input device (e.g., a stylus) or a particular function of the touch input stylus (e.g., a stylus tilt function). When the refresh rate exceeds a threshold, the host module 362 can enable a strobe function of the display panel, if supported by the display panel, to reduce ghosting artifacts.

[0111] Vision / imaging module 363 includes neural network accelerator 327, DSP 328, memory 329, fabric 312, AON block 318, I / O 334, and frame router 339. Vision / imaging module 363 interacts with user-facing camera 346. Vision / imaging module 363 can interact with multiple cameras and combine image data from multiple cameras into a single stream for transmission to integrated sensor hub 342 in SoC 340. In some embodiments, lid 301 can include one or more additional user-facing and / or world-facing cameras in addition to user-facing camera 346. In some embodiments, any of the user-facing cameras can be an in-display camera. Image sensor data generated by camera 346 is received by frame router 339, where it is preprocessed before being sent to neural network accelerator 327 and / or DSP 328. The image sensor data may further be sent to an image processing module 345 in the SoC 340 through a frame router 339. The neural network accelerator 327 and / or the DSP 328 enable face detection, head orientation detection, recognition of facial landmarks (e.g., eyes, cheeks, eyebrows, nose, mouth), generation of a 3D mesh that fits the detected face, and other image processing functions. In some embodiments, facial parameters (e.g., positions of facial landmarks, 3D mesh, physical dimensions of the face, head orientation) may be transmitted to the SoC at a rate of 30 frames per second (30 fps).

[0112] The audio module 364 includes a neural network accelerator 350, one or more DSPs 351, memory 352, fabric 313, AON block 319, and I / O 335. The audio module 364 receives audio sensor data from microphones 390. In some embodiments, there is one DSP 351 for each microphone 390. The neural network accelerator 350 and DSP 351 implement audio processing algorithms and AI models that improve audio quality. For example, the DSP 351 may perform audio preprocessing on the received audio sensor data to prepare it for processing by the audio AI model implemented by the neural network accelerator 350. One example of an audio AI model that can be implemented by the neural network accelerator 350 is a noise reduction algorithm that filters out background noise, such as a dog barking or a blaring siren. A second example is a model that enables voice activation or speaker ID functionality. A third example is a context awareness model. For example, an audio context model may be implemented that classifies audio event occurrences related to situations in which police or emergency medical providers should be called, such as glass breaking, a car crash, or a gun being fired. The LCH may provide information indicating the occurrence of such events to the SoC, and the SoC may query the user whether police or medical professionals should be called.

[0113] AON blocks 316-319 in LCH modules 361-364 contain various I / O, timers, interrupts, and control units to support LCH "always on" features such as voice activation, speaker ID, face activation, and face ID, and an always-on display that is visible and provides content when lid 301 is closed.

[0114] FIG. 4 shows a block diagram of the security module of the lid controller hub of FIG. 3. Vault 320 includes a cryptographic accelerator 400 capable of implementing cryptographic hash functions running on firmware stored in memory 353. In some embodiments, cryptographic accelerator 400 implements an Advanced Encryption Standard (AES)-compliant (AES-128) or a 384-bit Secure Hash Algorithm (SHA)-compliant (SHA-384) encryption algorithm with a 128-bit block size. Security processor 321 resides within security processor module 402, which further includes a platform specific function module (PUF) 405, an OTP generator 410, a ROM 415, and a direct memory access (DMA) module 420. PUF 405 can implement one or more security-related functions specific to a particular LCH implementation. In some embodiments, security processor 321 can be a DesignWare® ARC® SEM security processor. Fabric 310 enables communication between various components of security module 361 and includes an Advanced Extensive Interface (AXI) 440, an Advanced Peripheral Bus (APB) 440, and an Advanced High-Performance Bus (AHB) 445. AXI 440 communicates with Advanced Peripheral Bus 440 via an AXI to APB (AXI X2P) bridge 430 and with Advanced High-Performance Bus 445 via an AXI to AHB (AXI X2A) bridge 435. Always-on block 316 includes multiple GPIOs 450, a universal asynchronous receiver / transmitter (UART) 455, a timer 460, and a power management and clock management unit (PMU / CMU) 465. PMU / CMU 465 controls the supply of power and clock signals to the LCH components and can selectively supply power and clock signals to individual LCH components. This ensures that only those components used to support a particular LCH operating mode or function receive power and are clocked.I / O set 332 includes an I2C / I3C interface 470 and a Queued Serial Peripheral Interface (QSPI) 475 for communication with memory 353. In some embodiments, memory 353 is a 16MB Serial Peripheral Interface (SPI) to NOR flash memory that stores the LCH firmware. In some embodiments, the LCH security module can eliminate one or more of the components shown in FIG. 4. In some embodiments, the LCH security module can include one or more additional components beyond those shown in FIG. 4.

[0115] FIG. 5 shows a block diagram of the host module of the lid controller hub of FIG. 3. The DSP 325 is part of a DSP module 500, which further includes a level 1 (L1) cache 504, a ROM 506, and a DMA module 508. In some embodiments, the DSP 325 can be a DesignWare® ARC® EM11D DSP processor. The security processor 324 is part of a security processor module 502, which further includes a PUF module 510, which enables implementation of platform-specific functions, an OTP generator 512, a ROM 514, and a DMA module 516. In some embodiments, the security processor 324 is a Synopsis® DesignWare® ARC® SEM security processor. The fabric 311 enables communication between the various components of the host module 362 and includes components similar to the security component fabric 310. Always-on block 317 includes multiple UARTs 550, a Joint Test Action Group (JTAG) / I3C port 552 to support LCH debugging, multiple GPIOs 554, a timer 556, an interrupt request (IRQ) / wake-up block 558, and a PMU / CCU port 560 that provides a 19.2 MHz reference clock to camera 346. Sync signal 370 is connected to one of the GPIO ports. I / O 333 includes an interface 570 that supports I2C and / or I3C communication with camera 346, a USB module 580 that communicates with USB module 344 in SoC 340, and a QSPI block 584 that communicates with touch controller 385. In some embodiments, I / O set 333 provides touch sensor data to the SoC via QSPI interface 582. In another embodiment, touch sensor data is communicated to the SoC via a USB connection 583. In some embodiments, connection 583 is a USB 2.0 connection.By utilizing USB connection 583 to transmit touch sensor data to the SoC, hinge 330 does not need to include wires to support the QSPI connection supported by QSPI interface 582. Not having to support this additional QSPI connection reduces the number of wires running through the hinge by between four and eight wires.

[0116] In some embodiments, the host module 362 can support dual displays. In such embodiments, the host module 362 communicates with a second touch controller and a second timing controller. A second synchronization signal between the second timing controller and the host module enables processing of touch sensor data provided by the second touch controller and synchronization of the transmission of touch sensor data provided by the second touch sensor to the SoC with the refresh rate of the second display. In some embodiments, the host module 362 can support three or more displays. In some embodiments, the LCH host module can omit one or more of the components shown in FIG. 5 . In some embodiments, the LCH host module can include one or more additional components beyond those shown in FIG. 5 .

[0117] FIG. 6 shows a block diagram of the vision / imaging module of the lid controller hub of FIG. 3. The DSP 328 is part of a DSP module 600, which further includes an L1 cache 602, a ROM 604, and a DMA module 606. In some embodiments, the DSP 328 may be a DesignWare® ARC® EM11D DSP processor. The fabric 312 enables communication between various components of the vision / imaging module 363 and includes an Advanced Extensible Interface (AXI) 640 connected to an Advanced Peripheral Bus (APB) 640 by an AXI to APB (X2P) bridge 630. The always-on block 318 includes multiple GPIOs 650, multiple timers 652, an IRQ / wake-up block 654, and a PMU / CCU 656. In some embodiments, the IRQ / wake-up block 654 receives a wake-on-motion (WoM) interrupt from the camera 346. The WoM interrupt may be generated based on accelerometer sensor data generated by an accelerator located within or communicatively coupled to the camera, or may be generated in response to the camera performing motion detection processing within an image captured by the camera. The I / O 334 includes an I2C / I3C interface 674 that transmits metadata to an integrated sensor hub 342 within the SoC 340, and an I2C3 / I3C interface 670 that connects to the camera 346 and other lid sensors 671 (e.g., lidar sensors, radar sensors, ultrasonic sensors, time-of-flight cameras, infrared sensors). The vision / imaging module 363 may receive sensor data from additional lid sensors 671 via the I2C / I3C interface 670.In some embodiments, the metadata includes information such as information indicating whether the information provided by the lid controller hub is valid, information indicating the operating mode of the lid controller hub (e.g., off, a "face-activated" low-power mode in which some LCH components are disabled but the LCH continuously monitors image sensor data to detect a user's face), auto-exposure information (e.g., the exposure level automatically set by the vision / imaging module 363 for the camera 346), and information related to faces detected in images or videos captured by the camera 346 (e.g., information indicating a confidence level that a face is present, information indicating a confidence level that the face matches the face of an authenticated user, bounding box information indicating the location of the face in the captured image or video, orientation information indicating the orientation of the detected face, and facial landmark information).

[0118] The frame router 339 can receive image sensor data from the camera 346 and process the image sensor data before sending it to the neural network accelerator 327 and / or the DSP 328 for further processing. The frame router 339 can also send the received image sensor data to the image processing module 345 within the SoC 340, bypassing the frame router processing. The image sensor data can be sent to the image processing module 345 while being processed by the frame router processing stack 699. The image sensor data generated by the camera 346 is received in the frame router 339 by the MIPI D-PHY receiver 680, where it is sent to the MIPI CSI2 receiver 682. The multiplexer / selector block 684 can enable the image sensor data to be processed by the frame router processing stack 699, sent directly to the CSI2 transmitter 697 and D-PHY transmitter 698 for transmission to the image processing module 345, or both.

[0119] The frame router processing stack 699 includes one or more modules that can perform pre-processing of the image sensor data and perform additional image processing on the image sensor data to prepare it for processing by the neural network accelerator 327 and / or the DSP 328. The frame router processing stack 699 includes a sampler / cropper module 686, a lens shading module 688, a motion detector module 690, an auto-exposure module 692, an image pre-processing module 694, and a DMA module 696. The sampler / cropper module 686 can reduce the frame rate of video represented by the image sensor data and / or crop the size of the image represented by the image sensor data. The lens shading module 688 can apply one or more lens shading effects to the image represented by the image sensor data. In some embodiments, the lens shading effects applied to the image represented by the image sensor data are user-selectable. The motion detector 690 can detect motion between multiple images represented by the image sensor data. The motion detector may indicate any motion or the motion of a particular object (eg, a face) across multiple images.

[0120] The auto-exposure module 692 can determine whether the image represented by the image sensor data is overexposed or underexposed and adjust the exposure of the camera 346 to improve the exposure of future images captured by the camera 346. In some embodiments, the auto-exposure module 362 can modify the image sensor data to address the overexposure or underexposure and improve the quality of the image represented by the image sensor data. The image pre-processing module 694 performs image processing of the image sensor data to further prepare the image sensor data for processing by the neural network accelerator 327 and / or the DSP 328. After being processed by one or more modules in the frame router processing stack 699, the image sensor data can be sent via the fabric 312 to another component in the vision / imaging module 363. In some embodiments, the frame router processing stack 699 includes more or fewer modules than shown in FIG. 6 . In some embodiments, the frame router processing stack 699 is configurable in that the image sensor data is processed by selected modules in the frame processing stack. In some embodiments, the order of operations on the image sensor data of the modules in the frame processing stack is also configurable.

[0121] Once the image sensor data has been processed by the frame router processing stack 699, the processed image sensor data is provided to the DSP 328 and / or neural network accelerator 327 for further processing. The neural network accelerator 327 enables a face wake-up function by detecting the presence of a face in the processed image sensor data and a face ID function by detecting the presence of an authenticated user's face in the processed image sensor data. In some embodiments, the NNA 327 is capable of detecting the presence of multiple faces in the image sensor data and multiple authenticated users in the image sensor data. The neural network accelerator 327 is configurable and can be updated with information that enables the NNA 327 to identify one or more authenticated users or to identify new authenticated users. In some embodiments, the NNA 327 and / or DSP 328 enable one or more adaptive dimming functions. One example of an adaptive dimming function is dimming areas of an image or video that are not occupied by a human face, a feature useful for video conferencing or video calling applications. Another example is globally dimming the screen while the computing device is active and a face is no longer detected in front of the camera, and then undimming the display when a face is detected again. Extending adaptive dimming functionality to incorporate Face ID means that the screen will only be undimmed if the authenticated user is detected again.

[0122] In some embodiments, the frame router processing stack 699 includes a super-resolution module (not shown) that can upscale or downscale the resolution of the image represented by the image sensor data. For example, in an embodiment in which the image sensor data represents a 1 megapixel image, the super-resolution module can upscale the 1 megapixel image to a higher resolution image before sending it to the image processing module 345. In some embodiments, the LCH vision / imaging module may not have one or more of the components shown in FIG. 6. In some embodiments, the LCH vision / imaging module may include one or more additional components in addition to those shown in FIG. 6.

[0123] FIG. 7 shows a block diagram of the audio module 364 of the lid controller hub of FIG. 3. In some embodiments, the NNA 350 can be an artificial neural network accelerator. In some embodiments, the NNA 350 can be an Intel® Gaussian & Neural Accelerator (GNA) or other low-power neural coprocessor. The DSP 351 is part of a DSP module 700, which further includes an instruction cache 702 and a data cache 704. In some embodiments, each DSP 351 is a Cadence® Tensilica® HiFi DSP. The audio module 364 includes one DSP module 700 for each microphone in the lid. In some embodiments, the DSP 351 can perform dynamic noise reduction on the audio sensor data. In other embodiments, more or fewer microphones than four can be used, and the audio sensor data provided by the multiple microphones can be processed by a single DSP 351. In some embodiments, the NNA 350 implements one or more models to improve audio quality. For example, the NNA350 can implement one or more "smart mute" models that eliminate or reduce background noise that may be distracting during an audio or video call.

[0124] In some embodiments, the DSP 351 can enable far-field functionality. For example, a lid containing multiple forward-facing microphones distributed across the bezel (or across the viewing area if in-display microphones are used) can perform beamforming or spatial filtering on audio signals generated by the microphones to enable far-field functionality (e.g., improved detection of sounds generated by distant sources). The audio module 364, utilizing the DSP 351, can determine the location of a distant sound source to improve detection of sounds received from that location. In some embodiments, the DSP 351 can determine the location of the sound source by determining delays to be applied to audio signals generated by the microphones so that the audio signals overlap in time and estimating the distance of the sound source from each microphone based on the delays applied to each audio signal. Applying the determined delays to the audio signals provided by the microphones can improve audio detection in the direction of the distant sound source. The enhanced audio can be provided to the NNA 350 for voice detection to enable voice activation or speaker ID functions. Further processing by the DSP 351 can also be performed on the enhanced audio. The identified location of the audio source may be provided to the SoC for use by the operating system or applications running on the operating system.

[0125] In some embodiments, the DSP 351 can detect information encoded in audio sensor data at near-ultrasonic (e.g., 15 kHz to 20 kHz) or ultrasonic (e.g., >20 kHz) frequencies, thereby realizing a low-frequency, low-power communication channel. Information detected at near-ultrasonic / ultrasonic frequencies can be sent to the audio capture module 343 in the SoC 340. For example, the ultrasonic communication channel can be used to communicate conference connection or Wi-to-Fi connection information to a mobile computing device to another computing device in a conference room (e.g., a Wi-to-Fi router, repeater, presentation equipment). The audio module 364 can further drive one or more microphones 390 to transmit information at ultrasonic frequencies. Thus, the audio channel can be used as a bidirectional, low-frequency, low-power communication channel between computing devices.

[0126] In some embodiments, the audio module 364 can enable adaptive cooling. For example, the audio module 364 can determine an ambient noise level and send information indicative of the ambient noise level to the SoC. The SoC can use this information as a factor in determining the operating level of a cooling fan in the computing device. For example, the speed of the cooling fan can be scaled up or down to increase or decrease the ambient noise level. This can enable improved cooling performance in noisier environments.

[0127] Fabric 313 enables communication between the various components of audio module 364. Fabric 313 includes an Open Core Protocol (OCP) interface 726 that connects NNA 550, DSP module 700, memory 352, and DMA 748 to APB 740 via OCP-to-APB bridge 728. Always-on block 319 includes multiple GPIOs 750, a pulse density modulation (PDM) module 752 that receives audio sensor data generated by microphone 390, one or more timers 754, a PMU / CCU 756, and a MIPI SoundWire® module 758 for sending and receiving audio data to and from audio capture module 343. In some embodiments, the audio sensor data provided by microphone 390 is received by DesignWare® SoundWire® module 760. In some embodiments, the LCH audio module may not have one or more of the components shown in FIG. 7. In some embodiments, the LCH audio module may include one or more additional components beyond those shown in FIG.

[0128] FIG. 8 shows a block diagram of a timing controller, integrated display panel, and additional electronics used with the lid controller hub of FIG. 3. The timing controller 355 receives video data from the display module 341 of the SoC 340 via an eDP connection including multiple main link lanes 800 and an auxiliary (AUX) channel 805. The video data and auxiliary channel information provided by the display module 341 are received at the TCON 355 by an eDP main link receiver 812 and an auxiliary channel receiver 810. The timing controller processing stack 820 includes one or more modules responsible for pixel processing and conversion of video data transmitted from the display module 341 into signals that drive control circuitry in the display panel 380 (e.g., row driver 882, column driver 884). Video data may be processed by the timing controller processing stack 820 without being stored in the frame buffer 830, or the video data may be stored in the frame buffer 830 before processing by the timing controller processing stack 820. The frame buffer 830 stores pixel information for one or more video frames (or frames; as used herein, the terms "image" and "frame" are used interchangeably). For example, in some embodiments, the frame buffer can store color information for pixels in a video frame to be displayed on a panel.

[0129] The timing controller processing stack 820 includes an autonomous low refresh rate module (ALRR) 822, a decoder panel self-refresh (decoder PSR) module 824, and a power optimization module 826. The ALRR module 822 can dynamically adjust the refresh rate of the display 380. In some embodiments, the ALRR module 822 can adjust the display refresh rate between 20 Hz and 120 Hz. The ALRR module 822 can implement various dynamic refresh rate techniques, such as adjusting the display refresh rate based on the frame rate of received video data, which can vary depending on the complexity of the images being rendered in gaming applications. The refresh rate determined by the ALRR module 822 can be provided to the host module as a synchronization signal 370. In some embodiments, the synchronization signal includes an indication that a display refresh is about to occur. In some embodiments, the ALRR module 822 can dynamically adjust the panel refresh rate by adjusting the length of a blank period. In some embodiments, the ALRR module 822 can adjust the panel refresh rate based on information received from the host module 362. For example, in some embodiments, the host module 362 can send information to the ALRR module 822 indicating that the refresh rate is to be reduced when the vision / imaging module 363 determines that a user is not in front of the camera. In some embodiments, the host module 362 can send information to the ALRR module 822 indicating that the refresh rate is to be increased when the host module 362 determines that there has been a touch interaction on the panel 380 based on touch sensor data received from the touch controller 385.

[0130] In some embodiments, the decoder PSR module 824, which decodes video data encoded using the VDSC compression standard, may include a Video Electronics Equipment Standards Association (VESA) Display Streaming Compression (VDSC) decoder. In another embodiment, the decoder-to-panel self-refresh module 824 may include a panel self-refresh (PSR) implementation that, when enabled, refreshes all or a portion of the display panel 380 based on video data stored in the frame buffer utilized in the previous refresh cycle. This may enable the portion of the display pipeline leading to the frame buffer to enter a lower power state. In some embodiments, the decoder-to-panel self-refresh module 824 may be the PSR function implemented in eDP v1.3 or the PSR2 function implemented in eDP v1.4. In some embodiments, the TCON may achieve additional power savings by entering a zero or low refresh state during a mobile computing device operating system upgrade. In the zero refresh state, the timing controller does not refresh the display. In the low refresh state, the timing controller refreshes the display at a low rate (e.g., 20 Hz or less).

[0131] In some embodiments, the timing controller processing stack 820 may include a super-resolution module 825 that can downscale or upscale the resolution of video frames provided by the display module 341 to match that of the display panel 380. For example, if the built-in panel 380 is a 3Kx2K panel and the display module 341 provides 4K video frames rendered at 4K, the super-resolution module 825 can downscale the 4K video frames to 3Kx2K video frames. In some embodiments, the super-resolution module 825 can upscale the video resolution. For example, if a gaming application renders images at 1360x768 resolution, the super-resolution module 825 can upscale the video frames to 3Kx2K to maximize the resolution capabilities of the display panel 380. In some embodiments, the super-resolution module 825 that upscales the video frames can utilize one or more neural network models to perform the upscaling.

[0132] The power optimization module 826 includes additional algorithms that reduce the power consumed by the TCON 355. In some embodiments, the power optimization module 826 includes a local contrast enhancement and global dimming module that reduces the power consumption of the display panel 380 by improving local contrast and applying global dimming to individual frames.

[0133] In some embodiments, the timing controller processing stack 820 may include more or fewer modules than those shown in FIG. 8 . For example, in some embodiments, the timing controller processing stack 820 includes an ALRR module and an eDP PSR2 module, but does not have a power optimization module. In other embodiments, additional modules to those shown in FIG. 8 may be included in the timing controller stack 820. The modules included in the timing controller processing stack 820 may depend on the type of built-in display panel 380 included in the lid 301. For example, if the display panel 380 is a backlit liquid crystal display (LCD), the timing controller processing stack 820 may not include modules that include the global dimming and local contrast power reduction techniques described above, as these techniques are more suitable for use in emissive displays (displays in which light-emitting elements are arranged in individual pixels, such as QLED, OLED, and microLED displays) than in backlit LCD displays. In some embodiments, the timing controller processing stack 820 includes color and gamma correction modules.

[0134] After the video data is processed by the timing controller processing stack 820, a P2P transmitter 880 converts the video data into signals that drive the control circuitry for the display panel 380. The control circuitry for the display panel 380 includes a row driver 882 and a column driver 884 that drive rows and columns of pixels in the display 380 within the embedded display panel 380, controlling the color and brightness of individual pixels.

[0135] In embodiments where the built-in panel 380 is a backlit LCD display, the TCON 355 may include a backlight controller 835 that generates signals to drive a backlight driver 840 to control the backlight of the display panel 380. The backlight controller 835 sends signals to the backlight driver 840 based on video frame data representing the image to be displayed on the panel 380. The backlight controller 835 can implement low-power features, such as turning off or reducing the brightness of the backlight for areas of the panel (or the entire panel) when large portions of the displayed image (or the entire image) appear dark. In some embodiments, the backlight controller 835 reduces power consumption by adjusting the saturation values ​​of pixels while reducing the brightness of the backlight so that the viewer perceives little or no visual degradation. In some embodiments, the backlight is controlled based on signals sent to the lid via the eDP auxiliary channel. This can reduce the number of wires routed across the hinge 330.

[0136] The touch controller 385 is responsible for driving the touchscreen technology of the built-in panel 380 and collecting touch sensor data from the display panel 380. The touch controller 385 can sample touch sensor data periodically or aperiodically and can receive control information from the timing controller 355 and / or the lid controller hub 305. The touch controller 385 can sample touch sensor data at a sampling rate similar to or close to the display panel refresh rate. The touch sampling can be adjusted in response to adjustments in the display panel refresh rate. Thus, if the display panel is being refreshed at a slower rate or not at all, the touch controller can be placed in a low-power state in which its touch sensor data is sampled at a slower rate or not at all. For example, in response to the vision / imaging module 363 detecting a user in image data being continuously analyzed by the vision / imaging module 363, the touch controller 385 can increase its touch sensor sampling rate or resume sampling touch sensor data when the computing device exits the low-power state. In some embodiments, the sampling of touch sensor data can be synchronized with the display panel refresh rate, as described in more detail below. This can enable a smooth and responsive touch experience. In some embodiments, the touch controller is capable of sampling touch sensor data at a rate that is independent of the display refresh rate.

[0137] Although the timing controllers 400 and 351 in Figures 2 and 3 are shown as being separate from the lid controller hubs 260 and 305, respectively, any of the timing controllers described herein may be integrated on the same die, package, or printed circuit board as the lid controller hub. Thus, a reference to a lid controller hub may refer to components that include the timing controller, and a reference to a timing controller may refer to components within the lid controller hub. Figures 10A through 10D show various possible physical relationships between the timing controllers and the lid controller hubs.

[0138] In some embodiments, the lid controller hub may have more or fewer components and / or implement fewer features or functionality than the LCH embodiments described herein. For example, in some embodiments, a mobile computing device may include an LCH without an audio module and perform processing of audio sensor data within the base. In another example, a mobile computing device may include an LCH without a vision / imaging module and perform processing of image sensor data within the base.

[0139] FIG. 9 shows a block diagram illustrating an example physical arrangement of components in a mobile computing device including a lid controller hub. The mobile computing device 900 includes a base 910 connected to a lid 920 via a hinge 930. The base 910 includes a motherboard 912 on which an SoC 914 and other computing device components are disposed. The lid 920 includes a bezel 922 that extends around the periphery of a display area 924, e.g., the active area of ​​an integrated display panel 926 disposed within the lid, which is the content-displaying portion of the integrated display panel. The lid 920 further includes a pair of microphones 926 at the top left and right corners of the lid 920 and a sensor module 928 disposed along the top center portion of the bezel 922. The sensor module 928 includes a forward-facing camera 932. In some embodiments, the sensor module 928 is a printed circuit board on which the camera 932 is mounted. The lid 920 further includes panel electronics 940 and lid electronics 950 disposed at the bottom portion of the lid 920. The lid electronics 950 includes a lid controller hub 954, and the panel electronics 940 includes a timing controller 944. In some embodiments, the lid electronics 950 includes a printed circuit board on which the LCH 954 is mounted. In some embodiments, the panel electronics 940 includes a printed circuit board on which the TCON 944 and additional panel circuitry, such as row and column drivers, a backlight driver (if the integrated display is an LCD backlit display), and a touch controller, are mounted. The timing controller 944 and the lid controller hub 954 communicate through a connector 958, which may be a cable connector connecting the two circuit boards. The connector 958 can transmit synchronization signals that allow touch sampling operations to be synchronized with the display refresh rate. In some embodiments, the LCH 954 can send power to the TCON 944 and other electronic components that are part of the panel electronics 940 through the connector 958.A sensor data cable 970 carries image sensor data generated by the camera 932, audio sensor data generated by the microphone 926, and touch sensor data generated by touchscreen technology to the lid controller hub 954. Wires carrying audio signal data generated by the microphone 926 may extend from the microphone 926 at the top left and right corners of the lid to the sensor module 928, where they are combined with wires carrying image sensor data generated by the camera 932 and sent via the sensor data cable 970 to the lid controller hub 954.

[0140] The hinge 930 includes a left hinge portion 980 and a right hinge portion 982. The hinge 930 physically couples the lid 920 to the base 910, allowing the lid 920 to rotate relative to the base. Wires connecting the lid controller hub 954 to the base 910 pass through one or both of the hinge portions 980 and 982. While the hinge 930 is shown as including two hinge portions, it may have a variety of different configurations in alternative embodiments. For example, the hinge 930 may include a single hinge portion or three or more hinge portions, and the wires connecting the lid controller hub 954 to the SoC 914 may cross the hinge at any of the hinge portions. By having fewer wires cross the hinge 930 than existing laptop devices, the hinge 930 may be a less expensive and simpler component than existing laptop hinges.

[0141] In other embodiments, the lid 920 may have a different sensor arrangement than that shown in FIG. 9 . For example, the lid 920 may include additional sensors, such as additional forward-facing cameras, forward-facing depth-sensing cameras, infrared sensors, and one or more world-facing cameras. In some embodiments, the lid 920 may include additional microphones located on the bezel or only one microphone located on the sensor module. The sensor module 928 may assemble wires carrying sensor data generated by additional sensors located within the lid and route them to the sensor data cable 970, which routes this additional sensor data to the lid controller hub 954.

[0142] In some embodiments, the lid includes in-display sensors, such as in-display microphones or in-display cameras, that generate light for each pixel and are located in the display area 924 in pixel areas not utilized by light-emitting elements, as described in more detail below. Data generated by the in-display camera sensors and in-display microphones can be compiled by the sensor module 928 and other sensor modules located in the lid that send sensor data generated by the in-display sensors to the lid controller hub 954 for processing.

[0143] In some embodiments, one or more microphones and cameras may be located in a location within the lid that is convenient for use in an "always on" usage scenario, such as when the lid is closed. For example, one or more microphones and cameras may be located on the "A-cover" of a laptop or other world-facing surface of a mobile computing device (such as a top or side edge of the lid) when the device is closed. This allows audio or image data to be captured and monitored to detect the utterance of an activation word or phrase or the presence of a person within the camera's field of view.

[0144] 10A through 10E show block diagrams of exemplary physical arrangements of a timing controller and a lid controller hub within a lid. FIG. 10A shows a lid controller hub 1000 and a timing controller 1010 located on a first module 1020 that is physically separate from a second module 1030. In some embodiments, the first and second modules 1020 and 1030 are printed circuit boards. The lid controller hub 1000 and the timing controller 1010 communicate via a connection 1034. FIG. 10B shows a lid controller hub 1042 and a timing controller 1046 located on a third module 1040. The LCH 1042 and the TCON 1046 communicate via a connection 1044. In some embodiments, the third module 1040 is a printed circuit board, and the connection 1044 comprises one or more printed circuit board traces. One advantage of applying a modular approach to lid controller hub and timing controller design is that it allows timing controller vendors to offer a single timing controller that works with multiple LCH designs with different feature sets.

[0145] FIG. 10C shows a timing controller separated into front-end and back-end components. The timing controller front-end (TCON FE) 1052 and lid controller hub 1054 are integrated or co-located on a first common component 1056. In some embodiments, the first common component 1056 is an integrated circuit package, and the TCON FE 1052 and LCH 1054 are separate integrated circuit dies integrated in a multi-chip package or separate circuits integrated on a single integrated circuit die. The first common component 1056 is located on a fourth module 1058, and the timing controller back-end (TCON BE) 1060 is located on a fifth module 1062. The timing controller front-end and back-end components communicate via connection 1064. Separating the timing controller into front-end and back-end components allows flexibility in developing timing controllers with various timing controller processing stacks. For example, the timing controller back-end may include a module that drives an integrated display. 8 , as well as other modules that may be common to various timing controller frame processor stacks, such as the P2P transmitter 880 of the timing controller processing stack 820 of FIG. 8 , and decoder or panel self-refresh modules. The timing controller front end may include modules specific to a particular mobile device design. For example, in some embodiments, the TCON FE includes a power optimization module 826 that performs global dimming and local contrast improvement desired for implementation in a particular laptop model, or an ALRR module that may be convenient to have the timing controller and lid controller hub components located nearby, operating synchronously (e.g., via synchronization signal 370), to reduce latency.

[0146] FIG. 10D illustrates an embodiment in which the second common component 1072 and the timing controller backend 1078 are located on the same module, i.e., the sixth module 1070, and the second common component 1072 and the TCON BE 1078 communicate via connection 1066. FIG. 10E illustrates an embodiment in which the lid controller hub 1080 and the timing controller 1082 are integrated on a third common component 1084 located on a seventh module 1086. In some embodiments, the third common component 1084 is an integrated circuit package, and the LCH 1080 and the TCON 1082 are individual integrated circuit die packages integrated into a multi-chip package or circuits located on a single integrated circuit die. In embodiments in which the lid controller hub and the timing controller are located on separate physical modules (e.g., FIGS. 10A and 10C), the connections between the modules may include wires, flexible printed circuits, printed circuits, or one or more other components that enable communication between the modules.

[0147] The modules and components in FIGS. 10C-10E, including the lid controller hub and timing controller (eg, the fourth module 1058, the second common component 1072, and the third common component 1084), may be referred to as a lid controller hub.

[0148] 11A-11C show a table of hinge wire count breakdowns for various lid controller hub embodiments. Display wires carry video data from the SoC display module to the LCH timing controller. Image wires carry image sensor data generated by one or more lid cameras to the SoC image processing module. Touch wires provide touch sensor data to the SoC built-in sensor hub. Audio and sensing wires provide audio sensor data to the SoC audio capture module and other types of sensor data to the built-in sensor hub. Additional "LCH" wire pairs enable additional communication between the LCH and the SoC. The types of sensor data provided by the audio and sensing wires can include visual sensing data generated by vision-based input sensors such as fingerprint sensors, vein sensors, etc. In some embodiments, vision sensing data can be generated based on information generated by one or more general-purpose cameras rather than dedicated biometric sensors such as fingerprint sensors.

[0149] Table 1100 shows the wire breakdown for a 72-wire embodiment. Display wires include 19 data wires and 16 power wires for a total of 35 wires to support four eDP HBR2 lanes and six signals for original equipment manufacturer (OEM) use. Image wires include six data wires and eight power wires for a total of 14 wires to carry image sensor data generated by a single 1-megapixel camera. Touch wires include four data wires and two power wires for a total of six wires to support an I2C connection to carry touch sensor data generated by a touch controller. Audio and sensing wires include eight data wires, two power wires, and a single interrupt (INT) wire for a total of 10 wires to support a DMIC and I2C connection to support audio sensor data generated by four microphones. Seven additional data wires carry additional information for communication between the LCH and the SoC via USB and QSPI connections.

[0150] Table 1110 shows a wire breakdown for a 39-wire embodiment. Here, providing dedicated wires to power the lid components and omitting various data signals contributes to the wire count reduction. Display wires include 14 data wires and four power wires for a total of 18 wires supporting two eDP HBR2 lanes, six OEM signals, and power delivery to the lid. Power provided via the four power wires powers the lid controller hub and other lid components. Power resources within the lid receive power provided via dedicated power wires from the base and control power delivery to the lid components. Image wires, touch wires, and audio and sensing wires include the same number of data wires as the embodiment shown in Table 1100, but do not include power wires because power is provided separately to the lid. Three additional data wires carry additional information between the LCH and SoC, reduced from seven in the embodiment shown in Table 1100.

[0151] Table 1120 shows the wire breakdown for a 29-wire embodiment. Here, further wire count reduction is achieved by utilizing the existing USB bus to also carry touch sensor data and eliminating the six display data wires that carry OEM signals. The display wires include eight data wires and four power wires for a total of 12 wires. The image wires include four data wires for each of the two cameras: a 2-megapixel RGB (red-green-blue) camera and an infrared (IR) camera. The audio and sensing wires include four wires (less than half the number of wires in the embodiment shown in Table 1110) to support a SoundWire® connection to carry audio data for four microphones. There are no wires dedicated to touch sensor data transmission; five wires are used to communicate touch sensor data. Additional information is communicated between the LCH and SoC via the USB connection. Thus, Tables 1100 and 1120 show that wire count reduction is possible by powering the lid over a dedicated power wire set, reducing the number of eDP channels, utilizing an existing connection (USB) to send touch sensor data, and omitting OEM-specific signals. Further reduction in hinge wire count can be achieved by streaming video data from the base to the lid, and audio, touch, image, and sensing data from the lid to the base through a single interface. In some embodiments, this single connection can include a PCIe connection.

[0152] In embodiments different from those summarized in Tables 1100, 1110, and 1120, the hinge may have more or fewer total wires, more or fewer wires to carry each type of signal listed (display, image, touch, audio and sensing, etc.), and may utilize different connection and interface technologies than those shown in Tables 1100, 1110, and 1120.

[0153] As described above, the lid may include an in-display camera and an in-display microphone in addition to the camera and microphone located in the lid bezel. Figures 12A through 12C show example placements of in-display microphones and cameras on the lid. Figure 12A shows a lid 1200 including a bezel 1204, an in-display microphone 1210, and a display area 1208. The bezel 1204 bounds the display area 1208, which is defined by a plurality of pixels located on a display substrate (not shown). The pixels extend to an inner edge 1206 of the bezel 1204, such that the display area 1208 extends from one inner bezel edge 1206 to the opposite bezel edge 1206 in both the horizontal and vertical directions. The in-display microphone 1210 shares area with the pixel display element, as described in more detail below. The microphones 1210 include a set of microphones located in the peripheral region of the display area 1208 and a microphone located approximately in the center of the display area 1208. Figure 12B shows a lid 1390 in which the in-display microphones 1400 include a set of microphones located in the peripheral region of the display area 1270, a microphone located approximately in the center of the display area 1270, and four additional microphones distributed across the display area 1270. Figure 12C shows a lid 1280 in which an array of in-display microphones 1290 is located within the display area 1295 of the lid 1280. In other embodiments, a display may have a different number and arrangement of in-display microphones than the exemplary configuration shown in Figures 12A-12C.

[0154] 12A-12C further illustrate exemplary placements of forward-facing cameras in built-in display panels, where 1210, 1400, and 1290 indicate in-display cameras rather than microphones. In some embodiments, built-in display panels may include a combination of in-display microphones and cameras. Built-in displays may include different numbers and placements of in-display cameras or combinations of in-display cameras and in-display microphones than the exemplary configurations shown in FIGS. 12A-12C.

[0155] 13A-13B show simplified cross-sections of pixels in an exemplary emissive display. Figure 13A shows a simplified diagram of a cross-section of a pixel in an exemplary microLED display. MicroLED pixel 1300 includes a display substrate 1310, a red LED 1320, a green LED 1321, a blue LED 1322, electrodes 1330-1332, and a transparent display media 1340. LEDs 1320-1322 are individual light-generating elements for pixel 1300, where the amount of light generated by each LED 1320-1322 is controlled by the associated electrode 1330-1332.

[0156] The LED stacks (red LED stack (layers 1320 and 1330), green LED stack (layers 1321 and 1331), and blue LED stack (layers 1322 and 1332)) can be fabricated on a substrate using microelectronic fabrication techniques. In some embodiments, the display substrate 1310 is a different substrate than the substrate on which the LED stacks are fabricated, and the LED stacks are transferred from the fabrication substrate to the display substrate 1310. In another embodiment, the LED stacks are built directly on the display substrate 1310. In both embodiments, multiple pixels can be placed on a single display substrate, and multiple display substrates can be assembled, to achieve a display of the desired size.

[0157] Pixel 1300 has a pixel width 1344, which may depend, for example, on the display resolution and display size. For example, for a given display resolution, the display size may increase with pixel width 1344. For a given display size, the pixel width 1344 may decrease as the resolution increases. Pixel 1300 has an unused pixel area 1348, which is a portion of the black matrix area of ​​the display. In some displays, the combination of LED size, display size, and display resolution allows the unused pixel area 1348 to be large enough to accommodate integration of components such as a microphone within the pixel.

[0158] 13B shows a simplified diagram of a cross-section of a pixel in an exemplary OLED display. OLED pixel 1350 includes a display substrate 1355 and organic light-emitting layers 1360 through 1362 capable of generating red (layer 1360), green (layer 1361), and blue (layer 1362) light, respectively. OLED pixel 1350 further includes cathode layers 1365 through 1367, electron-injection layers 1370 through 1372, electron-transport layers 1375 through 1377, anode layers 1380 through 1382, hole-injection layers 1385 through 1387, hole-transport layers 1390 through 1392, and a transparent display medium 1394. OLED pixel 1350 generates light by applying a voltage across the cathode layers 1365 through 1367 and anode layers 1380 through 1382. This results in electrons and holes being injected into the electron injection layers 1370 to 1372 and hole injection layers 1385 to 1387, respectively. The injected electrons and holes traverse the electron transport layers 1375 to 1377 and hole transport layers 1390 to 1392, respectively, and the electron-hole pairs recombine in the organic light-emitting layers 1360 to 1362, respectively, to generate light.

[0159] Similar to the LED stacks in a micro LED display, the OLED stacks (red OLED stack (layers 1365, 1370, 1375, 1360, 1390, 1385, 1380), green OLED stack (layers 1366, 1371, 1376, 1361, 1391, 1386, 1381), and blue OLED stack (layers 1367, 1372, 1377, 1362, 1392, 1387, 1382) can be fabricated on a substrate separate from the display substrate 1355. In some embodiments, the display substrate 1355 is The display substrate 1355 is a different substrate than the substrate on which the OLED stack is transferred from the manufacturing substrate. In another embodiment, the OLED stack is built directly on the display substrate 1355. In both types of embodiments, multiple display substrate components can be assembled to achieve a desired display size. The transparent display media 1340 and 1394 can be any transparent medium, such as glass, plastic, or film. In some embodiments, the transparent display medium can include a touch screen.

[0160] Again, similar to the microLED pixel 1300, the OLED pixel 1350 has a pixel width 1396, which may depend on factors such as the display resolution and display size. The OLED pixel 1350 has an unused pixel area 1398, and in some displays, the combination of OLED stack width, display size, and display resolution allows the unused pixel area 1398 to be large enough to accommodate integration of components such as a microphone within the pixel.

[0161] As used herein, the term "display substrate" can refer to any substrate used in a display on which pixel display elements are fabricated or placed. For example, the display substrate can be fabricated separately from the pixel display elements (e.g., the micro LEDs / OLEDs in pixels 1300 and 1350) and can be a backplane on which the pixel display elements are attached, or a substrate on which the pixel display elements are fabricated.

[0162] FIG. 14A shows an example set of pixels with built-in microphones. Pixels 1401 through 1406 each have a red display element 1411, a green display element 1412, and a blue display element 1413, which may be, for example, a microLED or OLED. Each of pixels 1401 through 1406 occupies a pixel area. For example, pixel 1404 occupies pixel area 1415. The amount of pixel area occupied by display elements 1411 through 1413 in each pixel leaves enough black matrix space for a miniature microphone to be included. Pixels 1401 and 1403 house forward-facing microphones 1420 and 1421, respectively, positioned along display elements 1411 through 1413. Because the rear-facing microphones are located on the backside of the display substrate, they are not limited by unused pixel area or display element size and can be located anywhere on the backside of the display substrate. For example, rear-facing microphone 1422 straddles pixels 1402 and 1403.

[0163] Figure 14B shows a cross-section of the example pixel of Figure 14A along line AA'. Cross-section 1450 shows a cross-section of pixels 1401-1403. Green display element 1412 and corresponding electrodes 1430 for pixels 1401-1403 are disposed on a display substrate 1460. Pixels 1401-1403 are covered by a transparent display medium 1470 that has holes 1474 over microphones 1420 and 1421, allowing acoustic vibrations to reach display surface 1475 and reach microphones 1420 and 1421. Rear-facing microphone 1422 is disposed on the back side of display substrate 1460. In some embodiments, the display housing (not shown) in which pixels 1401-1403 are disposed has vents or other openings to allow acoustic vibrations to reach the back of the housing and reach rear-facing microphone 1422.

[0164] In some embodiments, the microphones used in the techniques described herein may be discrete microphones that are manufactured or created independently of the pixel display elements and transferred from the manufacturing substrate or otherwise attached to the display substrate. In another embodiment, the microphones may be fabricated directly on the display substrate. While the forward-facing microphones are shown in FIG. 14B as being located on the surface of the display substrate 1460, they may reside at least partially within the display substrate in embodiments in which the microphones are fabricated on the display substrate.

[0165] As used herein, the term "disposed on" with respect to any sensor (microphone, piezoelectric element, temperature sensor) relative to the display substrate refers to a sensor physically coupled to the display substrate in any manner (e.g., an individual sensor attached directly to the substrate, an individual sensor attached to the substrate via one or more intervening layers, a sensor fabricated on the display substrate). As used herein, the term "disposed on" with respect to an LED relative to the display substrate refers similarly to an LED physically coupled to the display substrate in any manner (e.g., an individual LED attached directly to the substrate, an individual LED attached to the substrate via one or more intervening layers, an LED fabricated on the display substrate). In some embodiments, the front-facing microphones are positioned in the peripheral area of ​​the display to reduce any visual obstruction that holes in the display above the front-facing microphones (such as hole 1474) may present to a user. In other embodiments, the holes on the microphones may be small enough, or infrequent enough, to present no or little obstruction to the visual experience.

[0166] While forward-facing microphones 1420 and 1421 are each shown as being within a single pixel, in another embodiment, the forward-facing microphones may span multiple pixels. This may allow, for example, larger microphones to be integrated into the display area or microphones to be integrated into displays with smaller pixels. Figures 14C-14D show exemplary microphones spanning multiple pixels. Figure 14C shows adjacent pixels 1407 and 1408 having the same size as pixels 1401-1406, and a forward-facing microphone 1440 that is larger than front-facing microphones 1420-1421 and occupies pixel area unused by the display elements in the two pixels. Figure 14D shows adjacent pixels 1409 and 1410 that are narrower than pixels 1401-1406, and a forward-facing microphone 1426 that spans both pixels. Using larger microphones may enable improved acoustic performance of the display, such as improved sound detection. A display with many built-in miniature microphones distributed across the display area can have sound detection capabilities that exceed those of a display with only one or a few individual microphones located within the display bezel.

[0167] In some embodiments, the microphones described herein are MEMS (microelectromechanical systems) microphones. In some embodiments, the microphones generate analog audio signals that are provided to an audio processing component, while in other embodiments, the microphones provide digital audio signals to the audio processing component. Microphones that generate digital audio signals may include a local AD converter and provide digital audio output in pulse density modulation (PDM), I2S (Inter-IC Sound), or other digital audio signal formats. In embodiments in which the microphone generates digital audio signals, the audio processing component may not include an AD converter. In some embodiments, the built-in microphone is a MEMS PDM microphone having dimensions of approximately 3.5 mm (width) x 2.65 mm (length) x 0.98 mm (height).

[0168] Microphones can be integrated into individual pixels or across several pixels using the techniques described herein. Thus, a variety of microphone configurations can be incorporated into the display. Figures 12A-12C and 14A-14D show some microphone configurations. Many more are possible. The display-integrated microphones described herein generate audio signals that are sent to an audio module in the lid controller hub (e.g., audio module 364 in Figures 3 and 7).

[0169] Displays described herein with microphones integrated within the display area can perform various audio processing tasks. For example, a display with multiple forward-facing microphones distributed across the display area can perform beamforming or spatial filtering on audio signals generated by the microphones to enable far-field functionality (e.g., improved detection of sounds generated by distant sound sources). The audio processing component can determine the location of a distant sound source, select a microphone subset based on the source location, and use audio signals from the selected microphone subset to improve detection of sounds received by the display from the sound source. In some embodiments, the audio processing component can determine the location of the sound source by determining delays to be applied to audio signals generated by various combinations of microphones so that the audio signals overlap in time, and estimating the distance of the sound source from each microphone in the combination based on the delays applied to each audio signal. Applying the determined delays to the audio signals provided by the microphones can improve audio detection in the direction of the distant sound source. The total number of microphone subsets in the display can be used for beamforming or spatial filtering, and microphones not included in the subset can be turned off to reduce power. Beamforming can also be achieved using rear-facing microphones distributed across the backside of the display substrate. Compared to displays with some microphones integrated into the display bezel, displays with microphones integrated within the viewing area can provide improved beamforming due to the greater number of microphones integrated within the display and distributed over a wider area.

[0170] In some embodiments, the display is configured with a set of rear-facing microphones distributed across the display area. This allows a closable device incorporating the display to have audio detection capabilities when the display is closed. For example, a closed device can enter a low-power mode in which the rear-facing microphones and audio processing components are enabled, enabling activation expression or term detection, or specific user identification (speaker ID).

[0171] In some embodiments, a display with both front- and rear-facing microphones can utilize both types of microphones for noise reduction, improved audio detection (far-field audio), and improved audio recording. For example, if a user is operating a laptop in a noisy environment, such as a coffee shop or cafeteria, audio signals from one or more rear-facing microphones that pick up ambient noise can be used to reduce noise from audio signals provided by the front-facing microphones, including the laptop user's voice. In another example, audio recordings made by a device incorporating such a display can include audio received by both the front- and rear-facing microphones. By including audio captured by both the front- and rear-facing microphones, the recording can provide a more accurate audio representation of the recording environment. In a further example, a display including both front- and rear-facing microphones can achieve 360-degree far-field audio reception. For example, the beamforming or spatial filtering techniques described herein can be applied to audio signals provided by both the front- and rear-facing microphones to provide improved audio detection.

[0172] Displays with built-in microphones located within the display area have advantages over displays with microphones located within the display bezel. Displays with microphones located within the display area can have narrower bezels because bezel space is not required to accommodate the built-in microphones. Displays with reduced bezel widths may have a better aesthetic appearance for viewers and allow for a larger display area within a given display housing size. Integrating microphones within the display area allows for a greater number of microphones to be included within the device, which may enable improved audio detection and noise reduction. Furthermore, displays with microphones located across the display area enable displays with improved audio detection capabilities through the use of beamforming or spatial filtering of received audio signals, as described above. Furthermore, the cost and complexity of routing audio signals from microphones located within the display area to audio processing components also located within the display area may be less than wiring individual microphones located in the display bezel to audio processing components located outside the display.

[0173] FIG. 15A shows an example set of pixels along with an in-display camera. Pixels 1501 through 1506 each have a red display element 1511, a green display element 1512, and a blue display element 1513. In some embodiments, the display elements are microLEDs or OLEDs. Each of pixels 1501 through 1506 occupies a pixel area. For example, pixel 1504 occupies pixel area 1515. In each pixel, the amount of pixel area occupied by display elements 1511 through 1513 leaves enough black matrix area for a small camera or other sensor to be included. Pixels 1501 and 1503 house cameras 1520 and 1521, respectively, positioned along display elements 1511 through 1513. As used herein, the term "in-display camera" refers to a camera positioned within the pixel area of ​​one or more pixels within a display. Cameras 1520 and 1521 are in-display cameras.

[0174] FIG. 15B shows a cross-section of the example pixel of FIG. 15A taken along line AA′. Cross-section 1550 shows a cross-section of pixels 1501 through 1503. Green display element 1512 and corresponding electrodes 1530 for pixels 1501 through 1503 are disposed on display substrate 1560 and behind transparent display media 1570. A camera disposed in the display area can receive light that has or has not passed through the transparent display media. For example, the area of ​​transparent display media 1570 on camera 1520 is perforated, allowing light to enter the image sensor of camera 1520 without having to pass through the transparent display media. The area of ​​transparent display media 1570 on camera 1521 is not perforated, allowing light reaching the image sensor in camera 1521 to pass through the transparent display media.

[0175] In some embodiments, the in-display camera may be a separate camera fabricated independently of the pixel display elements, with the separate camera attached to the display substrate after fabrication. In another embodiment, one or more camera components, such as an image sensor, may be fabricated directly on the display substrate. While cameras 1520-1521 are shown in FIG. 15B as being disposed on front surface 1580 of display substrate 1560, in embodiments in which the camera components are fabricated on the display substrate, the cameras may reside at least partially within the display substrate.

[0176] As used herein, the term "disposed on" with respect to any sensor or component (e.g., camera, temperature sensor) relative to the display substrate refers to a sensor or component physically coupled to the display substrate in any manner, such as a discrete sensor or other component attached directly to the substrate, a discrete sensor or component attached to the substrate via one or more intervening layers, or a sensor or component fabricated on the display substrate. As used herein, the term "disposed on" with respect to an LED relative to the display substrate similarly refers to an LED physically coupled to the display substrate in any manner, such as a discrete LED attached directly to the substrate, a discrete LED attached to the substrate via one or more intervening layers, or an LED fabricated on the display substrate.

[0177] While cameras 1520 and 1521 are each shown as residing within a single pixel in FIGS. 15A and 15B, in other embodiments, the in-display camera can span multiple pixels. This may allow, for example, a larger camera to be integrated into the display area or a camera to be incorporated into a display with pixels having smaller black matrix areas. FIGS. 15C-15D show exemplary cameras that span multiple pixels. FIG. 15C shows adjacent pixels 1507 and 1508 that are the same size as pixels 1501-1506, and camera 1540 that is larger than cameras 1520-1521 and occupies a portion of the pixel area within pixels 1507-1508. FIG. 15D shows adjacent pixels 1509 and 1510 that are narrower than pixels 1501-1506, and camera 1526 that spans both pixels. Using larger cameras may allow for improved image or video capture, such as allowing higher resolution images and video to be captured with each individual camera.

[0178] Because cameras can be integrated within individual pixels or across several pixels, a variety of camera configurations can be incorporated within a display. Figures 12A-12C and 15A-15D show just a few example camera configurations; many more are possible. In some embodiments, thousands of cameras can be located across the display area. A display with multiple in-display cameras distributed across the display area can have superior image and video capture capabilities than a display with only one or a few cameras located within the bezel.

[0179] The in-display cameras described herein generate image sensor data that is sent to a vision / imaging module in a lid controller hub, such as the vision / imaging module 363 in FIGS. 3 and 6. Image sensor data is data output from the camera to another component. The image sensor data can be image data, which is data representing an image, or video data, which is data representing a video. The image data or video data can be in compressed or uncompressed format. The image sensor data can also be data where the image data or video data is generated by another component (e.g., the vision / imaging module 363 or any component thereof).

[0180] Interconnects that provide image sensor data from the camera to the lid controller hub may be located within the display substrate. The interconnects may be fabricated on the display substrate, attached to the display substrate, or physically coupled to the display substrate in any other manner. In some embodiments, manufacturing a display involves fabricating separate display substrate portions onto which pixels are attached and assembling the display substrate portions to achieve the desired display size.

[0181] FIG. 16 shows an exemplary camera that can be incorporated into an integrated display. The camera 1600 is disposed on a display substrate 1610 and includes an image sensor 1620, an aperture 1630, and a microlens assembly 1640. The image sensor 1620 can be a CMOS photodetector or any other type of photodetector. The image sensor 1620 includes multiple camera pixels, the individual elements used to capture light within the camera, and the number of pixels within the camera can be used as a measure of the camera's resolution (e.g., 1 megapixel, 12 megapixels, 20 megapixels). The aperture 1630 has an aperture width 1635. The microlens assembly 1640 includes one or more microlenses 1650 that focus light to a focal point. It can be made of glass, plastic, or other transparent material. The microlens assembly 1640 typically includes multiple microlenses to accommodate various types of aberrations (such as chromatic aberration) and distortion.

[0182] Camera 1655 is further disposed on display substrate 1610 and includes image sensor 1660, aperture 1670, and metalens 1680. Camera 1655 is similar to camera 1600, except that camera 1655 uses a metalens instead of a microlens assembly as the focusing element. Generally, a metalens is a planar lens that includes physical structures on its surface that act to manipulate different wavelengths of light so that they reach the same focal point. Metalenses do not produce chromatic aberrations, as can occur with a single existing microlens. Metalenses can be significantly thinner than glass, plastic, or other types of microlenses and can be fabricated using MEMS (microelectromechanical systems) or NEMS (nanoelectromechanical systems) techniques. Thus, a camera including a single, thin metalens, such as camera 1655, can be thinner than a camera including a microlens assembly containing multiple microlenses, such as camera 1600. Aperture 1670 has an aperture width 1675.

[0183] The distances from microlens assembly 1640 to image sensor 1620 and from metalens 1680 to image sensor 1660 define the focal length of cameras 1600 and 1655, respectively. The ratio of the focal length to the aperture width (1635, 1675) defines the camera's f-stop, which is a measure of the amount of light reaching the surface of the image sensor. The f-stop, in turn, is a measure of the camera's depth of field. A camera with a small f-stop will have a shallower depth of field, while a camera with a large f-stop will have a deeper depth of field. Depth of field can have a significant effect on the captured image. In an image with a shallow depth of field, often only the subject of the photograph is in focus. On the other hand, in an image with a deep depth of field, most objects are typically in focus.

[0184] In some embodiments, cameras 1600 and 1655 are fixed-focus cameras; that is, their focal lengths are not adjustable. In other embodiments, the focal lengths of cameras 1600 and 1655 can be adjusted by moving microlens assembly 1640 or metalens 1680 either closer to or farther away from the associated image sensor. In some embodiments, the distance of microlens assembly 1640 or metalens 1680 relative to their respective image sensors can be adjusted using MEMS-based actuators or other techniques.

[0185] In-display cameras can be distributed across the display area at various densities. For example, cameras can be placed every 100 pixels, every 10 pixels, adjacent pixels, or other densities. A certain level of camera density (how many cameras per unit area of ​​display area) can be desirable in certain usage situations. For example, if cameras are used for image and video capture, a lower camera density may be sufficient than if cameras are used for touch detection or touch location determination.

[0186] In some embodiments, image data corresponding to images captured by multiple individual cameras can be used to generate a composite image. The composite image can have a higher resolution than any image the individual cameras can capture. For example, a system can generate a 6-megapixel image using image data corresponding to images captured by several 3-megapixel cameras. In some embodiments, the composite image or video generated from images or videos captured by individual in-display cameras can have an ultra-high resolution, such as in the gigapixel range. The composite images and videos can be used for ultra-high resolution selfie images or videos, ultra-high resolution security monitors, or other applications.

[0187] Generating higher resolution images from image data corresponding to images captured by multiple individual cameras may allow individual cameras with fewer megapixels per individual pixel. This may allow cameras to be incorporated into higher resolution displays of a given screen size or into smaller displays of a given resolution (more pixels per unit area of ​​display size, and therefore less free pixel area available for cameras that can be incorporated at the pixel level). A composite image may be generated in real time along with image capture, in which case only the image data for the composite image is stored. Alternatively, image data for images captured by individual cameras may be stored, and the composite image may be generated during post-processing. Similarly, a composite video may be generated using video data corresponding to video generated by multiple individual cameras, where the composite video is generated in real time or during post-processing.

[0188] In some embodiments, an in-display camera may be used instead of a touchscreen to detect an object (e.g., a finger, a stylus) touching the display surface and determine where on the display the touch occurred. While some existing touchscreen technologies (e.g., resistive-based, capacitive-based) can increase display thickness by adding multiple layers on top of the transparent display media, others use in-cell or on-cell touch technology to reduce display thickness. As used herein, the term "transparent display media" includes the touchscreen layer, regardless of whether the touchscreen layer is disposed on the transparent display media or whether the transparent display media is used within the touchscreen layer. Some existing touchscreen technologies employ transparent conductive surfaces stacked together while separated by an insulating layer. These additional layers may add thickness to the display and reduce the display's optical transparency. Eliminating the use of a separate touchscreen layer may reduce display costs, as the transparent conductors used in touchscreens are typically made of indium tin oxide, which can be expensive.

[0189] For example, touch detection and touch location determination can be performed using an in-display camera by detecting the blocking of visible or infrared light caused by an object touching or proximate to the display. A touch detection module, which may be located within the display or otherwise communicatively coupled to the display, can receive images captured by the in-display camera and process the image data to detect one or more touches on the display surface and determine the touch location. Touch detection can be performed, for example, by determining whether image sensor data indicates that light received by the image sensor has decreased below a threshold value. In another example, touch detection can be performed by determining whether image sensor data indicates that light received by the camera has decreased by a determined percentage or amount. In yet another example, touch detection can be performed by determining whether image sensor data indicates that light received by the camera has decreased by a predetermined percentage or amount within a predetermined time period.

[0190] Touch location determination can be performed, for example, by using as the touch location the location of a camera whose associated image sensor data indicates that a touch has been detected on the display (e.g., the associated image sensor data indicates that the light received at the camera's image sensor has decreased below a threshold, decreased by a predetermined percentage or amount, or decreased by a predetermined percentage or amount within a predetermined time). In some embodiments, the touch location is based on the location within the image sensor where the lowest level of light was received. When image sensor data associated with multiple adjacent cameras indicates a touch, the touch location can be determined by determining the centroid location of the multiple adjacent cameras.

[0191] In some embodiments, a touch-enabled display that utilizes in-display cameras for touch detection and touch location determination may have a higher camera density than a non-touch display that includes an in-display camera. However, a display that is touch-enabled by using in-display cameras does not need to have a camera located at every pixel. The touch detection module may use image sensor data from one or more cameras to determine touch location. The density of the in-display cameras may depend in part on the touch detection algorithm used.

[0192] Information indicating the presence of a touch and touch location information may be provided to the operating system, an application, or any other software or hardware component of the system, including or communicatively coupled to the display. Multiple touches may also be detected. In some embodiments, the in-display camera may provide updated image sensor data to the touch detection module frequently enough to achieve the touch display experience users expect from modern touch-enabled devices. The touch detection functionality of the display may be temporarily disabled when the in-display camera is being utilized for other purposes described herein.

[0193] In some embodiments, in situations where the system prompts a user to touch the display with their finger, thumb, or palm for user authentication, when the system detects the touch, the system may illuminate one or more pixel display elements located at or near the location where the touch was detected, thereby illuminating the area touched by the user's finger, thumb, or palm. This illumination may enable capture of a fingerprint, thumbprint, or palmprint, making the print features more distinguishable or more easily extractable for the system or device.

[0194] The use of an in-display camera enables touches on a display surface to be detected by a wider variety of objects than can be detected by existing capacitive touchscreen technology. Capacitive touchscreens detect touches on a display by detecting localized changes in the electrostatic field generated by the capacitive touchscreen. Thus, capacitive touchscreens can detect when a conductive object, such as a finger or a metallic stylus, is touching or in proximity to the display surface. Because the in-display camera relies on light occlusion to detect touch rather than sensing changes in capacitance at the display surface, the in-display camera-based approach to touch detection can detect touches from a wider variety of objects, including a passive stylus. There is no limitation that the touching object must be conductive or capable of generating changes in the electrostatic field of another display.

[0195] In some embodiments, an in-display camera can be used to detect gestures that a user can use to interface with a system or device. A display incorporating an in-display camera can enable recognition of two-dimensional (2D) gestures (e.g., swipes, taps, pinches, unpinch) with one or more fingers or other objects on the display surface, or three-dimensional (3D) gestures with a stylus, finger, hand, or other object performed within the volume of space in front of the display. As used herein, the phrase "3D gesture" or "air gesture" refers to a gesture that is performed, at least in part, within the volume of space in front of the display without touching the display surface.

[0196] A twist gesture may be mapped to an action performed by an operating system or an application running on the system. For example, a twist gesture may result in an object manipulation in a computer-aided design (CAD) application. For example, a twist gesture may result in a deformation of a selected object in the CAD application. That is, the application holds one end of the object fixed and rotates the opposite end of the object by an amount determined to correspond to the amount the user has twisted the physical object. For example, in response to the system detecting that a user has twisted a stylus in front of a display, a 3D cylinder in a CAD program may be selected and twisted about its longitudinal axis. The resulting deformed cylinder may appear like a twisted piece of licorice. The amount of rotation, distortion, or other manipulation performed on a selected object in response to detecting the rotation of a physical object in front of the display need not correspond one-to-one to the detected amount of rotation of the physical object. For example, in response to detecting that the stylus has been rotated 360 degrees, the selected object may be rotated 180 degrees (half the detected amount of rotation), 720 degrees (twice the detected amount of rotation), or any other amount proportional to the detected amount of rotation of the physical object.

[0197] A system incorporating or communicatively coupled to a display with an in-display camera can capture 3D gestures across a larger spatial volume in front of the display than can be captured by just a few cameras positioned within the display bezel. This is because the in-display cameras, which can be positioned across the display, have a larger total viewing area than the total viewing area of ​​a few bezel cameras. If a display only has one or more cameras positioned within the display bezel, these cameras are unlikely to be able to capture 3D gestures made away from the bezel (e.g., in the central region of the display) or near the display surface. Multiple cameras positioned within the display area can also be used to capture depth information for 3D gestures.

[0198] The ability to recognize 3D gestures in front of the display area enables gesture detection and recognition not possible with resistive or capacitive touchscreens or displays that include bezel cameras. For example, a system incorporating an in-display camera can detect 3D gestures that begin or end with a touch on the display. For example, a "pick-up-and-place" gesture may include a user performing a pinch gesture on the display surface to select an object shown at or near the pinched finger position (the pinch position), lifting the object by moving the pinched fingers away from the display surface, moving the object by moving the pinched fingers along a path from the pinched finger position to a destination position, and placing the object by moving the pinched fingers back toward the display surface until they touch the display surface and un-pinch the fingers at the destination position.

[0199] During a "pick up and place" gesture, the selected object may change from an unselected appearance to a selected appearance upon detection of the pinch portion of the gesture, the selected object may move across the display from the pinch location to a destination location upon detection of the movement portion of the gesture, and the selected object may change back to an unselected appearance upon detection of the place portion of the gesture. The gesture may be used to manipulate objects in a three-dimensional environment rendered on the display. The three-dimensional environment may be part of a CAD application or a game. The three-dimensional nature of the gesture may be manifested, for example, by the selected object not interacting with other objects in the environment positioned along the selected object's path of movement during movement between the pinch location and the destination location. That is, the selected object is lifted and floats above other objects in the application via the 3D "pick up and place" gesture.

[0200] Variations of this gesture may also be recognized. For example, a "pick up and drop" gesture may involve a user grasping an object with a pinch gesture, then lifting the object by moving the pinched fingers away from the display surface, and then "dropping" the object by unpinching the fingers while still positioned above the display. An application may generate a response to detecting that the picked object has been dropped. The magnitude of the response may correspond to the "height" at which the object was dropped. The height corresponds to the distance from the display surface at which the pinched fingers were determined to be located when they were unpinched. In some embodiments, the application's response to the object being dropped may correspond to one or more attributes, such as the weight of the dropped object.

[0201] For example, in a gaming application, a system may detect that a user has lifted a boulder by detecting a pinch gesture at a location on the display where the boulder is indicated, detect that the user has moved the pinched fingers a certain distance from the display surface, and detect that the user has unpinched the pinched fingers a certain distance away from the display surface. The application may interpret the unpinched fingers a certain distance away from the display screen as the boulder falling from a certain height. The gaming application may modify the gaming environment to an extent corresponding to the "height" at which the boulder was dropped, which corresponds to the distance from the display surface at which the system determined the pinched fingers were unpinched, and the weight of the boulder. For example, if the boulder is dropped from a low height, a small crater may be formed in the environment, and the application may generate a soft thud as the boulder hits the ground. If the boulder is dropped from a higher height, a larger crater may be formed, nearby trees may be knocked down, and the application may generate a loud crash as the boulder hits the ground. In another embodiment, when determining the magnitude of the reaction, the application may take into account the attributes of the boulder, such as its weight: heavier boulders will cause greater changes to the game environment when they fall.

[0202] In some embodiments, the measurement of the distance of an unpinched or pinched finger from the display surface may be determined by the size of the fingertip extracted from image sensor data generated by an in-display camera. A larger extracted fingertip size indicates that the fingertip is closer to the display surface. The determined distance of an unpinched or pinched finger need not be determined according to a standardized measurement system (e.g., metric, imperial). It may be any metric such that fingers positioned further away from the display surface are at a greater distance from the display surface than fingers positioned closer to the display surface.

[0203] FIG. 17 shows a block diagram illustrating an exemplary software / firmware environment for a mobile computing device including a lid controller hub. The environment 1700 includes a lid controller hub 1705 and timing controller 1706 located in a lid 1701 that communicate with components located in a base 1702 of the device. The LCH 1705 includes a security module 1710, a host module 1720, an audio module 1730, and a vision / imaging module 1740. The security module 1710 includes a boot module 1711, a firmware update module 1712, a flash file system module 1713, a GPIO privacy module 1714, and a CSI privacy module 1715. In some embodiments, any of the modules 1711-1715 may operate on or be implemented in one or more of the security module 1710 components shown in FIGS. 2-4 or other components disclosed herein. The boot module 1711 activates the security module in response to the computing device being booted. In some embodiments, boot module 1711 activates additional components of LCH 1705 in response to the computing device being powered up. Firmware update module 1712 updates the firmware used by security module 1710, allowing modules 1711 and 1713 through 1715 to be updated. Flash file system module 1713 implements a file system for firmware and other files stored in flash memory accessible by security module 1710. GPIO and CSI privacy modules 1714 and 1715 control accessibility by LCH components to image sensor data generated by a camera located within the lid.

[0204] Host module 1720 includes a debug module 1721, a telemetry module 1722, a firmware update module 1723, a boot module 1724, a virtual I2C module 1725, a virtual GPIO 1726, and a touch module 1727. In some embodiments, any of modules 1721 through 1727 may run on or be implemented in one or more of the host module components shown in FIGS. 2, 3, and 5 or other components disclosed herein. Boot module 1724 activates host module 1720 in response to the computing device being powered up. In some embodiments, boot module 1724 activates additional components of LCH 1705 in response to the computing device being powered up. Firmware update module 1723 updates firmware used by host module 1720, thereby enabling the updating of modules 1721 through 1722 and 1739 through 1727. Debug module 1721 provides debug functionality for host module 1720. In some embodiments, debug module 1721 utilizes a JTAG port to provide debug information to base 1702. In some embodiments, telemetry module 1722 generates telemetry information that can be used to monitor LCH performance. In some embodiments, telemetry module 1722 may provide information generated by a power management unit (PMU) and / or clock controller unit (CCU) located within the LCH. Virtual I2C module 1725 and virtual GPIO 1726 allow host processor 1760 to remotely control the GPIO and I2C ports on the LCH as if they were part of the SoC. Providing control of the LCH GPIO and I2C ports to the host processor via a low-pin USB connection allows for a reduced number of wires in the device hinge. Touch module 1727 processes touch sensor data provided by the touch sensor controller to host module 1720 and drives the display's touch controller.The touch module 1727 may, for example, determine the location on the display of one or more touches to the display and gesture information (e.g., information indicating the type of gesture, the location of the gesture on the display). The information determined by the touch module 1727 and other information generated by the host module 1720 can be communicated to the base 1702 via the USB connection 1728.

[0205] The audio module 1730 includes a voice-activated module 1731, an ultrasound module 1732, a noise reduction module 1733, a far-field preprocessing module 1734, an acoustic context awareness module 1735, a topic detection module 1736, and an audio core 1737. In some embodiments, any of the modules 1731 through 1737 may operate on or be implemented in one or more of the audio module components shown in FIGS. 2, 3, and 6 or other components disclosed herein. The voice-activated module 1731 implements the voice-activated functionality described above and, in some embodiments, may also implement the speaker ID functionality described above. The ultrasound module 1732 may support a low-power, low-frequency ultrasound channel by detecting information at near-ultrasonic / ultrasonic frequencies in audio sensor data. In some embodiments, the ultrasound module 1732 may drive one or more speakers disposed within the computing device to transmit information to other computing devices via ultrasound communication. The noise reduction module 1733 can implement one or more noise reduction algorithms on the audio sensor data. The far-field preprocessing module 1734 performs preprocessing on the audio sensor data to enhance audio signals received from distant sound sources. The acoustic context awareness module 1735 can implement algorithms or models to process the audio sensor data based on the determined audio context of the audio signal (e.g., detect unwanted background noise and filter the unwanted background noise from the audio signal).

[0206] The topic detection module 1736 determines one or more topics in the audio detected in the audio sensor data. In some embodiments, the topic detection module 1736 includes a natural language processing algorithm. In some embodiments, the topic detection module 1736 may determine topics discussed prior to a user's voice question and provide a response to the user based on the tracked topics. For example, the topic detection module 1736 may determine people, places, or other topics discussed in a period prior to the question (e.g., the past 30 seconds, the past minute, the past 5 minutes) and answer the question based on the topics. For example, if a user was talking with another person about Hawaii, the topic detection module 1736 may determine that the topic of the conversation is "Hawaii." If the user then asks the computing device, "What's the weather like there?" the computing device may provide a response providing the weather in Hawaii. The audio core 1737 is a real-time operating system and infrastructure upon which the audio processing algorithms implemented in the audio module 1730 are built. The audio module 1730 communicates with a sound capture module 1780 in the base 1702 via a SoundWire® connection 1738 .

[0207] The vision / imaging module 1740 includes a vision module 1741, an imaging module 1742, a vision core 1743, and a camera driver 1744. In some embodiments, any of the components 1741-1744 may operate on or be implemented in one or more of the vision / imaging module components shown in FIGS. 2, 3, and 7 or other components disclosed herein. The vision / imaging module 1740 communicates with an integrated sensor hub 1790 in the base 1702 via an I3C connection 1745. The vision module 1741 and the imaging module 1742 can implement one or more of the algorithms disclosed herein to operate on image sensor data provided by one or more cameras of the computing device. For example, the vision / imaging module can implement, separately or in concert, one or more of the face activation, face ID, head orientation detection, facial landmark tracking, and 3D mesh generation functions described herein. The vision core 1743 is a real-time operating system and infrastructure. It builds upon video and image processing algorithms implemented in the vision / imaging module 1740. The vision / imaging module 1740 interacts with one or more lid cameras via a camera driver 1744. In some circumstances, the camera driver 1744 can be a microdriver.

[0208] Components within the base include a host processor 1760, an audio capture module 1780, and an integrated sensor hub 1790. In some embodiments, these three components are integrated on an SoC. The audio capture module 1780 includes an LCH audio codec driver 1784. The integrated sensor hub 1790 may be an Intel® integrated sensor hub or any other sensor hub capable of processing sensor data from one or more sensors. The integrated sensor hub 1790 communicates with the LCH 1705 via an LCH driver 1798, which in some embodiments may be a microdriver. The integrated sensor hub 1790 further includes a biometric presence sensor 1794. The biometric presence sensor 1794 may include a sensor disposed within the base 1702 capable of generating sensor data used by the computing device to determine the presence of a user. The biometric presence sensor 1794 may include, for example, a pressure sensor, a fingerprint sensor, an infrared sensor, or a galvanic skin reflex sensor. In some embodiments, the built-in sensor hub 1790 can determine the presence of a user based on image sensor data received from the LCH and / or sensor data generated by a biometric presence sensor located within the lid (e.g., a lid-based fingerprint sensor, a lid-based infrared sensor).

[0209] The host processor 1760 includes a USB root complex 1761 that connects a touch driver 1762 and an LCH driver 1763 to the host module 1720. The host module 1720 communicates data determined from the image sensor data, such as the presence of one or more users in an image or video, facial landmark data, 3D mesh data, etc., to one or more applications 1766 on the host processor 1760 via a USB connection 1728 to the USB root complex 1761. The data is routed from the USB root complex 1761 through the LCH driver 1763, the camera sensor driver 1764, and the intelligent collaboration module 1765 to reach the one or more applications 1766.

[0210] The host processor 1760 further includes a platform framework module 1768 that enables power management at the platform level. For example, the platform framework module 1768 facilitates management of power to individual platform-level resources, such as the host processor 1760, SoC components (e.g., GPU, I / O controller), LCH, display, etc. The platform framework module 1768 also facilitates management of other system-level settings, such as clock rates for controlling the operating frequencies of various components and fan settings for improved cooling performance. The platform framework module 1768 communicates with an LCH audio stack 1767 to enable control of audio settings and with a graphics driver 1770 to enable control of graphics settings. The graphics driver 1770 provides video data to the timing controller 1706 via an eDP connection 1729. The graphics controller 1772 facilitates user control over the graphics settings of the computing device. For example, a user can configure graphics settings for performance, image quality, or battery life optimization. In some embodiments, the graphics controller is an Intel® Graphics Control Panel application instance.

[0211] Low-power always-on display

[0212] In some embodiments, a mobile computing device includes a foldable display that includes an "always-on" portion that is visible to the user and displays content when the device is closed and in a low-power mode. To reduce power consumption when the device is in the low-power mode, portions of the display that are not visible to the user when the device is closed are disabled. This may be achieved by placing one or more components of the display pipeline (e.g., a frame buffer, an image processing module, a row driver, and a column driver) in a low-power state. When the device is closed, the content displayed on the always-on portion of the display may be periodically updated with new images provided by the SoC display module, although in some embodiments, a second display module that is part of the low-power subsystem may provide images to the lid. This allows the SoC display module to remain in a low-power state. Extending an existing display to provide always-on display functionality by selectively disabling portions of the display may be less expensive than adding a second display and a second timing controller, thereby resulting in lower device costs.

[0213] An embodiment provides a mobile computing device having a base, a foldable display, and a lid rotatably attached to the base. The lid may include a timing controller including a frame buffer. The timing controller of the computing device may enable a first display portion of the foldable display when the computing device is in a full display mode, disable the first display portion when the computing device is in a partial display mode, and enable a second display portion of the foldable display. The first display portion may, for example, be visible when the computing device is in an open configuration and invisible when the computing device is in a closed configuration. In contrast, the second display portion may be visible when the computing device is in a closed configuration, and the computing device may be in a closed configuration when the computing device is in the partial display mode. The timing controller of the computing device may further place at least a portion of the frame buffer in a frame buffer low power state when the computing device is in the partial display mode. The frame buffer may store image data corresponding to an image displayed on the foldable display. Optionally, the timing controller may enable the second display portion when the mobile computing device is in the full display mode. In one example, the computing device may be in the partial display mode. In this mode, the timing controller may place one or more row drivers and / or one or more column drivers, which respectively drive the rows and columns of the first display portion, into a low power state.

[0214] 18A and 18B show top views of a mobile computing device in open and closed configurations, respectively. Here, a first exemplary foldable display includes a portion that can operate as an always-on display. The mobile computing device 1800 includes a base 1810 rotatably attached to a lid 1820 via a hinge 1812. The base 1810 includes a physical keyboard 1814 and a trackpad 1818. The lid 1820 includes a foldable display 1830 wrapped around the lid's upper edge 1835. The foldable display 1830 includes a first display portion 1840 disposed on a surface 1850 of the lid 1820 that is user-facing when the device 1800 is in the open configuration, and a second display portion 1860 on a surface 1870 of the lid. The second display portion 1860 can operate as an always-on portion of the display 1830 because it is visible when the device 1800 is in the closed configuration, as shown in FIG. 18B. With device 1800 closed, notification 1880 is displayed on second display portion 1860. Referring to Figure 1B, lid surfaces 1850 and 1870 correspond to the B-cover and A-cover of mobile computing device 1800, respectively.

[0215] FIG. 19A shows a top view of a mobile computing device in an open configuration having a second exemplary foldable display that includes a portion that can operate as an always-on display. The mobile computing device 1900 includes a base 1910 rotatably mounted to a lid 1920. The device 1900 includes a foldable display 1930 that covers a portion of a lid surface 1950 and a portion of a base surface 1960. The lid surface 1950 is the user-facing surface, and the base surface is the upward-facing surface when the device 1900 is in the open configuration. A virtual keyboard 1934 is displayed on a base display portion 1964 of the foldable display 1930. FIGS. 19B and 19C show a cross-sectional view and a top view, respectively, of the mobile computing device of FIG. 19A in a closed configuration. The length 1974 of the lid 1920 from the lid top edge 1976 to the hinge end portion 1978 is shorter than the length 1970 of the base 1910 extending from the base front edge 1984 to the hinge end portion 1978. This allows the display portion 1980 of the foldable display 1930 to remain visible when the device 1900 is closed. The display portion 1980 may operate as an always-on display. From a top view, the top surface 1990 of the lid 1920 appears adjacent to the always-on display portion 1980. Notification content 1994 may be displayed on the always-on display portion 1980 when the device 1900 is closed. Referring to FIG. 1B , the lid surface 1950, the base surface 1960, and the lid top surface 1990 correspond to the B-cover, C-cover, and A-cover of the mobile computing device 1900, respectively.

[0216] 20A through 20L show various views of a mobile computing device including a foldable display having a display portion that can be operated as an always-on display. 20A through 20C show A-cover, B-cover, and C-cover views, respectively, of a mobile computing device 2000 including a base 2012, a lid 2014, and a foldable display wrapped around a top edge 2018 of the lid 2014. The lid top edge 2018 is positioned away from a hinge end portion 2019 at which the mobile computing device 2000 is rotatably attached to the base 2012. The foldable display includes a first display portion 2020 and a second display portion 2022. The first display portion 2020 is user-facing when the device 2000 is in the open configuration, and the second display portion 2022 is world-facing when the device 2000 is in either the open or closed configuration. The second display portion 2022 may operate as an always-on display portion as it is visible when the device 2000 is in the closed configuration. A camera 2039 may be located on the A-cover of the device 2000, below the second display portion 2022.

[0217] 20D-20F show A-cover, B-cover, and C-cover views, respectively, of a mobile computing device 2030 including a base 2032, a lid 2034, and a foldable display wrapped around a top edge 2038 of the lid 2034. The lid top edge 2038 is positioned away from a hinge end portion 2039 where the mobile computing device 2030 is rotatably attached to the base 2032. The foldable display includes a first display portion 2040 and a second display portion 2042. The first display portion 2040 is user-facing when the device 2030 is in the open configuration, and the second display portion 2042 is world-facing when the device 2030 is in either the open or closed configuration. The second display portion 2042 may operate as an always-on display portion because it is viewable when the device 2030 is in the closed configuration. The camera 2044 may be located on the A-cover of the device 2030, beside the second display portion 2042.

[0218] 20G-20I show views of the A-cover, B-cover, and C-cover, respectively, of a mobile computing device 2050 including a base 2052, a lid 2054, and a foldable display wrapped around a side edge 2058 of the lid 2054. The foldable display includes a first display portion 2060 and a second display portion 2062. The first display portion 2060 is user-facing when the device 2050 is in the open configuration, and the second display portion 2062 is world-facing when the device 2050 is in either the open or closed configuration. The second display portion 2062 may operate as an always-on display portion because it is viewable when the device 2050 is in the closed configuration. A camera 2064 may be located on the A-cover of the device 2050, beside the second display portion 2062. In another embodiment, the foldable display may include two world-facing second display portions that wrap around both lateral edges of the lid, one or both of which may operate as always-on displays.

[0219] 20J-20L show a top view, a B-cover view, and a C-cover view, respectively, of a mobile computing device 2070 including a base 2072, a lid 2074, and a foldable display extending across the base 2072 and the lid 2074. The foldable display includes a first display portion 2076 disposed within the lid and a second display portion 2078 disposed within the base. The first display portion 2076 is user-facing, and the second display portion 2078 faces upward when the device 2070 is in the open configuration. A portion 2082 of the second display portion 2078 is visible when the device 2070 is in the closed configuration and may therefore operate as an always-on display portion. A camera 2084 may be located on the A-cover of the device 2070. 20J shows a top view of mobile computing device 2070 in a closed configuration, showing always-on display portion 2082 of second display portion 2078, which is visible when mobile computing device 2070 is closed due to lid length 2086 being shorter than base length 2088. Always-on display portion 2082 appears adjacent to A-cover 2090 of computing device 2070 when viewed from above.

[0220] 21 shows a block diagram of an exemplary timing controller and additional display pipeline components associated with a foldable display having a portion that can operate as an always-on display. The timing controller 2100 includes a video data receiver 2110, a frame buffer 2120, a processing stack 2122, and a display driver 2130. The timing controller 2100 is located within the lid of a mobile computing device along with an integrated foldable display panel 2140 and display panel control circuitry (e.g., row driver 2150 and column driver 2160). The video data receiver 2110 is part of the SoC 2186 and receives video data via connection 2182 from a display module 2164 located within the base of the mobile computing device. The frame processing stack 2122 includes a decoder 2139, an RGB module 2140, and a VDSC encoder 2126. The decoder 2139 decodes the encoded frames before they are processed by the RGB module 2140, and the encoder 2126 re-encodes the frames before they are processed by the color and gamma correction module 2128, which performs color and gamma correction on the frames. In some embodiments, the decoder 2139 and encoder 2126 decode and encode the frames according to the VESA® Display Compression Standard (VDSC).

[0221] In one or more embodiments, the timing controller 2100 represents an example implementation of a TCON 150, 400, 355. The built-in panel 2190 represents an example implementation of a built-in display panel 145, 280, 380, any of which may be used in an example implementation of a lid 120, 220, 301, 1820, which may be part of an example implementation of a mobile computing device (user device) 100, 200, 300. In one or more embodiments, the display module 2164 represents an example implementation of a display module 391, 341, which may be used in an example implementation of a SoC 140, 390, which may be disposed in a base 210, 315, 1810, which may be part of a mobile computing device (user device) 100, 200, 300.

[0222] In some embodiments, the frame processing stack 2122 includes one or more modules in addition to those shown in FIG. 21. The additional image processing modules may include any of the modules shown in FIG. 8 or any other image processing modules described or referenced herein. In some embodiments, the frame processing stack 2122 may include more or fewer modules than shown in FIG. 21. In some embodiments, the display module 2164 communicates with the TCON 2100 over an eDP connection, and the video data receiver 2110 includes an eDP main link receiver. In some embodiments, the display driver 2130 is a P2P transmission module.

[0223] The foldable display panel 2140 includes a first display portion 2190 and a second display portion 2194. The first display portion 2190 is viewable when the mobile computing device is in the open configuration and is not viewable when the mobile computing device is in the closed configuration (e.g., first display portion 1840 in FIG. 18A , and display portions 2020, 2040, 2060, and 2076 in FIGS. 20B, 20E, 20H, and 20K, respectively). The second display portion 2194 is viewable when the mobile computing device is in the closed configuration (e.g., second display portion 1860 in FIG. 18B , and display portions 2022, 2042, 2062, and 2082 in FIGS. 20A, 20D, 20G, and 20J, respectively). The second display portion 2194 can be operated as an always-on display portion when the mobile device is in the closed configuration. The first display portion 2190 and the second display portion 2194 are enabled based on the display mode of the mobile computing device. When the mobile computing device is in the open configuration, the device can be in a full display mode in which both the first and second display portions 2190 and 2194 are enabled. In some embodiments, there can be multiple full display modes. For example, in a first full display mode, the first display portion 2190 is enabled and the second display portion 2194 is disabled. In a second full display mode, the first and second display portions 2190 and 2194 are enabled and display content. When the mobile computing device is in a closed configuration, the device can be in a partial display mode in which the first display portion 2190 is disabled and does not display content, and the second display portion 2194 is enabled and operates as an always-on display.

[0224] In some embodiments, the display mode is based on the physical configuration of the mobile computing device and also on user-configurable settings. For example, a user may be provided with user-configurable display settings that allow the user to select whether the second display portion is enabled or disabled when the device is in the open configuration and whether the second display portion is enabled when the device is in the closed configuration. The timing controller 2100 receives display mode information 2196 from a component in the base of the mobile computing device, such as a lid controller hub or display module 2164.

[0225] To reduce device power consumption, one or more components of the device's display pipeline can be placed into a lower power state when the device is in partial display mode. The display pipeline includes components associated with generating, processing, and displaying images displayed on the display panel 2140. In the embodiment shown in FIG. 21 , the display pipeline components include a display module 2164, a video data receiver 2110, a frame buffer 2120, a frame processing stack 2122, a display driver 2130, a row driver 2150, and a column driver 2160. Placing the mobile computing device into a lower power state may include placing any of its constituent components into a lower power state. For example, the row driver can be placed into a row driver low power state, the column driver can be placed into a column driver low power state, the frame buffer can be placed into a frame buffer low power state, the display module can be placed into a display module low power state, etc. Placing a component or a portion of a component into a lower power state may include, for example, reducing its operating voltage, operating the component at a lower frequency, disabling the component, or disabling functionality of the component (such as disabling a portion of a memory array or disabling an output driver). Display pipeline components can be put into a low power state by timing controller control logic or by display mode information (e.g., display mode information 2196) being provided directly to individual display pipeline components to cause the modules to enter a low power state.

[0226] In some embodiments, when the device is in partial display mode, the row driver 2154 that drives the first display portion 2190 can be placed in a low power state, while the row driver 2158 that drives the second display portion 2194 remains enabled. In another embodiment, the column driver 2160 can include a first set of column drivers that drive the first display portion and a second set of column drivers that drive the always-on display portion, where the first set of column drivers are disabled and the second set of column drivers are enabled when the device is in partial display mode. In some embodiments, the portion of the timing controller display driver (e.g., display driver 2130) that drives the row or column drivers that drive the first display portion can be placed in a low power state when the device is in partial display mode.

[0227] In some embodiments, when the device is in partial display mode, a portion of the frame buffer 2120 is placed in a low power state. Placing a portion of the frame buffer 2120 in a low power state may include disabling a portion of a memory array in the frame buffer 2120 that stores images to be displayed on the first display portion 2190, disabling control circuitry that controls the portion of the memory array that stores images to be displayed on the first display portion 2190, or simply storing only image data corresponding to images to be displayed on the second display portion 2194 of the display.

[0228] In some embodiments, the TCON 2100 includes a first frame buffer that stores image data corresponding to images displayed on the first display portion 2190 and a second frame buffer that stores image data corresponding to images displayed on the second display portion 2194. The first and second frame buffers are sized based on the size of the first and second display portions, respectively, and are therefore smaller than the size of a single frame buffer (e.g., frame buffer 2120) required to store image data for images displayed across the display portions 2190 and 2194, respectively. The first frame buffer may be placed in a low power state when the device is in partial display mode.

[0229] In some embodiments, the TCON 2100 has a first frame buffer that stores image data corresponding to an image to be displayed across the first display portion 2190 and the second display portion 2194, and a second frame buffer that stores image data corresponding to an image to be displayed on the second display portion 2194. The second frame buffer is smaller than the first frame buffer. When the mobile computing device is in full display mode, the TCON 2100 utilizes the first frame buffer and the second frame buffer is placed in a low power state. In partial display mode, the TCON 2100 places the first frame buffer in a low power state and utilizes the second frame buffer.

[0230] In some embodiments, to reduce display pipeline power consumption, one or more image processing components of the frame processing stack 2122 may be bypassed when the mobile computing device is in partial display mode. For example, if content displayed on the second display portion 2194 when the device is in partial display mode appears black and white or does not otherwise benefit from processing by the modules in the frame processing stack, those modules may be bypassed when the device is in partial display mode. Bypassing components may include placing at least some of the bypassed components in a lower power state. As an example, some of the bypassed modules may remain enabled, allowing image data to pass through the modules while the remaining bypassed components may be disabled.

[0231] In some embodiments, display pipeline power consumption may be reduced by having components in the pipeline operate only on portions of image data corresponding to content to be displayed on the second display portion 2194. In components that operate in parallel on pixel data for pixels, portions of the component that perform operations on pixel data corresponding to the first display portion may be disabled. In modules that operate serially on pixel data for multiple pixels, the modules may be configured (e.g., by timing controller logic) to operate only on pixel data for pixels that correspond to the second display portion 2194.

[0232] In some embodiments, when the device is in partial display mode, the content displayed on the second display 2194 may include notification content updated at a rate slower than the typical refresh rate (e.g., 30 to 120 Hz) of the built-in panel when operating in full display mode. For example, if the notification content includes a clock showing hours and minutes, the display module 2164 may exit a low-power state once per minute to generate a new image to be displayed on the second display portion showing the current time. After generating the updated image, the display module 2164 may return to a low-power state. Examples of other types of information that may be included in notification content and displayed on the second display portion 2194 when the device is in partial display mode include information about upcoming meetings, remaining battery life, number of unread messages (e.g., email, text, social media), and context-based greetings (e.g., "Good morning!"). Many other types of information may be included in notification content. Depending on the information being displayed, the notification content may be updated at various intervals (e.g., once per minute for displaying a clock, once every 5 or 10 minutes to update unread message information, and once every 60 minutes to update meeting information). In some embodiments, a timing controller may signal the display module when to wake up and generate updated notification content. In another embodiment, the activation of the display module to generate updated information may be controlled by a component within the base.

[0233] In some embodiments, notification content may be provided to the TCON 2100 via an out-of-band connection 2184 by a display module 2198 that is part of a low-power subsystem 2199 located within the device base (e.g., 210, 315, 1810). The low-power subsystem 2199 may generate the notification content at a lower power level than the SoC 2186. The notification content may be provided as video data provided to the TCON 2100 via a connection 2184 that is separate from the connection 2182 that the display module 2164 uses to provide video data to the video data receiver 2110. Using the low-power subsystem 2199 to generate the notification content allows the SoC 2186 and display module 2164 to remain in a low-power state, reducing power consumption of the mobile computing device. In some embodiments, the out-of-band connection 2184 is a USB connection or a MIPI Display Serial Interface (MIPI DSI) connection.

[0234] FIG. 22 illustrates an exemplary method for operating a foldable display of a mobile computing device operable as an always-on display. Method 2200 may be performed, for example, by a laptop including a foldable display wrapped around the top of the laptop's lid. The foldable display has a user-facing portion when the lid is open and a world-facing portion that operates as an always-on display when the laptop lid is closed. At 2210, a first display portion of the foldable display of the mobile computing device is enabled when the mobile computing device is in full display mode. In an example, when the laptop transitions to full display mode in response to the lid being opened, the user-facing portion of the laptop is enabled and displays content. At 2220, when the mobile computing device is in partial display mode, the first display portion is disabled and a second display portion of the foldable display is enabled. The first display portion is visible when the mobile computing device is in the open configuration and is invisible when the mobile computing device is in the closed configuration. The second display portion is visible when the mobile computing device is in a closed configuration, and the mobile computing device is in a closed configuration when in a partial display mode. In an example, when the lid is closed, the user-facing portion of the display is disabled and the world-facing portion of the display is enabled. The second display portion operates as an always-on display. At 2230, when the mobile computing device is in the partial display mode, at least a portion of the frame buffer is placed in a frame buffer low power state. The frame buffer stores image data corresponding to images displayed on the foldable display. In an example, a portion of the memory array in the frame buffer used to store images for display on the foldable display is placed in a lower power state. That is, only the portion of the memory array in the frame buffer that stores images to be displayed on the always-on display is enabled.

[0235] Additional examples of low-power always-on embodiments described herein include the following non-limiting implementations: Each of the following non-limiting examples may stand alone or may be combined in any permutation or with any one or more of the other examples provided below or throughout this disclosure.

[0236] Example AA1 is a method that includes enabling a first display portion of a foldable display of a mobile computing device when the mobile computing device is in a full display mode; disabling the first display portion and enabling a second display portion of the foldable display when the mobile computing device is in a partial display mode, the first display portion being viewable when the mobile computing device is in an open configuration and not viewable when the mobile computing device is in a closed configuration, and the second display portion being viewable when the mobile computing device is in the closed configuration and is in the closed configuration when the mobile computing device is in the partial display mode; and placing at least a portion of a frame buffer in a frame buffer low power state when the mobile computing device is in the partial display mode, the frame buffer storing image data corresponding to an image displayed on the foldable display.

[0237] Example AA2 includes the method of Example AA1, further including enabling the second display portion when the mobile computing device is in the full display mode.

[0238] Example AA3 includes the method of Example AA1 or AA2, further including placing one or more row drivers that drive rows of the first display portion into a row driver low power state when the mobile computing device is in the partial display mode.

[0239] Example AA4 includes the method of any one of Examples AA1 to AA3, further including placing one or more column drivers that drive columns of the first display portion into a column driver low power state when the mobile computing device is in the partial display mode.

[0240] Example AA5 includes the method of any one of Examples AA1 to AA4, wherein the frame buffer is a first frame buffer, the mobile computing device further includes a second frame buffer smaller than the first frame buffer, and the method further includes using the second frame buffer to store image data corresponding to an image displayed on the second display portion when the mobile computing device is in the partial display mode.

[0241] Example AA6 includes the method of any one of Examples AA1-AA5, further including bypassing one or more image processing components in a display pipeline when the mobile computing device is in the partial display mode.

[0242] Example AA7 includes the method of any one of Examples AA1 to AA6, further including configuring one or more image processing components in a display pipeline to not process image data corresponding to a portion of an image displayed on the first display portion when the mobile computing device is in the partial display mode.

[0243] Example AA8 includes the method of any one of Examples AA1 to AA7, further including: displaying a first image on the first display portion when the mobile computing device is in the full display mode, the first image corresponding to video data provided by a first display module; and displaying a second image on the second display portion when the mobile computing device is in the partial display mode, the second image corresponding to video data provided by a second display module.

[0244] Example AA9 includes the method of Example AA8, further including placing the first display module in a first display module low power state when the mobile computing device is in the partial display mode, and periodically waking up the second display module from the second display module low power state to generate new images to be displayed on the second display portion.

[0245] Example AA10 includes the method of any one of Examples AA1 to AA9, wherein the mobile computing device includes a lid rotatably attached to a base, the foldable display covering at least a portion of a first surface of the lid and at least a portion of a second surface of the lid, and the first surface of the lid is a user-facing surface and the second surface of the lid is a world-facing surface when the mobile computing device is in an open configuration.

[0246] Example AA11 includes the method of Example 10, wherein the lid includes an upper edge spaced apart from a hinge end portion where the lid is rotatably attached to the base, and the foldable display is wrapped around the upper edge of the lid.

[0247] Example AA12 includes the method of example 10, wherein the lid includes side edges and the foldable display is wrapped around at least one of the side edges.

[0248] Example AA13 includes the method of any one of Examples AA1 to AA9, wherein the mobile computing device includes a lid rotatably attached to a base, the foldable display covering at least a portion of a first surface of the lid and at least a portion of a first surface of the base, the first surface of the lid and the first surface of the base facing each other when the mobile computing device is in a closed configuration, the lid having a lid length extending from a hinge end portion where the lid is rotatably attached to the base to a lid top edge spaced apart from the hinge end portion, and the base having a base length extending from a hinge end portion where the base is rotatably attached to the lid to a base front edge spaced apart from the hinge end portion, the lid length being shorter than the base length, and the second display portion being disposed on the first surface of the base.

[0249] Example AA14 includes the method of any one of Examples AA1-AA13, wherein placing at least a portion of the frame buffer in a frame buffer low power state includes disabling the frame buffer.

[0250] Example AA15 includes the method of any one of Examples AA1-AA13, wherein placing at least a portion of the frame buffer in a frame buffer low power state includes disabling a portion of a memory array of the frame buffer.

[0251] Example AA16 includes the method of any one of Examples AA1-AA13, wherein placing at least a portion of the frame buffer in a frame buffer low power state includes reducing an operating voltage of the frame buffer.

[0252] Example AB1 includes a mobile computing device comprising a base, a foldable display, and a lid rotatably attached to the base, the lid including a timing controller including a frame buffer, the timing controller enabling a first display portion of the foldable display when the mobile computing device is in a full display mode, disabling the first display portion when the mobile computing device is in a partial display mode, and enabling a second display portion of the foldable display, the first display portion being viewable when the mobile computing device is in an open configuration and not viewable when the mobile computing device is in a closed configuration, the second display portion being viewable when the mobile computing device is in the closed configuration, the mobile computing device being in the closed configuration when in the partial display mode, and placing at least a portion of a frame buffer in a frame buffer low power state when the mobile computing device is in the partial display mode, and the frame buffer storing image data corresponding to an image displayed on the foldable display.

[0253] Example AB2 includes the mobile computing device of Example AB1, wherein the timing controller further enables the second display portion when the mobile computing device is in the full display mode.

[0254] Example AB3 includes the mobile computing device of Examples AB1 or AB2, wherein the timing controller further places one or more row drivers that drive rows of the first display portion into a row driver low power state when the mobile computing device is in the partial display mode.

[0255] Example AB4 includes the mobile computing device of any one of Examples AB1 to AB3, wherein the timing controller further places one or more column drivers that drive columns of the first display portion into a column driver low power state when the mobile computing device is in the partial display mode.

[0256] Example AB5 includes the mobile computing device of any one of Examples AB1 to AB4, wherein the frame buffer is a first frame buffer, the timing controller further includes a second frame buffer smaller than the first frame buffer, and the timing controller further uses the second frame buffer to store image data corresponding to an image displayed on the second display portion when the mobile computing device is in the partial display mode.

[0257] Example AB6 includes the mobile computing device of any one of Examples AB1 to AB5, wherein the timing controller further includes one or more image processing components, and the timing controller further bypasses one or more image processing components in a display pipeline when the mobile computing device is in the partial display mode.

[0258] Example AB7 includes the mobile computing device of any one of Examples AB1 to AB6, wherein the timing controller further includes one or more image processing components, and wherein the timing controller further configures one or more image processing components in a display pipeline to not process image data corresponding to a portion of an image displayed on the first display portion when the mobile computing device is in the partial display mode.

[0259] Example AB8 includes the mobile computing device of any one of Examples AB1 to AB7, wherein the timing controller further displays a first image on the first display portion when the mobile computing device is in the full display mode and a second image on the second display portion when the mobile computing device is in the partial display mode, and the mobile computing device further includes a first display module that provides the first image to the timing controller and a second display module that provides the second image to the timing controller.

[0260] Example AB9 includes the mobile computing device of any one of Examples AB1 to AB8, wherein the lid includes a top edge positioned away from a hinge end portion where the lid is rotatably attached to the base, and the foldable display is wrapped around the top edge of the lid.

[0261] Example AB10 includes the mobile computing device of any one of Examples AB1-AB8, wherein the lid includes one or more side edges and the foldable display wraps around at least one of the side edges.

[0262] Example AB11 includes the mobile computing device of any one of Examples AB1 to AB8, wherein the foldable display covers at least a portion of a first surface of the lid and at least a portion of a first surface of the base, the first surface of the lid and the first surface of the base facing each other when the mobile computing device is in a closed configuration, the lid has a lid length extending from a hinge end portion where the lid is rotatably attached to the base to a lid top edge spaced apart from the hinge end portion, and the base has a base length extending from a hinge end portion where the base is rotatably attached to the lid to a base front edge spaced apart from the hinge end portion, the lid length being shorter than the base length, and the second display portion is disposed on the first surface of the base.

[0263] Example AB12 includes the mobile computing device of any one of Examples AB1-AB11, wherein placing at least a portion of the frame buffer in a frame buffer low power state includes disabling the frame buffer.

[0264] Example AB13 includes the mobile computing device of any one of Examples AB1-AB11, wherein placing at least a portion of the frame buffer in a frame buffer low power state includes disabling a portion of a memory array of the frame buffer.

[0265] Example AB14 includes the mobile computing device of any one of Examples AB1-AB11, wherein placing at least a portion of the frame buffer in a frame buffer low power state includes reducing an operating voltage of the frame buffer.

[0266] Example AC1 includes one or more non-transitory computer-readable storage media storing computer-executable instructions for causing a mobile computing device to perform the method of any one of Examples AA described herein above.

[0267] Example AD1 includes a system including one or more means for implementing one or more methods of Example AA described herein above.

[0268] Display management for a multiple display computing system

[0269] Display power, which may include backlight and panel electronics, consumes a significant amount of power in systems today. Displays in computing systems may use between forty and sixty percent (40-60%) of the total system power. SoC and system power increases significantly when multiple external displays are present. For example, connecting two 4K monitors may incur significantly greater power costs to render the additional high-resolution displays.

[0270] Many current computing devices switch between power modes to conserve power, extend battery life, and / or prevent burn-in of certain display screens. However, energy efficiency techniques implemented in computing systems can adversely affect the user experience if the techniques impair the responsiveness or performance of the system.

[0271] Display management solutions for display power and energy conservation have involved user presence detection from a single display system, where the display panel is dimmed or turned off if no user is detected. For example, the backlight may be dimmed to reduce brightness, or the backlight may be turned off completely. For example, software-based solutions may determine whether a user's face is facing a single display and dim or turn off that display accordingly. However, software-based solutions use large amounts of power, e.g., amounts in the watt range. Furthermore, because software-based solutions can only handle built-in displays, they must be more careful in determining when to turn off the display. Furthermore, these single-display solutions only have an accuracy of the single display's field of view.

[0272] Single-display user presence solutions cannot adequately manage battery life, responsiveness gains, privacy, and security features together or efficiently. In single-display systems, only one input is obtained from one display. This limits the amount of data available to effectively manage multiple display scenarios. Depending on where a user presence-enabled system is located, the system may not effectively receive accurate information about when a user is approaching the computing system (e.g., workstation, desk) and, if present, where the user is looking. Furthermore, if a user closes a laptop with a single-display user presence solution, it is no longer possible to manage the external monitor display to save power when the user is not looking at the external monitor. When the laptop is closed, the external monitor cannot respond to any user presence behavior.

[0273] External high-resolution displays (e.g., 4K displays) are increasingly being used in extended display scenarios. However, such displays significantly increase display and system power and energy. These solutions do not have a way to handle user presence in a way that conserves power and energy. This can affect compliance with certain state and / or federal programs, such as the California Energy Policy and Energy Star. High resolution can also affect performance by fifty percent (50%) or more. Therefore, the user experience can be further impaired.

[0274] In another example, authentication software (e.g., Microsoft® Windows Hello® authentication software) allows users to place a clip-on camera on each monitor and perform facial authentication on the monitor that the user's attention is directed to. The solution is available for authentication (e.g., via facial recognition) and user login only when the user is at the appropriate distance and facing the display panel. These authentication solutions do not address display power and brightness management based on the user's presence.

[0275] Recent developments include low-power components that provide human presence and attention sensing, which provides privacy and security through different operating modes based on the user's presence and attention. While these advances are important for laptop or other single device implementations, they do not solve the problems associated with multiple display modules in today's typical computing environments.

[0276] In today's computing environment, it is common for users to dock their laptops to their workstations, whether in the office or at home. Studies have shown that enterprise users work in a docked scenario approximately eighty percent (80%) of the time. A common scenario for users is to dock their laptop and work primarily in a docking station with an external monitor, where the external monitor may be larger than the main display that the user engages with most of the time while docked.

[0277] An embodiment provides a computing system including a first display device including a first display panel, a first camera, and first circuitry for generating first image metadata based on first image sensor data captured by the first camera, and a second display device including a second display panel, a second camera, and second circuitry for generating second image metadata based on second image sensor data captured by the second camera. The computing system may further include a processor operatively coupled to the first and second display devices. The processor is configured to select an operating mode for the display device based on the image metadata. For example, the processor may select a first operating mode for the first display device based on the first image metadata and a second operating mode for the second display device based on the second image metadata.

[0278] The first and second image metadata may indicate, for example, whether a user is engaged with the first display device or the second display device, respectively, and an operational mode for the respective display device may be selected based on this indication. For example, detecting a user's engagement or disengagement with the display device may be based on facial recognition. For example, first circuitry may detect a user's face in the first image sensor data, determine a user is present in a first field of view of the first camera based on detecting the user's face in the first image sensor data, determine a first orientation of the user's face based on the first image sensor data, and determine whether the user is engaged or disengaged with the first display device based at least in part on the first orientation of the user's face. Similar operations may be performed by second circuitry to determine whether a user is engaged or disengaged with a second display device. If a user is not engaged with a particular one of the display devices, an operating mode for that one display device may be selected such that the brightness of the backlight for the display panel of that one display device gradually decreases over a period of time until a user event occurs or the backlight is reduced to a predetermined minimum level of brightness (or even until the backlight is turned off).

[0279] The presence of the user may be used to unlock the computing system and / or authenticate the user. For example, the processor of the computing system may further determine that access to the computing system is locked, determine that an authentication mechanism is not currently running on the second display device, trigger authentication of the user via the display device with which the user is engaged by the authentication mechanism, and keep the other display devices turned off until the user is authenticated.

[0280] 23A-23B illustrate possible user attention scenarios in a computing system in which a laptop is connected to one additional external monitor. FIG. 23A includes a user 2302, a laptop 2312, and an external monitor 2339 communicatively coupled to the laptop 2312. The laptop 2312 includes a primary display panel 2316 on the laptop's lid 2314 and a user-facing camera 2310 coupled to the lid. The external monitor 2339 includes a secondary display panel 2326. Only the laptop 2312 has user presence and attention detection enabled. That is, a user presence policy based on user disengagement or absence from the single built-in primary display panel 2316 may cause the display to be dimmed and / or turned off completely. Because the laptop 2312 is the only system for which user presence and attention detection is enabled, only that system can dim or turn off its built-in display panel based on the user's attention, i.e., based on its display viewpoint. Thus, display management can be applied only to the primary display panel in a laptop, or it can be applied equally to all screens.

[0281] In FIG. 23A , when the user's attention is directed to the primary display panel 2316, as shown at 2304A, the laptop 2312 can detect the user's face and presence and initiate (or maintain) an appropriate operating mode that allows use of the laptop and its primary display panel 2316 while the external monitor 2339 also remains on and uses power and energy. When the user's attention is directed to the secondary display panel 2326 of the external monitor 2339, as shown at 2304B in FIG. 23B, the laptop 2312 can apply display management to its primary display panel 2316. If display management is applied equally to both monitors, any changes (e.g., dimming, sleep mode) applied to the primary display panel 2316 will also be applied to the secondary display panel 2326, even if the user is engaged with the secondary display panel 2326.

[0282] Multiple screen and docking configurations are possible. In another example, FIGS. 24A through 24C illustrate possible user attention scenarios in a computing system in which a laptop is connected to two additional external monitors. FIG. 24A includes a user 2402, a laptop 2412, and first and second external monitors 2439 and 2434 communicatively coupled to the laptop 2412. The laptop 2412 includes a primary display panel 2416 in the laptop's lid 2414 and a camera 2410 including an image sensor coupled to the primary display panel. The external monitor 2439 includes a secondary display panel 2426, and the external monitor 2434 also includes a secondary display panel 2436. User presence and attention detection is enabled only for the laptop 2412. The user may be engaged and focused on any of the three monitors. Because the laptop 2412 is the only system where user presence and attention detection is enabled, only that system can dim or turn off its built-in display panel based on the user's attention, i.e., based on the display's viewpoint. If the user remains engaged only with the laptop display, the other two monitors remain powered because they do not provide any user presence-based input to the policy. Thus, display management can be applied only to the display panel on the laptop, or equally to all three screens.

[0283] In FIG. 24A , when a user's attention is directed to the laptop 2412's primary display panel 2416, as shown at 2404A, the laptop 2412 may detect the user's face and presence and initiate (or maintain) an appropriate operating mode that allows use of the system while both external monitors 2439 and 2434 remain on and use power and energy. When the user's attention is directed to the central screen, as shown at 2404B in FIG. 24B, the laptop 2412 may apply display management to its primary display panel 2416 while the secondary display panel 2436 remains on and uses power and energy. This may also result in a degraded user experience because only one display can handle the dimming policy while the other external monitors remain on. Similarly, when the user's attention is directed to the central screen, as shown at 2404C in FIG. 24C, the laptop 2412 may apply display management to its primary display panel 2416 while the central display panel 2426 remains on and uses power and energy.

[0284] Display power management for multiple displays and docking scenarios based on user presence and attention, as disclosed herein, can solve these problems. The embodiments described herein extend a single display policy to handle multiple displays to seamlessly manage each individual display panel according to an overall, comprehensive policy. The embodiments disclosed herein enable a primary display device (e.g., a lid with an integrated display panel in a mobile computing device, a monitor connected to a desktop) in a multiple-display computing system to perform user presence and attention detection and detection-based individual display management for one or more secondary displays (e.g., an external monitor) in the multiple-display computing system. Thus, any display panel of a display device can be dimmed and / or turned off depending on user behavior. Policies can be implemented to consistently manage multiple display panels (e.g., a primary display panel of a computing device and one or more other display panels in an external monitor operatively coupled to the computing device). Examples may include policies to accommodate waking the system upon detection of a face from any of multiple display devices, adaptively dimming a display panel (e.g., by reducing the backlight) based on user attention to a particular display device, preventing the display panel from locking when a user is detected (even if the user is not interacting with the computing system), and locking the computing system when use is no longer detected from any of the display devices.

[0285] In one or more embodiments, a lid controller hub, such as LCH 155, 260, 305, 954, 1705, 1860, 2830A, or 2830B, or at least certain features thereof, can be used to implement user presence and attention-based display management for multiple displays and docking scenarios. The embodiments disclosed herein can intelligently handle input received from all displays (e.g., via their respective LCHs), seamlessly dimming or turning off each display based on user presence and attention data. In one or more embodiments, the system can be triggered to wake up even before a user sits down. Furthermore, the use of cameras on each display can expand the area in which a user can be detected. The system can also be triggered to wake up even before any use of the system if the user is already logged on. The embodiments herein further provide for preventing the system from dimming the display panel or setting the system to a low-power state based on the user's presence on any of the multiple displays, even if the user is not actively interacting with the system. Thus, power and energy savings and improved user experience are possible when two or more (or all) displays are capable of providing user presence and attention detection.

[0286] Reference is made to FIG. 25, which is a simplified block diagram illustrating possible details of a multiple display system 2500 capable of implementing an embodiment of user presence-based display management to apply an overall policy for handling multiple display devices. In one or more embodiments, each display device is configured to provide its own user presence and attention input. The multiple display system 2500 may include a computing device 2505, such as a laptop or any other mobile computing device connected to one or more additional display devices. In at least one example, the computing device 2505 (and its components) represents exemplary implementations of other computing devices (and their components) disclosed herein (e.g., 100, 122, 200, 300, 900, 1700-2300, 2800A, 2800B). The additional display devices in the exemplary system 2500 are implemented in a first external monitor 2520 and a second external monitor 2530. In one possible implementation, the external monitors 2520 and 2530 may be docked to the computing device 2505 via a docking station 2550, although it will be apparent that other implementations are possible. For example, the external monitors 2520 and 2530 may be connected directly to the computing device 2505 (e.g., via an HDMI port on the computing device), or may be connected to the computing device 2505 using any other suitable means.

[0287] The computing device 2505 may comprise a base 2506 and a lid 2510. A processing element 2508, such as a system-on-chip (SoC) or central processing unit (CPU), may be disposed within the base 2506. A display panel 2512 and a user-facing camera 2514 may be disposed within the lid 2510. The external monitors 2520 and 2530 may also comprise respective display panels 2522 and 2532 and respective cameras 2539 and 2534.

[0288] Each display device, including the primary display device (e.g., 2512) of the computing device and one or more additional external (or secondary) display device(s) connected to the computing device (e.g., 2520, 2530), may be configured with its own vision-based analyzer integrated circuit (IC), which may include some or all of the features of one or more lid controller hubs (e.g., LCH155, 260, 305, 954, 1705, 1860, 2830A, 2830B) described herein. For example, vision-based analyzer IC 2540A is located within lid 2510 of computing device 2505. Vision-based analyzer IC 2540B is located within first external monitor 2520. Vision-based analyzer IC 2540C is located external to second external monitor 2530.

[0289] Each of the vision-based analyzer ICs 2540A, 2540B, and 2540C includes circuitry for executing machine learning algorithms capable of performing user presence detection, face detection, and face orientation detection in the field of view from their respective cameras 2514, 2539, and 2534 based on image sensor data generated by the associated camera. The vision-based analyzer ICs 2540A, 2540B, and 2540C may generate respective image metadata based on the respective image sensor data generated by their respective cameras 2514, 2539, and 2534. The image metadata generated by a given vision-based analyzer IC associated with a given camera may indicate whether a user is present within the field of view of the given camera, engaged with a display panel associated with the given camera, disengaged from a display panel associated with the given camera, and / or not present within the field of view of the given camera.

[0290] Image metadata generated by the vision-based analyzer ICs 2540A, 2540B, and 2540C may be provided to the SoC 2508, which can selectively control the settings of certain performance parameters that affect the power and / or performance levels of the multiple display devices 2510, 2520, and 2530, as well as the computing device 2505 to which the multiple display devices are docked or otherwise connected. The settings for the display devices may be controlled according to defined policies based on user presence, user absence, user engagement, and / or user inactivity for each display device. Because dedicated cameras and vision-based analyzer ICs are implemented for each display device (e.g., the primary display device of the computing device and the secondary display devices of the external monitor(s)), user presence-based policies may be more precisely managed. Examples of user presence-based policies may include, but are not necessarily limited to, face and head presence and orientation detection to wake the system when a user is detected, adaptively dim the display panel if the user is not paying attention, not lock the system when the user is present, and lock the system when the user is absent. Each vision-based analyzer IC coupled to a camera for a display panel can provide input from its unique field of view indicating a user's proximity to that display device. This can improve accuracy and speed with respect to system wake-up and triggering facial recognition. Display devices can also be more precisely managed, individually and collectively (e.g., dimming or turning off the backlight of a display panel when the user is disengaged or absent). Importantly, power and performance can be improved if one or more display panels can be dimmed or turned off by adjusting the backlight, and / or if the display panel refresh rate and rendering can be reduced for one or more display panels.At least in the context of embodiments of display management for a multiple display computing system, if a backlight is not provided for the display panel, the display panel can be dimmed by adjusting the backlight of the display panel to a lower brightness and turning it off. However, if no backlight is provided for the display panel such that the display panel is effectively turned off, cameras 2514, 2539, and 2534 can be configured for “always on” use such that each camera can continue to capture images and generate image sensor data, and vision-based analyzer ICs 2540A, 2540B, and 2540C can continue to evaluate the image sensor data.

[0291] In one or more embodiments, a dedicated vision-based analyzer IC may be provided for each display device in a multiple display system (e.g., 2500). Embodiments allow the vision-based analyzer IC to be integrated into the display device or configured as an add-on device to the display device. For example, in some cases, a dedicated vision-based analyzer IC may be built into the lid of a computing device and into an external monitor. However, in other scenarios, the vision-based analyzer IC may be configured as a dongle or other small device that can be connected to and shared with any display device in a computing system. For example, in system 2500, the lid 2510 of computing device 2505 and the first external monitor 2520 may have integrated vision-based analyzer ICs 2540A and 2540B, respectively. However, the second external monitor 2530 may have an add-on vision-based analysis IC 2540C. In one example, the add-on vision-based analyzer IC2540C may be integrated with its own camera and attached to the housing of the second external monitor 2530 such that the external camera is positioned to capture images of an appropriate field of view of the second external monitor 2530. In another example, the second external monitor 2530 may be configured with a port that allows the add-on vision-based analyzer IC2540C to be connected to a camera built into the second external monitor, such as camera 2534.

[0292] Figure 26 is a top plan view illustrating possible fields of view of cameras in a multiple display system. In this exemplary scenario, a top plan view of a first display device 2610, a second display device 2620, and a third display device 2630 is shown. At least one of the display devices may be configured as part of a computing device (e.g., the lid 2510 of the computing device 2505). The other two display devices may be implemented as external monitors (e.g., 2520, 2530) or other devices including display panels operatively coupled to the computing device. The first display device 2610 includes a first display panel 2612 and a first user-facing camera 2614. The second display device 2620 includes a second display panel 2622 and a second camera 2639. The third display device 2630 includes a third display panel 2632 and a third camera 2634. Each camera may be configured for always-on use and may point toward its associated display panel. Cameras 2614, 2639, and 2634 may each be coupled to the respective display panel 2612, 2622, or 2632, attached externally to the respective display device 2610, 2620, or 2630, and / or positioned on the respective display device or other portion of the associated computing device, within a bezel area surrounding the respective display panel 2612, 2622, or 2632.

[0293] In at least one embodiment, each camera is associated with an imaging field of view (FoV) and may include (or be operatively coupled to) a suitable image sensor for detecting movement and / or light, which may indicate the presence of a user within the camera's imaging FoV. For example, a first camera 2614 is associated with a first imaging FoV 2618 that spans between dashed lines 2617 and 2619. A second camera 2639 is associated with a second imaging FoV 2628 that spans between dashed lines 2627 and 2629. A third camera 2634 is associated with a third imaging FoV 2638 that spans between dashed lines 2637 and 2639. In the example of FIG. 26 , the first camera 2614 generates image sensor data representing an image of the area around the first display device 2610 that is viewable within the imaging FoV 2618. The second camera 2639 generates image sensor data representing images of the area around the second display device 2620 viewable within the second imaging FoV 2628. The third camera 2634 generates image sensor data representing images of the area around the third display device 2630 viewable within the third imaging FoV 2638.

[0294] As shown in FIG. 26 , depending on the arrangement of multiple display systems, the imaging FoVs may overlap. In this example, imaging FoVs 2618, 2628, and 2638 associated with display devices 2610, 2620, and 2630 may overlap to create an extended area larger than the FoV of a single camera. Here, a user's presence may be detected by one or more of cameras 2614, 2639, and 2634. Note that while the multiple devices shown in FIG. 26 are arranged along a generally straight line, numerous other configurations are possible. For example, any of the display devices may be angled or rotated based on the user's particular desires and preferences. Different arrangements (e.g., angle, tilt, position) may result in different fields of view and different overlapping portions of the fields of view of the multiple cameras.

[0295] 27A through 27C are top plan views illustrating possible user head / face orientations relative to a display device. In this example scenario, a display device 2710 is shown including a display panel 2712 and a user-facing camera 2714. Additionally, a user 2702 is shown positioned facing the display panel 2712 so that the user can view the display panel. In this example, the user 2702 is within the field of view (FoV) of the camera 2714. Thus, image sensor data generated by the camera 2714 may include data representing the user 2702. For illustrative purposes, face orientations 2704A through 2704D show example gaze directions of the user 2702 relative to the display panel 2712. In one or more examples, the face orientation or gaze direction extends perpendicularly from the user's face (e.g., the nose, the center point between the eyes, the center point of the lips, etc.). Display device 2710 represents an exemplary lid for a computing device (e.g., lid 2510 of computing device 2505) or an external monitor (e.g., 2520, 2530) or other device having a display panel operatively coupled to the computing device.

[0296] In one or more embodiments, image sensor data generated by camera 2714 may be analyzed by a vision-based analyzer IC (e.g., 2540A, 2540B, 2540C) to determine whether a user is present within the camera's field of view (FoV), as described with reference to FIG. 26 . Furthermore, upon determining that a user is present within the camera's 2714 FoV, the vision-based analyzer IC may determine the facial orientation of user 2702 relative to display panel 2712. Using machine learning algorithm(s), the vision-based analyzer IC may be trained to recognize human facial features and, based on the recognition of the facial features, determine the user's facial orientation and the user's head / face position within the associated camera's field of view (FoV). In at least one embodiment, the user's intent (e.g., engaged, disengaged) can be inferred from the identified facial orientation and parameters defining the maximum rotation angle of the user's face and / or gaze direction.

[0297] In one or more embodiments, determining whether the user 2702 is engaged or disengaged with the display panel 2712 may be accomplished by determining whether the rotation angle of the user's face relative to the display panel 2712 is within a first-level region between the user 2702 and the display panel 2712. In one example, the rotation angle of the user's face relative to the display panel 2712 may be calculated as an angle defined between a direct display path (e.g., 2703) and the user's facial orientation or gaze direction (e.g., 2704A to 2704D). The direct display path (e.g., 2703) may be defined as a generally direct path from the camera 2714 to the user 2702 because the camera is typically close to the display panel. The user's head orientation (e.g., 2704A to 2704D) may correspond to a vertical direction extending from the user's face (e.g., the center of the user's face, such as the nose, the center point between the eyes, the center point of the lips, etc.). The first-level region may be defined as the region between the first-level maximum rotation angle of the user's face to the left of the direct display path and the first-level maximum rotation angle of the user's face to the right of the direct display path. In one or more implementations, the first-level maximum rotation angle to the left may be the same as the first-level maximum rotation angle to the right, for example, 45 degrees (45°). The first-level maximum rotation angle may be user and / or system configurable in at least one embodiment.

[0298] 27A through 27C , the direct display path 2703 and face orientations 2704A through 2704D may indicate possible scenarios of a user's face orientation relative to the display panel 2712. From this, the engagement or disengagement of the user 2702 with the display panel 2712 can be inferred. Generally, when the user 2702 is facing the display panel 2712 (whether the user is directly in front of the display panel 2712 or at some other position within the imaging FoV of the camera 2714), the vision-based analyzer IC may infer that the user is interested in (e.g., engaged with) the content rendered on the display panel and does not want the operation of the computing device or the brightness of the display panel 2712 to be adversely affected. In another scenario, if the user 2702 turns to either side relative to the direct display path from the user to the display panel 2712, but does not face completely away from the display panel 2712, the vision-based analyzer IC may infer that the user is currently not interested in (e.g., present but unengaged with) the content rendered on the display panel 2712, but is present and engaged in the operation of the computing device. For example, the user 2702 may be sitting at their desk within the imaging FoV associated with the camera 2714, but may have turned aside to look at a second display panel of a docked external monitor. Thus, while the user may be unengaged with the display panel 2712, the user may be interested in the content rendered on the second display panel of the external monitor and may therefore desire continued operation of the computing device. In this case, the display panel 2712 may be gradually dimmed over time (e.g., the backlight brightness is reduced), assuming the user does not return their attention to the display panel 2712. However, a vision-based analyzer IC associated with the external monitor may determine that the user 2702 is present and engaged with a second display panel on the external monitor.Thus, in this scenario, operation of the computing device and the second display panel of the external monitor are not adversely affected (e.g., dimming the display panel, transitioning the SoC to a lower power state). In general, the performance of the computing device, the display panel 2712, and the second display panel of the external monitor may be adjusted based on the inferred user 2702 engagement, thereby increasing performance to improve the user experience (e.g., when the user is engaged), or decreasing performance to conserve power (e.g., when the user is either present, disengaged, or absent).

[0299] In at least one embodiment, image metadata indicative of a user's presence, absence, engagement, or disengagement with respect to the display panel 2712 may be generated based on identifying the user's presence / absence and the facial orientation of the detected human face. This image metadata may be provided to a processing element of a computing device (e.g., SoC 2508 of computing device 2505) communicatively coupled to the vision-based analyzer IC of the display device 2710. Based on the indication of whether the user is engaged, disengaged, present, and / or absent and on one or more predefined rules, the computing device may control settings for certain performance parameters that affect the power and / or performance levels of the display device 2710.

[0300] 27A , the vision-based analyzer IC may determine that the user 2702 is present and engaged with the display panel 2712 based on determining that the user's facial orientation 2704A is within the first level region 2718. In one implementation, the first level region 2718 may extend forty-five degrees (45°) to the left and forty-five degrees (45°) to the right of the direct display path 2703. In this example, the first level region 2718 may span approximately ninety degrees (90°) between the dashed line 2717 indicating the first-level maximum rotation angle to the right and the dashed line 2719 indicating the first-level maximum rotation angle to the left. The user's facial orientation 2704A forms a rotation angle with the direct display path 2703 that is less than the first-level maximum rotation angle to the right indicated by the dashed line 2717. Thus, the user's facial orientation 2704A is within the first level region 2718. Thus, the vision-based analyzer IC may infer that the user 2702 is engaged with the display panel 2712 (e.g., interested in the content rendered on the display panel 2712) even if the user 2702 is not currently interacting with a user interface (e.g., keyboard, touchpad, mouse, touchscreen).

[0301] 27B , the vision-based analyzer IC may determine that user 2702 is present but not engaging display panel 2712 based on determining that user's facial orientation 2704B or 2704C is within second level region 2728B or 2728C. In one implementation, second level regions 2728B and 2728C reside outside first level region 2718 but do not extend beyond a second level maximum angle of rotation to the left or right from the direct display path. In one example, the second level maximum angle of rotation from the direct display path 2703 is ninety degrees (90°) to the left, as shown by dashed line 2729, and ninety degrees (90°) to the right, as shown by dashed line 2727. The left and right second level regions may each span approximately forty-five degrees (45°) and be defined between dashed lines 2719 and 2729 and between dashed lines 2717 and 2727. In one scenario of FIG. 27B , user's facial orientation 2704C forms a rotation angle with direct display path 2703 between a first-level maximum rotation angle to the right, indicated by dashed line 2717, and a second-level maximum rotation angle to the right, indicated by dashed line 2727. Thus, user's facial orientation 2704C is within second-level region 2728A. Thus, in this scenario, the vision-based analyzer IC may infer that user 2702 is not engaged with display panel 2712, but may be present and engaged with other display devices. In another scenario of FIG. 27B , user's facial orientation 2704B forms a rotation angle with direct display path 2703 between a first-level maximum rotation angle to the left, indicated by dashed line 2719, and a second-level maximum rotation angle to the left, indicated by dashed line 2729. Thus, user's facial orientation 2704B is within second-level region 2728B. Thus, in this scenario, the vision-based analyzer IC may infer that the user 2702 is not engaged with the display panel 2712, but may be present and engaged with other display devices.

[0302] 27C , the vision-based analyzer IC may determine that the user 2702 is not present, or that the user is present but the user's face is not detectable relative to the display panel 2712, based on determining that the user's facial orientation 2704D is within the third-level region 2738. In one implementation, the third-level region 2738 may extend beyond the second-level maximum rotation angle to the left or right from the direct display path. In one example, the third-level region 2738 may span approximately one hundred and eighty degrees (180°) between the dashed line 2727 indicating the second-level maximum rotation angle to the right and the dashed line 2729 indicating the second-level maximum rotation angle to the left. Because the user's 2702 face is not pointing toward the camera 2714 and the display panel 2712, the user's facial orientation 2704D is not identified because it is not captured by the camera 2714. Thus, the vision-based analyzer IC may infer that the user 2702 is not present. However, in some scenarios, the vision-based analyzer IC may infer that a user 2702 is present (e.g., based on other detectable features such as the body, back of the head, etc.), but the user's face is not detectable. This may make it useful, for example, to implement a more aggressive dimming policy than the one applied when the user's face orientation is within the second level region.

[0303] In one or more embodiments, additional level regions may be defined. In this embodiment, if the user's facial orientation is determined to be within the additional level region, the user's attention becomes unknown. In one possible implementation, additional level regions may be created to the left and right between the first and second level regions shown in FIGS. 27A through 27C by reducing the size of the first level region 2718 and the second level regions 2728A and 2728B. For example, the first level region (e.g., 2718) may span seventy-four degrees (74°) between the first level maximum rotation angle to the right (e.g., 37°) and the first level maximum rotation angle to the left (e.g., 37°). The second (additional) level region to the right may span fifteen degrees (15°) between the first level maximum rotation angle to the right (e.g., 37°) and the second level maximum rotation angle to the right (e.g., 31°). The second (additional) level region to the left may span fifteen degrees (15°) between the first level maximum rotation angle to the left (e.g., 37°) and the second level maximum rotation angle to the left (e.g., 31°). The third level region to the right (e.g., 2728A) may span thirty-eight degrees (38°) between the second level maximum rotation angle to the right (e.g., 31°) and the third level maximum rotation angle to the right (e.g., 90°). The third level region to the left may span thirty-eight degrees (38°) between the second level maximum rotation angle to the left (e.g., 31°) and the third level maximum rotation angle to the left (e.g., 90°). The fourth level region (e.g., 2738) may span one hundred eighty degrees (180°) between the third level maximum rotation angle to the right (e.g., 90°) and the third level maximum rotation angle to the left (e.g., 90°).

[0304] In this example, if the user's facial orientation is within one of the additional second-level regions, a determination may be made that the user's engagement status with the display device is unknown. In this scenario where the user has an unknown status, display management may be handled in any appropriate manner depending on the particular implementation and needs. In one example, the unknown status may prevent the display panel from being gradually dimmed to ensure that the dimming is not premature and / or undesirable. In some implementations, a timer may be started to monitor the time the user remains in the unknown status orientation. Once the threshold time has elapsed, the display brightness may be immediately reduced to a predetermined minimum display brightness level for a present but unengaged user, or a gradual dimming may be applied to the display device. Other level regions may be evaluated in a manner similar to that described with reference to FIGS. 27A through 27C. Specifically, if the user's facial orientation is within the first-level region, the user is inferred to be engaged with the display panel, and a normal / default level of brightness may be provided to the display panel. If the orientation of the user's face is within a third level region, it is inferred that the user is present but not engaged with the display panel associated with the captured image sensor data, while the user may be engaged with other display devices. In this scenario, backlight dimming may be gradually applied to the display panel. If the orientation of the user's face is within a fourth level region, it is inferred that the user is not present or that the user's face is undetectable. In this scenario, the display panel may be immediately turned off (e.g., if all other display devices provide image sensor data indicating the user is not present), or active dimming may be gradually applied to the display panel until the display panel is turned off and no backlight is provided.

[0305] It should be noted that the specific values ​​(e.g., degrees, distances) provided herein for the user's field of view are for illustrative purposes only. While such given values ​​may actually be implemented in one or more scenarios, the values ​​may be adjusted to any other appropriate values ​​based on specific needs and implementations. Furthermore, the values ​​may be adjusted on a per-display device basis. For example, in some scenarios, it may be desirable for certain display devices, such as those at the edge of several display devices within a single workstation, to have the rotation range of the first-level region expanded to ensure that the display panel is not dimmed too frequently.

[0306] 28A-28B show an example of a two-display system 2800 in which an embodiment of user presence-based display management is implemented. The two-display system 2800 includes a computing device 2805 and an external monitor 2820 operatively coupled to the computing device 2805. The computing device 2805 may include a base 2806 and a lid 2810. In this example, the lid 2810 includes an integrated primary display panel 2812 and a user-facing camera 2814 including an image sensor. The external monitor 2820 includes a secondary display panel 2822 and a user-facing camera 2839 including an image sensor. In this example, the cameras 2814 and 2839 are built into the bezel areas of their respective lids and monitors. However, it will be understood that the cameras may be integrated into their respective lids and monitors or externally coupled in any other suitable manner. 28A-28B also show a user 2802 and the possible attention directed to two display panels 2812 and 2822. FIG.

[0307] Each of the user-facing cameras 2814 and 2839 may be coupled to a respective vision-based analyzer IC (e.g., an LCH described herein, such as LCHs 155, 260, 305, 954, 1705, 1860, 2830A, 2830B, or circuitry including some of its features), which may be located within the lid / monitor housing or operatively coupled to the lid / monitor by an external connector (e.g., a dongle). Each vision-based analyzer IC may be configured to provide input to the computing device 2805 indicating whether a user is present, engaged, disengaged, or not within the field of view of its associated user-facing camera. The input from a given vision-based analyzer IC may be generated based on image sensor data generated for images captured by its associated camera 2814 or 2839.

[0308] FIG. 28A shows user 2802's attention directed toward primary display panel 2812, as shown at 2804A. Camera 2814 can capture images within a first field of view specific to camera 2814. A first vision-based analyzer IC coupled to camera 2814 can use image sensor data from the captured images to detect the user's presence and the user's facial orientation. The first vision-based analyzer IC can then provide input to computing device 2805 indicating whether the user is present, engaged, disengaged, or absent based on the first field of view of camera 2814. Camera 2839 can capture images within a second field of view specific to camera 2839. A second vision-based analyzer IC coupled to camera 2839 can use image sensor data from the captured images to detect the user's presence and the user's facial orientation from the second field of view. The second vision-based analyzer IC may then provide input to the computing device 2805 indicating whether the user is present, engaged, disengaged, or absent based on a second field of view of the camera 2839.

[0309] 28A-28B are capable of dimming and turning off the appropriate displays depending on the user's orientation. For example, in FIG. 28A , the vision-based analyzer IC associated with the display panel 2812 in the lid 2810 of the computing device 2805 may detect the user's presence and facial orientation as shown in 2804A. Meanwhile, the vision-based analyzer IC associated with the display panel 2822 of the external monitor 2820 may detect the user's presence but not the user's attention because the user's face is instead directed toward the computing device 2805. Thus, the computing device may initiate (or maintain) an appropriate operating mode to enable use of the computing device and its primary display panel 2812. The computing device may also adaptively dim the display panel 2822 of the external monitor 2820 until a threshold time has elapsed without detecting user attention to the external monitor. Once the threshold time has elapsed, the display panel 2822 may be turned off while the user continues to use the computing device and its built-in display panel 2812.

[0310] 28B , the vision-based analyzer IC associated with the display panel 2822 of the external monitor 2820 may detect the user's presence and facial orientation as shown in 2804B. Meanwhile, the vision-based analyzer IC associated with the display panel 2812 in the lid 2810 of the computing device 2805 may detect the user's presence but not the user's attention because the user's face is instead directed toward the external monitor 2820. Accordingly, the computing device may initiate (or maintain) an appropriate mode of operation to enable use of the computing device and the external display panel 2822. The computing device may also adaptively dim the display panel 2812 of the computing device 2805 until a threshold time has elapsed without detecting user attention to the computing device. Once the threshold time has elapsed, the display panel 2812 may be turned off while the user continues to use the computing device and the external monitor 2820.

[0311] If a computing device is connected to two or more additional external monitors, user presence-based policies may apply to any number of displays. Figures 29A-29C show an exemplary multiple display system 2900 in which user presence and attention-based display management is implemented. The multiple display system 2900 includes a computing device 2905, a first external monitor 2920 operatively coupled to the computing device 2905, and a second external monitor 2930 operatively coupled to the computing device 2905. The computing device 2905 may include a base 2906 and a lid 2910. In this example, the lid 2910 includes an integrated primary display panel 2912 and a user-facing camera 2914 that includes an image sensor. The first external monitor 2920 includes a secondary display panel 2922 and a user-facing camera 2939 that includes an image sensor. The second external monitor 2930 also includes a secondary display panel 2932 and a user-facing camera 2934 that includes an image sensor. In this example, cameras 2914, 2939, and 2934 are integrated into the bezel areas of their respective lids / monitors. However, it will be understood that the cameras may be integrated into their respective lids / monitors or externally coupled in any other suitable manner. Figures 29A-29C also illustrate possible attention directed to user 2902 and multiple display panels 2912, 2922, and 2932.

[0312] Each of the user-facing cameras 2914, 2939, and 2934 may be coupled to a respective vision-based analyzer IC (e.g., an LCH disclosed herein, such as LCH 155, 260, 305, 954, 1705, 1860, 2830A, 2830B, or circuitry including some of its features), which may be disposed within the lid / monitor housing or operatively coupled to the lid / monitor by an external connector (e.g., a dongle). Each vision-based analyzer IC may be configured to provide input to the computing device 2905 indicating whether a user is present, engaged, disengaged, or not within the field of view of its associated user-facing camera. The input from a given vision-based analyzer IC may be generated based on image sensor data generated for images captured by its associated camera 2914, 2939, or 2934.

[0313] FIG. 29A shows a user 2902 directed at a primary display panel 2912, as shown at 2904A. A camera 2914 can capture images within a first field of view specific to the camera 2914. A first vision-based analyzer IC coupled to the camera 2914 can use image sensor data from the captured images to detect the presence of the user and the orientation of the user's face. The first vision-based analyzer IC can then provide input to the computing device 2905 indicating whether the user is present, engaged, disengaged, or absent based on the first field of view of the camera 2914. A camera 2939 can capture images within a second field of view specific to the camera 2939. A second vision-based analyzer IC coupled to the camera 2939 can use image sensor data from the captured images to detect the presence of the user and the orientation of the user's face from the second field of view. The second vision-based analyzer IC may then provide input to the computing device 2905 indicating whether a user is present, engaged, disengaged, or absent based on the second field of view of the camera 2939. The camera 2934 may capture images within a third field of view specific to the camera 2934. A third vision-based analyzer IC coupled to the camera 2934 may use image sensor data from the captured images to detect the presence of a user and the orientation of the user's face from the third field of view. The third vision-based analyzer IC may then provide input to the computing device 2905 indicating whether a user is present, engaged, disengaged, or absent based on the third field of view of the camera 2934.

[0314] The multiple-display implementations of Figures 29A-29C are capable of dimming and turning off appropriate displays depending on where the user is facing. For example, in Figure 29A, a first vision-based analyzer IC associated with display panel 2912 in lid 2910 of computing device 2905 can detect a user's presence and facial orientation, as shown in 2904A. A second vision-based analyzer IC associated with display panel 2922 of external monitor 2920 detects a user's presence but may not detect the user's attention because the user's face is instead facing the computing device 2905. Similarly, a third vision-based analyzer IC associated with display panel 2932 of external monitor 2930 detects a user's presence but may not detect the user's attention because the user's face is instead facing the computing device 2905. Accordingly, the computing device may initiate (or maintain) an appropriate operating mode to enable use of the computing device and its primary display panel 2912. The computing device can also adaptively dim the display panel 2922 of the external monitor 2920 until a threshold time has elapsed without detecting user attention to the external monitor 2920. After the threshold time has elapsed, the display panel 2922 can be turned off while the user continues to use the computing device and its built-in display panel 2912. Similarly, the computing device can adaptively dim the display panel 2932 of the external monitor 2930 until a threshold time has elapsed without detecting user attention to the external monitor 2930. After the threshold time has elapsed, the display panel 2932 can be turned off while the user continues to use the computing device and its built-in display panel 2912.

[0315] 29B , a vision-based analyzer IC associated with a display panel 2922 of an external monitor 2920 can detect a user's presence and facial orientation, as shown in 2904B. A first vision-based analyzer IC associated with a display panel 2912 in a lid 2910 of a computing device 2905 detects a user's presence but may not detect the user's attention because the user's face is instead directed toward the external monitor 2920. Similarly, a third vision-based analyzer IC associated with a display panel 2932 of an external monitor 2930 detects a user's presence but may not detect the user's attention because the user's face is instead directed toward the external monitor 2920. Accordingly, the computing device may initiate (or maintain) an appropriate mode of operation to enable use of the computing device and the display panel 2922 of that external monitor 2920. The computing device may also adaptively dim the display panel 2912 of the computing device 2905 until a threshold time has elapsed without detecting user attention to the computing device. Once the threshold time has elapsed, the display panel 2912 may be turned off while the user continues to use the external monitor 2920. The computing device may also adaptively dim the display panel 2932 of the external monitor 2930 until the threshold time has elapsed without detecting user attention to the external monitor 2930. Once the threshold time has elapsed, the display panel 2932 may be turned off while the user continues to use the external monitor 2920.

[0316] 29C , a vision-based analyzer IC associated with the display panel 2932 of the external monitor 2930 can detect the presence of a user and the facial orientation shown at 2904C. A first vision-based analyzer IC associated with the display panel 2912 in the lid 2910 of the computing device 2905 detects the presence of a user but may not detect the user's attention because the user's face is instead directed toward the external monitor 2930. Similarly, a second vision-based analyzer IC associated with the display panel 2922 of the external monitor 2920 detects the presence of a user but may not detect the user's attention because the user's face is instead directed toward the external monitor 2930. Accordingly, the computing device may initiate (or maintain) an appropriate operating mode to enable use of the computing device and the display panel 2932 of the external monitor 2930. The computing device may also adaptively dim the display panel 2912 of the computing device 2905 until a threshold time has elapsed without detecting user attention to the computing device. Once the threshold time has elapsed, the display panel 2912 may be turned off while the user continues to use the external monitor 2930. The computing device may also adaptively dim the display panel 2922 of the external monitor 2920 until the threshold time has elapsed without detecting user attention to the external monitor 2920. Once the threshold time has elapsed, the display panel 2922 may be turned off while the user continues to use the external monitor 2930.

[0317] 30 is a block diagram illustrating additional possible details of a vision-based analyzer IC 3020 operatively coupled to an SoC 3002 in a multiple display computing system, such as multiple display computing system 2500. In one or more examples, SoC 3002 illustrates possible details of SoC 2508 in computing device 2505 of multiple display computing system 2500 of FIG. 25. In one or more examples, vision-based analyzer IC 3020 illustrates possible details of vision-based analyzer ICs 2540A, 2540B, and 2540C implemented within lid 2510 of computing device 2505, within first external monitor 2520, and external to second external monitor 2530 of multiple display computing system 2500, respectively. Accordingly, display device 3010 is shown in FIG. 30 to represent any one of possible display devices in which vision-based analyzer IC 3020 may be implemented. The camera 3014 may be integrated with the display device 3010 or coupled to the display device as an add-on and communicatively coupled to the vision-based analyzer IC 3020. The camera 3014 may be positioned to be user-facing and have a unique imaging field of view that extends out from the display panel in the display device 3010. Hardened indicators and controls 3012 may be provided within the display device 3010 and may include light-emitting diodes (LEDs) to indicate how the camera 3014 is being used (e.g., by software running on the SoC 3002, by the vision-based analyzer IC 3020).

[0318] Generally, in one or more embodiments, the SoC 3002 and the vision-based analyzer IC 3020 may perform different functions associated with display management for the display computing systems described herein. In some examples, the SoC 3002 includes an input / output (I / O) interface (IF) 3004, an integrated sensor hub (ISH) 3006, and an image processing module 3008. In some examples, the vision-based analyzer IC 3020 includes a vision / imaging module 3026, a security module 3022, and a selector 3024. The vision / imaging module 3026 is an artificial intelligence-based vision processing unit that supports processing image sensor data to detect human face(s) and head / face orientation. Image sensor data may be generated for each frame of a sequence of images captured by the user-facing camera 3014 and streamed into the vision / imaging module 3026, where, for each frame of image sensor data, a human face(s) and head / face orientation are detected, image metadata is generated (e.g., indicating user presence, absence, engagement, and / or disengagement), and the image metadata is transmitted to the ISH 3006 of the SoC 3002. In some embodiments, the image metadata may indicate when a user is present but their face is undetectable, such as when the user has turned away from the display screen (e.g., the user's face orientation is within the third-level region 2738). In one or more examples, the camera 3014 can transmit a stream of image data sensor files (or frames) to the vision / imaging module 3026, which may include a neural network accelerator (NNA) 3030 and a storage unit such as a database 3040.In one or more examples, SoC3002 and vision-based analyzer IC3020, and components thereof, may be configured with at least some of the same features as provided in one or more other SoCs (e.g., 140, 390, 340, 914, 1840, 2840A, 2840B, 2508) and / or lid controller hubs (e.g., 155, 260, 305, 954, 1705, 1860, 2830A, 2830B) disclosed herein.

[0319] The NNA 3030 may be configured to perform an initial analysis of image sensor data generated by a camera 3014 embedded in or coupled to the display device 3010 to determine whether a user is present or absent within the camera's field of view and whether the user is engaged or disengaged with the display device. The NNA 3030 may utilize machine learning algorithms (e.g., neural networks) to detect human faces, facial orientation, and / or multiple faces in the image sensor data received from the camera 3014. The NNA 3030 may include hardware, firmware, software, or any suitable combination thereof to perform human face(s) and facial orientation detection. Image metadata for the display device 3010 may be generated each time image sensor data is generated by the camera 3014 and analyzed by the NNA 3030. The image metadata may indicate whether a user is present or absent within the camera's field of view and whether the user is engaged or disengaged with the display device 3010.

[0320] The exemplary vision-based analyzer IC 3020 may be implemented as a separate die from the SoC 3002 and is specifically designed to perform this vision-based analysis at relatively low power (e.g., approximately 10 mW) for an "always on" implementation. The vision-based analyzer IC 3020 may be implemented within the display device 3010, in one exemplary implementation of the vision-based analyzers 2540A, 2540B, 2540C. The vision-based analyzer IC 3020 may be configured in or as part of a lid controller hub (LCH) disclosed herein (e.g., LCH 155, 260, 305, 954, 1705, 1860, 2830A, 2830B). After analyzing the image sensor data and generating image metadata responsive to the analysis, the vision / imaging module 3026 may transmit the image metadata to a lid controller hub (LCH) (e.g., LCH 155, 260, 305, 954, 1705, 1860, 2830A, 2830B) for further processing. 2 The image metadata may be transmitted to an integrated sensor hub (ISH) 3006 of the SoC 3002 (via the C serial bus). In one optimization, the image metadata may be stored in a database 3040 for comparison with subsequent image metadata generated for new image sensor data captured by the camera 3014. In this optimization, the image metadata is transmitted to the SoC only in response to a determination that an event has occurred based on a comparison of the newly generated image metadata with previously generated image metadata stored in the database 3040.

[0321] When ISH3006 receives image metadata from the vision-based analyzer IC3020 of the display device 3010, the ISH may use the image metadata to identify an appropriate operational mode for the SoC 3002, the display device associated with the received image metadata, and other display devices operatively coupled to the SoC 3002, and adjust corresponding performance parameters accordingly. Example operational modes may include, but are not necessarily limited to, one or more of: 1) present and engaged, 2) present and passively engaged, 3) present and disengaged, and 4) absent.

[0322] As shown in the depicted example, the vision-based analyzer IC 3020 is communicatively coupled between the camera 3014 and the SoC 3002 to enable the vision / imaging module 3026 to perform an initial low-power analysis of the image sensor data. However, when a user initiates an activity that involves use of the camera 3014 (e.g., for a video conference call), the example selector 3024 may forward the image sensor data directly to the image processing unit 3008 of the SoC 3002, bypassing the vision / imaging module 3026.

[0323] In some examples, the vision-based analyzer IC 3020 includes a security module 3022 to maintain the security and / or integrity of the vision-based analyzer IC 3020. The security module 3022 may provide the end user with full visibility and control of the user-facing camera 3014. In some implementations, the security module 3022 may communicate with the SoC 3002 via the I / O interface 3004. However, the security module 3022 ensures that image sensor data indicative of what is captured within the camera's 3014 field of view is not disclosed to the SoC 3002 without authentication. For example, a user may have the option to authenticate video frames of a video call. Furthermore, the security module 3022 ensures that the image sensor data cannot be accessed by potential malware without the user's knowledge (e.g., via the hardened indicators and controls 3012) or control (e.g., via a privacy switch). Specifically, hardened indicators and controls 3012 may ensure that the current state of selector 3024 (e.g., normal, ULP vision, or privacy mode) is appropriately reflected in the indicators. In addition to receiving image sensor data, vision-based analyzer IC 3020 may also receive inputs for hardened indicators and controls 3012 and provide appropriate signals to LEDs via general-purpose inputs / outputs (GPIOs).

[0324] 31 is a block diagram illustrating additional possible details of the vision / imaging module 3026 of the vision-based analyzer IC 3020 and the integrated sensor hub (ISH) 3006 of the SoC 3002. The vision / imaging module 3026 may include an NNA 3030, a database 3040, and image processing algorithms 3039. The ISH 3006 may include a display management policy 3001, a performance controller 3003, an operating mode selector 3005, and operating mode definitions / rules 3007.

[0325] The NNA 3030 may implement one or more deep neural networks (DNNs), such as a convolutional neural network (CNN), tuned for human face, head / face orientation, and multi-face detection. In at least one embodiment, the neural network may be implemented using a machine learning model trained to recognize human faces and face orientations. In at least one example, the machine learning model is trained to identify the orientation of a human face by degrees of rotation. In at least one embodiment, the degrees of rotation may be measured from the path between the human face and a display device with an embedded or coupled camera. The machine learning engine 3050 may train the machine learning model using training image data 3052. Exemplary training image data 3052 may include past image data for a particular user or users and / or multiple other human subjects. The machine learning engine 3050 may operate within a computing device (e.g., 2505) associated with the vision-based analyzer IC using the model, within a local or remote server, in the cloud, or any other suitable system or device where the trained model can be provided or accessible by a suitable vision-based analyzer IC such as the vision-based analyzer IC 3020.

[0326] The training image data 3052 may be stored in any suitable storage unit or memory. In some examples, the training image data 3052 may be stored on the same system or device on which the machine learning engine 3050 is stored and / or operating. In another example, the training image data 3052 may be stored outside the system or device on which the machine learning engine 3050 is stored and / or operating, while being in a location accessible to the machine learning engine 3050.

[0327] The trained models may be stored in database 3040 for use by NNA 3030 to detect human face(s) and determine human face orientation. For example, face detection model(s) 3044 and face orientation model(s) 3046 may be stored in database 3040.

[0328] In one or more examples, the NNA 3030 may include a human face detector 3034, a face orientation detector 3036, and a multi-face detector 3038. The human face detector 3034 may use face detection model(s) 3044 to identify human faces from the image sensor data generated by the user-facing camera 3014, from a unique imaging FoV encompassing the front portion of the display device on which the vision-based analyzer IC 3020 is located. Detecting a human face in the image sensor data indicates the presence of a user in the imaging FoV. Not detecting a human face in the image sensor data indicates the absence (or non-existence) of a user in the imaging FoV. The human face detector 3034 may further generate information that can be used to determine the distance of the detected user's face from the camera that generated the image sensor data. In one example, the neural network of the human face detector 3034 may be trained to predict a bounding box of a human head as part of the human face detection. Once the bounding box is determined and a human face is recognized, the distance of the user (e.g., the user's face) to the camera can be determined based on the size of the bounding box. In some implementations, this determination can be performed by the NNA 3030. In other implementations, the bounding box information can be provided to an image processing algorithm 3039 to determine the distance of the detected human face to a camera associated with a display device.

[0329] When a human face is detected, the face orientation detector 3036 may use the face orientation model(s) 3046 to determine the face orientation. In one example, the face orientation may be provided by the degree to which the user is facing the camera 3014 relative to the path between the detected human face. The face orientation may be determined based on identifying facial features in the image sensor data (e.g., number of visible ears, number of visible eyes). The face orientation may be used to infer whether the user is engaged or disengaged with a particular display device associated with the camera that generated the image sensor data. In one example, as described above herein, if the user's face orientation is within a level 1 region, it may be inferred that the user is engaged with the display device associated with the camera 3014. If the user's face orientation is within a level 2 region, it may be inferred that the user is not engaged with the display device associated with the camera 3014, but may be engaged with other display devices docked or otherwise connected to the same computing device. If the user's facial orientation is within the Level 3 region, it may be inferred that the user is not engaging with the display device associated with the camera 3014 or any other display devices docked or otherwise connected to the same computing device. In some implementations, these inferences may be determined by the NNA 3030. In other implementations, these inferences may be determined by the image processing algorithms 3039.

[0330] In one or more embodiments, the NNA 3030 may further include a multi-face detector 3038 to detect multiple faces within the image sensor data generated by the user-facing camera 3014 from its unique imaging FoV. In at least one embodiment, the face detection model(s) 3044 may be trained to detect multiple human faces in the image sensor data of a single image. In another embodiment, the face detection model(s) 3044 may be trained to detect a single human face, and other models may be trained for multi-face detection. Once multiple human faces are detected, in at least one embodiment, each detected face may be analyzed to determine the facial orientation and whether a user associated with the detected face is engaged with the display device. Dimming and / or turning off the display panel of the display device associated with the camera 3014 may be performed only if all detected human faces are determined to be disengaged from the display device.

[0331] The image processing algorithm 3039 may be provided within the vision imaging module 3026 as part of the NNA 3030 or may be implemented separately. The image processing algorithm 3039 may be implemented within circuitry and may include hardware, firmware, software, or any suitable combination thereof. The image processing algorithm 3039 may use information generated by the human face detector 3034, face orientation detector 3036, and multi-face detector 3038 based on the current frame of image sensor data to generate new image metadata for the image captured by the camera 3014. The new image metadata may indicate whether a user is present, absent, engaged, or disengaged in the current image sensor data representing the captured image. Once the new image metadata is generated, the image processing algorithm 3039 may compare the new image metadata with previous image metadata 3042 stored in the database 3040 to determine whether any changes between the two image metadata binaries (or files) indicate the occurrence of an event related to the user's presence, absence, engagement, or disengagement with a display device associated with the camera 3014. Previous image metadata 3042 may indicate image metadata generated based on a frame of image sensor data received by vision / imaging module 3026 immediately prior to receiving the frame of image sensor data for which new image metadata was generated.

[0332] If, based on the comparison, it is determined that an event has occurred, the vision / imaging module 3026 may store a copy of the new image metadata in database 3040 as previous image metadata 3042 and transmit the new image metadata to ISH 3006. If, based on the comparison, it is determined that an event has not occurred, in at least one embodiment, to conserve processing resources, the new image metadata may not be transmitted to ISH 3006. However, the new image metadata may still be stored in database 3040 as previous image metadata 3042.

[0333] In one or more embodiments, ISH3006 of SoC3002 is configured to receive image metadata from vision / imaging module 3026, apply appropriate policies based on the image metadata, select an appropriate operating mode for a display device associated with the received image metadata and possibly a computing device docked or otherwise connected to the display device, and, if necessary, adjust performance parameters based on the selected operating mode.

[0334] The operational mode definitions / rules 3007 may include various operational modes that may be applied to each display device (e.g., 2510, 2520, 2530) based on the image metadata generated for each display device individually. Based on the image metadata and policies applicable to the received metadata, an appropriate operational mode may be selected and applied to the display device associated with the received image metadata. In one example, operational modes that may be applied to a particular display device may include, but are not necessarily limited to:

[0335] Engaged operating mode - Image metadata indicates the user is present and engaged. The user is detected and the gaze direction is within the Level 1 region relative to the display device (e.g., the user may be reading or watching a video on the display device).

[0336] Adaptive dimming operating mode (normal or aggressive) - Image metadata indicates a user is present and disengaged. A user is detected but is not engaged with the display device (but may be engaged with other display devices in the computing system).

[0337] Unattended operating mode - Image metadata indicates that the user is not present or the user's face is not detectable. The user's face is not detected.

[0338] It should be understood that other modes of operation, such as modes of operation for an SoC or other processing element of a computing device, may also be used, and these other modes of operation may be used in conjunction with the modes of operation for a display device described herein.

[0339] In one or more embodiments, the performance controller 3003 may control performance parameters governing power consumption, performance, and / or system responsiveness of the computing device 2505 and multiple display devices 2510, 2520, and 2530 docked or otherwise connected to the computing device. The performance parameters may be adjusted in response to a determination of user presence and / or engagement with one or more display devices 2510, 2520, 2530 in the computing system 2500. For example, the performance controller 3003 may control the performance parameters to wake the system and trigger face-based authentication when one of the display devices in the system detects that a user is approaching. In another case, the performance controller 3003 may control the performance parameters to dim the backlight for a particular display device if it is determined that the user is not engaged with the particular display device. The backlight may be gradually dimmed (e.g., the brightness of the backlight is reduced) based on a predetermined period of user inactivity with the display device. In another example, the performance controller 3003 may control performance parameters to lock the computing system immediately upon determining that a user has walked away from multiple displays in the computing system. In yet another example, the performance controller 3003 may control performance parameters to stop any attempts by the SoC to lock the system when at least one display device in the computing system detects that a user is present and engaged, even if the user is not interacting with a user interface such as a keyboard, mouse, touchpad, etc.

[0340] Various display management policies 3001 may be implemented by the SoC 3002 based on image metadata received from the vision / imaging module 3026. Exemplary display management policies 3001 may include, but are not limited to, a face-activation policy, an adaptive dimming policy, a presence-activated unlock policy, and an unattended lock policy. In a multiple-display computing system, a face-activation policy may be invoked in response to a user approaching the computing system and entering the field of view of one or more cameras associated with each display device. When the face-activation policy is invoked, the computing system wakes up and triggers face-based authentication (e.g., Windows Hello® authentication software). Thus, the computing system is ready before the user sits in front of any of the display devices or interacts with the computing system (e.g., via user interface mechanisms, voice). For a multiple-display computing system, the display device that the user first engages with (e.g., draws their attention to that display device) triggers the computing system to wake up. Other display devices in the multiple-display computing system remain off until the user successfully logs into the computing system. A multiple-display face activation policy covers a larger area that a user views in close proximity to a computing system. Because each display device has an associated camera with a different field of view, the area around a multiple-display computing system where movement and light can be detected and images can be captured is larger than with a single-display computing system. This creates a more robust solution, especially for users who may enter the work area from different directions.

[0341] In one or more embodiments, an adaptive dimming policy can be invoked in multiple display computing systems to gradually dim the backlight of a display panel over a defined period of user inattention and to turn off the display device when the user is no longer present. When the adaptive dimming policy is invoked, a normal or aggressive dimming mode of operation can be selected for a particular display device. Implementing this policy can significantly extend battery life. For each display device implemented with a vision-based analyzer IC, power savings and responsiveness can be optimized for each display device. For example, when a user is engaged and looking at a first display device directly in front of the user, dimming of any other displays the user is not looking at can be adjusted. In one implementation, the adaptive dimming policy can include gradually dimming display panels that the user is no longer engaging with over a period of time and turning off display panels that the user is no longer facing (e.g., face orientation > 90° in either direction). This policy can be applied to any number of additional external display devices that have an integrated or add-on vision-based analyzer IC3020 (or LCH155, 260, 305, 954, 1705, 1860, 2830A, 2830B, etc.). The display panels of the display devices can be dimmed and turned off, and the refresh rate can also be managed. For example, the refresh rate can be reduced for any display panel that the user has disengaged. Thus, the SoC3002 can apply an adaptive dimming policy to reduce the refresh rate and reduce rendering to optimize performance.

[0342] In one or more embodiments, a no-lock-on-presence policy may be invoked in a multiple-display computing system to prevent the computing system from locking and the display device from being turned off when a user is present, even if the user is not actively typing or moving the mouse, or the user is disengaged. When the no-lock-on-presence policy is invoked, an engaged mode of operation may be selected (or not changed) for a particular display device if the user is engaged. This scenario may occur, for example, when a user is reading a long document or watching an embedded video. If the user is disengaged, an unengaged mode of operation may be selected (or not changed) for a particular display. On the other hand, a no-lock-on-presence policy may be invoked to prevent the computing system from locking and to prevent the backlight from being turned off completely. Thus, in at least one embodiment of a multiple-display computing system, the computing system is not locked if image metadata from at least one display device indicates that the user is either present but engaged (e.g., face orientation between ≦45° in either direction) or disengaged (e.g., face orientation >45° and ≦90° in either direction).

[0343] In one or more embodiments, an unattended lock policy may be invoked in a multiple display computing system to turn off the display devices and quickly lock the computing system if the user walks away before the inactivity timeout expires. When the unattended lock policy is invoked, an unattended operating mode may be selected for each specific display device. The inactivity timeout may be typically about 5 minutes and may be implemented based on the user's lack of interaction with the computing system (e.g., via a user interface such as a mouse or touchpad). When certain applications (e.g., embedded video) are running, no inactivity timeout may occur. In one or more embodiments, a multiple display computing system may be configured to lock the computing system and turn off all display devices when all display devices indicate user absence (e.g., user is absent or face orientation > 90° in any direction).

[0344] Embodiments of display management for multiple display computing systems provide an improved user experience. With improved accuracy from additional user presence inputs, embodiments enable faster and seamless detection of a user's proximity to their system, as the first display capable of detecting user proximity can wake the system. Furthermore, improved accuracy is provided for multiple inputs. Adaptive dimming provides energy and power savings by turning off either the laptop display or additional external displays that the user is not engaging. This power savings can be on the order of 29 to 100 W. This significant energy savings can also be applied to wireless display scenarios.

[0345] Embodiments of display management in a multiple display computing system eliminate the need to drive additional external monitors when the user is not engaged with one or more of those external monitors, thereby enabling reduced refresh rates and rendering, thereby optimizing performance. High-resolution displays typically come with higher costs, including rendering tax on GPU overhead for processing and rendering, display tax on graphics and display engines for data organization and transmission, and CPU budget impact on thermally constrained form factors. Analysis using battery life measurement tools has shown a performance loss of ≥ 50% when attaching an external 4K display.

[0346] Without the need to unnecessarily drive additional display devices, rendering and refresh rates can be significantly reduced. By timely reducing refresh rates and display rendering when a user is disengaged or absent, display power and energy can meet (and even exceed) certain state policies (e.g., California Energy Policy and Energy Star Standards). Furthermore, correlation policy issues can be appropriately addressed to optimize the user experience for specific features (e.g., face activation, adaptive dimming, present-unlock, unattended-lock). Correlation policies can include when there is a user input event (HID) that correlates with whether the user is present and engaged, disengaged, or absent. Embodiments can eliminate this issue for multiple display computing system scenarios, accurately indicating whether the user is present and engaged or disengaged / absent without having to wait for a user input event. Specifically, embodiments can indicate which displays are not correlated for effective management of power, energy, and performance.

[0347] 32 through 37, simplified flowcharts illustrate exemplary hardware logic, machine-readable instructions, firmware, software, or any suitable combination thereof, that may be associated with an embodiment of a multiple display computing system 2500 in which user presence and attention-based display management is implemented. In at least one embodiment, the set of operations corresponds to the operations illustrated in the flowcharts of FIGS. 32 through 37. In one example, a lid control hub (e.g., 155, 260, 305, 954, 1705, 1860, 2830A, 2830B), or a portion thereof (e.g., a vision-based analyzer integrated circuit (e.g., 2540A, 2540B, 2540C, 3020)) may utilize or perform at least some of the operations, and an SoC (e.g., 140, 390, 340, 914, 1840, 2840A, 2840B, 2508, 3002) may utilize or perform at least some of the operations. For ease of illustration, the flowcharts of Figures 32 through 37 may be described with reference to the components of Figures 25 and 30. However, these components are further illustrated and described throughout this application, and it will be understood and accepted that one or more of these illustrations and descriptions may be applicable to the components referenced with respect to Figures 32 through 37.

[0348] 32 is a high-level flowchart of an exemplary process 3200 associated with detecting a user's presence in a multiple display computing system (e.g., 2500) according to at least one embodiment. In at least one embodiment, a set of operations corresponds to the operations of exemplary process 3200. In one example, the operations may be performed by a vision / imaging module (e.g., 172, 263, 363, 1740, 1863, 2832A, 2832B, 3026) disposed within a display device (e.g., 391, 341, 2510, 2520, 2530, 3010) of the multiple display computing system and a camera (e.g., 2514, 3014) associated with the display device. More specifically, a neural network accelerator (e.g., 276, 327, 1740, 1863, 2834A, 2834B, 3030) may perform one or more of the operations.

[0349] At 3202, movement within a field of view (FoV) of a camera 3014 associated with the display device 3010 may be detected. In some implementations, the movement detection may be performed by an imaging sensor integrated into the camera or communicatively coupled to the camera 3014. At 3204, the camera captures a new image (or frame) within the FoV associated with the camera. The camera generates image sensor data for the new image (or frame), and the image sensor data is provided to a vision-based analyzer IC 3020 associated with the display device 3010. More specifically, in at least one embodiment, the image sensor data is provided to a neural network accelerator 3030 to perform user presence detection, face detection, face orientation detection, and optionally multi-face detection.

[0350] At 3205, the NNA 3030 may detect a human face(s) and, if a human face is detected, may run one or more machine learning algorithms to detect the orientation of the human face(s). At 3206, the vision / imaging module 3026 may generate new image metadata based on the face, user presence, and head orientation detection performed by the NNA 3030. The new image metadata may include, for example, an indication of whether a user is present or absent in a new image captured in front of the display device 3010 by the camera 3014. User presence may be determined based on face detection. If a human face is detected, the user is present. If a user is determined to be present, the new image metadata of head orientation may further include an indication of whether the user is engaged with the display device 3010 associated with the camera or is disengaged (e.g., passively engaged) with the display device 3010.

[0351] At 3208, the new image metadata can be compared to the previous image metadata to determine whether any user events have occurred since the previous image associated with the previous image metadata was captured. To determine whether a user event has occurred, new image metadata generated based on image sensor data for the new image captured by the camera 3014 can be compared to stored previous image metadata previously generated based on image sensor data for images captured before the new image was captured. Thus, the new image metadata is compared to the previous image metadata to identify differences and determine whether the differences correspond to a user event. A user event can occur when a change in user presence is detected in the new image metadata compared to the previous image data. For example, the user may be absent in the previous image, but the user may be present in the new image. Or, the user may be present in the previous image, but the user may be absent in the new image. Another possible user event is a change in the user's facial orientation. In this scenario, a human face may have been detected in the previous image, and the facial orientation may have been identified. If the new image metadata indicates that the detected user's facial orientation is different from the facial orientation indicated in the previous image metadata, a user event has occurred. For example, the previous image metadata may indicate that the user was not engaging with the display device (determined based on facial orientation), and the new image metadata may indicate that the user is engaging with the display device (determined based on facial orientation), or vice versa.

[0352] At 3210, it is determined whether a user event has occurred based on the comparison performed at 3208. If a user event has not occurred (e.g., the previous image metadata and the new image metadata are the same), then, as an optimization, at 3212, new image metadata may not be sent to SoC 3002. If a user event has occurred based on the comparison performed at 3208, then, at 3214, new image metadata is sent to SoC 3002 to determine an operating mode for the display device (and optionally, the computing device to which the display device is connected). At 3216, the new image metadata may be stored as the previous image metadata for comparison against next new image metadata generated for the display device.

[0353] 33 is a high-level flowchart of an exemplary process 3300 associated with processing new image sensor data in a multiple display computing system (e.g., 2500) to detect user presence, human face(s), and head / face orientation, according to at least one embodiment. The exemplary process 3300 may provide additional details associated with one or more operations (e.g., 3205, 3206) of process 3200 of FIG. 32. In at least one embodiment, a set of operations corresponds to the operations of the exemplary process 3300. In one example, at least some of the operations may be performed by image processing algorithms 3039 in a neural network accelerator (e.g., 276, 327, 1740, 1863, 2834A, 2834B, 3030) and, optionally, a vision / imaging module (e.g., 172, 263, 363, 1740, 1863, 2832A, 2832B, 3026) of a vision-based analyzer IC (e.g., 2540A, 2540B, 2540C, 3020).

[0354] At 3302, the NNA 3030 may run one or more machine learning algorithms on the image sensor data of a new image captured by the camera 3014 associated with the display device 3010. An example of a machine learning algorithm may be a neural network model that is trained to detect human faces and, when executed, is capable of detecting the presence of a human face in the image sensor data. At 3304, the neural network model for face detection determines whether a human face is detected in the image sensor data. If no human face is detected, at 3306, new image metadata is generated to indicate that no user is present. The new image metadata may further indicate that no user is engaging with the display device 3010.

[0355] If the neural network model for face detection determines at 3304 that a human face has been detected in the image sensor data, then at 3308, another machine learning algorithm may be executed to determine the head / face orientation of the detected human face. One example of a machine learning algorithm may be a neural network model that is trained to detect the orientation of a human face (e.g., in degrees) and that, when executed, is capable of detecting the orientation of the human face detected in the image sensor data. In at least one embodiment, the face orientation may be determined in degrees of rotation relative to a direct display path defined between the user's face and the display device. The direct display path may be calculated from the detected face to the center of the display panel of the display device, to the camera that captured the image, or to any other suitable point associated with the display device.

[0356] At 3310, the orientation of the user's face may be evaluated to determine if the inferred engagement with the display device 3010 is within a maximum rotation angle. In one example, the maximum rotation angle is a first-level maximum rotation angle (e.g., 24°) of the user's face in either direction relative to a direct display path between the user's face and the display device 3010 (e.g., at the center of the display panel of the display device, the camera, or any other location on the display device).

[0357] If, at 3312, it is determined that the user's facial orientation is within a first level maximum rotation angle (e.g., 24°), it may be inferred that the user is engaged with the display panel of the display device 3010 (e.g., the user may be reading or looking at something on the display panel). In this scenario, at 3316, new image metadata is generated to indicate that the user is present and engaged with the display device 3010.

[0358] If the orientation of the user's face is determined to exceed the first-level maximum rotation angle but not exceed the second-level maximum rotation angle (e.g., 90°), it may be inferred that the user is disengaged from the display panel of display device 3010 but may be engaged with other display panels in a computing system to which display device 3010 is docked or otherwise connected. In this scenario, new image metadata is generated in 3314 to indicate that the user is present but is not engaged with (i.e., disengaged from) display device 3010.

[0359] Note that if the user's face orientation exceeds the second level maximum rotation angle or is otherwise undetectable, it may be determined that the user is not present, as evaluated in 3306.

[0360] It should also be noted that in another embodiment, the SoC (e.g., 3002) may determine whether the user's facial orientation is within a certain threshold level of rotation to infer engagement or disengagement. In this embodiment, the metadata may include an indication of the determined facial orientation, and this information may be used by the SoC to determine whether the user is engaged or disengaged with the display device.

[0361] 34 is a high-level flowchart of an example process 3400 associated with processing new image metadata generated by a vision-based analyzer IC (e.g., 3020) of a display device (e.g., 3010) in a multiple display computing system (e.g., 2500). In at least one embodiment, the set of operations and / or instructions correspond to the operations of the example process 3400 for receiving new image metadata generated by a vision-based analyzer IC (or LCH) in a computing system and applying face activation policies, if applicable, when the computing system is locked (e.g., in an unattended mode of operation). In one example, at least some of the operations may be performed by an SoC (e.g., 140, 390, 340, 914, 1840, 2840A, 2840B, 4140, 3002, 2508) of a multiple display computing system (e.g., 2500). In a more specific example, one or more operations of process 3400 may be performed by an internal sensor hub (e.g., 392, 342, 1790, 1842, 3006) of the SoC.

[0362] In 3402, when the display devices are turned off (e.g., no backlight is provided to the display panel) and the computing system is locked, the multiple display computing system SoCs 3002 receive new image metadata from the vision-based analyzer IC 3020 in the display devices 3010. In at least one embodiment, the computing system may be locked when an unattended mode of operation is selected for all display devices.

[0363] At 3404, it is determined whether the new image metadata indicates that a user is present. If a machine learning algorithm (e.g., NNA 3030) detects a human face in the image sensor data of the new image captured by camera 3014, a user may be indicated as present in the metadata. If the new image metadata does not indicate that a user is present, display device 3010 remains off and the computing system remains locked.

[0364] If the new image metadata indicates a user is present, then in 3406, it is determined whether the new image metadata indicates that the user is engaged with the display device. If the orientation of the user's face (e.g., relative to a direct display path to the display device) is determined to be less than or equal to the first-level maximum rotation angle in either direction, the new image metadata may indicate that the user is engaged with the display device. If the new image metadata does not indicate that the user is engaged with the display device, the display device 3010 remains off and the computing system remains locked. However, the user may be engaged with another display device in a multiple-display computing system. In that case, new image metadata received from the other display device (which may indicate that the user is present and engaged with the display device) may cause the SoC to trigger an authentication mechanism.

[0365] If, in 3406, the new image metadata indicates that a user is engaged with the display device, then, in 3408, it may be determined whether another display device in the multiple display computing system has already caused SoC 3002 to trigger an authentication mechanism. If so, display device 3010 remains off until the user is authenticated through another display device.

[0366] If, in 3408, it is determined that there is no other display device in the multiple display computing system that has already caused SoC 3002 to trigger an authentication mechanism, in 3410, a face activation policy may be invoked and the SoC may trigger an authentication mechanism on display device 3010. The other display devices in the multiple display computing system may remain off until the user is authenticated through display device 3010.

[0367] 35 illustrates a high-level flowchart of an exemplary process 3500 associated with processing new image metadata generated by a vision-based analyzer IC (e.g., 3020) of a display device (e.g., 3010) in a multiple display computing system (e.g., 2500). In at least one embodiment, a set of operations corresponds to those of the exemplary process 3500 for receiving new image metadata and invoking a dimming policy or an unattended lock policy, if applicable, when the display device is in an engaged mode of operation (e.g., a display panel of the display device has an initial brightness). In one example, at least some of the operations may be performed by an SoC (e.g., 140, 390, 340, 914, 1840, 2840A, 2840B, 4140, 3002, 2508) of a multiple display computing system (e.g., 2500). In a more specific example, one or more operations of process 3400 may be performed by an internal sensor hub (e.g., 392, 342, 1790, 1842, 3006) of the SoC.

[0368] In 3502, when the display device is in an engaged operating mode (e.g., the display panel of the display device has an initial brightness), the SoC 3002 of the multiple display computing system receives new image metadata from the vision-based analyzer IC 3020 in the display device 3010.

[0369] At 3504, it is determined whether the new image metadata indicates a user is present. If the new image metadata indicates a user is present, then at 3506, it is determined whether the new image metadata indicates a user is engaged with the display device. If the user is engaged with the display device, the orientation of the user's face may be within a first level region (e.g., less than or equal to a first level maximum rotation angle in either direction). If the new image metadata indicates a user is engaged with the display device, then the display device remains in an engaged mode of operation (e.g., full display brightness or initial display brightness), as shown at 3508.

[0370] If it is determined in 3506 that the new image metadata does not indicate a user is engaged with the display device 3010, then it is determined in 3510 whether the new image metadata indicates a user is disengaged (but present) at the display device. If it is determined that a user is disengaged (but present), then in 3514 an adaptive dimming policy may be invoked and a normal dimming mode of operation may be selected for the display device 3010. In one example, the normal dimming mode of operation may reduce the backlight of a display panel of the display device 3010 by a predetermined brightness percentage until a predetermined minimum level of brightness is reached. For example, the brightness of the display panel may be reduced five percent (5%) after five seconds and then reduced five percent (5%) every second until twenty percent (20%) is reached. The display panel may be maintained at twenty percent (20%) brightness until a user event occurs.

[0371] If, at 3510, it is determined that the new image metadata indicates a user is present but not engaged or disengaged, the user's facial orientation may be in a third-level region (e.g., exceeding the second-level maximum rotation angle in either direction). Thus, the user's face may be undetectable. In this scenario, different approaches may be desirable for different users. First, at 3512, an aggressive adaptive dimming policy may be invoked, and an aggressive dimming mode of operation may be selected for the display device 3010. In one possible implementation (or system / user configuration), the aggressive dimming mode of operation may reduce the backlight of a display panel of the display device 3010 by a predetermined percentage of brightness until the backlight is turned off. For example, the brightness of the display panel may be reduced twenty percent (20%) after five seconds and reduced one percent (1%) every second until it reaches zero percent (0%) and the backlight can be turned off. In another possible implementation (or system / user configuration), an aggressive dimming mode of operation may reduce the brightness of the display panel of the display device 3010 by a predetermined percentage until the backlight is reduced to a predetermined minimum level of brightness. In implementations where the backlight eventually turns off, a no-lock-on-presence policy may be invoked (described further herein) to prevent the computing system from locking in the presence of a user, even if the user's face is undetectable. However, in other implementations, the system may be configured to lock after the backlight is turned off if the user's face is not detected (e.g., the user's face is oriented within a third-level region).

[0372] Referring again to 3504, if the new image metadata does not indicate that a user is present, then at 3520, an unattended locking policy may be evaluated to determine whether it should be invoked. At 3520, it is determined whether the last image metadata received from each of the other display devices in the multiple display computing system indicates that a user is not present. If so, then at 3522, an unattended locking policy may be invoked and an unattended mode of operation may be selected for the display device 3010. When the unattended mode of operation is selected, the display device is turned off and no backlight is provided. If the unattended mode of operation is selected for the display device 3010 and all other display devices are already in the unattended mode of operation (or an aggressive dimming mode of operation), then the SoC of the computing system locks and the user must re-authenticate to unlock and use the computing system.

[0373] If, at 3520, it is determined that the last image metadata from at least one of the other display devices in the computing system indicates that a user is present, then, at 3539, an aggressive adaptive dimming policy may be invoked and an aggressive dimming operating mode may be selected for the display device 3010 as described above.

[0374] FIG. 36 illustrates a high-level flowchart of an exemplary process 3600 associated with processing new image metadata generated by a vision-based analyzer IC (e.g., 3020) of a display device (e.g., 3010) in a multiple display computing system (e.g., 2500). In at least one embodiment, a set of operations corresponds to those of the exemplary process 3600 for receiving new image metadata when the display device is in a normal or aggressive dimming mode of operation. In one example, at least some of the operations may be performed by an SoC (e.g., 140, 390, 340, 914, 1840, 2840A, 2840B, 4140, 3002, 2508) of the multiple display computing system (e.g., 2500). In a more specific example, one or more operations of process 3600 may be performed by an integrated sensor hub (e.g., 392, 342, 1790, 1842, 3006) of the SoC.

[0375] At 3602, a multiple display computing system SoC 3002 receives new image metadata from a vision-based analyzer IC 3020 in a display device 3010 when the display device is in a dimmed mode of operation.

[0376] At 3604, it is determined whether the new image metadata indicates a user is present. If the new image metadata indicates a user is present, then at 3606 it is determined whether the new image metadata indicates a user is engaged with the display device. If the new image metadata indicates a user is engaged with the display device, then at 3608 the dim mode of operation for the display device is changed to an engaged mode of operation. Thus, the brightness of the display panel ...

Claims

1. A method comprising: a lid controller hub receiving touch sensor data from a touch display controller; in response to receiving by the lid controller hub a display refresh notification from a display subsystem to refresh a touch display; generating conditioned touch sensor data based on the touch sensor data by the lid controller hub; the lid controller hub transmitting the conditioned touch sensor data to an operating system; A method comprising: the steps of receiving the touch sensor data, generating the conditioned touch sensor data, and transmitting the conditioned touch sensor data are performed within a lid of a mobile computing device, and the operating system is performed on one or more processing units disposed within a base of the mobile computing device. method.

2. 2. The method of claim 1 , wherein the touch sensor data includes touch location data, and generating the adjusted touch sensor data includes generating adjusted touch location data based on the touch location data, and the adjusted touch sensor data includes the adjusted touch location data.

3. 3. The method of claim 1, wherein the touch sensor data includes touch intensity data, and generating the adjusted touch sensor data includes generating adjusted touch intensity data based on the touch intensity data, and the adjusted touch sensor data includes the adjusted touch intensity data.

4. The method of claim 1 , wherein generating the conditioned touch sensor data comprises smoothing the touch sensor data.

5. The method of claim 1, further comprising: receiving, by the lid controller hub, previous touch sensor data from the touch display controller prior to receiving the touch sensor data; receiving a previous display refresh notification from the display subsystem prior to receiving the display refresh notification by the lid controller hub; 5. The method of claim 1, further comprising: a step of: generating the adjusted touch sensor data by averaging the touch sensor data with the previous touch sensor data; and wherein the previous touch sensor data is received before receiving the previous display refresh notification.

6. A method described in any one of claims 1 to 5, further comprising a step in which the lid controller hub notifies the display subsystem to change the refresh rate of the touch display.

7. generating, by the operating system or an application running on the mobile computing device, frame information based on the conditioned touch sensor data; generating a new frame based on the frame information by the one or more processing units; the display subsystem refreshing the touch display to display the new frame; The method of any one of claims 1 to 6, further comprising:

8. Receives touch sensor data from the touch display controller; receiving a display refresh notification from the display subsystem that refreshes the touch display; In response to receiving the display refresh notification, generating adjusted touch sensor data based on the touch sensor data; transmitting the conditioned touch sensor data to a host processing unit; An apparatus comprising circuitry such as The apparatus is used within a lid of a mobile computing device. Device.

9. 9. The device of claim 8, wherein the touch sensor data includes touch location data, and generating the adjusted touch sensor data includes generating adjusted touch location data based on the touch location data, and the adjusted touch sensor data includes the adjusted touch location data.

10. 10. The device of claim 8 or 9, wherein the touch sensor data includes touch intensity data, and generating the adjusted touch sensor data includes generating adjusted touch intensity data based on the touch intensity data, and the adjusted touch sensor data includes the adjusted touch intensity data.

11. 11. The device of claim 8, wherein generating the adjusted touch sensor data comprises averaging the touch sensor data with touch sensor data received prior to a previous display refresh.

12. 12. The device of claim 8, wherein the circuitry further signals the display subsystem to increase the refresh rate of the touch display.

13. 13. The device of claim 8, wherein the circuitry further signals the display subsystem to increase the refresh rate of the touch display up to a maximum refresh rate.

14. 14. The device of claim 12 or 13, wherein the circuitry further signals the display subsystem to increase the refresh rate of the touch display, or to increase the refresh rate of the touch display up to a maximum refresh rate, in response to receiving the touch sensor data.

15. a base including one or more host processing units running an operating system; a lid rotatably coupled to the base, Touch display and A touch display controller; a display subsystem that refreshes the touch display; 1. A circuit arrangement comprising: receiving touch sensor data from the touch display controller; receiving a display refresh notification from the display subsystem; In response to receiving the display refresh notification, generating adjusted touch sensor data based on the touch sensor data; circuitry for transmitting the conditioned touch sensor data to the operating system; a lid including A mobile computing device comprising:

16. 16. The mobile computing device of claim 15, wherein the touch sensor data includes touch location data, and generating the adjusted touch sensor data includes generating adjusted touch location data based on the touch location data, and the adjusted touch sensor data includes the adjusted touch location data.

17. 17. The mobile computing device of claim 15 or 16, wherein the touch sensor data includes touch intensity data, and generating the adjusted touch sensor data includes generating adjusted touch intensity data based on the touch intensity data, and the adjusted touch sensor data includes the adjusted touch intensity data.

18. The circuitry further comprises: receiving previous touch sensor data from the touch display controller prior to receiving the touch sensor data; 18. The mobile computing device of claim 15, wherein a previous display refresh notification is received from the display subsystem prior to receiving the display refresh notification, the previous touch sensor data being received prior to receiving the previous display refresh notification, and generating the adjusted touch sensor data includes averaging the touch sensor data with the previous touch sensor data.

19. 19. The mobile computing device of claim 15, wherein the circuitry further signals the display subsystem to change a refresh rate of the touch display.

20. 20. The mobile computing device of claim 15, wherein the circuitry further signals the display subsystem to change a refresh rate of the touch sensitive display in response to receiving the touch sensor data.

21. 21. The mobile computing device of claim 15, wherein the display subsystem includes a timing controller that causes video data received from the one or more host processing units to be displayed on the touch display.

22. 22. The mobile computing device of claim 15, wherein the one or more host processing units execute applications, and the operating system provides the conditioned touch sensor data to the applications.

23. The operating system or the application generates frame information based on the adjusted touch sensor data, the one or more host processing units generates a new frame based on the frame information, and the display subsystem further comprises: receiving video data from the base; 23. The mobile computing device of claim 22, wherein the touch display is refreshed with the new frame.

24. a base including one or more host processing units running an operating system; a lid rotatably coupled to the base, Touch display and A touch display controller; a display subsystem that refreshes the touch display; touch sensor data conditioning means for conditioning touch sensor data received from the touch display controller and transmitting the conditioned touch sensor data to the operating system in synchronization with a refresh rate of the touch display; a lid including A mobile computing device comprising:

25. the one or more host processing units execute an application, the operating system or the application generates frame information based on the adjusted touch sensor data, the operating system generates a new frame based on the frame information, and the display subsystem further comprises: receiving video data from the base; 25. The mobile computing device of claim 24, wherein the touch display is refreshed with the new frame.

Images