Methods, apparatuses, and systems for 3D integrated heterogeneous micro-display system architecture

Abstract

The present disclosure relates to methods, devices, and systems for a 3D integrated heterogeneous micro-display system architecture. A method of displaying an image at an electronic device having a display that includes an array of pixels and a plurality of compute tiles stacked with the array of pixels. The method includes receiving image data corresponding to a frame to be displayed. The method also includes dividing the image data into sets of tile data corresponding to different portions of the array of pixels. The method further includes concurrently processing the sets of tile data by the plurality of compute tiles. Concurrently processing the sets of tile data includes, for each tile data in the sets of tile data: processing the tile data to generate processed tile data; and driving a corresponding subset of the array of pixels based on the processed tile data.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 792,231, filed April 21, 2025, entitled “Methods, Devices, and Systems for 3D-Integrated Heterogeneous Microdisplay System Architecture,” and U.S. Application No. 19 / 647,847, filed April 14, 2026, which are incorporated herein by reference in their entirety. Technical Field

[0002] This disclosure generally relates to three-dimensional display architectures, including but not limited to integrated heterogeneous microdisplay system architectures. Background Technology

[0003] Display backplane silicon is one of the most challenging design points in optimizing display form factors, resolution, power consumption, and pixel performance. Specifically, due to the large amount of data communication between the system-on-chip (SoC) and the display, data transfer accounts for a significant portion of total power consumption in display pipelines operating at high frame rates (e.g., approximately 90% in some systems).

[0004] Therefore, it is necessary to address one or more of the challenges mentioned above. The following is a brief overview of the solutions to the problems mentioned above. Summary of the Invention

[0005] Several embodiments described herein relate to three-dimensional (3D) integration techniques for display systems, such as microdisplays for augmented reality glasses. The display system may include vertically stacked dedicated layers: a digital chip (or computing chip) at the bottom performing pixel processing, an analog chip in the middle driving the pixels, and a display chip at the top containing the pixel array. By vertically stacking these layers instead of arranging components side-by-side, peripheral circuitry and input / output logic can be repositioned below the display, reducing bezel area and achieving a more compact form factor. In some embodiments, the digital chip includes an array of computing tiles, where each computing tile is spatially proximate to a corresponding portion of the pixel array and is responsible for driving that corresponding portion of the pixel array. This local pixel driving approach can reduce off-chip data transfer between the external on-chip system and the display, which can help reduce power consumption. In some embodiments, a tag-based pulse width modulation mechanism is employed to reduce redundant memory reads during pixel illumination, thereby achieving energy savings during pixel memory accesses. In some embodiments, a lightweight pixel-shifting architecture based on a single-bit multiplexer array enables a display to track real-time eye movements at subframe intervals without incurring the energy and latency costs of copying complete pixel data between computation tiles. These characteristics combined offer advantages in power efficiency, system performance, and compact form factor for integrated heterogeneous microdisplay system architectures.

[0006] In one aspect, this paper discloses a method for displaying an image at an electronic device having a display comprising a pixel array and a plurality of computational tiles stacked with the pixel array. The method includes receiving image data corresponding to a frame to be displayed. The method further includes partitioning the image data into a tile dataset corresponding to different portions of the pixel array. The method also includes processing the tile dataset concurrently through the plurality of computational tiles. Concurrently processing the tile dataset includes: processing the tile data for each tile in the tile dataset to generate processed tile data, and driving a corresponding subset of the pixel array based on the processed tile data.

[0007] On the other hand, a circuit package for a display used to drive an electronic device includes a digital chip, an analog chip vertically stacked on top of the digital chip, and a display chip vertically stacked on top of the analog chip, the display chip including a pixel array. The digital chip is electrically coupled to the analog chip via a through-silicon via (TSV), and the digital chip is configured to drive the analog chip. The analog chip is heterogeneously integrated to the display chip via hybrid bonding, and the analog chip is configured to drive the pixel array of the display chip.

[0008] Instructions for performing the methods and operations described herein can be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may be contained in a single electronic device or distributed across multiple electronic devices in a system (computing system). A non-exhaustive list of electronic devices that can perform the methods and operations described herein individually or in combination (e.g., a system) includes extended-reality (XR) headsets / glasses (e.g., mixed-reality (MR) headsets or augmented-reality (AR) glasses, as two examples), wrist-worn devices, intermediate processing devices, textile-based smart clothing, etc. For example, instructions may be stored on an AR pair of glasses, or on a combination of an AR pair of glasses and an associated input device (e.g., a wrist-worn device), such that instructions for detecting input operations can be executed at the input device, and instructions for changing the displayed user interface in response to these input operations can be executed at the AR glasses. The devices and systems described herein can be configured for use in conjunction with methods and operations for providing XR experiences. Methods and operations used to provide XR experiences can be stored on non-transitory computer-readable storage media.

[0009] The devices and / or systems described herein can be configured to include instructions that cause the execution of methods and operations associated with: the presentation of an XR head-mounted viewer and / or interaction with an XR head-mounted viewer. These methods and operations can be stored on a non-transitory computer-readable storage medium of the device or system. It should also be noted that the devices and systems described herein can be part of a larger overall system comprising multiple devices. A non-exhaustive list of electronic devices (which may individually or in combination (e.g., systems) include instructions that cause the execution of methods and operations associated with: the presentation of an XR experience and / or interaction with an XR experience) includes: extended reality head-mounted viewers (e.g., MR head-mounted viewers or a pair of AR glasses, as two examples), wrist-worn devices, intermediate processing devices, textile-based smart clothing, etc. For example, when describing an XR head-mounted viewer, it can be understood that the XR head-mounted viewer can communicate with one or more other devices (e.g., wrist-worn devices, servers, intermediate processing devices), which together can include instructions for performing methods and operations associated with: presentation to and / or interaction with an extended reality system (i.e., the XR head-mounted viewer will be part of a system including one or more additional devices). Various combinations with different associated devices are envisioned, but for brevity, they will not be elaborated upon.

[0010] The features and advantages described in this specification are not necessarily all-encompassing; in particular, certain additional features and advantages will be apparent to those skilled in the art from the accompanying drawings, specification, and claims. Furthermore, it should be noted that the terminology used in this specification has been chosen primarily for readability and guidance purposes.

[0011] Having outlined the examples above, a brief description of the accompanying drawings will now be presented. Attached Figure Description

[0012] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, in which similar reference numerals indicate corresponding parts in all drawings.

[0013] Figure 1A A comparison between 2D and 3D integrated graphics and display pipelines according to some embodiments is shown.

[0014] Figure 1B Cross-sectional views of 2D and 3D integration of displays according to some embodiments are shown.

[0015] Figure 2 A high-level system overview of a 3D stacked microdisplay system according to some embodiments is shown.

[0016] Figure 3 A top-level schematic diagram of a digital die according to some embodiments is shown.

[0017] Figure 4 A block diagram of a computed tile is shown according to some embodiments.

[0018] Figure 5 Design space exploration of tiled static random-access memory (SRAM) is illustrated according to some embodiments.

[0019] Figure 6A Pulse Width Modulation (PWM) operation for pixel rows is illustrated according to some embodiments.

[0020] Figure 6B The illustration shows a tag-based PWM operation for a pixel row according to some embodiments.

[0021] Figure 7 An overview of a pixel shifting mechanism according to some embodiments is shown.

[0022] Figure 8 A single-bit pixel shifting circuit based on a lightweight multiplexer array in a computational tile is shown according to some embodiments.

[0023] Figure 9A , Figure 9B , Figure 9C-1 and Figure 9C-2 Example MR and AR systems according to some embodiments are shown.

[0024] By convention, the features shown in the accompanying drawings may not be drawn to scale. Accordingly, the dimensions of the features may be arbitrarily scaled up or down for clarity. Additionally, some of these drawings may not depict all parts of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0025] In traditional display systems, the system-on-a-chip (SoC) performs graphics processing and sends image data to individual display components via high-bandwidth connections. This approach can result in significant power consumption because a large amount of pixel data is transferred between the SoC and the display for each frame. Furthermore, traditional displays typically include peripheral circuitry (e.g., analog pixel drivers and input / output logic) arranged around the edges of the pixel array, leading to large bezel areas and a large overall device size.

[0026] The embodiments described herein employ a 3D stacking approach to address these challenges. In some embodiments, the display system comprises three stacked layers: a digital chip (also known as a computing chip) that performs pixel processing, an analog chip that drives the pixels, and a display chip containing the pixel array. The digital chip performs pixel processing tasks, the analog chip converts digital signals into analog driving signals for the pixels, and the display chip contains the actual light-emitting elements that form the visible image. This heterogeneous 3D stacking architecture allows for the selection of the most suitable transistor technology for each layer. In some embodiments, the display chip can employ cost-effective and small-area technologies, such as those suitable for micro-LEDs or liquid crystal on silicon (LCoS) pixel arrays. In some embodiments, the analog chip can use high-voltage transistors to support the voltage requirements for pixel driving. In some embodiments, the digital chip can use miniaturized process nodes to achieve high-performance processing and high-density memory storage, such as high-density SRAM for storing pixel values. By stacking these layers instead of arranging components side-by-side, the peripheral circuitry and input / output logic can be repositioned below the display, which can reduce bezel area and achieve a more compact form factor suitable for applications such as augmented reality glasses.

[0027] In some embodiments, the digital chip includes an array of computational tiles, wherein each computational tile is spatially aligned with a corresponding portion of a pixel array located above it and is responsible for driving that corresponding portion of the pixel array. This localization approach means that pixel data can be stored and processed near the location where it is displayed, rather than transferring pixel data from a distant SoC for each frame. Therefore, off-chip data transfer between the external SoC and the display can be reduced, which can help reduce power consumption.

[0028] To further reduce power consumption, some implementations employ a tag-based pulse width modulation (PWM) mechanism. In a typical PWM method, the pixel value is read from memory each time the PWM counter increments to determine when to switch pixel illumination. The tag-based method, however, first checks a smaller tag buffer to determine whether pixels in a row need to change state at the current counter value. If no pixel needs to be changed, main memory reads can be skipped entirely, resulting in significant power savings during pixel memory accesses.

[0029] Some embodiments also include a lightweight pixel-shifting architecture that enables the display to track real-time eye movements. In augmented reality and virtual reality applications, a user's eyes may move rapidly between display refresh cycles. To reduce motion blur and improve perceived image quality, display systems may change the position of the displayed image multiple times within a single frame cycle. Unlike copying the complete multi-bit pixel brightness value between computed tiles with each shift (which would consume significant energy), the pixel-shifting architecture operates based on a single-bit on / off state, indicating whether each pixel is illuminated at any given moment. This approach enables energy-efficient positional pixel shifting without the energy and latency costs required to transfer complete pixel data between blocks.

[0030] This document describes numerous details to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, some embodiments may be practiced without many of these specific details, and the scope of the claims is limited only to those features and aspects specifically recited in the claims. Furthermore, well-known processes, components, and materials are not necessarily described in full detail to avoid obscuring relevant aspects of the embodiments described herein.

[0031] Overview Embodiments of this disclosure may include various types of extended reality (XR) systems, or a combination of various types of XR systems, such as MR and AR systems. As described herein, MR and AR are any overlay of functional and / or sensorily detectable presentations provided by MR and AR systems in a user's physical environment. Such MR may include and / or represent virtual reality (VR) and VR, wherein at least some aspects of the surrounding environment are reconstructed in the virtual environment (e.g., a virtual reconstruction of physical objects in the physical environment to avoid collisions between the user and physical objects in the surrounding physical environment). In the case of MR, the surrounding environment presented through a display is acquired by one or more sensors configured to acquire the surrounding environment (e.g., camera sensors, time-of-flight (ToF) sensors). While the wearer of an MR headset can see full detail of the surrounding environment, what they see is a reconstruction of the environment reproduced using data from one or more sensors (i.e., the user cannot directly see physical objects). MR headsets may also forgo displaying reconstructions of objects in the physical environment, thereby providing the user with a fully VR experience. On the other hand, AR systems provide an experience in which information is provided, for example, by using waveguides and by directly viewing at least some of the surrounding environment through one or more transparent or semi-transparent waveguides and / or lenses of AR glasses. Throughout this application, the term "extended reality (XR)" is used as a general term encompassing both AR and MR. Furthermore, this application sometimes uses the term head-worn device or head-mounted viewer as a general term encompassing XR head-mounted viewers such as AR glasses and MR head-mounted viewers.

[0032] As described above, the MR environment described herein may include, but is not limited to, non-immersive VR environments, semi-immersive VR environments, and fully immersive VR environments. As described above, AR environments may include marker-based AR environments, markerless AR environments, location-based AR environments, and projection-based AR environments. The above description is not exhaustive and allows intentional ambient lighting to pass through to reach the user; any other environment will fall within the AR range, but not the MR range.

[0033] AR and MR content can include video, audio, haptic events, sensory events, or some combination thereof, any of which can be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional (3D) effect for the viewer). Furthermore, AR and MR can be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in an AR or MR environment and / or otherwise for use in an AR or MR environment (e.g., to perform activities in an AR or MR environment).

[0034] The interactions with these AR and MR environments described in this article can occur in several different modes, and the resulting outputs can also occur across multiple different modes. In one example of an AR or MR system, a user can perform an air swipe gesture to skip a song via an application programming interface (API) provided by a song that is played, for example, at a home speaker.

[0035] As described herein, gestures can include air gestures, surface contact gestures, and / or other gestures that can be detected and determined based on: single-hand movement (e.g., a single-hand gesture performed by a user's hand, detected by one or more sensors of a wearable device (e.g., an electromyography (EMG) and / or an inertial measurement unit (IMU) of a wrist wearable device and / or one or more sensors included in a smart textile wearable device), and / or the single-hand gesture is detected via image data acquired by an imaging device of the wearable device (e.g., a camera of a head wearable device, an external tracking camera positioned in the surrounding environment). "Air" generally includes gestures in which a portion of the user's hand does not contact a surface, object, or electronic device (e.g., a head wearable device or other communication-coupled device, such as a wrist wearable device); in other words, the gesture is performed in an open space in 3D space without contacting a surface, object, or electronic device. More generally, surface contact gestures (contact at a surface, object, part of the user's body, or electronic device) are also considered, in which contact is detected at the surface. Touch (or intention to touch) (e.g., a single or two-finger tap on a table, on a user's hand or another finger, on a user's leg, a sofa, or a steering wheel). The various gestures disclosed herein can be detected using image data and / or sensor data (e.g., neuromuscular signals sensed by one or more biopotential sensors (e.g., EMG sensors) or other types of data from other sensors (e.g., proximity sensors, time-of-flight (ToF) sensors, IMU sensors, capacitive sensors, strain sensors), which are detected by wearable devices worn by the user and / or other electronic devices owned by the user (e.g., smartphones, laptops, imaging devices, intermediate devices, and / or other devices described herein).

[0036] The input modes described above can be different and depend on the user experience. For example, in interactions using a wrist-worn wearable device, a user can provide input using air gestures or surface contact gestures detected by the neuromuscular signal sensors of the wrist-worn wearable device. In the absence of a wrist-worn wearable device, alternative and fully interchangeable input modes can be used, such as one or more cameras located on a head-mounted viewer / glasses or elsewhere to detect air gestures, or surface contact gestures or inputs at an intermediate processing device (e.g., via physical input components such as buttons and touchpads). These different input modes can be interchanged based on the desired user experience, portability, and / or product feature set (e.g., low-cost products may not include a handheld tracking camera).

[0037] When the inputs are diverse, the outputs generated from those inputs are also diverse. For example, an air gesture input detected by a camera on a head-mounted wearable device can cause the output to occur at the head-mounted wearable device or to control another electronic device different from the head-mounted wearable device. In another example, input detected using data from a neuromuscular signal sensor can also cause the output to occur at the head-mounted wearable device or to control another electronic device different from the head-mounted wearable device. While only a few examples have been described above, those skilled in the art will understand that different input patterns and different output patterns in response to those inputs are interchangeable.

[0038] The specific operations described above may occur depending on the specific hardware. The devices described are not limiting, and features may be removed from these devices or additional features may be added to them. Different devices may include one or more similar hardware components. For the sake of brevity, similar devices and components are described herein. Any differences between the devices and components will be described in their respective sections below.

[0039] As described herein, a processor (e.g., a central processing unit (CPU) or microcontroller unit (MCU)) is an electronic component responsible for executing instructions and controlling the operation of electronic devices (e.g., wrist-worn devices, head-worn devices, handheld intermediary processing devices (HIPDs), textile-based smart clothing, or other computer systems). Various types of processors exist, which may be used interchangeably or are specifically required by the embodiments described herein. For example, the processor can be: (i) a general-purpose processor designed to perform a wide range of tasks, such as running software applications, managing operating systems, and performing arithmetic and logical operations; (ii) a microcontroller designed for specific tasks, such as controlling electronic devices, sensors, and motors; (iii) a graphics processing unit (GPU) designed to accelerate the creation and rendering of images, videos, and animations (e.g., VR animations, such as 3D modeling); (iv) a field-programmable gate array (FPGA) that can be programmed and reconfigured post-manufacturing and / or customized to perform specific tasks, such as signal processing, encryption, and machine learning; or (v) a digital signal processor (DSP) designed to perform mathematical operations on signals (e.g., audio, video, and radio waves). Those skilled in the art will understand that one or more processors of one or more electronic devices can be used in the various embodiments described herein.

[0040] As described herein, a controller is an electronic component that manages and coordinates the operation of other components within an electronic device (e.g., controlling inputs, processing data, and / or generating outputs). Examples of controllers may include: (i) microcontrollers, which include small, low-power controllers commonly used in embedded systems and Internet of Things (IoT) devices; (ii) programmable logic controllers (PLCs), which can be configured for use in industrial automation systems to control and monitor manufacturing processes; (iii) system-on-a-chip (SoC) controllers, which integrate multiple components such as processors, memory, input / output (I / O) interfaces, and other peripherals into a single chip; and / or (iv) DSPs. As described herein, a graphics module is a component or software module designed to handle graphics computations and / or graphical processes, and such a graphics module may include hardware modules and / or software modules.

[0041] As described herein, memory refers to electronic components in a computer or electronic device that store data and instructions for access and operation by a processor. The devices described herein may include volatile and non-volatile memory. Examples of memory may include: (i) random access memory (RAM) (e.g., DRAM, SRAM, DDR RAM, or other random access solid-state memory devices) configured to temporarily store data and instructions; (ii) read-only memory (ROM) configured to permanently store data and instructions (e.g., in one or more portions of system firmware and / or bootloader); (iii) flash memory, disk storage devices, optical disk storage devices, and other non-volatile solid-state storage devices that may be configured to store data in electronic devices (e.g., universal serial bus (USB) drives, memory cards, and / or solid-state drives (SSDs)); and (iv) cache memory configured to temporarily store frequently accessed data and instructions. As described herein, memory may include structured data (e.g., SQL databases, MongoDB databases, GraphQL data, or JSON data). Other examples of storage may include: (i) profile data, which includes user account data, user settings and / or other user data stored by the user; (ii) sensor data detected by one or more sensors and / or otherwise acquired; (iii) media content data, which includes stored image data, audio data and documents, etc.; (iv) application data, which may include data collected and / or otherwise acquired and stored during use of the application; and / or (v) any other types of data described herein.

[0042] As described herein, the power system of an electronic device is configured to convert input power into a form usable for operating the device. The power system may include various components, including: (i) a power source, which may be an alternating current (AC) adapter power source or a direct current (DC) adapter power source; (ii) a charger input, which may be configured to use wired and / or wireless connections (which may be part of a peripheral interface, such as USB, microUSB, near-field magnetic coupling, magnetic induction and magnetic resonance charging, and / or radio frequency (RF) charging); (iii) a power management integrated circuit configured to distribute power to various components of the device and ensure that the device operates within safety limits (e.g., regulating voltage, controlling current, and / or managing heat dissipation); and / or (iv) a battery configured to store power to provide usable power to components of one or more electronic devices.

[0043] As described herein, a peripheral interface is an electronic component (e.g., an electronic component of an electronic device) that allows the electronic device to communicate with other devices or peripheral devices and can provide means for inputting and outputting data and signals. Examples of peripheral interfaces may include: (i) USB and / or micro USB interfaces configured to connect devices to an electronic device; (ii) Bluetooth interfaces configured to allow devices to communicate with each other, including Bluetooth Low Energy (BLE); (iii) Near-Field Communication (NFC) interfaces configured as short-range wireless interfaces for operations such as access control; (iv) Pogo pins, which may be small, spring-loaded pins configured to provide a charging interface; (v) Wireless charging interfaces; (vi) Global Positioning System (GPS) interfaces; (vii) Wi-Fi interfaces for providing connectivity between a device and a wireless network; and (viii) Sensor interfaces.

[0044] As described herein, a sensor is an electronic component (e.g., an electronic component in an electronic device (e.g., a wearable device) and / or an electronic component that otherwise communicates electronically with the electronic device). Examples of sensors may include: (i) imaging sensors for collecting imaging data (e.g., including one or more cameras arranged on a corresponding electronic device, such as a Simultaneous Localization and Mapping (SLAM) camera); (ii) biopotential signal sensors; (iii) IMUs for detecting, for example, changes in angular velocity, force, magnetic field, and / or acceleration; (iv) heart rate sensors for measuring a user's heart rate; (v) peripheral oxygen saturation (SpO2) sensors for measuring a user's blood oxygen saturation and / or other biometric data; (vi) capacitive sensors for detecting potential changes at a part of the user's body (e.g., a sensor-skin interface) and / or near other devices or objects; (vii) sensors for detecting certain inputs (e.g., capacitive and force sensors); and (viii) light sensors (e.g., TFO sensors, infrared sensors, or visible light sensors, etc.), and / or sensors for sensing data from the user or the user's environment. As described herein, biopotential signal sensing components are devices for measuring electrical activity within the body (e.g., biopotential signal sensors). Some types of bioelectric potential signal sensors include: (i) electroencephalography (EEG) sensors, which are configured to measure electrical activity in the brain to diagnose neurological diseases; (ii) electrocardiography (ECG or EKG) sensors, which are configured to measure electrical activity in the heart to diagnose heart problems; (iii) EMG sensors, which are configured to measure electrical activity in muscles and diagnose neuromuscular diseases; and (iv) electrooculography (EOG) sensors, which are configured to measure electrical activity in eye muscles to detect eye movements and diagnose eye diseases.

[0045] As described herein, applications (e.g., software) stored in the memory of an electronic device include instructions stored in the memory. Examples of such applications include: (i) games; (ii) word processors; (iii) messaging applications; (iv) media streaming applications; (v) financial applications; (vi) calendars; (vii) clocks; (viii) web browsers; (ix) social media applications; (x) camera applications; (xi) web-based applications; (xii) health applications; (xiii) AR and MR applications; and / or (xiv) any other applications that may be stored in memory. These applications may operate in conjunction with one or more components of a data, and / or device or communication-coupled device to perform one or more operations and / or functions.

[0046] As described herein, a communication interface module may include hardware and / or software capable of data communication using any of the following protocols: custom wireless protocols or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, or MiWi); custom wired protocols or standard wired protocols (e.g., Ethernet or HomePlug); and / or any other suitable communication protocol (including those not yet developed as of the date of this submission). A communication interface is a mechanism that enables different systems or devices to exchange information and data with each other, including hardware, software, or a combination of both. For example, a communication interface may refer to a physical connector and / or port on a device that enables communication with other devices (e.g., USB, Ethernet, HDMI, or Bluetooth). A communication interface may refer to a software layer that enables different software programs to communicate with each other (e.g., APIs and protocols such as HTTP and TCP / IP).

[0047] As described herein, a graphics module is a component or software module designed to handle graphics operations and / or graphical processes, and the graphics module may include hardware modules and / or software modules.

[0048] As described herein, a nontransitory computer-readable storage medium is a physical device or storage medium that can be used to store electronic data in a nontransitory form (e.g., such that the data is permanently stored until it is intentionally deleted or modified).

[0049] Display pipeline Figure 1A A comparison between 2D and 3D integrated graphics and display pipelines according to some embodiments is shown. Figure 1A The upper part depicts a conventional 2D integration method 100, while the lower part depicts a 3D integrated graphics and display pipeline 150. Figure 1A In the 2D integration method 100 shown in the upper part, a SoC 102 with graphics and display pipelines is connected to a separate display component via a high-bandwidth connection 104. The display component 106 includes a central pixel array region 108 surrounded by multiple peripheral regions 110. The peripheral regions 110 house analog pixel drivers and associated circuitry controlling the pixel array region 108. In this conventional approach, a significant portion of the display's footprint (e.g., approximately 58%) is occupied by the peripheral regions 110 surrounding the pixel array (e.g., peripheral circuitry).

[0050] exist Figure 1A In the 3D integrated graphics and display pipeline 150 shown in the lower part, the 3D stacked microdisplay system 156 includes multiple vertically stacked layers. A SoC 152 with graphics and display pipeline is connected to the 3D stacked microdisplay system 156 via a low-bandwidth connection 154. The 3D stacked microdisplay system 156 includes at least three vertically arranged layers. At the bottom is a first layer 158 containing pixel states and display pipeline functionality. In the middle is a layer containing pixel drivers 160 (e.g., analog drivers). At the very top is a pixel array layer 162, which represents the display pixels. The 3D stacked microdisplay system 156 demonstrates how 3D integration technology enables the repositioning of peripherals and I / O logic below the display, thereby reducing bezel area and achieving a more compact form factor compared to traditional 2D integration methods. The low-bandwidth connection 154 between the SoC and the 3D stacked display components indicates reduced off-chip data transfer, which can help reduce power consumption in the display pipeline.

[0051] Although Figure 1A A three-layer configuration is shown, but other embodiments may include additional layers. For example, a four-layer configuration may distribute digital processing functions across two distinct digital layers, or a dedicated memory layer may be included between the digital and analog layers. In some embodiments, analog driver functions and the display pixel array are combined into a single layer, forming a two-layer configuration. The number and arrangement of layers can be selected based on manufacturing considerations, performance requirements, or form factor constraints specific to the application.

[0052] Interconnects between multiple layers can be achieved using various interconnect technologies. In some embodiments, through-silicon vias (TSVs) provide electrical coupling between adjacent layers, enabling high-density vertical interconnects with relatively low parasitic capacitance. Hybrid bonding is a wafer-to-wafer bonding technique that can be used to achieve even higher interconnect densities between multiple layers, particularly between analog driver layers and display layers. Other interconnect methods (e.g., microbumps, copper pillars, or direct metal bonding) may also be used depending on spacing requirements and manufacturing process compatibility. In some embodiments, different interconnect technologies are used between different pairs of layers within the same stack.

[0053] Depending on the implementation method, the functional distribution of each layer may differ. Although Figure 1A The pixel states and display pipeline functions in the first layer 158 are described, but other embodiments may allocate these functions in different ways. For example, certain display pipeline operations (e.g., gamma correction, scaling, or color space conversion) may be performed in an external SoC 152 rather than in a 3D stacked component. In some embodiments, additional processing functions (e.g., image compression, decompression, or local dimming control) are integrated into the digital layer. In some embodiments, the digital layer includes calibration data storage and pixel compensation logic to support per-pixel brightness and color correction.

[0054] In some embodiments, the 3D integrated graphics and display pipeline 150 method can reduce peripheral overhead (e.g., to about 14% of the footprint). Figure 2 As shown, localizing pixel driving to both the digital and analog dies 220 minimizes off-chip interconnect and communication costs. For example, up to approximately 11% of system-level power consumption can be saved simply by minimizing off-chip interconnect traffic; while approximately 50% of system-level power consumption can be saved through additional architecture and circuit development. Figure 2As shown, the digital die 202 at the miniaturization process node can provide opportunities to increase pixel depth (e.g., from 6 bits to 8 bits or higher) stored in dense SRAM memory and achieve higher frame rates (e.g., up to about 2x) through high-performance dedicated circuitry. The analog die 220 can be used to implement a more powerful analog driver with area-intensive functions (e.g., current regulation and pixel measurement). In some embodiments, current regulation may include circuitry configured to adjust the drive current level of individual pixels to compensate for variations in pixel characteristics (e.g., differences in luminous efficiency or threshold voltage across the pixel array). In some embodiments, pixel measurement may include sensing circuitry configured to measure pixel output characteristics (e.g., current or luminance levels) that can be used for calibration purposes to improve the uniformity and accuracy of pixel illumination. These characteristics, combined with the higher pixel depth and better pixel calibration storage in the digital die 202, can potentially improve overall pixel performance. Functional decoupling can offload time-critical functions to the 3D stacked microdisplay system, thereby shortening the overall graphics-to-display main path (e.g., cross-die traffic can be reduced by up to about 50%).

[0055] Figure 1B A cross-sectional view is shown in a 2D integration for a display and a 3D stacked microdisplay system 156 according to some embodiments. Figure 1B It also presents a comparison between 2D and 3D ensemble methods. Figure 1B In the 2D integration method 100 depicted, a central array region representing a pixel array region 108 is surrounded by multiple peripheral regions 110. In some embodiments, the peripheral regions 110 include peripheral circuitry, such as analog pixel drivers and associated input / output logic. A cross-sectional view of the 2D integration configuration shows the pixel array region 108 situated on a substrate, with the peripheral regions 110 extending laterally beyond the array boundaries, resulting in a larger overall footprint.

[0056] Figure 1BA 3D integrated graphics and display pipeline 150 method is also illustrated. In this configuration, the pixel array region 108 occupies a larger proportion of the visible surface area, where the peripheral region 110 is repositioned below the pixel array region 108, rather than surrounding it. A cross-sectional view of the 3D integrated graphics and display pipeline 150 shows a vertically stacked arrangement where the pixel array layer 162 sits above a combined array containing the pixel driver 160 and a peripheral layer. This stacked configuration demonstrates how peripheral circuitry and input / output logic can be placed below the display, resulting in a smaller area and a more compact form factor. In some embodiments, the 3D stacked microdisplay system 156 utilizes this vertical integration method to maximize pixel density on the display die by moving the peripheral region 110 to a layer below the pixel array layer 162. In some embodiments, this architectural approach may be able to reduce bezel area compared to conventional 2D integration methods where the peripheral region 110 occupies a large amount of lateral space around the pixel array region 108.

[0057] Although Figure 1B A specific cross-sectional arrangement of the pixel array layer 162 above the pixel driver 160 is shown, but other layer arrangements are possible. In some embodiments, an additional intermediate layer, such as a color filter layer, an optical management layer, or an additional wiring layer, is included between the pixel array and the driver circuitry. The relative thicknesses of the layers may vary depending on manufacturing processes and performance requirements. For example, in some embodiments, the digital processing layer may be thinner than the analog driver layer, while in other embodiments, the layers have similar thicknesses.

[0058] These types of peripheral circuitry that can be relocated below the display may include analog pixel drivers, digital control logic, input / output interfaces, power distribution networks, clock generation and distribution circuitry, and data serialization and deserialization circuitry. In some embodiments, only a subset of these peripheral circuits is relocated below the display, while other circuitry remains at the edge of the pixel array or implemented in external components. The selection of which circuitry to relocate may be based on factors such as thermal requirements, signal integrity considerations, or manufacturing yield optimization.

[0059] Pixel array technologies compatible with this vertical integration approach can include micro-LED arrays, liquid crystal on silicon (LCoS) arrays, organic light-emitting diode (OLED) arrays, and other emitting or reflecting pixel technologies. In some embodiments, the pixel array includes a sub-pixel arrangement for color display (e.g., RGB sub-pixels or RGBW sub-pixels). Pixel pitch and array size can vary depending on the target application, ranging from high-density arrays for near-eye displays to low-density arrays for other display applications.

[0060] The reduction in bezel area achievable through vertical integration may vary depending on the specific implementation. Factors affecting bezel reduction include the size of the repositioned peripheral circuitry, the achievable interconnect density between layers, and any remaining edge circuitry that may not be repositioned below the pixel array. In some embodiments, the bezel area is reduced by repositioning almost all of the peripheral circuitry below the display. In other embodiments, the bezel area can be partially reduced by repositioning only selected peripheral circuitry.

[0061] The compact form factor achieved through this vertical integration approach can benefit a variety of applications beyond augmented reality glasses. In some embodiments, the 3D stacked display architecture is used in virtual reality headsets, smart glasses, head-up displays, camera viewfinders, medical imaging devices, or other applications where display size and weight are primary considerations. Reducing the bezel area enables edge-to-edge display designs, tiled display configurations, or integration into space-constrained enclosures.

[0062] Figure 2 A high-level system overview of a 3D stacked microdisplay system 200 according to some embodiments is shown. The 3D stacked microdisplay system 200 provides Figure 1A The 3D integrated graphics and display pipeline 150 described herein provides a detailed system overview. The 3D stacked microdisplay system 200 includes a three-layer vertically stacked structure comprising a bottom digital die 202 (e.g., a computing die housing computing tiles 212), a middle analog die 220 (e.g., an analog pixel driver), and a top display die 240 (e.g., a pixel array).

[0063] Digital die 202 (also referred to as a computing die or digital chip) houses one or more computing tiles 212 and includes several subsystems. A graphics and display subsystem 214 (e.g., via a C / D-PHY Rx 306 connection and a MIPI 304 interface in a bump and wire bonding configuration) receives input display data from an external source and processes the data to distribute it to the computing tiles. A tile controller 206 manages the distribution of pixel data to the computing tiles, thereby streaming data in a tile-by-tile row manner. A microcontroller unit (MCU) 208 with low-speed input / output (LSIO) capabilities handles system-level control tasks (e.g., startup and housekeeping). An interconnect structure 204 (e.g., a network-on-chip (NoC)) interconnects the subsystems and provides communication paths within digital die 202, enabling the computing tiles 212 to be integrated into the underlying subsystems. A crystal oscillator (XTAL) 216 with a phase-locked loop (PLL) generates a clock signal to synchronize operation on the digital die 202. Tile interconnect 210 provides communication between the tile controller 206 on the digital die 202 and the driver tiles 222 on the analog die 220, thereby enabling coordinated pixel driving across stacked layers. In some embodiments, LSIO and a reference clock are provided via a bump configuration on the digital die 202 and a wire bonding configuration on the analog die 220.

[0064] Analog die 220 includes analog pixel drivers disposed in driver tiles 222. Analog die 220 is located between digital die 202 and display die 240. Tile interconnect 210 provides communication between tile controller 206 on digital die 202 and driver tiles 222 on analog die 220. In some embodiments, tile interconnect 210 implements a 96-bit data bus for streaming data from tile controller 206 to compute tile rows, thereby enabling efficient data transfer along the write path. Digital die 202 and analog die 220 are connected via through-silicon vias (TSVs) 224, which provide electrical coupling between compute tiles 212 on digital die 202 and corresponding driver tiles 222 on analog die 220. In some embodiments, the number and location of TSVs 224 can be optimized to minimize their footprint while still providing the bandwidth required for data transfer between compute tiles 212 and driver tiles 222. In some embodiments, the pixel array is grouped into pixel tiles to allow sharing of common interfaces for bus, power, and clock, which can save power and area. In some embodiments, the pixel tiles on analog die 220 can be layout-planned to maximize the usable area of ​​the analog drivers within TSV 224 and driver tiles 222. Analog die 220 and display die 240 are connected via hybrid bonding technology 244, which provides high-density interconnects between driver tiles 222 and pixel array 242. The display die 240, located at the top of the stack, contains pixel array 242, which forms the visible display surface.

[0065] Analog die 220 includes analog pixel drivers disposed in driver tiles 222, wherein each driver tile 222 is configured to convert received digital pixel states from a corresponding compute tile 212 into analog drive signals to control pixel illumination. Analog die 220 is located between digital die 202 and display die 240 as an intermediate layer bridging the digital processing performed by compute tiles 212 and the physical pixel actuation at display die 240. Tile interconnect 210 provides communication between tile controller 206 on digital die 202 and driver tiles 222 on analog die 220, enabling tile controller 206 to coordinate and synchronize pixel drive operations on driver tiles 222. Digital die 202 and analog die 220 are connected via through-silicon vias (TSVs) 224, which provide electrical coupling between compute tiles 212 on digital die 202 and corresponding driver tiles 222 on analog die 220. Each compute tile 212 transmits pixel on / off states to its corresponding driver tile 222 via a TSV 224, which in turn converts these states into analog signals that control pixel illumination timing and intensity. The analog die 220 and the display die 240 are connected via a hybrid bonding technique 244, which provides a high-density interconnect between the driver tile 222 and the pixel array 242 on the display die 240. The display die 240, located at the top of the stack, contains the pixel array 242 that forms the visible display surface, where the pixels in the pixel array 242 are driven by analog signals generated by the driver tile 222 in the underlying analog die 220. In this way, the compute tile 212, driver tile 222, and pixel array 242 work together across three vertically stacked layers to achieve local pixel driving, where digital and analog components are separated in multiple heterogeneous layers while maintaining spatial proximity to their respective pixel regions.

[0066] The local pixel-driving architecture of the 3D stacked microdisplay system 200 enables various graphics and display processing functions to be implemented locally within the system. At the microarchitecture level, the miniaturization technology of the digital die 202 is used to customize the computation tiles 212 to locally perform operations such as reprojection, color gamut binding, subframe updates, zooming, gaze, and eye-tracking-based pixel shifting. The results of these operations are transmitted via TSV 224 to the driver tiles 222 to drive the pixel array 242. At the system level, the digital die 202 allows a portion of the graphics pipeline to be offloaded from the external SoC. This portion of the graphics pipeline is integrated into the digital die 202 as a functional block via interconnect structure 204 (e.g., NoC), thereby reducing off-chip bandwidth requirements and eliminating redundant latency in the main packetization or compression and decompression path. In some embodiments, the architectural features of the 3D stacked microdisplay system 200 include: pixel values ​​being locally stored in dense SRAM (e.g., 8-bit or higher depth resolution) within each compute tile 212, thereby enabling each compute tile 212 to independently manage pixel illumination of its corresponding driver tile 222; and a lightweight pixel shifting architecture based on a single-bit multiplexer array that operates on pixel on / off states transmitted via TSV 224, thereby achieving energy-efficient positional pixel shifting without requiring data copying and transmission between compute tiles 212. Energy and latency costs are reduced; a pixel value access mechanism based on SRAM tags reduces redundant SRAM reads by first checking the tag buffer before accessing the main SRAM, thereby achieving significant dynamic energy savings (e.g., more than 4x) during pixel memory accesses used for PWM generation; and a lightweight tile interconnect 210 implements a streaming architecture for tile-to-tile connections that pipes data from tile controller 206 to compute tile 212, eliminating the need for flow control and data congestion. In some embodiments, tile interconnect 210 implements a flat butterfly topology that allows hopping to more distant routers, minimizing latency losses for physically distant tiles. In some embodiments, for each router-to-router data transfer, data is transmitted within a single fast system clock cycle. For example, with 44 tiles in a tile row and hops implemented to connect to every 11 routers, the maximum initial delay for the farthest tile to receive new data can be less than 14 clock cycles (e.g., 3 hops plus 11 neighbor transmissions). For example, at an example 400 MHz system clock frequency, this corresponds to a maximum initial delay of less than approximately 35 nanoseconds.

[0067] Although Figure 2A specific three-layer configuration with digital die 202, analog die 220, and display die 240 is shown, but other configurations are also considered. In some embodiments, digital die 202 includes other functional blocks (e.g., dedicated image processing units, compression and decompression engines, or machine learning accelerators for display optimization) in addition to the functional blocks shown. The interconnect structure 204 can be implemented using various on-chip network topologies, including mesh networks, ring networks, or hierarchical bus structures, depending on the system's bandwidth and latency requirements.

[0068] The tile controller 206 and the computing tiles 212 can be configured in various arrangements. In some embodiments, the number of computing tiles 212 is related to... Figure 2 The arrangements shown vary, with fewer or more tiles used depending on the display resolution and processing requirements. Tile sizes can also vary, with larger tiles covering more pixels but requiring more local memory, or smaller tiles providing finer granularity but requiring more inter-tile communication. In some embodiments, the dimensions of the calculated tiles 212 are non-uniform, with tiles at the edges of the array being smaller than those at the center, to accommodate irregular display shapes or optimize for specific viewing areas.

[0069] The analog die 220 may include various types of driver circuitry depending on the display technology. For micro-LED displays, driver tile 222 may include a current-mode driver with a programmable current level for controlling pixel brightness. For LCoS displays, driver tile 222 may include a voltage-mode driver for controlling liquid crystal orientation. In some embodiments, driver tile 222 includes calibration circuitry for compensating for pixel-to-pixel variations, aging effects, or temperature-dependent characteristics. The analog die 220 may also include power management circuitry, a reference voltage generator, or an analog-to-digital converter for pixel sensing and feedback.

[0070] Display die 240 can support various pixel array configurations. In some embodiments, pixel array 242 includes monochrome pixels for applications where color is provided by external optics or sequential color illumination. Other embodiments include RGB pixel arrangements with independent red, green, and blue subpixels, or RGBW arrangements with added white subpixels for improved efficiency. Pixel pitch can range from less than 5 micrometers for high-density near-eye displays to larger pitches for other applications. In some embodiments, pixel array 242 includes redundant pixels or repair mechanisms (e.g., to improve manufacturing yield).

[0071] The interconnect technologies between multiple layers may differ. Although Figure 2The diagram depicts a TSV 224 between the digital die 202 and the analog die 220, and a hybrid bonding technique 244 between the analog die 220 and the display die 240, but other combinations are also possible. In some embodiments, hybrid bonding is used between all adjacent layers to achieve the highest interconnect density. In other embodiments, TSVs are used throughout the stack to simplify manufacturing. The spacing and density of vertical interconnects can be selected based on the data bandwidth requirements between layers and the manufacturing process capabilities.

[0072] In various implementations, the MCU 208 and the graphics and display subsystem 214 can be configured differently. In some embodiments, the MCU 208 is a simple microcontroller for basic housekeeping tasks, while in other embodiments, the MCU 208 includes additional processing capabilities for runtime calibration, power management, or system diagnostics. The graphics and display subsystem 214 can support a variety of input interfaces in addition to MIPI, such as DisplayPort, High-Definition Multimedia Interface (HDMI), or proprietary high-speed serial interfaces. In some embodiments, the graphics and display subsystem 214 includes frame buffers, scaling engines, or color processing blocks that offload work from an external SoC.

[0073] The 3D stacked microdisplay system 200 can be applied to a variety of applications. In some embodiments, the system is optimized for augmented reality glasses, with an emphasis on low power consumption and a compact form factor. Other embodiments are designed for virtual reality head-mounted displays, where higher resolution and faster refresh rates may be more important. The architecture can also be used in automotive head-up displays, medical imaging systems, industrial inspection equipment, or other applications requiring compact, high-performance displays. The modular nature of the stacked architecture allows for different combinations of digital dies, analog dies, and display dies to be assembled for various application needs.

[0074] Figure 3 A top-level schematic of a digital die 300 according to some embodiments is shown. The digital die 300 includes a graphics and display subsystem 302 located in the upper region. The graphics and display subsystem 302 receives input from external sources via a MIPI interface, which includes a C / D PHY Rx component 306 and a MIPI DSI-2 controller component 308. These interface components receive incoming display data (e.g., image frames from an external SoC) and pass the data to graphics and display function blocks 309 and pipelines within the subsystem. The graphics and display function blocks 309 process the incoming display data and prepare to distribute the display data to compute tiles within the digital die 300. In this way, the graphics and display subsystem serves as the entry point for image data into the digital die 300, thereby bridging the external graphics pipeline with the internal block-based processing architecture.

[0075] The clock and reset block 310 is located adjacent to the graphics and display subsystem 302 and includes a crystal oscillator (XTAL) 312 and a phase-locked loop (PLL) 314. The clock and reset block 310 receives external signals and includes an XTAL 316, a CLK_EXT 318, and a RESET 320. The clock and reset block 310 also generates clock signals (e.g., clock_low-speed I / O (clk_lsio) 322 and clock_system (clk_sys) 324) distributed throughout the digital die 300 to synchronize the operation of the various subsystems. The generated clock signals ensure that the basic subsystem, tile controller, and compute tiles operate in a coordinated manner.

[0076] The basic subsystem 326 occupies a portion of the digital die 300 and includes an on-chip network (NoC) that interconnects multiple components. The basic subsystem 326 includes a microcontroller (MCU) 328 and an associated MCU local memory (LMEM) 330 and a top control and status register (CSR) 332. The MCU 328 handles system-level control tasks such as startup and housekeeping operations. The basic subsystem 326 also includes peripheral interfaces 334 to 346, including an SPI slave device 334 that receives SPI_S input 336, a UART 338 that receives UART input 340 or provides UART output, an SPI master device 342 that provides SPI_M output 344, and a GPIO 346 that provides GPIO connections 348. These peripheral components are connected to the NoC 350 via interconnect nodes, enabling the MCU 328 to communicate with and manage the peripheral interfaces 334 to 346. In some embodiments, the basic subsystem 326 may be divided into two clock domains: a first region and a second region. The first region operates in the faster system clock domain and includes, for example, an MCU 328, a NoC 350, an LMEM330, and an SPI-to-AHB bridge controller that translates between the SPI interface and the internal bus. The second region operates in the slower clock domain and includes, for example, a UART 338, a GPIO 346, a watchdog timer (WDT), and a basic register file. Figure 3 The digital dies 300 shown are stacked vertically on the reference. Figure 2Below the analog die 220, the compute tile 364 in the tile subsystem 354 is connected to the corresponding driver tile 222 on the analog die 220 via TSV 224. In some embodiments, depending on the system use case, input / output connections to off-chip ports can be routed through the digital die 300 using a bump configuration (e.g., solder bumps on the digital die 300 directly connected to external components) and / or through the analog die 220 using a wire bonding configuration (e.g., wire connections from pads on the analog die 220 to external components).

[0077] Tile controller 352 is located between basic subsystem 326 and tile subsystem 354. Tile controller 352 is connected to NoC 350 within basic subsystem 326 and controls the connection from graphics and display subsystem 302 to computational tiles 364 (e.g., as...). Figure 3 The tile controller 352 manages the pixel data stream of the tile area shown in the diagram. It is responsible for receiving the pixel stream from the graphics and display subsystem 302, distributing the pixel data to the compute tiles 364, and synchronizing the lighting operations of multiple compute tiles 364. The tile controller 352 works in conjunction with multiple tile processing chains 356 (e.g., such as...). Figure 3 The tile processing chains 356a to 356c shown are interfaces, each tile processing chain 356 including a de-gamma lookup table (LUT) 358 (e.g., as shown in the image). Figure 3 The gamma-de-Gamma LUTs shown are 358a to 358c), and the scaler 360 (e.g., as shown) Figure 3 The scalers 360a to 360c shown) and the PWM clock 362 (e.g., as shown) Figure 3 The PWM clocks shown are 362a to 362c. A degamma LUT 358 applies gamma correction to the incoming pixel data to generate gamma-corrected pixel data. A scaler 360 adjusts the size of the gamma-corrected data to generate scaled pixel data, which is then distributed to computation tiles 364. Tile interconnect 210 corresponds to the reference... Figure 2 The tile interconnection described; although Figure 3While the tile interconnect 210 is not explicitly described, it represents a communication path through which the tile controller 352 distributes pixel data to compute tiles 364 via the tile processing chain 356. In some embodiments, the tile controller 352 implements two main control mechanisms: a first mechanism for sending new pixel data to individual tile rows via the tile interconnect 210 based on pixel addresses; and a second mechanism for illumination, where a gamma LUT generates non-linear or linear PWM increments (e.g., referred to as "tickers"), which are globally sent to each compute tile 364 to initiate pixel driving. Synchronization signals are distributed from the tile controller 352 to the tile region to synchronize illumination operations on multiple compute tiles 364. In some embodiments, the tile controller 352 may (e.g., for μLED and LCoS displays, respectively) receive pixel streams in a row-by-pixel or tile-by-tile manner. In some embodiments, the rolling shutter mechanism starts the pixel drive in row-by-row order, while the global shutter mechanism starts the pixel drive for all rows simultaneously, and either mechanism can be started by the tile controller 352.

[0078] The tile subsystem 354 (also called the tile region) occupies a portion of the digital die 300 and contains a K-by-N array of identical, adjacent computation tiles 364 arranged in rows and columns. The computation tiles 364 are related to the reference... Figure 2The described computational tile 212 corresponds to this. Each computational tile 364 is responsible for illuminating a corresponding set of pixels (e.g., an M×M pixel area) in the pixel array on the display die 240. Each computational tile 364 includes its own local controller and gamma lookup table (LUT) for independently managing the illumination of its corresponding pixel area. In some embodiments, each computational tile 364 includes its own local controller, SRAM block, and PWM generation circuitry, which operate independently of the other computational tiles 364. Tile controller 352 simultaneously broadcasts a global PWM timing signal (e.g., a tick) to all computational tiles 364, and each computational tile 364, upon receiving these timing signals, concurrently processes its locally stored pixel data and generates pixel on / off states independently of the other computational tiles 364. In some embodiments, the values ​​of K, N, and M can be selected to maximize the number of pixels on the display die and the local pixel value storage (e.g., K and N can be 44, and M can be 64). Each row of compute tiles 364 receives a PWM clock 362 signal from the tile controller 352 via a degamma LUT 358 and a scaler 360. Compute tiles 364 within each row are connected in series, with data flowing from one tile to the next adjacent tile via tile interconnects 210 in a pipelined streaming manner. The tile controller 352 streams pixel data to the compute tiles 364 row-by-row via tile interconnects 210. Upon receiving pixel data, each compute tile 364 processes the data locally and drives the corresponding pixel on / off state to the driver tile 222 on the analog die 220 via a TSV 224, which in turn controls the illumination of the pixel array 242 on the display die 240. The TSV 224, driver tile 222, analog die 220, display die 240, and pixel array 242 are connected to a reference... Figure 2 The components described correspond to each other. The internal architecture reference for each computational tile is 364. Figure 4 Further detailed description.

[0079] The digital die 300 is implemented as a flexible architecture that can integrate various additional functional blocks. Computation tile 364 connects to the NoC 350, which is related to the reference... Figure 2 Corresponding to the described interconnect structure 204, NoC 350 enables communication between the compute tiles 364 and other subsystems within the digital die 300. Similarly, functional blocks separated from the main SoC graphics pipeline can also be integrated into NoC 350 as part of the graphics and display subsystem 302. This architecture, by integrating such functional blocks into the digital die 300, enables the system to reduce off-chip bandwidth requirements between the external SoC and the 3D stacked microdisplay system, and eliminates redundant latency in the main path that would otherwise be caused by packetization, compression, and decompression operations.

[0080] Although Figure 3 The specific arrangement of the subsystems within the digital die 300 is shown, but other configurations are also considered. In some embodiments, the graphics and display subsystem 302 supports input interfaces in addition to MIPI, such as DisplayPort, HDMI, embedded DisplayPort (eDP), or proprietary high-speed serial interfaces. The graphics and display subsystem 302 may include additional processing blocks, such as an image decompression engine, frame rate conversion logic, or a display stream compression (DSC) decoder. In some embodiments, the graphics and display subsystem 302 includes multiple input channels to support stereoscopic display configurations or redundant data paths.

[0081] In several implementations, the basic subsystem 326 can be configured differently. In some embodiments, the MCU 328 is a simple microcontroller for basic startup and housekeeping tasks; while in other embodiments, the MCU 328 includes additional processing capabilities for runtime calibration, thermal management, or system diagnostics. The peripheral interfaces within the basic subsystem 326 can vary, employing different combinations of SPI, UART, I2C, GPIO, or other interface types depending on system requirements. The NoC 350 can be implemented using various topologies, including mesh networks, ring networks, crossbar switches, or hierarchical bus structures, depending on the system's bandwidth and latency requirements.

[0082] The tile controller 352 and the tile subsystem 354 can be configured in various arrangements. In some embodiments, the number of tiles 364 is calculated in relation to... Figure 3 The K×N arrangement shown varies, using fewer or more tiles depending on the display resolution and processing requirements. Tile sizes can differ, with larger tiles covering more pixels but requiring more local memory; or smaller tiles providing finer granularity but requiring more inter-tile communication. In some embodiments, the tile processing chain 356 includes additional processing stages, such as dithering logic, local dimming control, or pixel compensation circuitry, in addition to the de-gamma LUT 358 and scaler 360. The number of tile processing chains 356 can vary depending on the display's color configuration (e.g., monochrome, RGB, RGBW). Figure 3 The three chains shown are different.

[0083] The clock and reset block 310 can support multiple clock configurations. In some embodiments, it includes multiple PLLs to generate different clock frequencies for different subsystems within the digital die 300. The digital die 300 can operate with multiple clock domains, allowing different portions of the die to operate at different frequencies to optimize power consumption. In some embodiments, the clock and reset block 310 includes clock gating logic or a dynamic frequency adjustment unit, wherein the clock gating logic is used to disable the clock for inactive portions of the die, and the dynamic frequency adjustment unit is used to adjust the clock speed according to the workload.

[0084] Tile interconnect 210 can be implemented using a variety of topologies and wiring schemes. Although Figure 3 A row-based streaming architecture is described, where data flows from one tile to the next adjacent tile, but other data distribution patterns are also possible. In some embodiments, the tile interconnect 210 includes direct connections between non-adjacent tiles to reduce latency in certain data transmission patterns. The tile interconnect 210 can support bidirectional data streaming, enabling pixel shifting operations or inter-tile communication for image processing algorithms that require data from adjacent tiles. In some embodiments, the tile interconnect 210 includes separate data paths for pixel data and separate data paths for control signals to reduce contention and improve throughput.

[0085] Digital nude 300 except Figure 3 In addition to the functions shown, various additional processing functions may be supported. In some embodiments, the digital die 300 includes dedicated hardware for foveated rendering, where pixel resolution varies on the display based on the user's gaze direction. The digital die 300 may include motion compensation logic to reduce motion blur, or local tone mapping circuitry to optimize image quality for the display's dynamic range. In some embodiments, the digital die 300 includes test and debugging circuitry, such as built-in self-test (BIST) logic or scan chains, to facilitate manufacturing testing and troubleshooting.

[0086] Figure 4 A block diagram of a calculated tile is shown according to some embodiments. (See reference...) Figure 2 and Figure 3 As described, the 3D stacked microdisplay system 200 includes a digital die 202 that houses an array of computational tiles 400 responsible for local processing and driving pixel illumination. The computational tiles 400 represent functional units within the digital die 300 of the 3D stacked microdisplay system 200 and correspond to a reference... Figure 2 The description of the calculation tile 212 and reference Figure 3The computational tile 364 is described. Each computational tile 400 is responsible for managing the illumination of a corresponding set of pixels (e.g., an M×M pixel region) by: locally storing pixel values, generating PWM signals to control pixel illumination timing, and transmitting pixel on / off states to the analog die 220 to drive the display. The computational tile 400 includes three SRAM blocks (e.g., SRAM blocks 402, 404, and 406), each configured as a single-port, high-density SRAM (e.g., 384×96 bits), arranged to store pixel values ​​for corresponding pixel regions (e.g., a 64×64 pixel region). SRAM blocks 402, 404, and 406 collectively implement a frame buffer for storing processed tile data, which includes the pixel values ​​for the corresponding pixel regions managed by the computational tile 400. In some embodiments, SRAM blocks 402, 404, and 406 can be configured with various bit configurations depending on pixel region size and bit depth requirements. For example, a 384×96-bit configuration means that the SRAM is organized into 384 addressable rows, each 96 bits wide, resulting in a total storage capacity of 36,864 bits for each SRAM block 402, 404, and 406, sufficient to store pixel values ​​for the corresponding pixel region (e.g., a 64×64 pixel region) at the supported bit depth. SRAM blocks 402, 404, and 406 have a native read / write width (e.g., 96 bits), which can be programmed to different bit depths (e.g., 6 bits, 8 bits, or 10 / 12 bits), allowing computational tiles 400 to support different levels of pixel depth resolution. SRAM blocks 402, 404, and 406 are positioned to maximize the storage capacity of local pixel values ​​within computational tiles 400. In some embodiments for μLED displays, one SRAM block (e.g., SRAM block 402) stores calibration data, while the other two SRAM blocks (e.g., SRAM blocks 404 and 406) store pixel data. Alternatively, the same architecture is also compatible with LCoS displays, where each SRAM block can store a corresponding color field (e.g., R, G, and B color fields). This local data storage method allows for contention-free operation because read operations from SRAM blocks (e.g., PWM generation) and write operations to SRAM blocks (e.g., new frame data) are interleaved between operations. For example, one SRAM block can be in read mode while another SRAM block can be in write mode, enabling simultaneous read and write operations without contention. Thus, while one SRAM block stores incoming processed tile data as a frame buffer, another SRAM block can concurrently load its stored pixel values ​​into the PWM / Pulse Density Modulation (PDM) block 410 to generate pixel on / off states for a corresponding subset of the driving pixel array.

[0087] The calculation tile 400 includes a controller 408 component, which manages the overall operation of the tile. The controller 408 receives data from (see...) Figure 3 The tile controller (described) controls the pixel flow and coordinates the data flow within the tile, directing incoming pixel data to appropriate SRAM blocks 402, 404, and 406 for storage, and initiating lighting operation when the tile controller instructs lighting operation. Controller 408 is operatively coupled to a pulse width modulation (PWM) / pulse density modulation (PDM) block 410, which receives pixel values ​​read from SRAM blocks 402, 404, and 406 and converts these pixel values ​​into modulation signals that control the timing of pixel lighting. PWM / PDM block 410 checks each pixel value based on incoming ticks (e.g., received PWM counter increments from the tile controller) and generates single-bit on / off signals, which are ultimately driven to the corresponding driver tile 222 via TSV 224. TSV 224 and driver tile 222 are referenced... Figure 2 The described TSV corresponds to the driver tile. Adjacent to the PWM / PDM block 410 is a tag buffer 412, which implements a tag-based pixel value access mechanism. The tag buffer 412 stores the smallest pixel value in each row that is greater than the current PWM counter, allowing the PWM / PDM block 410 to check the tag buffer 412 before accessing the main SRAM block. If the value of the tag buffer 412 does not match the current PWM counter, the SRAM read is completely skipped, reducing redundant SRAM reads and achieving significant dynamic power savings (e.g., more than 4x) during pixel memory accesses for PWE generation (as referenced). Figure 6A and Figure 6B (For further details).

[0088] Computational tile 400 includes an interconnect 414 located at the center within the tile. This interconnect 414 serves as a communication hub, facilitating data transfer between controller 408, SRAM blocks 402, 404, and 406, PWM / PDM block 410, and an external tile interface. Interconnect 414 is coupled to an I / O formatter 416 component, which manages the formatting and packaging of input and output data streams to ensure compatibility between internal tile components and external tile-to-tile communication paths. I / O formatter 416 is connected to a router 418 component, which is configured for streaming operations, handling tile-to-tile data transfer in a pipelined manner. Router 418 receives a PWM timing strobe signal and tile data from an external source on one side and outputs a PWM timing strobe signal and tile data to adjacent tiles on the other side, thereby achieving [the desired result as described in the reference]. Figure 3 The described architecture is a per-tile row data distribution streaming architecture. At the bottom of compute tile 400, hybrid bonding of tile I / O 420 provides an interface for vertical connection to the upper analog die 220 layer. Analog die 220, TSV 224, and driver tile 222 are connected to the reference... Figure 2 The analog die, TSV, and driver tile are described in relation to each other. The hybrid bonding of tile I / O 420 enables computation tile 400 to transmit the single-bit on / off pixel state generated by PWM / PDM block 410 to the corresponding analog pixel driver in driver tile 222 via TSV 224.

[0089] Although Figure 4 A specific compute tile configuration with three SRAM blocks is shown, but other configurations are also considered. In some embodiments, compute tile 400 includes fewer than three SRAM blocks; for example, for a simpler implementation, compute tile 400 includes a single SRAM block; or, for double-buffered operation, compute tile 400 includes two SRAM blocks. Other embodiments include more than three SRAM blocks to support higher bit depths, additional color channels, or separate storage for calibration data and pixel data. The size and organization of each SRAM block may vary, with the number of rows, bit width, or access mode differing depending on the pixel region size and performance requirements.

[0090] In various implementations, the bit depth supported by the SRAM block may differ. While 6-bit, 8-bit, and 10 / 12-bit configurations are mentioned in the description, other bit depths are also possible. In some embodiments, the SRAM block supports 4-bit or 5-bit pixel values ​​to suit applications with acceptable lower color depths. Other embodiments support 14-bit, 16-bit, or higher bit depths to meet applications requiring extended dynamic range or accurate color reproduction. The SRAM block can be configured to store different bit depths for different color channels; for example, green pixels may have a higher bit depth to match human visual sensitivity.

[0091] In various implementations, the controller 408 and the PWM / PDM block 410 can be configured differently. In some embodiments, the PWM / PDM block 410 supports only pulse width modulation, while in other embodiments, it supports only pulse density modulation. The modulation scheme can be selected at runtime to optimize for different display characteristics or power consumption targets. In some embodiments, the PWM / PDM block 410 includes multiple modulation engines operating in parallel to improve throughput or simultaneously support different pixel groups. The controller 408 may include a programmable state machine, a microcode engine, or hardwired control logic, depending on flexibility and performance requirements.

[0092] The tag buffer 412 can be implemented in various configurations. In some embodiments, the tag buffer 412 stores a single tag value per row, while in other embodiments, the tag buffer 412 stores multiple tag values ​​per row to more finely skip SRAM reads. The tag value can represent the minimum pixel value, maximum pixel value, or other statistical measures in a row that can effectively determine whether SRAM access is needed. In some embodiments, the tag buffer 412 is omitted entirely, and the tile 400 reads pixel values ​​from the SRAM block each time a PWM counter increments. The size of the tag buffer 412 can vary depending on the number of rows in the pixel region and the bit depth of the tag values.

[0093] Router 418 and Interconnect 414 can support various data flow modes and topologies. Although Figure 4 The diagram describes a data flow architecture from one tile to the next adjacent tile, but other routing schemes are possible. In some embodiments, router 418 supports multi-hop routing to enable data transfer between non-adjacent tiles. Router 418 may include buffering for flow control, priority arbitration for different data types, or quality of service mechanisms to ensure timely delivery of time-sensitive data. In some embodiments, interconnect 414 includes separate buses or channels for different data types, such as pixel data, control signals, and synchronization signals.

[0094] The hybrid bonding of I / O formatter 416 and tile I / O 420 can be configured for different interconnect technologies and data formats. In some embodiments, I / O formatter 416 serializes parallel pixel data for transmission via hybrid bonding; in other embodiments, I / O formatter 416 transmits data in parallel format. The number and arrangement of hybrid bonds can vary depending on data bandwidth requirements and spacing limitations of the bonding technology. In some embodiments, the hybrid bonding of tile I / O 420 is replaced by through-silicon vias or other vertical interconnect technologies. I / O formatter 416 may include error detection or correction logic to improve data integrity in vertical interconnects.

[0095] Calculate the number of tiles 400, excluding Figure 4 In addition to the processing functions shown, various additional processing functions may be supported. In some embodiments, the computational tile 400 includes local image processing functions such as spatial filtering, edge enhancement, or noise reduction. The computational tile 400 may include pixel compensation logic that adjusts pixel values ​​based on calibration data to correct for differences in brightness or color between pixels. In some embodiments, the computational tile 400 includes temperature sensing and compensation circuitry to adjust pixel driving based on thermal conditions. The computational tile 400 may also include diagnostic circuitry for detecting and reporting pixel defects or other anomalies during operation.

[0096] Figure 5 Design space exploration of tile static random access memory (SRAM) according to some embodiments is illustrated. Design space exploration is a process of evaluating multiple possible configurations to find the optimal option that balances competing requirements. Design space exploration Figure 500 shows the PWM least significant bit (LSB) size in nanoseconds on the vertical axis and the memory area in square micrometers on the horizontal axis. The PWM LSB size represents the optimal timing resolution achievable for controlling pixel illumination. Simply put, the PWM LSB size indicates with what precision the system can control the timing of pixel on and off. A smaller PWM LSB size corresponds to finer control over pixel brightness levels. The memory area represents the silicon footprint allocated to each SRAM block within the computation tile 400, which directly affects the overall form factor of the digital die 300. For example, this represents the physical space the memory occupies on the chip.

[0097] This figure shows data points and trend lines for various bit depths, including 8-bit, 10-bit, 12-bit, and 14-bit configurations, where each bit depth is represented by different lines with different shapes along the lines (e.g., diamonds for 8-bit, circles for 10-bit, triangles for 12-bit, and squares for 14-bit). Each trend line connects to the Pareto-efficient configuration of the depth. Pareto-efficient configurations represent the optimal possible trade-offs, where improving one parameter (e.g., timing accuracy) requires sacrificing another parameter (e.g., memory area). These are, for example, more efficient design choices that can be implemented. The trend lines show an inverse relationship between PWM LSB size and memory area: as memory area increases, the achievable PWM LSB size decreases (i.e., finer timing resolution) because larger SRAM blocks can store more pixel data and support finer PWM operation. In short, allocating more chip space to memory allows for more precise control of pixel brightness. Higher bit depths shift the trend line to the right, indicating that for a given PWM LSB size, higher bit depths require a larger memory area because the more bits per pixel, the greater the storage capacity required.

[0098] The two horizontal dashed lines represent performance thresholds corresponding to the target specifications of the display system. The first dashed line at approximately 43 nanoseconds corresponds to a 16-bit PWM resolution at 360 frames per second; the second dashed line at approximately 21.5 nanoseconds corresponds to a 17-bit PWM resolution at 360 frames per second. These thresholds represent the PWM LSB size specified to achieve the corresponding PWM resolution at the target frame rate. Configurations below these lines meet or exceed the corresponding performance targets. Meeting these targets helps ensure smooth, flicker-free display performance at higher refresh rates, which is relevant to applications such as augmented reality, where visual quality and responsiveness are critical.

[0099] Several data points are highlighted using markers that indicate selected design points within the exploration space. Solid circles on the 8-bit curve represent the selected 8-bit configuration, hollow stars on the 10-bit curve represent the selected 10-bit configuration, and solid stars on the 12-bit curve represent the selected 12-bit configuration. These highlighted design points represent recommended configurations that balance competing requirements for memory area, PWM timing accuracy, and bit depth resolution while meeting target performance thresholds. From a practical perspective, these selected configurations represent design choices that provide the optimal combination of image quality, chip size, and performance for the intended display application. The background of the figure contains a grid of faded data points representing a broader design space evaluated during exploration, from which Pareto-efficient configurations are derived.

[0100] Figure 5 The design space exploration shown directly leads to (reference) Figure 4 The description of compute tile 400 indicates the SRAM configuration selection, while compute tile 400 is (refer to...) Figure 3 (Described) Tile subsystem 354 and (Reference) Figure 2 (Described) as part of digital die 202. In some embodiments, reference is made to... Figure 4 The described SRAM configuration (e.g., 384×96 bits) is chosen based on this design space exploration, taking into account trade-offs between target bandwidth, power consumption, PWM parameters, and silicon area to maximize the pixel values ​​stored for the corresponding pixel area (e.g., 64×64 pixels per tile) while achieving the target PWM performance under higher frame rate operation. This careful balancing of design parameters helps ensure that the 3D stacked microdisplay system 200 can deliver higher quality images with precise brightness control while maintaining a compact form factor suitable for augmented reality applications.

[0101] Figure 6AA pulse width modulation (PWM) operation method 600 for a pixel row is illustrated according to some embodiments. Method 600 illustrates a more general PWM operation method in which pixel values ​​stored in an SRAM block of a computation tile 400 are read and compared with a PWM counter to control pixel illumination. In some embodiments, one factor in the energy consumption of data movement within the computation tile 400 is the repeated reading of pixel data from the SRAM block to convert pixel values ​​into a PWM signal that controls the duration of pixel illumination. In some embodiments, method 600 includes incrementing a PWM counter value (602), a step that advances the counter to the next time step in the PWM cycle for comparison with stored pixel values. Method 600 also includes setting a column to zero (604), a step that initializes the column index to begin processing pixels from the beginning of the row. Method 600 also includes reading the pixel value at the current column position from the SRAM (606), a step that retrieves the stored luminance value of the pixel being processed. Method 600 also includes determining whether a pixel value is equal to a PWM counter (610), a step that compares the retrieved pixel value with the current counter value to determine whether the pixel needs to change state. If the pixel value matches the PWM counter (a branch), method 600 further includes turning off the pixel (614), a step that switches the pixel state from on to off over the PWM time period to control the duration of pixel illumination. After turning off the pixel, or if the pixel value does not match the PWM counter (a branch), method 600 further includes determining whether the current column is the last column in the row (612). If the current column is not the last column (a branch), method 600 further includes incrementing the column (608), a step that advances to the next pixel position and returns to reading the pixel value of the next column from SRAM (606). If the current column is the last column (a branch), method 600 advances to a "row complete" state (616), indicating that processing of the current row is complete, and the method can return to increment the PWM counter for the next iteration. This method involves reading all pixel values ​​each time the PWM counter increments to search for matches in the stored pixel values, thereby switching the pixel state. The total number of PWM counter increments is 2. L Where L is the bit depth used to represent the pixel value in memory, so 2 can be performed on all possible pixel values. L This can result in a large number of SRAM reads (e.g., high dynamic power consumption), which in turn cause relatively few PWM signal switching, especially when many pixel values ​​in a row do not match the current PWM counter value.

[0102] Although Figure 6AA specific PWM operation method with a particular sequence of steps is illustrated, but other methods are also considered. In some embodiments, the PWM counter linearly increments from zero to a maximum value determined by the bit depth (e.g., 0 to 255 for an 8-bit pixel value). Other embodiments employ non-linear counting schemes (e.g., logarithmic or gamma-corrected increments) to better match human visual perception of brightness levels. The PWM counter can count upwards from zero, downwards from the maximum value, or use a triangular counting mode that alternates between upward and downward counting.

[0103] The comparison operation in step 610 can be implemented in several ways. Although Figure 6A The description describes checking whether a pixel value equals a PWM counter, but other comparison operations are also possible. In some embodiments, the comparison checks whether the pixel value is less than or equal to the PWM counter, a step that determines whether the pixel should be turned on or off at the current time step. The comparison logic may include hysteresis or dejitter to prevent rapid switching of pixel states near a threshold. In some embodiments, multiple comparison thresholds are used to support multi-level pixel states or to implement a dithering mode.

[0104] In different implementations, the processing order of rows and columns may differ. Although Figure 6A The sequential processing of columns within a row is described, but other traversal patterns are also possible. In some embodiments, multiple columns are processed in parallel to improve throughput, where parallel comparison logic operates on multiple pixel values ​​simultaneously. To reduce memory access latency, the processing order may follow a serpentine pattern (alternating between left-to-right and right-to-left for consecutive rows). In some embodiments, pixels are processed in a block-based order rather than row-by-row to optimize memory locality or for alignment with tile-based architectures.

[0105] Pixel state transitions can be configured differently depending on display technology and application requirements. Although Figure 6A The diagram describes a pixel being turned off when its value matches a PWM counter, but other state transition schemes are also possible. In some embodiments, the pixel starts in an off state and is turned on when the counter reaches the pixel value. A pixel can undergo multiple state transitions within a frame period to implement pulse density modulation or hybrid PWM / PDM schemes. In some embodiments, pixel state transitions are interleaved between different rows to reduce transient current consumption and minimize electromagnetic interference.

[0106] In some implementations, alternative modulation schemes can be used in addition to PWM. Pulse density modulation (PDM) distributes pixel on-time across the entire frame period, rather than concentrating it in a single continuous pulse, which can reduce perceived flicker at low brightness levels. Hybrid approaches can combine PWM for coarse brightness control with PDM for fine brightness adjustment. In some embodiments, the modulation scheme is selected on a per-pixel or per-region basis to optimize for different content characteristics within the same frame.

[0107] Method 600 can be applied to different display technologies. For micro-LED displays, the PWM signal controls the duration of the current flowing through each LED element. For LCoS displays, the PWM signal controls the duration of the voltage applied to each liquid crystal cell. For OLED displays, the PWM signal can be combined with current stage control to achieve the desired brightness while controlling pixel aging. The timing parameters of the PWM operation (e.g., PWM period and minimum pulse width) can be adjusted based on the response characteristics of the specific display technology.

[0108] Figure 6B A tag-based pulse width modulation (PWM) method 650 according to some embodiments is illustrated. Method 650 uses a reference... Figure 4 The described tag buffer 412 implements a two-step access mechanism to solve the reference Figure 6A The description of redundant SRAM reads.

[0109] In some embodiments, method 650 includes incrementing a PWM counter value (618), a step that advances the counter to the next time step in the PWM cycle. Figure 6A Unlike method 600, which immediately reads all pixel values ​​from SRAM, method 650 also includes reading pixel values ​​from a flag buffer (620), a step that retrieves the smallest pixel value in each row that is greater than the current PWM counter. Method 650 also includes determining whether the pixel value read from the flag buffer is equal to the PWM counter (622), a step that checks whether any pixel in the row needs to be switched at the current counter value. If the pixel value does not match the PWM counter (no branch), in many practical cases the pixel value will not match, and method 650 proceeds directly to the row completion state (638), indicating that processing of the current row is complete without any SRAM access. This approach improves efficiency: by first checking the flag buffer, method 650 completely avoids accessing the main SRAM block when no pixel in the row needs to be switched at the current PWM counter value.

[0110] If the flag buffer value matches the PWM counter (is a branch), method 650 further includes entering a loop to read SRAM for all columns in that row (624). Within this loop, method 650 also includes setting the column to zero (626), a step that initializes the column index to begin processing pixels from the beginning of the row. Method 650 also includes reading the pixel value of the current column from SRAM (628), a step that retrieves the stored luminance value of the pixel being processed. Method 650 further includes determining if the pixel value is equal to the PWM counter (630), a step that compares the retrieved pixel value with the current counter value to determine if the pixel needs to change state. If the pixel value is equal to the PWM counter (is a branch), method 650 further includes turning off the pixel (634), a step that switches the pixel state from on to off. If the pixel value is not equal to the PWM counter (no branch), method 650 bypasses the turning-off step. Method 650 also includes determining if the current column is the last column (632). If the current column is not the last column (no branch), method 650 increments the column and returns to read the next pixel value from SRAM (628). If the current column is the last column (yes branch), method 650 further includes updating the marker buffer with the next minimum pixel value greater than the current PWM counter (636), a step that prepares the marker buffer for the next PWM counter increment. Method 650 then proceeds to the row completion state (638).

[0111] In the worst case, by employing the tag-based PWM method 650, the maximum number of reads from memory can be reduced by at least T times, where T=2. L / Display size (e.g., for a display size of 64 pixels and L=8 bits, this saves approximately 4x power). This worst-case scenario represents a situation where all pixel values ​​in a row are different. In real-world scenarios, the savings can be even greater because there may be duplicate pixel values ​​in a row (e.g., images have spatial pixel commonalities). In some embodiments, for pixel tiles (e.g., 64×64 pixels), the contribution to read power can be reduced by approximately 4x in the worst case and by approximately 64x in the best case.

[0112] Although Figure 6B A specific tag-based PWM method with a particular sequence of steps is illustrated, but other methods for reducing redundant memory accesses are also considered. In some embodiments, the tag buffer stores different types of tag values ​​depending on the implementation requirements. For example, the tag buffer may store the minimum pixel value, the maximum pixel value, or a bitmap indicating which PWM counter values ​​in that row have at least one matching pixel. The choice of tag representation may affect the granularity of SRAM access skipping and the size of the tag buffer.

[0113] The tag buffer checking mechanism can be implemented in several ways. In some embodiments, such as... Figure 6B As shown, the tag buffer is checked row by row as each PWM counter increments. Other embodiments check the tag buffer with different granularity (e.g., once per group of rows or once per tile). The comparison operation in step 622 may use different logic depending on the tag representation, such as an equality comparison of minimum tag values ​​or a bit lookup of a bitmap-based tag. In some embodiments, multiple tag buffers are maintained in parallel to support different access modes or to implement pipelined tag checking and SRAM access.

[0114] In several implementations, the SRAM access mode during tag matching can differ. Although Figure 6B The description outlines reading all columns in a row upon tag matching, but other access modes are possible. In some embodiments, a subset of multiple columns is read based on additional tag information indicating which columns contain matching pixel values. To reduce access latency, SRAM access can be performed in burst or pipelined manner. In some embodiments, the SRAM read width is configurable to balance access granularity with power consumption.

[0115] In various embodiments, the tag buffer update mechanism in step 636 can be implemented in different ways. In some embodiments, after each SRAM access, the tag buffer is incrementally updated by scanning the read pixel values ​​to find the next minimum value greater than the current PWM counter. Other embodiments recalculate the tag value periodically or at the beginning of each frame. Tag updates can be performed in parallel with other operations to hide update latency. In some embodiments, the tag buffer is pre-calculated during the frame data loading phase and stored in an SRAM block along with the pixel data.

[0116] Tagging-based methods can be combined with other energy-saving techniques. In some embodiments, clock gating is applied to the SRAM block when the tag buffer indicates that access is not required, thereby further reducing dynamic power consumption. Tagging-based methods can be combined with data compression techniques that reduce the amount of data stored in the SRAM block. In some embodiments, the tag-based PWM method is compared with a reference... Figure 7 and Figure 8 The described pixel shifting mechanism is used in combination, where the marker buffer is updated to reflect the shifted pixel position.

[0117] Tag-based PWM methods can adapt to different display configurations and pixel arrangements. In some embodiments, separate tag buffers are maintained for different color channels (e.g., red, green, and blue) to allow for independent optimization of SRAM access for each channel. For displays employing subpixel rendering, the tag buffers can track pixel values ​​at the subpixel level (rather than the full pixel level). This method can be applied to displays with different pixel pitches, resolutions, and aspect ratios by adjusting the size and organization of the tag buffers accordingly.

[0118] Figure 7 A pixel shifting mechanism 700 according to some embodiments is illustrated. Matching the user's real-time random eye movements at subframe intervals allows for a smoother overall display experience. The method used to match random subframe eye movements is called pixel shifting. In conventional methods, the entire pixel array in the display is updated using the shifted image position in each subframe, which can be difficult to achieve higher frame rates and is energy-intensive. Utilizing local pixel computation of the 3D stacked microdisplay system 200, the pixel shifting mechanism 700 implements a lightweight pixel shifting architecture based on a single-bit multiplexer array, thereby achieving energy-efficient positional pixel shifting without the energy and latency costs of copying and transmitting pixel data between computation tiles 212. Computation tiles 212 and reference... Figure 2 The description corresponds to the calculated tiles.

[0119] The pixel shifting mechanism 700 depicts a series of subframes illustrating how an image displayed on a pixel tile grid can be shifted to match the user's real-time eye movements. In augmented reality (AR) and virtual reality (VR) applications, the user's eyes move rapidly and unpredictably between display refresh cycles. To reduce motion blur and improve perceived image quality, the display system may shift the position of the displayed image multiple times within a single frame cycle, with each shift corresponding to a subframe. Multiple subframes 704 (e.g., subframe 0 704a, subframe 1 704b, subframe 2 704c) together constitute a complete display frame. The pixel shifting mechanism 700 illustrates that this process uses a pixel tile grid, where each tile corresponds to a computed tile 364 in a tile subsystem 354. An image position indicator shows the position of the displayed image within the pixel tile grid, and a shift direction indicator shows the direction and magnitude of each pixel shift between subframes. Subframe 0 704a shows the image at its original position on the display, within a pixel tile grid, with X and Y coordinates. Subframe 1 704b shows the same image after a pixel shift operation, where the image has been shifted in one or more directions (e.g., 2 pixels up, 4 pixels left), resulting in new coordinates (e.g., X-4 and Y+2), with a dashed outline indicating the previous position of subframe 0 704a. Subframe 2 704c shows a subsequent pixel shift, where the image has been shifted in a different direction relative to subframe 1 704b (e.g., 6 pixels down, 1 pixel right), resulting in updated coordinates (e.g., X-3 and Y-4), again with a dashed outline indicating the previous position of subframe 1 704b. Additional subframes with further pixel shifts may continue to appear in the sequence as the user's eye position changes.

[0120] To implement the pixel shifting mechanism per subframe, the tile array transmits pixel data in the up (U), down (D), left (L), and right (R) directions within the array. Pixel shifting mechanism 700 illustrates how pixel data can be transmitted in these directions within the tile array to achieve energy-efficient positional pixel shifting. In some embodiments, instead of copying the entire multi-bit pixel dataset between compute tiles 212 in each subframe (which would be energy-intensive and slow), pixel shifting mechanism 700 operates on single-bit on / off states driven to TSV 224. Since each pixel only needs one bit at any given time to indicate whether it is on or off, shifting these single-bit states between adjacent tiles is far more efficient than transmitting the entire pixel brightness value (e.g., 8 bits or higher). This approach avoids the high energy consumption of copying multi-bit pixel data from tile to tile in each subframe, enabling the display to track eye movements at high speeds without excessive power consumption. Compute tiles 212 and TSV 224 with reference Figure 2 The described calculation tiles correspond to TSVs.

[0121] Although Figure 7 A specific pixel shift sequence with a particular shift direction and magnitude is shown, but other configurations are also considered. The shift directions shown (up, down, left, right) represent an example; diagonal shifts can be achieved by combining horizontal and vertical shift components. In some embodiments, the shift magnitude is related to... Figure 7 The shift magnitudes shown vary, with smaller shifts (e.g., single-pixel increments) for fine eye tracking and larger shifts for fast eye movements. The maximum shift distance supported by the pixel shift mechanism can be configured based on the expected eye movement range and the size of the pixel tiles.

[0122] The number of subframes per frame can vary depending on application requirements and display refresh rate. Although Figure 7 Three subframes (subframe 0 704a, subframe 1 704b, and subframe 2 704c) are depicted, but other embodiments include fewer subframes (e.g., two subframes for a simpler implementation) or more subframes (e.g., four, eight, or more subframes for higher temporal resolution). In some embodiments, the number of subframes is dynamically adjusted based on the detected eye movement velocity, with more subframes used during rapid eye movements and fewer subframes used when the eye is relatively still. The subframe temporal sequence can be uniform (equal intervals between subframes) or non-uniform (intervals that vary according to the predicted eye position).

[0123] Several methods can be used to determine the shift value applied to each subframe. In some embodiments, the shift value is derived from real-time eye-tracking data provided by eye-tracking sensors (e.g., infrared cameras or other gaze detection systems) integrated into a display system or head-mounted device. Other embodiments employ predictive algorithms that estimate future eye positions based on historical eye movement patterns, head motion data, or content analysis. In some embodiments, fixed or predetermined shift patterns are used for applications where eye tracking is unavailable or unnecessary, such as anti-aliasing or resolution enhancement applications. Shift values ​​can also be provided by an external system-on-a-chip (SoC) or calculated locally within the digital die.

[0124] The timing of pixel shifts within each frame period can be configured in several ways. In some embodiments, pixel shifts occur at fixed intervals within a frame period, where the duration of each subframe is equal. Other embodiments employ adaptive timing, where the shift interval is adjusted based on detected or predicted eye movement speed. The pixel shift operation itself can be synchronized with the PWM illumination cycle to minimize visual artifacts during inter-frame transitions. In some embodiments, pixel shifts are performed during blanking when the pixel is not illuminated; while in other embodiments, the shift occurs through appropriate transition processing while the pixel remains illuminated.

[0125] Pixel shifting mechanisms can be combined with other display functions to enhance overall system performance. In some embodiments, pixel shifting is used in conjunction with foveated rendering, where higher resolution is provided in the display area corresponding to the user's gaze direction. Pixel shifting mechanisms can also be combined with local dimming or high dynamic range (HDR) processing to improve image quality. In some embodiments, the pixel shifting data path is shared with other inter-tile communication functions to reduce circuit area and power consumption.

[0126] Pixel shifting mechanisms can also be applied to applications beyond eye tracking. In some embodiments, pixel shifting is used for spatial anti-aliasing, where the displayed image is shifted by sub-pixel amounts between multiple subframes to achieve effective resolution enhancement. Other embodiments employ pixel shifting to reduce motion blur by shifting the image in the opposite direction of the detected motion. In some embodiments, pixel shifting is used for display calibration or testing purposes, where known shifting patterns are applied to verify pixel alignment and timing accuracy. The pixel shifting architecture can also support image stability in head-mounted displays by compensating for minute head movements between frame updates.

[0127] Figure 8 A single-bit pixel shifting circuit 800 based on a lightweight multiplexer array in a computational tile is illustrated according to some embodiments. The pixel shifting circuit 800 implements a reference by providing a lightweight shifting network for the single-bit pixel driver value. Figure 7 The pixel shifting mechanism 700 described above: This shifting network operates on top of the existing tile interconnect 210. The tile interconnect 210 and the reference... Figure 2The tile interconnection described corresponds to this. At a high level, the pixel shifting circuit 800 enables the display system to shift the position of the displayed image between subframes to track the user's eye movements without the high energy cost of copying complete pixel brightness data between adjacent computed tiles. Alternatively, the pixel shifting circuit 800 operates only on a single-bit on / off state, indicating whether each pixel is lit at any given time. This design avoids the high area cost of an additional 2D mesh network used to implement shifting in all directions. In some embodiments, each tile can drive itself and its neighboring tiles (e.g., up to eight neighboring tiles) in all directions. This approach reduces the high energy consumption of copying pixel data from tile to tile in each subframe.

[0128] At the bottom of the pixel shifting circuit 800, a tile SRAM+PWM 834 block stores pixel values ​​and generates pulse width modulation signals. The tile SRAM+PWM 834 block corresponds to the reference... Figure 4 The SRAM blocks 402, 404, and 406 and the PWM / PDM block 410 are described. The tile SRAM+PWM 834 block outputs a 1-bit pixel on / off state 832, representing a single bit driver value indicating whether each pixel should be lit at the current PWM time step. In some embodiments, the tile SRAM+PWM 834 block does not output a complete brightness value (e.g., an 8-bit value), but instead outputs a simple on or off signal for each pixel. These 1-bit pixel on / off states 832 are fed to a column shifting component 830, which performs a horizontal shift of the pixel data within the tile based on the received shift value of the current subframe. The column shifting component 830 shifts the pixel data left or right by a specified number of pixels within the tile boundaries.

[0129] The column-shifted data then enters the Arbitration X 826 unit, which handles the horizontal arbitration of pixel shifts in the left and right directions. The Arbitration X 826 unit determines whether a tile uses its own pixel data or receives shifted pixel data from a horizontally adjacent tile. The Arbitration X 826 unit receives tile data 828 from adjacent tiles via data input (left) and data input (right) paths, and outputs data to adjacent tiles via data output (left) 820 and data output (right) 822 paths. Since left and right shifts are mutually exclusive (i.e., left and right shifts cannot occur in the same subframe), horizontal arbitration requires only a single-bit arbiter.

[0130] The data path continues upward through buffer element 824, which temporarily stores intermediate data between the horizontal and vertical shift stages. A row shifting element 818, located above buffer 824, vertically shifts the pixel data within the tile, shifting the pixel data upward or downward by a specified number of pixels. The row-shifted data then enters arbitration element Y806, which handles the vertical shift arbitration of pixels in the up-down direction. Arbitration element Y806 determines whether a tile uses its own pixel data or receives shifted pixel data from adjacent tiles in the vertical direction. Arbitration element Y806 receives tile data 816 from adjacent tiles via data input (lower) 808 and data input (upper) 810 paths, and outputs data to adjacent tiles via data output (lower) 812 and data output (upper) 814 paths. Similar to horizontal arbitration, since the upward and downward shifts are mutually exclusive, vertical arbitration requires only a single-bit arbiter.

[0131] To handle diagonal shifts (i.e., shifts with both horizontal and vertical components), the pixel shifting circuit 800 decouples the horizontal and vertical shifts so that they are executed in a pipelined manner in two consecutive cycles. The horizontal shift is first performed via the arbitrary X826 component, and then the vertical shift is performed via the arbitrary Y806 component in the next cycle. A Ready-Valid-based backpressure mechanism, indicated by bidirectional flow control arrows, coordinates data transmission between adjacent tiles to ensure that each tile can drive itself and its adjacent tiles in all directions without data loss.

[0132] The shifted 1-bit pixel on / off state output from the arbitration Y 806 component is passed through the TSV FIFO 804 component, which buffers these shifted states. These shifted states are then passed via the TSV to the analog driver 802 on the upper analog die 220. The analog die 220, tile interconnect 210, TSV 224, and compute tile 212 are connected to the reference... Figure 2 The analog dies, tile interconnects, TSVs, and computational tiles described correspond to each other. The shift values ​​are global and are received via tile interconnect 210 at the beginning of each subframe (e.g., within approximately 10 nanoseconds), and these shift values ​​are then used to configure the multiplexer array for the current subframe. Thus, the pixel shifting circuit 800 implements a lightweight shifting network that operates on single-bit pixel drive values ​​as additional elements of the illumination path, thereby automatically performing pixel shifting using only a single bit of on / off state driven to TSV 224, without incurring the energy and latency costs of pixel data copying and transmission between computational tiles 212.

[0133] Although Figure 8 A specific pixel shifting circuit configuration with particular components and data paths is shown, but other configurations are also considered. In some embodiments, the multiplexer array architecture employs different multiplexer configurations, such as a wider multiplexer that can select from multiple two-input pairs to support larger shift distances within a single cycle. The shifting network can be implemented with different levels of parallelism, with some embodiments processing multiple pixel rows or columns simultaneously to improve throughput. In some embodiments, the pixel shifting circuitry is more tightly integrated with the PWM generation logic, rather than operating as a separate stage in the data path.

[0134] Arbitration mechanisms for horizontal and vertical displacement can be implemented in a variety of ways. Although Figure 8 Separate arbitration components X 826 and Y 806 are depicted for horizontal and vertical arbitration, but other embodiments combine these functions into a single arbitration unit. The arbitration logic can support different priority schemes, such as prioritizing certain shift directions or data from specific adjacent tiles. In some embodiments, arbitration includes conflict resolution mechanisms for situations where multiple tiles simultaneously attempt to drive the same output. Figure 8 The ready-to-effective backpressure mechanism shown represents one flow control method, while other embodiments use alternative handshake protocols or credit-based flow control.

[0135] In different implementations, the data flow and buffering in the pixel shifting circuit may differ. Although Figure 8 The buffer element 824 between the horizontal and vertical shift stages is depicted; however, other embodiments use different buffer sizes or omit intermediate buffers entirely to reduce latency. The TSV FIFO 804 can be configured with different depths depending on timing requirements and the variability in data arrival from the shift network. In some embodiments, additional buffers are included at tile boundaries to accommodate timing differences between adjacent tiles. A suitable buffer configuration can be selected to balance latency, throughput, and circuit area.

[0136] In several embodiments, the allocation of shift values ​​to the pixel shifting circuitry can be implemented in different ways. While the description indicates that shift values ​​are received via tile interconnect 210 at the beginning of each subframe, other timing arrangements may also be employed. In some embodiments, shift values ​​for multiple subframes are preloaded to reduce communication overhead. Shift values ​​can be distributed hierarchically, with a global shift value broadcast to all tiles and local adjustments applied to each tile. In some embodiments, different areas of the display receive different shift values ​​to support non-uniform shifting modes or compensate for optical distortions in the display system.

[0137] The interconnect topology used for inter-tile data transmission may differ during pixel movement. Although Figure 8 The diagram depicts connections to adjacent tiles in the top, bottom, left, and right directions, but other embodiments include connections to diagonally adjacent tiles or to tiles spaced multiple locations apart. In some embodiments, the shift network supports wraparound connections around the edges of the tile array, allowing pixels shifted from one edge to appear on the opposite edge. The data width of tile-to-tile connections may vary, with some implementations, for example... Figure 8 As shown, a single bit is transmitted per pixel, while other embodiments transmit multiple bits per pixel to support additional pixel state information. In some embodiments, the mobile network can be configured at runtime to enable or disable connectivity with specific neighboring tiles.

[0138] Pixel shifting circuitry can be adapted to different display configurations and applications. In some embodiments, the pixel shifting circuitry supports different shift granularities, such as subpixel shifting for displays with subpixel rendering, or multipixel shifting for coarser positioning adjustments. The pixel shifting circuitry can be configured to operate at different speeds, with higher speeds for displays with faster refresh rates or more subframes per frame. In some embodiments, the pixel shifting circuitry includes bypass modes that allow data to pass through without shifting when pixel shifting is not required, thereby reducing power consumption during static image display. In addition to translation, the pixel shifting architecture can be extended to support rotation or scaling operations, enabling more complex image transformations within tile arrays.

[0139] The following paragraphs describe embodiments of a 3D stacked microdisplay system in which multiple computational tiles perform local processing and drive pixel illumination on a vertically stacked display architecture. These embodiments further describe a circuit package comprising a digital chip, an analog chip, and a display chip, arranged in a three-layer stack and connected via through-silicon vias and hybrid bonding.

[0140] (A1) According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by a control circuit of an electronic device, cause the control circuit to perform operations. The electronic device has a display that includes a pixel array (e.g., as shown in reference 1). Figure 2 The described pixel array 242) and multiple computational tiles stacked with the pixel array (e.g., as referenced) Figure 2 The calculation of tile 212 described, as referenced Figure 3 The described calculation of tile 364 and as referenced Figure 4The described computational tiles 400. These operations include receiving image data corresponding to a frame to be displayed. These operations also include dividing the image data into tile datasets corresponding to different portions of a pixel array. These operations further include concurrently processing the tile datasets through the plurality of computational tiles, wherein concurrent processing of the tile datasets includes, for each tile data in the tile dataset: processing the tile to generate processed tile data, and driving a corresponding subset of the pixel array based on the processed tile data. For example, Figure 2 A 3D stacked microdisplay system 200 is illustrated, in which a digital die 202 houses computation tiles 212 that process pixel data locally and drive the corresponding pixel states to driver tiles 222 on an analog die 220 via a TSV 224. These driver tiles, in turn, control the illumination of the pixel array 242 on a display die 240. (See reference...) Figure 3 As described, the tile controller 352 receives a pixel stream from the graphics and display subsystem 302 and distributes the image data to the compute tiles 364 in a row-by-row manner via the tile interconnect 210. Each compute tile 364 processes the tile data locally and drives the corresponding pixel on / off state to the display.

[0141] (A2) In some embodiments of A1, processing the tile data to generate processed tile data includes: generating gamma-corrected tile data by applying gamma correction to the tile data; generating scaled tile data by adjusting the size of the gamma-corrected tile data; and generating processed tile data by converting the scaled tile data into a pulse width modulation (PWM) signal. For example, as referenced... Figure 3 As described, the tile controller 352 interfaces with a tile processing chain 356, which includes: a degamma LUT 358 for applying gamma correction to incoming pixel data to generate gamma-corrected pixel data; and a scaler 360 for resizing the gamma-corrected pixel data to generate scaled pixel data before distributing it to compute tiles 364. (See reference...) Figure 4 As described, the PWM / PDM block 410 within each compute tile 400 converts pixel values ​​into PWM signals that control the timing of pixel illumination.

[0142] (A3) In some embodiments of either A1 or A2, driving a corresponding subset of the pixel array based on the processed tile data includes storing the processed tile data in a corresponding computational tile among a plurality of computational tiles as a corresponding frame buffer, and concurrently loading the corresponding frame buffer from the corresponding computational tile to drive the corresponding subset of the pixel array. For example, as referenced Figure 4As described, computational tile 400 includes SRAM blocks 402, 404, and 406, which are used to store pixel values ​​for corresponding pixel regions. These SRAM blocks collectively implement a frame buffer for storing processed tile data. These SRAM blocks enable contention-free operation, where one SRAM block can be in read mode while another is in write mode, thus allowing simultaneous read and write operations without contention.

[0143] (A4) In some embodiments of any of A1-A3, the corresponding calculated tiles are spatially close to a corresponding subset of the pixel array, and configured to drive the corresponding subset of the pixel array via a corresponding analog driver. For example, as referenced Figure 2 As described, each computation tile 212 on the digital die 202 is vertically aligned with a corresponding driver tile 222 on the analog die 220, which in turn drives the corresponding pixel array 242 on the display die 240 via hybrid bonding technology 244.

[0144] (A5) In some embodiments of any of A1-A4, each computational tile in the plurality of computational tiles corresponds to a different subset of the pixel array. For example, as referenced Figure 3 As described, the tile subsystem 354 comprises a K×N array of computational tiles 364 arranged in rows and columns, wherein each computational tile 364 is responsible for illuminating a corresponding set of pixels (e.g., an M×M pixel region) in the pixel array on the bare die 240, such as Figure 2 As shown.

[0145] (A6) In some embodiments of any of A1-A5, the display further includes analog drivers, wherein the analog drivers are vertically stacked on a plurality of computational tiles, and the pixel array is vertically stacked on the analog drivers. For example, as referenced Figure 2 As described, the 3D stacked microdisplay system 200 includes a three-layer vertical stacked structure comprising a bottom digital die 202 (containing computing tiles 212), a middle analog die 220 (containing driver tiles 222), and a top display die 240 (containing a pixel array 242).

[0146] (A7) In some embodiments of any of A1-A6, a plurality of compute tiles are electrically coupled to an analog driver via through-silicon vias (TSVs) and configured to drive the analog driver, and the analog driver is heterogeneously integrated to the plurality of compute tiles via hybrid bonding and configured to drive a pixel array. For example, as referenced Figure 2 As described, the digital die 202 and the analog die 220 are connected via TSV 224, and the analog die 220 and the display die 240 are connected via hybrid bonding technology 244.

[0147] (B1) According to some embodiments, a circuit package for driving a display of an electronic device includes: a digital chip (e.g., as shown in reference 1) Figure 2 The described digital nub 202, as referenced Figure 3 The described digital die 300), and analog chips stacked vertically on top of the digital chips (e.g., as referenced). Figure 2 The analog die 220 described herein and the display chip (e.g., as shown in the reference) are stacked vertically on top of the analog chip. Figure 2 The described display die 240 includes a pixel array (e.g., as referenced). Figure 2 The pixel array 242 is described. The digital chip is located via a through-silicon via (TSV) (e.g., as referenced). Figure 2 The described TSV 224 is electrically coupled to the analog chip and configured to drive the analog chip; the analog chip is connected via hybrid bonding (e.g., as referenced). Figure 2 The described hybrid bonding technique 244) is heterogeneously integrated into the display chip and configured to drive the pixel array of the display chip. For example, Figure 2 A 3D stacked microdisplay system 200 is illustrated, comprising a three-layer vertically stacked structure, wherein a digital die 202 is connected to an analog die 220 via a TSV 224, and the analog die 220 is connected to a display die 240 via a hybrid bonding technology 244. As used herein, the terms “digital chip,” “digital die,” and “computing die” are used interchangeably to refer to the same component, and the terms “analog chip” and “analog die” are used interchangeably to refer to the same component.

[0148] (B2) In some embodiments of B1, the digital chip includes an array of computational tiles arranged in rows and columns, and the analog chip includes a plurality of analog drivers, wherein each of the plurality of analog drivers corresponds to a corresponding computational tile in the computational tile array, and each computational tile in the computational tile array is configured to drive a corresponding subset of the pixel array of the display chip via the corresponding analog driver, and is spatially adjacent to a corresponding subset of the pixel array of the display. For example, as referenced Figure 2 As described, the digital die 202 houses computation tiles 212, which are connected via TSV 224 to corresponding driver tiles 222 on the analog die 220. These driver tiles drive the pixel array 242 on the display die 240. (See reference...) Figure 3 As described, the tile subsystem 354 comprises a K-by-N array of computational tiles 364 arranged in rows and columns.

[0149] (B3) In some embodiments of either B1 or B2, the digital chip includes a tile controller (e.g., as referenced). Figure 2 The tile controller 206 described herein, as referenced Figure 3 The described tile controller 352 is electrically coupled to the computational tile array and configured to drive the computational tile array sequentially row by row. For example, as referenced... Figure 3 As described, the tile controller 352 manages the pixel data stream from the graphics and display subsystem 302 to the computation tile 364, thereby streaming data per tile row via the tile interconnect 210, such as Figure 2 As shown.

[0150] (B4) In some embodiments of any of B1-B3, the digital chip includes a system-on-a-chip (SoC) electrically coupled to a tile controller and configured to communicate image data with the tile controller. For example, as referenced... Figure 1A and Figure 2 As described, SoC 152 is connected to 3D stacked microdisplay system 200 via low-bandwidth connection 154, and graphics and display subsystem 214 receives incoming display data from external sources and processes the data to distribute it to computing tiles.

[0151] (B5) In some embodiments of any of B1-B4, the digital chip includes an on-chip network (NoC) (e.g., as referenced). Figure 3 The described NoC 350, as referenced Figure 2 The described interconnect structure 204, the NoC, is electrically coupled to the tile controller and configured to stream image data between the tile controller and the SoC. For example, as referenced... Figure 3 As described, NoC 350 enables communication between computational tiles 364 and other subsystems within the digital die 300, and corresponds to the reference... Figure 2 The described interconnect structure 204.

[0152] (C1) According to some embodiments, a method for displaying an image at an electronic device having a display comprising a pixel array (e.g., as referenced) Figure 2 The described pixel array 242) is used to stack multiple computational tiles (e.g., as in reference) with the pixel array. Figure 2 The calculation of tile 212 described, as referenced Figure 3 The described calculation of tile 364 and as referenced Figure 4The described computational tile (400) method includes receiving image data corresponding to a frame to be displayed. The method also includes partitioning the image data into tile datasets corresponding to different portions of a pixel array. The method further includes concurrently processing the tile datasets through the plurality of computational tiles, wherein concurrent processing of the tile datasets includes processing each tile data in the tile dataset to generate processed tile data, and driving a corresponding subset of the pixel array based on the processed tile data. For example, Figure 2 A 3D stacked microdisplay system 200 is illustrated, wherein a digital die 202 houses computation tiles 212, which process pixel data locally and drive the corresponding pixel states to driver tiles 222 on an analog die 220 via a TSV 224. The driver tiles then control the illumination of the pixel array 242 on a display die 240. (See reference...) Figure 3 As described, the tile controller 352 receives a pixel stream from the graphics and display subsystem 302 and distributes the image data to the computing tiles 364 row by row via the tile interconnect 210, such as Figure 2 As shown. Each computed tile 364 processes tile data locally and drives the corresponding pixel on / off state to the display.

[0153] As used herein, control circuitry refers to electronic components or combinations thereof configured to manage, coordinate, and direct the operation of other components in an electronic device or system. Control circuitry may include one or more controllers, processors, microcontrollers, tile controllers, or combinations thereof. For example, in a display system, control circuitry may include: a tile controller (e.g., as referenced...) Figure 2 The tile controller 206 described herein, as referenced Figure 3 The described tile controller 352 is configured to manage the distribution of pixel data to computational tiles; a microcontroller (e.g., as referenced) is configured to perform system-level control tasks. Figure 2 The MCU 208 described, as per the reference Figure 3 The described MCU 328); and / or the controller within each computing tile (e.g., as referenced). Figure 4The described controller 408 is configured to manage local tile operations. The control circuitry can be implemented using dedicated hardware, programmable logic, firmware, processor-executed software, or a combination thereof. In some embodiments, the control circuitry interfaces with the plurality of compute tiles to perform operations by: receiving image data corresponding to frames via a graphics and display subsystem; partitioning the image data into tile data via a tile controller; and distributing the tile data to compute tiles via a tile interconnect. The control circuitry can coordinate concurrent processing of tile data by the compute tiles, wherein each compute tile processes its corresponding tile data locally using its controller and stores the processed tile data in its SRAM block. The control circuitry can also coordinate the driving of the pixel array by causing each compute tile to generate a PWM signal based on the processed tile data and transmitting pixel on / off states to the corresponding driver tiles on the analog die via TSVs. As used in this document, the term "frame buffer" refers to memory configured to store pixel data for display (e.g., SRAM blocks in compute tiles that store processed tile data, including pixel values ​​for corresponding pixel regions).

[0154] (C2) In some embodiments of C1, processing the tile data to generate processed tile data includes: generating gamma-corrected tile data by applying gamma correction to the tile data; generating scaled tile data by adjusting the size of the gamma-corrected tile data; and generating processed tile data by converting the scaled tile data into a pulse width modulation (PWM) signal. For example, as referenced... Figure 3 As described, the tile controller 352 interfaces with a tile processing chain 356, which includes a degamma LUT 358 and a scaler 360. The degamma LUT 358 applies gamma correction to the incoming pixel data to generate gamma-corrected pixel data, and the scaler 360 adjusts the size of the gamma-corrected data to generate scaled pixel data, which is then distributed to compute tiles 364. (See reference...) Figure 4 As described, the PWM / PDM block 410 within each compute tile 400 converts pixel values ​​into PWM signals that control the timing of pixel illumination.

[0155] (C3) In some embodiments of either C1 or C2, driving a corresponding subset of the pixel array based on the processed tile data includes storing the processed tile data in the corresponding computed tiles among the plurality of computed tiles as a corresponding frame buffer (e.g., as referenced). Figure 4 The SRAM blocks 402, 404, and 406 are described, and corresponding frame buffers are concurrently loaded from the corresponding computed tiles to drive corresponding subsets of the pixel array. For example, as referenced... Figure 4 As described, computational tile 400 includes SRAM blocks 402, 404, and 406, which store pixel values ​​for corresponding pixel regions and collectively implement a frame buffer for storing processed tile data. These SRAM blocks enable contention-free operation, where one SRAM block can be in read mode while another is in write mode, thus enabling simultaneous read and write operations without contention.

[0156] (C4) In some embodiments of any of C1-C3, the corresponding calculated tiles are spatially close to a corresponding subset of the pixel array and are configured to drive the corresponding subset of the pixel array via a corresponding analog driver. For example, as referenced Figure 2 As described, each computation tile 212 on the digital die 202 is vertically aligned with a corresponding driver tile 222 on the analog die 220, which in turn drives the corresponding pixel array 242 on the display die 240 via hybrid bonding technology 244.

[0157] (C5) In some embodiments of any of C1-C4, each of the plurality of computational tiles corresponds to a different subset of the pixel array. For example, as referenced Figure 3 As described, the tile subsystem 354 includes a K×N array of computational tiles 364 arranged in rows and columns, wherein each computational tile 364 is responsible for illuminating a corresponding set of pixels (e.g., an M×M pixel region) in the pixel array on the bare die 240.

[0158] (C6) In some embodiments of any of C1-C5, the display further includes an analog driver, wherein the analog driver is vertically stacked on the plurality of compute tiles, and the pixel array is vertically stacked on the analog driver. For example, as referenced Figure 2 As described, the 3D stacked microdisplay system 200 includes a three-layer vertical stacked structure comprising a bottom digital die 202 (containing computing tiles 212), a middle analog die 220 (containing driver tiles 222), and a top display die 240 (containing a pixel array 242).

[0159] (C7) In some embodiments of any of C1-C6, the plurality of computing tiles are electrically coupled to an analog driver via a through-silicon via (TSV) and configured to drive the analog driver, and the analog driver is heterogeneously integrated to the plurality of computing tiles via hybrid bonding and configured to drive the pixel array. For example, as referenced Figure 2 As described, the digital die 202 and the analog die 220 are connected via TSV 224, and the analog die 220 and the display die 240 are connected via hybrid bonding technology 244.

[0160] (D1) According to some embodiments, the electronic device includes a display that includes a pixel array (e.g., as shown in reference 1). Figure 2 The described pixel array 242) and multiple computational tiles stacked with the pixel array (e.g., as referenced) Figure 2 The calculation of tile 212 described, as referenced Figure 3 The described calculation of tile 364 and as referenced Figure 4 The described computational tiles 400 and control circuitry are configured to: receive image data corresponding to a frame to be displayed; divide the image data into tile datasets corresponding to different portions of a pixel array; and concurrently process the tile datasets through the plurality of computational tiles, wherein concurrent processing of the tile datasets includes processing each tile data in the tile dataset to generate processed tile data, and driving a corresponding subset of the pixel array based on the processed tile data. For example, Figure 2 A 3D stacked microdisplay system 200 is illustrated, wherein a digital die 202 houses computation tiles 212, which process pixel data locally and drive the corresponding pixel states to driver tiles 222 on an analog die 220 via a TSV 224. The driver tiles then control the illumination of the pixel array 242 on a display die 240. (See reference...) Figure 3 As described, the tile controller 352 receives a pixel stream from the graphics and display subsystem 302 and distributes the image data tile-by-tile row to the computing tiles 364 via the tile interconnect 210.

[0161] (D2) In some embodiments of D1, the control circuitry is further configured to process the tile data to generate processed tile data by: applying gamma correction to the tile data to generate gamma-corrected tile data; adjusting the size of the gamma-corrected tile data to generate scaled tile data; and converting the scaled tile data into a pulse width modulation (PWM) signal to generate processed tile data. For example, as referenced... Figure 3 As described, tile controller 352 is connected to tile processing chain 356, which includes a degamma LUT 358 and a scaler 360. The degamma LUT 358 applies gamma correction to incoming pixel data to generate gamma-corrected pixel data, and the scaler 360 adjusts the size of the gamma-corrected data to generate scaled pixel data, which is then distributed to compute tiles 364. (See reference...) Figure 4 As described, the PWM / PDM block 410 within each compute tile 400 converts pixel values ​​into PWM signals that control the timing of pixel illumination.

[0162] (D3) In some embodiments of either D1 or D2, the control circuitry is further configured to drive a corresponding subset of the pixel array based on the processed tile data by storing the processed tile data in a corresponding computed tile as a corresponding frame buffer, and concurrently loading the corresponding frame buffer from the corresponding computed tile to drive the corresponding subset of the pixel array. For example, as referenced Figure 4 As described, the computation tile 400 includes SRAM blocks 402, 404, and 406, which are used to store pixel values ​​for corresponding pixel regions. These SRAM blocks enable contention-free operation, where one SRAM block can be in read mode while another SRAM block is in write mode, thus enabling simultaneous read and write operations without contention.

[0163] (D4) In some embodiments of any of D1-D3, the corresponding computed tiles spatially approximate a corresponding subset of the pixel array and are configured to drive the corresponding subset of the pixel array via a corresponding analog driver. For example, as referenced Figure 2 As described, each computation tile 212 on the digital die 202 is vertically aligned with a corresponding driver tile 222 on the analog die 220, which in turn drives the corresponding pixel array 242 on the display die 240 via hybrid bonding technology 244.

[0164] (D5) In some embodiments of any of D1-D4, each of the plurality of computational tiles corresponds to a different subset of the pixel array. For example, as referenced Figure 3 As described, the tile subsystem 354 includes a K×N array of computational tiles 364 arranged in rows and columns, wherein each computational tile 364 is responsible for illuminating a corresponding set of pixels (e.g., an M×M pixel region) in the pixel array on the bare die 240.

[0165] (D6) In some embodiments of any of D1-D5, the display further includes analog drivers, wherein the analog drivers are vertically stacked on the plurality of compute tiles, and the pixel array is vertically stacked on the analog drivers. For example, as referenced Figure 2 As described, the 3D stacked microdisplay system 200 includes a three-layer vertical stacked structure comprising a bottom digital die 202 (containing computing tiles 212), a middle analog die 220 (containing driver tiles 222), and a top display die 240 (containing a pixel array 242).

[0166] (E1) According to some embodiments, a system includes a wrist-worn wearable device (e.g., as referenced). Figures 9A to 9C-2The described wrist-worn wearable device 926 (or multiple wrist-worn wearable devices) and a pair of augmented reality glasses (e.g., as referenced) Figure 9A The described AR device 928), and the system is configured to perform an operation corresponding to any of A1-A7.

[0167] The devices described above, including wrist-worn wearable devices, head-mounted visual devices, systems, and haptic feedback devices, will be described in further detail below. The specific operations described above can occur as a result of specific hardware, which is described in more detail below. The devices described below are not limiting, and features on these devices can be removed or additional features can be added to them.

[0168] Example Extended Reality System Figure 9A , Figure 9B , Figure 9C-1 and Figure 9C-2 An example XR system, including an AR system and an MR system, is shown according to some embodiments. Figure 9A A first XR system 900a and a first example user interaction are shown, which uses a wrist wearable device 926, a head wearable device (e.g., an AR device 928), and / or a HIPD 942. Figure 9B A second XR system 900b and a second example user interaction are shown, the second example user interaction using a wrist wearable device 926, an AR device 928 and / or a HIPD 942. Figure 9C-1 and Figure 9C-2 A third MR system 900c and a third example user interaction are shown, which uses a wrist wearable device 926, a head wearable device (e.g., an MR device such as a VR device), and / or a HIPD 942. As those skilled in the art will understand upon reading the description provided herein, the above example AR and MR systems (described in detail below) can perform various functions and / or operations.

[0169] The wrist-worn wearable device 926, the head-worn wearable device, and / or the HIPD 942 can be communicatively coupled via a network 925 (e.g., cellular, near-field, Wi-Fi, personal area network, wireless local area network (LAN)). Additionally, the wrist-worn wearable device 926, the head-worn wearable device, and / or the HIPD 942 can also be communicatively coupled via the network 925 (e.g., cellular, near-field, Wi-Fi, personal area network, wireless LAN) to one or more servers 930, computers 940 (e.g., laptops or computers), mobile devices 950 (e.g., smartphones, tablets), and / or other electronic devices. Similarly, when textile-based smart clothing is used, it can also be communicatively coupled via the network 925 to the wrist-worn wearable device 926, one or more head-worn wearable devices, the HIPD 942, one or more servers 930, computers 940, mobile devices 950, and / or other electronic devices.

[0170] Go to Figure 9A The illustration shows a user 902 wearing a wrist-worn wearable device 926 and an AR device 928, with a HIPD 942 placed on their table. The wrist-worn wearable device 926, AR device 928, and HIPD 942 facilitate user interaction with the AR environment. Specifically, as shown in the first XR system 900a, the wrist-worn wearable device 926, AR device 928, and / or HIPD 942 enable the presentation of one or more avatars 904, digital representations 906 of one or more contacts, and one or more virtual objects 908. As discussed below, the user 902 can interact with the one or more avatars 904, digital representations 906 of one or more contacts, and one or more virtual objects 908 via the wrist-worn wearable device 926, AR device 928, and / or HIPD 942. Additionally, the user 902 can directly view physical objects in the environment, such as a physical table 929, through one or more transparent lenses and one or more waveguides of the AR device 928. Alternatively, an MR device can be used instead of the AR device 928, and a similar user experience can be produced, but the user will not directly see physical objects in the environment (such as table 929), but will be presented with a virtual reconstruction of table 929 based on one or more sensors of the MR device (e.g., an outward-facing camera capable of recording the surrounding environment).

[0171] User 902 may provide user input using any of the following: a wrist-worn wearable device 926, an AR device 928 (e.g., through physical input at the AR device and / or built-in motion tracking of the user's limbs), smart textile apparel, an externally mounted limb tracking device, or a HIPD 942. For example, user 902 may perform one or more gestures to provide user input, which may be detected by the wrist-worn wearable device 926 (e.g., using one or more EMG sensors and / or IMUs built into the wrist-worn wearable device) and / or by the AR device 928 (e.g., using one or more image sensors or cameras). Alternatively or additionally, user 902 may provide user input via one or more touch surfaces of the wrist-worn wearable device 926, the AR device 928, and / or the HIPD 942; and / or voice commands acquired by the microphones of the wrist-worn wearable device 926, the AR device 928, and / or the HIPD 942. The wrist-worn wearable device 926, AR device 928, and / or HIPD 942 include an artificial intelligence digital assistant to help the user provide user input (e.g., complete a series of actions, suggest different actions or commands, provide reminders, confirm commands). For example, the digital assistant can be invoked via input occurring at AR device 928 (e.g., via input at the temple of AR device 928). In some embodiments, user 902 can provide user input via one or more facial gestures and / or facial expressions. For example, the camera of wrist-worn wearable device 926, AR device 928, and / or HIPD 942 can track user 902's eyes to navigate the user interface.

[0172] The wrist-worn wearable device 926, AR device 928, and / or HIPD 942 can operate individually or in combination to allow user 902 to interact with the AR environment. In some embodiments, HIPD 942 is configured to operate as a central hub or control center for the following devices: wrist-worn wearable device 926; AR device 928; and / or another communication-coupled device. For example, user 902 can provide input at any of the wrist-worn wearable device 926, AR device 928, and / or HIPD 942 to interact with the AR environment, and HIPD 942 can identify one or more backend and frontend tasks to perform the requested interaction and distribute instructions to execute the one or more backend and frontend tasks at the wrist-worn wearable device 926, AR device 928, and / or HIPD 942. In some embodiments, backend tasks are user-insensible background processing tasks (e.g., rendering content, decompression, compression, application-specific operations), while frontend tasks are user-insensible user-facing tasks (e.g., presenting information to the user or providing feedback to the user). HIPD 942 can perform backend tasks and provide operational data corresponding to the backend tasks performed to the wrist wearable device 926 and / or AR device 928, enabling the wrist wearable device 926 and / or AR device 928 to perform frontend tasks. In this way, HIPD 942 (which has more computing resources and greater thermal headroom than the wrist wearable device 926 and / or AR device 928) performs computationally intensive tasks and reduces the computing resource utilization and / or power consumption of the wrist wearable device 926 and / or AR device 928.

[0173] In the example shown in the first XR system 900a, HIPD 942 identifies one or more backend and frontend tasks associated with a user request to initiate an AR video call with one or more other users (represented by avatar 904 and contact digital representation 906); and HIPD 942 issues instructions to execute the one or more backend and frontend tasks. Specifically, HIPD 942 performs backend tasks for processing and / or rendering image data (and other data) associated with the AR video call and provides operational data associated with the performed backend tasks to AR device 928, causing AR device 928 to perform frontend tasks for presenting the AR video call (e.g., presenting avatar 904 and contact digital representation 906).

[0174] In some embodiments, HIPD 942 serves as a focal point or anchor point for information presentation. This allows user 902 to generally know where the information is presented. For example, as shown in the first XR system 900a, an avatar 904 and a digital representation 906 of a contact are presented above HIPD 942. Specifically, HIPD 942 and AR device 928 operate in conjunction to determine the location for presenting avatar 904 and the digital representation 906 of the contact. In some embodiments, information may be presented within a predetermined distance from HIPD 942 (e.g., within five meters). For example, as shown in the first XR system 900a, a virtual object 908 is presented on a table at a distance from HIPD 942. Similar to the examples above, HIPD 942 and AR device 928 may operate in conjunction to determine the location for presenting virtual object 908. Alternatively, in some embodiments, the presentation of information is not constrained by HIPD 942. More specifically, avatar 904, the digital representation 906 of the contact, and virtual object 908 need not be presented within a predetermined distance of HIPD 942. Although the AR device 928 is described as working with HIPD, the MR head-mounted viewer can interact in the same way as the AR device 928.

[0175] The user input provided at the wrist wearable device 926, AR device 928, and / or HIPD 942 is coordinated to enable the user to initiate, continue, and / or complete an operation using any device. For example, user 902 can provide user input to AR device 928 to cause AR device 928 to present virtual object 908, and while AR device 928 presents virtual object 908, user 902 can provide one or more gestures via wrist wearable device 926 to interact with and / or manipulate virtual object 908. Although AR device 928 is described as working in conjunction with wrist wearable device 926, MR head-mounted viewer can interact in the same manner as AR device 928.

[0176] Integration of Artificial Intelligence and XR Systems Figure 9A The diagram illustrates an interaction where an AI virtual assistant can assist with requests made by user 902. This AI virtual assistant can be used to fulfill unlimited requests made by user 902 through natural language input. For example, in... Figure 9A In the middle, user 902 issues an audible request 944 to summarize the conversation and then share the summarized conversation with others in the meeting. Additionally, the AI ​​virtual assistant is configured to use the sensors of the XR system (e.g., the camera, microphone, and various other sensors from any device in the system) to provide contextual cues to the user to initiate tasks.

[0177] Figure 9A An example neural network 952 used in artificial intelligence applications is also shown. The uses of artificial intelligence (AI) are diverse and encompass many different aspects of the devices and systems described in this paper. AI capabilities cover a wide range of applications and enhance the interaction between user 902 and user devices (e.g., AR device 928, MR device 932, HIPD 942, wrist-worn wearable device 926). The AI ​​discussed in this paper can be obtained using many different training techniques. Although the primary AI model example discussed in this paper is a neural network, other AI models can also be used. Non-limiting examples of AI models include artificial neural networks (ANNs), deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), large language models (LLMs), long short-term memory networks, transformer models, decision trees, random forests, support vector machines, k-nearest neighbors, genetic algorithms, Markov models, Bayesian networks, fuzzy logic systems, and deep reinforcement learning, etc. The AI ​​models can be implemented on one or more user devices and / or any other devices among the user devices described in this paper. For devices and systems employing multiple AI models as described in this paper, different models can be used depending on the task. For example, an LLM can be used for a natural language AI virtual assistant, while a DNN can be used instead for object detection in a physical environment.

[0178] In another example, an AI virtual assistant may include many different AI models, and multiple AI models may be employed (concurrently, sequentially, or a combination of concurrent and sequential employment) based on the user's request. For example, an LLM-based AI model may provide instructions to help the user follow a recipe, and these instructions may be based in part on another AI model derived from ANNs, DNNs, RNNs, etc., capable of identifying which part of the recipe the user is performing (e.g., object and scene detection).

[0179] As AI training models evolve, the operations and experiences described herein may be performed using models different from those listed above, and those skilled in the art will understand that the list above is non-limiting.

[0180] User 902 can interact with the AI ​​model through natural language input, text input, or any other input modality that accepts natural language and / or the corresponding sound sensor module, acquired by a sound sensor. In another instance, input is provided by tracking user 902's eye gaze via a gaze tracker module. Additionally, the AI ​​model can receive input beyond that provided by user 902. For example, the AI ​​can further generate its response based on environmental input acquired by various types of sensors and / or their corresponding sensor modules in response to user requests (e.g., temperature data, image data, video data, ambient light data, audio data, GPS location data, inertial measurement results (i.e., user motion) data, pattern recognition data, magnetometer data, depth data, pressure data, force data, neuromuscular data, heart rate data, temperature data, sleep data). Sensor data can be acquired entirely from a single device (e.g., AR device 928) or from multiple devices communicating with each other (e.g., a system including at least two of AR device 928, MR device 932, HIPD 942, wrist-worn wearable device 926, etc.). The AI ​​model can also access additional information (e.g., one or more servers 930, computers 940, mobile devices 950 and / or other electronic devices) via network 925.

[0181] A non-limiting list of AI enhancements includes, but is not limited to, image recognition, speech recognition (e.g., automatic speech recognition), text recognition (e.g., scene text recognition), pattern recognition, natural language processing and understanding, classification, regression, clustering, anomaly detection, sequence generation, content generation, and optimization. In some embodiments, the AI ​​enhancements are executed, wholly or partially, on a cloud computing platform coupled to a user device (e.g., AR device 928, MR device 932, HIPD 942, wrist-worn wearable device 926) via one or more network communications. The cloud computing platform provides scalable computing resources, distributed computing, managed AI services, perturbation acceleration, pre-trained models, APIs, and / or other resources to support the full computational requirements of the AI ​​enhancements.

[0182] Example outputs derived from AI models may include natural language responses, mathematical calculations, charts displaying information, audio, images, videos, text, meeting minutes, predictions based on environmental factors, classification, pattern recognition, recommendations, evaluations, or other operations. In some embodiments, the generated outputs are stored on the local memory of a user device (e.g., AR device 928, MR device 932, HIPD 942, wrist-worn wearable device 926), storage options of external devices (servers, computers, mobile devices, etc.), and / or storage options of a cloud computing platform.

[0183] AI-based outputs can be presented across different modalities (e.g., audio-based modalities, vision-based modalities, haptic modalities, and any combination thereof) and across different devices within the XR system described herein. Some vision-based outputs may include XR enhancements on XR headsets, or information displayed on user interfaces on wrist-worn wearables, laptop devices, mobile devices, etc. Haptic feedback may provide information to user 902 on devices with or without displays (e.g., HIPD 942). The AI ​​model may also use the aforementioned inputs to determine the appropriate modality for presenting content to the user and one or more devices (e.g., audio output may be presented instead of visual output to a user walking on a busy road to avoid distracting user 902).

[0184] Example of augmented reality interaction Figure 9B The image shows user 902 wearing a wrist-worn wearable device 926 and an AR device 928, and holding a HIPD 942. In the second XR system 900b, the wrist-worn wearable device 926, the AR device 928, and / or the HIPD 942 are used to receive one or more messages and / or provide one or more messages to user 902's contacts. Specifically, the wrist-worn wearable device 926, the AR device 928, and / or the HIPD 942 detect and coordinate one or more user inputs to initiate a messaging application, and prepare a response to the received message via that messaging application.

[0185] In some embodiments, user 902 launches an application on wrist wearable device 926, AR device 928, and / or HIPD 942 via user input, causing the application to launch on at least one device. For example, in the second XR system 900b, user 902 performs a gesture associated with a command to launch a messaging application (represented by messaging user interface 912); wrist wearable device 926 detects the gesture and, based on determining that user 902 is wearing AR device 928, causes AR device 928 to present the messaging user interface 912 of the messaging application. AR device 928 may present the messaging user interface 912 to user 902 via its display (e.g., as shown in user 902's field of view 910). In some embodiments, the application is launched and can run on a device (e.g., wrist wearable device 926, AR device 928, and / or HIPD 942) that detects user input to launch the application, and that device provides operational data to another device to cause the messaging application to be presented. For example, the wrist-worn wearable device 926 can detect user input to launch a messaging application, launch and run the messaging application, and provide operational data to the AR device 928 and / or HIPD 942 to render the messaging application. Alternatively, the application can be launched and run on a different device than the one that detected the user input. For example, the wrist-worn wearable device 926 can detect gestures associated with launching the messaging application and enable the HIPD 942 to run the messaging application and coordinate its rendering.

[0186] Furthermore, user 902 can provide user input at the wrist wearable device 926, AR device 928, and / or HIPD 942 to continue and / or complete an operation initiated on another device. For example, after launching a messaging application via the wrist wearable device 926 and while the messaging user interface 912 is displayed on the AR device 928, user 902 can provide input at HIPD 942 to prepare a reply (e.g., as indicated by a swipe gesture performed on HIPD 942). The gesture performed by user 902 on HIPD 942 can be provided and / or displayed on another device. For example, the swipe gesture performed by user 902 on HIPD 942 is displayed on the virtual keyboard of the messaging user interface 912 displayed by AR device 928.

[0187] In some embodiments, the wrist wearable device 926, AR device 928, HIPD 942, and / or other communication-coupled devices may present one or more notifications to the user 902. The notification may be an indication of a new message, incoming call, application update, or status update, etc. The user 902 may select a notification via the wrist wearable device 926, AR device 928, or HIPD 942, causing an application or action associated with the notification to be presented on at least one device. For example, the user 902 may receive a notification of a received message at the wrist wearable device 926, AR device 928, HIPD 942, and / or other communication-coupled devices, and provide user input at the wrist wearable device 926, AR device 928, and / or HIPD 942 to view the notification. The device that detects the user input may cause the application associated with the notification to be launched and / or the application associated with the notification to be presented on the wrist wearable device 926, AR device 928, and / or HIPD 942.

[0188] While the examples above describe coordinated input for interaction with messaging applications, those skilled in the art will recognize upon reading this description that user input can be coordinated to interact with any number of applications, including but not limited to gaming applications, social media applications, camera applications, web-based applications, and financial applications. For example, AR device 928 can present gaming application data to user 902, while HIPD 942 can use a controller to provide input to the game. Similarly, user 902 can use wrist-worn wearable device 926 to activate the camera of AR device 928, and the user can use wrist-worn wearable device 926, AR device 928, and / or HIPD 942 to manipulate image acquisition (e.g., zoom in or out, or apply filters) and acquire image data.

[0189] While AR device 928 is shown to be capable of certain functions, it should be understood that AR devices can be AR devices with different functions based on cost and market demand. For example, an AR device may include a single output modality such as an audio output modality. In another example, an AR device may include a low-fidelity display as one of the output modalities, capable of presenting simple information (e.g., text and / or low-fidelity images / videos) to the user. In yet another example, an AR device may be configured with face-facing light-emitting diodes (LEDs) configured to provide information to the user; for example, when providing direction, an LED around the right lens may illuminate to notify the wearer to turn right, or an LED on the left may illuminate to notify the wearer to turn left. In another embodiment, an AR device may include an outward-facing projector, enabling the display of information (e.g., text information, media) on the user's palm or other suitable surface (e.g., a table, a whiteboard). In yet another embodiment, information may also be provided by locally darkening portions of the lens to emphasize portions of the environment where the user's attention can be directed. Some AR devices can present AR augmentations in a monocular or binocular manner (e.g., the AR augmentation can be presented on a single display associated with a single lens, rather than at two lenses to produce a binocular image). In some instances, AR devices capable of presenting AR augmentations in a binocular manner may also optionally display AR augmentations in a monocular manner (e.g., for energy-saving purposes or other presentation considerations). These examples are not exhaustive, and features of one AR device described above can be combined with features of another AR device described above. Although the features and experiences of AR devices have been described in general terms in the preceding sections, it should be understood that the described features and experiences can be applied in a similar manner to MR headsets, which will be described in the following sections.

[0190] Example of mixed reality interaction Go to Figure 9C-1 and Figure 9C-2The image shows a user 902 wearing a wrist-worn wearable device 926 and an MR device 932 (e.g., a device capable of providing a full VR or MR experience, which displays one or more objects from the physical environment on the device's display) and holding a HIPD 942. In the third MR system 900c, the wrist-worn wearable device 926, the MR device 932, and / or the HIPD 942 are used for interaction within an MR environment (e.g., a VR game or other MR / VR application). Although the MR device 932 presents a representation of the VR game to the user 902 (e.g., a first MR game environment 920), the wrist-worn wearable device 926, the MR device 932, and / or the HIPD 942 detect and coordinate one or more user inputs to allow the user 902 to interact with the VR game.

[0191] In some embodiments, user 902 may provide user input that elicits movement in the corresponding MR environment via wrist-worn wearable device 926, MR device 932, and / or HIPD 942. For example, user 902 in a third MR system 900c (such as...) Figure 9C-1 (As shown) The user 902 raises HIPD 942 in preparation for swinging it in the first MR game environment 920. MR device 932 responds to the user 902 raising HIPD 942 by causing the user's MR representation 922 to perform a similar action (e.g., raising a virtual object, such as virtual sword 924). In some embodiments, each device uses corresponding sensor data and / or image data to detect user input and provide an accurate representation of the user 902's movement. For example, the image sensor of HIPD 942 (e.g., a Simultaneous Localization and Mapping (SLAM) camera or other camera) can be used to detect the position of HIPD 942 relative to the user 902's body, allowing the virtual object to be properly positioned within the first MR game environment 920; sensor data from the wrist-worn wearable device 926 can be used to detect the speed at which the user 902 raises HIPD 942, synchronizing the user's MR representation 922 and virtual sword 924 with the user 902's movement; and the image sensor of MR device 932 can be used to represent the user 902's body, boundary conditions, or real-world objects within the first MR game environment 920.

[0192] exist Figure 9C-2In this scenario, user 902 performs a downward swing while holding HIPD 942. Wrist wearable device 926, MR device 932, and / or HIPD 942 detect the downward swing of user 902 and execute a corresponding action within the first MR gaming environment 920. In some embodiments, data collected by each device is used to enhance the user 902's experience within the MR environment. For example, sensor data from wrist wearable device 926 can be used to determine the speed and / or force of the downward swing, and image sensors from HIPD 942 and / or MR device 932 can be used to determine the location of the swing and how it should be represented within the first MR gaming environment 920. This can then be used as input to the MR environment (e.g., a game mechanic that can classify the user's input using the detected speed, force, location, and / or aspects of the user 902's action (e.g., the user performs a tap, a hit, a critical strike, a glancing strike, a miss) or can calculate an output (e.g., damage amount)).

[0193] Figure 9C-2 It is also shown that, when displaying the MR game environment 920, a portion of the physical environment is reconstructed and displayed on the monitor of the MR device 932. In this example, when one or more objects in the physical environment are potentially in the user's path (e.g., the user is likely to collide with objects in the physical environment), a reconstruction 946 of the physical environment is displayed instead of a portion of the MR game environment 920. Thus, this example MR game environment 920 includes: (i) an immersive VR portion 948 (e.g., an environment that has no necessary counterpart in the nearby physical environment); and (ii) a reconstruction 946 of the physical environment (e.g., a table 929 and a cup 951). Although the example shown here displays an MR environment where a reconstruction of the physical environment is shown to avoid collisions, other uses of the reconstruction of the physical environment can be employed, such as defining the characteristics of the virtual environment based on the surrounding physical environment (e.g., virtual pillars can be placed based on objects in the surrounding physical environment, such as trees).

[0194] Although the wrist-worn wearable device 926, MR device 932, and / or HIPD 942 are described as detecting user input, in some embodiments, the user input is detected at a single device (where this single device is responsible for distributing signals to other devices for executing the user input). For example, HIPD 942 may run an application for generating a first MR game environment 920 and provide the MR device 932 with corresponding data for rendering the first MR game environment 920, as well as detect movement of the user 902 (while holding HIPD 942) to cause corresponding actions to be performed within the first MR game environment 920. Additionally or alternatively, in some embodiments, operational data (e.g., sensor data, image data, application data, device data, and / or other data) of one or more devices is provided to a single device (e.g., HIPD 942) for processing, causing the corresponding device to perform an action associated with the processed operational data.

[0195] In some embodiments, user 902 may wear a wrist-worn wearable device 926, a wearable MR device 932, a textile-based smart garment 938 (e.g., a wearable haptic glove), and / or hold a handheld HIPD 942 device. In this embodiment, the wrist-worn wearable device 926, the MR device 932, and / or the textile-based smart garment 938 are used in an MR environment (e.g., as referenced above). Figure 9A and Figure 9B Interaction within any AR or MR system described. Although the MR device 932 presents a representation of an MR game to the user 902 (e.g., a second MR game environment 921), the wrist wearable device 926, the MR device 932, and / or the textile-based smart clothing 938 detect and coordinate one or more user inputs to allow the user 902 to interact with the MR environment.

[0196] In some embodiments, user 902 may provide user input that elicits movement in the corresponding MR environment via wrist wearable device 926, HIPD 942, MR device 932, and / or textile-based smart clothing 938. In some embodiments, each device uses corresponding sensor data and / or image data to detect user input and provide an accurate representation of user 902's movement. Although four different input devices are shown (e.g., wrist wearable device 926, MR device 932, HIPD 942, and textile-based smart clothing 938), each of these input devices can provide input for full interaction with the MR environment completely independently. For example, the wrist wearable device itself can provide sufficient input for interaction with the MR environment. In some embodiments, if multiple input devices (e.g., wrist wearable device and textile-based smart clothing 938) are used, sensor fusion can be utilized to ensure that the input is correct. Although multiple input devices are described, it should be understood that other input devices can be used in combination or alone, such as, but not limited to, external motion tracking cameras, other wearable devices mounted on different parts of the user's body, and devices that allow users to experience walking in the MR environment while remaining substantially stationary in the physical environment.

[0197] As described above, the data collected by each device is used to enhance the user's experience within the MR environment. Although not shown, the textile-based smart clothing 938 can be used in conjunction with MR devices and / or the HIPD 942.

[0198] Although some experiences are described as taking place on AR devices and others on MR devices, those skilled in the art will understand that experiences can be transferred from MR devices to AR devices and vice versa.

[0199] For ease of reference, some definitions of the following devices and components are defined herein: these devices and components may be included in some or all of the example devices discussed. It will be understood by those skilled in the art that certain types of components described may be more suitable for a particular set of devices and less suitable for different sets of devices. However, subsequent references to the components defined herein should be considered as being covered by the definitions provided.

[0200] In some embodiments, several example devices and systems, including electronic devices and systems, will be discussed. These example devices and systems are not intended to be limiting, and those skilled in the art will understand that alternative devices and systems to the example devices and systems described herein can be used to perform the operations and construct the systems and devices described herein.

[0201] As described herein, an electronic device is a device that uses electrical energy to perform a specific function. Such an electronic device can be any physical object containing electronic components, such as transistors, resistors, capacitors, diodes, and integrated circuits. Examples of electronic devices include smartphones, laptops, digital cameras, televisions, game consoles, and music players, as well as the example electronic devices discussed herein. As described herein, an intermediate electronic device is a device that is situated between two other electronic devices and / or between subsets of multiple components of one or more electronic devices, and facilitates communication, and / or data processing, and / or data transmission between the respective electronic devices and / or electronic components.

[0202] The above provided Figures 9A to 9C-2 The foregoing description is intended to expand upon the description provided in Appendix A, which is included herein. Although the terminology in Appendix A may differ from that used in the foregoing description, those skilled in the art will understand that these terms have the same meaning.

[0203] Any data collection performed by the device described herein and / or any device configured to perform or cause to perform the different embodiments described in any of the accompanying drawings (hereinafter referred to as the “Device”) is conducted with the user’s consent and in a manner that complies with all applicable privacy laws. The user is provided with options to allow the Device to collect data, and options to restrict or refuse the Device’s collection of data. The user can enable or opt out of any data collection at any time. Furthermore, the user is provided with the option to request the deletion of any collected data.

[0204] It will be understood that although the terms “first,” “second,” etc., may be used in this document to describe various elements, these elements are not limited by these terms. These terms are used to distinguish one element from another.

[0205] The terminology used herein is for the purpose of describing particular embodiments and is not intended to limit the claims. As used in the description of the embodiments and the appended claims, the singular forms “a,” “an,” and “the / said” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms “comprising” and / or “including” are used in this specification, they specify the presence of the stated features, uniformities, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, uniformities, steps, operations, elements, components, and / or groups thereof.

[0206] As used herein, depending on the context, the term "if" can be interpreted as meaning: "when the stated conditional precedent is true"; or "once" the stated conditional precedent is true; or "in response to determination" the stated conditional precedent is true; or "according to determination" the stated conditional precedent is true; or "in response to detection" the stated conditional precedent is true. Similarly, depending on the context, the phrases "if it is determined [the stated conditional precedent is true]"; "if [the stated conditional precedent is true]" or "when [the stated conditional precedent is true]" can be interpreted as meaning: "when it is determined that the stated conditional precedent is true"; "in response to determination" the stated conditional precedent is true; "according to determination" the stated conditional precedent is true; "when the stated conditional precedent is detected"; or "in response to detection" the stated conditional precedent is true.

[0207] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of operation and practical application, thereby enabling others skilled in the art to implement them.

Claims

1. A non-transitory computer-readable storage medium storing instructions that, when executed by a control circuit of an electronic device, cause the control circuit to perform an operation, the electronic device having a display including a pixel array and a plurality of computational tiles stacked with the pixel array, the operation comprising: Receive image data corresponding to the frame to be displayed; The image data is divided into tile datasets corresponding to different parts of the pixel array; as well as The tile dataset is processed concurrently using the plurality of computed tiles, wherein concurrent processing of the tile dataset includes processing data for each tile in the tile dataset: The tile data is processed to generate processed tile data; as well as The corresponding subset of the pixel array is driven based on the processed tile data.

2. The non-transitory computer-readable storage medium according to claim 1, wherein, Processing the tile data to generate the processed tile data includes: Gamma-corrected tile data is generated by applying gamma correction to the tile data; Scaled tile data is generated by adjusting the size of the gamma-corrected tile data; and The processed tile data is generated by converting the scaled tile data into a pulse width modulation (PWM) signal.

3. The non-transitory computer-readable storage medium according to claim 1, wherein, Driving the corresponding subset of the pixel array based on the processed tile data includes: The processed tile data is stored in the corresponding computed tile among the plurality of computed tiles, serving as a corresponding frame buffer; and The corresponding frame buffer is loaded concurrently from the corresponding computed tile to drive the corresponding subset of the pixel array.

4. The non-transitory computer-readable storage medium according to claim 1, wherein, The corresponding computational tile in the plurality of computational tiles is spatially close to the corresponding subset of the pixel array, and the corresponding computational tile in the plurality of computational tiles is configured to drive the corresponding subset of the pixel array through a corresponding analog driver.

5. The non-transitory computer-readable storage medium according to claim 1, wherein, Each of the plurality of computational tiles corresponds to a different subset of the pixel array.

6. The non-transitory computer-readable storage medium according to claim 1, wherein, The display also includes an analog driver, and wherein: The analog drivers are stacked vertically on the plurality of computing tiles; and The pixel array is stacked vertically on top of the analog driver.

7. The non-transitory computer-readable storage medium according to claim 6, wherein: The plurality of computing tiles are electrically coupled to the analog driver via through-silicon vias (TSVs), and the plurality of computing tiles are configured to drive the analog driver; and The analog driver is heterogeneously integrated into the plurality of computation tiles via hybrid bonding, and the analog driver is configured to drive the pixel array.

8. A method for displaying an image at an electronic device having a display, the display comprising a pixel array and a plurality of computational tiles stacked with the pixel array, the method comprising: Receive image data corresponding to the frame to be displayed; The image data is divided into tile datasets corresponding to different parts of the pixel array; as well as The tile dataset is processed concurrently using the plurality of computed tiles, wherein concurrent processing of the tile dataset includes processing data for each tile in the tile dataset: The tile data is processed to generate processed tile data; as well as The corresponding subset of the pixel array is driven based on the processed tile data.

9. The method according to claim 8, wherein, Processing the tile data to generate the processed tile data includes: Gamma-corrected tile data is generated by applying gamma correction to the tile data; Scaled tile data is generated by adjusting the size of the gamma-corrected tile data; and The processed tile data is generated by converting the scaled tile data into a pulse width modulation (PWM) signal.

10. The method of claim 8, wherein, Driving the corresponding subset of the pixel array based on the processed tile data includes: The processed tile data is stored in the corresponding computed tiles among the plurality of computed tiles as a corresponding frame buffer; and The corresponding frame buffer is loaded concurrently from the corresponding computed tile to drive the corresponding subset of the pixel array.

11. The method according to claim 8, wherein, Each of the plurality of computational tiles is spatially close to the corresponding subset of the pixel array, and each of the plurality of computational tiles is configured to drive the corresponding subset of the pixel array via a corresponding analog driver.

12. The method according to claim 8, wherein, Each of the plurality of computational tiles corresponds to a different subset of the pixel array.

13. The method of claim 8, wherein, The display also includes an analog driver, and wherein: The analog drivers are stacked vertically on top of the multiple computing tiles; and The pixel array is stacked vertically on top of the analog driver.

14. The method of claim 13, wherein, The plurality of computing tiles are electrically coupled to the analog driver via through-silicon vias (TSVs), and the plurality of computing tiles are configured to drive the analog driver; and The analog driver is heterogeneously integrated into the plurality of computation tiles via hybrid bonding, and the analog driver is configured to drive the pixel array.

15. An electronic device comprising: A display comprising a pixel array and a plurality of computational tiles stacked with the pixel array; as well as Control circuit, the control circuit being configured to: Receive image data corresponding to the frame to be displayed; The image data is divided into tile datasets corresponding to different parts of the pixel array; and The tile dataset is processed concurrently using the plurality of computed tiles, wherein concurrent processing of the tile dataset includes processing data for each tile in the tile dataset: The tile data is processed to generate processed tile data; as well as The corresponding subset of the pixel array is driven based on the processed tile data.

16. The electronic device according to claim 15, wherein, The control circuit is further configured to process the tile data to generate the processed tile data in the following manner: Gamma-corrected tile data is generated by applying gamma correction to the tile data; Scaled tile data is generated by adjusting the size of the gamma-corrected tile data; as well as The processed tile data is generated by converting the scaled tile data into a pulse width modulation (PWM) signal.

17. The electronic device according to claim 15, wherein, The control circuit is further configured to drive the corresponding subset of the pixel array based on the processed tile data in the following manner: The processed tile data is stored in the corresponding computed tile among the plurality of computed tiles as a corresponding frame buffer; as well as The corresponding frame buffer is loaded concurrently from the corresponding computed tile to drive the corresponding subset of the pixel array.

18. The electronic device according to claim 15, wherein, Each of the plurality of computational tiles is spatially close to the corresponding subset of the pixel array, and each of the plurality of computational tiles is configured to drive the corresponding subset of the pixel array via a corresponding analog driver.

19. The electronic device according to claim 15, wherein, Each of the plurality of computational tiles corresponds to a different subset of the pixel array.

20. The electronic device according to claim 15, wherein, The display also includes an analog driver, and wherein: The analog drivers are stacked vertically on top of the multiple computing tiles; and The pixel array is stacked vertically on top of the analog driver.

Images