Systems and methods for updating an image displayed on a display device

Abstract

Apparatus, systems, and methods are provided for updating an image based on a display device. The display device includes pixels in a pixel array. The system includes a display subsystem for executing commands and displaying images, the display subsystem including: a parser for receiving image frame data, wherein the parser extracts updated image data and commands; a storage device for storing the updated image data in an updated cache memory location according to the commands; a loader for reading the commands to identify and extract the updated image data from the storage device; and a display backplane circuitry for receiving the updated image data from the loader and for updating a pixel driver circuitry for pixels within the updated image data. Embodiments herein are ideal for driving microdisplays such as LCoS micro-LED displays.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 958,627, filed on Jan. 8, 2020, the entire content of which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to displays. More specifically, the present disclosure relates to updating an image displayed on a display device, such as a spatial light modulator and / or a display, such as a liquid crystal display, a liquid crystal on silicon (LCoS) display, an LED display, a micro-LED display, and a microdisplay, which are driven by a display driver circuitry and / or software (e.g., circuitry, logic components, or circuitry, ICs, and / or driver software). Background Art

[0004] In mobile systems such as augmented reality (AR) head-mounted displays, power consumption and battery usage are critical. In these systems, the generation (rendering) of video data on a host (e.g., a video source such as a phone, computer, tablet, or other mobile device), the transmission of the data (e.g., via a cable), and the processing / loading of the video data in a display driver IC and one or more backplane ICs consume power for every bit transmitted or switched in the hardware of the pipeline. These systems send an entire data frame from the host's frame buffer, including both unchanged data and changed data to be displayed. For example, some systems may only render the changing portions of the image being displayed and write these changing portions to the frame buffer on the host, and then send the entire frame to the display subsystem. Additionally, these systems also employ a fixed refresh rate. Partially due to this repetition and the power wasted dedicated to data transmission through the system, known techniques and conventional systems are currently lacking in providing lightweight, long-lasting systems that also minimize battery weight and volume.

[0005] Such known display systems having a data source or host for rendering image frame data render and transmit the entire image frame data from the frame buffer for updating the currently displayed image of the entire display. This includes sending repeated or unchanged data between two image frames. The data can be, for example, video, an image, or image data characterizing an image. The display can be a microdisplay such as an LCD or an LCoS microdisplay, or a larger direct-view display. Summary of the Invention

[0006] In an embodiment of the present disclosure, a display system for updating an image displayed on a display device is provided. The display device includes pixels in a pixel array. The system includes: a display subsystem for executing commands, where the commands indicate how the display subsystem updates the display. The display subsystem includes: a parser for receiving image frame data, where the parser extracts updated image data and commands; a storage device for storing the updated image data in an updated cache memory location according to the commands; a loader for reading the commands to identify and extract the updated image data from the storage device; and a display backplane circuitry for receiving the updated image data from the loader and for updating a pixel driver circuitry for pixels within the updated image data.

[0007] In an embodiment of the present disclosure, a method for updating an image displayed on a display device is provided. The display device includes pixels in a pixel array. The method includes the steps of: receiving image frame data in a display subsystem; receiving the image frame data in a parser, the parser extracting updated image data and associated commands from the image frame data, the image frame data including commands indicating how a loader updates the display; storing the updated image data in an updated cache memory location according to the commands; reading the commands to identify and extract the updated image data from the updated cache memory location; receiving the updated image data from the loader; and updating a pixel driver circuitry for only pixels within the updated image data.

[0008] In an embodiment of the present disclosure, the storage device may include a cache memory. The cache memory or the storage device may be a physical memory such as RAM, EEPROM, etc., or may be a software-based cache memory or storage device.

[0009] In an embodiment of the present disclosure, a display system for updating an image displayed on a display device is provided. The display device has a display including pixels in a pixel array. The system includes: a display subsystem for executing commands, where the commands include: instructions for the display device describing how to update the display according to updated image frame data. The display subsystem includes a parser for receiving the updated image frame data, where the parser extracts updated image data and commands; instructions for the storage device describing how to store the updated image data in an updated cache memory location; instructions for the loader describing how to identify and extract the updated image data from the cache memory location; and a display backplane circuitry for receiving the updated image data from the loader and for updating a pixel driver circuitry for pixels within the updated image data.

[0010] In a system according to an embodiment of the present disclosure, the display backplane portion reloads data only into a portion of the display while keeping the states of all other pixels static. This is achieved by causing the display to load its pixels by addressing groups of pixels (e.g., small rectangular blocks or subsets of pixels in the pixels of the display or a set or subset of blocks of pixels in the pixels of the display) through one or more column-directed load enable signals that define the horizontal extent of columns in the definition block 308 (e.g., control voltages that activate or deactivate (enable or disable) the pixel grouping), the one or more column-directed load enable signals working in combination with a row-directed load signal (e.g., a control voltage) to enable or disable loading new data into the pixels within the block 308 of pixels.

[0011] In an embodiment of the present disclosure, a display system for updating an image displayed on a display device, the display device may include pixels in a pixel array, and the system includes a display subsystem for executing commands, where the commands indicate to the loader how to update the display. The display subsystem includes: a parser for receiving image frame data, where the parser extracts updated image data and commands, and the commands include commands identifying parts of the display to be updated or not updated; a map composed of a set of data, where the set of data includes at least one of coordinates and block positions indicating active and areas to be updated in the display (e.g., coordinates of positions in the display, or positions of pixels or pixel elements in the display, or positions or coordinates of blocks of pixels or pixel elements in the display, or positions in the corresponding image or image data); a storage device for storing the updated image data in the storage device, updated storage locations, or cache memory according to the commands, and the storage device may selectively store or determine where incoming data will be stored based on the content of the map; a loader for reading the commands to identify and extract the updated image data from the storage device, updated storage locations, or cache memory, and the loader may selectively extract data according to the map; and a display backplane circuitry for receiving the updated image data from the loader and for updating the pixel driver circuitry for pixels within the updated image data. In an embodiment of the present disclosure, a display system for updating an image displayed on a display device, the display device includes pixels in a pixel array, and the system includes: a parser for receiving image frame data, where the parser extracts updated image data and commands, and the commands include commands identifying parts of the display for updating; a map composed of a set of data, where the set of data includes at least one of coordinates and block positions of the active area for updating of the display; a storage device for storing the updated image data based on at least one of the commands and the map; a loader for reading the commands, and where the loader identifies the updated image data and extracts the updated image data from the storage device according to the map; and a display backplane circuitry for receiving the updated image data from the loader and for updating the pixel driver circuitry, which uses the updated image data to update the pixels.

[0012] In an embodiment, the command includes a descriptor that identifies which location of the corresponding portion of the image is shown by the updated image data. In an embodiment, the parser performs at least one of the following operations: sending the command to the storage device and / or sending the command to the loader. In an embodiment, the loader reads the descriptor and extracts the updated image data from the updated storage location. In an embodiment, the loader identifies the pixel rows and / or pixel columns in the pixel array corresponding to the location in the descriptor. In an embodiment, the loader queues the reading of the descriptor until the loader extracts data from the storage device. In an embodiment, the loader queues the reading of the descriptor to occur at a predetermined interval. In an embodiment, the system further includes an image source, and wherein the image source identifies the location to be updated in the display, and renders the updated image data only for that location. In an embodiment, the image source is a video data source, and wherein the updated image data is stored in at least one of a memory and a buffer. In an embodiment, at least one of the memory and the buffer is a buffer, and the updated image data stored in the buffer is transmitted to the display system, and wherein the display system includes driver display logic for calculating an occupancy map of the active area of the display based on the updated image data. In an embodiment, the area of the display includes a rectangular tile of at least one of the pixels of the display, the rows of pixels, and the partial pixels of the pixels. In an embodiment, the occupancy map determines the active area by analyzing whether there are any non-black pixels in the area of the display. In an embodiment, only non-black data is written to the storage device. In an embodiment, only the updated image data corresponding to the active area of the display is retrieved from the storage device. In an embodiment, the display system includes a pixel array, and each pixel in the pixel array containing pixels includes: a pixel element coupled to pixel logic circuitry; and a pixel memory coupled to the pixel logic circuitry. In an embodiment, the storage element stores the current gray value of the pixel. In an embodiment, the pixel further includes pixel driver circuitry. In an embodiment, the pixel logic circuitry decodes the current gray value, modulates the pixel driver circuitry, and generates at least one of pixel intensity and brightness. In an embodiment, the memory element includes a latch circuitry for storing the image data for the pixel represented by a plurality of bits. In an embodiment, the system further includes an image source, wherein the image source performs at least one of the following operations: rendering the image frame data and transmitting the image frame data, the image frame data including: updated image data of at least a portion of the image, the updated image data being used to replace the corresponding portion of the image data of the currently displayed image; and an associated command describing how the updated image data replaces the image data. In an embodiment, the storage device is a cache memory.In an embodiment, a method of updating an image displayed on a display device, the display device including pixels in a pixel array, the method comprising the steps of: receiving image frame data in a display subsystem; receiving the image frame data in a parser that extracts updated image data and associated commands from the image frame data, the image frame data including commands that indicate to a loader how to update the display; storing the updated image data in an updated cache location according to the commands; reading the commands to identify and extract the updated image data from the updated cache location; receiving the updated image data from the loader; and updating a pixel driver circuitry for only the pixels within the updated image data.

[0013] The devices, systems, and methods of the present disclosure save power in a display system by optimizing the rendering, transmission, and refreshing of data (e.g., video, images, video data / image data). In this document, embodiments of the present disclosure allow for changing only what is changing at each stage, i.e., selectively refreshing only the portions of data that need to be changed relative to an entire previous frame or subframe and those portions that are non-blank. Thus, only these changing or active portions are rendered, transmitted, and refreshed while other portions of the image or image data remain unchanged or are not changed by the display driver circuitry and / or software. According to embodiments herein, this selective refreshing can occur at one or more stages of a pipeline, i.e., it can occur during host rendering, transmission, and / or updating of the display.

[0014] As a result of minimizing the amount of data sent and subsequently processed, transmitted, and refreshed at multiple stages of a pipeline (e.g., not sending a full video frame), the overall system improves efficiency and reduces the battery size and power consumption of the associated devices using such a display system and method. These savings (e.g., smaller batteries and smaller / lighter mobile systems) are desirable for applications that require longer battery life, such as head-mounted systems including AR, VR, etc. Additionally, the embodiments herein are desirable for driving displays (e.g., liquid crystal displays, LCoS displays, LED displays, micro-LED displays). In embodiments of the present disclosure, a display driver circuit system and / or software may be integrated, included, or coupled to a display to form a single device (e.g., a display device having a silicon backplane including a display driver IC (the display device may be bonded or otherwise coupled to the backplane)). In embodiments of the present disclosure, the display may be separate from the display driver circuit system and / or software (e.g., a display driver IC) and electrically coupled to the display device, for example, via a flexible cable, wire, or electrical connector element. The terms "coupled" and "connected" and their derivatives may be used. It should be understood that these terms are not intended to be synonyms of each other. Rather, in certain embodiments, "connected" may be used to indicate that two or more elements are in direct physical contact with each other. "Coupled" may mean that two or more elements are in direct physical contact. However, "coupled" may also mean that two or more elements are not in direct contact with each other but still cooperate or interact with each other.

[0015] These and other capabilities of the disclosed subject matter will be more fully understood after review of the following figures, detailed description, and claims. It should be understood that the language and terminology used herein is for the purpose of description and should not be regarded as limiting. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present disclosure is illustrated and described with reference to various drawings, in which like reference numerals are used to appropriately represent like system components, and in which:

[0017] Figure 1a A block diagram of a system and its components in accordance with the present disclosure is shown.

[0018] Figure 1b A micro-LED display system of a system incorporating Figure 1a is shown.

[0019] Figure 1c A LCOS display system of a system incorporating Figure 1a is shown.

[0020] Figure 2Illustration of an exemplary image shown in accordance with an embodiment of the present disclosure.

[0021] Figure 3 Illustration of an image divided into an active area and an inactive area in accordance with an embodiment of the present disclosure.

[0022] Figure 4 Shows a method in accordance with an embodiment of the present disclosure. Detailed Description

[0023] Figure 1a Shows an illustrative block diagram of a data source 102 such as a host, e.g., an image, image data, or video source or video data 104. In an embodiment of the present disclosure, the host may include a processor 105, such as an application processor, that executes applications, instructions, or commands 106 (e.g., software) stored in a storage device (e.g., memory device 107) of the host. In an embodiment of the present disclosure, the host may stream image data or an image (e.g., video data) via, for example, a central processing unit, a graphics processing unit, a display processing unit, a DMA engine, a MIPI DSI controller, and a MIPI PHY interface coupled to or included in the host. In an embodiment of the present disclosure, the software module 107a includes the command 106.

[0024] Figure 1b Illustration of a micro-LED display system 180, which may include, in an embodiment, the display system 100 of an embodiment of the present disclosure. The micro-LED display system 180 generally includes a pixel array 140 of individual pixel LED elements 184 disposed on a substrate 182. In an embodiment of the present disclosure, the substrate 182 may be coupled to a driver backplane or backplane IC 186, may be integrated with the driver backplane or backplane IC 186, or may be formed on the driver backplane or backplane IC 186. In an embodiment of the present disclosure, the pixel LED elements are disposed directly on the driver backplane or backplane IC 186. In an embodiment of the present disclosure, the driver backplane or backplane IC 186 includes circuitry for driving the pixel array 140, such as display driver logic 122 or an integrated circuit and / or display backplane logic 130 or an integrated circuit. The display driver logic 122 may include a parser 124 and / or a cache or storage device 153. The display backplane logic 130 may include a loader 162. A descriptor 152 may also be provided. Pixel drive circuitry 170 for the pixels of the pixel array 140 may also be provided.

[0025] Similarly, Figure 1cFIG. 0 is a diagram of an LCOS display system 190 having similar components, and in an embodiment, the LCOS display system 190 may also include the display system 100 of the present disclosure. In particular, the LCOS display device includes a substrate 120, and a display backplane or a display driver backplane IC 186 may be coupled to or integrated with the substrate 120. The substrate 120 is silicon. A layer 196 of a separate reflecting device is arranged, coupled to or integrated with the display backplane 120, and the separate reflecting device may be a separate reflecting device such as a mirror or an aluminum device (represented herein as a single layer). An alignment layer 140 is formed on top of the layer 196. A liquid crystal layer 193, an alignment layer 198, a transparent electrode (usually within an indium tin oxide layer) 199, and a glass or other transparent material layer 197 also form the structure of the display.

[0026] In an embodiment of the present disclosure, the display backplane IC 130 may be included, incorporated, or integrated in the display backplane 120 (the display backplane 120 may be coupled to, for example, a reflecting device 186; 196 in an LCOS display system or a micro-LED in an LED such as a micro-LED display system). The pixel array (i.e., the micro-LED or LED) may be on its own substrate, where each LED is coupled to a pixel driver on the backplane. In an embodiment of the present disclosure, the display backplane 120 is coupled to a display device 180, 190 (e.g., an LCOS device) or an LCOS pixel array 140 or pixel elements (e.g., a reflecting device 186; 196 or an LED or an LED array, such as a micro-LED). In an embodiment of the present disclosure, the display device 100 includes a display backplane 120, a display backplane circuitry 130, pixel circuitry 174 (e.g., as a pixel drive circuitry or pixel circuit), and pixel elements 140a, which are coupled (e.g., electrically coupled) to the DDIC 122, integrated into, included in, and / or positioned within the DDIC 122. In an embodiment of the present disclosure, the display device 100 (including the DDIC 122) is electrically coupled to the DDIC and forms a display subsystem 120, and is assembled together in the display module 100.

[0027] In an embodiment of the present disclosure, the display subsystem 120 includes a display 197. In an embodiment of the present disclosure (such as Figure 1cAs shown in [FIGURE REFERENCE], the display 197 is an array of pixel elements 140 disposed on the backplane 120 or coupled to a backplane circuit system (e.g., a pixel circuit system 170, a pixel drive circuit system, or a drive circuit system incorporated into, integrated into, or coupled to the backplane). In an embodiment of the present disclosure, the display subsystem 120 is an LCoS display subsystem of a liquid crystal on silicon (LCoS) display system 190. In an embodiment of the LCoS system 190 according to an embodiment of the present disclosure, the display 197 is a liquid crystal display or liquid crystal cell (including liquid crystal 193 placed between two substrates 196, 199), electrically coupled to the backplane 120 or backplane IC of the display subsystem or integrated with the backplane 120 or backplane IC of the display subsystem. In an embodiment of the present disclosure, one of the substrates 196 of the liquid crystal display or cell is the backplane 120 or backplane IC. In an embodiment of the present disclosure, the liquid crystal display or liquid crystal cell 193 is included in, integrated in, or electrically coupled to the display backplane integrated circuit 130 (IC) or its substrate. In an embodiment of the present disclosure, the display backplane IC 130 is electrically coupled to one of the sides of the liquid crystal cell, e.g., electrically coupled to the side of the display that does not emit light or light corresponding to an image. In an embodiment of the present disclosure, the liquid crystal cell is coupled to a silicon substrate 182, and the substrate is the same silicon substrate that forms one side of the display backplane IC 120 or a silicon substrate coupled to the display backplane IC (e.g., a substrate such as a silicon substrate including a drive circuit system, a pixel circuit system, pixel circuits 170, and / or drive circuits 166).

[0028] In an embodiment of the present disclosure (such as Figure 1bAs shown in [figure number not provided], the display subsystem 120 is a micro-LED display subsystem 180. In an embodiment of the present disclosure, the display subsystem 100 is a micro-LED system 180. In an embodiment of the micro-LED subsystem 180 according to an embodiment of the present disclosure, the display is an array of LEDs 184 or micro-LEDs, which is electrically coupled to the backplane 120 of the display subsystem 100 or the display backplane IC, or integrated into the backplane 120 of the display subsystem 100 or the display backplane IC, or integrated with the backplane 120 of the display subsystem 100 or the display backplane IC. In an embodiment of the present invention, the array of micro-LEDs 184 is included, integrated into, or electrically coupled to the display backplane integrated circuit (IC) 122 and / or its substrate 182. In an embodiment of the present disclosure, the display backplane IC 122 is electrically coupled to one of the sides of the array of LEDs (e.g., micro-LEDs), such as the side of the array of LEDs (e.g., micro-LEDs) that does not emit light or the light corresponding to the image. In an embodiment of the present disclosure, the array of LEDs 184 (e.g., micro-LEDs) is coupled to the silicon substrate 182, and the substrate is the same silicon substrate that forms one side of the display backplane IC 122, or the silicon substrate 182 that is coupled to the substrate 186 (e.g., the silicon substrate of the display backplane IC) of the display backplane IC 122.

[0029] In an embodiment of the present disclosure, the array of pixels 184 includes pixel elements assembled in the array 140 or pixel elements assembled on the substrate 182 in the array. In an embodiment of the present disclosure, the pixel array 180 is an array of light-emitting diodes (LEDs). In an embodiment of the present disclosure, the LED can be a micro-LED. In an embodiment of the present disclosure, the array 140 of pixel elements 184 is an array of reflective elements 196 that control the birefringence of the liquid crystal 193 in the LCoS device 190 or other liquid crystal devices. In an embodiment of the present disclosure, the pixel element 184 is a reflective element / device, such as a mirror. In an embodiment of the present disclosure, the pixel array 140 or the array of pixels is formed in, incorporated into, integrated into, or coupled to the display backplane IC 122. In an embodiment of the present disclosure, the DDIC or display driver circuitry drives the operation of the pixel elements (e.g., light-emitting diodes (LEDs), micro-LEDs 184, reflective elements) of the array of pixel elements arranged on the backplane 120 of the display subsystem 100.

[0030] In embodiments of the present disclosure, the data source or host can be, for example, a mobile device such as a smart phone, a tablet computer, a laptop computer, a head-up display system, a head-mounted device, or a hybrid device (or some combination of the foregoing devices). In embodiments of the present disclosure, images, image data, and / or video data are received by the real-time data software module 107a and / or the MUX 108 of the host 102, and the real-time data software or software module 107a combines the data with commands parsed from a master sequence file containing a desired drive sequence at the multiplexer (MUX) 108 and / or the real-time data software module 107a. Additionally or alternatively, the MUX 108 can be hardware controlled by the real-time data software module 107a. In embodiments of the present disclosure, the data combined with the commands is combined data output from the data source (e.g., host) 102 and transmitted to the display subsystem. In embodiments of the present disclosure, the combined data is output via the real-time data software module 107a and / or the MUX 108 of the data source 102.

[0031] Compared with known systems, the data source or host 102 of the embodiments herein renders and transmits the changed portions of the data 104 between each frame, rather than all of the image data (i.e., changed data and unchanged data). In an embodiment of the present disclosure, the only image data transmitted by the host is the changed data portion between each frame, and this changed image data is transmitted from the host 102 to the display 197 or display backplane 130 and / or driver circuitry 122 and / or software for driving the display backplane 130 along with one or more commands 106. Such savings can occur at five stages that will be described in more detail below: 1) transmitting data from the host 102 to the display 197; 2) placing data in the cache memory 153; 3) reading data from the cache memory 153; 4) writing data from the cache memory 153 to the display 130 via the display loader 162; and 5) modulating the active pixels of the display 197. Embodiments of the present disclosure take advantage of the nature of images presented in system applications such as AR systems, which typically have sparse image data (since the image is overlaid on the real world), the sparse image data having a high proportion of black background that becomes transparent in an AR headset. According to embodiments of the present disclosure, there may be situations where it may be necessary to refresh the entire display area and all of the data bandwidth on the cable transmitting data to the display, for example, when presenting full-motion video or when the user is moving and the graphics are being updated to align with the user's real-world position. In an embodiment of the present disclosure, the display 197 may show icons, graphics, or text in certain regions (such as strips along the bottom edge, top edge, or side edges of the display, while the central region is blank), and these may be static or occasionally changing, and in these types of situations, according to embodiments of the present disclosure, it is advantageous to send the changed data portion between each frame rather than all of the image data (i.e., changed data and unchanged data), as this saves power, for example. According to embodiments of the present disclosure, the data 104 may also be a moving symbol, icon, or indicator, but is small relative to the entire display area and thus only requires updating a subset of that area. For the case of small-area refreshes, the refresh rate can also be increased in cases where the motion needs to be fast to keep up with changes in the environment, since the available time and transmission bandwidth can be dedicated entirely to the changing portion of the display. Embodiments of the present disclosure address the minimization of data processing, transmission, and refresh at multiple stages of the pipeline. In one embodiment, the entire display is refreshed without refreshing the entire cache memory, while in another embodiment, the entire display is refreshed starting from the cache memory.In an embodiment of the present disclosure, the loader 162 may extract or read only the changed data from the cache memory 153 and refresh or update the display 197. Alternatively, for example, the loader 162 may utilize or read all of the content or image data in the cache memory 153 - rather than reading only the changed data from the cache memory 153 - to refresh or update the display 197 to update or refresh the display 197.

[0032] The display backplane portion 130 of the display subsystem 120 according to an embodiment of the present disclosure incorporates the ability to reload data only into a portion of the display while keeping the states of all other pixels 140 static. This is shown in Figure 3 and is achieved by column - directed load enable signals or control voltages that are enabled or disabled for each group of columns 301 that define the horizontal extent of columns in the defined block 308, such that the display 300 loads its pixels by addressing small rectangular blocks 308 or subsets of pixels within the pixels 140a, 140b of the display or sets or subsets of blocks of pixels within the pixels of the display, and the column - directed load enable signals or control voltages that are enabled or disabled work in combination with row - directed load signals or control voltages to enable or disable the loading of new data into the pixels within the block 308 of pixels. In an embodiment of the present disclosure, the pixels 140a incorporate sufficient memory 174 (e.g., pixel circuitry 170 such as a latch or latch circuitry) to fully describe the gray - scale value of the pixel and maintain the state of the static (unchanged) pixels 140a frame - by - frame.

[0033] Turning back again to Figure 1a , software applications on the host 102 (i.e., the host that is the source of image data such as images, videos, image data, and / or video data 104) render the portion of the image (or the entire image) that needs to be changed on the display 197 in the image data (e.g., within the frame buffer). In an embodiment of the present disclosure, the image data is formatted. In an embodiment of the present disclosure, such image data is formatted by combining, appending, or integrating the image data with a command 106 (e.g., a descriptor or image descriptor indicating the size and location of the image data to be changed) and transmitting the image data to the lower layer of the transport protocol 110 (e.g., MIPI) as if it were pure video data 104 (i.e., without any indication that the image data includes the command 106 or command data). In an embodiment of the present disclosure, the command 106 may not be processed subsequently after transmission until the command 106 is parsed out of the transmitted data or image data by the parser 124.

[0034] In an embodiment of the present disclosure, commands 106 such as descriptors (e.g., region descriptors) are used to identify which pixels and / or pixel blocks are to be reloaded (i.e., the changed data), and the host 102 can send the changed data (along with new image and / or video data 104) to the display subsystem 120 and / or the display loader logic 162 (i.e., loader circuitry and / or software) via the transmission channel device 110. The transmission channel (e.g., cable, physical layer electronics, and higher layers of the transmission protocol) 110 can be kept standardized for interoperability; however, the content of the data stream is changed to include both commands 106 and image and / or video data 104 that are disguised as image and / or video data 104 (i.e., pure image and / or video data 104 or data for which the commands 106 included therein are not recognized). Implementations of the software on the host 102 and the parser 124 in the display driver logic 122 (or a separate display driver IC) agree or operate according to the formats of both the image / video data packets and the commands 106 (i.e., the same format or a compatible format) and the refresh rate of the system 100, such commands as block descriptors 155 that provide the size and location of the regions to be refreshed. In an embodiment of the present disclosure, the host 102 and the parser 124 are designed to agree. In an embodiment of the present disclosure, the parser 124 can be hardware and / or software.

[0035] In the display subsystem 120 (or a monolithic display IC), a parser 124 parses the received data stream 110 to extract image data 152 (e.g., blocks of image or video data 104) and its accompanying commands 106 or identifiers (e.g., descriptors 155). In embodiments of the present disclosure, the descriptor 155 is used to identify where new data 104 will be stored in the local cache 153. In embodiments of the present disclosure, the descriptor 155 is sent to both the cache 153 and the loader 162 such that the driver logic (i.e., driver circuitry 122 and / or software) knows where to place the changed data and such that the loader 162 knows where to read the changed data from the cache 153 when it needs to write the changed data to the pixel array 140. In embodiments of the present disclosure, the cache 153 (which may be before the display loader 162) is used to stage or store image and / or video data 104 before the image and / or video data 104 is loaded onto the display 197. In embodiments of the present disclosure, only the updated portions, segments, or addresses in the cache 153 are those portions of the content of the cache 153 that correspond to or are related to the updated display area as identified or set by the descriptor 155 (e.g., a region descriptor). In some embodiments, only a portion of the overall scheme is used. For example, in an embodiment, the host 102 may identify the area in the display 197 to be updated and only send data for that area, and the cache 153 will be rewritten, and the display 197 may be set to start refreshing from the cache 153 if, for example, a multi-color display needs to sequentially display data from 3 separate colors (color sequence operation).

[0036] In an embodiment of the present disclosure, the parser 124 sends the descriptor 155 to the cache memory 153 and also to the loader 162, and the loader uses the descriptor to identify the block address 164, reads or extracts the changed data in the cache memory 153 from the block address 164 (such that the loader 162 can load the changed data into the pixel array 140), and then the loader uses the descriptor 155 to identify the rows 303 and columns 301 in the arrays 140, 300 for loading the data (such as image data). In an embodiment of the present disclosure, the descriptor 155 can be queued for the time when the loader 162 will extract data from the cache memory 153 for loading into the display pixels 140 (i.e., the descriptor 155 is placed in the buffer memory in the same order as the order of the regions to be updated on the pixel array or pixel elements of the display or the display 197). The queuing can occur at a predetermined interval. In an embodiment of the present disclosure, there can be a fixed refresh rate, where all the data to be displayed in the next frame time arrives at the cache memory 153 for updating the display 197 according to the refresh rate. In contrast, the system 100 according to an embodiment of the present disclosure can periodically update the display 197 or the pixel array or pixel elements of the display 197 at a predetermined interval (and the timing of such intervals can vary). Alternatively, in an embodiment of the present invention, the system 100 according to an embodiment of the present disclosure can push out the changed data when the display driver logic (i.e., the display driver circuitry 122 and / or software and / or the parser 124) receives the changed data.

[0037] In another embodiment of the present disclosure, the host processor 105 only renders the active or changed portions of the image into the video buffer, but for convenience, the entire video buffer is transmitted to the display 197, and the display driver logic 122 calculates an occupancy map 300 of the regions of the display based on whether there are any non - black pixels in each region 308, such as portions of rectangular tiles 308 or lines 303. This is in Figure 3Shown in. The occupancy map 300 may replace the descriptors 152 of the occupied and changed areas sent from the host 102. Only non-black data 303, 305, 308 are written to the cache 153, and only the areas 304 marked as active in the occupancy map are retrieved from the cache 153, and only the areas 304 marked as active are written to the display pixel memory 174. In an embodiment of the present disclosure, the occupancy map 300 is a collection of data and can be accessed by display driver software and / or circuit systems / hardware. In an embodiment, the occupancy map 300 can determine the active areas 304 (i.e., the active regions) by analyzing whether there are any non-black pixels in the respective areas of the display 197.

[0038] By using the systems and methods of the embodiments herein, power consumption is saved by only rendering, transmitting, and refreshing the data of the portions of the display that need to be changed while leaving the other portions unchanged, rather than sending a complete video frame. This also results in a higher frame rate compared to conventional systems and methods.

[0039] In one embodiment, each pixel 140a, 140b in the pixel array 140 includes a memory element 174 (e.g., a latch), logic 172 (e.g., hardware such as combinational gates, transistors, etc.), and pixel drive circuitry 176 (such as a horizontal shifter). Alternatively, one or more of each of the pixels 140b may include only the memory element 174 and pixel drive logic 172. This is shown in FIG. 1 by exploded views of two alternative pixel blocks 140a, 140b. The pixels themselves incorporate sufficient memory 174 to fully describe the gray-scale value of the pixel, which then enables the pixel value to remain static without load activity when the image does not change. In other words, there is sufficient memory 174 to store the information in the pixels 140a, 140b such that the pixels 140a, 140b maintain the correct color or brightness frame by frame without being refreshed. The logic 172 in the pixels 140a, 140b is used to decode the stored gray-scale value and modulate the pixel driver output 176 (e.g., set an analog value, or a series of binary values that give the desired light intensity during the course of a frame). In an embodiment of the present disclosure, the memory element 174 may be a latch or a latch circuit system that stores image data represented by multiple bits.

[0040] The pixel logic 172 can incorporate the information received from the G bus and determine which waveform to output to the pixel array 140 based on the image data. See, for example, FIGS. 10-11 of PCT file PCT / US2021 / 012262 titled "Dynamic Pixel Modulation", the entire content of which is incorporated herein by reference in its entirety as part of the present disclosure.

[0041] Figure 2 Shown is a display area 200 that would be viewed in an exemplary and non-limiting system application (i.e., an AR head-mounted display) according to embodiments herein. (An alternative could be a display for a vehicle's HUD). In embodiments of the present disclosure, Figure 2 Shown is an image 210 that is overlaid on the real world as seen by a user of a device (e.g., an AR head-mounted device). In Figure 2 the example, since the viewer is looking at the real world with information overlaid, most of the image 210, 214 is clear (no light is reflected from the display into the eye) and does not need to be updated. The moving indicator / symbol 212 occupies a small moving area and needs to respond to the external movement of the wearer in their environment. Shown is a strip area 220 with a fixed position in the image 210, and the content of this strip area 220 only needs to be updated as needed (as frequently as needed). For example, the update may occur once per second for a clock, while it may be very slow for a battery indicator 222. In embodiments of the present disclosure, the system 100 according to embodiments of the present disclosure will be effective in displaying changes to an image overlaid on the content seen by the viewer in the real world because the system will only update the changing parts of the overlaid image, such as Figure 2 the moving indicator arrow in

[0042] Figure 3 Shown is an exemplary sparse image 300 that is subdivided into regions 302, 303, 305, 307, 308 of a fixed size, which are marked as active 304 (containing non-black image data) or passive 302. For illustrative purposes, the background of the regions containing non-black pixels is shown as white. The occupancy map 300 marks the active regions in such a way that only the data of these regions will be stored in the cache 153, retrieved from the cache 153, and loaded into the memory 174 of each pixel.

[0043] As Figure 4As shown, a method 400 for operating a display system 100 according to an embodiment of the present disclosure is described. In an embodiment of the method for operating a display system, the method may include a step 410 of providing the display system 100. According to an embodiment of the present disclosure, a subsequent step 420 may include sending image frame data from an image or video data source 120, wherein the image frame data includes: updated image data 104 of at least a portion of an image 200, which is used to replace a corresponding portion 210 of the image data of the currently displayed image 200. Associated commands 106 (e.g., commands for updating aspects of a display drive sequence such as timing parameters) provide a descriptor of a region and / or describe how the updated image data replaces the image data. After being presented by the source 120 in the foregoing manner, the method may include a step 430 of receiving the image frame data in a display subsystem 120. The image frame data may be received and parsed by a parser 124, which extracts the updated image data 104 and the associated commands 106 from the image frame data in step 440 and stores the updated image data 104 in a storage device or cache 153. Then, in step 450, the content of the storage device or cache 153 (e.g., the updated image data 104) may be sent to a loader according to the associated commands 106. The loader 162 may then read 460 the associated commands 106 to identify and extract the updated image data 104 from the cache 253; and subsequently, a pixel array 140 receives 470 the updated image data 104 from the loader 162; and in step 480, pixel driver circuitry is updated for pixels only within the updated image data.

[0044] The subject matter described herein can be implemented in digital electronic circuitry, or in computer software, firmware, and / or hardware, including the structural means disclosed in this specification and their structural equivalents thereof, or in combinations of them. The subject matter described herein can be implemented as hardware, one or more computer program products tangibly embodied in an information carrier (e.g., a machine-readable storage device) and / or implemented in a propagated signal (e.g., a voltage waveform or a current waveform) for execution by a data processing apparatus (e.g., a programmable processor, a computer, or multiple computers) or to control the operation of a data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). A computer program (also referred to as a program, software, software application, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file. A program can be stored in a part of a file that holds other programs or data, in a single file dedicated to the program being discussed, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[0045] The processes and logical flows described in this specification, including the method steps of the subject matter described herein, can be performed by one or more programmable processors executing one or more computer programs to perform the functions of the subject matter described herein by operating on input data and generating output. The processes and logical flows can also be performed by dedicated logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), and the apparatus of the subject matter described herein can be implemented as dedicated logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0046] By way of example, processors suitable for executing computer programs include both general and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing the instructions and data. Generally, a computer will also include or be operatively coupled to one or more mass storage devices such as magnetic disks, magneto-optical disks, or optical disks for storing data from which it receives or to which it transfers data or both. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, by way of example including: semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and optical disks (e.g., CD and DVD disks). The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0047] The subject matter described herein may be implemented in a computing system that includes a back-end component (e.g., a data server), a middleware component (e.g., an application server), or a front-end component (e.g., a client computer mobile device, a wearable device having a graphical user interface or a web browser through which a user may interact with an implementation of the subject matter described herein), or any combination of such back-end, middleware, and front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”) such as the Internet.

[0048] It should be understood that the disclosed subject matter is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The disclosed subject matter is capable of other embodiments and of being practiced and carried out in various ways. Further, it should be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting. Thus, those skilled in the art will recognize that the concepts upon which this disclosure is based may readily be utilized as a basis for designing other structures, methods, and systems for carrying out some of the purposes of the disclosed subject matter. Accordingly, it is important that the claims be regarded as including such equivalent constructions insofar as they do not depart from the spirit and scope of the disclosed subject matter.

[0049] Although the disclosed subject matter has been described and shown in the foregoing exemplary embodiments, it should be understood that the present disclosure is by way of example only and that many changes may be made to the details of the implementation of the disclosed subject matter without departing from the spirit and scope of the disclosed subject matter, which is limited only by the appended claims.

Claims

1. A display system for updating an image displayed on a display device, the display device including pixels in a pixel array, the system including: A display backplane circuitry including display driver logic, the display driver logic including a parser adapted to receive image frame data, wherein the display driver logic is configured to compute a map of regions of the display, the map consisting of a set of data including at least one of coordinates and block positions of an active region of the display for updating, wherein the active region of the display is determined by analyzing whether any non-black pixels are included in each region of the display in the image frame data, the active region being a region where non-black pixels exist; A storage device adapted to store updated image data corresponding to the active region of the display, wherein the parser extracts the updated image data corresponding to the active region of the display from the image frame data and writes only the updated image data to the storage device based on the map; and A loader that identifies the updated image data corresponding to the active region of the display and extracts only the updated image data from the storage device according to the map, wherein The display backplane circuitry is configured to receive only the updated image data corresponding to the active region of the display from the loader and to update a pixel driver circuitry, the pixel driver circuitry using only the updated image data to update the pixels.

2. The system according to claim 1, further comprising an image source, and wherein, The image source identifies a location on the display to be updated and renders only updated image data for the location.

3. The system according to claim 2, wherein The image source is a video data source, and wherein the updated image data is stored in at least one of a memory or a buffer.

4. The system according to claim 3, wherein, The at least one of the memory or the buffer is a buffer, and the updated image data stored in the buffer is transmitted to the display system.

5. The system according to claim 4, wherein, The regions of the display include a rectangular tile of at least one of the pixels of the display, rows of pixels, or partial pixels of the pixels.

6. The system according to claim 1, wherein, The display system includes a pixel array, and each pixel in the pixel array includes: A pixel element coupled to a pixel logic circuitry; and A pixel memory coupled to the pixel logic circuitry.

7. The system according to claim 6, wherein The storage device includes storage elements for storing current gray scale values of the pixels.

8. The system according to claim 7, wherein, Each of the pixels further includes a pixel driver circuitry.

9. The system according to claim 8, wherein, The pixel logic circuitry decodes the current gray scale value, modulates the pixel driver circuitry, and generates at least one of pixel intensity or luminance.

10. The system according to claim 9, wherein, Each of the memory elements includes a latch circuitry for storing image data for the pixel represented by a plurality of bits.

11. The system according to claim 1, wherein The storage device is a cache memory.

12. A method for updating an image displayed on a display device, the display device including pixels in a pixel array, the method including: Receiving image frame data in a parser; A diagram of the area of the display is calculated by display driver logic including the parser, the diagram consisting of a set of data including at least one of coordinates and block positions of an active area of the display for updating, wherein the active area of the display is determined by analyzing whether any non-black pixels are included in each area of the display in the image frame data, and the active area is an area where non-black pixels exist; The parser extracts updated image data corresponding to the active area of the display from the image frame data and writes only the updated image data to a storage device based on the diagram; The updated image data corresponding to the active area of the display is stored in the storage device; The loader identifies the updated image data corresponding to the active area of the display; The loader extracts only the updated image data from the storage device according to the diagram; A display backplane circuit system including the display driver logic receives only the updated image data corresponding to the active area of the display from the loader; and The pixel driver circuit system is updated for pixels using only the updated image data.

Images