Systems, methods, and devices for providing a sequence-based display driver

Abstract

A display driver device (210) receives downloadable "sequences" for dynamically reconfiguring the displayed image characteristics in an image system. The display driver device includes one or more storage devices, such as a memory device (218) for storing image data and a memory device (219) for storing a portion of a drive sequence, which are downloaded and / or updated in real time according to various inputs (214).

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63024637, filed May 14, 2020, entitled "SYSTEMS, METHODS AND DEVICES FOR PROVIDING SEQUENCE BASED DISPLAY DRIVERS", the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to spatial light modulators, displays, and / or microdisplays. More particularly, the present disclosure relates to systems and methods for providing digital display driver circuitry and software modules for digital spatial light modulators, displays, and / or microdisplays including, but not limited to, digital displays, digital liquid crystal (LC) displays, digital liquid crystal on silicon (LCoS) displays, organic light emitting diode displays (OLEDs), and micro - versions (i.e., microdisplay versions) of the above - mentioned types of displays. Background Art

[0004] LCoS displays and microdisplays are used in many different applications. These can range from high - brightness projection systems to augmented reality (AR) or virtual reality (VR) headsets to phase - mode scientific applications. These different applications can place widely different and sometimes unexpected requirements on their display driver integrated circuit (DDIC) functions such as signal frequency, timing, detailed sequencing, and display - or application - specific data formatting.

[0005] Conventional LCoS displays and their associated DDIC chips typically have hard - coded drive algorithms. Generally, known DDIC chips perform the same sequence of operations when driving an attached display, and thus there is very little flexibility when the desired application is different from what the manufacturer originally anticipated. In particular, it is typically not possible to change characteristics such as the grayscale algorithm used, frame rate, bit depth, color sequence (in color - sequencing applications), timing of illumination, or other operational aspects. These display characteristics can be used to control the brightness, resolution, depth perception, and other visual effects of the displayed image.

[0006] When rendering image data in a display device of the prior art, the display device typically has few options for modifying the characteristics of the displayed image. These display devices generally only accept incoming video data and write it to a pixel array in the display. Such display devices typically have a fixed data format that they require from the display driver. This fixed data format may not be an industry-standard video data format, so the display driver needs to reformat the video data using dedicated hardware (logic) to match the requirements of the display. For many prior art displays, there are custom-designed display drivers that are designed to perform this reformatting. When creating a new or improved display design, it is often necessary to redesign the display driver as well.

[0007] When using some existing image rendering techniques to reconfigure display characteristics (e.g., frame rate, brightness of the display, etc.), these existing image rendering techniques involve turning off the display or otherwise interrupting the rendering of the content being displayed in order to reconfigure or update the display characteristics. Sometimes, these changes may take a time amount of several seconds and generally require temporarily terminating the image rendering. Thus, the prior art for reconfiguring display characteristics in an image system may not be able to achieve real-time and / or dynamic reconfiguration of display characteristics, e.g., while the image data is being rendered. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In this document, the present disclosure is illustrated and described with reference to the various drawings, in which like reference numerals are appropriately used to denote like system components, and in the drawings:

[0009] Figure 1 is a block diagram of an overview of a display system according to an example embodiment;

[0010] Figure 2 is a general block diagram of a display driver according to an example embodiment;

[0011] Figure 3 is a method performed by a display driver according to an example embodiment;

[0012] Figure 4 is another method performed by a display driver according to an example embodiment;

[0013] Figure 5 is a detailed block diagram of a display driver according to an example embodiment;

[0014] Figure 6 are instructions for a drive sequence according to an example embodiment;

[0015] Figure 7 is a detailed block diagram of another display driver according to an example embodiment; and

[0016] Figures 8A to 8B Depicts the pulse width modulation and pixel information of a pseudo-code display sequence according to an exemplary embodiment. Detailed implementation

[0017] Embodiments of a display driver device including circuitry, hardware, and / or software in an image system are disclosed herein. The display driver device utilizes a downloadable "sequence" to dynamically reconfigure the displayed image characteristics in the image system. The display driver device may be configured with one or more storage devices such as memory devices for storing image data, e.g., static or still images or dynamic / motion images (e.g., video data). The display driver device also includes a dedicated storage device for storing the "sequence". The sequence details a series of consecutive actions that occur in the display driver starting at each Vsync. These Vsync actions control the movement of image data from the input of the display driver into and out of various cache memories and to the output of the display driver. The sequence includes instructions that can modify the image data in a flexible manner and may also include instructions that can send a Serial Peripheral Interface (SPI) sequence out of the display driver circuitry to control other external devices (e.g., light sources). As further described below, additional actions that support the display operation may be included in the sequence.

[0018] As display designs continue to evolve, user expectations regarding the quality of the displayed images have increased significantly. Display characteristics can determine how a user perceives image data, but there are typically adjustments to the image data that can enhance or otherwise modify that user experience. The display driver device has characteristics that can be manipulated to enhance or otherwise change the way a user will experience the image data rendered on a display. The display driver device of the present disclosure is capable of dynamically (e.g., in real-time and without interruption) reconfiguring the display characteristics of the image data rendered in a display device of an image system in order to, for example, effect these enhancements. Hereinafter, the terms "driver", "display driver device", and "display driver" may be used interchangeably.

[0019] To illustrate, consider the advantages of dynamically reconfiguring display characteristics in a display driver implemented in an augmented reality (AR) headset. When a user wears an AR headset, the headset typically superimposes graphics, text, instructions, controls, or other information (i.e., overlay data) on an image or video of the user's real-time environment. The real-time environmental data can be captured by imaging (e.g., via a still, video, panoramic, or other imaging device), and when the user moves their head left, right, up, or down, the image overlay data is also updated such that the overlay data is translated left, right, up, or down in the user's environment accordingly. When the user moves their head left, right, up, or down, the real-time environmental data can also be updated. Using the ability to dynamically reconfigure display characteristics (e.g., the display characteristics of a still or video image), the AR headset can dim the areas of the user's eyes that are out of focus (e.g., change the brightness or gray level of a group of pixels) and can increase the brightness of the areas of the user's eyes that are in focus.

[0020] Similarly, using a display driver according to the disclosed embodiments, the AR headset can reduce the resolution and / or frame rate of the image data in the areas of the user's eyes that are out of focus and can increase the resolution and / or frame rate of the image data in the areas of the user's eyes that are in focus. Since the reconfiguration of the display driver characteristics is performed dynamically and does not interrupt the display of image content to the user, the reconfiguration of the display characteristics can appear seamless to the user and can be used to improve the overall visual experience quality of the user. Additionally, as a tangential benefit, adjusting the brightness, gray level, resolution, and / or frame rate of the focus or position (corresponding to the pixels of the headset display) of the headset display according to the user's preferences (e.g., pre-determined or based on data about the user's actual, known, or expected environment) can result in reduced power consumption of the headset display and / or improved visibility of the image portions focused by the user of the headset display. These example features are then described in detail in the context of embodiments of dynamically reconfiguring display driver characteristics by updating one or more memories of the display driver with image data and a drive sequence for processing the image data and configuring the display device. This reconfigurability of the display driver also enables it to modify the display data when necessary to correct for the undesirable effects of display temperature changes, ambient light changes, or other external factors. Another tangential benefit of this reconfigurability is that a common display driver can be designed for more than one display device and / or for subsequent generations of display devices.

[0021] In particular, a display driver or device (including, for example, display driver circuitry and / or display driver software) is configured to have the ability to receive (or "download") a "drive sequence", which can be incorporated into an instruction block, a downloadable file, or dedicated software code. The display driver can include a display driver integrated circuit (DDIC). Alternatively, the display driver can be incorporated or integrated into, for example, an application specific integrated circuit (ASIC) or similar circuitry. According to embodiments disclosed herein, one or more drive sequences are loaded from an external controller device such as a graphics processing unit (GPU), internal or external memory, or other processor of a host system into the display driver. In an example embodiment, one or more drive sequences are loaded into the display driver circuitry. In an example embodiment, the display driver circuitry includes a DDIC or is a DDIC. In an example embodiment, one or more drive sequences are loaded into the display driver circuitry via, for example, a DDIC. In an example embodiment, one or more drive sequences are loaded directly into the DDIC.

[0022] As further described herein, according to the present disclosure, a "drive sequence" represents a list of one or more operations or encoded instructions by which the detailed operation of a display driver is determined or changed. By defining this detailed operation of the display driver, the display operation mode, power level, timing characteristics, and details of image rendering applied to the display device can be (directly or indirectly) manipulated so that the display device displays image data in a specific manner. The drive sequence will hereinafter be referred to as the "sequence", "main sequence", and / or "sequence list". In an example embodiment, one or more drive sequences are loaded into the display driver after the display driver is powered on. In an example embodiment, one or more drive sequences are pre-loaded into one or more memories of the display driver. Once loaded, the drive sequence can be executed starting from each new data frame, or sub-frame, or vertical synchronization (Vsync) event of a data source (e.g., GPU, digital camera device, video recording, or other source of images or video images). Additionally, the drive sequence can include "opcodes" and various data parameters that determine what data (e.g., video image data, commands, register writes, and / or other data) is sent to the display device of the display system, when the data is sent, and what manipulation or other filters are applied to the data (e.g., to provide the desired modulation of LCoS). The drive sequence also enables driving external control outputs from the display driver such as laser / LED enable, and allows any serial peripheral interface (SPI) commands (embedded in the instructions of the drive sequence) to be transmitted to the backplane and other system chips (e.g., analog support chips, laser driver devices) at different time points as desired by the operator. Since the drive sequence can coordinate the execution of the commands and instructions in the drive sequence, these commands and instructions can be in the form of an instruction table or list, and since each instruction or event in the sequence includes an execution time (e.g., relative to Vsync), the events occur at a controlled timing. This timing can be defined by the user or system designer. Additionally, the timing is implemented by one or more time base circuits incorporated into the display driver, which enables the execution of one or more drive sequences (and the instructions therein) to be synchronized with one or more internal times (e.g., "ticks") as well as video synchronization (VSync) events (e.g., frame intervals). For the purposes of the present disclosure, VSync is a standard video interface that includes 27 lines, including 8-bit red data (represented as R[7:0]), 8-bit green data (represented as G[7:0]), 8-bit blue data (represented as B[7:0]), a video clock associated with this data (commonly represented as "CLK"), a vertical synchronization pulse that occurs at the start of each frame (represented as "VSync" or "VS"), and a horizontal synchronization pulse that occurs at the start of the data for each line (represented as "Hsync" or simply "HS").In addition, a "Vsync event" can include detecting a pulse on the Vsync line. Alternatively, the Vsync pulse can be encoded as a data packet or transmitted via other means. However, generally speaking, a Vsync event or pulse can be included in most different types of video interfaces and can take different forms as long as it performs the function of indicating to the interface the start of a new data frame. For example, in a 60HZ video, the Vsync pulse is transmitted 60 times per second.

[0023] The exemplary embodiments are described herein in the context of driving a spatial electromagnetic radiation (e.g., light) modulator including, but not limited to, a display and a microdisplay, and providing a sequence-based circuit system and / or software module for a driver system or device that can be within, for example, a display system such as a digital display system. Although LCoS displays are used herein for exemplary purposes, those of ordinary skill in the art will understand that the disclosed embodiments include and are applicable to other types of digital display systems, including but not limited to digital liquid crystal (LC) displays, organic light emitting diode displays (OLEDs), micro-LED displays, and the like.

[0024] In the following detailed description, reference is made to the accompanying drawings, which form a part of the detailed description and in which are shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope. Accordingly, the following detailed description should not be taken in a limiting sense, and the scope of the embodiments is defined by the appended claims and their equivalents.

[0025] The various operations may be described serially in a number of discrete operations in a manner that helps to understand the embodiments; however, the order of description should not be construed as implying that these operations are order-dependent.

[0026] Descriptions may use perspective-based descriptions, such as up / down, back / front, and top / bottom. Such descriptions are only used to facilitate the discussion and are not intended to limit the application of the disclosed embodiments.

[0027] The terms "coupled" and "connected" and their derivatives (e.g., "communicatively coupled") may be used. It should be understood that these terms are not intended as synonyms for each other. Rather, in a particular embodiment, "connected" may be used to indicate that two or more elements are in direct physical contact with each other. "Coupled" may mean that two or more elements are in direct physical contact. However, "coupled" may also mean that two or more elements are not in direct contact with each other but still cooperate or interact with each other.

[0028] For purposes of description, a phrase of the form "A / B", "A or B", or "A and / or B" means (A), (B), or (A and B). For purposes of description, a phrase of the form "at least one of A, B, and C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). For purposes of description, a phrase of the form "(A)B" means (B) or (AB), i.e., A is an optional element.

[0029] These descriptions may use the terms "embodiment" or "each embodiment", each of which may refer to one or more of the same or different embodiments. Additionally, terms such as "comprising", "including", and "having" as used with respect to embodiments are synonyms and generally are to be construed as "open" terms (e.g., the term "including" should be interpreted as "including but not limited to", the term "having" should be interpreted as "having at least", and the term "comprising" should be interpreted as "including but not limited to", etc.).

[0030] Regarding the use of any plural and / or singular terms herein, those skilled in the art can convert from plural to singular and / or from singular to plural according to the context and / or application. For clarity, various singular / plural permutations may be set forth explicitly herein.

[0031] Various embodiments are now described with reference to the accompanying drawings, wherein like reference numerals are always used to refer to like elements. In the following description, for purposes of explanation, numerous specific details are set forth in order to facilitate a thorough understanding of one or more embodiments. However, in some or all cases, it is apparent that any of the embodiments described below may be practiced without the use of the specific design details described below.

[0032] Figure 1FIG. 0 shows a simplified view of an image system 100 configured to dynamically configure and / or reconfigure display characteristics in accordance with embodiments of the present disclosure. To support dynamic reconfiguration of display characteristics, the image system 100 may include a controller 140, a display driver 110, and a display device 150. The controller 140 may include a graphics processing unit (GPU) and may be configured to combine image data 143 with one or more drive sequences 142 into an image data frame 141 and / or sub-frames. As used herein, a "sub-frame" refers to a situation where more than one version of an image is sent to a display during one image frame. For example, in a "color sequencing" system, all of the red information in an image is typically packed together and sent to the display as a separate sub-frame. This is then typically followed by a green sub-frame and then a blue sub-frame. These sub-frames occur in such a short period of time that the eye blends the colors together and perceives the image as a full-color image. Those of ordinary skill in the art will understand that the colors may be different. Those of ordinary skill in the art will also understand that all of the colors need not be different from one another (i.e., one color may be the same as another or more of the colors). As discussed herein, one or more drive sequences 142 may include settings for reconfiguring, updating, establishing, initiating, adjusting, changing, and / or modifying the data transmitted to the display device 150 and / or the image data 113 displayed thereon. The image data 143 is merged or combined with one or more drive sequences 142 into the image data frame 141, and thus, the settings included in the one or more drive sequences 142 may be transmitted to the display driver 110 without interrupting the transmission of the image data 143. Alternatively, the drive sequence 142 may be transmitted to the display driver 110 via a separate interface such as a Serial Peripheral Interface (SPI). Those of ordinary skill in the art will understand that other interfaces may be used. The image data frame 141 may be formatted in accordance with, for example, one or more MIPI ("Mobile Industry Processor Interface") or modified MIPI interfaces or communication protocols. The controller 140 may be configured to send the image data frame 141 to the display driver 110 via a communication channel (e.g., a conductive bus, a network, a wireless interface, etc.). The controller 140 may include additional features and may be configured to define or select one or more drive sequences 142 based at least in part on information received from one or more sensors 160.

[0033] For example, the displayed image may benefit from changes in the drive sequence based on the temperature of the display. As is well known, especially liquid crystal displays are extremely sensitive to temperature and may require adjustments to the display voltage or timing to compensate for the actual device temperature and maintain image quality. In such cases, the controller 140 may be programmed to periodically command a temperature measurement of the display 150, read the result via the sensor input 160, and use an internal software "look-up table" or equivalent (e.g., another cross-referenced method) to determine the best sequence 142 for that temperature from a list (stored in local memory). This new sequence would then be transmitted to the display as described above. In another example, the controller 140 may be configured to brighten or dim the display in response to ambient lighting - in this case, via a light sensor attached to the controller 140. As previously mentioned, the ambient lighting can be used to select one of the stored available drive sequences 170 that will be transmitted to the display driver 110.

[0034] The display driver 110 may be configured to operate the display 150 using the image data frame 141 received from the controller 140. The display driver 110 may use information (e.g., one or more drive sequences 142 contained in the image data frame 141) to operate the display 150. The controller or GPU 140 reads the drive sequences from memory or an external storage device as needed to update the stored drive sequences 142. The display driver 110 may separate or parse the image data 143 and one or more drive sequences 142 from the image data frame 141. The display driver 110 may temporarily store the image data 143 and one or more drive sequences 142 in one or more storage devices of the display driver 110, as further described herein, e.g., with reference to Figure 2。The display driver 110 can configure the formatting of the display data of the display 150 using display characteristics included within one or more drive sequences 142, and the display driver 110 can provide image data 143 to the display 150 for display along with the display drive characteristics from one or more drive sequences 142. The drive sequence used at any given time will specify the image data, and, if necessary, the time and manner in which a new sequence is sent to the display. New sequence data is typically sent to the display between blocks of image data, or in some embodiments the new sequence is appended to the front of a data transfer block (e.g., for the transfer of image data, video data, and / or sequences or sequence data transfer blocks). In systems that use a MIPI interface or other packet-based video interface to input video data, special packets can be defined and used for non-image data, such as sequence data. By receiving, parsing, and applying one or more drive sequences 142 for the display 150, the display driver 110 supports dynamically reconfiguring the display characteristics with the display 150. The display driver 110 can include additional features to facilitate dynamically reconfiguring the display characteristics with the display 150, as further described herein. These additional features can include alternative interfaces for configuring and sequence data, such as SPI. They can also include a "dual buffer" or "alternate" version of the internal configuration memory in the display controller such that one configuration memory can be updated while the alternate version is in use. Then, internal multiplexing will be used to swap or replace the newly updated version for the currently in-use version at the next Vsync (vertical sync) event.

[0035] The display driver 110 implements a personalized implementation of the selection and / or definition of one or more drive sequences 142. Because the display driver 110 can be configured to receive and interpret the display drive characteristics or the displayed / display image parameters (e.g., resolution, power level, etc.) defined by one or more drive sequences 142, developers can create unique applications that define one or more drive sequences 142. In other words, the controller 140 can be implemented as a process, software application, and / or circuitry independent of the display driver 110, allowing one or more developers to update one or more drive sequences 142 according to their preferences. This feature of the display driver 110 and one or more drive sequences 142 enables different and customized applications for the dynamic reconfiguration of the displayed / display image characteristics supported by the image system 100.

[0036] The operations of the controller 140 and / or the display driver device 110 may be controlled by one or more processors, which are not shown herein but are understood to be incorporated by those of ordinary skill in the art in accordance with the present disclosure. In an example embodiment, the one or more processors may include a GPU (“Graphics Processing Unit”), an SoC (“System on Chip”), a CPU (“Central Processing Unit”), a DSP (“Digital Signal Processor”), and an ASIC (“Application Specific Integrated Circuit”), among others. Additionally, those of ordinary skill in the art may consider additional components of the system 100 not shown herein in accordance with the present disclosure. For example, the controller 140 may include a sensor data acquisition module for obtaining, receiving, and / or storing sensor data acquired from various sensors 160. In an example embodiment, the drive sequence 142 may be updated, switched, or modified in real time in response to sensor data, which includes data from inertial measurement sensors, ambient light sensors, temperature sensors, image sensors, and / or eye tracking sensors as specific illustrative and non-exhaustive examples of sensors. Other sensors may also be used. Generally, the method of updating the drive sequence based on sensor data may include: 1) obtaining sensor readings, 2) using the sensor readings to determine which one of a plurality of pre-stored sequences 170 is most suitable for the sensor readings, 3) loading the selected sequence into the current sequence 142, 4) determining in which upcoming frame or sub-frame the new sequence will start to be used, 5) sending sequence information to the controller 140. In step 2 above, the step of “determining which sequence to use” may use a lookup list. Note that any GPU or other processor (140) will always include many other modules not shown in the figure, and as previously mentioned, it may be assumed that any GPU or other processor (140) includes general computing capabilities, programming and program storage, additional memory available to the processor or GPU, and generally also includes other hardware.

[0037] In addition, the controller 140 may include an image data module (not shown) or generate instructions to acquire and format the image data 143. For example, according to an embodiment, the image data module may acquire and / or receive image data such as raw image data and apply format instructions to generate the formatted image data 143. The image data module (not shown) may cause the image system 100 to acquire image data from one or more image sensors that generate at least a portion of the sensor data. The image data 143 may also be acquired from one or more other sources such as, but not limited to, digital imaging devices, downloaded over the Internet, received from a wireless connection (e.g., Wi-Fi, LTE, etc.), received from a storage device (e.g., hard disk drive, solid state drive, etc.), read from a memory (e.g., random access memory), etc., as would be understood by one of ordinary skill in the art. The image data may be in one or more image formats, and the controller 140 may execute format instructions to convert the image data into one or more other image formats. As is known to those skilled in the art, the image data 143 may include, for example, the red, green, and blue (RGB) values of each pixel of each image that makes up the image data. A non-exhaustive list of image formats from which the image data 143 may be converted may include, but is not limited to, VP8, VP9, AV1, VP6, Sorenson Spark, H.264, H.262, MPEG-1, MPEG-2, Theora, Dirac, MPEG-4, Windows Media Image, RealVideo, H.263, Adobe Flash platform, and any other image format known to one of ordinary skill in the art. According to an embodiment, the image data module may use one or more commercially available, open source, or otherwise developed image data conversion algorithms.

[0038] The image data module (not shown) is typically implemented as software that runs in the GPU using an array of its parallel processors and is configured to convert the format of incoming video image data into one of three fixed compressed image data formats suitable for direct transmission to a display driver. In an example embodiment, this transmission may occur via the MIPI data link as previously described. The MIPI specification defines three data types that a display driver may support.

[0039] The compressed data formats include: 1) bit-plane data, 2) nibble data, and 3) byte data. Bit-plane data (1) is obtained by selecting one bit from a typical 8 bits of a given color and compressing the selected bits from each group of 8 adjacent pixels into an 8-bit word. Once the selected bits from each pixel in the entire image are sent to the display driver 110, another bit is selected and the process is repeated. Since each write of the selected bits from each pixel's group transmits only 1 bit of information per pixel, this process must be repeated multiple times to transmit a full-color image. For example, for 8-bit RGB video, each pixel has, for example, 8 bits each for red, green, and blue, and the bit-plane transmission needs to repeat this process 24 times. Nibble data (2) is a format in which, for example, 4 bits of each pixel are sent (by definition, 4 bits is a nibble). These 4 bits will be either the upper 4 bits or the lower 4 bits of each pixel. Two of these 4-bit nibbles corresponding to the nibbles selected from two adjacent pixels are compressed into each 8-bit word sent to the display driver 110. This process is repeated until the 4 bits of each pixel in the image are transmitted. Since only half of the color information per pixel can be sent at a time, this process must be repeated for the other half of the color information of that color and other colors. Since 8-bit RGB video is typically transmitted, this nibble transmission must be repeated a total of 6 times. Byte data (3) is a format in which 8 bits (or all the pixel information of a given color) are sent for each pixel in the display. Since there are three colors, this process must be repeated 3 times - once for each color.

[0040] In an example embodiment, the controller 140 may define or select one or more drive sequences 142 to apply to the image data 143 at least in part based on the sensor data 102. The controller 140 may merge one or more drive sequences 142 with the image data 143 to effect a dynamic reconfiguration of the image characteristics displayed in the display 150. As used herein, the terms "one or more drive sequences" and "drive sequence" may be used interchangeably and refer to a series of step-by-step instructions sent to the display driver 110 that cause the display driver to manipulate or process the data sent to the display in such a way that the resulting image on the display has the desired displayed image characteristics.

[0041] The displayed image characteristics that can be changed or altered by one or more drive sequences 142 include, but are not limited to, the color duration of pixels, frame rate, color sub-frame rate, bit depth, color sequence duty cycle, timing, color gamut, gamma, brightness, persistence, drive voltage, illumination timing and intensity, and the timing of each bit plane sent to the display (which can determine when the liquid crystal display changes state for each gray level, which can be adjusted based on bit depth and temperature), a look-up table (LUT) that can determine which liquid crystal display state changes occur for each possible gray level, and / or serial port interface (SPI) commands sent to the display or other system components (including the timing and literal values of various SPI commands), which are image characteristics understood by those of ordinary skill in the art. To define or select one or more drive sequences and merge one or more drive sequences with image data, the controller 140 can execute a drive sequence algorithm to generate merged image data. It should be noted that typically only one drive sequence will be used in any given video frame. However, if more than one sequence has been sent to the display driver 110, a different drive sequence can be used for the next frame.

[0042] The drive sequence algorithm can cause the image system 100 to define or select one or more drive sequences 142 at least in part based on sensor data. As discussed above, examples of sensor data include, but are not limited to, inertial measurement data, ambient light data, temperature data, and eye tracking data. One or more drive sequences can be determined by mapping predetermined sensor data characteristics to predetermined display characteristics. For example, data from an eye tracking sensor can indicate that a user of the image system 100 is viewing the left visible region of the display 150. The user's eyes looking left can be a predetermined sensor data characteristic that is mapped to a predetermined display characteristic, such as reducing the resolution of the right visible region of the display 150 and increasing the resolution of the left visible region of the display 150. A suitable sequence that causes the display driver to format the image data corresponding to this situation will be selected and used. Other predetermined sensor characteristics can be mapped to correspond to other display characteristics such that a combination of values of sensor data from the sensor 160 results in a combination of display characteristics that form one or more drive sequences 142. One or more drive sequences 142 can also be defined at least in part based on one or more modes or settings. Example modes or settings can include a power saving mode, a 3D enhancement mode, an augmented reality (AR) mode, a virtual reality (VR) mode, a mixed reality (MR) mode, etc.

[0043] According to one embodiment, the controller 140 formats the image data frame 141 into a MIPI image frame. The MIPI image frame is modified to resemble the image frame of a larger display, such as a larger display, such that there are additional or extra pixel "rows" that do not actually exist. These extra rows can contain some header information (e.g., a description of subsequent data, such as the quantity and type of data for a display driver device or other data) and a driving sequence that is being updated. After the display driver receives it, the parser component in the display driver strips or removes these extra rows, extracts the driving sequence information, and applies it to the display driver logic. However, the MIPI image frame is a specific example implementation, and the controller 140 can also use other image or video formats. Examples of other image or video formats that can be used for or modified for parallel transmission of one or more driving sequences and image data include, but are not limited to, HDMI (High-Definition Multimedia Interface), DP (DisplayPort), PCI-express, USB, Ethernet, and Wi-Fi. Each of the image data frames 141 for the image data to be merged can include multiple bits or bytes reserved for one or more driving sequences 142 and multiple bits or bytes reserved for the image data 143. According to an embodiment, when each of the image data frames 141 is transmitted from the controller 140 to the display driver 110, one or more driving sequences 142 are transmitted together with the image data 143. By receiving one or more driving sequences 142 and the image data 143, the display driver 110 can achieve dynamic reconfiguration of the display characteristics by selecting one of the received driving sequences. The display driver 110 can be configured to receive the image data frame 141 and control the display 150 using one or more driving sequences 142 included in the image data frame 141. The display driver 110 can also provide the image data 143 to the display 150, so that the image data 143 can be displayed by the display 150 for the user to view. The display driver 110 can be configured to reconfigure the display 150 using the display characteristics included in one or more driving sequences 142 while providing the image data 143 without interrupting the display.

[0044] Additional details regarding the features of the controller, display driver device, display, and the image data and / or sequences transmitted therebetween are described in further detail in International Application No. PCT / US2019 / 033809, entitled "SYSTEMS AND METHODS FOR DRIVING A DISPLAY", the entire disclosure of which is incorporated herein by reference.

[0045] Figure 2FIG. 0 is a block diagram of a display driver 210 according to an example embodiment. The display driver 210 is similar to the display driver 110 and additional components are shown herein. For example, to dynamically reconfigure display characteristics within an image system, the display driver 210 includes a parser 215 and an image output 217. The parser 215 includes a parser algorithm or software module that can perform several operations within the parser 215 as follows, which can enable the display driver 210 to process both image data and one or more drive sequences to support dynamically updating the display without interrupting the display of image data. The parser 215 can cause the display driver 210 to receive an image data frame and parse or separate one or more drive sequences and image data from the image data frame. The parser 215 can cause the display driver 210 to temporarily store one or more drive sequences and image data, for example, before providing the image data to a display (not shown herein). The image data can be stored in one or more buffer memories 218, and the drive sequences can be stored in one or more sequence memories 219, one or more LUT memories 220, and one or more SPI memories 221. These components of the display driver 210 can be integrated into a display driver integrated circuit (DDIC) disposed on, for example, an application specific integrated circuit (ASIC) or a similar circuitry.

[0046] The parser 215 reads header information added to the front of the image data that defines what sequence information (if any) exists and where the sequence information should be stored, and can include instructions for the display driver 210 to perform a plurality of operations for separating one or more drive sequences and image data from the image data frame. Examples of operations can include, but are not limited to: receiving the data frame, searching the data frame for one or more synchronization bytes that identify a portion of the data frame (e.g., the first row), and storing a portion of the data frame into a sequence memory or a buffer memory. The operations can include performing sub-operations using variables, such as separating one or more drive sequences from the image data. According to an embodiment, after separating one or more drive sequences from the image data frame, the parser 215 can cause the display driver 210 to store one or more drive sequences in one or more of the memories 219 to 221. One or more of the memories 218 to 221 can be a volatile or non-volatile memory or memory structure within the display driver 210. One or more of the memories 218 to 221 can also be implemented as a volatile or non-volatile memory allocated for use by the display driver 210 within the image system. After separating the image data from the image data frame, the parser 215 can cause the image system to store the image data in the buffer memory 218 or an external image data storage device (not shown herein).

[0047] The display driver 210 includes an image output 217 that provides image data to a display device. The image output 217 obtains data from one or more cache memories 218 or from a LUT memory 220, as determined by the current sequence.

[0048] The timing module 216 enables the execution of one or more drive sequences to be synchronized with one or more timers, time intervals, or synchronization signals such as video sync (VSync). In an example embodiment, a VSync pulse is received via input 214, indicating the start of each video frame. The VSync pulse may be received in MIPI format. At each VSync pulse, the drive sequence is initialized to the first instruction in the drive sequence, and at the same time, incoming data begins to be written to a reserved location in the cache memory 218. The timing base 216 is initialized to 0 at each VSync and counts up in time increments hereinafter referred to as "ticks". When the timer reaches the time specified in the "start time" field of the first instruction, the command or opcode ("Op-code") specified in that instruction is executed with the parameters and flags specified in the remainder of the instruction. Once the execution of the operation is complete, the next instruction in the drive sequence is loaded. This continues until the last "sequence end" command or opcode is encountered. Generally, to achieve synchronization, each instruction in the drive sequence is configured with a unique start time, and the instructions are sorted in the sequence of their start times. In addition, a configurable time period may be set between sequence instructions such that there is sufficient time between sequence instructions for the commands to be executed. When instructions do not require the use of conflicting hardware (such as a cache), some instructions may be executed in parallel and / or asynchronously. All read and write operations may be associated with an entire bitplane. These features provide separate control over the timing of each event that occurs during a frame. In the case where existing display drivers typically have certain mechanisms such as fixed timers for controlling when to send data to the display, the disclosed embodiments send each event, particularly each bitplane, to the display at separately configurable times. This facilitates many benefits, such as setting the timing of individual falling edges in, for example, a PWM scheme in order to achieve any desired linear or non-linear gamma. It also allows for easy modification of the color sequence algorithm to best suit the needs of a particular application.

[0049] Therefore, in combination with Figure 1Other components described in can show that the operation of display driver 210 can enable an image system to dynamically reconfigure the image display settings of a display device without interrupting the image data being displayed by the display device. In an example embodiment, the operation of display driver 210 can be implemented by one or more processors that execute each module set therein. According to various embodiments, and as understood by those of ordinary skill in the art, one or more processors represent one or more system-on-chips (SoCs), digital signal processors (DSPs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and / or other processors. According to various embodiments, one or more processors are configured to read and execute the modules described herein from any one of memories 218 to 221, which may include shared or independently implemented RAM, flash memory, other volatile memory, other non-volatile memory, or memory structures, hard disk drives, and / or solid state drives. In an example embodiment, a drive sequence is executed on the display driver without a processor. In other words, a combination of programmable circuits (e.g., on an ASIC or FPGA) is provided on the DDIC to execute one or more drive sequences.

[0050] Figure 3 is a method executed by a display driver according to an example embodiment. For example, the method can be executed by Figure 2 display driver 210 shown or Figure 1 display driver 110 shown.

[0051] At 31, a first drive sequence is received at a display driver, and at 32, a video frame is processed using the first drive sequence. As described herein, the display driver includes circuitry and / or software and other components, including a parser and one or more memories for storing downloadable drive sequences. In an example embodiment, the first drive sequence includes a "main sequence" downloaded from an external controller, such as a GPU, to the display driver. The main sequence can include a list of instructions that specify some or all of the actions to be taken for processing the video frame, i.e., the actions that the display drive circuitry takes for each frame of incoming data, such as image and / or video data. In an example embodiment, these instructions are executed by a main timing component of the display driver, such as circuitry embedded on the DDIC. In an example embodiment, one of the fields in each instruction specifies the time (relative to a VSync event) at which each instruction of the drive sequence is to be executed. In an example embodiment, instructions that can be included and executed at a specified time in the drive sequence include one or more of the following: writing incoming bitplane data to an external memory (if present) and / or one or more of a plurality of on-chip cache memories; reading bitplane data from an external memory (if present) and / or from one or more of a plurality of on-chip cache memories; combining bitplane data from an external memory and / or from various caches, such as one or more of a plurality of on-chip cache memories, into a 1-bit output "logic plane" using logic defined by a LUT also included in the sequence instructions; writing the result of the logical combination to another cache or caches and / or to an output first-in, first-out (FIFO) buffer and / or memory; formatting the contents of the output FIFO buffer and / or memory and sending it to a display device; sending SPI commands of any data and format to an external display device (including LC or LCoS display devices), where the content and time are determined by the drive sequence; activating other dedicated control lines having system functions, such as a laser enable pin, an external trigger pin, etc.; controlling data conversion by an output formatter and other on-chip dedicated hardware, e.g., optionally, inverting outgoing data before the outgoing data from the output FIFO is sent to a backplane integrated circuit; and / or performing other miscellaneous on-chip tasks, such as instructions to clear a FIFO or cache.

[0052] In addition, as described herein, a portion of the downloaded drive sequence can be modified or replaced with other downloadable drive sequences. Thus, at 33, a second drive sequence is received at the display driver, and at 34, the video frames are processed using the second drive sequence. The second drive sequence can be downloaded in response to a timer event and can be triggered based on, for example, user input, sensor readings, system variables (e.g., temperature), or external instructions. For example, an external controller (e.g., a GPU or a host CPU) can determine that a completely different drive sequence is needed based on sensor readings or user input. For example, a portion of the drive sequence can, for example, via hardware in the DDIC, direct the digital drive circuitry and / or software to conform to the requirements of various spatial light modulators, such as display devices, or to operate in multiple operating modes. In an exemplary embodiment, the second drive sequence differs from the first drive sequence in that pulse width modulation is used instead of duty cycle modulation to achieve grayscale in the display device, or pseudo-analog modulation is used instead of pulse width modulation for a phase display device. Those of ordinary skill in the art can envision other variations of the drive sequence and its portions in accordance with the present disclosure.

[0053] Figure 4 is another method performed by a display driver according to an example embodiment. For example, the method can be performed by Figure 2 the display driver 210 shown or Figure 1 the display driver 110 shown.

[0054] At 41, a drive sequence is received at the display driver. As described herein, the display driver includes circuitry and / or software and other components, including a parser and one or more memories for storing downloadable drive sequences. In an example embodiment, the drive sequence includes a "main sequence" downloaded from an external controller (e.g., a GPU) to the display driver. The main sequence can include a list of instructions that specify some or all of the actions to be taken for processing video frames, i.e., the actions that the display drive circuitry takes for each frame of incoming data, such as image and / or video data, at step 42. In an example embodiment, these instructions are executed relative to the time defined by the main timing component of the display driver, such as circuitry embedded on the DDIC. In an example embodiment, one of the fields in each instruction specifies the time (relative to the VSync event) at which each instruction of the drive sequence is to be executed. In addition, in an exemplary embodiment, the main drive sequence can include at least a first drive sequence and a second drive sequence of a plurality of drive sequences, and the first drive sequence and the second drive sequence can be invoked at different times in response to changes in video information or external commands.

[0055] Accordingly, at 43, the display driver switches to using a second drive sequence in the main drive sequence. The second drive sequence can be executed in response to a timer event and can be triggered based on, for example, user input, sensor readings, system variables (e.g., temperature), or external instructions. For example, an external controller (e.g., a GPU or a host CPU) can determine, based on sensor readings or user input, that a completely different drive sequence is needed. For example, a portion of the drive sequence can, for example, via hardware in the DDIC, direct the digital drive circuitry and / or software to conform to the requirements of various spatial light modulators such as display devices, or to operate in multiple operating modes. In an exemplary embodiment, the second drive sequence differs from the first drive sequence in that pulse width modulation is used instead of duty cycle modulation to implement grayscale in the display device, or pseudo-analog modulation is used instead of pulse width modulation for a phase display device. Those of ordinary skill in the art can envision other variations of the drive sequence and its portions in accordance with the present disclosure.

[0056] Figure 5Is a detailed schematic diagram of a display driver according to an exemplary embodiment. The display driver 510 is similar to the display drivers 110 and 210 and is shown here with components. For example, to dynamically reconfigure display characteristics within an image system, the display driver 510 includes a main sequencer 515 and an image output 517. The main sequencer 515 includes a parser algorithm that can perform several operations to enable the display driver 510 to process both image data and one or more drive sequences to support dynamically updating the display without interrupting the display of image data. The main sequencer 515 can cause the display driver 510 to receive an image data frame and parse or separate one or more drive sequences and image data from the image data frame. The main sequencer 515 can cause the display driver 510 to temporarily store one or more drive sequences and image data, for example, before providing the image data to a display (not shown herein). The image data can be stored in one or more cache memories 518, and the drive sequences can be stored in one or more sequence memories 519. These components of the display driver 510 can be integrated into a display driver integrated circuit (DDIC) disposed on, for example, an application specific integrated circuit (ASIC) or similar circuitry. The main sequencer 515 can include instructions for a plurality of operations for separating one or more drive sequences and image data from an image data frame. Examples of operations can include, but are not limited to, receiving a data frame, searching the data frame for one or more synchronization bytes, or in some embodiments, looking for a header data structure that identifies which image data and / or sequence data will directly follow the header and where it should be stored. The operations can include performing sub-operations using variables, such as separating one or more drive sequences from the image data. After separating one or more drive sequences from the image data frame, the main sequencer 515 can cause the display driver 510 to store the one or more drive sequences in the memory 519 and store the image data frame in the memory 518. The memories 519, 518 can be volatile or non-volatile memories within the display driver 510, or can be external memories of the display driver 510. After separating the image data from the image data frame, the main sequencer 515 can cause the image system to store the image data in the cache memory 518 or an external image data storage device.

[0057] In an exemplary embodiment, the drive sequence is downloaded to the sequence memory 519 via the "SPI Cntrl" input 514. The download of the drive sequence can occur immediately after the display driver 510 is powered on. In an exemplary embodiment, the drive sequence includes a list of up to 1024 128-bit words, and the sequence memory 519 is constructed for such a drive sequence. In other embodiments, other sizes of drive sequences and memories can be envisioned. For example, as described below with respect to Figure 7As shown, the word width and sequence memory arrangement are different from this embodiment. In this embodiment, a word width of 128 bits is used, and each word has multiple dedicated fields, such as Figure 6 as shown.

[0058] Referring to Figure 6 , a 128-bit instruction for an example drive sequence. In this example embodiment, the first 2 bits are not used. The field labeled "start time 'tick'" is the time to perform the specified action in the instruction of the drive sequence after the VSync pulse. This 20-bit number indicates a "tick", which is an internal time unit in use and can be synchronized with the system clock and a time input such as the VSync pulse. Referring to Figure 5 for the display driver, this time can include 30 ns, but can be adjusted by writing to the time base control register for other applications. The field labeled "opcode" specifies the operation being commanded. Since this is a 4-bit field, there are 16 possible encoded actions. The list of possible actions for the opcode can include but is not limited to the following actions:

[0059] · Read from main memory bitplane <n> (where <n> is specified in the "plane address" field)

[0060] ;

[0061] · Write to cache (cache number specified in the "cache R / W flag" field);

[0062] · Read from cache (cache number specified in the "cache R / W flag" field);

[0063] · Use a logical expression with up to 7 variables to combine data from one or more caches and optionally from main memory into a single output bitstream. This is controlled via the LUT specified in the "LUT code field";

[0064] · Write data to the output FIFO;

[0065] · Send the contents of the output FIFO to an additional imager via the "output formatting" and "output data interface" blocks;

[0066] · Send arbitrary data to an arbitrary address via the output SPI interface (the data and address are specified by reusing parts of the "LUT code" field;

[0067] · Set or clear output pins using the GPIO block;

[0068] · Other actions, such as "clear FIFO", etc.;

[0069] · Control the power-on, standby, or power-off modes of various internal memories or memory structures in the display driver; particularly perform the above operations when not in use to reduce power consumption;

[0070] · If appropriate, enable a special "full-byte" data stream to the display, which can include moving the entire pixel data from the cache to the display instead of moving individual bit planes to the display when the display is at a higher bit depth;

[0071] · Configure the read and write depths of a single cache to reduce power;

[0072] · Configure the SPI output to operate as an I2C master in some applications;

[0073] · Mark the last used instruction;

[0074] · Turn on / off selected lighting sources at specific times in the sequence.

[0075] The field marked as "Miscellaneous Flags" is for bits used for multiplexing (mux) control inside the cache array and some arithmetic control flags used with the "Read Data LUT Logic". The field marked as "LUT Code" is used to create any logical function for up to 7 different bit plane data streams, as further described below. The field marked as "Cache R / W Flags" determines which cache memories are being written to, from which source, and from which source to read. The field marked as "Output Multiplexing" determines whether the input to the output FIFO is from one of the caches, from the LUT logic block, or directly from the main memory. The "Pol" bit determines whether the data sent out is inverted.

[0076] At each Vsync pulse (occurring at the start of each video frame), the drive sequence is initialized to the first instruction, and at the same time, under the control of one or more "write-side" registers set in the display driver 510, incoming data begins to be written to reserved locations in the main memory 518. The time base 516 is initialized to t = 0 at each Vsync and is incremented in "ticks". When the timer reaches Figure 6When the time specified in the "start time" field of the first drive sequence mentioned is reached, the opcode specified in the instruction is executed with the parameters and flags specified in the remainder of the instruction. Once the operation is complete, the next drive sequence instruction is loaded, and the main sequencer 515 waits for the time specified in the next drive sequence instruction. This continues until the last "sequence end" opcode is encountered. In an example embodiment, each instruction in the drive sequence is configured with a unique start time, and the instructions are sorted in the sequence of their start times. Additionally, a configurable time period can be provided between the sequence instructions so that there is sufficient time between the sequence instructions for the commands to be executed. When the instructions do not need to use conflicting hardware (such as a cache), some instructions can be executed in parallel. All read and write operations can be associated with an entire bitplane. These features provide separate control of the timing of each event that occurs during a frame. In a situation where existing display drivers typically have some mechanism such as a fixed timer for controlling when to send data to the display, the disclosed embodiments send each event and in particular each bitplane to the display at separately selectable times. This helps with many benefits, such as setting the timing of a single falling edge in, for example, a PWM scheme to achieve any desired linear or non-linear gamma. It also allows for easy modification of the color sequence algorithm to best suit the needs of a particular application.

[0077] In addition, the LUT logic block and the associated 64-bit field in the drive sequence are capable of generating any logical function of up to 6 variables, where these variables are understood to be the bits of a pixel data word. Up to 6 inputs as 6-bit addresses can be used for this 64-bit string. Since 2 to the power of 6 is 64, any bit in this 64-bit string can be considered as the result of one of the possible combinations of the 6 inputs. By correctly selecting the pattern of 1s and 0s in this 64-bit string, the logical function can be determined. This ability enables efficient use of the bitplane data typically stored in cache memory for performing calculations. In an example embodiment, if any one of the 6 inputs is true, a true output can be generated using this array - effectively creating a 6-wide "or" function. Alternatively or additionally, a function can be created that outputs true only when one input is true and the other inputs are false. These logical calculations can be used as part of a process for determining when to end pulse width modulation based on a particular gray level represented by the bitplane data. In an embodiment, the 64-bit look-up table can handle any logical function of 6 inputs corresponding to 6 cache memories. In an embodiment, there are 7 caches. The 7th cache can have logic that allows the data therein to be "anded" or "ored" with the result of a previous logical operation. This provides the flexibility typically required of a 128-bit LUT. In another embodiment, the result of the 6-bit LUT function is written to the 7th cache, and in the next instruction, the result of the 6-bit LUT function is combined with new data to form further extended calculations.

[0078] Other functions can be programmed into the display driver 510. For example, the time base 516 that determines the duration of a "tick" is programmable, such that a faster or slower drive sequence can be performed according to user and / or device requirements. In one embodiment, the duration of a tick can be based on the length or complexity of commands in the drive sequence. In another embodiment, the duration of a tick can be based on the fastest frame rate of the display device. Because the SPI output interface 517 is programmable, the SPI output interface 517 can communicate with devices other than the display device. In an example embodiment, the SPI output interface 517 can control an SPI programmable digital-to-analog controller (DAC), such as setting the current level for illuminating an LED or LASER, etc. In another example embodiment, arbitrary control of the output pins on the display driver 510 is provided.

[0079] Figure 7Another detailed schematic diagram of a display driver according to an example embodiment. Display driver 710 is similar to display drivers 110 and 210 and additional components are shown herein. Note that in this embodiment, the "sequence" memory is separated into, for example, 3 parts such that, for example, only a part of the sequence can be updated at a time. These three parts are command FIFO 719, LUT FIFO 720, and SPI memory 721. For example, to dynamically reconfigure image characteristics within an image system, parser 715 reads header information from incoming data, determines that updated sequence information exists, and writes the result to appropriate memories 719, 720, and 721. Those of ordinary skill in the art will understand that the number of parts can vary. Parser 715 includes a parser algorithm that can perform several operations within parser 715 as follows, which can enable display driver 710 to process both image data and one or more drive sequences to support dynamically updating the display without interrupting the display of image data. Parser 715 can cause display driver 710 to receive an image data frame and parse or separate one or more drive sequences and image data from the image data frame. Parser 715 can cause display driver 710 to temporarily store, for example, one or more drive sequences and image data before providing the image data to a display (not shown herein). The image data can be stored in one or more buffer memories 718, and the drive sequences can be stored in one or more sequence memories 719 (shown herein as "command FIFO"), one or more LUT memories 720 (shown herein as "LUT FIFO"), and one or more SPI memories 721. These components of display driver 710 can be integrated into a display driver integrated circuit (DDIC) disposed on, for example, an application specific integrated circuit (ASIC) or similar circuitry. Additionally, display driver 710 is shown as having two channels, i.e., a single parser 715 and a time base 716 control the operation of at least every two of memories 719, 720, 721. The two channels achieve high data throughput, or in some applications drive more than one display device, and channels can be added and removed according to the specific application without departing from the scope and spirit of the disclosed embodiments.

[0080] Contrary to Figure 5 the display driver 510 shown, Figure 7The present embodiment shown eliminates the external memory and increases the number of cache memories 718. For example, 64 cache memories can be provided. In an example embodiment, all bitplane data can be stored in these caches, so there is no need for a main / external memory. Further, the cache memories 718 are configured such that they can be used individually, or in pairs, or in groups of 4 or 8. This enables the cache memories to be suitable for bitplanes that will be used for different sized displays, including larger displays expected to be available with future technology advancements. Further compared with Figure 5 and Figure 6 embodiments, the drive sequence can be partitioned or separated into different fields or types of instructions. In an example embodiment, the first field of the drive sequence holds only instructions, the second field of the drive sequence holds only LUT content, and the third field of the drive sequence holds only SPI instructions, where separately configured memories are provided on the display driver 710. Thus, the display driver 710 can be adapted and configured to download new versions of these portions (or "sub-portions") of the drive sequence at any point in time without interrupting the video playback output via output 717. Further, compared with Figure 5 and Figure 6 previous embodiments, the individual portions or sub-portions of the drive sequence can be increased both in depth (number of words) and width. In an example embodiment, the LUT memory 720 can be 256 bits wide, such that it generates any function up to 8 bits. Since standard video is typically 8 bits wide, this enables calculations to be performed on the full width of the incoming video in a single step. Further, the individual drive sequences can be increased in depth to, for example, 4096.

[0081] Pseudocode

[0082] The following section describes "pseudocode" representing a limited version of the drive sequence. It should be understood that while the drive sequence itself is in binary, hexadecimal, or other machine-readable format, the following pseudocode is presented in a human-readable format. A person of ordinary skill in the art will further understand that while this pseudocode section clearly does not represent a real-world drive sequence, it is intended to enable a person of ordinary skill in the art to program such downloadable sequences that can benefit from the novel display driver circuitry and devices / methods described herein.

[0083] For this part, the display driver system or device can be configured with 3 cache memories, each cache memory containing or storing 1-bit words, thus restricting the LUT string to only 8 bits. Those of ordinary skill in the art will understand that the embodiments herein can include additional cache memories, larger look-up tables, and wider data words, as presented in the above L-chip and N-chip designs. In this embodiment, linear PWM modulation is created, where the "on" duration of LC is equal to the grayscale value, as Figure 8A shown.

[0084] In this embodiment, the 3 cache memories will load values 0 to 7 in the first 7 pixel addresses. Figure 8B Shown are the pixel number (which will also be the cache memory address in this embodiment), the corresponding grayscale value, and the bits that will be present in Cache0 (C[0]), Cache1 (C[1]), and Cache2 (C[2]). It should be understood that Figure 8A the time shown in is shown and is entirely arbitrary. In the embodiment, it is assumed that according to the following alignment convention: when using the LUT, the C[0] value selects the LSB of the LUT address, the C[l] value selects the LSB+1 of the LUT address, and the C[2] value selects the LSB+2 (or MSB) of the LUT address. Since the LUT is 8 bits wide, these 3 bits from the cache form a 3-bit address that can select any one of these 8 bits.

[0085] At the start of the frame, instructions in the drive sequence to fetch cache data from the main memory and load it into the three caches are provided. In pseudocode, it can be described as follows:

[0086] · At T = 0: Read data from the main memory or parser bitplane 0 and write the data to C[0];

[0087] · At T = 10: Read data from the main memory or parser bitplane 1 and write the data to C[l];

[0088] · At T = 20: Read data from the main memory or parser bitplane 2 and write the data to C[2].

[0089] Next, the plane load data for the display is calculated. In the embodiment, a two-step process is used for each plane load. In the first step, data is read from the 3 caches, 1-bit results are obtained using the LUT, and these 1-bit results are written to the output FIFO. In the second step, which can start shortly after the first step, the contents of the output FIFO are written to the LCOS display.

[0090] In the next instruction, 1 is written to any pixel that has any value other than 0 as the desired grayscale value. This corresponds to Figure 5The rising edge provided in

[0091] · At T = 30: Read C[0], C[1], C[2], combine them using the LUT "11111110", and write to the output FIFO;

[0092] · At T = 35: Write one bit from the output FIFO to the LCOS.

[0093] Note the operation of the LUT here. For the first pixel, the bits from "C[2], C[1], C[0]" are "000". Interpreted as the address into the LUT string, this will select the first bit at the right - hand end of the LUT, which is 0. This corresponds to Figure 8A the top row in , which has no rising edge. Any other gray - scale value will result in a different address into the LUT string, which will produce 1.

[0094] Then, execute the subsequent or additional rows of the sequence as follows:

[0095] · At T = 40: Read C[0], C[1], C[2], combine them using the LUT "11111100", and write to the output FIFO;

[0096] · At T = 45: Write one bit from the output FIFO to the LCOS (note: pixel values "000"

[0097] or "001" will have a falling edge (LUT address 000 or 001) here, and all other pixel values will remain high);

[0098] · At T = 50: Read C[0], C[1], C[2], combine them using the LUT "11111000", and write to the output FIFO;

[0099] · At T = 55: Write one bit from the output FIFO to the LCOS;

[0100] · At T = 60: Read C[0], C[1], C[2], combine them using the LUT "11110000", and write to the output FIFO;

[0101] · At T = 65: Write one bit from the output FIFO to the LCOS;

[0102] · At T = 70: Read C[0], C[1], C[2], combine them using the LUT "11100000", and write to the output FIFO;

[0103] · At T = 75: Write one bit from the output FIFO to the LCOS;

[0104] · At T = 80: Read C[0], C[1], C[2], combine using LUT "11000000", and write to the output FIFO;

[0105] · At T = 85: Write one bit from the output FIFO to the LCOS;

[0106] · At T = 90: Read C[0], C[l], C[2], combine using LUT "10000000"

[0107] and write to the output FIFO;

[0108] · At T = 95: Write one bit from the output FIFO to the LCOS;

[0109] · At T = 100: Read C[0], C[1], C[2], combine using LUT "00000000", and write to the output FIFO;

[0110] · At T = 105: Write one bit from the output FIFO to the LCOS.

[0111] Note that the result of the last LUT logic comparison will be 0, regardless of the cache content, because the LUT is all 0. This makes sense because a falling edge is needed at this time if there is no previous falling edge.

[0112] For illustrative purposes, the above embodiments use many simplifying assumptions. In other embodiments, each entry in the cache memory is a whole word of bit values corresponding to multiple adjacent pixels - typically 256. The LUT is applied to each of these pixels simultaneously, and in some embodiments, each of these operations writes the values of 256 pixels to or from the output FIFO. Additionally, in other embodiments, in the sequence, the sequence can have additional commands, including illumination control Serial Peripheral Interface (SPI) commands.

[0113] Embodiments of the present subject matter disclosure overcome the above identified problems of conventional devices and methods and other disadvantages and deficiencies of the prior art. Embodiments herein have several benefits and advantages, including but not limited to the following. Embodiments of the present subject matter disclosure place almost all aspects of the display driving process under the control of a downloadable sequence. Compared to known systems / methods, adapting and configuring the sequence for design and download allows for flexibility in configuring features such as the gray scale algorithm used, frame rate, bit depth, color order, and timing of illumination.

[0114] Embodiments herein provide a DDIC that has maximum flexibility and configurability such that it can be used with existing display devices while maintaining sufficient flexibility to be used with future display chips; even those with features not currently anticipated by customers or end users. The flexible DDIC design of the embodiments herein does not require adding many external additional components. Additionally, the embodiments do not utilize built-in hardware features that limit its applicability to only certain applications.

[0115] In addition to configuration flexibility, the DDIC chips / devices fabricated according to the disclosed embodiments are scalable to future displays and display applications. The systems, methods, and DDICs fabricated according to the disclosed embodiments are adapted and configured to be flexible, configurable, and customizable to meet the specific application requirements of users. Thus, the embodiments have the benefit of reducing costs and also require fewer new DDIC chip designs because the embodiments herein achieve a broader applicability without penalizing simpler applications among these applications with additional features required for more advanced applications.

[0116] Aspects of the subject matter described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware—including the structural means disclosed in this specification and their structural equivalents—or in combinations of them. The subject matter described in this specification can be implemented as one or more computer program products, e.g., one or more computer programs tangibly embodied in an information carrier (e.g., in a machine-readable storage device) or implemented in a propagated signal, for execution by, or to control the operation of, a data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). A computer program (also referred to as a program, software, software application, or code) can be written in any form of programming language, including a compiled or interpreted language, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file. A program can be stored in a portion of a file that holds other programs or data, in a single file dedicated to the program being discussed, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers at one site, or distributed across multiple sites and interconnected by a communication network.

[0117] The processes and logical flows described in this specification, including the method steps of the subject matter described herein - may be performed by one or more programmable processors that execute one or more computer programs to perform the functions of the subject matter described herein by operating on input data and generating output. The processes and logical flows may also be performed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or ASIC (application specific integrated circuit), and the apparatus of the subject matter described herein may be implemented as special purpose logic circuitry, such as an FPGA (field programmable gate array) or ASIC (application specific integrated circuit).

[0118] By way of example, processors suitable for the execution of a computer program include both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, to receive data therefrom, or to transfer data thereto, or both. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, by way of example including: semiconductor memory devices (such as EPROM, EEPROM, and flash memory devices); magnetic disks (such as internal hard disks or removable disks); magneto-optical disks; and optical disks (such as CD and DVD disks). The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0119] The subject matter described herein may be implemented in a computing system that includes a back-end component (such as a data server), a middleware component (such as an application server), or a front-end component (such as a client computer, mobile device, wearable device having a graphical user interface or a web browser through which a user may interact with an implementation of the subject matter described herein), or any combination of such back-end, middleware, and front-end components. The components of the system may be interconnected by any form or medium of digital data communication (such as a communication network). Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0120] It should be understood that the disclosed subject matter is not limited in its application to the details of the construction and the arrangement of components set forth in the following description or shown in the drawings. The disclosed subject matter is capable of other embodiments and of being practiced and carried out in various ways. Further, it should be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. Thus, those skilled in the art will recognize that the concepts upon which this disclosure is based can readily be utilized as a basis for designing other structures, methods, and systems for carrying out the several purposes of the disclosed subject matter. Therefore, it is important that the claims be regarded as including such equivalent constructions insofar as they do not depart from the spirit and scope of the disclosed subject matter.

[0121] Although the disclosed subject matter has been described and illustrated in the foregoing exemplary embodiments, it should be understood that this disclosure has been made only by way of example, and that various changes in the details of the implementation of the disclosed subject matter may be made without departing from the spirit and scope of the disclosed subject matter, which is limited only by the appended claims.

Claims

1. A display driver, comprising: One or more inputs for receiving image data and a driving sequence from one or more external controllers; One or more cache memories for storing the image data; At least two sequence memories, including separate sequence memories configured to separately store one or more parts of the driving sequence; A parsing circuit configured to receive the image data and the driving sequence and to update the one or more cache memories and the at least two sequence memories in real time, and the separate sequence memories execute in parallel and asynchronously for different image data sets; A time base circuit for synchronizing the execution of the driving sequence with time events associated with the image data; And One or more output circuits for retrieving the image data from the one or more cache memories according to the execution of the synchronized driving sequence and for providing the image data to a display device via one or more output interfaces.

2. The display driver according to claim 1, wherein, At least the one or more inputs, the one or more cache memories, the at least two sequence memories, the parsing circuit, and the one or more output circuits are disposed on an integrated circuit.

3. The display driver according to claim 2, wherein, The integrated circuit includes an application specific integrated circuit (ASIC).

4. The display driver according to claim 3, wherein, The ASIC includes a display driver integrated circuit (DDIC).

5. The display driver according to claim 1, wherein, The one or more external controllers include image data processing circuits.

6. The display driver according to claim 5, wherein, The image data processing circuit includes a graphics processing unit (GPU).

7. The display driver according to claim 5, wherein, An updated driving sequence is received from the image data processing circuit in response to a change in the image data.

8. The display driver according to claim 1, wherein, The time events include video synchronization (VSync) intervals.

9. The display driver according to claim 8, wherein, The synchronization includes executing different combinations of driving sequence instructions at different VSync intervals.

10. The display driver according to claim 1, wherein, The one or more parts include signal modulation characteristics, color duration of pixels, frame rate, color sub-frame rate, bit depth, color sequence duty cycle, color gamut, gamma, persistence, driving voltage, illumination timing, illumination intensity, timing of each bit plane sent to the display, a look-up table (LUT), and serial port interface (SPI) commands.

11. The display driver according to claim 10, wherein, The separate sequence memories include one or more of a LUT memory, a main sequence memory, or an SPI memory.

12. The display driver according to claim 1, wherein, The image data is formatted in at least one of a Mobile Industry Processor Interface (MIPI) format, a High-Definition Multimedia Interface (HDMI) format, a DisplayPort (DP) format, a PCI-express format, a USB format, an Ethernet format, or a Wi-Fi format.

13. A method for driving a display, the method comprising: Receiving a first driving sequence at a display driver from a graphics processing unit (GPU); Storing the first driving sequence in a first sequence memory; Processing a first one or more image frames received from the GPU using the first driving sequence retrieved from the first sequence memory; Receiving a second driving sequence from the GPU in response to a timer event; Storing the second driving sequence in a second sequence memory; And Process a second one or more image frames received from the GPU using the second drive sequence retrieved from the second sequence memory in parallel with and asynchronously to retrieving the first drive sequence from the first sequence memory. Wherein, processing of the first one or more image frames using the first drive sequence is synchronized with a time event associated with the first one or more image frames, and processing of the second one or more image frames using the second drive sequence is synchronized with a time event associated with the second one or more image frames, and Wherein, the first one or more image frames and the second one or more image frames processed by the first drive sequence and the second drive sequence respectively are retrieved from a second memory of the display driver according to the synchronized processing of the first one or more image frames and the synchronized processing of the second one or more image frames, and passed from the display driver to a display device.

14. The method according to claim 13, wherein, The first sequence memory includes one or more memory structures, the one or more memory structures at least including a look-up table (LUT) memory, a main sequence memory, and a serial peripheral interface (SPI) memory.

15. The method according to claim 13, wherein The one or more image frames are stored on a second memory of the display driver.

16. The method according to claim 13, wherein Timer increments associated with processing of the first drive sequence and the second drive sequence include a video synchronization (VSync) signal.

17. The method according to claim 13, wherein, Timer increments associated with processing of the first drive sequence and the second drive sequence include a time interval that is a function of a part of a command of one of the first drive sequence or the second drive sequence.

18. The method according to claim 13, wherein, The one or more image frames include video frames.

19. A method for driving a display, the method comprising: Receiving, at a display driver, a plurality of drive sequences from a graphics processing unit (GPU); Storing a first drive sequence of the plurality of drive sequences in a first sequence memory; Storing a second drive sequence of the plurality of drive sequences in a second sequence memory; Processing a first one or more image frames received from the GPU using the first drive sequence retrieved from the first sequence memory; And Processing a second one or more video frames received from the GPU using a second drive sequence retrieved from the second sequence memory in parallel with and asynchronously to retrieving the first drive sequence from the first sequence memory, Wherein, processing of the first one or more image frames using the first drive sequence is synchronized with a time event associated with the image frames, and processing of the second one or more video frames using the second drive sequence is synchronized with a time event associated with the video frames, and Wherein, a switch from using the first drive sequence to using the second drive sequence is performed in response to a command from the GPU.

Images