Systems and methods for multi-primary wide gamut color systems

Abstract

The present invention includes a system and method for a multi-primary color system for a display. A multi-primary color system increases the number of primary colors available in a color system and color system device. Increasing the number of primary colors reduces metameric error from viewer to viewer. One embodiment of a multi-primary color system includes red, green, blue, cyan, yellow, and magenta primaries. The system of the present invention maintains compatibility with existing color systems and devices and provides a system for backward compatibility with older color systems.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is related to and claims priority to the following application: This application claims priority to U.S. Application No. 17 / 976,347, filed October 28, 2022, the entire disclosure of which is incorporated herein by reference. [Background technology]

[0002] 1. Field of the Invention

[0003] The present invention relates to color systems, and more particularly to wide gamut color systems with an increased number of primary colors.

[0004] 2. Description of the Prior Art

[0005] It is generally known in the prior art to provide systems that have an increased color gamut within a display device.

[0006] Prior art patent documents include:

[0007] U.S. Patent No. 10,222,263 (inventor: Shigezane Yasuyuki, filed February 6, 2017, granted March 5, 2019) relates to a microcomputer that divides an RGB circle into 6 x n equal parts (n is an integer greater than or equal to 1) and calculates the RGB values ​​of each divided color. (255, 0, 0) is stored in the microcomputer's ROM as the reference RGB value of the reference color. The microcomputer converts the reference RGB value according to the angle difference on the RGB circle between the RGB value of the desired color and the reference color, and sets the converted RGB value as the RGB value of the desired color.

[0008] U.S. Patent No. 9,373,305 (inventor: Kawaguchi Hiroshi, filed May 29, 2015, registered June 21, 2016) relates to an image processing device including a display panel capable of providing an input interface for adjustment values ​​of at least some of the color attributes of each vertex of the n-axis (n is an integer equal to or greater than 3), which serves as the adjustment axis in the RGB color space, and an adjustment data generation unit that calculates the influence of each n-axis vertex, which indicates an index described below, for each n-axis based on the distance between a target point, which is an arbitrary lattice point in the RGB color space, and each n-axis vertex, and is capable of calculating the adjusted coordinates of the target point in the RGB color space.

[0009] U.S. Patent Publication No. 20130278993 (Heikenfeld et al., filed September 1, 2011, published October 24, 2013) relates to a display pixel. The pixel includes first and second substrates arranged to define a channel. A fluid is disposed within the channel and includes a first colorant and a second colorant. The first colorant has a first charge and a color. The second colorant has a second charge opposite in polarity to the first charge and a color complementary to the color of the first colorant. A first electrode, along with a voltage source, is operatively connected to the fluid and configured to move either or both of the first and / or second colorants within the fluid, thereby changing at least one spectral characteristic of the pixel.

[0010] U.S. Patent No. 8,599,226 (inventors: Ben-Chorin et al., filed February 13, 2012, issued December 3, 2013) relates to a method and system for converting color image data from a three-dimensional color space format to a format usable by an n-primary display device (where n is 3 or greater). The system defines a two-dimensional subspace having a plurality of two-dimensional locations, each of which can represent a set of n-primary color values ​​and a third scalable coordinate value for generating an n-primary display input signal. Furthermore, the system can receive a three-dimensional color space input signal containing out-of-gamut pixel data that cannot be reproduced by a three-primary additive display device and convert it into wide-gamut color image pixel data suitable for driving a wide-gamut color display.

[0011] U.S. Patent No. 8,081,835 (inventors: Elliott et al., filed July 13, 2010, issued December 20, 2011) relates to a system and method for rendering image data to a multi-primary display device that adjusts image data across metamers, as disclosed herein. Metamer filtering can be based on input image content, and subpixel values ​​can be optimized to improve the accuracy or perception of image rendering. Optimization can be performed for a variety of desired effects. One embodiment includes a display system including a display device capable of selecting from a set of image data values ​​including at least one metamer; an input image data unit; a spatial frequency detection unit that extracts spatial frequency characteristics from the input image data; and a selection unit that selects image data from the metamers according to the spatial frequency characteristics.

[0012] U.S. Patent No. 7,916,939 (Roth et al., filed November 30, 2009, issued March 29, 2011) relates to a device including a color filter arrangement for producing at least four colors, each color produced by a filter on a color filter arrangement, and wherein the relative segment sizes of at least two of the primary colors are different.

[0013] U.S. Patent No. 6,769,772 (Roddy et al., filed October 11, 2002, issued August 3, 2004) relates to a display system for digital color images that provides an expanded color gamut using a six-color light source or two or more multicolor LED arrays or OLEDs. The device uses two or more spatial light modulators to switch between two or more color light sources or LED arrays to provide a six-color display output. Pairing the modulated colors using relative brightness minimizes flicker effects. Summary of the Invention

[0014] It is an object of the present invention to provide an improvement or alternative to the current RGB system.

[0015] In one embodiment, the present invention provides a system for displaying a color primary system, the system comprising: a set of image data including a set of color primary signals, the set of color primary signals corresponding to a set of values ​​in Commission Internationale de l'Eclairage (CIE) Yxy color space, the set of values ​​in the CIE Yxy color space including luminance (Y) and two colorimetric coordinates (x, y), the set of image data including medical image data; and an image data converter, the image data converter including a digital interface, the digital interface capable of encoding and decoding the set of values ​​in the CIE Yxy color space, the encoding and decoding including transferring processed data, the processed data including a first channel related to luminance (Y), a second channel related to a first colorimetric coordinate (x) of the two colorimetric coordinates (x, y), and a third channel related to a second colorimetric coordinate (y) of the two colorimetric coordinates (x, y), the image data converter capable of converting the set of image data for display on at least one display device.

[0016] In another embodiment, the present invention provides a system for displaying a color primary system, the system comprising: a set of image data including a set of color primary signals, the set of color primary signals corresponding to a set of values ​​in Commission Internationale de l'Eclairage (CIE) Yxy color space, the set of values ​​in the CIE Yxy color space including luminance (Y) and two colorimetric coordinates (x, y), the set of image data including medical image data, at least one image capture device, one or more of the at least one image capture device being capable of providing the medical image data, an image data converter, the image data converter including a digital interface, the digital interface being capable of encoding and decoding the set of values ​​in the CIE Yxy color space, the encoding and decoding including transferring processed data, the processed data including a first channel related to luminance (Y), a second channel related to a first of the two colorimetric coordinates (x, y), and two colorimetric coordinates (x, y). a third channel associated with a second colorimetric coordinate (y) among the at least one imager (y), wherein one or more of the at least one imager is incorporated into at least one medical device, and the image data converter is capable of converting the set of image data for display on at least one display device.

[0017] In yet another embodiment, the present invention provides a system for displaying a color primary system, the system comprising: a set of image data including a set of color primary signals, the set of color primary signals corresponding to a set of values ​​in Commission Internationale de l'Eclairage (CIE) Yxy color space, the set of values ​​in the CIE Yxy color space including luminance (Y) and two colorimetric coordinates (x, y), the set of image data including medical image data; at least one imaging device, one or more of the at least one imaging device capable of providing the medical image data; an image data converter, the image data converter including a digital interface, the digital interface capable of encoding and decoding the set of values ​​in the CIE Yxy color space; and at least one display device, the image data converter and the at least one display device capable of communication, the encoding and decoding including transferring processed data, the processed data including a first channel related to luminance (Y), a second channel related to a first of the two colorimetric coordinates (x, y), and two colorimetric coordinates (x, y). a third channel associated with a second colorimetric coordinate (y) among the first and second colorimetric coordinates (y), wherein the image data converter is capable of converting the set of image data for display on the at least one display device.

[0018] These and other aspects of the present invention will become apparent to those skilled in the art from the following description of the preferred embodiment, taken in conjunction with the drawings. [Brief explanation of the drawings]

[0019] The patent or application file contains at least one drawing executed in color. Copies of the color drawing(s) of this patent or patent application publication will be provided by the Office upon request and payment of the appropriate fee. [Figure 1] FIG. 1 illustrates a comparison of one embodiment of a six-primary system including red, green, blue, cyan, magenta, and yellow primaries (“6P-B”) with ITU-R BT.709-6. [Figure 2]FIG. 10 illustrates a comparison of another embodiment of a six-primary system including a red primary, a green primary, a blue primary, a cyan primary, a magenta primary, and a yellow primary (“6P-C”) with the Society of Motion Picture and Television Engineers (SMPTE) RP431-2 D60 white point. [Figure 3] FIG. 10 illustrates a comparison of yet another embodiment of a six-primary system including red, green, blue, cyan, magenta, and yellow primaries (“6P-C”) with the D65 white point of SMPTE RP431-2. [Figure 4] A diagram comparing Super 6Pa and 6P-C. [Figure 5] FIG. 1 shows a comparison of Super 6Pb with Super 6Pa and 6P-C. [Figure 6] FIG. 1 illustrates an embodiment of an encoding and decoding system for a multi-primary color system. [Figure 7] Figure 1 shows a sequential system ("System 2") in which the three primary colors are passed as full bit-level image data to the transfer format and inserted as normal. [Figure 8] FIG. 1 illustrates an embodiment of the encoding and decoding process for a system using a dual-link scheme ("System 3"). [Figure 9] FIG. 1 is a diagram illustrating an embodiment of an encoding process using a dual link method. [Figure 10] FIG. 10 is a diagram illustrating an embodiment of a decoding process using a dual link method. [Figure 11] FIG. 1 illustrates an embodiment of Yxy or Yu′v′ encoding using a nonlinear transfer function (NLTF). [Figure 12] FIG. 1 illustrates an embodiment of Yxy or Yu′v′ encoding without NLTF. [Figure 13] FIG. 1 illustrates an embodiment of Yxy or Yu′v′ decoding using an inverse nonlinear transfer function (NLTF-1). [Figure 14] FIG. 1 illustrates an embodiment of Yxy or Yu′v′ decoding without NLTF-1. [Figure 15]Figure 15A illustrates an embodiment of 4:2:2 Yxy encoding using NLTF, and Figure 15B illustrates an embodiment of 4:2:2 Yxy encoding without NLTF. [Figure 16] Figure 16A shows an embodiment of 4:2:2 Yxy encoding with NLTF applied to all three channels and linear scaling of x and y, and Figure 16B shows an embodiment of 4:2:2 Yxy encoding with linear scaling of x and y without NLTF. [Figure 17] Figure 17A shows an embodiment of 4:4:4 Yxy encoding using NLTF, and Figure 17B shows an embodiment of 4:4:4 Yxy encoding without NLTF. [Figure 18] Figure 18A shows an embodiment of 4:4:4 Yxy encoding with NLTF applied to all three channels and linear scaling of x and y, and Figure 18B shows an embodiment of 4:4:4 Yxy encoding with linear scaling of x and y without NLTF. [Figure 19] FIG. 1 illustrates a sample arrangement of Yxy system components in a 4:2:2 pixel mapping. [Figure 20] FIG. 10 illustrates a sample arrangement of Yxy system components in a 4:2:0 pixel mapping. [Figure 21] FIG. 1 illustrates one embodiment of SMPTE ST292 Yxy system mapping. [Figure 22] FIG. 1 illustrates one embodiment of SMPTE ST2082 Yxy system mapping. [Figure 23] FIG. 10 illustrates an embodiment of inserting Yxy into a CTA 861 stream. [Figure 24] Figure 24A illustrates an embodiment of Yxy decoding with an inverse nonlinear transfer function (NLTF-1) applied to only the Y channel, and Figure 24B illustrates an embodiment of Yxy decoding without applying NLTF-1 to either channel. [Figure 25]Figure 25A shows an embodiment of Yxy decoding with NLTF-1 applied to all three channels and rescaling of x and y, and Figure 25B shows an embodiment of Yxy decoding with no NLTF-1 applied to any of the channels and rescaling of the x and y channels. [Figure 26] Figure 26A illustrates an embodiment of IPT 4:4:4 encoding, and Figure 26B illustrates an embodiment of IPT 4:4:4 decoding. [Figure 27] Figure 27A illustrates an embodiment of IC_{T}C_{P} 4:2:2 encoding. Figure 27B illustrates an embodiment of IC_{T}C_{P} 4:2:2 decoding. [Figure 28] FIG. 1 illustrates one embodiment of a 1 / 2 gamma function. [Figure 29] FIG. 10 is a graph showing the maximum quantization error using a 1 / 2 gamma function. [Figure 30] FIG. 1 illustrates one embodiment of a 1 / 3 gamma function. [Figure 31] FIG. 1 illustrates an embodiment of an encoder. [Figure 32] FIG. 2 illustrates an embodiment of a decoder. [Figure 33] FIG. 1 illustrates one embodiment of a display engine capable of cooperating with a graphics processing unit (GPU) according to the present invention. [Figure 34] FIG. 1 illustrates one embodiment of a process flow diagram for converting images for display. [Figure 35] FIG. 1 illustrates one embodiment of a camera process flow. [Figure 36] FIG. 1 illustrates one embodiment of a display process flow. [Figure 37] FIG. 1 illustrates one embodiment of Payload ID metadata modifications applied to SMPTE ST352. [Figure 38]Figure 38A illustrates an embodiment of Payload ID modifications applied to SMPTE ST352 and ST292, Figure 38B illustrates an embodiment of Payload ID modifications applied to SMPTE ST352 and ST372, and Figure 38C illustrates an embodiment of Payload ID modifications applied to SMPTE ST352 and ST425. [Figure 39] FIG. 1 illustrates an embodiment of System 4 Yxy 10-bit 4:2:2 encoding applied to SMPTE ST292. [Figure 40] Figure 40A shows an embodiment of a first link of System 4 Yxy 10-bit 4:4:4 YC_{B}C_{R} encoding applied to SMPTE ST372. Figure 40B shows an embodiment of a second link of System 4 Yxy 10-bit 4:4:4 YC_{B}C_{R} encoding applied to SMPTE ST372. [Figure 41] Figure 41A shows an embodiment of a first link of System 4 Yxy 10-bit 4:4:4 RGB encoding applied to SMPTE ST372. Figure 41B shows an embodiment of a second link of System 4 Yxy 10-bit 4:4:4 RGB encoding applied to SMPTE ST372. [Figure 42] Figure 42A shows an embodiment of the first link of System 4 Yxy 12-bit 4:4:4 YC_{B}C_{R} encoding applied to SMPTE ST372. Figure 42B shows an embodiment of the second link of System 4 Yxy 12-bit 4:4:4 YC_{B}C_{R} encoding applied to SMPTE ST372. [Figure 43] Figure 43A shows an embodiment of a first link of System 4 12-bit Yxy 4:4:4 RGB encoding applied to SMPTE ST372. Figure 43B shows an embodiment of a second link of System 4 Yxy 12-bit 4:4:4 RGB encoding applied to SMPTE ST372. [Figure 44]Figure 44A shows an embodiment of a System 4 Yxy 10-bit 4:2:2 Level A encoded first data stream applied to SMPTE ST425. Figure 44B shows an embodiment of a System 4 Yxy 10-bit 4:2:2 Level A encoded second data stream applied to SMPTE ST425. [Figure 45] Figure 45A shows an embodiment of a first data stream in Yxy System 4 10-bit 4:4:4 Level A encoding applied to SMPTE ST425. Figure 45B shows an embodiment of a second data stream in Yxy System 4 10-bit 4:4:4 Level A encoding applied to SMPTE ST425. [Figure 46] Figure 46A shows an embodiment of a first data stream in Yxy System 4 12-bit 4:4:4 Level A encoding applied to SMPTE ST425. Figure 46B shows an embodiment of a second data stream in Yxy System 4 12-bit 4:4:4 Level A encoding applied to SMPTE ST425. [Figure 47] Figure 47A shows an embodiment of a first data stream in System 4 Yxy 12-bit 4:2:2 Level A encoding applied to SMPTE ST425. Figure 47B shows an embodiment of a second data stream in System 4 Yxy 12-bit 4:2:2 Level A encoding applied to SMPTE ST425. [Figure 48] Figure 48A is a diagram illustrating an embodiment of a first data stream for System 4 Yxy Level B multiplex dual stream (DS) encoding applied to SMPTE ST 425. Figure 48B is a diagram illustrating an embodiment of a second data stream for System 4 Yxy Level B multiplex dual stream (DS) encoding applied to SMPTE ST 425. [Figure 49]Figure 49A is a diagram illustrating an embodiment of a first data link of System 4 Yxy 10-bit Level B multiplex dual link (DL) encoding applied to SMPTE ST 425. Figure 49B is a diagram illustrating an embodiment of a second data link of System 4 Yxy 10-bit Level B multiplex dual link (DL) encoding applied to SMPTE ST 425. [Figure 50] Figure 50A illustrates an embodiment of a first data link of System 4 Yxy 12-bit Level B multiplex dual link (DL) encoding applied to SMPTE ST425. Figure 50B illustrates an embodiment of a second data link of System 4 Yxy 12-bit Level B multiplex dual link (DL) encoding applied to SMPTE ST425. [Figure 51] This table is a modification of SMPTE ST2036-1(2014) parameters to include System 4 (e.g., Yxy, Yu′v′). [Figure 52] This table is a modification of CTA 861 Table 6 - Colorimetric Transfer Characteristics to include System 4 (e.g., Yxy, Yu′v′). [Figure 53] Figure 53A is a table of Yxy 8-bit 4:2:2 encoding for 4 lanes, Figure 53B is a table of Yxy 8-bit 4:2:2 encoding for 2 lanes, and Figure 53C is a table of Yxy 8-bit 4:2:2 encoding for 1 lane. [Figure 54] Figure 54A is a table of Yxy 10-bit 4:2:2 encoding for 4 lanes, Figure 54B is a table of Yxy 10-bit 4:2:2 encoding for 2 lanes, and Figure 54C is a table of Yxy 10-bit 4:2:2 encoding for 1 lane. [Figure 55] Figure 55A is a table of Yxy 12-bit 4:2:2 encoding for 4 lanes, Figure 55B is a table of Yxy 12-bit 4:2:2 encoding for 2 lanes, and Figure 55C is a table of Yxy 12-bit 4:2:2 encoding for 1 lane. [Figure 56]Figure 56A is a table of Yxy 16-bit 4:2:2 encoding for 4 lanes, Figure 56B is a table of Yxy 16-bit 4:2:2 encoding for 2 lanes, and Figure 56C is a table of Yxy 16-bit 4:2:2 encoding for 1 lane. [Figure 57] Figure 57A is a table of Yxy 10-bit 4:4:4 encoding for 4 lanes, Figure 57B is a table of Yxy 10-bit 4:4:4 encoding for 2 lanes, and Figure 57C is a table of Yxy 10-bit 4:4:4 encoding for 1 lane. [Figure 58] Figure 58A is a table of Yxy 12-bit 4:4:4 encoding for 4 lanes, Figure 58B is a table of Yxy 12-bit 4:4:4 encoding for 2 lanes, and Figure 58C is a table of Yxy 12-bit 4:4:4 encoding for 1 lane. [Figure 59] Figure 59A is a table of Yxy 16-bit 4:4:4 encoding for 4 lanes, Figure 59B is a table of Yxy 16-bit 4:4:4 encoding for 2 lanes, and Figure 59C is a table of Yxy 16-bit 4:4:4 encoding for 1 lane. [Figure 60] 1 is a table of auxiliary video information (AVI) for InfoFrame version 4. [Figure 61] FIG. 1 illustrates RGB sampling transmission in a 4:4:4 sampling system. [Figure 62] FIG. 1 is a diagram showing RGBC / YM sampling transmission in a 4:4:4 sampling system. [Figure 63] FIG. 10 is a diagram showing an example of RGBC YM 4:4:4 transmission from system 2. [Figure 64] FIG. 1 illustrates Y Cb Cr sampling transmission using a 4:2:2 sampling system. [Figure 65] FIG. 1 illustrates Y Cr Cb Cc Cy sampling transmission using a 4:2:2 sampling system. [Figure 66] FIG. 10 shows an example of a Y Cr Cb Cc Cy 4:2:2 transmission from System 2 as a non-stationary luminance. [Figure 67] FIG. 1 illustrates Y Cb Cr sampling transmission using a 4:2:0 sampling system. [Figure 68] FIG. 1 illustrates Y Cr Cb Cc Cy sampling transmission using a 4:2:0 sampling system. [Figure 69] FIG. 1 shows a dual stack LCD projection system for a six-primary color system. [Figure 70] FIG. 1 illustrates an embodiment of a single projector. [Figure 71] FIG. 1 shows a six-primary color system using a single projector and a reflecting mirror. [Figure 72] FIG. 1 shows a dual-stack DMD projection system for a six-primary color system. [Figure 73] FIG. 1 illustrates an embodiment of a single DMD projector solution. [Figure 74] FIG. 1 shows an embodiment of a color filter array for a six-primary color system using a white OLED monitor. [Figure 75] FIG. 1 shows an embodiment of an optical filter array for a six-primary color system using a white OLED monitor. [Figure 76] FIG. 1 illustrates one embodiment of an LCD drive matrix for a six-primary color system using a backlit LCD monitor. [Figure 77] FIG. 1 shows an embodiment of an optical filter array for a six-primary color system using a backlit LCD monitor. [Figure 78] FIG. 1 shows an array for a quantum dot (QD) display device. [Figure 79] FIG. 1 shows one embodiment of an array for a six-primary color system for use in a direct emissive assembly display device. [Figure 80] FIG. 1 illustrates an embodiment of a six-primary color system in an emissive display device that does not incorporate color filter subpixels. [Figure 81]1 is a schematic diagram of a computer system illustrating one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] The present invention relates generally to multi-primary color systems.

[0021] In one embodiment, the present invention provides a system for displaying a color primary system, the system including a set of image data including a set of color primary signals, the set of color primary signals corresponding to a set of values ​​in the Commission Internationale de l'Eclairage (CIE) Yxy color space, the set of values ​​in the CIE Yxy color space including luminance (Y) and two colorimetric coordinates (x, y), the set of image data including medical image data, and an image data converter, the image data converter including a digital interface, the digital interface capable of encoding and decoding the set of values ​​in the CIE Yxy color space, the encoding and decoding including transferring processed data, the processed data including a first channel related to luminance (Y), a second channel related to a first colorimetric coordinate (x) of the two colorimetric coordinates (x, y), and a third channel related to a second colorimetric coordinate (y) of the two colorimetric coordinates (x, y), the image data converter capable of converting the set of image data for display on at least one display device. In one embodiment, the image data converter is capable of converting the set of values ​​in the CIE Yxy color space to multiple color gamuts. In one embodiment, the image data converter includes a lookup table. In one embodiment, the image data set includes colors outside the International Telecommunications Union Recommendation (ITU-R) BT.2020 color gamut. In one embodiment, the image data converter is capable of fully sampling the processed data on the first channel and sub-sampling the processed data on the second and third channels. In one embodiment, the processed data on the first, second, and third channels is fully sampled. In one embodiment, the encoding includes scaling two colorimetric coordinates (x, y) to generate first-scale colorimetric coordinates and second-scale colorimetric coordinates, and / or the decoding includes rescaling data associated with the first-scale colorimetric coordinates and data associated with the second-scale colorimetric coordinates.In one embodiment, the encoding includes converting a set of primary color signals to XYZ data and then converting the XYZ data to generate a set of values ​​in the CIE Yxy color space, and / or the decoding includes converting the processed data to XYZ data and then converting the XYZ data to a format displayable on at least one display device. In one embodiment, the system further includes at least one nonlinear function, the at least one nonlinear function including a data range reduction function having a value ranging from about 0.25 to about 0.9 and / or an inverse data range reduction function having a value ranging from about 1.1 to about 4. In one embodiment, the system further includes at least one image capture device, wherein one or more of the at least one image capture device are capable of providing the medical image data. In one embodiment, the system supports metadata of a medical imaging communication standard. In one embodiment, the system further includes at least one processor connected to at least one memory and at least one learning algorithm for image processing and comparison. In one embodiment, the set of image data further includes hyperspectral data, ultraviolet (UV) data, and / or infrared (IR) data. In one embodiment, the image data converter is capable of generating two different three-coordinate format elements, a first three-coordinate format element being Yxy and a second three-coordinate format element including a first coordinate associated with the UV data, a second coordinate associated with the IR data, and a third coordinate proportional to the intensities of the UV and IR data. In one embodiment, the system further includes at least one chip chart or at least one telemed chart having a plurality of colors and / or at least one reference for system calibration.

[0022] In another embodiment, the present invention provides a system for displaying a color primary system, the system comprising: a set of image data including a set of color primary signals, the set of color primary signals corresponding to a set of values ​​in a Commission Internationale de l'Eclairage (CIE) Yxy color space, the set of values ​​in the CIE Yxy color space including luminance (Y) and two colorimetric coordinates (x, y); the set of image data including medical image data; at least one imaging device, one or more of the at least one imaging device capable of providing the medical image data; and an image data converter, the image data converter including a digital interface, the digital interface capable of encoding and decoding the set of values ​​in the CIE Yxy color space, the encoding and decoding including transferring processed data, the processed data including a first channel related to luminance (Y), a second channel related to a first of the two colorimetric coordinates (x, y), and two colorimetric coordinates (x, y). a third channel associated with a second colorimetric coordinate (y) among the at least one imager (y), wherein one or more of the at least one imager is incorporated into at least one medical device, and the image data converter is capable of converting the set of image data for display on at least one display device.

[0023] In yet another embodiment, the present invention provides a system for displaying a color primary system, the system comprising: a set of image data including a set of color primary signals, the set of color primary signals corresponding to a set of values ​​in a Commission Internationale de l'Eclairage (CIE) Yxy color space, the set of values ​​in the CIE Yxy color space including luminance (Y) and two colorimetric coordinates (x, y); the set of image data including medical image data; at least one imaging device, one or more of the at least one imaging device capable of providing the medical image data; an image data converter, the image data converter including a digital interface, the digital interface capable of encoding and decoding the set of values ​​in the CIE Yxy color space; and at least one display device, the image data converter and the at least one display device capable of communication, the encoding and decoding including transferring processed data, the processed data including a first channel related to luminance (Y), a second channel related to a first of the two colorimetric coordinates (x, y), and two colorimetric coordinates (x, y). a third channel associated with a second colorimetric coordinate (y) among colorimetric coordinates (y), and the image data converter is capable of converting the set of image data for display on the at least one display device. In one embodiment, the at least one display device includes at least four color primaries. In one embodiment, the at least one display device is capable of displaying colors outside the International Telecommunication Union Recommendation (ITU-R) BT.2020 color gamut. In one embodiment, the at least one display device includes a headset configured for a virtual reality, augmented reality, and / or mixed reality environment.

[0024] The present invention relates to color systems. Many color systems are known, but many challenges remain. As imaging technology advances, there has been growing interest in expanding the range of colors reproduced on electronic displays. Television system improvements have expanded from the initial CCIR 601 standard to ITU-R BT.709-6, Society of Motion Picture and Television Engineers (SMPTE) RP431-2, and ITU-R BT.2020. Each standard has expanded the visible color gamut by extending the distance from a reference white point to the locations of the red (R), green (G), and blue (B) primaries (collectively "RGB") in chromaticity space. While this approach works, it has several drawbacks. When implemented in content presentation, technical techniques used to expand the color gamut (typically using narrower emission spectra) can increase metamerism errors for viewers and increase power consumption due to reduced output from lighting sources. These challenges increase both capital and operational costs.

[0025] With currently available technology, displays have a limited range of colors and light output. There are many misunderstandings between how viewers technically interpret display output and how the human eye perceives it in the real world. The reason the human eye can perceive more than three luminous primary colors is because the eye integrates the incoming spectral wavelengths into three bands. Humans interpret the radiant energy (spectrum and amplitude) from a display and perceive it as individual colors. Displays do not directly emit colors or specific wavelengths, but simply emit energy in the same spectrum that humans perceive as light and color. It is the observer who interprets this energy as color.

[0026] When the CIE 2° Standard Observer was established in 1931, the common understanding of color sensation was that the eye uses red, blue, and green cone receptors (James Maxwell & James Forbes 1855). Later, in his Munsell Model of Vision (Munsell 1915), Munsell described the visual system as three independent components: luminance, hue, and saturation. Using RGB illuminants or filters, these three primary colors are the components used to generate images on modern electronic displays.

[0027] There are three main physical variables that affect color perception: the spectral distribution of radiant energy absorbed by the retina, the eye's sensitivity to the intensity of light reaching the retinal pigment epithelium, and the distribution of cones within the retina. The distribution of cones (e.g., L-, M-, and S-cones) varies greatly among individuals.

[0028] Brightness improvements have been achieved through larger backlights and more efficient phosphors. High dynamic range encoding has been addressed with wider, perceptually uniform electro-optical transfer functions to support these brightness improvements. Meanwhile, wider color gamuts have been achieved through the use of narrow-band emitters. Narrow-band emitters allow the viewer to experience greater color saturation. However, there can be a disconnect between how saturation is generated and how it is controlled. The common misconception about changing saturation is that increasing the color value of the primary colors implies increased saturation, but this is incorrect. Changing saturation requires a parametric variation of the spectral output of the primary colors. Variable-spectrum displays that achieve this are not currently commercially available, and no new infrastructure to support them is under discussion.

[0029] Instead, the way a display changes the viewer's color perception is by changing the luminance of the color: as the data value increases, the primary colors become brighter. Changes in saturation are achieved by changing the luminance of all three primary colors, making use of dominant color theory.

[0030] Extending the number of primaries beyond RGB has been discussed in the past. Numerous multi-primary display designs exist. For example, SHARP attempted a four-color QUATTRON television system by adding a yellow primary and developing algorithms to drive it. Matthew Brennesholtz also proposed a four-primary display by adding a cyan primary, and Yan Xiong, Fei Deng, Shan Xu, and Sufang Gao (Changjiang University, School of Physics and Optoelectronic Engineering) described a six-primary display. AU OPTRONICS is also developing five-primary display technology. Sony has also recently disclosed camera designs with RGBCMY (red, green, blue, cyan, magenta, yellow) and RGBCMYW (red, green, blue, cyan, magenta, yellow, white) sensors.

[0031] Actual working displays were released in the late 1990s by Tokyo Polytechnic University, Nagoya City University, Genoa Technologies, etc. However, all of these systems are limited to each display, and the additional primary color information is limited to processing within the display.

[0032] Additionally, the Visual Arts System for Archiving and Retrieval of Images (VASARI) project has developed a colorimetric scanner system for direct digital imaging of paintings. This system provides more accurate color reproduction than traditional film and can replace film photography. The project began in 1989, but technology development continues. Additional information is available at https: / / www.southampton.ac.uk / ~km2 / projs / vasari / (last accessed March 30, 2020), which is incorporated herein by reference in its entirety.

[0033] None of the prior art discloses generating additional primary color information outside the display. Additionally, the systems that drive the displays are often proprietary and are for demonstration purposes only. None of these examples include anything to capture or generate additional primary color information within the workflow. The development of a multi-primary system is not complete if the only part of the display that supports the additional primaries is internal.

[0034] Reference will now be made to the drawings, which are for the purpose of illustrating preferred embodiments of the present invention and are not intended to limit the invention.

[0035] Additional details regarding multi-primary color systems can be found in U.S. Patent Nos. 10,607,527, 10,950,160, 10,950,161, 10,950,162, 10,997,896, 11,011,098, 11,017,708, 11,030,934, 11,037,480, 11,037,481, 11,037,482, 1 Nos. 1,043,157, 11,049,431, 11,062,638, 11,062,639, 11,069,279, 11,069,280, 11,100,838 and U.S. Published Patent Nos. 20200251039, 20210233454, and 20210209990, which are incorporated herein by reference in their entireties.

[0036] Conventional displays include three primary colors: red, green, and blue. The multi-primary color system of the present invention includes at least four primary colors. Preferably, the at least four primary colors include at least one red primary color, at least one green primary color, and / or at least one blue primary color. In one embodiment, the at least four primary colors include a cyan primary color, a magenta primary color, and / or a yellow primary color. In one embodiment, the at least four primary colors include at least one white primary color.

[0037] In one embodiment, the multi-primary color system includes six primary colors. In a preferred embodiment, the six primary colors are red (R), green (G), blue (B), cyan (C), magenta (M), and yellow (Y), often referred to as "RGBCMY." However, the systems and methods of the present invention are not limited to RGBCMY, and other primary colors are compatible with the present invention.

[0038] 6P-B

[0039] 6P-B is a color set that uses the same RGB values ​​defined in the ITU-R BT.709-6 television standard. The color gamut includes these RGB primaries and adds three orthogonal primaries based on the white point. The white point used in 6P-B is D65 (ISO 11664-2).

[0040] In one embodiment, the red primary has a dominant wavelength of 609 nm, the yellow primary has a dominant wavelength of 571 nm, the green primary has a dominant wavelength of 552 nm, the cyan primary has a dominant wavelength of 491 nm, and the blue primary has a dominant wavelength of 465 nm (see Table 1). In one embodiment, the dominant wavelengths are approximately (e.g., within ±10%) of the values ​​listed in the table below. Alternatively, the dominant wavelengths are within ±5% of the values ​​listed in the table below. In yet other embodiments, the dominant wavelengths are within ±2% of the values ​​listed in the table below.

[0041] [Table 1]

[0042] Figure 1 shows a comparison between 6P-B and ITU-R BT.709-6.

[0043] 6P-C

[0044] 6P-C is based on the same RGB primaries defined in the SMPTE RP431-2 projection recommendation. Each gamut includes these RGB primaries and adds three orthogonal primaries based on the white point. The white point used in 6P-B is D65 (ISO 11664-2). There are two versions of 6P-C: one optimized for the D60 white point (SMPTE ST2065-1) and one optimized for the D65 white point. Additional information regarding white points can be found in ISO 11664-2:2007, "Colorimetry - Part 2: CIE Standard Illuminants" (published in 2007) and "ST 2065-1:2012 - SMPTE Standard - Academy Color Encoding Specification (ACES)," in ST 2065-1:2012, pp. 1-23, April 17, 2012, doi: 10.5594 / SMPTE.ST2065-1.2012, which are incorporated herein by reference in their entireties.

[0045] In one embodiment, the red primary has a dominant wavelength of 615 nm, the yellow primary has a dominant wavelength of 570 nm, the green primary has a dominant wavelength of 545 nm, the cyan primary has a dominant wavelength of 493 nm, and the blue primary has a dominant wavelength of 465 nm (see Table 2). In one embodiment, the dominant wavelengths are approximately (e.g., within ±10%) of the values ​​listed in the table below. Alternatively, the dominant wavelengths are within ±5% of the values ​​listed in the table below. In yet other embodiments, the dominant wavelengths are within ±2% of the values ​​listed in the table below.

[0046] [Table 2]

[0047] Figure 2 shows a comparison of 6P-C and SMPTE RP431-2 at the D60 white point.

[0048] In one embodiment, the red primary has a dominant wavelength of 615 nm, the yellow primary has a dominant wavelength of 570 nm, the green primary has a dominant wavelength of 545 nm, the cyan primary has a dominant wavelength of 423 nm, and the blue primary has a dominant wavelength of 465 nm (see Table 3). In one embodiment, the dominant wavelengths are approximately (e.g., within ±10%) of the values ​​listed in the table below. Alternatively, the dominant wavelengths are within ±5% of the values ​​listed in the table below. In yet other embodiments, the dominant wavelengths are within ±2% of the values ​​listed in the table below.

[0049] [Table 3]

[0050] Figure 3 shows a comparison of 6P-C and SMPTE RP431-2 at the D65 white point.

[0051] SUPER 6P

[0052] One advantage of ITU-R BT.2020 is that it allows for the inclusion of all Pointer colors, as well as the ability to increase the saturation of primaries in a six-primary design. Pointer is described in "The Gamut of Real Surface Colors," MR Pointer, Colour Research and Application, Volume 5, Issue 3 (1980), which is incorporated herein by reference in its entirety. However, extending the 6P gamut beyond SMPTE RP431-2 ("6P-C") presents two challenges. The first challenge is the need to narrow the spectrum of the extended primaries. The second challenge is the complexity of designing backward-compatible systems using primaries not associated with the current standard. However, in some cases, it may be necessary to extend the gamut beyond 6P-C to avoid these challenges. If the goal is to encompass the Pointer data set, it is possible to maintain most of the 6P-C system and only change the position of the cyan primary. In one embodiment, the position of the cyan primary is positioned so that the edge of the gamut encompasses the entire Pointer data set. In another embodiment, the position of the cyan primary limits maximum saturation. In 6P-C, cyan is located at u' = 0.096, v' = 0.454. In one implementation of Super 6P (Super 6Pa (S6Pa)), cyan is moved to u' = 0.075, v' = 0.430. This results in a new color gamut that covers almost the entire Pointer dataset. Figure 4 shows a comparison between Super 6Pa and 6P-C.

[0053] Table 4 lists the values ​​for Super 6Pa. The definitions of x and y are given in ISO 11664-3:2012 / CIE S 014 Part 3, which is incorporated herein by reference in its entirety. The definitions of u′ and v′ are given in ISO 11664-5:2016 / CIE S 014 Part 5, which is incorporated herein by reference in its entirety. JPEG2026501043000103.jpg98 defines each primary color as a dominant wavelength for RGB and a complementary wavelength for CMY.

[0054] [Table 4]

[0055] In another embodiment, the saturation is extended over the same hue angles as 6P-C, as shown in FIG. This approach reduces the complexity of backward compatibility. However, it requires higher chroma (i.e., a narrower spectrum). In another implementation of Super 6P (Super 6Pb (S6Pb)), cyan is shifted to u' = 0.067, v' = 0.449. Furthermore, Figure 5 shows a comparison of Super 6Pb with Super 6Pa and 6P-C.

[0056] Table 5 lists the values ​​for Super 6Pb. The definitions of x and y are given in ISO 11664-3:2012 / CIE S 014 Part 3, published in 2012, and are incorporated herein by reference in their entirety. The definitions of u′ and v′ are given in ISO 11664-5:2016 / CIE S 014 Part 5, published in 2016, and are incorporated herein by reference in their entirety. JPEG2026501043000105.jpg98 defines each primary color as a dominant wavelength for RGB and a complementary wavelength for CMY.

[0057] [Table 5]

[0058] In a preferred embodiment, a matrix is ​​created from the XYZ values ​​of each primary color. As the XYZ values ​​of the primary colors change, the matrix also changes. Additional details regarding the matrix are provided below.

[0059] Multi-primary signal format and transmission

[0060] The present invention includes three different formatting methods for transmitting video: System 1, System 2, and System 3. System 1 consists of an encoding and decoding system and can be divided into a base encoder and digitizing, stacking image data, mapping to standard data transmission, readout, unstacking, and final image decoding. In one embodiment, the basic method of this system is to combine paired primary colors within three standard transmission channels and identify them by their code values.

[0061] System 2 uses a sequential method in which three primary colors are inserted sequentially into the transmission format as full bit-level image data and inserted normally. The additional three channels are delayed by one pixel and inserted into the transmission in place of the first color. This method is useful when quantization artifacts are important to image performance. In one embodiment, the system consists of six primary colors (e.g., a delayed insertion method for RGB and CMY colors), image resolution identification for pixel count synchronization, video start identification, and RGB delay.

[0062] System 3 utilizes a dual-link scheme using two wires. In one embodiment, the first three channels (e.g., RGB) are sent on link A, and the second three channels (e.g., CMY) are sent on link B. Once the image reaches its destination, they are recombined.

[0063] System 1, System 2, or System 3 can be used as described herein to transmit up to six color components (e.g., four, five, or six color components). When four color components are used, two channels are set to 0. When five color components are used, one channel is set to 0. This transmission method is valid for all primary color systems containing up to six color components described herein.

[0064] Comparison of the three systems

[0065] System 1 is compatible with traditional SDI, CTA and Ethernet transmission. Additionally, System 1 has zero latency processing for conversion to RGB displays. However, System 1 is limited to 11-bit words.

[0066] System 2 can transmit six channels using 16-bit words without compression. System 2 is also compatible with new SDI, CTA and Ethernet transmission formats. However, System 2 requires twice the bit rate. For example, a 4K image requires the same data rate as an 8K RGB image.

[0067] System 3, on the other hand, can transmit up to six channels with 16-bit words using compression, the same amount of data required for a particular resolution. For example, the data rate for an RGB image is the same as a 6P image using System 3. However, System 3 requires twin cable connections within the video system.

[0068] nomenclature

[0069] In one embodiment, a standard visual nomenclature is used to more clearly describe each system.

[0070] R represents red data as linear light (e.g., no nonlinear function is applied). G represents green data as linear light. B represents blue data as linear light. C represents cyan data as linear light. M represents magenta data as linear light. Y c and / or Y represents yellow data as linear light.

[0071] R' represents red data as a nonlinear light (e.g., when a nonlinear function is applied). G' represents green data as a nonlinear light. B' represents blue data as a nonlinear light. C' represents cyan data as a nonlinear light. M' represents magenta data as a nonlinear light. Y c′ and / or Y′ represents yellow data as nonlinear light.

[0072] Y6 represents the sum of the luminance values ​​of the RGBCMY data. RGB represents the encoding of System 2, which is a linear luminance sum of RGB data. CMY represents a System 2 encoding that is a linear luminance sum of CMY data.

[0073] C R represents the red data value after subtracting the linear image luminance. C B represents the blue data value after subtracting the linear image intensity. C C represents the cyan data value after subtracting the linear image luminance. C Y represents the yellow data value after subtracting the linear image luminance.

[0074] Y′ RGB represents the encoding of System 2, which is a nonlinear luminance sum of RGB data. CMY represents the System 2 encoding, which is a non-linear luminance sum of the CMY data. -Y represents Y6 minus the sum of the RGB data.

[0075] C' R represents the red data value after subtracting the nonlinear image luminance. B represents the blue data value after subtracting the nonlinear image intensity. C represents the cyan data value after subtracting the nonlinear image luminance. Y represents the yellow data value after subtracting the non-linear image intensity.

[0076] B+Y represents System 1 encoding with blue or yellow data. G+M represents System 1 encoding with green or magenta data. R+C represents System 1 encoding with green or magenta data.

[0077] C R +C C represents the encoding of System 1, including any chrominance data. C B +C Y represents the encoding of System 1 including any chrominance data.

[0078] 4:4:4 represents full-bandwidth sampling of color in the RGB system. 4:4:4:4:4:4 represents full sampling of color in the RGBCMY system. 4:2:2 represents encoding that carries image detail in the full-bandwidth luma channel (Y) and half-samples the remaining components as Cb Cr encoding. 4:2:2:2:2 represents encoding that carries image detail in the full-bandwidth luma channel (Y) and half-samples the remaining components as Cb Cr Cy Cc encoding. 4:2:0 is a component system similar to 4:2:2, but with Cr and Cb samples alternating every line. 4:2:0:2:0 is a component system similar to 4:2:2, but with Cr, Cb, Cy, and Cc samples alternating every line.

[0079] Constant luminance is a signal processing where the luminance (Y) values ​​are calculated with linear light. Non-constant luminance is a signal processing where the luminance (Y) values ​​are calculated with non-linear light.

[0080] Derivation of color components

[0081] When using the chrominance format (4:2:2), some components require specific processing to be usable for low-frequency transmission. These are derived as follows:

[0082]

number

[0083]

number

[0084]

number

[0085]

number

[0086]

number

[0087] The ratios of Cr, Cb, Cc, and Cy are also valid for linear light calculations.

[0088] Magenta can be calculated as follows:

[0089]

number

[0090] System 1

[0091] In one embodiment, the multi-primary system is compatible with legacy systems. A backward-compatible multi-primary system is defined by a sampling scheme. In one embodiment, the sampling scheme is 4:4:4. In one embodiment, the sampling scheme is 4:2:2. In another embodiment, the sampling scheme is 4:2:0. In one embodiment of a backward-compatible multi-primary system, a new encoding and decoding system is divided into the following steps ("System 1"): base encoding and digitizing, stacking image data, mapping to standard data transmission, readout, unstacking, and image decoding. In one embodiment, System 1 combines paired primary colors within three standard transmission channels, identified by code values. In one embodiment of a backward-compatible multi-primary system, the processing is analog. In another embodiment, the processing is digital.

[0092] In one embodiment, the multi-primary sampling scheme is 4:4:4 sampling. The black and white bits are redefined. In one embodiment, black is placed at a mid-level within each data word, allowing for the addition of CMY color data.

[0093] FIG. 6 illustrates one embodiment of a multi-primary encoding and decoding system. In one embodiment, the multi-primary encoding and decoding system is divided into a base encoder and digitization, image data stacking, mapping to standard data transmission, readout, unstacking, and final image decoding ("System 1"). In one embodiment, the system's method combines opposing primary colors within three standard transmission channels, identified by their code values. In one embodiment, the multi-primary encoding and decoding is analog-based. In another embodiment, the multi-primary encoding and decoding is digital-based. System 1 is designed for compatibility with low-bandwidth systems, allowing up to 11 bits per channel and limited to transmitting only three of the six maximum primary color channels simultaneously. In one embodiment, this is achieved by using a stacking system to decode either the color channel or the complementary color channel, depending on the bit level of each channel.

[0094] System 2

[0095] Figure 7 shows a sequential method ("System 2") in which the three primary colors are passed to the transmission format as full bit-level image data and inserted normally. The additional three channels are delayed by one pixel and then inserted into the transmission in place of the first color. This method is useful in situations where quantization artifacts are critical to image performance. In one embodiment, the system includes six primary colors (RGBCMY), a delay method for CMY color injection, image resolution identification for pixel count synchronization, video start identification, RGB delay, and logic to select the dominant primary color in the case of a YCCCCC system. The advantage of System 2 is that it can transmit full bit-level video but at twice the normal data rate.

[0096] System 2A

[0097] System 2 operates pixel by pixel sequentially. However, a quadrature system ("System 2A") is also possible, which can transmit six-primary stereo or twelve-primary image information. Each quadrant of the frame contains a set of three primary color data sets, which are combined by the display device. A first set of three primary color sets is displayed in the upper left quadrant, a second set of three primary color sets is displayed in the upper right quadrant, a third set of three primary color sets is displayed in the lower left quadrant, and a fourth set of three primary color sets is displayed in the lower right quadrant. In one embodiment, none of the first, second, third, and fourth sets of three primary color sets contain overlapping primary color sets (i.e., 12 different primary color sets). Alternatively, the first, second, third, and fourth sets of three primary color sets contain overlapping primary color sets (i.e., at least one primary color is included in multiple sets of three primary color sets). In one embodiment, the first and third sets of three primary color sets contain the same primary color sets, and the second and fourth sets of three primary color sets contain the same primary color sets.

[0098] System 3

[0099] Figure 8 shows one embodiment of the encoding and decoding process for a system using a dual link scheme ("System 3"). System 3 utilizes a dual link scheme using two wires. In one embodiment, RGB is sent on link A and CMY is sent on link B. After the image reaches its destination, the two links are recombined. Alternating primary colors are also compatible with the present invention.

[0100] System 3 is simpler and more straightforward than Systems 1 and 2. The advantage of this system is that non-RGB primaries (e.g., CMY) can be introduced simply by formatting them on the second link. As an example, in an SDI design, RGB is transmitted in the standard SDI stream, just as it is today. No changes are made to the transmission; this link only requires compensation for luminance differences, allowing it to be transmitted to an RGB display device that does not contain non-RGB (e.g., CMY) components. Data for non-RGB primaries (e.g., CMY data) is transmitted in the same manner as RGB data. This data is combined at the display device to form a 6P image. The disadvantage is that two wires are required to move one image. This system can work with almost all formats, including SMPTE ST292, 424, 2082, and 2110. It can also work with dual High-Definition Multimedia Interface (HDMI) / CTA connections. In one embodiment, the system includes at least one transfer function (e.g., OETF, EOTF).

[0101] One embodiment of the encoding process using the dual link scheme is shown in Figure 9. Alternative numbers of primaries and alternative colors are also compatible with the present invention.

[0102] One embodiment of the decoding process using the dual link scheme is shown in Figure 10. Alternative numbers of primaries and alternative colors are also compatible with the present invention.

[0103] System 4

[0104] Color is generally defined at the three-component data level (e.g., RGB, YCbCr). The serial data stream must accommodate a word for each color component (e.g., R, G, B). If more than three primary colors are used, adjustments must be made to accommodate this data based on the RGB concept. For this reason, System 1, System 2, and System 3 use stacking, sequencing, and / or dual link. This is inefficient because multiple words are required to define one pixel, and not all values ​​are required. In one embodiment, System 4 uses Yxy, L*a*b*, ICT C P , YCbCr, YUV, Yu'v', YPbPr, YIQ, OkLab, LMS, Mlm, and / or XYZ. All of the aforementioned color spaces are based on a set of three human spectral response functions.

[0105] In a preferred embodiment, colors are defined as colorimetric coordinates. Therefore, all colors are defined by three words. Serial systems are already based on three color components (e.g., RGB, YCbCr). System 4 preferably uses XYZ or Yxy as the three color components. System 4 more preferably uses Yxy as the three color components. In another preferred embodiment, System 4 uses Yu'v' as the three color components. System 4 preferably uses two colorimetric coordinates and luminance or luma. In a preferred embodiment, System 4 uses a color format described in a CIE and / or ISO colorimetric standard. In a preferred embodiment, System 4 uses color components that are independent of a white point and / or reference white value. Alternatively, System 4 uses color components that are dependent on a white point and / or reference white value (e.g., YCbCr, L*a*b*). In another embodiment, System 4 uses color components that require at least one known primary color.

[0106] Advantageously, Yxy does not require a reference to a white point and / or at least one known primary color. While YUV and / or L*a*b are reasonable solutions, both are based on the CIE 1931 standard observer and require additional processing without improving accuracy or gamut coverage compared to Yxy. XYZ is the basis for YUV and L*a*b, but both require more mathematical transformations than Yxy. For example, x and y must be calculated before a*b* can be calculated. Furthermore, YUV must be converted back to RGB and then converted back to YUV using a known white point and primary colors. The reliance on a known white point also requires additional processing (e.g., chromatic adaptation) if the display device's white point differs from the encoded white point. Furthermore, the 3x3 matrix used to convert from RGB to YUV has zero-based positive and negative values, resulting in negative values ​​for chroma components and only positive values ​​for luma. In contrast, Yxy is derived from XYZ, but advantageously handles only positive coefficients. Furthermore, since luma is only included in Y, reducing brightness does not affect the chroma components. However, in YUV, reducing brightness reduces the contrast of the chroma components. Since Y is independent, it does not need to be calculated within xy, these are color data points and are not used in the luma calculation.

[0107] In yet another embodiment, L*C*h or other non-Cartesian coordinate systems (e.g., cylindrical, polar) are compatible with the present invention. In one embodiment, a polar coordinate system is defined from Yxy by converting x and y to a hue angle (e.g., θ = arctan(y / x)) and a magnitude vector (e.g., r, similar to C* in L*C*h). However, when converting Yxy to a polar coordinate system, x and y are always non-negative, so θ is limited to 0 to 90 degrees. In one embodiment, the θ angle is expanded by applying a transformation (e.g., an affine transformation) that subtracts the x and y values ​​of the system's white point (e.g., D65) from the x and y data, allowing the x and y data to include negative values. Thus, θ goes from 0 to 360 degrees, allowing a polar plot of the Yxy data to span multiple quadrants.

[0108] XYZ has been used in the motion picture industry for over 10 years. The Digital Cinema Initiative (DCI) defined a file format for theatrical distribution using the XYZ format. The reason for adopting XYZ was to enable adaptability to new display technologies in the future. By including all colors in three-dimensional space, this system ensures that legacy content can be adapted to any new display methods in the future. This system has been in operation since 2005.

[0109] While XYZ works very well within the closed infrastructure of digital cinema, it has drawbacks when used in other applications (e.g., broadcasting, streaming). The reason is that many applications have limited signal bandwidth. Both RGB and XYZ contain luminance in all three channels, necessitating a system in which each subpixel carries separate image information. To get around this, techniques are used to distribute color information across multiple pixel regions. The logic is that (1) image detail is preserved in the luminance component of the image, and (2) the resolution in the color regions can be significantly reduced without significant loss of image quality. Hence, YP B P R , Y.C. B C R ,I C T C P Methods such as RGB and XYZ are used for image transmission. Chrominance encoding and image subsampling allow high-quality image transmission even at low bandwidths. For this reason, RGB or XYZ only use the 4:4:4 sampling system, while YC B C R can be implemented as a 4:4:4, 4:2:2, 4:1:1, or 4:2:0 sampling system.

[0110] There is a long-standing and unmet need for a system capable of describing more than RGB images. In a preferred embodiment, the present invention advantageously uses Yxy or Yu'v' to describe images outside the RGB color gamut. Furthermore, the Yxy or Yu'v' system allows for data transmission using more than three primary colors (e.g., other than RGB). The Yxy or Yu'v' system advantageously presents all color possibilities to a display device. Furthermore, the Yxy or Yu'v' system bridges the gap between scene-referred and display-referred imaging. In an end-to-end system, with a defined white point and EOTF, image data from a camera or graphics generator must match a defined display device. With the emergence of new display devices and high-dynamic-range display devices, source image data (e.g., scene-referred) often needs to be reworked for a specific display device (e.g., display-referred). A scene-referred workflow refers to manipulating images before converting them from the camera color space to the display color space. Using XYZ or ACES 0 for color timing and then transitioning to Yxy or Yu'v' to meet display requirements allows for a smoother display experience without losing color values ​​and while preserving positive color values. This is an advantage of Yxy or Yu'v' even when images are manipulated only after they are converted from the camera color space to the display color space (display-referred imaging). The Yxy or Yu'v' system simplifies electronic image distribution because it is independent of both camera data and display device characteristics. The Yxy or Yu'v' system of the present invention does not increase the data payload and can be substituted into any RGB file or transmission system. Furthermore, the xy or u'v' information can be subsampled, allowing for 4:2:2, 4:1:1, and 4:2:0 packaging. The present invention does not require a specific media definition to accommodate display color gamut limitations. Display devices with different primaries (e.g., multi-primary display devices) can display the same image using the Yxy or Yu'v' system of the present invention, as long as the colors are within the display's range.The Yxy or Yu'v' system also allows for additional primaries to fill the visual spectrum, reducing metamerism errors. Color fidelity can be extended beyond the prior art R+G+B=W model. Display devices with any number of primaries and varying white points can benefit from using the Yxy or Yu'v' approach to define a single media source encoding for all display devices. Conversion from a wide-gamut camera to a multi-primary display device can be achieved using a multiple triple conversion method, which can be implemented within the display device, simplifying image data transmission.

[0111] Out-of-gamut information is managed by the individual display device, not the media definition. Luminance is described by only one channel (Y), and because xy or u'v' do not contain luminance information, changes in Y are independent of hue and saturation, facilitating conversion between SDR and HDR. Any camera gamut can be encoded into a Yxy or Yu'v' encoding, and only minor modifications are required to implement a Yxy or Yu'v' system. Conversion from Yxy or Yu'v' to RGB is straightforward, with minimal processing latency, and fully compatible with traditional RGB systems.

[0112] There is also a long-standing unmet need for a system that replaces the optical gamma function with a code-efficient nonlinear scheme (e.g., Data Range Reduction (DRR)). DRR can optimize data efficiency and simplify image display. Furthermore, DRR is media and display device independent. By substituting a data-efficient nonlinear representation of optical gamma, larger data words (e.g., 16-bit floats) can be stored as 12-bit, 10-bit, or 8-bit integer data words.

[0113] As mentioned above, adding primaries is simplified by the Yxy or Yu'v' process. Furthermore, the brightness of a display device can be advantageously increased by adding primaries. If brightness is provided in the range 0 to 1, the image brightness can be scaled to any desired display brightness using the DRR.

[0114] XYZ requires 16-bit float and 32-bit float encoding, or at least 12 bits for higher quality gamma or log-encoded images. Transmission of XYZ must occur in a 4:4:4 sample system. Sample systems less than 4:4:4 result in a loss of image detail because Y is used as a coordinate with X and Z to carry color information. Furthermore, because X and Z are not orthogonal to Y, luminance information is also included. Advantageously, the conversion to Yxy or Yu'v' concentrates luminance only in Y, leaving two independent pure chromaticity values. In the preferred embodiment, X, Y, and Z are used to calculate x and y. Alternatively, X, Y, and Z are used to calculate u' and v'.

[0115] However, when Y or an equivalent component is used as a luminance value and color is described using two independent colorimetric coordinates (e.g., x and y, u' and v', u and v, etc.), differences in visual sensitivity between color and luminance allow for a system that uses subsampling. In one embodiment, the I or L* component is used instead of Y. In one embodiment, the I and / or L* data is generated by a matrix transformation from XYZ to LMS values. In one embodiment, L* has a nonlinear form using a 1 / 3 power function. In one embodiment, a nonlinear curve (e.g., PQ, HLG) is applied to I. For example, in the case of ICtCp, in one embodiment, a 0.43 power function is applied to I (e.g., in the case of ITP). The system can use any two independent colorimetric coordinates with characteristics similar to x and y, u' and v', and / or u and v. In a preferred embodiment, the two independent colorimetric coordinates are x and y, and the system is a Yxy system. In another preferred embodiment, the two independent colorimetric coordinates are u' and v', and the system is a Yu'v' system. Advantageously, the two independent colorimetric coordinates (e.g., x and y, u' and v') are white-point independent. Furthermore, this reduces the system complexity compared to XYZ, which contains luminance values ​​in all three channels. It also favors subsampling (e.g., 4:2:2, 4:2:0, 4:1:1). In one embodiment, other systems (e.g., IC T C P and L*a*b*) require a white point for calculation. However, the transformation matrix using a white point of [1,1,1] is IC T C P and L*a*b*, thereby removing the white point reference, which can then be recaptured as the [1,1,1] white point in XYZ space. In a preferred embodiment, the image data includes a reference to at least one white point.

[0116] Current technology uses components derived from the legacy National Television System Committee (NTSC). Encoding described in SMPTE, International Telecommunications Union (ITU), and CTA standards includes 4:2:2, 4:2:0, and 4:1:1 subsampling schemes. Advantageously, this allows for color transmission of more than three primaries, i.e., at least four, at least five, at least six, at least seven, at least eight, at least nine, at least ten, at least eleven, and / or at least twelve (e.g., via SMPTE ST292 or HDMI 1.2 transmission). In one embodiment, color transmission of more than three primaries occurs via an SMPTE-defined serial digital interface (SDI), HDMI, or DisplayPort digital display interface. In one embodiment, color transmission of more than three primaries occurs via an imaging serial data stream format.

[0117] Systems 1, 2, and 3 use the YCbCr extension for transmission of the six-primary color data set. The same transmission (e.g., YCbCr extension) can carry image information as Yxy, where Y is luminance information and x and y describe CIE 1931 color coordinates in a half-sampled segment (e.g., 4:2:2) of the data stream. The same transmission (e.g., YCbCr extension) can also carry image information as Yu'v', where Y is luminance information and u' and v' describe CIE 1976 color coordinates in a half-sampled segment (e.g., 4:2:2) of the data stream. Alternatively, x and y or u' and v' are fully sampled (e.g., 4:4:4). In yet another embodiment, the sampling rate is 4:2:0 or 4:1:1. In yet another embodiment, the same transmission can carry information as luminance and colorimetric coordinates using colorimetric coordinates other than x and y (e.g., u' and v'). In one embodiment, the same transmission can carry a data set using one channel of luminance data and two channels of colorimetric data. Alternatively, the same transmission can carry image information at full sampling (e.g., 4:4:4) or partial sampling (e.g., 4:2:2, 4:2:0, 4:1:1) as Yu'v'. In one embodiment, the same transmission is used at full sampling (e.g., XYZ).

[0118] The advantage is that no additional channels are required, and no luminance information needs to be separated from the color components. Furthermore, for example, x and y do not have reference to the primaries, because x and y are explicit colorimetric locations. In Yxy space, x and y are chromaticity coordinates, and x and y can be used to define the gamut of visible colors. Similarly, in Yu'v' space, u' and v' are explicit colorimetric locations. Other formats (e.g., L*a*b*, IC T C P It is possible to define the gamut of visible colors in the L*a*b* and IC* formats, but this is not always easy. T C Pis colorimetric and can describe any visible color, whereas YCbCr is limited to the available colors within the RGB primary triplet. T C P requires gamut restriction / description before encoding color information.

[0119] To determine whether a color is visible in Yxy space, one must determine whether the sum of x and y is greater than or equal to zero; otherwise, the color is undefined. If the x,y point is within the CIE x,y locus (CIE horseshoe), then the color is visible; otherwise, the color is not visible. Similarly, if the u',v' point is within the CIE u',v' locus (CIE horseshoe), then the color is visible. The Yxy chromaticity diagram is nonlinear, and the difference between two chromaticities cannot be represented by a uniformly visible unit vector. Advantageously, Yu'v' reduces the nonuniformity present in the Yxy system and is more perceptually uniform than Yxy.

[0120] The Y value plays an especially important role in display devices. In one embodiment, the display device can reproduce x,y colors within a range of Y values ​​determined as a function of the primary colors. In another embodiment, the display device can reproduce u',v' colors within a range of Y values ​​determined as a function of the primary colors. Another advantage is that the image can be transmitted as linear data without applying a nonlinear function, and a nonlinear function (e.g., an electro-optical transfer function (EOTF)) can be added after the image is received, eliminating the need to apply a nonlinear function (e.g., an OETF) to the signal. This allows for a much simpler encoding and decoding system. In one embodiment, only Y, L*, or I is transformed with a nonlinear function. Alternatively, Y, L*, or I is transmitted linearly (e.g., no nonlinear function is applied). In a preferred embodiment, the nonlinear function is applied to all three channels (e.g., Yxy, Yu'v'). Advantageously, applying the nonlinear function to all three channels results in data compression.

[0121] FIG. 11 illustrates one embodiment of Yxy or Yu'v' encoding using a nonlinear transfer function (NLTF). Image data can be acquired in any format (e.g., RGB, RGBCMY, CMYK) and converted to XYZ linear data. The XYZ data is converted to Yxy or Yu'v' data and then processed through NLTF. The processed Yxy or Yu'v' data is converted to a standardized transmission format for mapping and readout. Advantageously, in one embodiment, x and y are maintained as independent colorimetric coordinates, and the nonlinear transfer function is applied only to Y, simplifying the decoding of the x and y values. In one embodiment, u' and v' are maintained as independent colorimetric coordinates, and the nonlinear transfer function is applied only to Y, simplifying the decoding of the u' and v' values. In another embodiment, advantageously, compression is obtained within the system by applying NLTF to all three channels. In one embodiment, NLTF is specified in ITU-R BT.2100 or ITU-R BT.1886. Advantageously, Y is orthogonal to x and y and remains orthogonal to x and y even when a nonlinear function is applied. Y is also orthogonal to u' and v' and remains orthogonal to u' and v' even when a nonlinear function is applied. Although the examples include Yxy and Yu'v' data, system 4 is compatible with multiple data formats, including data formats that use one luminance coordinate and two colorimetric coordinates.

[0122] Because the RGB set is diverse, the matrix used to convert image data from the RGB primary color set to XYZ requires a specific solution depending on the RGB values:

number

[0123] In embodiments where the image data is 6P-B data, the following formulas are used for conversion to XYZ data:

number

[0124] In an embodiment where the image data is 6P-C data with a D60 white point, the following formulas are used for conversion to XYZ data:

number

number

[0125] To convert XYZ data to Yxy data, the following formula is used:

number

[0126] To convert XYZ data to Yu′v′ data, the following formula is used:

number

[0127] To convert x,y data to u',v' data, the following formula is used:

number

[0128] In one embodiment, the LMS data is converted to a projective representation using the following formula:

number

[0129] In Yxy and Yu'v', Y is the tristimulus relative luminance, while the M channel (closest to the Y response) is not exactly Y. A projected representation can be used analogously to Yxy as Mlm, or alternatively, the projected representation can be used as Ylm, and lms can be converted back to XYZ with a 3x3 matrix.

[0130] In one embodiment, to convert XYZ data to LMS data under equal energy illumination, the following formulas are used:

number

[0131] In one embodiment, to convert D65 normalized XYZ data to LMS data, the following formulas are used:

number

[0132] In one embodiment, the following Hunt-Pointer-Estevez matrix is ​​used to convert the LMS data to XYZ data:

number

number

[0133] Figure 12 shows one embodiment of Yxy or Yu'v' encoding without NLTF. Image data can be acquired in any format (e.g., RGB, RGBCMY, CMYK) and converted to XYZ data.

[0134] The XYZ data is converted to Yxy or Yu'v' data and then converted to a standardized transmission format for mapping and readout. While the example in Figure 12 shows Yxy or Yu'v' encoding, system 4 can be used with multiple data formats.

[0135] Figure 13 shows the inverse nonlinear transfer function (NLTF) -1 ) after mapping and readout, the data is -1 The image is processed through a Yxy or Yu'v' decoder to obtain Yxy or Yu'v' data. The Yxy or Yu'v' data is converted to XYZ data. The XYZ data can be converted to multiple data formats, including RGB, CMYK, 6P (e.g., 6P-B, 6P-C), and color gamuts containing at least four to at least 12 primary colors. While the example in Figure 13 shows Yxy or Yu'v' decoding, system 4 can be used with multiple data formats.

[0136] Finally, the XYZ data must be converted to the correct standard color space. In embodiments where the color gamut is the 6P-B color gamut, the following formula is used:

number

[0137] In embodiments where the color gamut is the 6P-C color gamut with a D60 white point, the following formula is used:

number

[0138] In another embodiment where the color gamut is a 6P-C color gamut with a D65 white point, the following formula is used:

number

[0139] In an embodiment where the color gamut is the ITU-R BT709.6 color gamut, the matrix is ​​as follows:

number

[0140] In an embodiment where the color gamut is the SMPTE RP431-2 color gamut, the matrix is ​​as follows:

number

[0141] In an embodiment where the color gamut is the ITU-R BT.2020 / 2100 color gamut, the matrix is ​​as follows:

number

[0142] To convert the Yxy data to XYZ data, the following formula is used:

number

[0143] To convert the Yu′v′ data to XYZ data, the following formula is used:

number

[0144] FIG. 14 shows one embodiment of Yxy or Yu′v′ decoding without NLTF. After mapping and readout, the Yxy or Yu′v′ data is converted to XYZ data. The XYZ data can be converted to multiple data formats, including, but not limited to, RGB, CMYK, and 6P (e.g., 6P-B, 6P-C), and color gamuts containing at least four primaries to 12 or more primaries. While the example in FIG. 14 shows Yxy or Yu′v′ encoding, system 4 can be used with multiple data formats.

[0145] FIG. 15A illustrates one embodiment of 4:2:2 Yxy encoding using NLTF. The full-bandwidth luminance channel (Y) carries image detail, while the remaining color coordinate components (e.g., x, y) are half-sampled. In the example shown in FIG. 15A, the Yxy data undergoes 4:2:2 encoding. Other encoding schemes (e.g., 4:4:4, 4:2:0, 4:1:1) are also compatible with the present invention. Other quantization schemes and bit depths are also compatible with the present invention. In one embodiment, the bit depth is 8, 10, 12, 14, and / or 16 bits. In one embodiment, the Yxy values ​​are sampled as floating-point numbers (e.g., 16-bit floating-point representation, 32-bit floating-point representation). For example, but not limited to, floating-point numbers include those defined in IEEE 754. While the example in FIG. 15A illustrates Yxy decoding, system 4 can be used with multiple data formats.

[0146] In one embodiment, the NLTF is a DRR function ranging from about 0.25 to about 0.9. In another embodiment, the NLTF is a DRR function ranging from about 0.25 to about 0.7. In one embodiment, the NLTF is a 1 / 2 DRR function including values ​​from about 0.41 to about 0.7. In one embodiment, the NLTF is a 1 / 3 DRR function including values ​​from about 0.25 to about 0.499.

[0147] Figure 15B illustrates one embodiment of 4:2:2 Yxy encoding without NLTF. In the example shown in Figure 15B, the Yxy data undergoes 4:2:2 encoding. Other encoding schemes (e.g., 4:4:4, 4:2:0, 4:1:1) are also compatible with the present invention. While the example in Figure 15B illustrates Yxy encoding, System 4 can be used with multiple data formats.

[0148] FIG. 16A illustrates an embodiment of 4:2:2 Yxy encoding with NLTF applied to all three channels and linear scaling of x and y. The full-bandwidth luminance channel (Y) conveys image detail, while the remaining color coordinate components (e.g., x, y) are half-sampled. In the example shown in FIG. 16A, the Yxy data undergoes 4:2:2 encoding. Other encoding schemes (e.g., 4:4:4, 4:2:0, 4:1:1) are also compatible with the present invention. Other quantization schemes and bit depths are also compatible with the present invention. In one embodiment, the bit depth is 8, 10, 12, 14, and / or 16 bits. In one embodiment, the Yxy values ​​are sampled as floating-point numbers (e.g., 16-bit floating-point representation, 32-bit floating-point representation). For example, but not limited to, floating-point numbers include those defined in IEEE 754. Although the example in FIG. 16A shows Yxy decoding, the system 4 can be used with multiple data formats.

[0149] Figure 16B illustrates one embodiment of 4:2:2 Yxy encoding without NLTF and with linear scaling of x and y. In the example shown in Figure 16B, the Yxy data undergoes 4:2:2 encoding. Other encoding schemes (e.g., 4:4:4, 4:2:0, 4:1:1) are also compatible with the present invention. While the example in Figure 16B illustrates Yxy encoding, System 4 can be used with multiple data formats.

[0150] FIG. 17A illustrates one embodiment of 4:4:4 Yxy encoding using NLTF. The full-bandwidth luminance channel (Y) carries image detail, and the remaining color coordinate components (e.g., x, y) are also fully sampled. In the example shown in FIG. 17A, the Yxy data undergoes 4:4:4 encoding. Other encoding schemes (e.g., 4:2:2, 4:2:0, 4:1:1) are also compatible with the present invention. While the example in FIG. 17A illustrates Yxy encoding, System 4 can be used with multiple data formats.

[0151] Figure 17B illustrates one embodiment of 4:4:4 Yxy encoding without NLTF. In the example shown in Figure 17B, the Yxy data undergoes 4:4:4 encoding. Other encoding schemes (e.g., 4:2:2, 4:2:0, 4:1:1) are also compatible with the present invention. While the example in Figure 17B illustrates Yxy encoding, System 4 can be used with multiple data formats.

[0152] FIG. 18A illustrates one embodiment of 4:4:4 Yxy encoding with NLTF applied to all three channels and linear scaling of x and y. The full-bandwidth luminance channel (Y) carries image detail, while the remaining color coordinate components (e.g., x, y) are fully sampled. In the example shown in FIG. 18A, the Yxy data undergoes 4:4:4 encoding. Other encoding schemes (e.g., 4:2:2, 4:2:0, 4:1:1) are also compatible with the present invention. While the example in FIG. 18A illustrates Yxy encoding, System 4 can be used with multiple data formats.

[0153] Figure 18B illustrates one embodiment of 4:4:4 Yxy encoding without NLTF and with linear scaling of x and y. In the example shown in Figure 18B, the Yxy data undergoes 4:4:4 encoding. Other encoding schemes (e.g., 4:2:2, 4:2:0, 4:1:1) are also compatible with the present invention. While the example in Figure 18B illustrates Yxy encoding, System 4 can be used with multiple data formats.

[0154] Figure 19 shows a sample arrangement of Yxy system components in a 4:2:2 pixel mapping. 00 ~P 35 ) is shown. The first subscript number indicates the row number, and the second subscript number indicates the column number. Pixel P 00 Regarding Y INT00 ' is the luma and the color components are x INT00 and y INT00 Pixel P 01 Regarding Y INT01 ' is the luma. Pixel P10 Regarding Y INT10 ' is the luma and the color components are x INT10 and y INT10 Pixel P 11 Regarding Y INT11 ' is luma. In one embodiment, 00 ) are used to calculate the color and luminance of the subpixels. The example shown in FIG. 19 includes luma, but if the data is luminance (e.g., Y INT00 ) is equally possible. Additionally, while the example of FIG. 19 includes a Yxy system component, system 4 is usable with multiple data formats. While prior art systems often use x, y coordinates to map color gamuts, the present invention provides pixel mapping data as Yxy data, allowing the data to be transmitted as a Y channel, a first colorimetric coordinate channel (e.g., x), and a second colorimetric coordinate channel (e.g., y).

[0155] Figure 20 shows a sample arrangement of Yxy system components in a 4:2:0 pixel mapping. 00 ~P 35 ) is shown. The first subscript number indicates the row number, and the second subscript number indicates the column number. Pixel P 00 Regarding Y INT00 ' is the luma and the color components are x INT00 and y INT00 Pixel P 01 Regarding Y INT01 ' is the luma. Pixel P 10 Regarding Y INT10 ' is the luma. Pixel P 11 Regarding Y INT11 ' is luma. In one embodiment, 00 ) are used to calculate the color and luminance of the subpixel. The example shown in FIG. 20 includes luma, but if the data is luminance (e.g., Y INT00) can also be transmitted linearly. Furthermore, although the example of FIG. 20 includes Yxy system components, the system 4 can be used with multiple data formats.

[0156] In one embodiment, the set of image data includes pixel mapping data. In one embodiment, the pixel mapping data includes a subsample of a set of values ​​in a color space. In a preferred embodiment, the color space is a Yxy color space (e.g., 4:2:2). In one embodiment, the pixel mapping data includes an alignment of a set of values ​​in a color space (e.g., a Yxy color space, a Yu'v' color space).

[0157] Table 6 shows the mapping of Yxy and Yu'v data to SMPTE ST2110 for 4:2:2 sampling. Table 7 shows the mapping of Yxy and Yu'v data to SMPTE ST2110 for 4:4:4 linear and non-linear sampling. The present invention is compatible with multiple data formats and is not limited to Yxy and Yu'v data.

[0158] [Table 6]

[0159] [Table 7]

[0160] Figure 21 shows one embodiment of a SMPTE ST292 Yxy system mapping. To adapt the Yxy system to a SMPTE ST292 stream, the following substitutions are made: Y INT ' into the Y data segment, and x INT_ Place y in the Cr data segment. INTis placed in the Cb data segment. In a preferred embodiment, luminance or luma is placed in the Y data segment, the first colorimetric coordinate is placed in the Cr data segment, and the second colorimetric coordinate is placed in the Cb data segment. The example in Figure 21 shows a Yxy system mapping, but System 4 can be used with multiple data formats.

[0161] Figure 22 shows one embodiment of the SMPTE ST2082 Yxy system mapping. To adapt the Yxy system to a SMPTE ST292 stream, the following substitutions are made: Y INT ' is placed in the G data segment, and x INT Place in the R data segment, and INT is placed in the B data segment. In a preferred embodiment, luminance or luma is placed in the G data segment, the first colorimetric coordinate is placed in the R data segment, and the second colorimetric coordinate is placed in the B data segment. The example in Figure 22 shows a Yxy system mapping, but System 4 can be used with multiple data formats.

[0162] Figure 23 shows one embodiment of inserting Yxy into a CTA 861 data stream. The example in Figure 23 shows the Yxy system mapping, however System 4 can be used with multiple data formats.

[0163] Figure 24A shows the NLTF -1 1 shows an embodiment of Yxy decoding where a nonlinear function (e.g., NLTF) is applied to only the Y channel. -1 ) is applied to luma. No nonlinear function is applied to the two colorimetric coordinates. The example in Figure 24A shows Yxy decoding, but system 4 can be used with multiple data formats.

[0164] In one embodiment, the NLTF -1 is the inverse DRR function in the range of about 1.1 to about 4. In one embodiment, the NLTF -1 is the inverse DRR function in the range of about 1.4 to about 4. In one embodiment, the NLTF -1is the inverse DRR function in the range of about 1.4 to about 2.4. -1 is the inverse DRR function ranging from about 2 to about 4.

[0165] Figure 24B shows the NLTF for both channels. -1 1 shows an embodiment of Yxy decoding without applying a non-linear function (e.g., NLTF). In one embodiment, the data is transmitted linearly as luminance. -1 ) does not apply to luminance or the two colorimetric coordinates. The example in Figure 24B shows Yxy decoding, but system 4 can be used with multiple data formats.

[0166] Figure 25A shows NLTF for all three channels. -1 , and rescaling of x and y. In one embodiment, a non-linear function (e.g., NLTF) is applied. -1 ) is applied to luma and the two colorimetric coordinates. The example in FIG. 25A shows Yxy decoding, but system 4 can be used with multiple data formats.

[0167] Figure 25B shows the NLTF for both channels. -1 1 shows an embodiment of Yxy decoding without applying a non-linear function (e.g., NLTF) and with rescaling applied to the x and y channels. In one embodiment, the data is transmitted linearly as luminance. -1 ) does not apply to luminance or the two colorimetric coordinates. The example in FIG. 25B shows Yxy decoding, but system 4 can be used with multiple data formats (e.g., Yu'v').

[0168] Advantageously, XYZ is used as the basis for ACES for cinematographers, allowing the use of colors outside the ITU-R BT.709 and / or P3 color spaces, and encompassing the entire CIE color space. Because colorists often work in XYZ, there is widespread familiarity with it. Furthermore, XYZ is also used in other standards (e.g., JPEG 2000, Digital Cinema Initiative (DCI)), making it easily adaptable to System 4. Additionally, because most color spaces use XYZ as the basis for their conversion, conversions between XYZ and most color spaces are well understood and documented. Many professional monitors also offer XYZ as a selectable color reference feature.

[0169] In one embodiment, the image data converter includes at least one processor connected to at least one memory. In one embodiment, the image data converter includes at least one lookup table (LUT). In one embodiment, the at least one lookup table maps out-of-gamut colors to zero. In one embodiment, the at least one lookup table maps out-of-gamut colors to a neighborhood of visible colors. In one embodiment, out-of-gamut colors are mapped to a neighborhood on a line between the out-of-gamut color at its original location and the system white point (e.g., D65). In one embodiment, luminance and / or luma values ​​are preserved, and only colorimetric coordinates are affected by the mapping. In one embodiment, gamma transformation and / or scaling are added after mapping. In one embodiment, mapping is used to convert from Yxy to XYZ and vice versa. Alternatively, mapping is used to convert from Y'xy to X'Y'Z' and vice versa. In one embodiment, gamma function and / or scaling are maintained throughout the conversion. As a non-limiting example, a 2.6 gamma function is used to scale x by 0.74 and y by 0.84. Alternatively, gamma and / or scaling is removed after converting out-of-gamut colors to zero. In one embodiment, at least one lookup table maps out-of-gamut colors to a neighborhood of visible colors.

[0170] In one embodiment, the image data converter includes at least one lookup table (LUT). In one embodiment, the at least one lookup table maps out-of-gamut colors to zero. In one embodiment, the at least one lookup table maps out-of-gamut colors to a neighborhood of visible colors. In one embodiment, out-of-gamut colors are mapped to a neighborhood on a line between the out-of-gamut color at its original location and the system white point (e.g., D65). In one embodiment, luminance and / or luma values ​​are preserved, and only colorimetric coordinates are affected by the mapping. In one embodiment, gamma conversion and / or scaling are added after mapping. In one embodiment, mapping is used to convert from Yxy to XYZ and vice versa. Alternatively, mapping is used to convert from Y'xy to X'Y'Z' and vice versa. In one embodiment, gamma function and / or scaling are maintained throughout the conversion. As a non-limiting example, a 2.6 gamma function is used to scale x by 0.74 and y by 0.84. Alternatively, gamma and / or scaling are removed after conversion.

[0171] Additional details regarding System 4 are described in U.S. Patent Application No. 17 / 727,372 (filed April 22, 2022) and U.S. Patent Application No. 17 / 849,220 (filed June 24, 2022), the entire contents of which are incorporated herein by reference.

[0172] Transfer Function

[0173] The system design minimizes the limitations of using standard transfer functions for both the encoding and / or decoding processes. Transfer functions currently used in standards include, but are not limited to, ITU-R BT.1886, ITU-R BT.2020, SMPTE ST274, SMPTE ST296, SMPTE ST2084, and ITU-R BT.2100. These standards are compatible with the system and require no modification.

[0174] Encoding and decoding of multi-primary (e.g., 6P, RGBC) images is formatted into several different configurations to accommodate the limitations of image transmission frequencies. The highest quality transmission is obtained by keeping all components as multi-primary (e.g., RGBCMY) components. This uses the highest sampling frequency and requires the most signal bandwidth. An alternative is to sum the image detail into a full-bandwidth luminance channel and transmit the color difference signals at half or quarter sampling (e.g., Y Cr Cb Cc Cy). This allows a similar image to pass through on a lower-bandwidth transmission.

[0175] The IPT system is similar in concept to the Yxy system with some exceptions. IPT system or IC T C P The system is also an extension of XYZ, and can be derived from RGB and multi-primary (e.g., RGBCMY, RGBC) color coordinates. While IPT color descriptions can be substituted within the 4:4:4 sampling structure, XYZ is already established and does not require the same level of computation. T C P Similar substitutions are possible for transmission systems. However, both substitution systems have the restriction that nonlinear functions (e.g., OOTF) must be present in all three components. The nonlinear functions can be IPT or IC. T C P Although it can be removed for , the derivation is still based on a set of RGB primaries with a white point reference. Removing the nonlinear function may also change bit depth noise and compressibility.

[0176] In transmission, the basis described in XYZ transmission is used to explain the use of IPT and IC in the current system. T C P It is a simple substitution for the current standard used in

[0177] FIG. 26A shows one embodiment of IPT 4:4:4 encoding.

[0178] FIG. 26B shows one embodiment of IPT 4:4:4 decoding.

[0179] Figure 27A shows the IC T C P 1 illustrates an embodiment of 4:2:2 encoding.

[0180] Figure 27B shows the IC T C P 1 illustrates an embodiment of 4:2:2 decoding.

[0181] The transfer functions used in Systems 1, 2, and 3 are generally based on two basic implementations. For images displayed in standard dynamic range, the transfer functions are defined in two standards: OETF is defined in ITU-R BT.709-6, Table 1, line 1.2. Its inverse function, EOTF, is defined in ITU-R BT.1886. For high dynamic range images, the perceptual quantizer (PQ) and hybrid log-gamma (HLG) curves are given in ITU-R BT.2100-2: 2018, Table 4.

[0182] Previous techniques involved introducing nonlinearities based on selected optical performance. As imaging technology has advanced, various methods have evolved. In the past, computer displays used a simple 1.8 gamma, while televisions assumed the inverse of 0.045 gamma. When digital cinema was established, 2.6 gamma was used, and more recently, complex HDR solutions have been introduced. However, because these are embedded within an RGB structure, converting between formats can be very complex and requires extensive processing. Advantageously, Yxy or Yu'v' systems do not require complex conversions or extensive processing.

[0183] A review of the use of gamma and optically based transfer curves for data compression led to the development of Data Range Reduction (DRR) technology. While a form of DRR is similar to the use of gamma, the goal of DRR is to maximize the efficiency of the number of bits available in the display. The advantage of DRR is that it can be used to convert to and / or from any OOTF system using simple conversion methods, allowing any input transform to be displayed in any output transform with minimal processing.

[0184] Using the DRR process, images can be encoded within the source device. A common nonlinearity allows for faster and more accurate conversion. This nonlinearity is designed for data transmission efficiency, not as an optical conversion function. It only works if certain parameters are set for encoding. Optional preprocessing is allowed, but must guarantee accurate 16-bit linear results.

[0185] There are two ways to decode: (1) apply the inverse DRR to the input data and convert it to a linear data format, or (2) map the input data directly to the display using the difference between the DRR value and the desired display gamma (in the case of a simple display gamma).

[0186] Another requirement is computational simplicity: using DRR minimizes processing and reduces signal delay, and nonlinearities (e.g., DRR) are applied based on bit levels rather than image intensity.

[0187] System 4 can use any transfer function applicable to the Y component. However, to improve compatibility between standard transfer functions and simplify conversion, a new method has been developed: the ½ DRR function. Advantageously, the ½ DRR function allows a single calculation from the luminance (e.g., Y) component of a signal (e.g., a Yxy signal, a Yu'v' signal) to a display. Advantageously, the ½ DRR function is designed for data efficiency and not as an optical transfer function. In one embodiment, the ½ DRR function is used in place of a nonlinear function (e.g., an OETF or EOTF). In one embodiment, the signal input to the ½ DRR function is assumed to be linear and constrained to values ​​between 0 and 1. In one embodiment, the ½ DRR function is optimized for 10-bit and / or 12-bit transmission. Alternatively, the ½ DRR function is optimized for 14-bit and / or 16-bit transmission. In another embodiment, the ½ DRR function is optimized for 8-bit transmission. A typical implementation applies the inverse of the 1 / 2 DRR function to linearize the signal, after which a transformation to the display gamut is applied.

[0188] FIG. 28 shows one embodiment of the 1 / 2 DRR function.

[0189] In one embodiment, the DRR is n=L for the source media. 1 / τ and the reverse DRR (DRR -1 ) is the display (or sink) relative to L=n τ where τ represents the exponent of inverse nonlinearity. In one embodiment, the system incorporates both the source gamma (e.g., OETF) and the display gamma (e.g., EOTF). For example, the following formula for DRR is used:

number

[0190] Advantageously, using the 1 / 2 DRR function in combination with the OOTF gamma allows the functions to be combined in a single step rather than using a two-stage conversion process. In one embodiment, at least one tone curve is applied after the 1 / 2 DRR function. The 1 / 2 DRR function has the advantage of facilitating conversion to and from linear values. Because all color and tone mapping must occur in the linear domain, it is desirable to have a conversion that is easy to implement, making conversion to and from linear values ​​simpler and easier.

[0191] 29 shows a graph of the maximum quantization error using a 1 / 2 DRR function. The graph shows the maximum quantization error from an original 16-bit image to a 10-bit (blue line) signal. In the embodiment shown in the graph, the maximum quantization error is less than 0.05% (e.g., 0.047%) for a 16-bit to 10-bit conversion using a 1 / 2 DRR function. The graph also shows the maximum quantization error from the original 16-bit image to a 12-bit (orange line) and 14-bit (gray line) signal.

[0192] While the 1 / 2 DRR is ideal for converting images with 16-bit (e.g., 16-bit floating point) values ​​to 12-bit (e.g., 12-bit integer) values, for other datasets the 1 / 3 DRR offers comparable performance in terms of peak signal-to-noise ratio (PSNR) and a wider luminance dynamic range (e.g., up to 1000 cd / m 2 ), a 1 / 3 DRR conversion from 16-bit floating point maintains the same performance as a 1 / 2 DRR. In one embodiment, the formula for determining the optimal value of τ is:

number

[0193] In one embodiment, the minimum floating-point value is based on the IEEE Standard for Floating-Point Arithmetic (IEEE 754) (July 2019), the entire contents of which are incorporated herein by reference. In one embodiment, the range of image values ​​is normalized to be between 0 and 1. The range of image values ​​is preferably normalized to be between 0 and 1 before the DRR function is applied.

[0194] For example, HDR systems (e.g., luminance dynamic range of 1000 to 4000 cd / m 2 ), the above formula becomes:

number

[0195] FIG. 30 shows one embodiment of the 1 / 3 DRR function.

[0196] In one embodiment, the DRR value preferably ranges from 0.25 to 0.9. Table 8 shows one embodiment of an evaluation of DRR versus bit depth versus full 16-bit floating point (equivalent to 24 f-stops). Table 9 shows a recommended application of DRR. Table 10 shows one embodiment of a DRR function optimized for 8-bit, 10-bit, and 12-bit based on the desired dynamic range (expressed in f-stops). Each f-stop represents two light values. The f-stop provides a tonal range in which noise (measured in f-stops, e.g., the inverse of the perceived signal-to-noise ratio, PSNR) remains below a specified maximum. The lower the maximum noise or the higher the PSNR, the better the image quality. In one embodiment, DRR is not applied to Yxy or Yu'v' 16-bit data. In one embodiment, Yxy or Yu'v' 16-bit data covers 24 f-stops. In one embodiment, a 0.6 DRR is applied to Yxy or Yu'v' 12-bit data, a 0.5 DRR is applied to Yxy or Yu'v' 10-bit data, and a 0.4 DRR is applied to Yxy or Yu'v' 8-bit data. In one embodiment, the Yxy or Yu'v' 12-bit data, the Yxy or Yu'v' 10-bit data, and / or the Yxy or Yu'v' 8-bit data cover 20 f-stops.

[0197] [Table 8]

[0198] [Table 9]

[0199] [Table 10]

[0200] Encoders and Decoders

[0201] In one embodiment, the multi-primary system includes an encoder capable of accepting image data input (e.g., RAW, SDI, HDMI, DisplayPort, Ethernet). In one embodiment, the image data input is from a camera, a computer, a processor, a flash memory card, a network (e.g., a local area network (LAN)), or other file storage or transfer medium capable of providing image data input. The encoder can send processed image data (e.g., Yxy, XYZ, Yu'v') to a decoder (e.g., via wired or wireless communication), and the decoder can send formatted image data (e.g., SDI, HDMI, Ethernet, DisplayPort, Yxy, XYZ, Yu'v', traditional RGB, multi-primary data (e.g., RGBC, RGBCMY, etc.)) to at least one display device (e.g., a display, monitor, projector) for display (e.g., via wired or wireless communication). In one embodiment, the decoder can send formatted image data to at least two display devices simultaneously. In one embodiment, two or more of the at least two display devices use different color spaces and / or formats. As one example, the decoder transmits the formatted image data to a first display device over HDMI and to a second display device over SDI. As another example, the decoder transmits the formatted image data as multi-primary color (e.g., RGBCMY, RGBC) to a first display device and as conventional RGB (e.g., Rec. 709) to a second display device. In one embodiment, the Ethernet-formatted image data conforms to SMPTE ST2022. Additionally or alternatively, the Ethernet-formatted image data conforms to SMPTE ST2110 and / or any Internet Protocol (IP)-based transmission protocol for image data.

[0202] The encoder and decoder preferably include at least one processor. By way of example, and not limitation, the at least one processor may be a general-purpose microprocessor (e.g., a central processing unit (CPU)), a graphics processing unit (GPU), a microcontroller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gate or transistor logic, discrete hardware components, or other suitable entity or combination thereof capable of performing calculations, processing instructions to execute and / or other manipulations of information. In one embodiment, one or more of the at least one processor is capable of executing a predetermined program stored in at least one memory of the encoder and / or decoder.

[0203] The encoder and / or decoder include hardware, firmware, and / or software. In one embodiment, the encoder and / or decoder are insertable into third-party software (e.g., via a dynamic link library (DLL)). In one embodiment, the functions and / or features of the encoder and / or decoder are integrated for efficiency.

[0204] FIG. 31 illustrates one embodiment of an encoder. The encoder includes at least one encoder input (e.g., SDI, HDMI, SMPTE ST2110, SMPTE ST2022, DisplayPort, Fiber, Ethernet) and at least one encoder output (e.g., SDI, HDMI, SMPTE ST2110, SMPTE ST2022, Yxy SDI, Yxy HDMI, Yu'v' SDI, Yu'v' HDMI, DisplayPort, Fiber, Ethernet). The encoder preferably includes an encoder operation programming port, which can provide updates to the encoder's firmware and / or software. For example, the encoder operation programming port can update library functions, internal formats, camera demosaicing (e.g., deBayer) pattern algorithms, and / or look-up tables within the encoder. In one embodiment, the encoder includes a metadata input. In one embodiment, the encoder includes an encoder configuration central processing unit (CPU) interfacing with at least one encoder memory.The encoder may further include an encoder equalizer, at least one encoder serial-to-parallel (S / P) converter (e.g., SDI S / P converter, HDMI S / P, Ethernet S / P converter), at least one encoder flash card reader, at least one Ethernet port, a demosaicing (e.g., debayer) engine, a linear converter, a scaler (e.g., 0-1), at least one custom encoder LUT, a color channel to XYZ converter (e.g., RGB (Rec. 709, P3, Rec. 2020), 6P, multi-primary, ACES, custom), an XYZ-Yxy converter, an XYZ-Yu'v' converter, a DRR function (e.g., 1 / 2 DRR), an xy scaler, a u'v' scaler, a sampling selector (e.g., 4:4:4, 4:2:2, 4:2:0), a metadata decoder, an encoder metadata formatter, and at least one encoder parallel-to-serial (P / S) converter (e.g., SDI P / S converter, HDMI In one embodiment, the input data may bypass any combination of processing stages and / or components within the encoder.

[0205] The at least one encoder input is, but is not limited to, an SDI input, an HDMI input, a DisplayPort input, an Ethernet input, and / or an SMPTE ST2110 input. The SDI input preferably complies with a modified version of the SMPTE ST352 payload identification (ID) standard. In one embodiment, the SDI input is SMPTE ST292, SMPTE ST425, and / or SMPTE ST2082. In one embodiment, the video signal from the SDI input is sent to an encoder equalizer to compensate for cable type and length. In one embodiment, the HDMI input is decoded by standard HDMI receiver circuitry. In one embodiment, the HDMI input is converted to a parallel format. In one embodiment, the HDMI input is defined within the CTA861 standard. In another embodiment, the at least one encoder input includes image data (e.g., RAW data) from a flash device. The configuration CPU has software that identifies the format and / or file type on the flash card and reads the image data and makes it available to the encoder.

[0206] In one embodiment, the encoder operating port is capable of connecting to an encoder control system (e.g., via micro USB or equivalent). In one embodiment, the encoder control system is capable of controlling at least one encoder memory that holds tables for a demosaicing (e.g., deBayer) engine, loading modifications to a linear converter and / or scaler, selecting at least one input, loading tables for at least one custom encoder LUT, bypassing one or more of the at least one custom encoder LUT, bypassing the demosaicing (e.g., deBayer) engine, adding or modifying a conversion table for an RGB-to-XYZ converter, modifying a DRR function (e.g., a 1 / 2 DRR function), turning a watermark engine on or off, modifying a digital watermark for the watermark engine, and performing functions for a flash memory player (e.g., play, stop, fast forward, fast rewind, frame selection).

[0207] In one embodiment, the metadata decoder can decode Extended Display Identification Data (EDID) (e.g., for HDMI input), SDP parameters (SMPTE ST2110), payload ID, and / or ancillary information (e.g., vertical ancillary data (VANC)). The encoder configuration CPU can process the data from the metadata decoder. Additionally, the encoder configuration CPU can select specific settings and send the selected data to the encoder metadata formatter. The metadata input can insert additional data and / or different data values, which can also be sent to the encoder metadata formatter. The encoder metadata formatter can receive information from the encoder configuration CPU and arrange to re-insert that information into the output of the process. In one embodiment, each encoder output formatter receives this formatted data and adapts it to the timing used in the serial stream.

[0208] In one embodiment, the at least one S / P converter supports up to n bits for increased processing efficiency. The at least one S / P converter preferably formats the processed image data to enable the encoder and / or decoder to use parallel processing. Advantageously, parallel processing speeds up processing and minimizes delays.

[0209] At least one encoder formatter is capable of organizing the serial stream into an appropriate format. In a preferred embodiment, the encoder includes a corresponding encoder formatter for each of at least one encoder output. For example, if the encoder includes at least one HDMI output in at least one encoder output, the encoder includes at least one HDMI formatter in at least one encoder formatter. If the encoder includes at least one SDI output in at least one encoder output, the encoder includes at least one SDI formatter in at least one encoder formatter. If the encoder includes at least one Ethernet output in at least one encoder output, the encoder includes at least one Ethernet formatter in at least one encoder formatter. And so on.

[0210] Inputting RAW camera images has the advantage of taking advantage of the extended dynamic range and wide color gamut compared to using standard video inputs. In one embodiment, a demosaicing (e.g., deBayer) engine can convert the RAW image data into a raster image. In one embodiment, the raster image is a three-channel image (e.g., RGB). In one embodiment, the demosaicing (e.g., deBayer) engine is bypassed for data that is not in a RAW image format. In one embodiment, the demosaicing (e.g., deBayer) engine is configured to support at least three primary colors (e.g., 3, 4, 5, 6, 7, 8, etc.) in a Bayer or stripe pattern. To support various demosaicing (e.g., deBayer) options, the operating programming port can load a file containing the necessary code to adapt to a particular pattern (e.g., Bayer). For non-RAW images, a bypass path is provided and switched by the encoder configuration CPU. In one embodiment, the encoder can recognize the image data format and automatically select the correct path. Alternatively, the image data format is included in the metadata.

[0211] The encoder configuration CPU can recognize input nonlinear values ​​and provide inverse values ​​to a linear converter to linearize the image data. The scaler can map out-of-gamut values ​​to in-gamut values.

[0212] In one embodiment, the at least one custom encoder LUT is capable of converting inputs (e.g., manufacturer standards) to XYZ, Yxy, or Yu'v'. Examples of inputs include, but are not limited to, RED Log3G10, ARRI log C, ACEScc, SONY S-Log, CANON Log, PANASONIC V Log, PANAVISION Panalog, and / or BLACK MAGIC CinemaDNG. In one embodiment, the at least one custom encoder LUT is capable of converting inputs to outputs according to artistic needs. In one embodiment, the encoder may not include a color channel-to-XYZ converter or an XYZ-to-Yxy converter; this functionality is incorporated into the at least one custom encoder LUT. In one embodiment, the at least one custom encoder LUT is a 65-cube look-up table. The at least one custom encoder LUT preferably conforms to ACES Common LUT Format (CLF) - A Common File Format for Look-Up Tables S-2014-006 (published July 22, 2021, incorporated herein by reference in its entirety). In one embodiment, the at least one custom encoder LUT is a multi-column LUT. The at least one custom encoder LUT is preferably loadable via an operational programming port. The encoder configuration CPU can bypass the at least one custom encoder LUT if the LUT is not needed.

[0213] In one embodiment, RGB or multi-primary (e.g., RGBCMY, RGBC) data is converted to XYZ data using a color channel-to-XYZ converter. In a preferred embodiment, the white point value of the original video data (e.g., RGB, RGBCMY) is stored in one or more of the at least one encoder memories. The encoder configuration CPU can perform adaptive calculations using the white point value. The XYZ-to-Yxy converter can convert XYZ data to Yxy data. Advantageously, the Yxy image data is divided into a set of luminance and colorimetric values, and the relationship between Y and x,y is manipulable for use at low data rates. Similarly, the XYZ-to-Yu'v' converter can convert XYZ data to Yu'v' data, and this conversion is also manipulable for use at low data rates. Any system with a set of luminance and colorimetric values ​​is compatible with the present invention. The configuration CPU can set a sample selector to match at least one encoder output. In one embodiment, the sampling selector sets the sampling structure (e.g., 4:4:4, 4:2:2, 4:2:0, 4:1:1). The sampling selector is preferably controlled by the encoder configuration CPU. In a preferred embodiment, the sampling selector places each component in the correct serial data position as shown in Table 8.

[0214] [Table 11]

[0215] The watermark engine can modify the image from the original image to include a digital watermark. In some embodiments, the digital watermark exists outside the ITU-R BT.2020 color gamut. In some embodiments, the digital watermark is compressed, reduced, and / or mapped to the edge of a smaller color gamut so that it is invisible and / or undetectable when displayed on a display device with a color gamut smaller than ITU-R BT.2020. In other embodiments, the digital watermark is invisible and / or undetectable when displayed on a display device with the ITU-R BT.2020 color gamut. In some embodiments, the digital watermark is a watermark image (e.g., a logo), alphanumeric text (e.g., a unique identification code), and / or pixel modification. In some embodiments, the digital watermark is invisible to the naked eye. In preferred embodiments, the digital watermark is perceptible upon decoding by an algorithm. In some embodiments, the algorithm decodes the digital watermark using a cryptographic key. In other embodiments, the digital watermark is visible in an unobtrusive manner (e.g., in the bottom right corner of the screen). The digital watermark is preferably detectable even after size compression, scaling, cropping, and / or screenshotting. In yet another embodiment, the digital watermark is an imperceptible change in audio and / or video. In one embodiment, the digital watermark is a pattern (e.g., a random pattern, a fixed pattern) using luminance differences (e.g., 1-bit luminance differences). In one embodiment, the pattern can change for each frame. The digital watermark is a dynamic digital watermark and / or a static digital watermark. In one embodiment, the dynamic digital watermark operates at full frame rate or partial frame rate (e.g., half frame rate). The watermark engine can accept commands from the encoder configuration CPU.

[0216] In another embodiment, at least one encoder input already includes a digital watermark upon input to the encoder. In one embodiment, the camera includes a digital watermark on the image signal that is input to the encoder as at least one encoder input.

[0217] The at least one encoder output is not limited to SDI, HDMI, DisplayPort, and / or Ethernet. In some embodiments, the at least one encoder formatter formats the image data and generates the at least one encoder output. The at least one encoder formatter is not limited to an SDI formatter, an SMPTE ST2110 formatter, and / or an HDMI formatter. In some embodiments, the SDI formatter formats the serial video data as an SDI package to a Yxy or Yu'v' output. The SMPTE ST2110 formatter formats the serial video data as an Ethernet package to a Yxy or Yu'v' output. The HDMI formatter formats the serial video data as an HDMI package to a Yxy or Yu'v' output.

[0218] Figure 32 illustrates one embodiment of a decoder. The decoder includes at least one decoder input (e.g., SDI, HDMI, Ethernet, Yxy SDI, Yxy HDMI, Yxy Ethernet, Yu'v' SDI, Yu'v' HDMI, Yu'v' Ethernet, DisplayPort, Fiber) and at least one decoder output (e.g., Yxy SDI, Yu'v' SDI, at least one SDI, X'Y'Z', HDMI, Ethernet, DisplayPort, Fiber). In one embodiment, the decoder includes a decoder configuration central processing unit (CPU) interfacing with at least one decoder memory. The decoder preferably includes a decoder operation programming port capable of providing decoder firmware and / or software updates.The decoder may further include a decoder equalizer, at least one decoder serial-to-parallel (S / P) converter (e.g., an SDI S / P converter, an HDMI S / P converter, or an Ethernet S / P converter), a watermark detection engine, a watermark removal engine, a DRR-linear converter (e.g., a 1 / 2 DRR-linear converter), an xy descaler, a u'v' descaler, at least one sampling converter (e.g., a 4:2:2 or 4:2:0 to 4:4:4 converter), at least one Yxy-to-XYZ converter, at least one Yu'v'-to-XYZ converter, a gamma library (e.g., linear, 2.2, 2.35, 2.4, 2.6, HLG, PQ, custom), an XYZ-color channel library (e.g., RGB (Rec. 709, P3, Rec. 2020, etc.); multi-primary data), a color channel-YUV ... 2020, etc.; multi-primary data), at least one sample selector, at least one transfer function, at least one custom decoder LUT, a metadata reader, a decoder metadata formatter, at least one decoder parallel-to-serial (P / S) converter (e.g., SDI X'Y'Z', at least one SDI, HDMI), and / or at least one decoder formatter (e.g., SDI X'Y'Z' formatter, SDI RGB formatter, SDI CMY formatter, HDMI formatter). In some embodiments, the X'Y'Z' output includes a non-linear function (e.g., gamma, PQ, HLG) applied to the XYZ data. In some embodiments, the processed image data can bypass any combination of processing stages and / or components within the decoder.

[0219] In some embodiments, the decoder operation port is connectable to a decoder control system (e.g., via micro USB or equivalent). In some embodiments, the decoder control system is capable of selecting at least one decoder input, performing flash memory player functions (e.g., play, stop, advance, fast-forward, rewind, fast-rewind, frame selection), turning watermark detection on / off, adding or changing gamma library and / or lookup table selection, adding or changing XYZ-RGB library and / or lookup table selection, loading data into at least one custom decoder LUT, bypassing one or more custom decoder LUTs, and / or modifying the Ethernet SDP. The gamma library preferably receives linear data and applies at least one nonlinear function to the linear data. The at least one nonlinear function may be, but is not limited to, a standard gamma (e.g., one used in standard dynamic range (SDR) and high definition range (HDR) formats) and / or at least one custom gamma. In some embodiments, the at least one standard gamma is defined in ITU BT.709 or ITU BT.2100.

[0220] In one embodiment, the output of the gamma library is fed into an XYZ-RGB library, where a table maps the XYZ data to a standard RGB or YCbCr output format. In another embodiment, the output of the gamma library bypasses the XYZ-RGB library, leaving the output XYZ data with gamma applied. The selection or bypass of the XYZ-RGB library is determined by the configuration CPU. If YCbCr is selected as the output format, the XYZ-RGB library flags the desired sampling scheme and provides that selection to the sample selector. The sample selector formats the YCbCr data into a 4:2:2, 4:2:0, or 4:1:1 sampling structure.

[0221] In one embodiment, the input to the decoder does not include full pixel sampling (e.g., 4:2:2, 4:2:0, 4:1:1). At least one sampling converter is capable of receiving a subsampled image and converting the subsampled image to full 4:4:4 sampling. In one embodiment, 4:4:4 Yxy image data is converted to XYZ using at least one Yxy-to-XYZ converter. In another embodiment, 4:4:4 Yu'v' image data is converted to XYZ using at least one Yu'v'-to-XYZ converter. The image data is converted from a parallel form to a serial stream.

[0222] The metadata reader can read Extended Display Identification Data (EDID) (e.g., for HDMI inputs), SDP parameters (SMPTE ST 2110), payload ID, and / or ancillary information (e.g., vertical ancillary data (VANC)). The decoder configuration CPU can process the data from the metadata reader. Additionally, the decoder configuration CPU can select a particular configuration and provide the selected data to a decoder metadata formatter. The decoder metadata formatter can receive information from the decoder configuration CPU and arrange to reinsert that information into the output of the process. In one embodiment, each decoder output formatter receives this formatted data and times it for use in the serial stream.

[0223] In some embodiments, the at least one SDI output includes multiple SDI outputs, which has the advantage of allowing output over multiple links (e.g., system 3). In some embodiments, the at least one SDI output includes a first SDI output and a second SDI output. In some embodiments, the first SDI output is used to transfer a first color channel data set (e.g., RGB) and the second SDI output is used to transfer a second color channel data set (e.g., CMY).

[0224] The watermark detection engine detects the digital watermark. In one embodiment, the digital watermark pattern is loaded into the decoder using an operational programming port. In one embodiment, the decoder configuration CPU is capable of switching the watermark detection engine on and off. The watermark removal engine removes the digital watermark from the image data before formatting it for display on at least one display device. In one embodiment, the decoder configuration CPU is capable of allowing bypass of the watermark removal engine, thereby leaving the digital watermark on the output image. In a preferred embodiment, the decoder requires the presence of a digital watermark in the processed image data sent from the encoder in order to provide at least one decoder output. Thus, if a digital watermark is not present in the processed image data, the decoder does not send color channel data to the at least one display device. In another embodiment, the decoder is capable of providing at least one decoder output even if a digital watermark is not present in the processed image data sent from the encoder. If a digital watermark is not present in the processed image data, the image displayed on the at least one display device preferably includes a visible watermark.

[0225] In one embodiment, the output from the watermark removal process includes data that includes nonlinearities (e.g., 1 / 2 DRR). The nonlinear data is applied by applying an inverse nonlinear transfer function (e.g., NLTF) to the Y channel and the xy or u'v' channels. -1 ) to convert it back to linear data. The xy or u'v' channels are rescaled and undergo a sampling transformation.

[0226] In one embodiment, at least one custom decoder LUT includes a nine-column LUT. In one embodiment, the nine-column LUT includes three columns for legacy RGB output (e.g., Rec. 709, Rec. 2020, P3) and six columns for six-primary multi-primary displays (e.g., RGBCMY). Other numbers of columns (e.g., seven columns) and alternative multi-primary displays (e.g., RGBC) are compatible with the present invention. In one embodiment, at least one custom decoder LUT (e.g., a nine-column LUT) can generate output values ​​using tetrahedron interpolation. Tetrahedron interpolation has the advantage of determining output values ​​using a smaller color space volume, resulting in more accurate color channel data. In one embodiment, each tetrahedron used in tetrahedron interpolation includes a neutral diagonal. This embodiment has the advantage of working with fewer than six color channels. For example, four-primary output (e.g., RGBC) or five-primary output (e.g., RGBCY) using an FPGA can be generated using tetrahedron interpolation. Furthermore, this embodiment allows the encoder to generate legacy RGB output in addition to multi-primary output. In another embodiment, the at least one custom decoder LUT is capable of generating output values ​​using cubic interpolation. The at least one custom decoder LUT is preferably capable of accepting linear XYZ data. In one embodiment, the at least one custom decoder LUT is a multi-column LUT. The at least one custom decoder LUT is preferably loadable via an operational programming port. The decoder configuration CPU can bypass the at least one custom decoder LUT if the LUT is not needed.

[0227] In one embodiment, the at least one custom decoder LUT can be used to improve the efficiency of HDMI transmission. In one embodiment, the at least one custom decoder LUT is a three-dimensional LUT. In one embodiment, the at least one custom decoder LUT can receive a three-column input (e.g., RGB, XYZ) and generate more than three columns of output (e.g., RGBC, RGBCY, RGBCMY). This system has the advantage of requiring only three channels of data as input to the at least one custom decoder LUT. In one embodiment, the at least one custom decoder LUT applies a non-linear function (e.g., inverse gamma) and / or curve to generate a linear output. In another embodiment, the at least one custom decoder LUT is a cropping LUT.

[0228] At least one decoder formatter is capable of organizing the serial stream into a format suitable for at least one output. In a preferred embodiment, the decoder includes a corresponding decoder formatter for each of at least one decoder output. For example, if the decoder includes at least one HDMI output in the at least one decoder output, the decoder includes at least one HDMI formatter in the at least one decoder formatter. If the decoder includes at least one SDI output in the at least one decoder output, the decoder includes at least one SDI formatter in the at least one decoder formatter. If the decoder includes at least one Ethernet output in the at least one decoder output, the decoder includes at least one Ethernet formatter in the at least one decoder formatter, etc.

[0229] The encoder and / or decoder may generate, insert, and / or restore metadata associated with the image signal. The metadata may include, but is not limited to, color space (e.g., 6P-B, 6P-C), image transfer function (e.g., DRR, gamma, PQ, HLG, 1 / 2 DRR), peak white value, white point (e.g., D65, D60, DCI), image signal range (e.g., narrow (SMPTE) or full), sampling structure (e.g., 4:4:4, 4:2:2, 4:2:0, 4:1:1), bit depth (e.g., 8, 10, 12, 16), and / or signal format (e.g., RGB, Yxy, Yu'v', multi-primary (e.g., RGBCMY, RGBC)). In one embodiment, the metadata is inserted into SDI or ST2110 using ancillary (ANC) data packets. In another embodiment, the metadata is inserted using vendor-specific InfoFrame (VSIF) data as part of the CTA 861 standard. In some embodiments, the metadata is compatible with SMPTE ST 2110-10:2017, SMPTE ST 2110-20:2017, SMPTE ST 2110-40:2018, SMPTE ST 352:2013, and / or SMPTE ST 352:2011, the disclosures of all of which are incorporated herein by reference in their entireties.

[0230] Additional details regarding multi-primary color systems and display devices are described in U.S. Application Nos. 17 / 180,441 and 17 / 209,959, and U.S. Patent Publication Nos. 20210027693, 20210020094, 20210035487, and 20210043127, all of which are incorporated by reference in their entireties.

[0231] Display Engine

[0232] In one embodiment, the present invention operates in conjunction with a graphics processing unit (GPU) to provide Yxy, XYZ, YUV, Yu'v', RGB, YCrCb and / or IC T C PIn one embodiment, the display engine and GPU are mounted on a video card. Alternatively, the display engine and GPU are integrated on a motherboard or central processing unit (CPU) die. The display engine and GPU are preferably included in and / or connected to at least one viewing device (e.g., a display, a video game console, a smartphone, etc.). Additional information related to GPUs can be found in U.S. Patent Nos. 9,098,323, 9,235,512, 9,263,000, 9,318,073, 9,442,706, 9,477,437, 9,494,994, 9,535,815, 9,740,611, 9,779,473, 9,805,440, 9,880,851, 9,971,959, 9,978,343, and 10,032,244. , No. 10,043,232, No. 10,114,446, No. 10,185,386, No. 10,191,759, No. 10,229,471, No. 10,324,693, No. 10,331,590, No. 10,460 ,417, No. 10,515,611, No. 10,521,874, No. 10,559,057, No. 10,580,105, No. 10,593,011, No. 10,600,141, No. 10,628,909, No. 10 ,705,846, 10,713,059, 10,769,746, 10,839,476, 10,853,904, 10,867,362, 10,922,779, 10,923,082, 10,963,299, and 10,970,805, and U.S. Patent Publication Nos. 20140270364, 20150145871, 20160180487, 20160350245, and 20170178275 , 20170371694, 20180121386, 20180314932, 20190034316, 20190213706, 20200098082, 20200183734, 20200279348, 20200294183, 20200301708, 20200310522, 20200379864 and 20210049030, the disclosures of all of which are incorporated herein by reference in their entireties.

[0233] In some embodiments, the GPU includes a render engine. In some embodiments, the render engine includes at least one render pipeline (RP), a programmable pixel shader, a programmable vector shader, a vector array processor, a curve engine, and / or a memory cache. The render engine can operate in conjunction with a memory controller interface, a command CPU, a host bus (e.g., Peripheral Component Interconnect (PCI), PCI Express (PCIe), Accelerated Graphics Port (AGP)), and / or adaptive full-frame anti-aliasing. The memory controller interface can operate in conjunction with a display memory (e.g., Double Data Rate (DDR) memory), a pixel cache, the command CPU, the host bus, and the display engine. The command CPU can exchange data with the display engine.

[0234] FIG. 33 illustrates one embodiment of a display engine operable in conjunction with a graphics processing unit (GPU) according to the present invention. In a preferred embodiment, the display engine operable in conjunction with the GPU is mounted on a video card. The video card is capable of interfacing with a computer. In a preferred embodiment, the video card is insertable into a connector (e.g., a PCIe connector, a PCI connector, an Accelerated Graphics Port (AGP) connector, etc.) provided within the computer. The computer includes a command central processing unit (CPU). The command CPU is dedicated to communication between the video card and the computer core. The command CPU is preferably capable of receiving commands from an application programming interface (API). The command CPU is further capable of distributing appropriate commands to each component within the video card. The video card further includes a memory controller interface. The memory controller interface is preferably a bus and includes hardware that manages the data allowed on the bus and where the data is routed.

[0235] In one embodiment, multiple video cards are connected to enable scaling of graphics processing. In one embodiment, the multiple video cards are connected via a PCIe connector. Other connectors are also compatible with the multiple video cards. In one embodiment, each of the multiple video cards has the same technical specifications. In one embodiment, the API includes a method for scaling graphics processing, and the command CPU can distribute the graphics processing to the multiple video cards. The command CPU can scale up and down the graphics processing based on the processing and / or power requirements of the system.

[0236] The display engine receives rendered data from the GPU and converts it into a format that can be displayed on at least one viewing device. The display engine includes a raster scaler, at least one video display controller (e.g., an XYZ video display controller, an RGB video display controller, an IC T C P a video display controller), a color channel to XYZ converter, a linear converter, a scaler and / or a limiter, a multi-column LUT having at least three columns (e.g., a three-dimensional (3D) LUT (e.g., a 129 3 The color channel-to-XYZ converter may include a Yu'v'-to-XYZ converter, an IC, an XYZ-to-Yxy converter, an XYZ-to-Yu'v' converter, a non-linear function and / or tone curve applicator (e.g., 1 / 2 DRR), a sampling selector, a video bus, and / or at least one output formatter and / or encoder (e.g., ST 2082, ST 2110, DisplayPort, HDMI). In some embodiments, the color channel-to-XYZ converter includes an RGB-to-XYZ converter. Additionally or alternatively, the color channel-to-XYZ converter may include a Yu'v'-to-XYZ converter, an IC T C PThe video bus may include an ACES-XYZ converter and / or an ACES-XYZ converter. The video bus may receive input from a graphics display controller and / or at least one input device (e.g., a cursor, a mouse, a joystick, a keyboard, a video game controller, etc.).

[0237] The video card can be connected via any number of lanes provided by the hardware on the computer. The video card can communicate via a communication interface, including, but not limited to, a PCIe physical layer (PHY) interface. In one embodiment, the communication interface is an API supported by the computer (e.g., OpenGL, Direct3D, OpenCL, Vulkan). Image data in the form of vector data or bitmap data is output from the communication interface to a command CPU. The communication interface can notify the command CPU that image data is available. The command CPU opens a bidirectional gate on the bus, instructing the memory controller interface to send the image data to double data rate (DDR) memory. The memory controller interface can open a path for the image data to be rendered from the DDR memory to the GPU. After rendering, the image data is stored back in the DDR memory while awaiting processing by the display engine for output.

[0238] After the image data is rendered and stored in DDR memory, the command CPU instructs the memory controller interface to allow the rendered image data to be loaded into the raster scaler. The command CPU loads framing information into the raster scaler. The framing information includes, but is not limited to, a start of file (SOF) identifier, an end of file (EOF) identifier, pixel count, pixel order, multi-primary color data (e.g., RGBCMY data), and / or frame rate. In some embodiments, the framing information includes HDMI and / or DisplayPort (e.g., CTA 861 format) information. In some embodiments, Extended Display Identification Data (EDID) can override API specifications. The raster scaler outputs image data formatted as rasters in the same format as the file being loaded. In some embodiments, the output of the raster scaler is RGB data, XYZ data, or Yxy data. Alternatively, the output of the raster scaler is Yu'v' data, IC T C P data or ACES data.

[0239] In some embodiments, the output of the raster scaler is sent to a graphics display controller. In some embodiments, the graphics display controller can provide display information for a graphical user interface (GUI). In some embodiments, the RGB video controller and the XYZ video controller block image data from entering the video bus. The raster data is not limited to synchronization data, SOF, EOF, frame rate, pixel order, multi-primary color data (e.g., RGBCMY data), and / or pixel count. In some embodiments, the raster data is limited to an RGB output that can be sent to at least one output formatter and / or encoder.

[0240] For general video display, an alternate path is provided. The alternate path can provide outputs including, but not limited to, SMPTE SDI, Ethernet, DisplayPort, and / or HDMI to at least one output formatter and / or encoder. At least one video display controller (e.g., an RGB video display controller) can limit and / or optimize video data for streaming and / or compression. In one embodiment, the RGB video display controller and the XYZ video display controller block image data from entering the video bus.

[0241] In a preferred embodiment, image data is provided from the raster scaler in the format provided in the file being played (e.g., RGB, multi-primary (e.g., RGBCMY), XYZ, Yxy, Yu'v'). In one embodiment, the raster scaler presets the XYZ video display controller as the format provided and stored within the raster size to be displayed. In one embodiment, non-linear information (e.g., OOTF) sent from the API via a command CPU is sent to the linear converter. The linear converter can use the non-linear information. For example, if the image data was created using OETF, the inverse of the OETF is available to the linear converter, or if the image information already has EOTF applied, the inverse of the EOTF is available to the linear converter. In one embodiment, the linear converter creates an EOTF map to linearize the input data (e.g., if EOTF data is available). In one embodiment, the linear converter uses an already available EOTF. After the linear data is loaded and the summation process is performed, the XYZ video display controller passes through the image data in its native format (e.g., RGB, multi-primary data (e.g., RGBCMY), XYZ, Yxy, Yu'v'), but without any nonlinearity applied to the luminance (e.g., Y) component. The color channel to XYZ converter can accept the native format (e.g., RGB, multi-primary data (e.g., RGBCMY), XYZ, Yxy, Yu'v') and convert it to XYZ format. In one embodiment, the XYZ format includes at least one chromatic adaptation (e.g., D60 to D65). In the case of RGB, the XYZ video display controller obtains the color gamut and white point specifications from the API and converts to XYZ output using data provided by the command CPU. In the case of multi-primary systems, a corresponding matrix or look-up table (LUT) is used to convert from the multi-primary system to XYZ. In one embodiment, the multi-primary color system is RGBCMY (e.g., 6P-B, 6P-C, S6Pa, S6Pb). For a Yxy system, a color channel to XYZ converter formats the Yxy data back to XYZ data.In the case of a Yu'v' system, the color channel to XYZ converter formats the Yu'v' data back into XYZ data. In another embodiment, the color channel to XYZ converter is bypassed. For example, if it is necessary to remain in a multi-primary system, the color channel to XYZ converter is bypassed. Additionally, in the case of XYZ data, the color channel to XYZ converter is bypassed.

[0242] In some embodiments, the input to the scaler and / or limiter is XYZ data or multi-primary color data. In some embodiments, the multi-primary color data is RGBCMY (e.g., 6P-B, 6P-C, S6Pa, S6Pb), RGBC, RG1G2B, RGBCW, RGBCY, RG1G2BW, RGBW R W G W B or R1R2G1G2B1B2. Other multi-primary data formats are also compatible with the present invention. The scaler and / or limiter can map out-of-gamut values ​​(e.g., negative values) to in-gamut values ​​(e.g., out-of-gamut values ​​resulting from the conversion to XYZ). In some embodiments, the scaler and / or limiter uses a gamut mapping algorithm to map out-of-gamut values ​​to in-gamut values.

[0243] In some embodiments, the input to the scaler and / or limiter is multi-primary color data, with all channels optimized to have values ​​between 0 and 1. For example, if the input is RGBCMY data, all six channels are optimized to have values ​​between 0 and 1. In some embodiments, the output of the scaler and / or limiter can be input to a three-dimensional (3D) multi-column LUT. In some embodiments, the 3D multi-column LUT includes one column for each channel. For example, if the output is RGBCMY data, the 3D multi-column LUT includes six columns (i.e., one column for each channel). Within the application that supplies the API, each channel can be selected to balance the white point and / or shade the image in a specific color channel direction. In some embodiments, if the output of the scaler and / or limiter is XYZ data, the 3D multi-column LUT is bypassed. The output of the 3D multi-column LUT is sent to an XYZ-to-Yxy converter, where the conversion is performed using a simple addition process. Alternatively, the output of the 3D multi-column LUT is sent to an XYZ-to-Yu'v' converter. In one embodiment, if the video data is RGBCMY, the processing of the XYZ-Yxy converter or the XYZ-Yu'v' converter is bypassed.

[0244] Since the image data is linear, it is possible to add any tone curve to luminance (e.g., Y). The advantage of this invention is that by using, for example, Yxy data or Yu'v' data, tone curve correction is only required for luminance. For L*a*b*, 1 / 3 gamma is applied to all three channels. IPT and IC T C Poperates gamma on all three channels. A tone curve can be added only to luminance (e.g., Y), while colorimetric coordinates (e.g., x and y channels, u' and v' channels) remain linear. The tone curve can be anything (e.g., a non-linear function), including currently used standard values. In one embodiment, the tone curve is an EOTF (e.g., one described for television and / or digital cinema). Additionally or alternatively, the tone curve includes HDR correction. In another embodiment, a non-linear transfer function is added to all three channels (e.g., Yxy or Yu'v').

[0245] In one embodiment, the output is processed through this process as 3-6 individual components (e.g., 3 components for Yxy, Yu'v', or XYZ, 6 components for RGBCMY, etc.). The present invention is compatible with other primary color and component counts. However, for some serial formats, this level of payload is too large. In one embodiment, the sampling selector sets the sampling structure (e.g., 4:4:4, 4:2:2, 4:2:0, 4:1:1). In one embodiment, the sampling selector can subsample the processed image data. The sampling selector is preferably controlled by the command CPU. In one embodiment, the command CPU obtains information from the API and / or display device EDID. In a preferred embodiment, the sampling selector places each component in the correct serial data location as shown in Table 11 (discussed above).

[0246] The output of the Sampling Select is provided to the Main Video Bus, where the SOF and EOF information is integrated into the image data. It is then distributed to at least one output formatter and / or encoder. In one embodiment, the output is RGBCMY. In one embodiment, the RGBCMY output is configured as 4:4:4:4:4:4 data. Formats to at least one viewing device include, but are not limited to, SMPTE ST2082 (e.g., 3G, 6G, and 12G serial data output), SMPTE ST2110 (e.g., via Ethernet), and / or CTA 861 (e.g., DisplayPort, HDMI). The video card preferably has appropriate connectors (e.g., DisplayPort, HDMI) for distribution via any external system (e.g., a computer) and connection to at least one viewing device (e.g., a monitor, television, etc.). The at least one viewing device may be, but is not limited to, a smartphone, a tablet, a laptop screen, a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a miniLED display, a microLED display, a liquid crystal display (LCD), a quantum dot display, a quantum nano light emitting diode (QNED) device, a personal gaming device, a virtual reality (VR) device and / or an augmented reality (AR) device, an LED wall, a wearable display, and the at least one projector. In some embodiments, the at least one viewing device is a single viewing device.

[0247] Figure 34 shows one embodiment of a process flow diagram for converting an image for display. An image from an image source undergoes a linear transformation and / or scaling (e.g., scaling from 0 to 1) to produce a processed image. The processed image undergoes an RGB-to-XYZ transformation and an XYZ-to-Yxy transformation. Alternatively, the processed image undergoes an XYZ-to-Yu'v' transformation. At least one nonlinear transfer function (NLTF) (e.g., 1 / 2 DRR) is applied to luminance (Y) to generate luma (Y'). In a preferred embodiment, the x and y colorimetric coordinates are scaled, and then at least one NLTF is applied. Alternatively, the u' and v' colorimetric coordinates are scaled, and then at least one NLTF is applied. The colorimetric coordinates (x and y, u' and v') are fully sampled or subsampled. The luma and colorimetric coordinates (e.g., full sampling, sub-sampling) are formatted in an output formatter (e.g., ST 2082, ST 2110, DisplayPort, HDMI) before signal transmission to generate formatted luma and colorimetric coordinates. The formatted luma and colorimetric coordinates are decoded using at least one inverse function (e.g., inverse 1 / 2 DRR), rescaling of chromaticity coordinates (x and y, u′ and v′), and resampling reconstruction to generate decoded image data. The decoded image data undergoes a Yxy-XYZ transformation or a Yu′v′-XYZ transformation and an XYZ-gamut (e.g., RGB, ITU-R BT.709, SMPTE RP431, ITU-R BT.2020, ACES, custom, multi-primary) transformation, followed by a gamma function to generate image data. The image data can be sent to a display device (e.g., to display the color gamut). In some embodiments, the image data undergoes further processing within the display device.

[0248] The top of the diagram shows the process typically found within a camera or image generating device. The bottom of the diagram shows the decoding process typically located within a display device. The image is acquired from a camera or generated from an electronic source. Typically, gamma is applied and must be removed to obtain a linear image. After the linear image is acquired, it is scaled from 0 to 1, allowing it to be scaled to the desired luminance on the display device. The source can detail information related to the image, including, but not limited to, the device's color gamut and / or the white point used during acquisition. Accurate XYZ conversion can be achieved using adaptation techniques (e.g., chromatic adaptation). After an image is encoded as XYZ, it can be converted to Yxy. Each component can be split into a Y and xy path or a Y and u'v' path. Nonlinearity (e.g., DRR) is applied to the Y component. In some embodiments, nonlinearity (e.g., DRR) is also applied to the scaled xy or u'v' component. The xy or u'v' components can be subsampled as needed, for example, to fit the application without loss of luminance information. These are recombined and input into a formatting process that formats the signal for transmission (e.g., SDI, IP packets).

[0249] After the signal reaches the receiver, it is decoded to output the separated Yxy or Yu'v' components. The Y channel preferably undergoes an inverse nonlinearity (e.g., inverse DRR) to restore it to linear space. If a nonlinearity was applied to the xy or u'v' channel, the inverse nonlinearity (e.g., inverse DRR) is also preferably applied to the xy or u'v' channel to restore the image data (i.e., Yxy, Yu'v') to linear space and then rescaled to its original value. The xy or u'v' channel is then returned to full subpixel sampling. These are converted from Yxy to XYZ or from Yu'v' to XYZ. XYZ can then be converted to the display color gamut (e.g., RGB). Because a linear image is used, any gamma can be applied by the display device. This advantage places image limitations on the maximum performance of the display device, not on the signal itself.

[0250] This method allows for image matching between display devices with different gammas, color gamuts, and / or primaries (e.g., multi-primary). Colorimetric information and luminance are presented as linear values. Any white point, gamma, and / or color gamut can be defined, for example, as a scene-referred or display-referred set. Furthermore, heterogeneous display devices can be connected and matched if the image parameters fit within the display device's limitations. This advantage allows for accurate comparison without conversion.

[0251] In any system, the camera settings and display device capabilities are known. Current methods adapt the acquired image to assumed display specifications. Even in advanced systems (e.g., ACES), the final output is adapted to known display specifications. The design intent of the Yxy or Yu'v' system is to avoid these processes by using image coding techniques that allow the display device to perform at its best while preserving the creative intent.

[0252] The system can be divided into simpler parts for explanation: (1) camera / acquisition, (2) file and storage, (3) transmission, and (4) display. Most professional cameras have documentation that lists the possible color gamuts, the OETF used by the camera, and / or the white point to which the camera was balanced. In an RGB system, these parameters must be tracked and corrected throughout the workflow.

[0253] However, in a Yxy or Yu'v' system, in one embodiment, these transformations are performed by the camera as part of the encoding process because the image parameters are known at the time of capture. Thus, the Yxy or Yu'v' system has inherent colorimetric and luminance information without carrying additional image metadata. Alternatively, the transformations can be performed outside the camera in a dedicated encoder (e.g., hardware) or in image processing (e.g., software) in a post-production application.

[0254] FIG. 35 illustrates one embodiment of a camera process flow. An image sensor (e.g., an RGB sensor) within a camera can acquire image data. In one embodiment, the image data is processed by the camera (e.g., via a camera manufacturer's proprietary process) to generate processed camera data. The image data or camera data undergoes a linear transformation and / or scaling (e.g., scaling from 0 to 1) to generate a processed image. In one embodiment, the processed image undergoes a transformation from the acquisition color gamut (e.g., RGB) to Yxy or Yu'v'. In one embodiment, a nonlinear transfer function (NLTF) (e.g., DRR) is applied to Y (e.g., for luma generation) and xy. In another embodiment, an NLTF (e.g., DRR) is applied to Y (e.g., for luma generation) and u'v'. The luma and colorimetric coordinates (e.g., full sampling, sub-sampling) are formatted in an output formatter (e.g., ST 2082, ST 2110, DisplayPort, HDMI) to provide output data. In one embodiment, the output data is sent to a display device and / or decoder. The example shown in Figure 35 uses Yxy or Yu'v' data, but the system can be used with multiple formats.

[0255] Images are acquired through a specific process designed by the camera manufacturer. Instead of using a RAW output format, the process begins by converting the RGB channels to a linear (e.g., 16-bit) data format, where the RGB data is normalized to 1. In one embodiment, this linear image is converted from RGB to XYZ (e.g., via a transformation matrix) and then processed to generate a Yxy or Yu'v' data stream. Y is maintained as a fully sampled value, but xy or u'v' can be subsampled (e.g., 4:2:2, 4:2:0). DRR values ​​are applied to the Yxy or Yu'v' and scaled x and y or u' and v' values ​​before being sent as a serial data stream or stored in an appropriate file container.

[0256] The greatest advantage of the Yxy or Yu'v' system is that it allows you to send a single signal format to any display device and obtain an accurate image. The signal contains all the image information, allowing the display device design to be optimized for optimal performance. Issues such as panel and backlight accuracy can be addressed with a tailored image color gamut and brightness based on the Yxy or Yu'v' data.

[0257] Prior art display devices use a specific color gamut. Typically, the specific color gamut is an RGB color gamut (e.g., Rec. 2020, P3, Rec. 709). Comparing different display devices using Yxy or Yu'v' inputs offers significant advantages. Images displayed on a BT.709 monitor will match on a P3 monitor and a BT.2020 monitor for all colors that fall within the BT.709 monitor's color gamut. Colors outside that gamut are controlled by individual monitors optimized for each device. If the image's color gamut falls within the P3 color space, it will match on the P3 monitor and the BT.2020 monitor, until the image's color gamut exceeds the capabilities of the P3 monitor.

[0258] The display input process is like the inverse of the camera process, except that the output of this process can be adapted to any display parameters using the same image data.

[0259] FIG. 36 illustrates one embodiment of a display process flow. In one embodiment, a Yxy or Yu'v' signal is input as a digital signal. In one embodiment, the digital signal undergoes equalization processing. The formatted luma and colorimetric coordinates are decoded using the inverse of at least one nonlinear function (e.g., inverse 1 / 2 DRR) to generate decoded image data. In one embodiment, the decoded image data undergoes a Yxy-to-XYZ or Yu'v' or XYZ conversion to generate XYZ data. The XYZ data can be converted from XYZ to a color gamut (e.g., RGB, ITU-R BT.709, SMPTE RP431, ITU-R BT.2020, ACES, custom, multi-primary) using a color gamut library to generate color gamut data. In one embodiment, a gamma library can apply at least one function (e.g., linear, 2.2, 2.35, 2.4, 2.6 gamma function, HLG, PQ, custom) to the color gamut data. In one embodiment, the color gamut data (which may or may not have had at least one function applied) undergoes a calibration process (e.g., using a LUT) before being sent to the display panel and / or modulator. While Figure 36 shows Yxy or Yu'v' signals, the system accommodates multiple data formats.

[0260] Most image file formats are based on storing RGB data, which usually only accommodate three datasets. A Yxy or Yu′v′ implementation has the advantage that it only requires three datasets, making it easy to substitute into any file format.

[0261] The ability to move Yxy or Yu'v' encoded image content in real time over transport systems commonly used in production, broadcast, and streaming applications is essential. The requirement calls for a simple system that requires minimal changes to the current infrastructure. Yxy or Yu'v' encoding of image data allows for a simple substitution, requiring only modification of the payload data used to identify the encoding type.

[0262] The design of the RGB system uses the information acquired from the camera to build a well-formed electrical representation within the signal. This means that each signal fed to a process or display device must be formatted or reformatted so that it can be displayed correctly. Yxy or Yu'v' redefines this, shifting the formatting process to the acquisition and display device, with the advantage of allowing for consistent signals across different devices. Connections within the system are simplified, as connection and display settings are independent of the signal format.

[0263] System 4 Replacement

[0264] In SMPTE and CTA serial data streams, as well as SMPTE Ethernet streams, the substitution of Yxy or Yu'v' formats preferably follows that shown in Table 12.

[0265] [Table 12]

[0266] In a preferred embodiment, the payload ID identifies Yxy or Yu'v' in byte 4 as shown in Figure 37. Figure 38A shows an embodiment of a payload ID based on SMPTE ST352:2013 and ST292:2018. Figure 38B shows an embodiment of a payload ID based on SMPTE ST352:2013 and ST372:2017. Figure 38C shows an embodiment of a payload ID based on SMPTE ST352:2013 and ST425:2017.

[0267] FIG. 39 illustrates one embodiment of System 4 Yxy 10-bit 4:2:2 encoding as applied to SMPTE ST292 (e.g., SMPTE ST292-1:2018).

[0268] 40A-40B show the Yxy 10-bit 4:4:4 YC coding scheme of System 4 as applied to SMPTE ST372 (e.g., SMPTE ST372:2017). B C R FIG. 40A illustrates an embodiment of encoding. In one embodiment, the encoding utilizes a first link (e.g., link A) and a second link (e.g., link B). FIG. 40A illustrates a Yxy 10-bit 4:4:4 YC encoding for System 4 as applied to SMPTE ST372. B C R Figure 40B shows an embodiment of the first link of encoding. System 4 Yxy 10-bit 4:4:4 YC applied to SMPTE ST372. B C R 10 illustrates an embodiment of a second link of encoding.

[0269] 41A-41B illustrate one embodiment of System 4 10-bit 4:4:4 Yxy RGB encoding as applied to SMPTE ST372 (e.g., SMPTE ST372:2017). In one embodiment, the encoding utilizes a first link (e.g., Link A) and a second link (e.g., Link B). FIG. 41A illustrates one embodiment of the first link of System 4 Yxy 10-bit 4:4:4 RGB encoding as applied to SMPTE ST372. FIG. 41B illustrates one embodiment of the second link of System 4 Yxy 10-bit 4:4:4 RGB encoding as applied to SMPTE ST372.

[0270] 42A-42B show the Yxy 12-bit 4:4:4 YC coding scheme of System 4 as applied to SMPTE ST372 (e.g., SMPTE ST372:2017). B C R FIG. 42A illustrates one embodiment of encoding. In one embodiment, the encoding utilizes a first link (e.g., link A) and a second link (e.g., link B). FIG. 42A illustrates a Yxy 12-bit 4:4:4 YC encoding for System 4 as applied to SMPTE ST372. B C R Figure 42B shows an embodiment of the first link of encoding. System 4 Yxy 12-bit 4:4:4 YC applied to SMPTE ST372. B C R 10 illustrates an embodiment of a second link of encoding.

[0271] 43A-43B illustrate one embodiment of System 4 12-bit 4:4:4 Yxy RGB encoding as applied to SMPTE ST372 (e.g., SMPTE ST372:2017). In one embodiment, the encoding utilizes a first link (e.g., Link A) and a second link (e.g., Link B). FIG. 43A illustrates one embodiment of the first link of System 4 Yxy 12-bit 4:4:4 RGB encoding as applied to SMPTE ST372. FIG. 43B illustrates one embodiment of the second link of System 4 Yxy 12-bit 4:4:4 RGB encoding as applied to SMPTE ST372.

[0272] 44A-44B illustrate an embodiment of System 4 Yxy 10-bit 4:2:2 Level A encoding applied to SMPTE ST425 (e.g., SMPTE ST425-1:2017) ("Mapping Structure 1"). FIG. 44A illustrates an embodiment of System 4 Yxy 10-bit 4:2:2 Level A encoding applied to SMPTE ST425 for a first data stream. FIG. 44B illustrates an embodiment of System 4 Yxy 10-bit 4:2:2 Level A encoding applied to SMPTE ST425 for a second data stream.

[0273] 45A-45B illustrate an embodiment of System 4 Yxy 10-bit 4:4:4 Level A encoding applied to SMPTE ST425 (e.g., SMPTE ST425-1:2017) ("Mapping Structure 2"). FIG. 45A illustrates an embodiment of System 4 Yxy 10-bit 4:4:4 Level A encoding applied to SMPTE ST425 for a first data stream. FIG. 45B illustrates an embodiment of System 4 Yxy 10-bit 4:4:4 Level A encoding applied to SMPTE ST425 for a second data stream.

[0274] 46A-46B illustrate an embodiment of System 4 Yxy 12-bit 4:4:4 Level A encoding applied to SMPTE ST425 (e.g., SMPTE ST425-1:2017) ("Mapping Structure 3"). FIG. 46A illustrates an embodiment of System 4 Yxy 12-bit 4:4:4 Level A encoding applied to SMPTE ST425 for a first data stream. FIG. 46B illustrates an embodiment of System 4 Yxy 12-bit 4:4:4 Level A encoding applied to SMPTE ST425 for a second data stream.

[0275] 47A-47B illustrate an embodiment of System 4 Yxy 12-bit 4:2:2 Level A encoding applied to SMPTE ST425 (e.g., SMPTE ST425-1:2017) ("Mapping Structure 4"). FIG. 47A illustrates an embodiment of System 4 Yxy 12-bit 4:2:2 Level A encoding applied to SMPTE ST425 for a first data stream. FIG. 47B illustrates an embodiment of System 4 Yxy 12-bit 4:2:2 Level A encoding applied to SMPTE ST425 for a second data stream.

[0276] 48A-48B illustrate an embodiment of System 4 Yxy Level B multiplex dual stream (DS) encoding applied to SMPTE ST425 (e.g., SMPTE ST425-1:2017). FIG. 48A illustrates an embodiment of a first data stream of System 4 Yxy Level B multiplex dual stream (DS) encoding applied to SMPTE ST425. FIG. 48B illustrates an embodiment of a second data stream of System 4 Yxy Level B multiplex dual stream (DS) encoding applied to SMPTE ST425.

[0277] 49A-49B illustrate an embodiment of System 4 Yxy 10-bit Level B multiplex dual link (DL) encoding as applied to SMPTE ST425 (e.g., SMPTE ST425-1:2017). FIG. 49A illustrates an embodiment of a first data link of System 4 Yxy 10-bit Level B multiplex dual link (DL) encoding as applied to SMPTE ST425. FIG. 49B illustrates an embodiment of a second data link of System 4 Yxy 10-bit Level B multiplex dual link (DL) encoding as applied to SMPTE ST425.

[0278] 50A-50B illustrate an embodiment of System 4 Yxy 12-bit Level B multiplex dual link (DL) encoding as applied to SMPTE ST425 (e.g., SMPTE ST425-1:2017). FIG. 50A illustrates an embodiment of a first data link of System 4 Yxy 12-bit Level B multiplex dual link (DL) encoding as applied to SMPTE ST425. FIG. 50B illustrates an embodiment of a second data link of System 4 Yxy 12-bit Level B multiplex dual link (DL) encoding as applied to SMPTE ST425.

[0279] In one embodiment, the format is compatible with SMPTE ST2022-6 (2012). Advantageously, the Yxy or Yu'v' identification is included in the mapped payload ID, so no additional identification is required. SMPTE ST2022 does not describe any mapping changes, so mapping to Ethernet only needs to follow the appropriate SDI standard. In one embodiment, map code 0x00 uses Level A direct mapping from SMPTE ST292 or SMPTE ST425. In one embodiment, map code 0x01 uses Level B direct mapping formatted as SMPTE ST372 DL. In one embodiment, map code 0x02 uses Level B direct mapping formatted as SMPTE ST292 DS.

[0280] Figure 51 is a table of SMPTE ST2036-1(2014) parameters modified to include System 4 (e.g., Yxy, Yu'v').

[0281] Table 13 shows the structure of a 4:4:4p group. Table 14 shows the structure of a 4:2:2p group. Table 15 shows the structure of a 4:2:0p group.

[0282] [Table 13-1] [Table 13-2]

[0283] [Table 14]

[0284] [Table 15]

[0285] In one embodiment, the SDP parameters are defined using SMPTE ST2110-20(2017). In one embodiment, the Yxy or Yu'v' system uses CIE S 014-3:2011 as the colorimetric standard. Table 16 shows one embodiment of the modifications to the SDP colorimetric flags.

[0286] [Table 16]

[0287] As an example, the SDP parameters for the Yxy system are as follows: m = video 30000 RTP / AVP 112, a = rtpmap:112 raw / 90000, a = fmtp:112, sampling = YCbCr-4:2:2, width = 1280, height = 720, exactframerate = 60000 / 1001, depth = 10, TCS (Transfer Characteristic System) = SDR, colorimetry = Yxy, PM = 2110GPM, SSN = ST2110-20:2017.

[0288] 52 is a table of CTA 861 Table 6 - Colorimetric Transfer Characteristics modified to include System 4 (e.g., Yxy, Yu'v'). In one embodiment, the CTA 861 standard conforms to CTA 861-H(2021), which is incorporated herein by reference in its entirety.

[0289] Figure 53A is a table of Yxy 8-bit 4:2:2 encoding for 4 lanes, Figure 53B is a table of Yxy 8-bit 4:2:2 encoding for 2 lanes, and Figure 53C is a table of Yxy 8-bit 4:2:2 encoding for 1 lane.

[0290] Figure 54A is a table of Yxy 10-bit 4:2:2 encoding for 4 lanes, Figure 54B is a table of Yxy 10-bit 4:2:2 encoding for 2 lanes, and Figure 54C is a table of Yxy 10-bit 4:2:2 encoding for 1 lane.

[0291] Figure 55A is a table of Yxy 12-bit 4:2:2 encoding for 4 lanes, Figure 55B is a table of Yxy 12-bit 4:2:2 encoding for 2 lanes, and Figure 55C is a table of Yxy 12-bit 4:2:2 encoding for 1 lane.

[0292] Figure 56A is a table of Yxy 16-bit 4:2:2 encoding for 4 lanes, Figure 56B is a table of Yxy 16-bit 4:2:2 encoding for 2 lanes, and Figure 56C is a table of Yxy 16-bit 4:2:2 encoding for 1 lane.

[0293] Figure 57A is a table of Yxy 10-bit 4:4:4 encoding for 4 lanes, Figure 57B is a table of Yxy 10-bit 4:4:4 encoding for 2 lanes, and Figure 57C is a table of Yxy 10-bit 4:4:4 encoding for 1 lane.

[0294] Figure 58A is a table of Yxy 12-bit 4:4:4 encoding for 4 lanes, Figure 58B is a table of Yxy 12-bit 4:4:4 encoding for 2 lanes, and Figure 58C is a table of Yxy 12-bit 4:4:4 encoding for 1 lane.

[0295] Figure 59A is a table of Yxy 16-bit 4:4:4 encoding for 4 lanes, Figure 59B is a table of Yxy 16-bit 4:4:4 encoding for 2 lanes, and Figure 59C is a table of Yxy 16-bit 4:4:4 encoding for 1 lane.

[0296] The identification of the connection in Yxy or Yu'v' format is preferably provided in the auxiliary video information (AVI) (e.g., for CTA 861). In one embodiment, the AVI is provided according to InfoFrame version 4, as shown in Figure 60. Additional information can be found in ANSI / CTA-861-H-2021, which is incorporated herein by reference in its entirety. See, for example, ANSI / CTA-861-H-2021 section 6.2. In one embodiment, the identification is located in data byte 14 (e.g., ACE3, ACE2, ACE1, ACE0). In one embodiment, ACE3=0, ACE2=0, ACE1=1, ACE0=1 identify a Yxy4:4:4 format image with no DRR applied; ACE3=0, ACE2=1, ACE1=0, ACE0=0 identify a Yxy4:2:2 format image with no DRR applied; ACE3=0, ACE2=1, ACE1=0, ACE0=1 identify a Yxy4:2:0 format image with no DRR applied; ACE3=0, ACE2=1, ACE1=1, ACE0=0 identify a Yxy4:4:4 format image with DRR applied; ACE3=0, ACE2=1, ACE1=1, ACE0=1 identify a Yxy4:2:2 format image with DRR applied; ACE3=1, ACE2=0, ACE1=0, ACE0=0 identify a Yxy4:2:0 format image with DRR applied; AC ACE3=1, ACE2=0, ACE1=0, ACE0=1 identifies a Yu′v′ 4:4:4 format image without DRR applied; ACE3=1, ACE2=0, ACE1=1, ACE0=0 identifies a Yu′v′ 4:2:2 format image without DRR applied; ACE3=1, ACE2=0, ACE1=1, ACE0=1 identifies a Yu′v′ 4:2:0 format image without DRR applied; ACE3=1, ACE2=1, ACE1=0, ACE0=0 identifies a Yu′v′ 4:4:4 format image with DRR applied; ACE3=1, ACE2=1, ACE1=0, ACE0=1 identifies a Yu′v′ 4:2:2 format image with DRR applied; ACE3=1, ACE2=1, ACE1=1, ACE0=0 identifies a Yu′v′ 4:2:0 format image with DRR applied.In another embodiment, ACE3=0, ACE2=0, ACE1=1, ACE0=1 identifies a Yxy4:4:4 format image; ACE3=0, ACE2=1, ACE1=0, ACE0=0 identifies a Yxy4:2:2 format image; and ACE3=0, ACE2=1, ACE1=0, ACE0=1 identifies a Yxy4:2:0 format image. In one embodiment, Data Byte 2 (C1, C0) is read as C1=1 and C0=1, and Data Byte 3 (EC2, EC1, EC0) is read as EC2=1, EC1=1, EC0=1. Table 17 shows the values ​​for Data Byte 2. Table 18 shows the values ​​for Data Byte 3. Table 19 shows the values ​​for Data Byte 14.

[0297] [Table 17]

[0298] [Table 18]

[0299] [Table 19]

[0300] Modification of the six-primary color system in the Session Description Protocol (SDP)

[0301] SDP is derived from IETF RFC 4566 and defines parameters including, but not limited to, bit depth and sampling parameters. IETF RFC 4566 (2006) is incorporated herein by reference in its entirety. In one embodiment, SDP parameters are included within the RTP payload. In another embodiment, SDP parameters are included within the media format and transport protocol. This payload information is transmitted as text. Therefore, modifying additional sampling identifiers requires the addition of new parameters for sampling description. SDP parameters include, but are not limited to, color channel data, image data, frame rate data, sampling standard, flag indicator, active picture size code, timestamp, clock frequency, frame count, scrambling indicator, and / or video format indicator. For non-constant luminance imaging, additional parameters include, but are not limited to, RGBCMY-4:4:4, YBRCY-4:2:2, and YBRCY-4:2:0. For constant luminance signals, additional parameters include, but are not limited to, CLYBRCY-4:2:2 and CLYBRCY-4:2:0.

[0302] Additionally, in one embodiment, differentiation by colorimetric identifier is included. For example, 6PB1 defines a 6P with a color gamut limited to ITU-R BT.709 as System 1, 6PB2 defines a 6P with a color gamut limited to ITU-R BT.709 as System 2, and 6PB3 defines a 6P with a color gamut limited to ITU-R BT.709 as System 3. 6PC1 defines a 6P with a color gamut limited to SMPTE RP 431-2 as System 1, 6PC2 defines a 6P with a color gamut limited to SMPTE RP 431-2 as System 2, and 6PC3 defines a 6P with a color gamut limited to SMPTE RP 431-2 as System 3. 6PS1 defines a 6P with a Super 6P gamut as System 1, 6PS2 defines a 6P with a Super 6P gamut as System 2, and 6PS3 defines a 6P with a Super 6P gamut as System 3.

[0303] Colorimetry can also be defined between six color primaries using the ITU-R BT.709-6 standard and the SMPTE ST431-2 standard, or it can remain standard-defined for the desired standard. For example, for a 1920x1080, six-color primaries, 10-bit signal using the ITU-R BT.709-6 standard, the SDP parameters for System 1 are: m = video 30000 RTP / AVP 112, a = rtpmap:112 raw / 90000, a = fmtp:112, sampling = YBRCY-4:2:2, width = 1920, height = 1080, exactframerate = 30000 / 1001, depth = 10, TCS = SDR, colorimetry = 6PB1, PM = 2110GPM, SSN = ST2110-20:2017.

[0304] In one embodiment, the six-primary system is integrated with a Consumer Technology Association (CTA) 861-based system. CTA-861 defines protocols, requirements, and recommendations for the use of uncompressed digital interfaces by consumer electronic devices, including but not limited to digital television (DTV), digital cable, satellite, or terrestrial set-top boxes (STBs), and DVD players and recorders, and other related sources or sinks.

[0305] These systems operate as parallel systems, splitting the video content across multiple line pairs. This allows each video component to have its own transition-minimized differential signaling (TMDS) path. TMDS is a high-speed serial data transmission technology used in Digital Visual Interface (DVI) and High-Definition Multimedia Interface (HDMI) video interfaces, as well as other digital communication interfaces. TMDS, like low-voltage differential signaling (LVDS), uses differential signaling to reduce electromagnetic interference (EMI) and enable faster and more accurate signal transmission. Additionally, TMDS uses twisted pairs for noise reduction, rather than the coaxial cable used for traditional video signal transmission. Like LVDS, data is transmitted serially over the data link. When transmitting video data, three TMDS twisted pairs are used to transmit video data when using HDMI.

[0306] In such systems, each pixel packet is limited to only 8 bits. For bit depths greater than 8 bits, split packing is used. This configuration is no different from what is already described in the current CTA-861 standard.

[0307] Based on CTA Extension Version 3, the identification of six-primary transmission is performed by the sink device (e.g., a monitor). Recognition of additional formats is flagged in the CTA Data Block Extension Tag Code (Byte 3). Codes 33 and above are reserved, so any two bits can be used to identify whether the format is RGB, RGBCMY, Y Cb Cr, or Y Cb Cr Cc Cy, or whether it is System 1 or System 2. If Byte 3 defines the six-primary sampling format and Block 5 Extension identifies Byte 1 as ITU-R BT.709, the logic assigns it as 6P-B. However, if Byte 4 bit 7 identifies the colorimetry as DCI-P3, the gamut is assigned as 6P-C.

[0308] In one embodiment, the system modifies the auxiliary video information (AVI) infoframe data to identify the content. The AVI infoframe data is shown in Table 10 of CTA 861-G. In one embodiment, Y2=1, Y1=0, Y0=0 identifies the content as 6P 4:2:0:2:0. In another embodiment, Y2=1, Y1=0, Y0=1 identifies the content as Y Cr Cb Cc Cy. In yet another embodiment, Y2=1, Y1=1, Y0=0 identifies the content as RGBCMY.

[0309] Byte 2 C1=1, C0=1 identify Extended Colorimetry in Table 11 of CTA 861-G. Byte 3 EC2, EC1, EC0 identify valid additional colorimetry extensions in Table 13 of CTA 861-G. Table 14 of CTA 861-G reserves additional extensions. In one embodiment, ACE3=1, ACE2=0, ACE1=0, ACE0=X identify 6P-B. In one embodiment, ACE3=0, ACE2=1, ACE1=0, ACE0=X identify 6P-C. In one embodiment, ACE3=0, ACE2=0, ACE1=1, ACE0=X identify System 1. In one embodiment, ACE3=1, ACE2=1, ACE1=0, ACE0=X identify System 2.

[0310] Figure 61 shows the current RGB sampling structure for 4:4:4 sampling video data transmission. In HDMI 4:4:4 sampling, video data is transmitted over three TMDS line pairs. Figure 62 shows a six-primary color sampling structure (RGBCMY) using System 1 for 4:4:4 sampling video data transmission. In one embodiment, the six-primary color sampling structure complies with CTA 861-G, published by the Consumer Technology Association in November 2016, and is incorporated herein by reference in its entirety. Figure 63 shows an example of RGBCMY 4:4:4 transmission from System 2. Figure 64 shows the current Y Cb Cr 4:2:2 sampling transmission as non-stationary luminance. Figure 65 shows a six-primary color system (System 1) using Y Cr Cb Cc Cy 4:2:2 sampling transmission as non-stationary luminance. Figure 66 shows an example of Y Cr Cb Cc Cy 4:2:2 transmission as non-stationary luminance from System 2. In one embodiment, the Y Cr Cb Cc Cy 4:2:2 sampling transmission complies with CTA 861-G by the Consumer Technology Association, November 2016. Figure 67 shows the current Y Cb Cr 4:2:0 sampling transmission. Figure 68 shows a six-primary system (System 1) using the Y Cr Cb Cc Cy 4:2:0 sampling transmission.

[0311] The HDMI sampling system includes Extended Display Identification Data (EDID) metadata. EDID metadata describes the capabilities of a display device to a video source. The data format is defined in standards published by the Video Electronics Standards Association (VESA). The EDID data structure includes, but is not limited to, the manufacturer name and serial number, product type, phosphor or filter type, timings supported by the display, display size, brightness data, and / or pixel mapping data. The EDID data structure is modifiable and does not require additional hardware or tools.

[0312] EDID information is transmitted between the source device and the display over the Display Data Channel (DDC), a set of digital communication protocols created by VESA. Both standards enable the exchange of information between the display and the source, with EDID providing the display information and DDC providing the link between the display and the source.

[0313] Additionally, VESA has assigned extensions to the EDID, including, but not limited to, Timing Extension (00), Additional Time Data Block (CEA EDID Timing Extension (02)), Video Timing Block Extension (VTB-EXT (10)), EDID 2.0 Extension (20), Display Information Extension (DI-EXT (40)), Localized String Extension (LS-EXT (50)), Microdisplay Interface Extension (MI-EXT (60)), Display ID Extension (70), Display Transfer Characteristics Data Block (DTCDB (A7, AF, BF)), Block Map (F0), Display Device Data Block (DDDB (FF)), and / or Monitor Manufacturer Defined Extension (FF).

[0314] In one embodiment, the SDP parameters include data corresponding to payload identification (ID) and / or EDID information. Additionally, VESA has assigned extensions to the EDID, including, but not limited to, Timing Extension (00), Additional Time Data Block (CEA EDID Timing Extension (02)), Video Timing Block Extension (VTB-EXT (10)), EDID 2.0 Extension (20), Display Information Extension (DI-EXT (40)), Localized String Extension (LS-EXT (50)), Microdisplay Interface Extension (MI-EXT (60)), Display ID Extension (70), Display Transfer Characteristics Data Block (DTCDB (A7, AF, BF)), Block Map (F0), Display Device Data Block (DDDB (FF)), and / or Monitor Manufacturer Defined Extension (FF). In one embodiment, the SDP parameters include data corresponding to payload identification (ID) and / or EDID information.

[0315] Multi-primary color system display

[0316] Figure 69 shows a dual-stack LCD projection system for a six-primary system. In one embodiment, the display is composed of a dual stack of projectors. This display uses two projectors stacked one on top of the other or arranged side-by-side. Each projector is similar, the only difference being the color filters in each unit. Refresh and pixel timing are synchronized, allowing for mechanical alignment between the two units so that each pixel is co-located between the projector units. In one embodiment, the two projectors are liquid crystal display (LCD) projectors. In another embodiment, the two projectors are digital light processing (DLP) projectors. In yet another embodiment, the two projectors are liquid crystal on silicon (LCOS) projectors. In yet another embodiment, the two projectors are light-emitting diode (LED) projectors.

[0317] In one embodiment, the display is comprised of a single projector. A single-projector six-primary system requires the addition of a second cross-block assembly for the additional color. One embodiment of a single projector (e.g., a single LCD projector) is shown in Figure 70. The single-projector six-primary system includes a cyan dichroic mirror, an orange dichroic mirror, a blue dichroic mirror, a red dichroic mirror, and two additional standard mirrors. In one embodiment, the single-projector six-primary system includes at least six mirrors. In another embodiment, the single-projector six-primary system includes at least two cross-block assembly units.

[0318] FIG. 71 illustrates a six-primary color system using a single projector and reciprocal mirrors. In one embodiment, the display is comprised of a single projector unit operating in combination with a first set of at least six reciprocal mirrors, a second set of at least six reciprocal mirrors, and at least six LCD units. Light from at least one light source is emitted toward the first set of at least six reciprocal mirrors. The first set of at least six reciprocal mirrors reflects the light to at least one of the at least six LCD units. The at least six LCD units may include, but are not limited to, a green LCD, a yellow LCD, a cyan LCD, a red LCD, a magenta LCD, and / or a blue LCD. The output of each of the at least six LCDs is received by the second set of at least six reciprocal mirrors. The output from the second set of at least six reciprocal mirrors is sent to the single projector unit. The image data output by the single projector unit is output as a six-primary color system. In other embodiments, there are more than two sets of reciprocal mirrors. In other embodiments, multiple projectors are used.

[0319] In another embodiment, the display is comprised of a dual-stack digital micromirror device (DMD) projector system. Figure 72 shows one embodiment of a dual-stack DMD projector system. In this system, two projectors are stacked one on top of the other. In one embodiment, the dual-stack DMD projector system uses a rotating wheel filter. In another embodiment, the dual-stack DMD projector system uses phosphor technology. In one embodiment, the filter system is illuminated by a xenon lamp. In another embodiment, the filter system uses a blue laser illumination system. One projector's filter system is RGB, while the other projector uses a CMY filter set. The wheels of each projector unit are synchronized using at least one of input video sync and inter-projector sync, and the timing is adjusted so that inverted colors are output from each projector simultaneously.

[0320] In one embodiment, the projector is a phosphor wheel system. A yellow phosphor wheel rotates in sync with the DMD imager, providing a continuous RG output. A second projector is similarly designed but uses a cyan phosphor wheel. The output from this projector is a continuous BG output. The combined output of both projectors is YRGGCB. Magenta is generated by synchronizing the yellow and cyan wheels and overlaying them with the DMD flashing.

[0321] In another embodiment, the display is a single DMD projector solution. The single DMD device is combined with an RGB diode light source system. In one embodiment, the DMD projector uses LED diodes. In one embodiment, the DMD projector includes CMY diodes. In another embodiment, the DMD projector uses double flashing technology to generate the CMY primaries. Figure 73 shows one embodiment of a single DMD projector solution.

[0322] Figure 74 shows one embodiment of a six-primary system using a white OLED display. In yet another embodiment, the display is a white OLED monitor. Current emissive monitor and / or television designs use white-emitting OLED arrays covered with color filters. Modification of this type of display requires only a change in pixel index and a change to a new six-primary filter. A different color filter array is used, with each subpixel positioned to provide minimal light restriction, color accuracy, and off-axis viewing.

[0323] FIG. 75 shows one embodiment of an optical filter array for a white OLED display.

[0324] Figure 76 shows one embodiment of an LCD drive matrix for a six-primary system with a backlit LCD monitor. In yet another embodiment, the display is a backlit LCD display. The design of the LCD display includes the addition of CMY subpixels. The drive for these subpixels is similar to the RGB matrix drive. With the advent of 8K LCD televisions, it is technically possible to modify the matrix drive and optical filters to create a 4K six-primary television.

[0325] Figure 77 shows an embodiment of an optical filter array for a six-primary system with a backlit LCD monitor. The optical filter array includes additional CMY sub-pixels.

[0326] In yet another embodiment, the display is a direct emissive assembly display. The direct emissive assembly display design includes a matrix of color emitters grouped as a six-color system, with each channel input driving a respective quantum dot (QD) element illuminator and / or micro LED element.

[0327] Figure 78 shows an array for a quantum dot (QD) display device.

[0328] Figure 79 shows one embodiment of an array for a six-primary system for use in a direct emissive assembly display.

[0329] Figure 80 shows one embodiment of a six-primary system for an emissive display that does not incorporate color filter subpixels. For LCD and WOLED displays, a six-primary system can be achieved by extending the RGB or WRGB filter arrangement to an RGBCMY matrix. In a WRGB system, the white subpixel can be eliminated because the luminance of the three additional primaries replaces it. SDI video is input through an SDI decoder. In one embodiment, the SDI decoder outputs to a YCrCbCcCy-to-RGBCMY converter. The converter outputs RGBCMY data with the luminance component (Y) subtracted. The RGBCMY data is converted to RGB data. This RGB data is sent to a scale sync generation component for image control, contrast, brightness, chroma, and saturation adjustments, and then to a color correction component, which outputs it as LVDS data to the display panel. In another embodiment, the SDI decoder outputs to an SDI YR switch component, which outputs the RGBCMY data. The RGBCMY data is sent to the scale synchronization generation component, where it is adjusted for image control, contrast, brightness, chroma, and saturation, then sent to the color correction component, and output to the display panel as LVDS data.

[0330] In one embodiment, the display is also capable of displaying ultra-saturated colors as described in U.S. Patent Application No. 17 / 748,655 (filed May 19, 2022, incorporated herein by reference in its entirety).

[0331] Single device image acquisition and display

[0332] In one embodiment, the present invention includes a device capable of acquiring, processing, and / or displaying image data. Such devices include, but are not limited to, cameras (e.g., digital video cameras, still cameras), mobile devices (e.g., smartphones), tablets, computers (e.g., desktop computers, laptops), monitors, wearable devices, personal digital assistants (PDAs), e-readers, digital media players, video game devices, video conferencing devices, video streaming devices, and / or augmented reality / virtual reality (AR / VR) devices (e.g., headsets, goggles, smart lenses). Such devices do not need to transfer data between separate components via wireless connections. Furthermore, such devices do not need to transfer data via long-distance wired and / or cabled connections (e.g., HDMI cables, SDI cables). Advantageously, the wired connections (e.g., soldered connections) of such devices can be short because they are within a single device. Therefore, such devices simplify the process of acquiring and displaying image data.

[0333] In one embodiment, the device includes at least one imager for acquiring image data. The at least one imager preferably includes at least one lens and at least one image sensor (e.g., a camera, video camera, camcorder, slow-motion camera, and / or high-speed camera). Charge-coupled device (CCD) image sensors and complementary metal-oxide semiconductor (CMOS) image sensors (e.g., active pixel sensors (APS), hybrid CCD / CMOS image sensors, n-type metal-oxide semiconductor (NMOS) image sensors, and quantum image sensors) are compatible with the present invention. In one embodiment, the at least one imager is a single imager with a stripe filter system. Alternatively, the at least one imager includes a red imager, a green imager, and a blue imager. The at least one lens directs light to the at least one image sensor. The at least one lens may include, but is not limited to, at least one convex lens and / or at least one concave lens. In one embodiment, the at least one image sensor is a wide color gamut image sensor, i.e., a wide color gamut camera. In one embodiment, the at least one image sensor is a single pixel image sensor. In one embodiment, the at least one image sensor does not include a detector array. In one embodiment, the at least one image sensor is a plurality of image sensors. In one embodiment, one or more of the at least one imager are replaceable, making the device compatible with multiple imagers. This modular design advantageously allows the at least one imager to be upgraded or replaced as image acquisition needs or technology evolve.

[0334] In one embodiment, the at least one imager includes multiple lenses for multiple image sensors. In one embodiment, the multiple lenses generate different focal lengths for each of the multiple image sensors. In one embodiment, the device is capable of changing focal lengths (e.g., by zooming). Alternatively, the device can interpolate signals from multiple image sensors with different focal lengths to generate hybrid sensor data. The device can combine sensor data from the multiple image sensors into a single image data set. In one embodiment, the device includes a stabilizer, such as a gyroscope system or an electronic image stabilization system. At least one imager is preferably disposed on the stabilizer, which moves the at least one imager to counteract motion that results in blurred images. In one embodiment, the at least one imager includes a lens mount (e.g., a screw mount, a bayonet mount, a breech lock, a tab lock, a double bayonet, a Z, an X, an electrofocus (EF), an EF-M, an EF-S, an AF, an E, an L, an RF, a G, an M, an SA, an A, a K, an F, an S, a PL, a T, a C, an H, or a 645 mount).

[0335] In one embodiment, the at least one imager includes at least one filter (e.g., an optical filter). In one embodiment, the at least one filter is disposed over the photosites on the at least one image sensor. In one embodiment, the at least one filter is an absorptive filter. Alternatively, the at least one filter is an interference filter or a dichroic filter. In one embodiment, the at least one filter has at least one cutoff wavelength and transmits or blocks light based on the cutoff wavelength (e.g., a long-pass filter, a short-pass filter, a band-pass filter, a multi-band-pass filter, or a notch filter). In another embodiment, the at least one filter varies the intensity of all wavelengths equally (e.g., a neutral density filter). In one embodiment, the at least one filter includes at least one color filter array (e.g., a Bayer filter, a quad-Bayer filter, a diamond-pattern color filter array, a mountain color filter array, a vertical stripe color filter array, a diagonal stripe color filter array, a pseudorandom color filter array, and / or a human visual system-based color filter array). Filter colors compatible with the present invention include, but are not limited to, RGB, CYGM, RGBE (red, green, blue, emerald), and / or CMY. At least one filter may be modifiable. As a non-limiting example, a Bayer filter may be modified to include a magenta filter. Alternatively, the element size within the Bayer filter may be adjusted to increase the sensitivity of at least one image sensor. In yet another embodiment, one or more of the at least one filter may be rotatable. In one embodiment, the at least one filter includes multiple filter layers. In one embodiment, the at least one filter includes at least one filter for light outside the visible wavelength range (e.g., an ultraviolet (UV) filter, an infrared (IR) filter). In one embodiment, the device may convert light captured through a non-visible wavelength filter into visible light to achieve visual effects such as UV / black light simulation.The at least one filter includes any number of color filters, hi one embodiment, the at least one filter includes an inverse color to increase the sensitivity of the at least one imager.

[0336] Single-device acquisition

[0337] In one embodiment, the device can capture raw image data as a RAW image file. RAW image files are considered unprocessed and cannot be edited or printed. RAW image files contain metadata and a header in addition to image data. Metadata includes, but is not limited to, image sensor parameters, imager parameters, timecode, frame data, HDR metadata, colorimetry metadata, aspect ratio, dimensions (e.g., pixel dimensions), and / or lens information (e.g., focal length, aperture, shutter speed, exposure time, sensitivity, white balance). RAW image formats include, but are not limited to, Digital Negative RAW (DNG), ISO 12234-2 (TIFF / EP), NIKON NEF, CANON Raw v2 (CR2), CR3, and / or REDCODE Raw (R3D) files. In one embodiment, the device can save the RAW image file before processing. The device can then convert the RAW image data to rendered image data, which can be displayed and / or edited. Rendering may include, but is not limited to, decoding, demosaicing (e.g., removing the effects of a Bayer filter), pixel removal (e.g., removing defective pixels), interpolation (e.g., replacing removed pixels), white balancing, noise reduction, color conversion, tone reproduction, optical correction, contrast manipulation, resizing, splitting, cropping, and / or compression. Alternatively, the device does not compress the raw image data. In one embodiment, the device can render the image data as a pipeline process, with each step performed sequentially. The order of the steps can be changed. Alternatively, the device can render the image data in parallel steps. In yet another embodiment, the device can render the image data by solving a single optimization problem. The device can store image prior data and / or image variation data and use them during rendering, processing, and / or display.

[0338] In one embodiment, the acquisition color gamut is the same as the display color gamut. In one embodiment, both the acquisition color gamut and the display color gamut are extended color gamuts and include at least four primary colors (e.g., 6P-B, 6P-C). Alternatively, the display color gamut (e.g., RGBCMY) has a larger volume than the acquisition color gamut (e.g., RGB). In yet another embodiment, the display color gamut (e.g., RGB) has a smaller volume than the acquisition color gamut (e.g., RGBCMY). Preferably, the device is capable of converting image data from the acquisition color gamut to the display color gamut.

[0339] In one embodiment, rendering includes converting the raw image data to a color space (e.g., CIE 1931, ITU-R BT.2020). In a preferred embodiment, the device is capable of rendering image data in a three-coordinate format, where the first coordinate is a luminance or luma value and the second and third coordinates are both colorimetry (chroma). As a non-limiting example, the three-coordinate format is Yxy, where Y is a luminance coordinate and x and y are orthogonal colorimetry coordinates. The device may also apply a transformation (e.g., gamma compression) to the luminance coordinate to generate a luma coordinate (e.g., Y'). Relative luminance values ​​are also compatible. Other three-coordinate formats include, but are not limited to, L*a*b*, ICtCp, YCbCr, YUV, Yu'v', YPbPr, and / or YIQ. Alternatively, the device may render the image data as XYZ data. In one embodiment, the device includes a user interface and accepts user input. In one embodiment, the raw image data is rendered based on the user input. In one embodiment, the device is capable of applying an optical-to-electrical transfer function (OETF) and an electrical-to-optical transfer function (EOTF) to the image data. Alternatively, the device is capable of applying at least one non-linear function (e.g., an OOTF) to the image data. In one embodiment, the device includes at least one look-up table (LUT). The LUT can be implemented in hardware (e.g., an FPGA) and / or software. In one embodiment, the rendering includes compressing the image data (e.g., 4:2:2 sampling, 4:2:0 sampling). In one embodiment, the rendering includes applying a gamut constraint of the target color gamut. Alternatively, the image data is uncompressed (4:4:4 sampling).

[0340] In one embodiment, the rendering further includes HDR processing to expand the visible luminance range of the image data. Displaying an HDR image typically requires the application of at least one transfer function (e.g., PQ, Hybrid Log Gamma (HLG)). In one embodiment, the device includes a PQ-enabled display and / or an HLG-enabled display and is capable of displaying HDR image data with at least one transfer function applied. In one embodiment, the device is capable of applying at least one tone mapping curve (e.g., an S-curve) to the image data to preserve highlight and shadow detail. In one embodiment, the metadata includes information regarding the at least one transfer function and / or the at least one tone mapping curve.

[0341] Single-device processing

[0342] In one embodiment, the device is also capable of processing and / or transforming rendered image data. In one embodiment, the device includes an encoder and decoder of the present invention in a single unit. In one embodiment, the device is capable of storing processed image data transmitted from the encoder to the decoder before decoding. Because the encoder and decoder are in the same device, data is transmitted between the encoder and decoder via a wired connection. The wired connection does not require an internet connection, Bluetooth, or any other type of wireless connection. Advantageously, storing data in an intermediate format allows for backup data to be generated in case the image data is corrupted or lost. Alternatively, the device can bypass the encoding and / or decoding steps because it is capable of both image acquisition and image display. For example, the device may encode image data as an HDMI input, eliminating the need for an HDMI receiving circuit to decode the HDMI input, thereby eliminating the need for an HDMI connection to display the image data. In another embodiment, the device is also capable of encoding image data for display on an additional display device separate from the device, and also displays the image data on the device's display screen. Advantageously, in one embodiment, the bit depth of the image data remains the same within the device throughout each step, from acquisition to display.

[0343] In one embodiment, the device can process and / or transform image data internally, such as with an embedded ARM (high-performance RISC (reduced instruction set computing) machine) processor. Alternatively, the device can perform remote image processing. For example, the device can be in network communication with a platform to send image data to the platform and receive image data from the platform. The platform can process the image data. In one embodiment, the platform is hosted on a server (e.g., a cloud-based server, a server on a distributed edge network). Alternatively, the device can be in wired communication with an external processor (e.g., a computer, a tablet) for image processing. In one embodiment, the device further includes a user interface, which can accept user input for editing the image data (e.g., brightness, saturation, contrast). In one embodiment, the device can edit the image data for specific characteristics (e.g., skin tone correction).

[0344] In one embodiment, the device is capable of subsampling image data for display. Advantageously, storing and processing image data in a three-coordinate system, such as Yxy, allows subsampling of chromaticity coordinates for display without affecting perception. By way of non-limiting example, 4:2:2, 4:2:0, and 4:1:1 subsampling are compatible with the present invention. Alternatively, the image data is fully sampled. In one embodiment, the device is capable of decompressing compressed image data.

[0345] In one embodiment, processing image data for display includes applying color matching functions (CMFs). CMFs are functions of three wavelengths:

number

[0346] Single Device View

[0347] In one embodiment, the device further includes a display device. The display device preferably displays image data using more than three primary colors. In one embodiment, the display device is capable of displaying colors outside the ITU-R BT.2020 color gamut. In one embodiment, the display device is capable of displaying at least 80% of the total gamut covered by the CIE-1931 color space. In one embodiment, the display device is described in U.S. Patent No. 11,030,934, filed October 1, 2020, and issued June 8, 2021, the entire contents of which are incorporated herein by reference in their entirety. In one embodiment, the display device is a screen, such as a liquid crystal display (LCD) screen, a light-emitting diode (LED) screen, an LED-backlit screen, an organic light-emitting diode (OLED) screen, an active matrix OLED (AMOLED) screen, a quantum dot (QD) display, an LCD display using a QD backlight, a perovskite display, and / or a laser display (e.g., using discrete modulation, diffraction grating modulation). In another embodiment, the display device includes at least one projector. The device may display the image data after it has been acquired, rendered, and / or processed. Additionally or alternatively, the device may receive image data for display from an external source. In another embodiment, the display device includes multiple display devices (e.g., screens, projectors).

[0348] In one embodiment, the device is capable of changing display parameters of the image data, including, but not limited to, color gamut, frame rate, sampling rate, aspect ratio, data format, metadata, and / or SDP parameters. In one embodiment, the display device of the device is interchangeable. In one embodiment, the device is capable of projecting image data onto a second display device, the second display device being separate from the device. For example, the device is capable of casting image data onto the second display device, which mirrors the display of the device (e.g., via a wireless or wired connection). Alternatively, the second display device extends the first display device. The device is further capable of applying tone curves, changing resolution, changing the color space of the image data, etc., to optimize display on the second display device.

[0349] Augmented Reality / Virtual Reality

[0350] In one embodiment, the system includes at least one headset (e.g., one headset, two headsets, etc.) configured for a virtual reality, augmented reality, and / or mixed reality environment ("AR / VR"). The headset preferably includes a display device, eyewear components, at least one power source, at least one image capture device, and / or control electronics. In one embodiment, the headset is a pair of goggles. Alternatively, the headset is a pair of eyeglasses. In one embodiment, the headset includes at least one strap and / or temples. In one embodiment, the power source includes at least one battery, at least one supercapacitor, or other similar power source. In another embodiment, the battery includes at least one rechargeable battery. In yet another embodiment, the at least one rechargeable battery includes a lithium-ion battery.

[0351] The headset is configured to receive and display images of virtual scenes, movies, and / or environments. The headset is also capable of receiving audio data and transmitting the audio data to the wearer via speakers, headphones, and other similar audio playback devices. In one embodiment, the headphones are noise-canceling headphones. The noise-canceling headphones are configured to block out external noise and allow the wearer to be fully immersed in the AR / VR environment.

[0352] Examples of headsets and / or AR / VR systems include U.S. Patent Nos. 8,217,856, 8,743,145, 9,094,677, 9,223,136, 9,635,450, 9,671,614, 9,733,480, 9,734,402, 9,766,462, 9,846,483, and 9,85 No. 8,703, No. 9,897,812, No. 9,989,998, No. 10,025,060, No. 10,037,084, No. 10,055,645, No. 10,055,8 No. 87, No. 10,061,352, No. 10,061,391, No. 10,102,674, No. 10,124,251, No. 10,133,305, No. 10,185,39 No. 0, No. 10,209,769, No. 10,244,226, No. 10,254,547, No. 10,261,579, No. 10,318,007, No. 10,419,731 No. 10,429,647, No. 10,540,003, No. 10,656,423, No. 10,656,822, No. 10,769,438, No. 10,825,255 , 10,838,206, 10,890,941, 10,911,734, 10,922,886, 10,928,613, 10,951,880, 11,106,276, 11,145,096 and 11,217,021, all of which are incorporated herein by reference in their entirety.

[0353] In one embodiment, the at least one strap is configured to surround the wearer's head and be attached to the eyewear member via at least one attachment mechanism. The at least one attachment mechanism may include hook-and-loop fasteners, latches, buttons, buckles, snaps, ties, clips, and other similar attachment mechanisms. The at least one strap is adjustable to fit the wearer's head. This advantageously allows the headset to be used by wearers with different head sizes. For example, but not by way of limitation, the at least one strap includes a tightening mechanism. In one embodiment, the tightening mechanism is configured to rotate in one direction to increase tension on the head strap and rotate in the opposite direction to loosen tension on the head strap. In yet another embodiment, the at least one strap includes at least two straps. In one embodiment, the at least two straps do not overlap and are positioned parallel around the wearer's head. Alternatively, the at least two straps are configured to intersect at a center behind the wearer's head, providing a more secure fit.

[0354] Advantageously, the headset is configured to apply minimal pressure to the wearer's face. In one embodiment, the headset includes a nose piece. In one embodiment, the wearer's nose can fit within the nose piece. In one embodiment, the nose piece is adjustable. In one embodiment, the nose piece is configured to be movable left and right and / or up and down. In one embodiment, the nose piece is expandable. Alternatively, the headset is designed to rest on the bridge of the wearer's nose.

[0355] In one embodiment, the at least one image capture device is a motion sensor camera. In one embodiment, the motion sensor camera is configured to capture images of the wearer's body movements. Additionally or alternatively, the at least one image capture device includes a LIDAR camera. The at least one image capture device is further capable of determining a position of the wearer and providing at least one recommendation for correcting the wearer's position based on the indication.

[0356] In one embodiment, the display device includes active matrix organic light emitting diode (AMOLED) technology. In one embodiment, the display device includes a diamond-shaped PenTile subpixel matrix. In one embodiment, the display device has a display panel size ranging from 5 inches to 9 inches (e.g., 7 inches). In one embodiment, the display device has a screen resolution of 2160 x 1200 and a resolution of 1080 x 1200 per eye. In one embodiment, the total number of pixels per eye is 1,296,000. In one embodiment, the display device has a refresh rate of 90 Hz.

[0357] In one embodiment, the system includes a 6-degrees-of-freedom constellation camera. In one embodiment, the system includes an optical 360-degree infrared (IR) LED tracking system. In one embodiment, the system has a 110-degree field of view. In another embodiment, the system includes a near-infrared CMOS sensor. See, e.g., Shafer DM, Carbonara CP, Korpi MF. Factors Affecting Enjoyment of Virtual Reality Games: A Comparison Involving Consumer-Grade Virtual Reality Technology. Games Health J. 2019 Feb;8(1):15-23. doi: 10.1089 / g4h.2017.0190. Epub 2018 Sep 8. PMID: 30199273, which is incorporated herein by reference in its entirety.

[0358] The control electronics preferably includes at least one processor. By way of example and not limitation, the processor may include a general-purpose microprocessor (e.g., a central processing unit (CPU)), a graphics processing unit (GPU), a microcontroller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gate or transistor logic, discrete hardware components, or any other suitable entity or combination thereof capable of performing calculations, executing instructions, and / or otherwise manipulating information. In one embodiment, one or more of the at least one processor is capable of executing predetermined programs stored in at least one memory of the control electronics.

[0359] The control electronics preferably includes at least one antenna, enabling the control electronics to receive and process input data (e.g., AR / VR settings) from at least one remote device (e.g., smartphone, tablet, laptop, desktop computer, gaming system). In a preferred embodiment, the at least one remote device is in wireless network communication with the control electronics. The wireless communication may be, by way of example and without limitation, radio frequency, BLUETOOTH®, ZIGBEE®, WI-FI®, wireless LAN, near field communication (NFC), infrared optical link, or other similar commercial standard. Alternatively, the at least one remote device is in wired communication with the control electronics via USB or equivalent means.

[0360] In one embodiment, the at least one processor is a microcontroller that includes a transceiver, a Bluetooth module, a Wi-Fi module, a microprocessor, an ultra-low power co-processor, read-only memory (ROM), random access memory (RAM) (e.g., static random access memory (SRAM)), flash memory, a power management unit, and / or a digital-to-analog converter.

[0361] In yet another embodiment, the AR / VR system can receive voice input data from the wearer. The AR / VR system includes a microphone capable of receiving and recording the wearer's voice. The headset can also change the display based on the wearer's speech. For example, but not by way of limitation, the AR / VR system can be configured to receive a speech from the wearer such as "start virtual reality game" and launch the virtual reality game.

[0362] The headset is capable of communicating, preferably wirelessly, with at least one remote device (e.g., a mobile phone (e.g., a smartphone), a tablet, a gaming system, at least one other headset, and / or a computer (e.g., a laptop)). The mobile phone may be any mobile phone that is (1) capable of running mobile applications and (2) capable of communicating with the headset. Examples of such mobile phones include an ANDROID phone, an APPLE® IPHONE®, or a SAMSUNG® GALAXY® phone. Similarly, the tablet may be any tablet that is (1) capable of running mobile applications and (2) capable of communicating with the headset. Examples of such tablets include a 3G or 4G version of an APPLE® IPAD®, or a 5G version of a Samsung Galaxy Tab S6.

[0363] Furthermore, in the AR / VR system, the remote device is in communication with a cellular network and / or a network, which may be any network that provides a wired or wireless connection to the Internet, such as a local area network (LAN) or a wide area network (WAN).

[0364] In one embodiment, an AR / VR application (e.g., an AR / VR mobile application) is installed and running on a remote device. The AR / VR system application is implemented depending on the type of remote device (i.e., operating system) on which it runs. The AR / VR system application is designed to receive wearer information from a headset. In one embodiment, the AR / VR application can provide graphical, auditory, and / or haptic feedback to the wearer. In one embodiment, the AR / VR system is configured to create a personalized profile based on the wearer's past AR / VR environments and reactions.

[0365] In one embodiment, the AR / VR system is further capable of displaying ultra-saturated colors as described in U.S. Patent Application No. 17 / 748,655, filed May 19, 2022, the entire contents of which are incorporated herein by reference in their entirety.

[0366] The AR / VR system may utilize multiple learning techniques, including, but not limited to, machine learning (ML), artificial intelligence (AI), deep learning (DL), neural networks (NN), artificial neural networks (ANN), support vector machines (SVM), Markov decision processes (MDP), and / or natural language processing (NLP). The AR / VR system may use any of the foregoing learning techniques, alone or in combination.

[0367] Additionally, the AR / VR system may utilize predictive analytics techniques, including, but not limited to, machine learning (ML), artificial intelligence (AI), neural networks (NNs) (e.g., long short-term memory (LSTM) neural networks), deep learning, historical data, and / or data mining, to predict and / or model the future. Preferably, the AR / VR system performs recommendations and / or actions based on historical data, external data sources, ML, AI, NNs, and / or other learning techniques. The AR / VR system may utilize predictive modeling and / or optimization algorithms, including, but not limited to, heuristic algorithms, particle swarm optimization, genetic algorithms, technical analysis descriptors, combinatorial algorithms, quantum optimization algorithms, iterative methods, deep learning techniques, and / or feature selection techniques.

[0368] Medical Imaging and Diagnostics

[0369] In one embodiment, the present invention is incorporated into medical imaging and / or diagnostics. Medical imaging and / or diagnostics are beginning to move away from grayscale and are increasingly using color in these fields. For example, medical imaging and / or diagnostics include, but are not limited to, telemedicine, digital microscopy, whole slide imaging (WSI), histopathology, endoscopy, laparoscopy, retinal imaging, ophthalmology, dermatology, digital dermatoscopy, echocardiography, radiology (e.g., magnetic resonance imaging (MRI) (e.g., 3D MRI), computed tomography (CT) scan, positron emission tomography (PET) scan, color flow imaging, ultrasound (e.g., 2D flow imaging)), surgery (e.g., robotic surgery), veterinary applications, wound management, cellular spectroscopy, digital pathology, short-wave infrared fluorescence triple-band imaging (SWIFTI), hyperspectral imaging (HSI), and / or dentistry (e.g., cosmetic dentistry, implants, dentures). In one embodiment, the robotic surgery includes at least one robotic surgical component, including, but not limited to, a surgeon console, a patient cart, and / or a vision cart. In one embodiment, the at least one robotic surgical component is a DA VINCI SYSTEMS robotic surgical component. See, e.g., (1) Mingels C, Sachpekidis C, Bohn KP, Hunermund JN, Schepers R, Fech V, Prenosil G, Rominger A, Afshar-Oromieh A, Alberts I. The influence of color scale in lesion detection and patient-based sensitivity in [68Ga]Ga-PSMA-PET / CT. Nucl Med Commun. 2021 May 1;42(5):495-502. doi: 10.1097 / MNM.0000000000001364.PMID: 33481506, (2) Papadacci C, Finel V, Villemain O, Goudot G, Provost J, Messas E, Tanter M, Pernot M. 4D simultaneous tissue and blood flow Doppler imaging: revisiting cardiac Doppler index with single heart beat 4D ultrafast echocardiography. Phys Med Biol. 2019 Apr 10;64(8):085013. doi: 10.1088 / 1361-6560 / ab1107. PMID: 30889552, (3) McKeown, L. A. (May 13, 2021). Color-coded radiation monitoring: Coming to a cath lab near you. TCTMD.com. Retrieved October 27, 2022, from https: / / www.tctmd.com / news / color-coded-radiation-monitoring-coming-cath-lab-near-you, (4) Wild, T., Prinz, M., Fortner, N. et al. Digital measurement and analysis of wounds based on colour segmentation. Eur Surg 40, 5-10 (2008). https: / / doi.org / 10.1007 / s10353-008-0378-0, (5) Sisson, Christye & Farnand, Susan & Fairchild, Mark & Fischer, Bill. (2014). Analysis of Color Consistency in Retinal Fundus Photography: Application of Color Management and Development of an Eye Model Standard. Analytical Cellular Pathology. 2014. 1-2. 10.1155 / 2014 / 398462, (6) Spatial Team (August 19, 2020). Applications of 3D printing in the medical field. Retrieved October 27, 2022, https: / / blog.spatial.com / the-future-of-3d-printing-in-the-medical-field, (7) Petrie, S. (November 17, 2021). Hyperspectral imaging captures spatial and spectral data of the human landscape. Laser Focus World. Retrieved October 27, 2022, https: / / www.laserfocusworld.com / detectors-imaging / article / 14212197 / hyperspectral-imaging-captures-spatial-and-spectral-data-of-the-human-landscape. All of these are incorporated herein by reference in their entirety.

[0370] In one embodiment, the present invention provides true color. Alternatively, the present invention additionally provides false color (e.g., pseudocolor, density slicing, choropleth). In one embodiment, the system is capable of combining greater bit depth (e.g., greater than 8 bits) and additional primaries to provide pseudocolor from a single band of data or a single filter source. Advantageously, the use of greater bit depth and additional primaries enables the system to provide pseudocolor in RGB false color applications. See, e.g., Zabala-Travers S, Choi M, Cheng WC, Badano A. Effect of color visualization and display hardware on the visual assessment of pseudocolor medical images. Med Phys. 2015;42(6):2942-2954. doi:10.1118 / 1.4921125 and Plaxco, J. The Difference Between True Color, False Color and Pseudo Color. Mars Art Gallery. Retrieved October 27, 2022, http: / / www.marsartgallery.com / color.html, all of which are incorporated herein by reference in their entireties.

[0371] In one embodiment, the system incorporates at least one imaging device (e.g., camera), at least one spectrometer, at least one scanner, at least one medical instrument, at least one illumination source, at least one sensor, at least one lens, at least one processor, at least one memory, at least one computing device, at least one database, at least one learning algorithm (e.g., deep neural network) for image processing and comparison, and / or at least one display device.

[0372] In one embodiment, the at least one illumination source is white light, which is effective for detecting lesions. In another embodiment, the at least one illumination source is a variable frequency illumination source, including, but not limited to, visible light, infrared light, and / or ultraviolet light. Advantageously, visible light, infrared light, and / or ultraviolet light are effective for detecting bacterial infections, lipid plaques, precancerous lesions, and subtle inflammatory conditions. In one embodiment, the system preferably uses a narrowband illumination source (e.g., 300-3400 Hz) in endoscopic diagnostic examinations to provide clearer and / or better contrast images. In another embodiment, the system preferably uses an infrared illumination source to provide percutaneous illumination of tissue. See, e.g., Hermanowski, J. Light Source Helps Endoscopes Get Smaller and Smaller. Photonics. Retrieved October 27, 2022, https: / / www.photonics.com / Articles / Light S Source H elps E endoscopic G et S Maller a nd / a51902, which is incorporated herein by reference in its entirety.

[0373] In one embodiment, the system further includes at least one contrast dye and / or at least one contrast agent. In one embodiment, the at least one contrast dye and / or at least one contrast agent is capable of enhancing the contrast of internal structures or fluids in medical images. In one embodiment, the at least one contrast dye and / or at least one contrast agent is radioactive. In one embodiment, the at least one contrast dye and / or at least one contrast agent includes, but is not limited to, iodine, barium (e.g., barium sulfate), gadolinium, and / or microbubbles (e.g., saline). In one embodiment, the at least one contrast dye and / or at least one contrast agent is a fluorescent dye (e.g., indocyanine green, methylene blue). In another embodiment, the at least one contrast dye and / or at least one contrast agent includes at least one magnetic microstructure. See, e.g., U.S. Patent Nos. 10,188,755, 9,084,820, and 10,215,825. all of which are incorporated herein by reference in their entirety.

[0374] In one embodiment, the system includes at least one imaging device for acquiring image data. The at least one imaging device preferably includes at least one lens and at least one image sensor (e.g., a camera, video camera, camcorder, slow-motion camera, and / or high-speed camera). The at least one lens directs light to the at least one image sensor. The at least one lens may include, but is not limited to, at least one convex lens and / or at least one concave lens. In one embodiment, the at least one image sensor is a wide color gamut image sensor, such as a wide color gamut camera. In one embodiment, the at least one image sensor is a single-pixel image sensor. In one embodiment, the at least one image sensor does not include a detector array. In one embodiment, the at least one image sensor is a plurality of image sensors.

[0375] In one embodiment, one or more of the at least one imaging device are integrated into at least one medical device (e.g., an endoscope). In one embodiment, one or more of the at least one imaging device are replaceable, such that the at least one medical device is compatible with multiple imaging devices. Advantageously, this modular design allows the at least one imaging device to be upgraded or replaced depending on image acquisition needs or technological advances. In one embodiment, the at least one imaging device includes an array sensor, such that the at least one imaging device can provide multispectral capture or operate as a light field sensor.

[0376] In one embodiment, the at least one imaging device includes multiple lenses for multiple image sensors. In one embodiment, the multiple lenses generate different focal lengths for each of the multiple image sensors. In one embodiment, the at least one imaging device is capable of changing its focal length, such as by zooming. Alternatively, the system can interpolate signals from multiple image sensors with different focal lengths to generate hybrid sensor data. The system can combine sensor data from each of the multiple image sensors into a single image data set. In one embodiment, the system includes a stabilizer, such as a gyroscope system. The at least one imaging device is preferably disposed on a stabilizer, which moves the at least one imaging device to counteract motion that causes image blur. In one embodiment, the multiple lenses are constructed from metamaterials, which enable a negative refractive index. This advantageously optimizes focal length while minimizing physical length.

[0377] In one embodiment, the at least one image capture device includes at least one optical filter. In one embodiment, the at least one optical filter is disposed over the at least one image sensor. In one embodiment, the at least one filter is an absorptive filter. Alternatively, the at least one filter is an interference filter or a dichroic filter. In one embodiment, the at least one filter has at least one cutoff wavelength and transmits or blocks light based on the at least one cutoff wavelength (e.g., a long-pass filter, a short-pass filter, a band-pass filter, a multi-band-pass filter, or a notch filter). In another embodiment, the at least one filter changes the intensity of all wavelengths equally (e.g., a neutral density filter). In one embodiment, the at least one filter is a filter array, such as a Bayer filter. Bayer filter arrays compatible with the present invention include, but are not limited to, RGB, CYGM, RGBE (red, green, blue, emerald), and / or CMY. The Bayer filter can be modified. As a non-limiting example, the Bayer filter includes a magenta filter. Alternatively, the size of the elements of the Bayer filter are differentially adjusted to increase the sensitivity of the at least one image sensor. In yet another embodiment, one or more of the at least one filter are rotatable, thereby allowing the system to advantageously use different filters for different exposures. In one embodiment, the at least one filter includes multiple filter layers. In one embodiment, the at least one filter includes multiple filters. In one embodiment, the multiple filters are arranged radially. In one embodiment, the multiple filters form a rotating filter wheel.

[0378] In one embodiment, the acquisition color gamut is the same as the display color gamut. In one embodiment, both the acquisition color gamut and the display color gamut are extended color gamuts and / or include at least four primary colors (e.g., 6P-B, 6P-C). Alternatively, the display color gamut (e.g., RGBCMY) has a larger volume than the acquisition color gamut (e.g., RGB). In yet another embodiment, the display color gamut (e.g., RGB) has a smaller volume than the acquisition color gamut (e.g., RGBCMY, RGBCEY (i.e., two greens—green (G) and emerald (E))). Preferably, the system converts image data from the acquisition color gamut to the display color gamut.

[0379] In one embodiment, the system can capture raw image data as RAW image files. RAW image files are considered unprocessed and therefore cannot be edited or printed. RAW image files contain metadata and headers in addition to image data. Such metadata can include, but is not limited to, patient health information (PHI) (e.g., patient name, medical record number, date of birth), anatomical origin and / or tissue type of the image, image acquisition parameters (e.g., image dimensions, voxel size, repetition time, voxel data type), image sensor parameters, imager parameters, time code, imager ISO, aperture measurements (e.g., f / number, aperture size), exposure data, color space parameters, lens data (e.g., focal length and lens type), filter data, file format, compression data, frame rate data, flash data, light source, position data, and / or frame data.

[0380] In one embodiment, the system is compliant with the Digital Imaging and Communications in Medicine (DICOM) standard for metadata. DICOM metadata differs based on different image modalities (e.g., CT scans and ultrasounds) because each modality has its own unique metadata. DICOM metadata defines information object definitions (IODs), which are object-oriented metadata organized into modules. DICOM also standardizes the structure of data values ​​(e.g., date, time, patient name) and provides a predefined list of attributes (e.g., gender, body part). Additional information regarding DICOM can be found in ISO 12052:2017 Health informatics - Digital imaging and communication in medicine (DICOM) including workflow and data management (August 2017), which is incorporated herein by reference in its entirety. See also Caffery, Liam J et al, “Transforming Dermatologic Imaging for the Digital Era: Metadata and Standards,” Journal of Digital Imaging vol. 31,4 (2018): 568-577, doi:10.1007 / s10278-017-0045-8, which is incorporated herein by reference in its entirety.

[0381] RAW image formats include, but are not limited to, Digital Negative Raw (DNG), ISO 12234-2 (TIFF / EP), Nikon NEF, Canon Raw v2 (CR2), CR3, Sony S-Log, ARRI Log, Panasonic V-Log, and / or Redcode Raw (R3D) files. In one embodiment, the system can save the RAW image file before processing. The system can then convert the RAW image data into rendered image data, which can be displayed and / or edited. Rendering can include, but is not limited to, decoding, demosaicing (e.g., removing the effects of a Bayer filter), pixel removal (e.g., removing bad pixels), interpolation (e.g., replacing removed pixels), white balance, noise reduction, color conversion, tone reproduction, optical correction, contrast manipulation, resizing, splitting, and / or compression. Alternatively, the system does not compress the RAW image data. In one embodiment, the system can render the image data as a pipeline process, where each step is performed sequentially. The order of the steps can be changed. Alternatively, the system may render the image data in parallel steps. In yet another embodiment, the system may render the image data by solving a single optimization problem. The system may store image prior data and / or image variation data and use the image prior data and / or image variation data in rendering, processing, and / or displaying the image data.

[0382] In one embodiment, rendering includes converting the raw image data to a color space (e.g., CIE 1931, ITU-R BT.2020). In a preferred embodiment, the system is capable of rendering the image data in a three-coordinate format, where the first and second coordinates are both colorimetric (chroma) and the third coordinate is a luminance or luma value. As a non-limiting example, the three-coordinate format is Yxy, where x and y are orthogonal colorimetric coordinates and Y is a luminance coordinate. The system can also apply a transformation (e.g., gamma compression) to the luminance coordinate to generate a luma coordinate (e.g., Y'). Relative luminance values ​​are also supported. Alternative three-coordinate formats include, but are not limited to, L*a*b*, ICtCp, YCbCr, YUV, Yu'v', YPbPr, and / or YIQ. Alternatively, the system can render the image data as XYZ data. In one embodiment, the system includes a user interface and accepts user input. In one embodiment, the user input determines how the raw image data is rendered. In one embodiment, the system is capable of applying an optical-to-electrical transfer function (OETF) and an electro-optical transfer function (EOTF) to the image data. Alternatively, the system is capable of applying at least one non-linear function (e.g., an OOTF) to the image data. In one embodiment, the system includes at least one look-up table (LUT). The LUT can be implemented in hardware (e.g., in an FPGA) and / or software.

[0383] In one embodiment, the system can remap colors to include perceptible differences on a display device. For example, if a first color and a second color are perceptually indistinguishable to a human observer, the system can remap the second color to a modified second color, thereby enabling the observer to perceive a difference between the first color and the modified second color. Additionally, the system can retain the original information, thereby enabling multiple learning and / or predictive analytics techniques to be performed using the unmodified data. The system can also retain information about the remapping. In one embodiment, the retained information is stored in metadata.

[0384] In one embodiment, the at least one imager acquires data as Yxy data, XYZ data, RGB data, and / or multi-primary color data (e.g., RGBC, RGBCY, RGBCMY, RGBEYC). Additionally or alternatively, the at least one imager may acquire data including, but not limited to, hyperspectral data, ultraviolet (UV) data, and / or infrared (IR) data. See, e.g., Lu, Guolan, and Baowei Fei. "Medical hyperspectral imaging: a review." Journal of Biomedical Optics vol. 19,1 (2014): 10901. doi:10.1117 / 1.JBO.19.1.010901, which is incorporated herein by reference in its entirety.

[0385] In one embodiment, the present invention is used to process hyperspectral data, UV data, and / or IR data outside the visible spectrum. In one embodiment, the system augments colorimetric visible data with data outside the visible spectrum. For example, the system generates two different three-coordinate format elements for each data point acquired from the at least one imager, one element related to Yxy (e.g., Y'xy) and the other element related to WUI (e.g., W'UI), where W (or W') is a function similar to Y (or Y') but encompasses more UV and IR wavelengths, U is the UV channel, I is the IR channel, and W (or W') is proportional to the intensities of U and I. The system can calculate a projection using WUI values ​​so that the projection plane is wUI. The system can evaluate the relevance of the acquired spectral channels and their rapid changing trends. The system can assign the acquired elements to specific primary colors. Alternatively, the system can assign the acquired elements to a mathematical combination of the acquired elements, such as UV*Red and IR*Cyan. Although the system is described in terms of nonlinear forms including Y'xy and W'UI, the system is not limited to these forms. In one embodiment, a nonlinear function is applied to xy and / or UI. In one embodiment, xy and / or UI are scaled.

[0386] In one embodiment, the present invention further includes at least one spectrometer, including, but not limited to, a filtered camera, a whisk broom scanner, a push broom scanner, an integral field spectrometer, a wedge imaging spectrometer, a Fourier transform imaging spectrometer, a computed tomography imaging spectrometer (CTIS), an image replicating imaging spectrometer (IRIS), a coded aperture snapshot spectroscopic imager (CASSI), an image mapping spectrometer (IMS), and / or a spectrophotometer. For example, a spectrophotometer may be used to obtain color data for shade matching of dental implants.

[0387] In a preferred embodiment, the present invention incorporates luminance (e.g., Y) and two independent colorimetric coordinates (e.g., x and y, u' and v'). This advantageously allows for luminance and the two independent colorimetric coordinates to be processed separately. In one embodiment, a nonlinear function, algorithm, and / or LUT can be applied to luminance only. In another embodiment, a nonlinear function, algorithm, and / or LUT can be applied to luminance and the two independent colorimetric coordinates. In one embodiment, the system can include at least two luminance levels (e.g., standard and increased luminance). Temporarily increasing luminance advantageously improves image visibility while minimizing impact on display device lifetime. Furthermore, the system can utilize Yxy and / or Yu'v' to plot the two independent colorimetric coordinates x and y outside the boundaries of the CIE-1931 color space. This advantageously allows for the conversion of raw ultraviolet-visible-near-infrared (UV-Vis-NIR) spectrophotometric data to rendered image data. See, e.g., U.S. Patent No. 9,685,109; Tom Kimpe and Albert Xthona, "Quantification of detection probability of microcalcifications at increased display luminance levels," Breast Imaging, Springer Berlin Heidelberg, 2012, pp. 490-497; and Ultraviolet Visible Near Infrared Spectrophotometry (UV-Vis-NIR). Covalent Metrology. Retrieved October 27, 2022, https: / / covalentmetrology.com / techniques / ultraviolet-visible-near-infrared-spectrotrophotometry-uv-vis-nir / , all of which are incorporated herein by reference.

[0388] As mentioned above, the system further includes at least one display device. In one embodiment, the at least one display device is a multi-primary display device (e.g., RGBC, RGBCY, RGBCMY). The addition of a cyan primary color provides a wider color gamut, which has the advantage of improving the visibility of green-cyan pseudocolor, a current challenge in medical imaging. See, e.g., Zabala-Travers, Silvina et al., "Effect of color visualization and display hardware on the visual assessment of pseudocolor medical images." Medical Physics vol. 42,6 (2015): 2942-54. doi:10.1118 / 1.4921125, which is incorporated herein by reference. Furthermore, multi-primary display devices can provide additional colors not available in RGB devices. For example, but not limited to, an RGB 12-bit display device can provide (2 12 ) 3 = 68.7 trillion colors, and a 12-bit display device with four primary colors (e.g., RGBC) can display (2 12 ) 4 = 281.5 trillion colors can be displayed.

[0389] In one embodiment, the at least one display device is capable of displaying colors outside the ITU-R BT.2020 color gamut. In one embodiment, the at least one display device is capable of displaying at least 80% of the total area covered by the CIE-1931 color space. In one embodiment, the at least one display device is described in U.S. Patent No. 11,030,934, filed October 1, 2020, and issued June 8, 2021, which is incorporated herein by reference. In one embodiment, the at least one display device is a screen, such as a liquid crystal display (LCD) screen, a light-emitting diode (LED) screen, an LED-backlit screen, an organic light-emitting diode (OLED) screen, an active matrix OLED (AMOLED) screen, a quantum dot (QD) display, a perovskite display, a stereoscopic display (e.g., head-mounted and / or autostereo), a virtual reality (VR) display, and / or an augmented reality display. In another embodiment, the at least one display device includes at least one projector. The at least one display device is capable of displaying image data after it has been acquired, rendered, and / or processed by the system. In another embodiment, the at least one display device includes multiple display devices (e.g., screens, projectors).

[0390] In one embodiment, the at least one display device is capable of modifying display parameters of image data, including, but not limited to, color gamut, frame rate, sampling rate, aspect ratio, data format, metadata, tone curve (e.g., perceptual quantizer (PQ), hybrid log gamma (HLG), gamma), and / or SDP parameters.

[0391] In one embodiment, the system further includes additional health information, including but not limited to vital signs, test results (e.g., blood test results, genetic test results), previous images or scans, and / or health history.

[0392] In one embodiment, the system incorporates multiple learning techniques in imaging and / or diagnosis. In a preferred embodiment, the multiple learning techniques use a training set to train the system during a learning period. The multiple learning techniques include, but are not limited to, machine learning (ML), artificial intelligence (AI), deep learning (DL), neural networks (NN), artificial neural networks (ANN), support vector machines (SVM), Markov decision processes (MDPs), decision trees, linear regression, logistic regression, naive Bayes, k-nearest neighbors, random forests, adaptive boosting (i.e., AdaBoost), and / or natural language processing (NLP). In one embodiment, the multiple learning techniques are capable of extracting at least one intensity, at least one edge, at least one texture, at least one segment, at least one color, at least one luminance, at least one color / tone gradient, at least one histogram / heat diagram, and / or at least one wavelength from the image. In one embodiment, the ANN includes, but is not limited to, a multilayer perceptron (MLP). Alternatively, the ANN comprises a Single Layer Perceptron (SLP). In one embodiment, the learning system comprises a feedforward network trained by a backpropagation algorithm. In one embodiment, the plurality of learning techniques comprises a supervised learning technique.See, e.g., Mahmood F, Bendayan S, Ghazawi FM, Litvinov IV. Editorial: The Emerging Role of Artificial Intelligence in Dermatology. Front Med (Lausanne). 2021 Nov 17;8:751649. doi: 10.3389 / fmed.2021.751649. PMID: 34869445; PMCID: PMC8635630 and Lavars, N. (2021, February 19). AI uses "Ugly duckling" technique to spot melanoma with high accuracy. New Atlas. Retrieved October 27, 2022, https: / / newatlas.com / medical / ai-ugly-duckling-melanoma-skin-cancer / , all of which are incorporated herein by reference.

[0393] Alternatively, the plurality of learning techniques may include unsupervised learning techniques, including, but not limited to, K-means, mean shift, affinity propagation, hierarchical clustering, density-based spatial clustering of noisy applications (DBSCAN), Gaussian mixture modeling, Markov random fields, iterative self-organizing data (ISODATA), and fuzzy C-means systems. In one embodiment, the plurality of learning techniques may include reinforcement learning techniques (e.g., Maja, Teaching-Box). The system may use any of the foregoing learning techniques alone or in combination.

[0394] Additionally, the system may utilize predictive analytics techniques, including, but not limited to, machine learning (ML), artificial intelligence (AI), neural networks (NNs) (e.g., long short-term memory (LSTM) neural networks), deep learning, historical data, and / or data mining. Preferably, the system performs recommendations and / or actions based on historical data, external data sources, ML, AI, NNs, and / or other learning techniques. The system may utilize predictive modeling and / or optimization algorithms, including, but not limited to, heuristic algorithms, particle swarm optimization, genetic algorithms, technical analysis descriptors, combinatorial algorithms, quantum optimization algorithms, iterative methods, deep learning techniques, and / or feature selection techniques. In one embodiment, the predictive modeling and / or optimization algorithm includes extracting at least one intensity, at least one edge, at least one texture, at least one segment, at least one color, at least one brightness, and / or at least one wavelength from the image. See, e.g., Erickson, Bradley J et al., "Machine Learning for Medical Imaging." Radiographics: a review publication of the Radiological Society of North America, Inc. vol. 37,2 (2017): 505-515. doi:10.1148 / rg.2017160130, which is incorporated by reference in its entirety.

[0395] In one embodiment, the system is used for digital dermoscopy and / or imaging, displaying, and / or diagnosing skin conditions. The display device preferably displays tissue images, live tissue manipulation (e.g., real-time or near-real-time), and / or images of various tissue types within or on the body with high color accuracy. The ability to distinguish and detect variations and differences in tissue color is particularly important for diagnostic imaging related to the skin and other organs (e.g., the brain, lungs, etc.). While human skin tone varies slightly due to various factors, health and emotion are key contributors. The human visual system is optimized to detect minute changes in skin reflectance due to blood flow and oxygenation. M (green) and L (red) cones are capable of detecting these variations. There is a long-standing unmet need for an expanded color gamut to provide more accurate skin color. In a preferred embodiment, the display device is a multi-primary (e.g., RGBCMY) display device. In a preferred embodiment, the display device includes a cyan primary. In another preferred embodiment, the display device includes a yellow primary. In one embodiment, the display device has a red primary at wavelengths greater than 615 nm. Skin tones often appear yellowish or reddish after color correction. Furthermore, skin often appears shiny after color correction. Increasing the cyan and / or magenta components has the advantage of improving color accuracy of skin tones and reducing the appearance of shiny skin.

[0396] In one embodiment, the present invention is used to ensure color fidelity in imaging and / or diagnosis. In one embodiment, the system includes at least one chip chart having multiple colors and / or at least one reference for calibrating the system. For example, digital pathology scanners often incorporate at least one sensor, at least one light source, at least one lens, and at least one processor. In one embodiment, the digital pathology scanner is calibrated using at least one reference slide having multiple known colors. In one embodiment, the at least one reference slide mimics at least one stained tissue sample and / or at least one fluorescent marker. An example of a reference slide is disclosed in U.S. Pat. No. 10,241,310, which is incorporated herein by reference. In a preferred embodiment, the digital pathology scanner is capable of scanning at least one reference slide and / or at least one sample as Yxy data and / or XYZ data. Alternatively, the digital pathology scanner is calibrated using at least one color patch. In one embodiment, the spectral transmittance of each of the at least one color patch is measured (e.g., using a spectroradiometer). Scanning as Yxy and / or XYZ data has the advantage of providing a wider color gamut than RGB data. Color fidelity during scanning is critical for individual diagnostics as well as multi-site studies. Additional information regarding colorimetry is contained in ISO CIE 11664-6:2014 and ISO CIE 15076-1:2010, which are incorporated herein by reference.

[0397] In one embodiment, the present invention is used to ensure standardization of images captured for telemedicine, regardless of the light source used in the images. In one embodiment, the system includes at least one telemed chart having multiple colors and / or at least one reference for normalizing and equalizing various light sources and calibrating at least one imaging device. The system is capable of digitally evaluating images of the telemed chart captured by at least one imaging device to characterize the overall light field within the image. The system is capable of digitally standardizing images captured for telemedicine purposes using image data of the captured telemed chart and data from the at least one imaging device. This advantageously eliminates ambiguity regarding which light source was used when evaluating telemedicine images. In one embodiment, the telemed chart includes at least one UV fluorescent dye that emits in the visible spectrum. In another embodiment, the telemed chart is transparent, with different transparent dyes selected to attenuate different wavelengths. Advantageously, a transparent telemed chart has clear segments that do not contain any dye, which can be used as a reference.

[0398] In one embodiment, at least one display device has a built-in calibrator for calibrating the individual primary color channels of the at least one display device over time. The system can calibrate the channels of the display device as the relative intensities of the individual primary color channels vary over time. For example, the system can adjust the relative intensity ratios of the individual primary color channels to a desired standard point. This advantageously improves the stability of the display device's primary colors over time.

[0399] FIG. 81 is a schematic diagram of a computer system 800 illustrating one embodiment of the present invention, including a network 810, multiple computing devices 820, 830, 840, a server 850, and a database 870.

[0400] Server 850 is configured, set up, and connected to enable communication with multiple computing devices 820, 830, 840 over network 810. Server 850 includes a processing unit 851 having an operating system 852. Operating system 852 enables server 850 to communicate with remote, distributed user devices over network 810. Database 870 can store operating system 872, memory 874, and programs 876.

[0401] In one embodiment of the present invention, system 800 includes a network 810 for distributed communications via a wireless communications antenna 812 and processing by at least one mobile communications computing device 830. Alternatively, wireless and wired communications and connections between devices and components described herein may include Wi-Fi, Worldwide Interoperability for Microwave Access (WIMAX), radio frequency (RF) communications (including RF identification (RFID)), near field communications (NFC), Bluetooth (including Bluetooth Low Energy (BLE)), ZIGBEE, infrared (IR) communications, cellular communications, satellite communications, Universal Serial Bus (USB), Ethernet communications, fiber optic cable, coaxial cable, twisted pair cable, and / or any other type of wireless or wired communications. In another embodiment of the present invention, system 800 is a virtualized computing system capable of executing any or all of the software and / or application components described herein on computing devices 820, 830, and 840. In certain aspects, computer system 800 may be implemented using hardware or a combination of software and hardware, such as a dedicated computing device, integrated into other entities, or distributed across multiple entities or computing devices.

[0402] By way of example, computing devices 820, 830, and 840 are intended to represent various forms of electronic devices having at least a processor and memory, such as servers, blade servers, mainframes, mobile phones, personal digital assistants (PDAs), smartphones, desktop computers, notebook computers, tablet computers, workstations, laptops, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0403] In one embodiment, the computing device 820 includes components such as a processor 860, a system memory 862 having a random access memory (RAM) 864 and a read-only memory (ROM) 866, and a system bus 868 connecting the memory 862 to the processor 860. In another embodiment, the computing device 830 may further include components such as a storage device 890 for storing an operating system 892 and one or more application programs 894, a network interface unit 896, and / or an input / output controller 898. The components may be interconnected via at least one bus 868. The input / output controller 898 may receive and process input from or provide output to other devices 899, such as an alphanumeric input device, a mouse, an electronic stylus, a display unit, a touchscreen, a game controller, a joystick, a touchpad, a signal generating device (e.g., a speaker), an augmented reality / virtual reality (AR / VR) device (e.g., an AR / VR headset), or a printer. By way of example, the processor 860 may be a general-purpose microprocessor (e.g., a central processing unit (CPU)), a graphics processing unit (GPU), a microcontroller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gate or transistor logic, discrete hardware components, or any other suitable entity or combination thereof capable of performing calculations, executing instructions and / or other manipulations of information.

[0404] In another embodiment, multiple processors 860 and / or multiple buses 868 may be used, as appropriate, along with multiple types of memories 862 (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, or a combination of one or more microprocessors and a DSP core), as shown at 840 in FIG.

[0405] Additionally, multiple computing devices may be connected together, each providing a portion of the required processing (e.g., a server bank, a cluster of blade servers, or a multiprocessor system), or some steps or methods may be performed by circuitry specialized for a particular function.

[0406] According to various embodiments, computer system 800 can operate in a networked environment using logical connections to local and / or remote computing devices 820, 830, 840 via network 810. Computing device 830 can connect to network 810 via a network interface unit 896 connected to bus 868. The computing devices can communicate over a wired network, a direct wired connection, or wirelessly via a communications medium, for example, antenna 897 in communication with network antenna 812 and network interface unit 896, which can include digital signal processing circuitry as needed. Network interface unit 896 can provide for communications under various modes or protocols.

[0407] In one or more exemplary aspects, the instructions may be implemented in hardware, software, firmware, or any combination thereof. The computer-readable medium may store, in a volatile or non-volatile manner, one or more sets of instructions for an operating system, data structures, program modules, applications, or other data embodying any or more of the methodologies or functions described herein. The computer-readable medium may include the memory 862, the processor 860, and / or the storage medium 890, and may be a single medium or multiple media (e.g., a centralized or distributed computer system) capable of storing one or more sets of instructions 900. Non-transitory computer-readable media includes all computer-readable media, with the sole exception being the transitory propagating signal itself. The instructions 900 may be transmitted or received over the network 810 via the network interface unit 896 as a communications medium. Communications media includes a modulated data signal, such as a carrier wave, or other transmission mechanism, and includes any distribution medium. A "modulated data signal" means a signal in which one or more characteristics of the signal are changed or set in such a manner as to encode information in the signal.

[0408] Storage devices 890 and memory 862 include, but are not limited to, cache, RAM, ROM, EPROM, EEPROM, flash memory, other solid-state memory technology, disk (e.g., digital versatile disk (DVD), HD-DVD, BLU-RAY, compact disk (CD), or CD-ROM) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage, floppy disk, or other magnetic storage device, or other medium that stores computer-readable instructions and is accessible by computer system 800.

[0409] In one embodiment, computer system 800 is in a cloud-based network. In one embodiment, server 850 is the designated physical server for distributed computing devices 820, 830, and 840. In one embodiment, server 850 is a cloud-based server platform. In one embodiment, the cloud-based server platform hosts serverless functions for distributed computing devices 820, 830, and 840.

[0410] In another embodiment, computer system 800 is in an edge computing network. Server 850 is an edge server and database 870 is an edge database. Edge server 850 and edge database 870 are part of an edge computing platform. In one embodiment, edge server 850 and edge database 870 are assigned to distributed computing devices 820, 830, 840. In one embodiment, edge server 850 and edge database 870 are not assigned to computing devices 820, 830, 840. Distributed computing devices 820, 830, 840 are connected to edge servers in the edge computing network based on proximity, availability, latency, bandwidth, and / or other factors.

[0411] It is also contemplated that computer system 800 may not include all of the components shown in FIG. 81 , may include other components not explicitly shown in FIG. 81 , or may utilize an entirely different architecture than that shown in FIG. 81 . The various logical blocks, modules, elements, circuits, and algorithms associated with the embodiments described herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functions in varying ways (e.g., arranging them in a different order or in a different way) for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the invention.

[0412] It will be apparent to those skilled in the art that the above examples are provided to clarify aspects of the present invention and are not intended to limit the scope of the present invention. The present invention is highly tunable, customizable, and adaptable in nature. The above examples are only a few of the many configurations that the components may assume. All modifications and improvements are omitted herein for brevity and readability, but would fall squarely within the scope of the present invention.

Claims

1. 1. A system for displaying a primary color system, comprising: a set of image data including a set of primary color signals, the set of primary color signals corresponding to a set of values ​​in a Commission Internationale de l'Eclairage (CIE) Yxy color space, the set of values ​​in the CIE Yxy color space including luminance (Y) and two colorimetric coordinates (x, y), the set of image data including medical image data, the medical image data including false color data; an image data converter, the image data converter including a digital interface, the digital interface being capable of encoding and decoding a set of values ​​in the CIE Yxy color space; at least one display device; Equipped with The processed Yxy data is transferred between the encoder and decoder; the image data converter is capable of converting the set of image data for display on the at least one display device. system.

2. 2. The system of claim 1, wherein the image data converter is capable of converting the set of values ​​in the CIE Yxy color space to multiple color gamuts.

3. 10. The system of claim 1, wherein the image data converter comprises a lookup table.

4. 10. The system of claim 1, wherein the set of image data includes colors outside the International Telecommunication Union Recommendation (ITU-R) BT.2020 color gamut.

5. 2. The system of claim 1, wherein the image data converter is capable of fully sampling the processed Yxy data in a first channel and sub-sampling the processed Yxy data in a second channel and a third channel.

6. 10. The system of claim 1, wherein the processed Yxy data in the first, second, and third channels is fully sampled.

7. 2. The system of claim 1, wherein the encoding includes scaling the two colorimetric coordinates (x, y) to generate first-scale colorimetric coordinates and second-scale colorimetric coordinates, and / or the decoding includes rescaling data associated with the first-scale colorimetric coordinates and data associated with the second-scale colorimetric coordinates.

8. 2. The system of claim 1, wherein the encoding comprises converting the set of primary color signals to XYZ data and then converting the XYZ data to generate a set of values ​​in the CIE Yxy color space, and / or the decoding comprises converting the processed Yxy data to XYZ data and then converting the XYZ data to a format displayable on the at least one display device.

9. 10. The system of claim 1, further comprising at least one non-linear function, wherein the at least one non-linear function comprises a data range reduction function having a value in the range of about 0.25 to about 0.9 and / or an inverse data range reduction function having a value in the range of about 1.1 to about 4.

10. The system of claim 1 , further comprising at least one imaging device, one or more of the at least one imaging device being capable of providing the medical image data.

11. The system of claim 1 , wherein the system supports metadata of a medical image communication standard.

12. 10. The system of claim 1, further comprising at least one processor connected to at least one memory and at least one learning algorithm for image processing and comparison.

13. The system of claim 1 , wherein the image data set further comprises hyperspectral data, ultraviolet (UV) data, and / or infrared (IR) data.

14. 14. The system of claim 13, wherein the image data converter is capable of generating two different three-coordinate format elements, a first three-coordinate format element being Yxy and a second three-coordinate format element including a first coordinate associated with the UV data, a second coordinate associated with the IR data, and a third coordinate proportional to the intensity of the UV data and the IR data.

15. 10. The system of claim 1, further comprising at least one chip chart or at least one Telemed chart having a plurality of colors and at least one reference for calibration of the system.

16. 1. A system for displaying a primary color system, comprising: a set of image data including a set of primary color signals, the set of primary color signals corresponding to a set of values ​​in the Commission Internationale de l'Eclairage (CIE) Yxy color space, the set of values ​​in the CIE Yxy color space including luminance (Y) and two colorimetric coordinates (x, y), the set of image data including medical image data; at least one imaging device, one or more of which are capable of providing the medical image data; an image data converter, the image data converter including a digital interface, the digital interface being capable of encoding and decoding a set of values ​​in the CIE Yxy color space; Equipped with The processed Yxy data is transferred between the encoder and decoder; one or more of the at least one imaging device is incorporated into at least one medical device; the medical image data includes false color data; the image data converter is capable of converting the set of image data for display on at least one display device; system.

17. 1. A system for displaying a primary color system, comprising: a set of image data including a set of primary color signals, the set of primary color signals corresponding to a set of values ​​in the Commission Internationale de l'Eclairage (CIE) Yxy color space, the set of values ​​in the CIE Yxy color space including luminance (Y) and two colorimetric coordinates (x, y), the set of image data including medical image data; at least one imaging device, one or more of which are capable of providing the medical image data; an image data converter, the image data converter including a digital interface, the digital interface being capable of encoding and decoding a set of values ​​in the CIE Yxy color space; at least one display device; Equipped with the image data converter and the at least one display device are in communication; The processed Yxy data is transferred between the encoder and decoder; the image data converter is capable of converting the set of image data for display on the at least one display device. system.

18. 20. The system of claim 17, wherein the at least one display device includes at least four primary colors.

19. 20. The system of claim 17, wherein the at least one display device is capable of displaying colors outside the International Telecommunications Union Recommendation (ITU-R) BT.2020 color gamut.

20. 20. The system of claim 17, wherein the at least one display device comprises a headset configured for a virtual reality, augmented reality, and / or mixed reality environment.

Images