Display controller and image display method

Abstract

To suitably composite and easily display a plurality of images regardless of conditions.SOLUTION: A display controller reads out data of a first image 52a and a second image 52b from a frame buffer, and with a conversion formula depending on the characteristics for the brightness of the images, converts the data into data in a blend space A having common characteristics (S10a). The display controller performs alpha blending of the converted data in the blend space A (S12), further converts the data into a space having characteristics suitable for a display (S14), and outputs the data to the display.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a display controller and an image display method for displaying an image on a display device. [Background technology]

[0002] Various technologies have been developed to improve the image quality of video displays such as television broadcasts and streaming videos. In recent years, in addition to technologies for improving resolution and color gamut, technologies for processing HDR (High Dynamic Range) signals, which expand the range of brightness, are becoming increasingly popular. Compared to the conventional SDR (Standard Dynamic Range), HDR allows for a brightness range that is approximately 100 times larger, making it possible to more realistically represent objects that are perceived as dazzling in the real world, such as reflected sunlight, in images. Not only in television broadcasts and streaming videos, but also in the world of computer graphics such as game images, HDR representations can add a sense of realism to virtual worlds (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-58848 Summary of the Invention [Problem to be solved by the invention]

[0004] The processing required for image display is becoming more complex as a result of the combination of various conditions, such as the brightness range defined for the original image data, the brightness range supported by the connected display, etc. In particular, when multiple images are composited and displayed, changes in these conditions can change the composite result, and the displayed image may differ from the original intention.

[0005] The present invention has been made in view of the above problems, and its purpose is to provide a technique that can appropriately control the synthesis results regardless of the conditions when synthesizing and displaying a plurality of images. [Means for solving the problem]

[0006] One aspect of the present invention relates to a display controller, which includes a plurality of blending conversion circuits that convert data of a plurality of images read from a memory into data having a common luminance characteristic, a blending circuit that performs alpha blending on the data of the plurality of images having the common characteristic output from the blending conversion circuits, and an output conversion circuit that outputs the blended image data as a signal for display.

[0007] Another aspect of the present invention relates to an image display method, the image display method including the steps of: converting, by a display controller, data of a plurality of images read from a memory into data having a common luminance characteristic; alpha-blending the data of the plurality of images having the common characteristic; and outputting the blended image data to a display as a display signal.

[0008] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, or recording medium on which a computer program is recorded, are also valid aspects of the present invention. [Effects of the Invention]

[0009] According to the present invention, when a plurality of images are synthesized and displayed, the synthesis result can be appropriately controlled regardless of the conditions. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment of the present invention. [Figure 2]1A and 1B are diagrams schematically illustrating an example of an image generated by the information processing device in the present embodiment. [Figure 3] 1 is a diagram showing an internal circuit configuration of an information processing device according to an embodiment of the present invention; [Figure 4] FIG. 1 is a diagram for explaining a general processing flow relating to a video signal. [Figure 5] FIG. 10 is a diagram illustrating an example of processing in which a display controller reads images from two frame buffers, performs alpha blending, and outputs the images to a display. [Figure 6] 10A and 10B are diagrams illustrating examples of conversion functions used when quantizing pixel values ​​of image data in the present embodiment. [Figure 7] 10A and 10B are diagrams comparing a composite image in linear space and a composite image in gamma space, which are generated in this embodiment. [Figure 8] FIG. 10 is a diagram showing a procedure of processing in which a display controller combines two images and outputs the combined images in this embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of an image synthesized by the procedure of FIG. 8. [Figure 10] 10 is a diagram illustrating a procedure of processing in which a display controller in the present embodiment combines and outputs three images. [Figure 11] 11A and 11B are diagrams illustrating images synthesized by the procedure of FIG. 10. [Figure 12] FIG. 10 is a diagram showing the characteristics of a conversion formula used for an HDR image in this embodiment and the synthesis results of images converted using this conversion formula. [Figure 13] FIG. 1 is a diagram comparing gamma space characteristics across various luminance ranges. [Figure 14] FIG. 10 is a diagram illustrating a processing procedure in which a display controller displays a composite image on an HDR-compatible display in the present embodiment. [Figure 15] 10 is a diagram illustrating a processing procedure in which a display controller displays a composite image on an SDR-compatible display in this embodiment. [Figure 16]FIG. 2 is a diagram schematically illustrating a circuit configuration of a display controller according to the present embodiment. [Figure 17] FIG. 10 is a diagram comparing images obtained by combining images in linear and gamma spaces as blend spaces when the images are displayed on SDR and HDR displays in this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1 shows an example of the configuration of an information processing system according to this embodiment. The information processing system includes an information processing device 10, an input device 14, and a display 16. As shown in the figure, the information processing device 10 may be connectable to a server or the like that provides various types of content via a network 8 such as the Internet. The input device 14 may be a general input device that can be operated by a user, such as a controller, keyboard, mouse, joystick, or touchpad, or may be an imaging device that captures images of the real world, such as a user, a microphone that captures audio, a sensor that detects various physical values, or any combination thereof.

[0012] The display 16 is realized by a liquid crystal display, a plasma display, an organic EL display, or the like that displays images. It may further include a speaker that outputs sound. The input device 14 and the display 16 may be connected to the information processing device 10 by a wired cable, or may be connected wirelessly via a wireless LAN (Local Area Network) or the like. Furthermore, the external shapes of the input device 14, the display 16, and the information processing device 10 are not limited to those shown in the drawings, and for example, two or more of them may be integrally formed.

[0013] The information processing device 10 receives a signal related to a user operation from the input device 14, performs processing in response to the signal, generates data for a display image, and outputs the data to the display 16. The information processing device 10 may be any of a game console, a personal computer, a tablet terminal, a mobile terminal, a mobile phone, etc. The content of the processing performed by the information processing device 10 may vary depending on the form of the information processing device 10, the application selected by the user, etc.

[0014] For example, the information processing device 10 progresses through an electronic game specified by a user in response to user operations, and generates and outputs game screen data at a predetermined frame rate. Alternatively, the information processing device 10 may acquire video data from a server via the network 8, sequentially decode it, and output it. As described above, the information processing device 10 may be used for a variety of purposes, and the details of the information processing performed vary accordingly, so detailed explanations will be omitted. The following description will focus on a method for suitably displaying images of content generated as a result of such information processing, images representing information to be presented, and the like.

[0015] 2 schematically shows an example of an image generated by information processing device 10 in this embodiment. In this example, main image 200a is an image that is primarily displayed, such as a game or video. Additional image 200b is an image that is temporarily displayed as needed, and the example shown includes a dialog box that prompts the user to enter a login address and password. When it becomes necessary to display such a dialog box, information processing device 10 superimposes additional image 200b including the dialog box on main image 200a that was originally displayed, and generates and outputs display image 202.

[0016] In this case, by making the main image 200a visible through the additional image 200b over as large an area as possible, it is possible to suitably integrate necessary information, such as games and videos, without interrupting the worldview of the main image 200a. Furthermore, by changing the transparency of the additional image 200b over time, it is possible to create the effect of a dialog box gradually appearing and disappearing.

[0017] Those skilled in the art will understand that there are various other cases in which multiple images are superimposed and displayed besides the illustrated example. For example, in the case of a racing game, an additional image showing an overhead view of the entire course may be displayed in addition to a main image showing the driver's field of view. When displaying a movie, an additional image showing bibliographic information such as a plot summary or cast, or an operation panel for play, pause, fast forward, etc. may be displayed.

[0018] When generating such a display image 202, the color value C of each pixel of the display image 202 is calculated by alpha blending expressed by the following equation: out can be determined. C out =(1-α)C1+αC2(Equation 1) Here, C1 and C2 are the color values ​​of the corresponding pixels in the main image 200a and the additional image 200b, respectively, and α is a general α value set for the pixel in the additional image 200b, i.e., a value between 0 and 1.0 indicating transparency.

[0019] For example, if the α value is changed from 0 to 1.0 for the entire image, the color of the additional image 200b gradually becomes darker from a state in which only the main image 200a is displayed, until finally the additional image 200b is displayed opaque. If the α value is set to an intermediate value greater than 0 and less than 1.0, the additional image 200b becomes semi-transparent with a darkness corresponding to that value, allowing the main image 200a to be seen through.

[0020] If the main image 200a and the additional image 200b are RGB images, color values ​​C1 and C2 are set for each of the three channels, but in this embodiment, these are collectively referred to as color values ​​C1 and C2. Also, because the color values ​​C1, C2 and the α value are set for each pixel, strictly speaking, they depend on the two-dimensional position coordinates (x, y) on the image plane. However, since Equation 1 assumes calculations related to pixels at the same position, the position coordinates are not shown. The same applies to the following descriptions.

[0021] 3 shows the internal circuit configuration of information processing device 10. Information processing device 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are interconnected via a bus 30. Further connected to bus 30 are a communication unit 32 including a peripheral device interface such as USB or IEEE1394, or a wired or wireless LAN network interface for connecting to network 8, a storage unit 34 such as a hard disk drive or nonvolatile memory, a display controller 36 that outputs video signals to display 16, an input unit 38 that inputs data from input device 14, and a recording medium drive unit 40 that drives a removable recording medium such as a magnetic disk, optical disk, or semiconductor memory.

[0022] The CPU 22 controls the entire information processing device 10 by executing an operating system stored in the storage unit 34. The CPU 22 also executes various programs that have been read from removable recording media and loaded into the main memory 26, or that have been downloaded via the communication unit 32. The communication unit 32 may also establish communication with an external device such as a server via the network 8 to obtain data of electronic content such as moving images, or to transmit data generated within the information processing device 10.

[0023] The main memory 26 is configured with RAM (Random Access Memory) and includes frame buffers 70a and 70b that store image data to be displayed. However, the number of frame buffers is not limited. The image data to be stored includes an alpha plane that represents an alpha value for alpha blending on an image plane. The alpha value may be set for each pixel or region, or may be a fixed value for the entire image. The setting may also change over time. The main memory 26 also stores programs and data required for processing.

[0024] The GPU 24 functions as both a geometry engine and a rendering processor, performs drawing processing in accordance with drawing commands from the CPU 22, and stores the resulting data in frame buffers 70a and 70b. The display controller 36 acquires image data from the frame buffer 70a and other sources and outputs the data as a video signal to the display 16 at appropriate times. The display controller 36 also performs the above-mentioned image synthesis processing as necessary.

[0025] 4 is a diagram illustrating a general processing flow related to a video signal. In (a), the input values ​​are pre-quantized pixel values ​​of an image rendered by the GPU 24 or the like of the information processing device 10, and the output values ​​are color values ​​to be expressed by these values; the two naturally have a linear relationship. When data having such linear characteristics is stored in the frame buffer 70a of the information processing device 10, it is quantized using an optical-electronic transfer function (OETF) as shown in (b). By using different functions for images whose luminance ranges are defined by SDR and HDR, SDR or HDR video signals having a predetermined color depth are generated.

[0026] The display controller 36 reads the video signal from the frame buffer 70a and supplies it to the display 16. The display 16 converts the video signal into a luminance value using an electro-optical transfer function (EOTF) as shown in (c). By appropriately selecting the EOTF, it is possible to obtain luminance values ​​in different ranges for each pixel, even for signals with the same color depth, such as 0 to 100 nit for an SDR image and 0 to 10,000 nit for an HDR image. These luminance values ​​are further corrected according to the luminance characteristics of the display panel and then output sequentially at appropriate times, thereby displaying images in each luminance range.

[0027] FIG. 5 shows an example of a process in which the display controller 36 reads the first and second images from the two frame buffers 70a and 70b, performs alpha blending on each pixel, and outputs the blended image to the display 16. Here, the first and second images correspond to the main image 200a and the additional image 200b in FIG. 2, respectively, and assume that the luminance range of the second image is fixed at SDR. Furthermore, the pixel values ​​of each image are generally quantized to values ​​within the range of 0 to 1.0. (a) shows a case in which a display with a corresponding luminance range of SDR is connected, and the first image is also generated in SDR. In this case, the display controller 36 can obtain the color value of the blended image and output it to the display 16 by directly substituting the corresponding pixel values ​​of both images into Equation 1.

[0028] (b) shows the case where a display with an HDR compatible luminance range is connected, and the first image is also generated in HDR. On the other hand, as described above, the second image is generated in SDR, so the display controller 36 converts the luminance range of the second image into HDR values ​​and then substitutes them into Equation 1 to obtain the color values ​​of the composite image. Conversion of the luminance range from SDR to HDR can be achieved by mapping the SDR luminance range to a portion of the HDR luminance range. For example, SDR peak luminance (1.0) is associated with a brightness of approximately 260 nits, and SDR values ​​of 0 to 1.0 are mapped to the HDR range of 0 to 260 nits. However, the brightness to be associated is not limited.

[0029] In the diagram shown in (b), the signal represented in the HDR luminance range is indicated by a thick line, and the signal represented in the SDR luminance range is indicated by a thin line. However, since the values ​​read by the display controller 36 are quantized using the OETF described above, if the two images to be combined are converted using different OETFs, this will affect the combined result. This can occur regardless of whether the luminance range is SDR or HDR. The principle behind this is explained below.

[0030] Figure 6 shows examples of conversion functions used when quantizing pixel values ​​of image data. (a) is a linear conversion function that determines the color value C after quantization so that it is directly proportional to the pixel value P represented by the original image, and if the values ​​before and after conversion are normalized, the relationship C=P holds. Hereinafter, the space of color values ​​quantized by such a function will be referred to as a "linear space." (b) is a gamma curve with γ=2.2, and C=P (1 / 2.2) Hereinafter, the space of color values ​​quantized by such a function will be called "gamma space."

[0031] Equation 1 generally assumes a pixel value P, but similar results can be obtained by combining color values ​​in a linear space transformed by the linear equation shown in (a) using Equation 1. On the other hand, in an image transformed by the nonlinear function shown in (b), the characteristics of the color value C change depending on the range of the pixel value P. This type of nonlinear transformation affects the composition using Equation 1.

[0032] Figure 7 compares a composite image in linear space with a composite image in gamma space. (a) is the composite result of two images expressed in linear space as shown in Figure 6(a), and (b) is the composite result of two images expressed in gamma space as shown in Figure 6(b). As shown above each image, the ratio C / P of the color value C to the original pixel value P is 1.0 in (a) regardless of the range of pixel values, whereas in (b) it increases as the pixel value P decreases. Therefore, when composited using Equation 1, an image containing a dark image effectively receives a larger weight, and the bright image to be composited with it receives a relatively smaller weight.

[0033] For example, comparing regions 50a and 50b, which are a semi-transparent composite of a scattered light region in a first image representing a certain scene and a dark region in a second image such as a system menu, the first image is highly transparent in (a), whereas it is almost completely opaque in (b). As mentioned above, (a) is similar to the result of combining the original pixel P, but whether (a) or (b) is preferable depends on the application. For example, the weaker the visual stimulus, the more sensitive people are to changes in brightness, so nonlinear conversion may have a higher affinity with vision.

[0034] Furthermore, when images generated using different conversion formulas are combined, various changes may occur in the combined results. Therefore, the display controller 36 of this embodiment converts the color values ​​of the images to be combined stored in the frame buffers 70a and 70b into values ​​in a common space before combining them. By allowing the application to specify the common space, it is possible to intentionally display either of the combined images shown in Figure 7(a) or (b). Hereinafter, the common space of the color values ​​converted for combining will be referred to as the "blend space."

[0035] 8 shows the processing steps performed by the display controller 36 of this embodiment to combine and output two images. First, data for the first image 52a and the second image 52b are stored in the frame buffers 70a and 70b, respectively. This data represents color values ​​C that have been converted and quantized using some conversion formula, such as the one shown in FIG. 6. The display controller 36 converts this data into data in a common blend space A (S10a, S10b).

[0036] Here, blending space "A" is, for example, the linear space shown in FIG. 6(a) or the gamma space shown in FIG. 6(b). If the original image stored in frame buffers 70a and 70b is nonlinear data converted using the OETF shown in FIG. 4, the nonlinear characteristics are first eliminated, i.e., the color values ​​are restored to have linear changes with respect to pixel values ​​(luminance), and then the image is converted to a common blending space. The process of eliminating the characteristics is qualitatively equivalent to the conversion using the OETF shown in FIG. 4. This results in color values ​​proportional to the output luminance of the display panel.

[0037] If the original image stored in the frame buffers 70a and 70b is linear data as shown in Fig. 4(a), it is converted into nonlinear data using the OETF shown in Fig. 4, and then converted using the EOTF before being converted into a common blending space. This is because the EOTF is generally not the inverse of the OETF, and Fig. 4(a) is different from the color values ​​proportional to the output luminance of the display panel.

[0038] Assuming that blend space “A” is a linear space, the color value C(linear) proportional to the output luminance at the display panel can be converted to color value C(blending space) in blend space by the following calculation: C(blending space) = C(linear) (Equation 2) If blend space "A" is a gamma space, the calculation formula is as follows: C(Blend Space) = (C(Linear)) (1 / γ) (Formula 3)

[0039] That is, the color value C in gamma space (gamma space) and the color value C in linear space (linear space) can be converted to each other by the following calculation: C(gamma space) = (C(linear space)) (1 / γ) (Formula 4) C(linear space) = (C(gamma space)) γ (Formula 5) Display controller 36 acquires blend space information specified by the application being executed from CPU 22 and switches the conversion formula accordingly. Display controller 36 then composites the two images, converted into the common blend space A, pixel by pixel using Formula 1 (S12). This allows for appropriate control of the composite result.

[0040] The display controller 36 then converts the data into a space suitable for the display 16 (S14). That is, color values ​​proportional to the output luminance of the display panel are calculated from the blend space, and data equivalent to the output of the OETF conversion is calculated by performing the inverse conversion of the EOTF shown in Figure 4 on the color values. The display controller 36 supplies the converted color values ​​to the display 16, allowing the display 16 to display an image using the same processing, regardless of whether it is a composite image or not.

[0041] FIG. 9 shows an example of an image composited using the procedure in FIG. 8. (a) shows the composite image when the blending space is linear, and (b) shows the composite image when the blending space is gamma space with γ=2.2. Looking at regions 54a and 54b, where the first and second images are semi-transparently composited, region 54a of the image composited in linear space allows the first image to be clearly visible, while region 54b of the image composited in gamma space shows the black of the second image prominently, making the first image less visible. In this way, regardless of the space in which color values ​​are stored in frame buffers 70a and 70b, the composite results can be suitably controlled by specifying the blending space.

[0042] 10 illustrates a process performed by the display controller 36 of this embodiment to combine and output three images. A first image 56a, a second image 56b, and a third image 56c are stored in three frame buffers, including the frame buffer 70a. The first image 56a and the second image 56b are generated by an application, while the third image 56c is provided independently of the application, for example, by a system. In this case, the first image 56a and the second image 56b, both generated by the application, are combined in a blend space specified by the application, similar to the process illustrated in FIG. 8.

[0043] That is, the display controller 36 converts these into data in a common blend space A (S16a, S16b) and then composites them pixel by pixel using Equation 1 (S18). This generates the composite image intended by the application creator. On the other hand, even if the blend space specified by the application is given some flexibility, it is desirable that the image provided by the system is not affected by such changes. Therefore, as shown in the figure, the third image 56c is converted into data in a unique blend space B, and if this differs from the blend space A of the application image, it is further converted so that the blend spaces of both are common.

[0044] That is, display controller 36 converts third image 56c into blend space B (S20), and converts the image composited in S18 from blend space A to data in blend space B (S22). Then, display controller 36 composites the composite image of the application with third image 56c (S24). The conversion from blend space A to blend space B can also be calculated using any of the above-mentioned equations 2 to 5. Display controller 36 further converts the image composited in blend space B into a space suitable for display 16 (S26) and outputs it.

[0045] Figure 11 shows an example of an image synthesized using the procedure in Figure 10. (a) and (b) respectively show images in which a separately generated semi-transparent fill image is superimposed on the composite images (a) and (b) shown in Figure 9. Even when the semi-transparent image is superimposed, the image visible underneath remains unchanged from the images (a) and (b) in Figure 9. Specifically, when examining regions 57a and 57b where the first and second images are semi-transparently synthesized, the first image is highly transparent in the linear space shown in (a), whereas the first image is less transparent in the gamma space shown in (b). Furthermore, the transparency of the overall image, resulting from the additionally superimposed, fully semi-transparent third image, is almost identical between (a) and (b). In this way, by using multiple blend spaces, it is possible to generate composite images that allow flexibility in application expression while not affecting the system image.

[0046] Considering the difference in luminance range of the original images, such as SDR and HDR, the conversion formulas become even more diverse. Figure 12 shows the characteristics of the conversion formulas used in HDR images and the composite results of images converted using these formulas. As shown in (a), HDR is generally quantized using a function called a PQ (Perceptual Quantization) curve. Compared to a gamma curve, the PQ curve has a larger increase in color values ​​in the low-luminance range. Therefore, when images converted in this way are composited, as shown in (b), the second image becomes even more dominant than in the composite result shown in Figure 7(b), and the first image does not show through (e.g., area 58).

[0047] Therefore, as mentioned above, the data that has been converted using the PQ curve must first have its characteristics removed, then converted into the blend space specified by the application, and then composited. Furthermore, as mentioned above, depending on the luminance range supported by the display, it may be necessary to convert HDR images to the SDR luminance range, or vice versa.

[0048] Figure 13 compares the characteristics of gamma space across various brightness ranges, assuming γ = 2.2. (a) shows the characteristics when an SDR image is converted to gamma space, (b) shows the characteristics when an SDR image is converted to gamma space after being expressed in the HDR brightness range, and (c) shows the characteristics when an HDR image is converted to gamma space. For example, by mapping the brightness on the horizontal axis in (c), which ranges from 0 to 260 nits, to the SDR brightness range of 0 to 1.0, the pixel values ​​of the SDR image can be converted to values ​​within the HDR brightness range. The horizontal axis in (b) represents the converted brightness.

[0049] As shown in the figure, in all cases, the behavior of the converted color values ​​relative to the pre-conversion color values ​​is the same. This means that as long as the same gamma space is used as the blend space, whether or not the luminance range is converted does not affect the converted color values, and therefore the image weights do not change when substituted into Equation 1. As a result, even if the luminance range is converted, the post-combination color values ​​can be appropriately controlled. Naturally, if the blend space is a linear space, converting the luminance range does not affect the blending results. Therefore, as shown in Figure 5, the blending process in the blend space described so far can be applied as is, except for converting the luminance range of the original image to the luminance range supported by the display.

[0050] 14 illustrates a processing procedure in which the display controller 36 of this embodiment displays a composite image on the HDR-compatible display 16. In this example, three frame buffers, including the frame buffer 70a, store a first image 60a, a second image 60b, and a third image 60c, respectively. Of these, the first image 60a and the second image 60b are images generated by an application, while the third image 60c is an image provided independently of the application, for example, by the system. The brightness range of each image may be SDR or HDR.

[0051] When the connected display 16 is detected as HDR-compatible, the display controller 36 converts the SDR images stored in the frame buffer into images with an HDR brightness range (S30a, 30b, 30c). Naturally, if the images are originally HDR, conversion is not necessary. The display controller 36 then converts the first image 60a and the second image 60b into data in a blend space specified by the application (S32a, S32b), and combines them pixel by pixel using Equation 1 (S34). In the illustrated example, the application specifies a linear space, so the combined image is linear space data.

[0052] Meanwhile, the display controller 36 converts the third image 60c to be further composited into data in the blend space set in the system (S36). In the illustrated example, it is converted into data in gamma space. In this case, because the composite image generated in S34 and the third image 60c are in different spaces, the former space is converted from linear space to gamma space (S38), and the two are then composited (S40). The display controller 36 further converts the image composited in gamma space into a space suitable for the display (S42) and outputs it.

[0053] If the original image is HDR, it has been converted using a PQ curve. To remove this characteristic and give it the characteristics of the blend space, if the blend space is a linear space, use the following formula: C(blending space) = PQ_EOTF(C(PQ)) (Equation 6) If the blend space is gamma space, the following formula is used: C(blend space) = PQ_EOTF(C(PQ)) (1 / γ) (Formula 7)

[0054] Here, C(PQ) is the color value obtained by conversion of the PQ curve, and PQ_EOTF corresponds to the EOTF in Figure 4 and is a function for obtaining a luminance value from a signal quantized by the PQ curve. Display controller 36 performs space conversion by selecting an appropriate formula from Formulas 2 to 7 depending on whether the original image is HDR or SDR, and in the case of SDR, whether the conversion formula applied to the original image is linear or a gamma curve, and whether the blend space is linear space or gamma space.

[0055] 15 illustrates a processing procedure in which the display controller 36 of this embodiment displays a composite image on an SDR-compatible display. As in FIG. 14, a first image 62a, a second image 62b, and a third image 62c are stored in a frame buffer 70a or the like, with the first image 62a and the second image 62b being generated by an application and the third image 62c being provided by a system or the like. The brightness ranges of the respective images are not limited.

[0056] When the connected display 16 is detected to be SDR compatible, the display controller 36 converts the HDR images stored in the frame buffer into values ​​in the SDR luminance range (S50a, 50b, 50c). The display controller 36 then converts the first image 62a and the second image 62b into data in a space specified by the application, in the illustrated example, linear space (S52a, S52b), and combines them pixel by pixel using Equation 1 (S54).

[0057] Meanwhile, the display controller 36 converts the third image 62c to be composited into the blend space set in the system, gamma space in the illustrated example (S56). Because the composite image generated in S54 and the third image 62c are in different spaces in this example, the former space is converted from linear space to gamma space (S58). The two are then composited (S60). The display controller 36 then converts the image composited in gamma space into a space suitable for display (S62) and outputs it.

[0058] 14 and 15, it can be seen that the procedures are similar except for the conversion of the luminance range as needed and the formula used for conversion to the blend space. The driver of the display controller 36 can easily switch between the procedures of FIG. 14 and FIG. 15 by appropriately setting the destination luminance range depending on the performance of the connected display 16. For example, even if the display 16 is switched to another display 16 that supports a different luminance range while an electronic game is in progress, the displayed luminance range can be adapted without affecting the composite result itself.

[0059] Note that the blend spaces shown in Figures 14 and 15 are merely examples, and may vary depending on factors such as the application developer and system settings. Furthermore, the conversion formulas used for quantization of original SDR and HDR images are not limited to linear, gamma curve, or PQ curve. Regardless of the conversion formula, as long as color values ​​proportional to the display's output luminance are obtained using the EOTF, subsequent conversions will be similar. Furthermore, by converting to a common blend space before compositing, the same effect can be achieved regardless of the number of images to be composited.

[0060] 16 is a schematic diagram showing the circuit configuration of the display controller 36 of this embodiment. In the illustrated example, four frame buffers 70a, 70b, 70c, and 70d are provided in the main memory 26. In contrast, the display controller 36 includes brightness range conversion circuits 72a, 72b, 72c, and 72d for the frame buffers 70a, 70b, 70c, and 70d, respectively, as well as synthesis circuits 78a, 78b, and 78c that sequentially weight and add the output values ​​of the brightness range conversion circuits. However, the number of brightness range conversion circuits and synthesis circuits is not limited.

[0061] The luminance range conversion circuits 72a, 72b, 72c, and 72d respectively read image data for each pixel from the frame buffers 70a, 70b, 70c, and 70d and convert, as necessary, the data into values ​​in the luminance range supported by the connected display 16. The luminance range conversion circuits 72a, 72b, 72c, and 72d also include synthesis conversion circuits 76a, 76b, 76c, and 76d, which convert color values ​​that correspond to the luminance range of the display 16 into values ​​in a predetermined blend space.

[0062] The combining circuit 78c calculates the color value of a combined image in blend space by adding the color values ​​of corresponding pixels output from the luminance range conversion circuits 72d and 72c with a weighting according to the α value. The combining circuit 78b calculates the color value of a combined image by weighting and adding the color value of a pixel output from the luminance range conversion circuit 72b and the color value of a pixel of the combined image output from the preceding combining circuit 78c. The combining circuit 78a calculates the color value of a combined image by weighting and adding the color value of a pixel output from the luminance range conversion circuit 72a and the color value of a pixel of the combined image output from the preceding combining circuit 78b.

[0063] The combining circuit 78a sequentially outputs the color values ​​of the final combined image thus obtained to the display 16. Here, the combining circuits 78b and 78c each include a combined image conversion circuit 80a or 80b, which converts the space of the image they have combined to match the blend space of the image to be combined in a subsequent stage. If the combined image has the same blend space as the image to be combined in a subsequent stage, the processing of the combined image conversion circuits 80a or 80b is omitted. In the illustrated example, the conversion by the combined image conversion circuit 80b to the same blend space as that of the combining conversion circuit 76b in the luminance range conversion circuit 72b is indicated by an open circle, and the conversion by the combined image conversion circuit 80a to the same blend space as that of the combining conversion circuit 76a in the luminance range conversion circuit 72a is indicated by a shaded circle.

[0064] The blending circuit 78a also includes an output conversion circuit 82, which converts the color values ​​of the pixels of the final composite image into values ​​in a space suitable for output to the display 16. The blending conversion circuits 76a, 76b, 76c, and 76d, the composite image conversion circuits 80a and 80b, and the output conversion circuit 82 refer to lookup tables (not shown) for each of the red, green, and blue pixel color values ​​and perform spatial conversion using a standard interpolation circuit. The blending space to which these circuits are to convert and whether conversion is necessary are controlled by the CPU 22.

[0065] Figure 17 compares images composited in linear and gamma blend spaces when displayed on SDR and HDR displays. Focusing on the central area where the first and second images are semi-transparently composited, as mentioned above, the result of compositing in gamma space places more weight on the darker areas than the result of compositing in linear space, and in this example, the transparency of the first image is reduced. On the other hand, when compositing is performed using the same blend space, the degree of transparency is similar regardless of whether the display is SDR or HDR. In other words, by selecting the blend space, the degree of transparency can be controlled appropriately regardless of the brightness range.

[0066] According to the embodiment described above, when multiple images are composited in the display controller, the conversion characteristics used for quantization are temporarily canceled and then converted again using common characteristics for compositing. This prevents a composite image with unintended transparency from being displayed due to a change in the balance of pixel values ​​to be composited due to factors other than the set alpha value in general alpha composite calculations. Furthermore, when compositing images from an application, the application itself can specify the compositing space, and the space of the composited image is further converted to match the image space of a separately provided system, etc.

[0067] This allows for flexibility in the degree of compositing of application images without affecting images from systems that display them overlaid. Furthermore, even with a nonlinear conversion formula, the characteristics themselves do not change before and after converting the luminance range between SDR and HDR, so conversion to the compositing space is performed after converting the luminance range. This allows for easy switching even when displays with different performance are connected, and allows for stable display of composite images that are not affected by display performance. Because compositing that accommodates various conditions can be achieved using the display controller hardware, application developers no longer need to take into account various user environments.

[0068] The present invention has been described above based on the embodiments. The above embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the respective components and treatment processes, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0069] 8 Network, 10 Information processing device, 14 Input device, 16 Display, 22 CPU, 24 GPU, 26 Main memory, 36 Display controller, 38 Input unit, 70a Frame buffer, 72a Brightness range conversion circuit, 76a Synthesis conversion circuit, 78a Synthesis circuit, 80a Synthesis image conversion circuit, 82 Output conversion circuit.

Claims

1. A system comprising: a plurality of brightness range conversion circuits that convert a plurality of image data, each of which has a different defined brightness range and whose pixel values ​​have been converted and quantized by different functions, into data represented in a brightness range supported by a connected display; a plurality of synthesis conversion circuits that convert the data of the plurality of images, which are represented by the brightness range conversion circuit in the brightness range supported by the display, into data in a common blend space having characteristics that change according to the same function expressed as a linear or gamma curve with respect to pixel values ​​before conversion according to the different functions; a blending circuit that performs alpha blending on the data of the plurality of images expressed in the common blend space output from the blending conversion circuit; an output conversion circuit that outputs the synthesized image data as a signal for display; A display controller comprising:

2. 2. The display controller according to claim 1, wherein the compositing circuit includes a composite image conversion circuit that converts data of the alpha-composited image into data of the common blend space between the data of the image to be alpha-composited next.

3. 3. The display controller according to claim 1, wherein when the data of the plurality of images includes data of an image generated by an application, the synthesis conversion circuit converts the data into data of the blend space specified by the application.

4. 4. The display controller according to claim 1, wherein the synthesis conversion circuit converts the data of the plurality of images into data in the common blend space by eliminating nonlinear characteristics of the pixel values ​​before conversion that were imparted to the image data and then performing conversion processing with preset characteristics.

5. 5. The display controller according to claim 1, wherein the synthesis conversion circuit converts the data using different conversion formulas depending on a luminance range defined by the image data.

6. A step of converting data of a plurality of images, each having a different defined luminance range and pixel values ​​converted and quantized by a different function, into data represented in a luminance range corresponding to the connected display; A step of converting data of the plurality of images represented in the luminance range supported by the display into data of a common blending space having characteristics that change with the same function represented by a linear or gamma curve with respect to pixel values ​​before conversion with the different functions; alpha blending data of a plurality of images expressed in the common blend space; outputting the synthesized image data to a display as a display signal; 10. A method for displaying an image by a display controller, comprising:

Images