Control circuit, display device, and display system

Abstract

[Problem] To make it possible to perform image processing on a region based on a viewing point while suppressing a data transmission amount. [Solution] This control circuit comprises: an acquisition unit that acquires first image data; an image processing unit that is capable of performing either first processing for generating a second pixel value using the pixel values of a plurality of first pixels in the first image data and / or second processing for increasing resolution by dividing each first pixel into a plurality of pixels while maintaining the pixel value of the first pixel, performs at least the first processing, and generates second image data having the second pixel value; and a display control unit that causes a display unit to display an image based on the second image data.

Description

Control Circuit, Display Device, and Display System

[0001] The present disclosure relates to a control circuit, a display device, and a display system.

[0002] There is known a display system that generates a CG image by rendering an object in a virtual space visible in the line-of-sight direction from the viewpoint position of a user wearing a head-mounted display. Such a display system is generally divided into an image generation device that generates an image and a display device that displays the image (see Patent Document 1).

[0003] Japanese Patent No. 6961018

[0004] [[ID=I12]] However, if all the image generation processes for the image to be displayed on the display device are executed by the image generation device, there is a risk that the data transmission amount to the display device will increase.

[0005] Therefore, the present disclosure provides a control circuit, a display device, and a display system capable of executing image processing on a region based on the viewpoint while suppressing the data transmission amount.

[0006] In order to solve the above problems, according to the present disclosure, an acquisition unit that acquires first image data, a first process that generates a second pixel value using the pixel values of a plurality of first pixels in the first image data, and at least one of a second process that maintains the pixel values of the first pixels and divides them into a plurality of pixels to increase the resolution, and an image processing unit that executes at least the first process and generates second image data having the second pixel value, and a display control unit that causes a display unit to display an image based on the second image data are provided.

[0007] The size of the pixel having the second pixel value may be the same as or different from the first pixel.

[0008] The first process may be at least one of a resolution increasing process and a jaggy suppression process.

[0009] The first process may generate the second pixel value by executing a convolution operation process in which coefficients are multiplied by the pixel values within the filter range.

[0010] The acquisition unit acquires information of a first region relating to the viewpoint, the first region of the first image data has a first resolution, the second region of the first image data excluding the first region has a second resolution lower than the first resolution, and the image processing unit may convert the image data in the second region to the first resolution.

[0011] The image processing unit is capable of performing at least one of a high-resolution processing and a jagged edge suppression processing. When the high-resolution processing and the jagged edge suppression processing are performed simultaneously, the filter ranges for the high-resolution processing and the jagged edge suppression processing within the second region may be set wider than the filter range for performing only the high-resolution processing within the second region.

[0012] The first region of the first image data has a first resolution, and the second region of the first image data excluding the first region has a second resolution lower than the first resolution. The image processing unit may perform the first processing on the first region and the second processing on the second region.

[0013] The acquisition unit can further acquire first depth data having depth information corresponding to the first image data, and the image processing unit may change at least one of the filter range and the distribution of the coefficients based on the depth information.

[0014] The first region of the first image data has a first resolution, and the second region of the first image data excluding the first region has a second resolution lower than the first resolution. The acquisition unit acquires the first image data of the first region with the first resolution and the second region with the second resolution, the third region with a third resolution corresponding to the first region, and the first depth data of the fourth region which corresponds to the second region and has a fourth resolution lower than the third region. The image processing unit may perform high-resolution processing and jagged edge suppression processing based on the first depth data.

[0015] The number of bits in the pixel values ​​of the first depth data may be smaller than the number of bits in the pixel values ​​of the first image data.

[0016] All pixels of the first depth data may be composed of the third resolution.

[0017] The first image data is either RGB image data or YCbCr image data, and in the case of YCbCr image data, the number of bits in the pixel values ​​of the CbCr image data may be smaller than the number of bits in the pixel values ​​of the Y image data.

[0018] The data of the first region of the first image data is image data from which the jagged edge suppression process has been performed, and the first depth data may consist only of the data of the fourth region.

[0019] When the image processing unit performs the high-resolution processing and the jagged edge suppression processing, it may reduce the resolution of the peripheral region away from the first region to less than the first resolution, and increase only the resolution of the image data of the reduced region by region division.

[0020] The data of the fifth region, which is a peripheral region of the first image data that is far from the first region, has been processed to suppress jaggies, and the first depth data may consist of data excluding the region corresponding to the fifth region.

[0021] The image processing unit may perform the high-resolution processing and the jagged edge suppression processing on the first region, and the image data excluding the first region may have its resolution increased only by region division.

[0022] The image processing unit may be capable of performing processing using a neural network.

[0023] The first image data may be image data divided into vertical strips.

[0024] The acquisition unit acquires information regarding the viewpoint position, and the image processing unit performs a convolution operation using coefficients. The degree of frequency reduction of the convolution operation may be changed based on at least one of the distance from the viewpoint position, the amount of jaggedness suppression, and the resolution.

[0025] To solve the above problems, the present disclosure provides a display device comprising: a control circuit; a pixel array unit having a plurality of pixel circuits arranged in the horizontal and vertical directions; and a light emission control unit that writes a signal voltage to the pixel circuits and performs light emission control based on second image data generated by the control circuit, and is also capable of simultaneously writing one pixel value to n (where n is a natural number of 2 or more, including 4) of the pixel circuits, wherein the control circuit is capable of performing at least one of a first process of generating a second pixel value using the pixel values ​​of a plurality of first pixels in the first image data, and a second process of increasing the resolution by maintaining the pixel value of the first pixel and dividing it into a plurality of pixels, and performs at least the first process and generates the second image data having the second pixel value; and a display control unit that causes the pixel array unit to display an image based on the second image data via the light emission control unit.

[0026] To solve the above problems, the present disclosure provides a display system comprising: an image generation device that generates at least first image data; and a control circuit, wherein the image generation device generates first image data of a region and a resolution based on the region, according to the degree of gaze based on information of the gaze point; the control circuit is capable of performing at least one of the following: an acquisition unit that acquires the first image data having information of the region; a first process that generates a second pixel value using the pixel values ​​of a plurality of first pixels in the first image data; and a second process that increases the resolution by maintaining the pixel value of the first pixel and dividing it into a plurality of pixels; an image processing unit that performs at least the first process and generates second image data having the second pixel value; and a display control unit that causes a display unit to display an image based on the second image data.

[0027] A diagram showing the configuration of the display system according to this embodiment. A block diagram showing an example of the configuration of the display system. An external view when the display device is configured as a head-mounted display. A diagram showing an example of the configuration of the display unit and the gaze point detection unit of the display device. A block diagram showing an example of the configuration of the image generation device. A diagram showing an example of area information. A diagram showing an example of the judgment process of the gaze intensity determination unit. A diagram showing an example of polygons used by the image generation unit. A diagram showing an example of the transmission mode of image data. A block diagram showing an example of the configuration of the display device of the display device. A diagram showing an example of the calculation process of the calculation processing unit. A diagram showing an example of high-resolution processing. A diagram showing an example of convolution calculation for high-resolution processing. A diagram showing an example of jagged edge suppression processing. A diagram showing an example of convolution calculation for jagged edge suppression processing. A diagram showing an example of high-resolution processing and jagged edge suppression processing. A block diagram showing an example of the configuration of the image generation device according to the second embodiment. A block diagram showing an example of the configuration of the display device according to the second embodiment. A diagram showing an example of high-resolution processing and jagged edge suppression processing using depth information. Figure (2) showing an example of high-resolution processing and jagged edge suppression processing using depth information. Figure (3) showing an example of high-resolution processing and jagged edge suppression processing using depth information. Figure (4) showing an example of high-resolution processing and jagged edge suppression processing using depth information. Figure (5) showing an example of high-resolution processing and jagged edge suppression processing using depth information. Figure (6) showing an example of high-resolution processing and jagged edge suppression processing using depth information. Figure (7) showing an example of high-resolution processing and jagged edge suppression processing using depth information. Figure (8) showing an example of high-resolution processing and jagged edge suppression processing using depth information. Figure (9) showing an example of high-resolution processing and jagged edge suppression processing using depth information. Flowchart showing an example of processing for an image generation device. Figure schematicly showing an example of image data in the processing of an image generation device. Flowchart showing an example of processing for a display device. Figure schematicly showing an example of image data in the processing of a display device. Figure showing an example of processing for an image display system according to the third embodiment. Flowchart showing an example of processing for an image generation device according to the third embodiment. Flowchart showing an example of processing for a display device according to the third embodiment. A diagram showing an example of concentric circles with the viewpoint as the center point. A diagram representing one example of pixel configuration. A diagram representing one example of pixel configuration. A diagram representing one example of pixel configuration. A diagram representing one example of pixel configuration. A diagram representing one example of pixel configuration. A diagram representing one example of pixel configuration. A diagram representing one example of pixel configuration.A diagram showing an example of a pixel configuration. A diagram showing an example of a pixel configuration. A diagram showing an example of the appearance of a head-mounted display. A diagram showing an example of the appearance of another head-mounted display. A front view showing an example of the appearance of a digital still camera. A side view showing an example of the appearance of a digital still camera. A diagram showing an example of the appearance of a smartphone. A diagram showing an example of the interior of a vehicle viewed from the rear. A diagram showing an example of the interior of a vehicle viewed from the left rear.

[0028] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0029] (First Embodiment) Figure 1 is a diagram showing the configuration of a display system 1 according to this embodiment. The display system 1 comprises an image generation device 100 and a display device 200. The image generation device 100 and the display device 200 communicate with each other via an interface 300, such as MIPI (Mobile Industry Processor Interface), as an example. The display device 200 is, for example, a head-mounted display.

[0030] Figure 2 is a block diagram showing an example configuration of the display system 1. The image generation device 100 is configured to include a CPU (central processing unit). The display device 200 has a control circuit 210 and a display unit 220. The image generation device 100 transmits first image data having region information based on the viewpoint analyzed by the image analysis device 59A (see Figure 4) to the control circuit 210 via MIPI.

[0031] The control circuit 210 is configured, for example, to include a CPU. The control circuit 210 is configured, for example, as a display element driving LSI. As described above, the control circuit 210 performs at least one of a high-resolution processing and a jagged edge suppression processing on the image data transmitted from the image generation device 100. When the jagged edge suppression processing is performed, there are cases in which the high-resolution processing is performed and cases in which it is not. That is, the control circuit 210 is capable of performing at least one of the following: a process that generates pixel values ​​using the pixel values ​​of multiple pixels in the image data based on region information, and a process that increases the resolution by dividing the pixel values ​​into multiple pixels. The control circuit 210 then performs control to display the second image data on the display unit 220.

[0032] The image generation device 100 predicts the position and orientation information of the display device 200 from the current position and orientation information of the display device 200, taking into account the delay from image generation to display, and then draws the image to be displayed on the display device 200 based on the predicted position and orientation information of the display device 200 and transmits it to the display device 200.

[0033] An example of the image generation device 100 is a game console. The image generation device 100 may also be connected to a server via a network. In that case, the server may provide the image generation device 100 with online applications such as games that multiple users can participate in via the network. The display device 200 may be connected to a computer, mobile terminal, television, etc., instead of a head-mounted display.

[0034] Figure 3 is an external view of the display device 200 when configured as a head-mounted display. The display device 200 is an image display device worn on the user's head for viewing still images and videos displayed on the screen and listening to audio and music output from headphones. The display device 200 is a device capable of performing at least one of the following on image data transmitted from the image generation device 100: high-resolution processing and jagged edge suppression processing. As described above, when performing jagged edge suppression processing, there are cases where high-resolution processing is performed and cases where it is not.

[0035] The display device 200 can measure the position information of the user's head and orientation information such as the rotation angle and tilt of the head, using a gyro sensor or accelerometer that is built into or attached to the display device 200.

[0036] The display device 200 is equipped with a camera unit, which can capture images of the outside world while the user is wearing the display device 200. It also has a gaze point detection unit 60 (see Figure 4), which can generate information regarding the wearer's viewpoint.

[0037] The display device 200 is an example of a "wearable display." Here, it is not limited to head-mounted displays in the narrow sense, but can also be applied when wearing glasses, glasses-type displays, glasses-type cameras, headphones, headsets (headphones with microphones), earphones, earrings, ear-hook cameras, hats, hats with cameras, hairbands, etc.

[0038] (Example of configuration of display unit and gaze point detection unit) Figure 4 is a diagram showing an example of the configuration of the display unit 220 and gaze point detection unit 60 of the display device 200. Referring to Figure 4, an example of the configuration of the gaze point detection unit 60 will be described. The display unit 220 is provided inside the display device 200.

[0039] The display unit 220 is, for example, a graphic display having a wide field of view (FOV). It has an optical element for display in front of each eye (binocular HMD). In this embodiment, the example of a binocular HMD is described, but it is not limited to this. For example, a single-eye (monocular HMD) may also be used.

[0040] This light-emitting element is, for example, an LED (Light Emitting Diode). LEDs include LEDs used in μLED displays and OLEDs (Organic Light Emitting Diodes) used in organic EL displays. The display unit 220 according to this disclosure is not necessarily limited to LEDs, but can be applied to various display units 220 such as liquid crystal displays.

[0041] For example, the display unit 220 includes a pixel array unit having a plurality of pixel circuits arranged in the horizontal and vertical directions, and a light emission control unit that writes a signal voltage to the pixel circuits and executes light emission control based on the second image data generated by the control circuit 210. The light emission control unit can also write one pixel value to n pixel circuits simultaneously (for example, n = 4). Thereby, for example, the display of the second image data region having the same pixel value generated by the resolution adjustment unit 218 (see FIG. 10 described later) of the control circuit 210 described later can be accelerated. Thus, the control circuit 210 causes the pixel array unit to display an image based on the second image data via the light emission control unit.

[0042] In addition, in the display unit 220, the screen area around the main fixation point requires a high resolution, while the area outside the main fixation point can be displayed with a low resolution because it is only seen by the peripheral vision.

[0043] The lower part of FIG. 4 schematically shows the display device 200 and the head F of the user wearing it as seen from above. A display image 61P is displayed on the display device 200, and the user is viewing it with the left and right eyes EL and ER. The display image 61P is composed of images 61PL and 61PR viewed by the left and right eyes EL and ER respectively. In this example, as the fixation point detection unit 60, an eye imaging sensor 59SR L, 59SR R composed of infrared LEDs 59ST L, 59ST R, an infrared camera, or a PSD (Position Sensitive Detector) sensor, and an image analysis device 59A are provided.

[0044] The infrared LEDs 59ST L and 59ST R irradiate infrared rays to the left and right eyes EL and ER of the user respectively. The eye imaging sensors 59SR L and 59SR R image the left and right eyes EL and ER of the user respectively, and supply the data to the image analysis device 59A.

[0045] The image analysis device 59A identifies the reflection position of the infrared rays on the cornea and the position of the pupil from the captured images of the left and right eyes, and identifies the user's line of sight from the positional relationship. This method has been put into practical use in the field of line of sight detection technology as the corneal reflection method.

[0046] Note that the method for detecting the line of sight is not limited to this. For example, any general method such as a technique for imaging the left and right eyes with a visible light camera and specifying the line of sight from the positional relationship between the inner corner of the eye and the iris may be adopted.

[0047] The image analysis device 59A uses the intersections RPL and RPR between the line of sight detected by the eye imaging sensors 59SRL and 59SRR and the display unit 220 (display panel) on which the display image 61P is displayed as the user's fixation points, detects their position coordinates, and supplies them to the image generation device 100. Any general method such as a technique for specifying the line of sight from the positional relationship between the inner corner of the eye and the iris may be adopted.

[0048] FIG. 5 is a block diagram showing a configuration example of the image generation device 100. The image generation device 100 includes a region determination unit 102, an image generation unit 106, a first storage unit 110, and a video signal conversion unit 112.

[0049] The region determination unit 102 determines region information based on the viewpoint analyzed by the image analysis device 59A. FIG. 6 is a diagram showing an example of the region information. The first region Z1 is the area of interest of the wearer. The second region Z2 is the peripheral region of the first region Z1. The region determination unit 102 generates information indicating the first region Z1 as the region information. The details of the region determination unit 102 will be described later. Also, the area of interest may be referred to as the foveal region.

[0050] The image generation unit 106 generates, for example, CG (Computer Graphics) image data having region information. This CG image data is processed by the display device 200 and displayed on the display unit 220. The details of the image generation unit 106 will also be described later.

[0051] The first storage unit 110 stores data related to the objects used for generating the CG image data. In this embodiment, for example, an object has data of a plurality of polygons defined as coordinates in a three-dimensional space. These polygons constitute the surface of the object. Also, texture data for the CG image corresponding to the plurality of polygons is stored.

[0052] The video signal conversion unit 112 transmits the first image data containing region information to the display device 200. The video signal conversion unit 112 is, for example, an encoder, which converts the data into MIPI format and outputs it to the display device 200. Details of the video signal conversion unit 112 will be described later.

[0053] Here, the details of the region determination unit 102 will be explained. The region determination unit 102 has a gaze intensity determination unit 104. The region determination unit 102 can change the range of the first region Z1 based on the determination result of the gaze intensity determination unit 104. For example, the amount of viewpoint shift output by the image analysis device 59A (see Figure 3) is measured for a predetermined measurement period t100. The gaze intensity determination unit 104 determines that the smaller the amount of viewpoint shift, the higher the degree of gaze.

[0054] The region determination unit 102 narrows the first region Z1 as the degree of fixation increases. The region determination unit 102 also generates resolution information for the first region Z1 and the second region Z2. As the first region Z1 is narrowed, the amount of computation required by the image generation device 100 and the display device 200 can be reduced, which improves processing speed and reduces power consumption.

[0055] For example, the first region Z1 has a resolution where pixel value information is stored for every 1x1 pixel, and the second region Z2 has a resolution where pixel value information is stored for every 2x2 pixels. The region determination unit 102 can also widen the first region Z1 and lower the resolution as the degree of fixation decreases.

[0056] Figure 7 shows an example of the determination process of the gaze intensity determination unit 104. The horizontal axis represents time, and the range is determined for each measurement time T100, with the first region Z1 as the ROI. More specifically, the gaze intensity determination unit 104 determines the degree of gaze CE = 0 to 1 (0: not gazed upon, 1: gazed upon) based on the amount of deviation of the measured line of sight within the measurement time T100. Next, the gaze intensity determination unit 104 changes the size of the ROI. The center of the ROI is updated, for example, in real time.

[0057] Here, the details of the image generation unit 106 will be explained. Figure 8 shows an example of a polygon used by the image generation unit 106. The image generation unit 106 reduces the number of vertices of the polygon in the second region Z2 using tessellation. The image generation unit 106 can generate CG image data corresponding to this number of vertices.

[0058] Alternatively, it is possible to generate CG image data that matches the resolution of the second region Z2. In this case, it can be transmitted to the display device 200 with low capacity. As will be described later, the display device 200 can use region information to increase the resolution of the image data in the first region Z1. In this way, the image generation unit 106 generates, for example, CG image data corresponding to the latest viewpoint position and line of sight of the display device 200 as the first image data.

[0059] Furthermore, the image generation unit 106 includes a first resolution conversion unit 108. The first resolution conversion unit 108 is capable of converting a first resolution corresponding to a first region Z1 of the first image data and a second resolution corresponding to a second region Z2 to different resolutions.

[0060] Here, a detailed example of the video signal conversion unit 112 will be explained. Figure 9 shows an example of the image data transmission mode. Figure 9(a) is the first mode, in which the image is transmitted column by column. Figure 9(b) is the second mode, in which the image is divided into data areas Da100 for each vertical strip and transmitted. Dividing the image into vertical strips improves the cache efficiency of the convolution image processing on the display device 200 side. In other words, if you want to perform subsequent processing quickly with less memory, you can achieve higher speed by transferring the image in vertical strips. As mentioned above, this is because the buffer utilization efficiency in the convolution operation improves, and processing can be done sequentially from the data that is sent.

[0061] Furthermore, if the data type (DataType) does not conform to the MIPI data structure, the video signal conversion unit 112 can transmit data using bits such as "User Defined 8-bit Data Type 1" to "User Defined 8-bit Data Type 8" of the DataTypeID. Also, as will be described later, when transmitting YCbCr format data, if the resolution differs for the Y data, CbCr data, and depth data, it is possible to send the data in separate frames.

[0062] Furthermore, by sending RGB data or YCbCr data as a mixed stream without separating frames, processing on the subsequent display device 200 can be sped up. In other words, if frames are not separated, processing can be performed before all frames are received.

[0063] Figure 10 is a block diagram showing an example configuration of the control circuit 210 of the display device 200. As shown in Figure 10, the control circuit 210 includes a video signal conversion unit 212, an image processing unit 214, a second storage unit 222, and a display control unit 224.

[0064] The video signal conversion unit 212 acquires first image data having region information. This video signal conversion unit 212 is, for example, a decoder and restores the transmitted MIPI format data. The restored data is stored in the second storage unit 222. In this way, the video signal conversion unit 212 acquires first image data having region information. Note that the video signal conversion unit 212 in this embodiment corresponds to the acquisition unit.

[0065] The image processing unit 214 performs at least one of the following on the image data based on the region information: a high-resolution processing process and a jagged edge suppression process. When the jagged edge suppression process is performed, the high-resolution processing process may or may not be performed. As described above, the image processing unit 214 is capable of performing at least one of the following on the region information: a process that generates pixel values ​​using the pixel values ​​of multiple pixels in the image data, and a process that increases the resolution by maintaining the pixel values ​​of pixels and dividing them into multiple pixels. The image processing unit 214 includes an arithmetic processing unit 216 and a resolution adjustment unit 218.

[0066] The arithmetic processing unit 216 performs arithmetic processing based on region information. Figure 11 shows an example of arithmetic processing by the arithmetic processing unit 216. Figures 10(a) to (c) show examples of counting in a convolution operation. The x and y axes correspond to the x and y axes of the first image data. The vertical axis shows the coefficient value. Note that the processing of the arithmetic processing unit 216 is not limited to so-called filtering processes such as convolution operations. For example, any process that outputs at least one pixel value based on multiple pixel values ​​is acceptable, and it is also possible to use a neural network, for example.

[0067] As shown in Figure 11, the arithmetic processing unit 216 can perform a convolution operation as an example of arithmetic processing. This convolution operation changes the combination of the range of the convolution filter window, which is the range of the convolution operation, and the coefficients, depending on the purpose. The range of the filter window increases from (a) to (c) in Figure 11. Also, the maximum value of the coefficients decreases from (a) to (c). As can be seen from this, the low-frequency components in the processed image increase from (a) to (c). As the low-frequency components increase, the image becomes what is called blurred.

[0068] The arithmetic processing unit 216 performs a convolution operation (c) that increases the low-frequency component in the region of the jaggies suppression process, for example, to suppress jaggies. On the other hand, the arithmetic processing unit 216 performs a convolution operation (a) in the high-resolution processing (upsampling) that increases the resolution. Furthermore, when performing jaggies suppression and high-resolution processing simultaneously, the coefficients and mask range are adaptively changed within the range of (a) to (c). Jaggies refer to the jagged, step-like edges that appear on edges and contours. Jaggies suppression corresponds to so-called "anti-aliasing" or "smoothing" processing. In other words, jaggies suppression is a process that makes jaggies less noticeable.

[0069] The resolution adjustment unit 218 performs a process that increases only the resolution by dividing the pixel data into regions. For example, it divides image data for one pixel into image data for four pixels while maintaining the same pixel value, thereby increasing the resolution.

[0070] The second storage unit 222 stores control parameters, image data acquired by the video signal conversion unit 212, and image data generated by the image processing unit 214. The display control unit 224 executes control to display the image data generated by the image processing unit 214 on the display unit 220.

[0071] (Example of high-resolution processing) Figure 12 shows an example of high-resolution processing. The image generation device 100 transmits the first image data G100 to the control circuit 210 of the display device 200 via MIPI. The video signal conversion unit 212 of the control circuit 210 acquires the first image data G101 through restoration processing. In the following explanation, an example of data from one eye will be used, but the data from the other eye is similar. That is, the object is displayed three-dimensionally on the display unit 220 using the CG data from both eyes.

[0072] Figure 13 shows an example of a convolution operation for high resolution. Based on region information, the arithmetic processing unit 216 of the image processing unit 214 sets the center coordinates M100 of the window F100 of the convolution filter to the center coordinates M100 of pixels obtained by dividing the region of one pixel in the image data of the second region Z2 of the first image data G101 into four equal parts. Based on the center coordinates M100, the arithmetic processing unit 216 increases the resolution of the image data of the second region Z2 by performing a convolution operation, for example (a) (see Figure 11), and generates the second image data G102. The display control unit 224 displays the second image data G102 as the third image data G103 for display on the display unit 220.

[0073] This processing makes it possible to increase the resolution of the image data in the second region Z2 while suppressing jagged edges. Furthermore, since the image resolution is increased after transmission from the image generation device 100, the data capacity during transmission can be reduced. In addition, the amount of signal processing on the image generation device 100 side can be reduced, resulting in power savings. As can be seen from these points, the resolution of the second region Z2 can be increased while maintaining a reduction in the computational load of the image generation device 100 and a reduction in the amount of video transmitted to the display device 200. This suppresses the impairment of immersion and realism.

[0074] (Example of jagged edge suppression processing) Figure 14 shows an example of jagged edge suppression processing. The processing is the same as in Figure 11 up to the acquisition of the first image data G101. Here, we will explain an example in which jagged edge suppression processing is performed on the first region Z1 of the first image data G101.

[0075] Figure 15 shows an example of a convolution operation for jagged edge suppression. Based on region information, the arithmetic processing unit 216 of the image processing unit 214 sets the center coordinates M100 of the pixels in the first region Z1 of the first image data G101 as the center M100 of the window F100 of the convolution filter. Based on the center M100, the arithmetic processing unit 216 blurs the image data by performing a convolution operation, for example, (c) (see Figure 10), to generate the second image data G102a. In the jagged edge suppression process, based on the pixel values, a convolution operation, for example, (c) (see Figure 10), is performed on the regions corresponding to jagged areas (regions with differences in pixel values, such as edges).

[0076] The resolution adjustment unit 218 performs a process to increase the resolution by dividing the pixel data of the second region Z2 of the second image data G102a into regions, thereby generating the third image data G103a. The display control unit 224 displays the third image data G103a on the display unit 220.

[0077] This processing suppresses jaggies in the first region Z1 of the first image data G101. Furthermore, since jaggies are suppressed and the image resolution is increased after transmission from the image generation device 100, the data capacity during transmission can be reduced. In addition, the amount of signal processing on the image generation device 100 side can be reduced, enabling power saving. Moreover, since convolution calculations are not performed in the peripheral region, the second region Z2, the amount of signal processing on the display device 200 side can be reduced, enabling power saving and higher speed. In this way, the resolution of the second region Z2 can be increased while maintaining the reduction in the amount of calculations on the image generation device 100 and the amount of video transmission to the display device 200 side. As a result, the impairment of immersion and realism is suppressed.

[0078] (Examples of high-resolution processing and jagged edge suppression processing) Figure 16 shows examples of high-resolution processing and jagged edge suppression processing. The processing up to the acquisition of the first image data G101 is the same as in Figure 12. Here, we will explain the jagged edge suppression processing for the entire first image data G101 and the high-resolution processing of the pixel data in the second region Z2.

[0079] Figure 17 shows an example of a convolution operation for high-resolution processing and jagged edge suppression processing. Based on region information, the arithmetic processing unit 216 of the image processing unit 214 sets the center coordinates M100 of pixels obtained by dividing the region of one pixel in the image data of the second region Z2 of the first image data G101 into four equal parts as the center of the window F100 of the convolution filter. Based on the center coordinates M100, the arithmetic processing unit 216 performs a convolution operation, for example (c) (see Figure 11), to increase the resolution of the image data of the second region Z2 and also performs jagged edge suppression processing. For pixels that only undergo high-resolution processing of the second region Z2, the same processing as in Figure 12 is performed. That is, the range and coefficients of the convolution differ between pixels that undergo both high-resolution processing and jagged edge suppression processing and pixels that only undergo high-resolution processing.

[0080] For the image data of the first region Z1 of the first image data G101, the same processing as in Figure 15 is performed. Through this processing, the arithmetic processing unit 216 generates the second image data G102b. The display control unit 224 displays the second image data G102b as the third image data G103b for display on the display unit 220.

[0081] This processing suppresses jaggies across the entire range of the first image data G101 while increasing the resolution of the image data in the second region Z2. Furthermore, since jaggies are suppressed and resolution is increased after the image is transmitted from the image generation device 100, the data capacity during transmission can be reduced. In addition, the amount of signal processing on the image generation device 100 side can be reduced, enabling power saving.

[0082] As described above, in this embodiment, the image processing unit 214 outputs at least one pixel value based on the pixel values ​​of multiple pixels in the first image data G101 based on region information, and generates second image data G102, G102a, and G102b by converting at least one pixel value among the pixel values ​​and resolution of the first image data G101. As a result, jaggies are suppressed and the resolution is increased after the first image data G100 is transmitted from the image generation device 100 to the display device 200, thereby reducing the data capacity during transmission. Furthermore, the resolution of the second region Z2 can be increased while maintaining the reduction in the computation amount of the image generation device 100 and the reduction in the amount of video transmitted to the display device 200. This suppresses the impairment of immersion and realism.

[0083] (Second Embodiment) The display system 1 according to the second embodiment differs from the display system 1 according to the first embodiment in that the image generation device 100 is also capable of transmitting first depth data D101 corresponding to the first image data G101. The differences from the display system 1 according to the first embodiment will be described below.

[0084] Figure 18 is a block diagram showing an example configuration of the image generation device 100 according to the second embodiment. As shown in Figure 18, the image generation unit 106 according to the second embodiment further includes a depth information assignment unit 114, a depth information conversion unit 116, and a calculation processing unit 118. It also further includes a data format conversion unit 120.

[0085] The image generation unit 106 generates, for example, a CG image corresponding to the latest viewpoint position and line of sight of the display device 200 as the first image data G100, and also generates depth information corresponding to the CG image. The depth information assignment unit 114 generates this depth information as the first depth data D100. The pixel values ​​of the first image data G100 correspond to the depth values ​​of the first depth data D100. That is, the first depth data D100 has a depth value for each two-dimensional pixel. For this reason, in this embodiment, the first depth data D100 may be referred to as image data.

[0086] The depth information conversion unit 116 converts the resolution of the first depth data D100 based on the region information of the region determination unit 102. The depth information conversion unit 116 can also convert the number of bits for each depth value.

[0087] The arithmetic processing unit 118 performs the same processing as the arithmetic processing unit 216 (see Figure 10) on the image generation device 100 side. In other words, the arithmetic processing unit 118 can perform jagged edge suppression processing on the image generation device 100 side.

[0088] Figure 19 is a block diagram showing an example configuration of the control circuit 210 according to the second embodiment. As shown in Figure 19, the control circuit 210 according to the second embodiment further comprises a region indicator unit 226 and a data format conversion unit 228.

[0089] The region indication unit 226 uses depth data to instruct the image processing unit 214 on regions where jagged areas are likely to occur. For example, if the difference between adjacent pixels in the depth data exceeds a predetermined threshold, the region indication unit 226 indicates the region of those adjacent pixels as a jagged area. The data format conversion unit 228 converts YCbCr format data to RGB format data. Alternatively, the data format conversion unit 228 can also convert RGB format data to YCbCr format data.

[0090] (Example of high-resolution processing and jagged edge suppression processing using depth information) Figure 20 shows an example of high-resolution processing and jagged edge suppression processing using depth information. The processing is the same as in Figure 11 up to the acquisition of the first image data G101. The first depth data D100 has a high-resolution region of the first region Z1 and a low-resolution region of the second region Z2.

[0091] The image generation device 100 transmits the first depth data D100 to the control circuit 210 of the display device 200 via MIPI. The video signal conversion unit 212 of the control circuit 210 acquires the first depth data D101 through restoration processing. Here, we will explain the jaggedness suppression processing of the entire first image data G101 using the first depth data D101 and the high-resolution processing of the pixel data of the second region Z2. This differs from the processing example in Figure 16 in that the region instruction unit 226 instructs the arithmetic processing unit 216 to designate pixels where the absolute value of the difference between adjacent pixels of the first depth data D100 is greater than or equal to a predetermined value as jagged regions.

[0092] The arithmetic processing unit 216 of the image processing unit 214 uses the center coordinates M100 of the convolution filter window F100 as the center of the pixel obtained by dividing the area of ​​one pixel in the image data of the second region Z2 of the first image data G101 into four equal parts, based on the jagged area indicated by the region indication unit 226. Based on the center M100, the arithmetic processing unit 216 performs a convolution operation, for example (c) (see Figure 11), to increase the resolution of the image data of the second region Z2 and perform jaggedness suppression processing to generate the second image data G102c. For pixels that only undergo the resolution enhancement processing of the second region Z2, processing equivalent to that in Figure 12 is performed. That is, the range and coefficients of the convolution differ between pixels that undergo both resolution enhancement processing and jaggedness suppression processing and pixels that only undergo resolution enhancement processing.

[0093] For the image data of the first region Z1 of the first image data G101, the arithmetic processing unit 216 of the image processing unit 214 sets the center coordinates M100 of the pixels in the image data of the first region Z1 of the first image data G101 as the center M100 of the window F100 of the convolution filter, based on the jagged region indicated by the region indication unit 226. Based on the center M100, the arithmetic processing unit 216 blurs the image data of the first region Z1 by a convolution operation, for example, (c) (see Figure 11), and generates the second image data G102c.

[0094] Through this process, the arithmetic processing unit 216 generates the second image data G102c. The display control unit 224 displays the second image data G102c as the third image data G103c for display on the display unit 220.

[0095] This processing allows for the suppression of jaggies across the entire range of the first image data G101 while increasing the resolution of the image data in the second region Z2. Since the region indication unit 226 performs jaggie suppression processing on the jaggy regions indicated based on the first depth data D101, processing accuracy can be further improved. Furthermore, because jaggies are suppressed and resolution is increased after the image is transmitted from the image generation device 100, the data capacity during transmission can be reduced. Additionally, the amount of signal processing on the image generation device 100 side can be reduced, resulting in power savings. Moreover, the resolution of the second region Z2 can be increased while maintaining the reduction in the computational load of the image generation device 100 and the reduction in the amount of video transmitted to the display device 200. This suppresses the impairment of immersion and realism.

[0096] (Example of high-resolution processing and jagged edge suppression processing using depth information (2)) Figure 21 shows an example of high-resolution processing and jagged edge suppression processing (2) using depth information. The processing is the same as in Figure 20 up to the acquisition of the first image data G101 and the first depth data D100.

[0097] The arithmetic processing unit 216 of the image processing unit 214 differs from the processing in Figure 20 in that it generates the second image data G102d without performing high-resolution processing on the data in region Z3 of the first image data G101.

[0098] The resolution adjustment unit 218 of the image processing unit 214 increases the resolution of the data in region Z3 of the second image data G102d by region division, and generates the third image data G103d.

[0099] This processing makes it possible to generate the third image data G103d faster than the processing shown in Figure 20. In this case as well, since region Z3 is the peripheral area of ​​the line of sight, the wearer observing the third image data G103d can be prevented from recognizing the difference in resolution. This processing reduces the processing load on the display device 200 and suppresses power consumption.

[0100] (Example of high-resolution processing and jagged edge suppression processing using depth information (3)) Figure 22 shows an example of high-resolution processing and jagged edge suppression processing using depth information (3). The arithmetic processing unit 118 generates first image data G100a by performing jagged edge suppression processing on the data in the peripheral region Z3 of the first image data G100. Therefore, jagged edge suppression processing for the peripheral region Z3 is unnecessary on the display device 200 side. This processing reduces the processing load of the display device 200 and suppresses power consumption.

[0101] The depth information conversion unit 116 of the image generation device 100 transmits the first depth data D100a, which is obtained by removing the data in region Z3 of the first depth data D100, to the control circuit 210. The calculation processing unit 216 of the image processing unit 214 performs the same processing as in Figure 21 on the region of the first image data G100a excluding the peripheral region Z3, and generates the second image data G102e. The resolution adjustment unit 218 of the image processing unit 214 increases the resolution of the data in region Z3 of the second image data G102ed by region division, and generates the third image data G103e.

[0102] This processing method further reduces the amount of data in the first depth data D100a compared to the processing shown in Figure 21.

[0103] (Example of high-resolution processing and jagged edge suppression processing using depth information (4)) Figure 23 shows an example of high-resolution processing and jagged edge suppression processing (4) using depth information. The depth information conversion unit 116 reduces the number of bits in the peripheral region Z2 of the first depth data D100b from 8 bits in the first depth data D100 in Figure 20 to 4 bits. Except for this point, the processing is equivalent to that in Figure 20. This makes it possible to further reduce the amount of data in the first depth data D100b.

[0104] (Example of high-resolution processing and jagged edge suppression processing using depth information (5)) Figure 24 shows an example of high-resolution processing and jagged edge suppression processing (5) using depth information. The processing is equivalent to that in Figure 23, except that the first depth data D100c is composed entirely of high resolution data. This makes it possible to perform jagged edge suppression processing with higher accuracy.

[0105] (Example of high-resolution processing and jagged edge suppression processing using depth information (6)) Figure 25 shows an example of high-resolution processing and jagged edge suppression processing (6) using depth information. The data format conversion unit 120 performs the same processing as in Figure 24, except that it converts the first image data G100 in RGB format to the first image data G100b in YCbCr format. For example, the processing is the same as in Figure 24, except that the first image data G100 in RGB format, which is 8 bits, is composed of the first image data G100b in Y format with 8 bits and the first image data G100b in cb,cr format with 4 bits. This makes it possible to further reduce the amount of data to be transmitted.

[0106] (Example of high-resolution processing and jagged edge suppression processing using depth information (7)) Figure 26 shows an example of high-resolution processing and jagged edge suppression processing (7) using depth information. The arithmetic processing unit 118 performs jagged edge suppression processing on the Z1 region of the first image data G100. The data format conversion unit 120 converts the first image data G100c, after jagged edge suppression processing on the Z2 region, into first image data G100c in YCbCr format. Therefore, jagged edge suppression processing for the surrounding region Z2 is unnecessary on the display device 200 side. The depth information conversion unit 116 of the image generation device 100 generates first depth data D100d, which excludes the data in region Z2 of the first depth data D100, and transmits it to the control circuit 210. Except for these points, the processing is equivalent to that in Figure 24. This makes it possible to further reduce the amount of data to be transmitted.

[0107] (Example of high-resolution processing and jagged edge suppression processing using depth information (8)) Figure 27 shows an example of high-resolution processing and jagged edge suppression processing (8) using depth information. The first image data G100e differs from the processing in Figure 25 in that it is composed entirely of low-resolution RGB image data.

[0108] Furthermore, the arithmetic processing unit 216 of the image processing unit 214 differs from the processing in Figure 25 in that, based on the region information, it generates the second image data G102k without performing high-resolution processing on the data in region Z2 of the first image data G101e.

[0109] The resolution adjustment unit 218 of the image processing unit 214 increases the resolution of the data in region Z2 of the second image data G102k by region division, and generates the third image data G103k. This process reduces the processing load of the display device 200 and suppresses power consumption. In addition, the display pixels can be driven in groups of four in areas where jagged edge suppression processing is not required. Depth information does not need to be sent to areas where jagged edge suppression processing is not required, so the amount of video signal can also be saved.

[0110] (Example of high-resolution processing and jagged edge suppression processing using depth information (9)) Figure 28 shows an example of high-resolution processing and jagged edge suppression processing (9) using depth information. The first depth data D100e differs from the processing in Figure 27 in that only region Z1 consists of high-resolution data. This processing reduces the processing load of the display device 200 and suppresses power consumption. It also reduces the amount of data transmitted.

[0111] Figure 29 is a flowchart showing an example of processing by the image generation device 100. Here, we will explain an example of processing by the image generation device 100 in high-resolution processing using depth information and jagged edge suppression processing example (7). Figure 30 is a schematic diagram showing an example of image data in the processing of the image generation device 100.

[0112] First, the region determination unit 102 of the image generation device 100 determines the first region Z1 (step S100). The first region Z1 at this time is shown as the gray area in Figure 30(a).

[0113] Next, the image generation unit 106 generates polygon data and computer CG data as first image data G100e as a rendering process according to the position, orientation, and line of sight information of the display device 200, and the depth information assignment unit 114 generates first depth data D100 corresponding to the computer CG data. (Step S102). The polygon data of the object at this time is shown in Figure 30(d), and the first depth data D100 is shown in Figure 30(e).

[0114] Next, the first resolution conversion unit 108 generates first image data G100c by increasing the resolution of the first region Z1 and decreasing the resolution of the second region Z2 (step S104). The first image data G100c at this time is shown in Figure 30(b).

[0115] Next, the arithmetic processing unit 118 performs jagged edge suppression processing on the first region Z1 of the first image data G100c (step S106). The first image data G100c at this time is shown in Figure 30(c). Next, the data format conversion unit 120 converts the first image data G100c in RGB format to the first image data G100c in YCbCr format (step S108).

[0116] Next, the depth information conversion unit 116 generates first depth data D100d by removing the data in the first region Z1 of the first depth data D100 (step S110). The first depth data D100d at this time is shown in Figure 30(f). Next, the video signal conversion unit 112 encodes the first image data G100c and the first depth data D100d in NIPI format and transmits them to the display device 200 (step S112).

[0117] Figure 31 is a flowchart showing an example of processing by the display device 200. Here, we will explain an example of processing by the display device 200 in the high-resolution processing using depth information and the jagged edge suppression processing example (7). Figure 32 is a schematic diagram showing an example of image data in the processing of the display device 200.

[0118] First, the video signal conversion unit 212 of the display device 200 decodes and obtains the first image data G101c and the first depth data D101d (step S200). The first image data G101c at this time is shown in Figure 32(a), and the first depth data D101d is shown in Figure 32(d).

[0119] Next, the data format conversion unit 228 converts the first image data G101c in YCbCr format to the first image data G100c in RGB format (step S202). The first image data G101c at this time is shown in Figure 32(b).

[0120] Next, the arithmetic processing unit 216 performs jaggedness suppression processing and high-resolution processing on the image data of the second region Z2 of the first image data G101c according to the jagged region indicated by the region indication unit 226 based on the first depth data D101d, and generates the second image data G102j (step S204). The first image data G101c at this time is shown in Figure 32(c).

[0121] Next, the display control unit 224 causes the display unit 220 to display the second image data G102j as the third image data G103j for display (step S206). The third image data G103j at this time is shown in Figure 32(e).

[0122] As described above, according to this embodiment, the arithmetic processing unit 216 performs jagged edge suppression processing on the first image data G101 to G101e based on the jagged edge regions indicated by the region indication unit 226 using depth information. This makes it possible to perform jagged edge suppression processing with even higher precision, in addition to the same effects as the display system 1 according to the first embodiment.

[0123] (Third Embodiment) The display system 1 according to the third embodiment differs from the display system 1 according to the second embodiment in that it can use a neural network (VDSR) as a filtering process, at least for the high-resolution processing. The differences from the display system 1 according to the second embodiment will be explained below.

[0124] The arithmetic processing unit 216 according to the third embodiment can execute any of the processes A, B, or C depending on the purpose.

[0125] As part of process A, the arithmetic processing unit 216 performs high-resolution processing using a Very-Deep Super-Resolution (VDSR) neural network. VDSR is an example of a convolutional neural network architecture designed to perform single-image super-resolution processing. The (VDSR) neural network learns a mapping between low-resolution and high-resolution images. That is, this mapping is possible because low-resolution and high-resolution images have similar image content, differing mainly in fine high-frequency components. For example, the (VDSR) neural network is trained to estimate the residual image using a so-called residual learning method. In this case, processing without using the first depth data D101 is also possible.

[0126] As part of process B, the arithmetic processing unit 216 executes a process using a so-called improved shock filter that sharpens edges (excluding jagged areas) that have become dull during the execution of the high-resolution processing and jaggedness suppression processing. The improved shock filter sharpens edges while maintaining the edge structure. In other words, the arithmetic processing unit 216 can also execute a process using a so-called improved shock filter. As part of process B, the arithmetic processing unit 216 can use the area generated by the area indication unit 226 to identify jagged areas.

[0127] As part of the C processing, the arithmetic processing unit 216 performs adaptive filter-type superresolution processing when executing high-resolution processing and jagged edge suppression processing. In other words, the arithmetic processing unit 216 can use an enlargement processing technique that suppresses blurring when generating a high-resolution image from a low-resolution image and increases resolution with less computation to increase the resolution of areas excluding jagged areas. As part of the C processing, the arithmetic processing unit 216 can use the area generated by the area indication unit 226 to identify jagged areas.

[0128] Figure 33 shows an example of processing of the display system 1 according to the third embodiment. The image generation unit 106 generates CG image data corresponding to the latest viewpoint position and line of sight of the display device 200 as first image data G100. The first resolution conversion unit 108 generates first image data G100e by reducing the resolution of the first image data G100. The depth information assignment unit 114 generates depth information corresponding to the first image data G100e as first depth data D100c at high resolution. The data format conversion unit 120 converts the first image data G100e into YCbCr format data. For example, the Y format data is set to 8 bits and the cbcr format data to 4 bits. The video signal conversion unit 112 converts the first image data G100e and the first depth data D100c into MIPI format and transmits them to the display device 200.

[0129] The video signal conversion unit 212 of the display device 200 decodes and obtains the first image data G101e and the first depth data D101c. Subsequently, the data format conversion unit 228 converts the first image data G101e in YCbCr format to the first image data G101e in RGB format.

[0130] The arithmetic processing unit 216 performs a high-resolution processing using a Very-Deep Super-Resolution (VDSR) neural network as processing A, and generates the second image data G101n. In this case, high resolution is possible without using the first depth data D100c. The display control unit 224 displays the second image data G101n as the third image data G103n for display on the display unit 220.

[0131] Figure 34 is a flowchart showing an example of processing by the image generation device 100 according to the third embodiment. Here, the arithmetic processing unit 216 describes an example in which it performs high-resolution processing using a Very-Deep Super-Resolution (VDSR) neural network as process A.

[0132] First, the image generation unit 106 of the image generation device 100 generates polygon data and computer CG data as first image data G100e as a rendering process, according to the position, orientation, and line-of-sight information of the display device 200 (step S102). The polygon data of the object at this time is shown in Figure 30(d).

[0133] Next, the first resolution conversion unit 108 generates a first image data G100e with reduced resolution from the first image data G100e (step S104). The first image data G100c at this time is shown in Figure 30(b).

[0134] Next, the data format conversion unit 120 converts the first image data G100e in RGB format to the first image data G100e in YCbCr format (step S108). Then, the video signal conversion unit 112 encodes the first image data G100e in NIPI format and transmits it to the display device 200 (step S112). In this way, since a low-resolution first image data G100e is transmitted, the amount of data transmitted can be further reduced.

[0135] Figure 35 is a flowchart showing a processing example of the display device 200 according to the third embodiment. It differs from the processing example in Figure 34 in that the processing in step S204a uses the processing of the arithmetic processing unit 216 for the high-resolution processing. When using processing A, it is also possible to perform processing without using the first depth data D101c. This type of processing makes it possible to increase the resolution while maintaining the edge structure.

[0136] (Fourth Embodiment) The display system 1 according to the fourth embodiment differs from the display system 1 according to the third embodiment in that the calculation processing unit 216 of the display device 200 can further change the processing effect according to the distance from the viewpoint area. The differences from the display system 1 according to the third embodiment will be described below.

[0137] The arithmetic processing unit 216 according to the fourth embodiment performs three processes—high resolution enhancement, jagged edge suppression, and concentric blurring—by changing the convolution coefficients. More specifically, it has multiple groups of convolution coefficients ranging from those with a small degree of blurring to those with a large degree of blurring. The second storage unit 222 stores, for example, a group of 30 convolution coefficients (0 to 29) used by the arithmetic processing unit 216.

[0138] The arithmetic processing unit 216 performs a high-resolution processing. Next, for each pixel after the high-resolution processing, the arithmetic processing unit 216 selects a convolution coefficient from a group of 30 coefficients (0 to 29) according to the evaluation value Vk obtained by equation (1). In other words, the arithmetic processing unit 216 can change the degree of frequency reduction of the convolution operation by selecting a coefficient from the group of coefficients.

[0139] Vk = αxA + βxB + γxC (1) A is the anti-aliasing amount, and A = 0 or 1 (0 means no anti-aliasing, 1 means anti-aliasing is performed). For example, the region indicated by the region indicator 226 is A = 1, and all other regions are A = 0. α, β, and γ are constants.

[0140] The amount of concentric blurring is set to a range of B = 0 to 1 (0 is no blurring, 1 is the most blurred). B is set to 0 to 1 according to the distance from the viewpoint. The arithmetic processing unit 216 takes the centroid of the first region Z1 as the viewpoint, for example.

[0141] Figure 36 shows an example of concentric circles A100 to A104 with the viewpoint as the central point. For example, B changes from 0 to 1 as you move from A100 towards A104. This results in a blurred image as you move towards the peripheral field of view. C is 1 if it is within the first region Z1 (see Figure 6), and 0 if it is outside the first region Z1.

[0142] The arithmetic processing unit 216 performs a convolution operation on each pixel using coefficients based on equation (1). This convolution operation makes it possible to gradually reduce the perceived resolution in a concentric manner. By controlling the perceived resolution in this way, such as by controlling the strength of anti-aliasing, it is possible to suppress the disruption of the sense of immersion in the image. Furthermore, since the magnitude of the coefficients is controlled according to the distance from the viewpoint, anti-aliasing is unnecessary for pixels that are a predetermined distance from the viewpoint, while anti-aliasing with a greater processing effect can be performed on pixels that are close to the viewpoint.

[0143] <Pixel Circuit Configuration> The display unit 220 in the display device 200 in Figure 2 has a pixel circuit for each pixel, which includes a light-emitting element such as an organic EL element. Various modifications are possible for the specific circuit configuration of the pixel circuit. A typical circuit configuration will be described below.

[0144] (First example)

[0145] Figure 37 shows an example configuration of a pixel PIX. The pixel PIX includes a capacitor C01, transistors MN02 to MN03, and a light-emitting element EL. Transistors MN02 to MN03 are N-type MOSFETs (Metal Oxide Semiconductor Field Effect Transistors). The gate of transistor MN02 is connected to the control line WSL, the other of its source and drain is connected to the signal line SGL, and one of its source and drain is connected to the gate of transistor MN03 and one end of capacitor C01. One end of capacitor C01 is connected to one of the source and drain of transistor MN02 and the gate of transistor MN03, and the other end is connected to one of the source and drain of transistor MN03 and the anode of the light-emitting element EL. The gate of transistor MN03 is connected to one of the source and drain of transistor MN02 and one end of capacitor C01, the other source and drain is connected to the power line VCCP, and one of the source and drain is connected to the other end of capacitor C01 and the anode of light-emitting element EL. The anode of light-emitting element EL is connected to one of the source and drain of transistor MN03 and the other end of capacitor C01, and its cathode is connected to the power line Vcath. The voltage of the power line VCCP is switched as appropriate between a first voltage and a second voltage lower than the first voltage.

[0146] In this configuration, when transistor MN02 is turned on in a pixel PIX, the voltage across capacitor C01 is set based on the pixel signal supplied from signal line SGL. During the period when the voltage of power line VCCP is the first voltage, transistor MN03 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C01. The light-emitting element EL emits light based on the current supplied from transistor MN03. In this way, the pixel PIX emits light with a brightness corresponding to the pixel signal. During the period when the voltage of power line VCCP is the second voltage, the light-emitting element EL is extinguished.

[0147] (Second Specific Example) Figure 38 shows another example of the configuration of a pixel PIX. This pixel PIX has a capacitor C21, transistors MN22 to MN25, and a light-emitting element EL. Transistors MN22 to MN25 are N-type MOSFETs. The gate of transistor MN22 is connected to the control line WSL, the other of its source and drain is connected to the signal line SGL, and one of its source and drain is connected to the gate of transistor MN24 and one end of capacitor C21. One end of capacitor C21 is connected to one of the source and drain of transistor MN22 and the gate of transistor MN24, and the other end is connected to one of the source and drain of transistor MN24, the other of the source and drain of transistor MN25, and the anode of the light-emitting element EL. The gate of transistor MN23 is connected to the control line DSL, the other of its source and drain is connected to the power line VCCP, and one of its source and drain is connected to the other of the source and drain of transistor MN24. The gate of transistor MN24 is connected to one of the source and drain of transistor MN22 and one end of capacitor C21, the other source and drain is connected to one of the source and drain of transistor MN23, and one source and drain is connected to the other end of capacitor C21, the other source and drain of transistor MN25 and the anode of light-emitting element EL. The gate of transistor MN25 is connected to the control line AZSL, the other source and drain is connected to one of the source and drain of transistor MN24, the other end of capacitor C21 and the anode of light-emitting element EL, and one source and drain is connected to the power line VSS.

[0148] In this configuration, in a pixel PIX, when transistor MN22 is turned on, the voltage across capacitor C21 is set based on the pixel signal supplied from signal line SGL. Transistor MN23 is turned on and off based on the signal from control line DSL. Transistor MN24 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C21 while transistor MN23 is on. The light-emitting element EL emits light based on the current supplied from transistor MN24. In this way, the pixel PIX emits light with brightness corresponding to the pixel signal. Transistor MN25 is turned on and off based on the signal from control line AZSL. While transistor MN25 is on, the voltage at the anode of the light-emitting element EL is initialized by being set to the voltage of power line VSS.

[0149] Transistors MN22 to MN25 may be transistors made of low-temperature polycrystalline silicon (LTPS). Furthermore, at least one of transistors MN22 and MN25 may be a transistor made of oxide semiconductor.

[0150] (Third Specific Example) Figure 39 shows another example of the configuration of a pixel PIX. This pixel PIX has a capacitor C21, transistors MN22 to MN25, and a light-emitting element EL. Transistors MN22 to MN25 are N-type MOSFETs. The gate of transistor MN22 is connected to the control line WSL, the other of its source and drain is connected to the signal line SGL, and one of its source and drain is connected to the gate of transistor MN24 and one end of capacitor C21. One end of capacitor C21 is connected to one of the source and drain of transistor MN22 and the gate of transistor MN24, and the other end is connected to one of the source and drain of transistor MN24, the other of the source and drain of transistor MN25, and the anode of the light-emitting element EL. The gate of transistor MN23 is connected to the control line DSL, the other of its source and drain is connected to the power line VCCP, and one of its source and drain is connected to the other of the source and drain of transistor MN24. The gate of transistor MN24 is connected to one of the source and drain of transistor MN22 and one end of capacitor C21, the other source and drain is connected to one of the source and drain of transistor MN23, and one source and drain is connected to the other end of capacitor C21, the other source and drain of transistor MN25 and the anode of light-emitting element EL. The gate of transistor MN25 is connected to the control line AZSL, the other source and drain is connected to one of the source and drain of transistor MN24, the other end of capacitor C21 and the anode of light-emitting element EL, and one source and drain is connected to the power line VSS.

[0151] In this configuration, in a pixel PIX, when transistor MN22 is turned on, the voltage across capacitor C21 is set based on the pixel signal supplied from signal line SGL. Transistor MN23 is turned on and off based on the signal from control line DSL. Transistor MN24 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C21 while transistor MN23 is on. The light-emitting element EL emits light based on the current supplied from transistor MN24. In this way, the pixel PIX emits light with brightness corresponding to the pixel signal. Transistor MN25 is turned on and off based on the signal from control line AZSL. While transistor MN25 is on, the voltage at the anode of the light-emitting element EL is initialized by being set to the voltage of power line VSS.

[0152] Transistors MN22 to MN25 may be transistors made of low-temperature polycrystalline silicon (LTPS). Furthermore, at least one of transistors MN22 and MN25 may be a transistor made of oxide semiconductor.

[0153] (Fourth specific example)

[0154] Figure 40 shows another example of a pixel PIX configuration. This pixel PIX includes a capacitor C31, transistors MP32 to MP36, and a light-emitting element EL. Transistors MP32 to MP36 are P-type MOSFETs. The gate of transistor MP32 is connected to the control line WSL, one of its source and drain is connected to the signal line SGL, and the other of its source and drain is connected to the gate of transistor MP33, the other of its source and drain of transistor MP34, and the other end of capacitor C31. One end of capacitor C31 is connected to the power line VCCP, and the other end is connected to the other of its source and drain of transistor MP32, the gate of transistor MP33, and the other of its source and drain of transistor MP34. The gate of transistor MP34 is connected to control line AZSL1, one of its source and drain is connected to the other of the source and drain of transistor MP33, and one of the source and drain of transistor MP35, and the other of its source and drain is connected to the other of the source and drain of transistor MP32, the gate of transistor MP33, and the other end of capacitor C31. The gate of transistor MP35 is connected to control line DSL, one of its source and drain is connected to the other of the source and drain of transistor MP33, and one of the source and drain of transistor MP34, and the other of its source and drain is connected to one of the source and drain of transistor MP36 and the anode of the light-emitting element EL. The gate of transistor MP36 is connected to control line AZSL2, one of its source and drain is connected to the other of the source and drain of transistor MP35, and the anode of the light-emitting element EL, and the other of its source and drain is connected to power line VSS.

[0155] In this configuration, in a pixel PIX, when transistor MP32 is turned ON, the voltage across capacitor C31 is set based on the pixel signal supplied from signal line SGL. Transistor MP35 is turned ON or OFF based on the signal from control line DSL. Transistor MP33 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C31 while transistor MP35 is ON. The light-emitting element EL emits light based on the current supplied from transistor MP33. In this way, the pixel PIX emits light with a brightness corresponding to the pixel signal. Transistor MP34 is turned ON or OFF based on the signal from control line AZSL1. While transistor MP34 is ON, the drain and gate of transistor MP33 are connected to each other. Transistor MP36 is turned ON or OFF based on the signal from control line AZSL2. While transistor MP36 is ON, the voltage of the anode of the light-emitting element EL is initialized by being set to the voltage of power line VSS.

[0156] Transistors MP32 to MP36 may be transistors made of low-temperature polycrystalline silicon (LTPS). Furthermore, at least one of transistors MP32, MP34, and MP36 may be a transistor made of oxide semiconductor.

[0157] (Fifth specific example)

[0158] Figure 41 shows another example of the pixel PIX configuration. One end of capacitor C48 is connected to the signal line SGL1, and the other end is connected to the power line VSS. One end of capacitor C49 is connected to the signal line SGL1, and the other end is connected to the signal line SGL2. Transistor MP49 is a P-type MOSFET, with its gate connected to the control line WSL2, one of its source and drain connected to the signal line SGL1, and the other of its source and drain connected to the signal line SGL2.

[0159] Each pixel PIX includes a capacitor C41, transistors MP42 to MP46, and a light-emitting element EL. Transistors MP42 to MP46 are P-type MOSFETs. The gate of transistor MP42 is connected to the control line WSL1, one of its source and drain is connected to the signal line SGL2, and the other of its source and drain is connected to the gate of transistor MP43 and the other end of capacitor C41. One end of capacitor 41 is connected to the power line VCCP, and the other end is connected to the other of its source and drain of transistor MP42 and the gate of transistor MP43. The gate of transistor MP43 is connected to the other of its source and drain of transistor MP42 and the other end of capacitor C41, one of its source and drain is connected to the power line VCCP, and the other of its source and drain is connected to one of its source and drain of transistors MP44 and MP45. The gate of transistor MP44 is connected to control line AZSL1, one of its source and drain is connected to the other of the source and drain of transistor MP43, and one of the source and drain of transistor MP45, and the other of its source and drain is connected to signal line SGL2. The gate of transistor MP45 is connected to control line DSL, one of its source and drain is connected to the other of the source and drain of transistor MP43, and one of the source and drain of transistor MP44, and the other of its source and drain is connected to one of the source and drain of transistor MP46 and the anode of the light-emitting element EL. The gate of transistor MP46 is connected to control line AZSL2, one of its source and drain is connected to the other of the source and drain of transistor MP45 and the anode of the light-emitting element EL, and the other of its source and drain is connected to power line VSS.

[0160] In this configuration, in a pixel PIX, when transistor MP42 is turned ON, the voltage across capacitor C41 is set based on the pixel signal supplied to signal line SGL1. Transistor MP45 is turned ON or OFF based on the signal on control line DSL. Transistor MP43 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C41 while transistor MP45 is ON. The light-emitting element EL emits light based on the current supplied by transistor MP43. In this way, the pixel PIX emits light with a brightness corresponding to the pixel signal. Transistor MP44 is turned ON or OFF based on the signal on control line AZSL1. While transistor MP44 is ON, the drain of transistor MP43 and signal line SGL2 are connected to each other. Transistor MP46 is turned ON or OFF based on the signal on control line AZSL2. While transistor MP46 is ON, the voltage of the anode of the light-emitting element EL is initialized by setting it to the voltage of power line VSS.

[0161] Furthermore, transistors MP42 to MP46 and MP49 may be transistors using low-temperature polycrystalline silicon (LTPS). Also, at least one of transistors MP42, MP46, and MP49 may be a transistor using an oxide semiconductor. (Specific Example 6)

[0162] Figure 42 shows another example of a pixel PIX configuration. Multiple pixels PIX are arranged in a matrix in the display area 1000, and the display area 1000 is located between the first control unit 40 and the second control unit 70.

[0163] The first control unit 40 includes transmission gates TG45 and TG46, transistors MP56 and MP57, and capacitor C61. Transistors MP56 and MP57 are P-type MOSFETs. A pixel signal is supplied to one end of transmission gate TG45, and the other end of transmission gate TG45 is connected to signal line 14a. One end of transmission gate TG46 is connected to signal line 14b, and the other end of transmission gate TG46 is connected to power line Vorst. One end of capacitor C61 is connected to signal line 14a, and the other end is connected to power line VSS1. The gate of transistor MP56 is connected to control line INIL, one of its source and drain is connected to power line Vini, and the other of its source and drain is connected to signal line 14b. The gate of transistor MP57 is connected to control line ELL, one of its source and drain is connected to power line Vel, and the other of its source and drain is connected to signal line 14b.

[0164] The second control unit 70 includes a transmission gate TG72, a transistor MP73, and a capacitor C82. The transistor MP73 is a P-type MOSFET. One end of the transmission gate TG72 is connected to the signal line 14a, and the other end is connected to the other of the source and drain of the transistor MP73 and to one end of the capacitor C82. The gate of the transistor MP73 is connected to the control line REFL, one of the source and drain is connected to the power line Vref, and the other of the source and drain is connected to the other end of the transmission gate TG72 and one end of the capacitor C82. One end of the capacitor C82 is connected to the other end of the transmission gate TG72 and the other of the source and drain of the transistor MP73, and the other end is connected to the signal line 14b.

[0165] Each pixel PIX includes a capacitor C132, transistors MP121 to MP125, and a light-emitting element EL. Transistors MP121 to MP125 are P-type MOSFETs. The gate of transistor MP122 is connected to the control line WSL, one of its source and drain is connected to the signal line 14b, and the other of its source and drain is connected to the gate of transistor MP121 and the other end of capacitor C132. One end of capacitor C132 is connected to the power line Vel, and the other end is connected to the other of its source and drain of transistor MP122 and the gate of transistor MP121. The gate of transistor MP121 is connected to the other of its source and drain of transistor MP122 and the other end of capacitor C132, one of its source and drain is connected to the power line Vel, and the other of its source and drain is connected to one of its source and drain of transistors MP123 and MP124. The gate of transistor MP123 is connected to the control line AZSL, one of its source and drain is connected to the other of the source and drain of transistor MP121, and one of the source and drain of transistor MP124, and the other of its source and drain is connected to the signal line 14b. The gate of transistor MP124 is connected to the control line DSL, one of its source and drain is connected to the other of the source and drain of transistor MP121, and one of the source and drain of transistor MP123, and the other of its source and drain is connected to one of the source and drain of transistor MP125 and the anode of the light-emitting element 130. The gate of transistor MP125 is connected to the control line AZSL, the other of its source and drain is connected to the power line Vorst, and one of its source and drain is connected to the other of the source and drain of transistor MP124 and the anode of the light-emitting element 130.

[0166] In this configuration, in a pixel PIX, when transistor MP122 is turned ON, the voltage across capacitor C132 is set based on the pixel signal supplied to one end of transmission gate TG45. Transistor MP124 is turned ON or OFF based on the signal on control line DSL. Transistor MP121 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C132 during the period when transistor MP124 is ON. The light-emitting element EL emits light based on the current supplied from transistor MP121. In this way, the pixel PIX emits light with brightness corresponding to the pixel signal. Transistors MP123 and MP125 are turned ON or OFF based on the signal on control line AZSL. During the period when transistor MP123 is ON, the other of the source and drain of transistor MP121 and one of the source and drain of transistor MP124 are connected to signal line 14b. During the period when transistor MP125 is ON, the voltage of the anode of the light-emitting element EL is initialized by setting it to the voltage of power line Vorst. Furthermore, transistor MP56 is switched on and off based on the signal of control line INIL, transistor MP57 is switched on and off based on the signal of control line ELL, and transistor MP73 is switched on and off based on the signal of control line REFL. When transistor MP56 is turned on, signal line 14b is set to the voltage of power line Vini, and when transistor MP57 is turned on, signal line 14b is set to the voltage of power line Vel. When transistor MP73 is turned on, one end of capacitor C82 is initialized by being set to the voltage of power line Vref.

[0167] Furthermore, transistors MP121 to MP125, MP56, and MP57 may be transistors using low-temperature polycrystalline silicon (LTPS). Also, at least one of transistors MP122 and MP125 may be a transistor using an oxide semiconductor.

[0168] (Seventh example)

[0169] Figure 43 shows another example of a pixel PIX configuration. This pixel PIX includes a capacitor C51, transistors MP52 to MP60, and a light-emitting element EL. Transistors MP52 to MP60 are P-type MOSFETs. The gate of transistor MP52 is connected to the control line WSL, one of its source and drain is connected to the signal line SGL, and the other of its source and drain is connected to the other of its source and drain of transistor MP53 and one of its source and drain of transistor MP54. The gate of transistor MP53 is connected to the control line DSL, one of its source and drain is connected to the power line VCCP, and the other of its source and drain is connected to the other of its source and drain of transistor MP52 and one of its source and drain of transistor MP54. The gate of transistor MP54 is connected to one of the source and drain of transistor MP55, the other of the source and drain of transistor MP57, and the other end of capacitor C51. One of the source and drain is connected to the other of the source and drain of transistors MP52 and MP53, and the other of the source and drain is connected to one of the source and drain of transistors MP58 and MP59. One end of capacitor C51 is connected to the power line VCCP, and the other end is connected to the gate of transistor MP54, one of the source and drain of transistor MP55, and the other of the source and drain of transistor MP57. Capacitor C51 may include two capacitors connected in parallel with each other. The gate of transistor MP55 is connected to the control line AZSL1, one of the source and drain is connected to the gate of transistor MP54, the other of the source and drain of transistor MP57, and the other end of capacitor C51, and the other of the source and drain is connected to one of the source and drain of transistor MP56. The gate of transistor MP56 is connected to the control line AZSL1, one of its source and drain is connected to the other of its source and drain, and the other of its source and drain is connected to the power line VSS.The gate of transistor MP57 is connected to the control line WSL, the other of its source and drain is connected to the gate of transistor MP54, one of the source and drain of transistor MP55, and the other end of capacitor C51, and one of its source and drain is connected to the other of its source and drain of transistor MP58. The gate of transistor MP58 is connected to the control line WSL, the other of its source and drain is connected to one of the source and drain of transistor MP57, the other of its source and drain is connected to the other of its source and drain of transistor MP54, and one of its source and drain is connected to one of its source and drain of transistor MP59. The gate of transistor 59 is connected to the control line DSL, the other of its source and drain is connected to the other of its source and drain of transistor MP54, and one of its source and drain is connected to one of its source and drain of transistor MP58, and the other of its source and drain is connected to one of its source and drain of transistor MP60, and the anode of the light-emitting element EL. The gate of transistor MP60 is connected to the control line AZSL2, one of its source and drain is connected to the other of its source and drain of transistor MP59 and to the anode of the light-emitting element EL, and the other of its source and drain is connected to the power line VSS.

[0170] In this configuration, in the pixel PIX, the voltage across capacitor C51 is set based on the pixel signal supplied from signal line SGL when transistors MP52, MP54, MP58, and MP57 are turned ON. Transistors MP53 and MP59 are turned ON and OFF based on the signal from control line DSL. Transistor MP54 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C51 while transistors MP53 and MP59 are ON. The light-emitting element EL emits light based on the current supplied from transistor MP54. In this way, the pixel PIX emits light with brightness corresponding to the pixel signal. Transistors MP55 and MP56 are turned ON and OFF based on the signal from control line AZSL1. While transistors MP55 and MP56 are ON, the gate voltage of transistor MP54 is initialized by setting it to the voltage of power line VSS. Transistor MP60 is turned ON and OFF based on the signal from control line AZSL2. During the period when transistor MP60 is ON, the voltage of the anode of the light-emitting element EL is initialized by setting it to the voltage of the power line VSS.

[0171] Transistors MP52 to MP60 may be transistors using low-temperature polycrystalline silicon (LTPS). Also, at least one of transistors MP55 to MP58 and MP60 may be a transistor using an oxide semiconductor.

[0172] (Example 8)

[0173] Figure 44 shows another example of a pixel PIX configuration. The signals of control line WSNL and control line WSPL are inverted signals of each other.

[0174] Each pixel PIX includes capacitors C61 and C62, transistors MN63, MP64, MN65-MN67, and light-emitting element EL. Transistors MN63 and MN65-MN67 are N-type MOSFETs, and transistor MP64 is a P-type MOSFET. The gate of transistor MN63 is connected to the control line WSNL, and the other of its source and drain is connected to the signal line SGL and one of the source and drain of transistor MP64. The other of its source and drain is connected to the other of the source and drain of transistor MP64, one end of capacitors C61 and C62, and the gate of transistor MN65. The gate of transistor MP64 is connected to the control line WSPL, and the other of its source and drain is connected to the signal line SGL and the other of the source and drain of transistor MN63. The other of its source and drain is connected to one of the source and drain of transistor MN63, one end of capacitors C61 and C62, and the gate of transistor MN65. Capacitor C61 is constructed using, for example, a MO (Metal Oxide Metal) capacitor, with one end connected to one of the source and drain of transistor MN63, the other of the source and drain of transistor MP64, one end of capacitor C62, and the gate of transistor MN65, and the other end connected to the power line VSS2. Capacitor C61 may also be constructed using, for example, a MOS capacitor or a MIM (Metal Insulator Metal) capacitor. Capacitor C62 is constructed using, for example, a MOS capacitor, with one end connected to one of the source and drain of transistor MN63, the other of the source and drain of transistor MP64, one end of capacitor C61, and the gate of transistor MN65, and the other end connected to the power line VSS2. Capacitor C62 may also be constructed using, for example, a MO capacitor or a MIM capacitor. The other end of capacitor C62 may also be connected to the power line VSS3 (not shown).The gate of transistor MN65 is connected to one of the source and drain of transistor MN63, the other of the source and drain of transistor MP64, and one end of capacitors C61 and C62. The other of the source and drain is connected to the power line VCCP, and the other of the source and drain is connected to the other of the source and drain of transistors MN66 and MN67. The gate of transistor MN66 is connected to the control line AZL, and the other of the source and drain is connected to one of the source and drain of transistor MN65 and the other of the source and drain of transistor MN67, and the other of the source and drain is connected to the power line VSS1. The gate of transistor MN67 is connected to the control line DSL, and the other of the source and drain is connected to one of the source and drain of transistor MN65 and the other of the source and drain of transistor MN66, and the other of the source and drain is connected to the anode of the light-emitting element EL. Alternatively, transistor MN67 and control line DSL may be omitted, and one of the source and drain of transistor MN65 may be connected to the other of the source and drain of transistor MN66, and to the anode of the light-emitting element EL.

[0175] In this configuration, at least one of transistors MN63 and MP64 is turned on in the pixel PIX, setting the voltage across capacitors C61 and C62 based on the pixel signal supplied from signal line SGL. Transistor MN67 is turned on and off based on the signal from control line DSL. Transistor MN65 supplies a current to the light-emitting element EL corresponding to the voltage across capacitors C61 and C62 during the period when transistor MN67 is on. The light-emitting element EL emits light based on the current supplied from transistor MP65. In this way, the pixel PIX emits light with a brightness corresponding to the pixel signal. Transistor MN66 may be turned on and off based on the signal from control line AZL. Transistor MN66 may also function as a resistive element having a resistance value corresponding to the signal from control line AZL. In this case, transistors MN65 and MN66 constitute a so-called source follower circuit.

[0176] Transistors MN63, MP64, and MN65-MN67 may be transistors using low-temperature polycrystalline silicon (LTPS). Furthermore, at least one of transistors MN63, MP64, and MN66 may be a transistor using an oxide semiconductor.

[0177] (Specific example #9)

[0178] Figure 45 shows another example of a pixel PIX configuration. This pixel PIX includes a capacitor C71, transistors MN72 to MN77, and a light-emitting element EL. Transistors MN72 to MN77 are N-type MOSFETs. The gate of transistor MN72 is connected to the control line WSL, and the other of its source and drain is connected to the signal line SGL. One of its source and drain is connected to one of the source and drain of transistor MN74 and the other of the source and drain of transistor MN75. One end of capacitor C71 is connected to the gate of transistor MN74 and one of the source and drain of transistor MN76, and the other end is connected to the other of the source and drain of transistor MN77, one of the source and drain of transistor MN75, and the anode of the light-emitting element EL. The gate of transistor MN73 is connected to control line DLS1, the other of its source and drain is connected to power line VCCP, and one of its source and drain is connected to the other of its source and drain of transistor MN74 and the other of its source and drain of transistor MN76. The gate of transistor MN74 is connected to one of its source and drain of transistor MN76 and one end of capacitor C71, the other of its source and drain is connected to one of its source and drain of transistor 73 and the other of its source and drain of transistor MN76, and one of its source and drain is connected to one of its source and drain of transistor MN72 and the other of its source and drain of transistor MN75. The gate of transistor MN75 is connected to control line DSL2, the other of its source and drain is connected to one of its source and drain of transistor MN72 and the other of its source and drain of transistor MN74, and one of its source and drain is connected to the other end of capacitor C71, the other of its source and drain of transistor MN77 and the anode of light-emitting element EL.The gate of transistor MN76 is connected to the control line AZSL, and the other of its source and drain is connected to one of the source and drain of transistor MN73 and the other of the source and drain of transistor MN74, with one of its source and drain connected to the gate of transistor MN74 and one end of capacitor C71. The gate of transistor MN77 is connected to the control line AZSL, and the other of its source and drain is connected to the other end of capacitor C71, one of the source and drain of transistor MN75 and the anode of light-emitting element EL, with one of its source and drain connected to the power line VSS.

[0179] In this configuration, in the pixel PIX, the voltage across capacitor C71 is set based on the pixel signal supplied from signal line SGL when transistors MN72, MN74, and MN76 are turned on. Transistor MN73 is turned on and off based on the signal from control line DSL1, and transistor MN75 is turned on and off based on the signal from control line DSL2. Transistor MN74 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C71 while transistors MN73 and MN75 are turned on. The light-emitting element EL emits light based on the current supplied from transistor MN74. In this way, the pixel PIX emits light with brightness corresponding to the pixel signal. Transistor MN77 is turned on and off based on the signal from control line AZSL. While transistor MN77 is turned on, the voltage of the anode of the light-emitting element EL is initialized by being set to the voltage of power line VSS.

[0180] Transistors MN72 to MN77 may be transistors using low-temperature polycrystalline silicon (LTPS). Transistor MN76 may be a transistor using an oxide semiconductor. <2. Application Examples> Next, application examples of the display system described in the above embodiments and modifications will be explained.

[0181] (Application Example 1) Figure 46 shows an example of the appearance of a head-mounted display 1100. The head-mounted display 1100 has, for example, a glasses-shaped display unit 1110 and ear hooks 1120 on both sides for attachment to the user's head. The technology according to the above embodiment can be applied to such a head-mounted display 1100.

[0182] (Application Example 2) Figure 47 shows an example of the appearance of another head-mounted display 1200. The head-mounted display 1200 is a transmissive head-mounted display having a main body 121, an arm 122, and a lens barrel 123. This head-mounted display 1200 is attached to eyeglasses 128. The main body 121 has a control board and a display unit for controlling the operation of the head-mounted display 1200. This display unit emits image light of the displayed image. The arm 122 connects the main body 121 and the lens barrel 1230 and supports the lens barrel 1230. The lens barrel 123 projects the image light supplied from the main body 1200 via the arm 122 towards the user's eyes through the lens 129 of the eyeglasses 128. The technology according to the above embodiment can be applied to such a head-mounted display 1200.

[0183] This head-mounted display 1200 is a so-called light guide plate type head-mounted display, but is not limited to this; for example, it may be a so-called birdbath type head-mounted display. This birdbath type head-mounted display includes, for example, a beam splitter and a partially transparent mirror. The beam splitter outputs light encoded with image information toward the mirror, and the mirror reflects the light toward the user's eyes. Both the beam splitter and the partially transparent mirror are partially transparent. This allows light from the surrounding environment to reach the user's eyes.

[0184] (Application Example 3) Figures 48A and 48B show an example of the external appearance of the digital still camera 1300, with Figure 48A showing a front view and Figure 48B showing a rear view. This digital still camera 1300 is a single-lens reflex type camera with interchangeable lenses and has a camera body 131, an imaging lens unit 132, a grip 133, a monitor 134, and an electronic viewfinder 135. The imaging lens unit 312 is an interchangeable lens unit and is located near the center of the front of the camera body 311. The grip 133 is located on the left side of the front of the camera body 311, and the photographer holds this grip 133. The monitor 134 is located to the left of the center of the rear of the camera body 131. The electronic viewfinder 135 is located on the rear of the camera body 131, above the monitor 14. The photographer can look through the electronic viewfinder 135 to see the light image of the subject guided by the shooting lens unit 132 and determine the composition. The technology according to the above embodiment can be applied to the electronic viewfinder 135.

[0185] (Application Example 4) Figure 49 shows an example of the appearance of a smartphone 150. The smartphone 150 has a display unit 151 that displays various information and an operation unit 152 that includes buttons and the like for receiving user input. The technology according to the above embodiment can be applied to this display unit 151.

[0186] (Application Example 5) Figures 50A and 50B show an example of a vehicle configuration to which the technology of this disclosure is applied. Figure 50A shows an example of the interior of the vehicle as seen from the rear, and Figure 50B shows an example of the interior of the vehicle as seen from the left rear.

[0187] The vehicles in Figures 50A and 50B have a center display 201, a console display 202, a head-up display 203, a digital rear mirror 204, a steering wheel display 205, and a rear entertainment display 206.

[0188] The center display 201 is positioned on the dashboard 261, facing the driver's seat 262 and the passenger seat 263. Figure 51A shows an example of a horizontally elongated center display 201 extending from the driver's seat 262 to the passenger seat 263, but the screen size and placement of the center display 2010 are not limited to this. The center display 2010 can display information detected by various sensors. As a specific example, the center display 201 can display images captured by an image sensor, distance images to obstacles in front of and to the side of the vehicle measured by a ToF sensor, and the body temperature of occupants detected by an infrared sensor. The center display 201 can be used to display at least one of the following: safety-related information, operation-related information, life logs, health-related information, authentication / identification-related information, and entertainment-related information.

[0189] Safety-related information includes data based on sensor detection results, such as drowsiness detection, distraction detection, detection of mischief by passengers, seatbelt usage status, and detection of unattended occupants. Operation-related information is information on occupant gestures detected using sensors. Gestures may include operation of various in-vehicle equipment, such as air conditioning, navigation systems, AV (Audio Visual) systems, and lighting systems. Lifelogs include the lifelogs of all occupants. For example, lifelogs include records of each occupant's actions. By acquiring and saving lifelogs, it is possible to check the state of the occupants in the event of an accident. Health-related information includes the occupant's body temperature detected using temperature sensors, and information on the occupant's health status inferred from the detected body temperature. Alternatively, information on the occupant's health status may be inferred based on the occupant's face captured by an image sensor. Furthermore, information on the occupant's health status may be inferred based on the occupant's responses obtained by conversing with the occupant using automated voice. Authentication / identification-related information includes information such as keyless entry functions that use sensors for facial recognition and functions that automatically adjust seat height and position based on facial recognition. Entertainment-related information includes information on AV equipment operation by occupants detected by sensors, and information on content to be displayed that is appropriate for occupants detected and recognized by sensors.

[0190] The console display 202 can be used, for example, to display life log information. The console display 202 is located near the shift lever 265 in the center console 264 between the driver's seat 262 and the passenger seat 263. The console display 202 can also display information detected by various sensors. In addition, the console display 202 may display images of the area around the vehicle captured by an image sensor, or it may display distance images to obstacles around the vehicle.

[0191] The head-up display 203 is virtually displayed behind the windshield 266 in front of the driver's seat 262. The head-up display 203 can be used to display, for example, at least one of safety-related information, operation-related information, life logs, health-related information, authentication / identification-related information, and entertainment-related information. Because the head-up display 203 is often virtually positioned in front of the driver's seat 262, it is suitable for displaying information directly related to vehicle operation, such as vehicle speed, fuel level, and battery level.

[0192] The digital rearview mirror 204 can not only display the area behind the vehicle, but also show the condition of the rear-seat passengers. Therefore, it can be used, for example, to display life log information of rear-seat passengers.

[0193] The steering wheel display 205 is positioned near the center of the vehicle's steering wheel 267. The steering wheel display 205 can be used to display at least one of the following: safety-related information, operation-related information, life log, health-related information, authentication / identification-related information, and entertainment-related information. In particular, because the steering wheel display 205 is located near the driver's hands, it is suitable for displaying life log information such as the driver's body temperature, or information related to the operation of AV equipment, air conditioning equipment, etc.

[0194] The rear entertainment display 206 is mounted on the back of the driver's seat 262 and the passenger seat 263, and is intended for viewing by rear-seat passengers. The rear entertainment display 206 can be used to display at least one of the following: safety-related information, operation-related information, life logs, health-related information, authentication / identification-related information, and entertainment-related information. In particular, because the rear entertainment display 206 is in front of the rear-seat passengers, it displays information relevant to the rear-seat passengers. For example, the rear entertainment display 206 may display information related to the operation of AV equipment or air conditioning equipment, or it may display the results of measurements of the rear-seat passenger's body temperature, etc., taken by the temperature sensor 5.

[0195] The technologies described in the above embodiments can be applied to these center displays 2010, console displays 202, head-up displays 203, digital rear mirrors 204, steering wheel displays 205, and rear entertainment displays 206.

[0196] Furthermore, this technology can take the following configuration.

[0197] (1) A control circuit comprising: an acquisition unit for acquiring first image data; an image processing unit capable of performing at least one of a first process for generating second pixel values ​​using the pixel values ​​of a plurality of first pixels in the first image data, and a second process for increasing resolution by maintaining the pixel values ​​of the first pixels and dividing them into a plurality of pixels, and which performs at least the first process to generate second image data having the second pixel values; and a display control unit for displaying an image based on the second image data on a display unit.

[0198] (2) The control circuit according to (1), wherein the size of the pixel having the second pixel value is the same as or different from that of the first pixel.

[0199] (3) The control circuit according to (1), wherein the first process is at least one of a high-resolution process and a jagged edge suppression process.

[0200] (4) The control circuit according to (1), wherein the first process generates the second pixel value by performing a convolution operation by multiplying the pixel value in the filter range by a coefficient.

[0201] (5) The acquisition unit acquires information of a first region relating to the viewpoint, the first region of the first image data has a first resolution, the second region of the first image data excluding the first region has a second resolution lower than the first resolution, and the image processing unit converts the image data in the second region to the first resolution, as described in (4).

[0202] (6) The control circuit according to (5), wherein the image processing unit is capable of performing at least one of a high-resolution processing and a jagged edge suppression processing, and when the high-resolution processing and the jagged edge suppression processing are performed simultaneously, the filter range for the high-resolution processing and the jagged edge suppression processing in the second region is set to be wider than the filter range for performing only the high-resolution processing in the second region.

[0203] (7) The control circuit according to (1), wherein the first region of the first image data has a first resolution, the second region of the first image data excluding the first region has a second resolution lower than the first resolution, and the image processing unit performs the first processing on the first region and the second processing on the second region.

[0204] (8) The control circuit according to (4), wherein the acquisition unit is capable of further acquiring first depth data having depth information corresponding to the first image data, and the image processing unit changes at least one of the filter range and the distribution of the coefficients based on the depth information.

[0205] (9) The control circuit according to (8), wherein the first region of the first image data has a first resolution, the second region of the first image data excluding the first region has a second resolution lower than the first resolution, the acquisition unit acquires the first image data of the first region having a first resolution and the second region having a second resolution, the third region having a third resolution corresponding to the first region and the fourth region having a fourth resolution lower than the third region, and the image processing unit performs a high-resolution processing and a jagged edge suppression processing based on the first depth data.

[0206] (10) The control circuit according to (9), wherein the number of bits of the pixel values ​​of the first depth data is smaller than the number of bits of the pixel values ​​of the first image data.

[0207] (11) The control circuit according to (10), wherein all pixels of the first depth data are composed of the third resolution.

[0208] (12) The control circuit according to (11), wherein the first image data is either RGB image data or YCbCr image data, and in the case of YCbCr image data, the number of bits in the pixel values ​​of the CbCr image data is smaller than the number of bits in the pixel values ​​of the Y image data.

[0209] (13) The control circuit according to (12), wherein the data of the first region of the first image data is image data from which the jaggedness suppression process has been performed, and the first depth data consists only of the data of the fourth region.

[0210] (14) The control circuit according to (13), wherein when the image processing unit performs the high-resolution processing and the jagged edge suppression processing, it reduces the resolution of a peripheral region away from the first region to less than the first resolution, and increases only the resolution of the image data of the reduced region by region division.

[0211] (15) The control circuit according to (13), wherein the data of the fifth region, which is a peripheral region of the first image data that is far from the first region, has been subjected to jaggedness suppression processing, and the first depth data consists of data excluding the region corresponding to the fifth region.

[0212] (16) The control circuit according to (8), wherein the first region of the first image data has a first resolution, the second region of the first image data excluding the first region has a second resolution lower than the first resolution, the image processing unit performs a high-resolution processing and a jagged edge suppression processing on the first region, and increases only the resolution of the image data excluding the first region by region division.

[0213] (17) The control circuit according to (1), wherein the image processing unit is capable of performing processing using a neural network.

[0214] (18) The control circuit according to any one of (1) to (17), wherein the first image data is image data divided into vertical strips.

[0215] (19) The control circuit according to any one of (1) to (17), wherein the acquisition unit acquires information relating to the viewpoint position, the image processing unit performs a convolution operation using coefficients, and changes the degree of frequency reduction of the convolution operation based on at least one of the distance from the viewpoint position, the amount of jaggedness suppression, and the resolution.

[0216] (20) A display device comprising: a control circuit; a pixel array section having a plurality of pixel circuits arranged in the horizontal and vertical directions; and a light emission control section that writes a signal voltage to the pixel circuits and performs light emission control based on second image data generated by the control circuit, and is also capable of simultaneously writing one pixel value to n (where n is a natural number of 2 or more, including 4) of the pixel circuits, wherein the control circuit is capable of performing at least one of a first process that generates a second pixel value using the pixel values ​​of a plurality of first pixels in the first image data, and a second process that increases the resolution by maintaining the pixel value of the first pixel and dividing it into a plurality of pixels, and performs at least the first process and generates the second image data having the second pixel value; and a display control section that causes the pixel array section to display an image based on the second image data via the light emission control section.

[0217] (21) A display system comprising: an image generation device for generating at least first image data; and a control circuit, wherein the image generation device generates first image data of a region and a resolution based on the region according to the degree of gaze based on information of the gaze point; the control circuit is capable of performing at least one of the following: an acquisition unit for acquiring the first image data having information of the region; a first process for generating a second pixel value using the pixel values ​​of a plurality of first pixels in the first image data; and a second process for increasing the resolution by maintaining the pixel value of the first pixel and dividing it into a plurality of pixels; an image processing unit for performing at least the first process and generating second image data having the second pixel value; and a display control unit for displaying an image based on the second image data on a display unit.

[0218] The aspects of this disclosure are not limited to the individual embodiments described above, but include various modifications that a person skilled in the art could conceive, and the effects of this disclosure are not limited to those described above. In other words, various additions, modifications, and partial deletions are possible, as long as they do not depart from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents.

[0219] 1: Display system, 60: Point of gaze detection unit, 100: Image generation device, 102: Region determination unit, 104: Gaze degree determination unit, 106: Image generation unit, 200: Display device, 212: Video signal conversion unit, 214: Image processing unit, 216: Calculation processing unit, 218: Resolution adjustment unit, 220: Display unit, T100: Measurement time, Z1: First region, Z2: Second region

Claims

1. A control circuit comprising: an acquisition unit for acquiring first image data; an image processing unit capable of performing at least one of a first process for generating second pixel values ​​using the pixel values ​​of a plurality of first pixels in the first image data, and a second process for increasing resolution by maintaining the pixel values ​​of the first pixels and dividing them into a plurality of pixels, and which performs at least the first process to generate second image data having the second pixel values; and a display control unit for displaying an image based on the second image data on a display unit.

2. The control circuit according to claim 1, wherein the size of the pixel having the second pixel value is the same as or different from that of the first pixel.

3. The control circuit according to claim 1, wherein the first process is at least one of a high-resolution process and a jagged edge suppression process.

4. The control circuit according to claim 1, wherein the first process generates the second pixel value by performing a convolution operation by multiplying the pixel value in the filter range by a coefficient.

5. The control circuit according to claim 4, wherein the acquisition unit acquires information of a first region relating to the viewpoint, the first region of the first image data has a first resolution, the second region of the first image data excluding the first region has a second resolution lower than the first resolution, and the image processing unit converts the image data in the second region to the first resolution.

6. The control circuit according to claim 5, wherein the image processing unit is capable of performing at least one of a high-resolution processing and a jagged edge suppression processing, and when the high-resolution processing and the jagged edge suppression processing are performed simultaneously, the filter range for the high-resolution processing and the jagged edge suppression processing in the second region is set to be wider than the filter range for performing only the high-resolution processing in the second region.

7. The control circuit according to claim 1, wherein the first region of the first image data has a first resolution, the second region of the first image data excluding the first region has a second resolution lower than the first resolution, and the image processing unit performs the first processing on the first region and the second processing on the second region.

8. The control circuit according to claim 4, wherein the acquisition unit is capable of further acquiring first depth data having depth information corresponding to the first image data, and the image processing unit changes at least one of the filter range and the distribution of the coefficients based on the depth information.

9. The control circuit according to claim 8, wherein the first region of the first image data has a first resolution, the second region of the first image data excluding the first region has a second resolution lower than the first resolution, the acquisition unit acquires the first image data of the first region having a first resolution and the second region having a second resolution, the third region having a third resolution corresponding to the first region and the fourth region having a fourth resolution lower than the third region, and the image processing unit performs a high-resolution processing and a jagged edge suppression processing based on the first depth data.

10. The control circuit according to claim 9, wherein the number of bits of the pixel values ​​of the first depth data is smaller than the number of bits of the pixel values ​​of the first image data.

11. The control circuit according to claim 10, wherein all pixels of the first depth data are composed of the third resolution.

12. The control circuit according to claim 11, wherein the first image data is either RGB image data or YCbCr image data, and in the case of YCbCr image data, the number of bits in the pixel values ​​of the CbCr image data is smaller than the number of bits in the pixel values ​​of the Y image data.

13. The control circuit according to claim 12, wherein the data of the first region of the first image data is image data from which the jaggedness suppression process has been performed, and the first depth data consists only of the data of the fourth region.

14. The control circuit according to claim 13, wherein, when the image processing unit performs the high-resolution processing and the jagged edge suppression processing, it reduces the resolution of a peripheral region away from the first region to less than the first resolution, and increases only the resolution of the image data of the reduced region by region division.

15. The control circuit according to claim 13, wherein the data of a fifth region, which is a peripheral region of the first image data that is far from the first region, has been subjected to jaggedness suppression processing, and the first depth data consists of data excluding the region corresponding to the fifth region.

16. The control circuit according to claim 8, wherein the first region of the first image data has a first resolution, the second region of the first image data excluding the first region has a second resolution lower than the first resolution, the image processing unit performs a high-resolution processing and a jagged edge suppression processing on the first region, and increases only the resolution of the image data excluding the first region by region division.

17. The control circuit according to claim 1, wherein the image processing unit is capable of performing processing using a neural network.

18. The control circuit according to claim 1, wherein the first image data is image data divided into vertical strips.

19. The control circuit according to claim 1, wherein the acquisition unit acquires information regarding the viewpoint position, the image processing unit performs a convolution operation using coefficients, and the degree of frequency reduction of the convolution operation is changed based on at least one of the distance from the viewpoint position, the amount of jaggedness suppression, and the resolution.

20. A display device comprising: a control circuit; a pixel array section having a plurality of pixel circuits arranged horizontally and vertically; and a light emission control section that writes a signal voltage to the pixel circuits and performs light emission control based on second image data generated by the control circuit, and is also capable of simultaneously writing one pixel value to n (where n is a natural number of 2 or more, including 4) of the pixel circuits, wherein the control circuit is capable of performing at least one of the following: an acquisition section that acquires first image data; an image processing section that performs at least the first processing and generates the second image data having the second pixel values; and a display control section that causes the pixel array section to display an image based on the second image data via the light emission control section.

21. A display system comprising: an image generation device for generating at least first image data; and a control circuit, wherein the image generation device generates first image data of a region and a resolution based on the region, according to the degree of gaze, based on information of the gaze point; the control circuit comprises: an acquisition unit for acquiring the first image data having information of the region; an image processing unit capable of performing at least one of a first process for generating a second pixel value using the pixel values ​​of a plurality of first pixels in the first image data, and a second process for increasing the resolution by maintaining the pixel value of the first pixel and dividing it into a plurality of pixels; and an image processing unit that performs at least the first process and generates second image data having the second pixel value; and a display control unit for displaying an image based on the second image data on a display unit.

Images