Display control unit, and on-vehicle display system

Abstract

To achieve a display control unit and a display system that achieve both visibility of an image and power saving without using a sensor that detects movements of user's eyes.SOLUTION: A display control unit and a display system comprise: a saliency map generation unit 12c that generates a saliency map on the basis of image data to be displayed on a display 20, and a luminance control unit 12d that executes luminance control on the basis of the saliency map. The luminance control unit 12d executes the luminance control of determining the luminance level of an image according to a gaze frequency on the basis of the saliency map.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a display control device and an in-vehicle display system that perform brightness control of a display. [Background technology]

[0002] Conventionally, there is known a display device that achieves power saving by controlling the brightness of the display (for example, Patent Document 1). The display device described in Patent Document 1 includes a backlight section in which an image display section is divided into a matrix, a backlight driving section that controls the backlight to be turned on and off for each divided matrix element, and a viewpoint detection section that detects the movement of the user's eyes. This display device achieves a power saving effect by selectively lighting only the matrix elements that correspond to the viewpoint position and the vicinity of the viewpoint position on the LCD section that are estimated from the detected eye movement, out of all the matrix elements that make up the backlight section. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-209610 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, in the field of in-vehicle display devices, detailed graphics have come to be used, and from the viewpoint of improving display quality, devices in which pixels are made up of self-luminous elements such as organic light-emitting diodes (OLEDs) are seen as promising. When a self-luminous display is used, it is considered that a display device that can achieve both high display quality and power saving can be obtained. However, the display device described in Patent Document 1 requires a viewpoint detection unit that detects the movement of the user's eyes.

[0005] In view of the above, the present disclosure aims to provide a display control device and an in-vehicle display system that can reduce power consumption by reducing the brightness of areas that are viewed infrequently by the user, without using a sensor that detects the user's eye movements. [Means for solving the problem]

[0006] A display control device according to one aspect of the present disclosure is a display control device that performs brightness control of a display (20), and includes a saliency map generation unit (12c) that generates a saliency map based on image data to be displayed on the display, and a brightness control unit (12d) that performs brightness control of the display based on the gaze frequency in the saliency map.

[0007] This display control device calculates the frequency of gazes in the display area based on a saliency map generated from image data to be displayed on the display, and performs brightness control according to the frequency of gazes. This eliminates the need for a sensor to detect the movement of the user's eyes, and enables power saving by performing brightness control according to the frequency of gazes and turning off or dimming areas that are viewed less than a certain frequency.

[0008] An in-vehicle display system according to one aspect of the present disclosure comprises a display (20) mounted on a vehicle, a saliency map generation unit (12c) that generates a saliency map based on image data to be displayed on the display, and a brightness control unit (12d) that performs brightness control of the display in accordance with the frequency of gaze on the saliency map.

[0009] This in-vehicle display system calculates the frequency of gazes in the display area based on a saliency map using image data for the display mounted on the vehicle, and controls the brightness according to the frequency of gazes. This eliminates the need for a sensor to detect the movement of the user's eyes, and enables the brightness control to be performed according to the frequency of gazes, making it possible to turn off or dim areas that are viewed less than a certain frequency, thereby enabling power saving.

[0010] The reference symbols in parentheses attached to each component indicate an example of the correspondence between the component and specific components described in the embodiments described below. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a display system according to an embodiment; [Diagram 2] FIG. 1 is a diagram illustrating an example of a display mounted on a vehicle. [Diagram 3] FIG. 1 illustrates an example of image-based saliency map generation. [Figure 4] 10 is a flowchart showing an example of a control flow in a first luminance control. [Diagram 5] FIG. 11 is an explanatory diagram of a first luminance control based on a saliency map. [Figure 6] 10 is a flowchart showing an example of a control flow in a second luminance control. [Figure 7] FIG. 11 is an explanatory diagram of a second luminance control based on a saliency map. [Figure 8] 13 is a flowchart showing an example of a control flow in a third luminance control. [Figure 9] FIG. 11 is an explanatory diagram of a third luminance control based on a saliency map. [Figure 10] 13 is a flowchart showing an example of a control flow in a fourth luminance control. [Figure 11] 13 is a flowchart showing an example of a control flow in a fifth luminance control. [Figure 12] FIG. 13 is a block diagram showing another example configuration of a display system that executes the fifth luminance control. [Figure 13] 13 is a flowchart showing an example of a control flow in a sixth luminance control. [Figure 14] FIG. 13 is an explanatory diagram for determining a mask area in the sixth brightness control. [Figure 15]FIG. 13 is an explanatory diagram of a sixth brightness control based on a saliency map and a mask area. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following embodiments, parts that are the same or equivalent to each other will be denoted by the same reference numerals.

[0013] (Embodiment) A display system to which a display control device according to an embodiment is applied will be described. In this embodiment, a case in which the display system is an in-vehicle display system mounted on a vehicle such as an automobile will be described as a representative example, but the display system can of course be applied to other uses. Hereinafter, the vehicle on which the display system is mounted will be referred to as the "own vehicle."

[0014] [Basic configuration] 1, the display system of this embodiment includes a drawing controller 10, a display 20, an in-vehicle device 30, and an in-vehicle LAN 40. The display system is configured such that, for example, various signals are input to the drawing controller 10 from the in-vehicle device 30 directly or via the in-vehicle LAN 40, and display control of the display 20 is performed based on these various signals. Note that LAN is an abbreviation for Local Area Network, and the in-vehicle LAN 40 may also be referred to as an in-vehicle communication bus.

[0015] The drawing controller 10 corresponds to a display control device that controls image display by the display 20, and controls image display on the display 20 based on various signals transmitted from the various in-vehicle devices 30. The drawing controller 10 is, for example, directly connected to the display 20 via a connection wire, and is connected to some or all of the various in-vehicle devices 30 via an in-vehicle LAN 40. The detailed configuration of the drawing controller 10 will be described later.

[0016] The display 20 is a self-luminous display having a plurality of pixels constituted by self-luminous elements such as OLED. The display 20 displays an image corresponding to a video output signal from the drawing controller 10. The display 20 is constituted by a pixel group consisting of a main pixel constituted by a plurality of sub-pixels having different luminous colors such as red (R), green (G), and blue (B). The display 20 is configured such that the luminance level can be adjusted by applying a variable driving voltage to each of the self-luminous elements constituting the display area for displaying an image, that is, to each of the pixels. For example, as shown in FIG. 2, the display 20 is mounted on the instrument panel of the vehicle V, and is formed into a large-area, approximately rectangular shape extending from the front of the passenger seat to the front of the driver's seat when the display area 20a is viewed from the front, but is not limited thereto. The display 20 can be formed into various shapes such as a square, a vertically elongated rectangle, a curved shape, and the arrangement in the vehicle V can be changed as appropriate. Furthermore, the display 20 may be configured with a plurality of display panels arranged side by side, or may be configured with a single large-area display panel.

[0017] In this specification, the display 20 will be described as an OLED display as a representative example, but since self-luminous displays such as OLED displays are well known, detailed description of the display itself will be omitted. In addition, although OLED will be described as a representative example of the self-luminous element, the self-luminous element is not limited to this, and other self-luminous elements such as inorganic EL and micro LED may be used.

[0018] Furthermore, the display 20 may be, for example, a liquid crystal display that uses a plurality of micrometer-sized micro LEDs as a backlight and can control the brightness for each minute area by local dimming, i.e., partial driving. In other words, the display 20 may be a display body that can control the brightness for each individual pixel or for each pixel group occupying an area of ​​a predetermined size or less. An example of an area of ​​a predetermined size or less is, but is not limited to, a 1 mm square size. When the above-mentioned liquid crystal display is used as the display 20, the first to sixth brightness controls described below are performed for each pixel group to which one of the multiple backlights is assigned.

[0019] The in-vehicle device 30 is configured to have a plurality of devices, such as a device for inputting a video signal to the drawing controller 10 and a device for inputting vehicle information to the drawing controller 10. In this specification, the in-vehicle device 30 will be described as a representative example including a navigation device 31, a multimedia device 32, an air conditioner (hereinafter referred to as "air conditioner") 33, and a vehicle ECU 34. ECU is an abbreviation for Electronic Control Unit. Note that the above configuration of the in-vehicle device 30 is merely an example, and some or all of the devices may be devices other than those shown here.

[0020] The navigation device 31 inputs video signals showing the current position of the vehicle and a map image to the drawing controller 10 based on map information stored in a map database. The navigation device 31 also inputs video signals showing, for example, an image for setting a destination and an image related to facility and store information around the vehicle or the destination to the drawing controller 10 based on a user's operation. Furthermore, the navigation device 31 obtains information related to the latitude, longitude, current time, and direction of the vehicle by using, for example, a known GPS, and inputs this information to the drawing controller 10. Note that GPS is an abbreviation for Global Positioning System.

[0021] The multimedia 32 can handle various types and formats of media, such as characters, images, videos, and audio, in a composite manner. Media is a device or medium for recording, transmitting, storing, etc., and in the case of a vehicle, it corresponds to a medium for providing images to be displayed, such as a television broadcasting station or a video site, or a recording medium such as a USB memory on which videos are recorded. USB is an abbreviation for Universal Serial Bus. The multimedia 32 inputs video signals from these media to the drawing controller 10 via wireless communication or a communication bus such as a USB cable.

[0022] The air conditioner 33 controls the air conditioning of the vehicle, and inputs various information related to the air conditioner control from an air conditioner ECU, which corresponds to the control unit, to the drawing controller 10. For example, a control signal corresponding to a mode selection of the air conditioner 33 is input to the drawing controller 10.

[0023] The vehicle ECU 34 acquires various information related to the host vehicle V and inputs the information to the drawing controller 10. For example, the vehicle ECU 34 inputs to the drawing controller 10 driving information related to the driving state of the host vehicle V, such as whether the host vehicle V is being driven automatically or manually, and vehicle information indicating various physical quantities applied to the host vehicle V, such as the vehicle speed, acceleration, and angular velocity of the host vehicle V.

[0024] As described above, in this embodiment, for example, latitude and longitude, current time, driving information, and vehicle information are input from the navigation device 31 to the drawing controller 10, but there are also cases where these various pieces of information are handled by the vehicle ECU 34. In such cases, these various pieces of information may be input from the vehicle ECU 34 to the drawing controller 10. In addition, the vehicle ECU 34 can acquire driving information from an output signal from a driving mode changeover switch (not shown) and can acquire vehicle information such as vehicle speed from a physical quantity sensor such as a vehicle speed sensor.

[0025] The above is the basic configuration of the display system.

[0026] [Drawing Controller] Next, the detailed configuration of the drawing controller 10 will be described.

[0027] As described above, the drawing controller 10 controls the image display by the display 20, and includes, as functional units for realizing this, for example, the communication unit 11 and the control unit 12.

[0028] The communication unit 11 is a part that acquires information from the in-vehicle device 30 through the in-vehicle LAN 40. Signals and information transmitted from the in-vehicle device 30 to the in-vehicle LAN 40 are acquired by the communication unit 11. In FIG. 1, for example, for the multimedia 32, an acquisition path through which a video signal can be directly input to the drawing controller 10 and an acquisition path through which the video signal can be input through the communication unit 11 are shown. In this way, the acquisition path of various information and video signals from the in-vehicle device 30 is arbitrary, and the various information and video signals may be directly input to the drawing controller 10 or may be input from the in-vehicle LAN 40 through the communication unit 11.

[0029] The control unit 12 is configured by, for example, a microcomputer equipped with a CPU, a ROM, a RAM, an I / O, and the like. CPU is an abbreviation for Central Processing Unit, ROM is an abbreviation for Read Only Memory, RAM is an abbreviation for Random Access Memory, and I / O is an abbreviation for Input / Output. The control unit 12 reads and executes various programs stored in a storage unit (not shown), for example, to output a display image on the display 20 based on a video signal and various information transmitted from the in-vehicle device 30. The control unit 12 is configured to include, for example, a drawing unit 12a, a video input / combining unit 12b, a saliency map generating unit 12c, a luminance control unit 12d, and a video output unit 12e.

[0030] The drawing unit 12a inputs, for example, various information related to air conditioner control transmitted from the air conditioner 33 and various information related to the vehicle V transmitted from the vehicle ECU 34, which are acquired by the communication unit 11, and generates image data for drawing the information on the display 20. The image data generated by the drawing unit 12a is output to the video input / combination unit 12b.

[0031] For example, the video input / combining unit 12b receives a video signal directly from the in-vehicle device 30 or via the in-vehicle LAN 40, generates image data indicated by the video signal, and outputs the image data to the saliency map generating unit 12c. In addition, when there is image data drawn by the drawing unit 12a, the video input / combining unit 12b inputs the image data and combines it with the image data indicated by the video signal, and outputs the combined image data to the saliency map generating unit 12c.

[0032] As shown in FIG. 3, for example, the saliency map generating unit 12c generates a saliency map SM based on the image data P1 generated by the video input / combining unit 12b, visualizing the frequency at which the user naturally directs their gaze in the image, that is, the distribution of gaze frequencies. The saliency map is described in, for example, "J. Harel, C. Koch, and P. Perona. Graph-Based Visual Saliency," NIPS 2006. The saliency map SM can be generated by a known method based on various characteristics of the image, such as brightness, color, orientation, and direction. For example, the image data P1 includes various contents C1 to C9 obtained by wireless communication with the in-vehicle device 30 or the outside, but the type, number, and arrangement of the contents can be changed as appropriate. The saliency map generating unit 12c generates a saliency map SM for each image data P1 generated by the video input / combining unit 12b as needed, based on the characteristics of the contents C1 to C9 included in the image data P1, such as color, brightness, and arrangement. The data of the saliency map SM generated by the saliency map generating unit 12c is output to, for example, the luminance control unit 12d and used for luminance control in the luminance control unit 12d.

[0033] In addition, in order to easily understand the difference in gaze frequency in the saliency map SM, in FIG. 3, although it does not show a cross section, different hatching is applied to each area with a different gaze frequency. Also, in FIG. 3, the higher the gaze frequency in the saliency map SM, the whiter the hatching, and the lower the gaze frequency, the blacker the hatching. This is the same for the saliency maps SM shown in FIG. 5, FIG. 7, FIG. 9, FIG. 14, and FIG. 15 described later. Also, the gaze frequency being "high" or "low" is, for example, a relative difference, and is determined by a predetermined threshold value set in advance for the calculated frequency.

[0034] The luminance control unit 12d performs signal output of the luminance level of the pixel group constituting the display 20 based on, for example, the image data generated by the video input / combining unit 12b and the saliency map generated by the saliency map generating unit 12c. For example, the luminance control unit 12d adjusts the luminance value and then outputs a control signal for the luminance value to the display 20. The luminance control unit 12d executes control to lower the luminance of an area that is not frequently gazed upon based on the saliency map. Details of this luminance control will be described later.

[0035] The video output unit 12e outputs a video output signal so that the image data transmitted from the video input / combination unit 12b is displayed on the display 20.

[0036] The basic configuration of the drawing controller 10 has been described above.

[0037] [First Brightness Control] Next, the first luminance control based on the saliency map will be described with reference to FIGS.

[0038] For convenience, in FIG. 5, the longitudinal direction of the saliency map SM, i.e., the direction along the left-right direction of the paper, is indicated as the "X direction", and the lateral direction of the saliency map SM perpendicular to the X direction is indicated as the "Y direction". In addition, in FIG. 5, the determination of the luminance level is shown as an example of a position "Y1" in the Y direction, which is a position along the X direction. The same applies to FIG. 7 and FIG. 9 described later. For example, in the case of an in-vehicle application, the X direction corresponds to the vehicle width direction.

[0039] The display system starts the control flow for brightness control shown in FIG. 4 when a predetermined start condition is satisfied, for example, when the ignition of the display 20 or the vehicle V is turned on.

[0040] In step S110, for example, the drawing controller 10 acquires an image signal from the in-vehicle device 30, and the drawing unit 12a and the image input / combining unit 12b generate image data.

[0041] Next, in step S120, for example, the saliency map generating unit 12c generates a saliency map based on the image data in step S110.

[0042] Next, in step S130, for example, the luminance control unit 12d determines the luminance level according to the distribution of gaze frequencies in the image based on the saliency map in step S120. The luminance control unit 12d determines the luminance level according to the gaze frequencies, for example, as shown in FIG. 5. Specifically, the luminance control unit 12d maintains the luminance level of the area with the highest gaze frequency, and determines a lower luminance level for the area with a lower gaze frequency. Note that FIG. 5 shows only the determination of the luminance level for the position Y1 in the Y direction, but in step S130, the luminance control unit 12d similarly determines the luminance level according to the gaze frequencies for other positions.

[0043] In the next step S140, for example, the luminance control unit 12d determines a gradation value of a sub-pixel (e.g., RGB) in each main pixel according to the luminance level determined in step S130, and outputs a signal corresponding to the determined gradation value to the video output unit 12e. Then, for example, the video output unit 12e outputs a signal corresponding to the image data after the luminance control in step S140 to the display 20. After the process of step S140, the control unit 12 returns the luminance control process to step S110, and repeats the above series of processes until a predetermined end condition is satisfied.

[0044] The above is an overview of the first brightness control. As a result, the display 20 displays an image in which an area that is frequently gazed upon is maintained in an easy-to-view state, and an area that is not frequently gazed upon is displayed in a lower brightness state. As a result, the display system can achieve both easy-to-view images on the display 20 and power saving without using a sensor that detects the movement of the user's eyes.

[0045] In the above luminance control, an example has been described in which the main pixel group displaying an area with a low gaze frequency among the main pixel groups constituting the display 20 is dimmed, but the present invention is not limited to this, and control to turn off the main pixel group may be performed. For example, in the luminance control, control may be performed to turn off the main pixel group displaying the area with the lowest gaze frequency or a plurality of areas with a gaze frequency of a predetermined frequency or less, that is, to display black with a luminance level of zero. Also, in FIG. 5, an example has been shown in which luminance control is performed on the entire area of ​​the wide horizontal display 20, but the present invention is not limited to this. For example, luminance control may be performed on a partial area of ​​the display 20, such as a part that functions as a center information display or a meter display that displays meters, etc. The above-mentioned turning off and luminance control of partial areas may be similarly applied to the second to sixth luminance controls described later.

[0046] [Second Brightness Control] Next, the second luminance control based on the saliency map will be described with reference to Fig. 6 and Fig. 7, but here, differences from the first luminance control will be mainly described. Note that in Fig. 7, a part of the luminance level determined by the luminance control when the lower limit value of the luminance gradient described later is not set is shown by a dashed line.

[0047] In the second luminance control, the drawing controller 10 executes, for example, the processes of steps S110 to S140 similar to those in the first luminance control, but executes the process of step S101 before step S110, as shown in FIG.

[0048] In step S101, for example, the luminance control unit 12d sets a lower limit value of the luminance gradient. The luminance gradient here means a gradient of the luminance level between regions with different gaze frequencies in the saliency map when luminance control is performed for each of the regions. In the luminance control, if a lower limit value of the luminance gradient is not set, the luminance level may change suddenly between regions with high and low gaze frequencies on the display 20, as shown by the dashed line in FIG. 7, for example. In this case, a user looking at the display 20 may feel discomfort due to display unevenness between regions with different gaze frequencies.

[0049] Therefore, in the second brightness control, a lower limit of the brightness gradient between areas with different gaze frequencies is determined, and a brightness level is determined so that the brightness gradient is equal to or greater than the lower limit. The lower limit of the brightness gradient determined in step S101 is used in the brightness control in step S130. The lower limit of the brightness gradient can be set in advance by any method, such as performing a sensory evaluation on a large number of people to examine the relationship between the brightness gradient and the presence or absence of discomfort in the image.

[0050] In the second brightness control, for example, the drawing controller 10 sequentially executes the processes of steps S110 to S140 after step S101. For example, in step S130, the drawing controller 10 determines a brightness level for each gaze frequency based on the lower limit of the brightness gradient determined in step S101 so that the difference in brightness level between adjacent regions with high and low gaze frequencies is equal to or less than a predetermined value. At this time, for example, the brightness level of the region with the highest gaze frequency is used as a reference to determine the brightness levels of the region adjacent to the region and its surrounding region so that they are equal to or greater than the lower limit of the brightness gradient.

[0051] Then, for example, after step S140 ends, the drawing controller 10 returns the process to step S110 and repeats the processes of steps S110 to S140 until a predetermined end condition is satisfied.

[0052] 7 shows a representative example in which the luminance levels of a region with a high gaze frequency and its surrounding region in step S130 are controlled to change linearly, but the present invention is not limited to this. Regarding the determination of the luminance levels between a plurality of adjacent regions with different gaze frequencies in the second luminance control, it is sufficient that the luminance gradient is equal to or greater than the lower limit value and the discomfort felt by the user is reduced, and the change may be step-like or curved, or other known methods may be used.

[0053] The above is an overview of the second brightness control. In addition to the same effects as the first brightness control, the second brightness control can suppress the change in brightness level in multiple adjacent regions with different gaze frequencies to a predetermined level or less, thereby reducing the discomfort caused by display unevenness.

[0054] [Third Brightness Control] Next, the third luminance control based on the saliency map will be described with reference to Fig. 8 and Fig. 9, but here, differences from the first luminance control will be mainly described. Note that in Fig. 9, a part of the luminance level determined by the luminance control when the lower limit value of the luminance level described later is not set is shown by a dashed line.

[0055] In the third luminance control, the drawing controller 10 executes, for example, the processes of steps S110 to S140 similar to those in the first luminance control, but executes the process of step S102 before step S110, as shown in FIG.

[0056] In step S102, for example, the luminance control unit 12d sets a lower limit value of the luminance level. The lower limit value of the luminance level here means the luminance level of the area in the saliency map when luminance control is performed on the area with the lowest gaze frequency. If the lower limit value of the luminance level is not set in the luminance control, the luminance level may change suddenly between areas with high and low gaze frequencies on the display 20, as shown by the dashed line in FIG. 9, for example. In this case, a user looking at the display 20 may feel discomfort due to display unevenness between areas with different gaze frequencies.

[0057] Therefore, in the third luminance control, a lower limit value of the luminance level in the area with the lowest gaze frequency is determined. The lower limit value of the luminance level determined in step S102 is used in the luminance control in step S130. The lower limit value of the luminance level can be set in advance by any method, such as performing a sensory evaluation on a large number of people to examine the relationship between the luminance difference between the part with the highest luminance level and the part with the lowest luminance level and the presence or absence of discomfort in the image. The lower limit value of the luminance level is determined, for example, by the above-mentioned method so that the luminance difference between the part with the highest luminance level and the part with the lowest luminance level is equal to or less than a predetermined threshold value at which many people do not feel discomfort.

[0058] In the third luminance control, for example, the drawing controller 10 sequentially executes the processes of steps S110 to S140 after step S102. For example, in determining the luminance level in step S130, the drawing controller 10 determines the luminance level for each gaze frequency based on the lower limit value of the luminance level determined in step S101 so that the region with the lowest gaze frequency is equal to or higher than the lower limit value of the luminance level. Then, in step S140, for example, the luminance control unit 12d determines the RGB luminance value for each main pixel according to the luminance level determined in step S130. As a result, the difference in luminance level between the region with the highest gaze frequency and the region with the lowest gaze frequency becomes equal to or lower than a predetermined threshold value, and the luminance gradient of the region connecting the region with the high luminance level and the region with the low luminance level becomes equal to or higher than a predetermined value, resulting in an image in which the user does not feel discomfort due to display unevenness.

[0059] Then, for example, after step S140 ends, the drawing controller 10 returns the process to step S110 and repeats the processes of steps S110 to S140 until a predetermined end condition is satisfied.

[0060] The above is an overview of the third brightness control. In addition to the same effects as the first brightness control, the third brightness control can keep the change in brightness level in multiple areas with different gaze frequencies below a predetermined level, and can also reduce discomfort caused by display unevenness.

[0061] [Fourth Brightness Control] Next, the fourth luminance control based on the saliency map will be described with reference to FIG. 10. Here, the differences from the first luminance control will be mainly described.

[0062] In the fourth luminance control, the drawing controller 10 executes, for example, the processes of steps S110 to S140 similar to those of the first luminance control, but executes the processes of steps S111 to S114 between steps S110 and S120 as shown in Fig. 10. In the fourth luminance control, a process is executed to determine whether an image has a white base tone or a black base tone, and to determine the setting value of the lower limit of the luminance gradient or the luminance level to a value corresponding to the black base tone or the white base tone. This is because display unevenness is more noticeable in a white base tone image than in a black base tone image, and the user feels uncomfortable due to the difference in luminance level, and therefore the lower limit of the luminance gradient or the luminance level required to reduce the discomfort is different.

[0063] In step S111 following step S110, for example, the luminance control unit 12d calculates the average value of the gradation values ​​of each sub-pixel in the main pixel group that displays this image, based on the image data in step S110. For ease of explanation, this average value will be simply referred to as the "average RGB value." The average RGB value is a value obtained by adding up all the gradation values ​​of the sub-pixels (R, G, B) of each main pixel and dividing the result by the total number of sub-pixels, and for example, when the gradation values ​​are 0 to 256, the average RGB value data will be a value between 0 and 256. The data of the average RGB values ​​calculated in step S111 is used in the process of step S112.

[0064] Next, in step S112, for example, the luminance control unit 12d determines whether or not the average value of the RGB values ​​calculated in step S111 is equal to or greater than a predetermined threshold value, and if the determination is affirmative, the process proceeds to step S113, whereas if the determination is negative, the process proceeds to step S114. Hereinafter, for convenience of explanation, the predetermined threshold value corresponding to the average value of the RGB values ​​is referred to as the "average threshold value." The average threshold value in step S112 is for determining whether the image acquired in step S110 is primarily white or primarily black, and is set to 128, for example, when the gradation value is in the range of 0 to 256, but is not limited to this.

[0065] According to the inventors' research, regardless of the ratio of RGB gradation values ​​of each pixel, the larger the average RGB values, the more noticeable the display unevenness tends to be. Therefore, it is effective to use the average RGB values ​​to determine whether an image is primarily white or black.

[0066] In step S113 where it is determined that the image is based on white, for example, the luminance control unit 12d determines a set value for the white base. The set value here is the lower limit of the luminance gradient or the lower limit of the luminance level. In step S113, for example, the set value is preset to a value similar to that of the second luminance control described above when it is the lower limit of the luminance gradient, and is determined to this value, but is not limited to this. Also, for example, in the case of the lower limit of the luminance level, the set value is preset to 75%, with 100% being the state in which the image is displayed in white, that is, all sub-pixels have the maximum gradation value of 256, and is determined to this value, but is not limited to this. The set value determined in step S113 is used for the luminance control in the next step S130.

[0067] On the other hand, in step S114 where it is determined that the image is black-based, for example, the luminance control unit 12d determines a set value for black-based. In step S114, for example, when the set value is the lower limit of the luminance gradient, no lower limit is set and an arbitrary value is set. This is because it is assumed that the luminance gradient is equal to or less than a predetermined value without any particular restriction in a black-based image, and the display unevenness is unlikely to be so large that the user feels uncomfortable. For example, when the set value is the lower limit of the luminance level, the white display is set to 100%, and the set value is set to 50%, which is smaller than the white-based tone, in advance, and is determined to this value, but is not limited to this. The set value determined in step S114 is used for the luminance control in the next step S130.

[0068] Next, in step S130, for example, the luminance control unit 12d executes determination of the luminance level according to the gaze frequency for a plurality of areas with different gaze frequencies so as to satisfy the set value determined in step S113 or step S114. Then, in step S140, for example, the luminance control unit 12d determines the RGB luminance value for each main pixel according to the luminance level determined in step S130. This determines the lower limit value of the luminance gradient or luminance level according to whether the image is based on white or black, making it possible to ensure that display unevenness is less noticeable in either case and to ensure ease of viewing of areas with high gaze frequencies.

[0069] Then, for example, after step S140 ends, the drawing controller 10 returns the process to step S110 and repeats the processes of steps S110 to S140 until a predetermined end condition is satisfied.

[0070] The above is an overview of the fourth luminance control. In addition to the same effects as the first luminance control, the fourth luminance control determines a setting value according to whether the image is based on white or black, and also reduces the discomfort caused by display unevenness between multiple areas with different gaze frequencies.

[0071] [Fifth Brightness Control] Next, the fifth luminance control based on the saliency map will be described with reference to FIG. 11 and FIG. 12. Here, the differences from the first luminance control will be mainly described.

[0072] In the fifth luminance control, the drawing controller 10 executes, for example, the processes of steps S110 to S140 similar to those of the first luminance control, but executes the processes of steps S115 to S119 between steps S110 and S120 as shown in Fig. 11. In the fifth luminance control, information on the driving state and vehicle state of the host vehicle V is acquired, and a process is executed to determine a setting value that suppresses the user's attention to the image in a predetermined situation from the viewpoint of ensuring safety.

[0073] In step S115, for example, the brightness control unit 12d acquires information on the driving mode of the host vehicle V, that is, whether the host vehicle V is in manual driving or automatic driving, from a sensor or switch (not shown) of the in-vehicle device 30.

[0074] In step S116, for example, the luminance control unit 12d acquires the traveling speed of the host vehicle V, that is, information on the vehicle speed, from a vehicle speed sensor (not shown) of the in-vehicle device 30 or the like.

[0075] Next, in step S117, for example, the luminance control unit 12d determines whether or not "the vehicle V is being manually driven and the vehicle speed is equal to or greater than a predetermined threshold" based on the information acquired in steps S115 and S116. Note that the predetermined threshold for the vehicle speed in step S117 is set as a speed threshold, and the speed threshold is appropriately set based on, for example, the width of the driver's effective visual field according to the vehicle speed. For example, if the luminance control unit 12d determines in step S117 that the vehicle is being manually driven, the luminance control unit 12d advances the process to step S118, and if the luminance control unit 12d determines in step S117 that the vehicle is being manually driven, the luminance control unit 12d advances the process to step S119.

[0076] In step S118, since the driver must look ahead of the vehicle V, the luminance control unit 12d, for example, determines a set value that suppresses the user from being attracted to the screen from the viewpoint of ensuring safety. The set value here is the lower limit of the luminance gradient or the lower limit of the luminance level. In step S118, since the need to ensure the visibility of the screen is low, for example, the set value is set to any value when it is the lower limit of the luminance gradient, and is determined to 50% when it is the lower limit of the luminance level, but is not limited thereto.

[0077] On the other hand, in step S119, since it is considered that the driver has enough time to see the screen, for example, the luminance control unit 12d determines the set value of the normal mode. The normal mode here means a display mode of the display 20 corresponding to a situation other than a situation in which the driver must watch the front of the vehicle V, such as, for example, "the vehicle V is being manually driven and the vehicle speed is equal to or higher than a predetermined threshold value." In step S119, for example, the set value is determined to be equal to or higher than a predetermined value when it is the lower limit value of the luminance gradient, and 75% when it is the lower limit value of the luminance level, but is not limited thereto. The set value determined in step S118 or S119 is used for the luminance control in the next step S130.

[0078] Next, in step S130, for example, the luminance control unit 12d executes determination of the luminance level according to the gaze frequency for a plurality of regions with different gaze frequencies so as to satisfy the set value determined in step S118 or step S119. Then, in step S140, for example, the luminance control unit 12d determines the RGB luminance value for each main pixel according to the luminance level determined in step S130. Thereby, the display system executes luminance control that suppresses the attraction to the image in a situation where the driver must gaze ahead of the vehicle V, and executes luminance control that achieves both easy viewing of the screen and power saving in other situations.

[0079] Then, for example, after step S140 ends, the drawing controller 10 returns the process to step S110 and repeats the processes of steps S110 to S140 until a predetermined end condition is satisfied.

[0080] The above is an overview of the fifth brightness control. According to the fifth brightness control, in addition to the same effects as the first brightness control, the fifth brightness control also contributes to safe driving by executing brightness control that suppresses distracting eyes in a situation where the driver must look ahead of the vehicle V.

[0081] The processing of steps S115 and S116 may be performed in the reverse order or in parallel. In the above, the information acquisition in steps S115 and S116 and the judgment processing in step S117 are all performed by the brightness control unit 12d, but the present invention is not limited to this. For example, as shown in FIG. 12, the drawing controller 10 may further include an information acquisition unit 12f that acquires various information on the driving state and the vehicle state, and executes a judgment process on the situation of the vehicle. In this case, the information acquisition unit 12f acquires various information on the vehicle V from the in-vehicle device 30, judges the situation of the vehicle V, and outputs a signal according to the judgment result to the brightness control unit 12d. In this way, the configuration of the drawing controller 10 and the parts that execute various processes may be appropriately changed.

[0082] [6th Brightness Control] Next, the sixth luminance control based on the saliency map will be described with reference to Figs. 13 to 15, but here, differences from the first luminance control will be mainly described.

[0083] In the sixth luminance control, the drawing controller 10 executes, for example, the processes of steps S110 to S140 similar to those of the first luminance control, but executes the process of step S121 in parallel with step S120 as shown in Fig. 13. Also, in the sixth luminance control, the drawing controller 10 executes the process of step S122 between steps S120 and S130. In the sixth luminance control, a process is executed to maintain the luminance level for a portion displaying important information, even in an area of ​​an image that is not frequently gazed upon.

[0084] In step S121, for example, the luminance control unit 12d determines a predetermined region including a portion displaying important information as a mask region that is not subject to luminance control based on the saliency map SM based on the image data in step S120. For example, as shown in Fig. 14, a case will be described as a representative example in which the image P2 acquired in step S110 has icons G1 to G5, which are GUIs, and various contents C10 to C14.

[0085] For example, in step S120, the saliency map generating unit 12c generates a saliency map SM based on various characteristics of the icons G1 to G5 and the contents C10 to C14. For example, the obtained saliency map SM is one in which the gaze frequency of the portion displaying the icon G3 and the contents C10, C11, and C12 is higher than that of other regions, and the gaze frequency of the other regions is lower. In this case, if the content C13 displayed in the portion determined to be the region with a low gaze frequency indicates the remaining battery level, and the content C14 indicates the radio wave strength for communication, these pieces of information are highly important among those presented to the user. In such a case, if the brightness control based on the saliency map SM is simply executed, the brightness level of the portion displaying the important information will decrease, and the important information may become difficult to see. Note that, in the above, an example in which the remaining battery level and the radio wave strength are set as important information has been described, but the important information is not limited to these, and other information such as information on / off of Bluetooth (registered trademark) may be set as important information.

[0086] Therefore, in the sixth luminance control, in step S121, the luminance control unit 12d preliminarily masks a portion of the display area 20a of the display 20 that displays important information as a mask area R, for example, as shown in FIG. M The mask area R M is, for example, preset for each image.

[0087] In the next step S122, for example, the luminance control unit 12d adds a mask region R to the saliency map SM as shown in FIG. M applies.

[0088] Next, in step S130, for example, the luminance control unit 12d calculates the saliency map SM and the mask area R M The luminance control unit 12d determines the luminance level according to the distribution of gaze frequencies in the image based on the above. For example, as shown in FIG. MExcept for the pixel groups corresponding to the main pixels, the luminance level of the pixel groups is determined for each region with a different gaze frequency according to the gaze frequency. Then, in step S140, for example, the luminance control unit 12d determines the RGB luminance value for each main pixel according to the luminance level determined in step S130. In this way, the display system ensures the visibility of the portion displaying important information, while adjusting the luminance level of other regions according to the gaze frequency, thereby enabling both visibility of the image and power saving.

[0089] Then, for example, after step S140 ends, the drawing controller 10 returns the process to step S110 and repeats the processes of steps S110 to S140 until a predetermined end condition is satisfied.

[0090] The above is an overview of the sixth brightness control. In addition to the same effects as the first brightness control, the sixth brightness control also has the effect of maintaining the brightness of the display portion of important information in the image and ensuring the visibility of the display portion.

[0091] According to this embodiment, the display system generates a saliency map based on an image to be displayed on the display 20, and executes brightness control to determine a brightness level according to the gaze frequency based on the distribution of gaze frequencies in the image. Therefore, without using a sensor to detect the movement of the user's eyes, the brightness level is maintained for areas to which the user frequently directs natural gaze, making them easy to see, while the brightness level is lowered for areas to which the user frequently gazes, thereby achieving the effect of saving power.

[0092] In addition, when this display system executes the second or third brightness control that determines the lower limit of the brightness gradient or the lower limit of the brightness level, the change or difference in brightness level between areas with different gaze frequencies is kept below a predetermined level, thereby suppressing display unevenness and reducing the sense of discomfort felt by the user.

[0093] Furthermore, when this display system performs a fourth brightness control in which the average value of the RGB values ​​of the image is calculated and the setting value is changed based on the average value depending on whether the image is based on white or black, it is possible to achieve both easy viewing and power saving regardless of whether the image is based on white or black.

[0094] In addition, when used in an automobile, this display system acquires information on the driving condition and vehicle condition, and in situations where the user must look ahead of the vehicle V, executes a fifth brightness control that suppresses distraction from the view, thereby achieving both ease of viewing and power saving, while also ensuring safety.

[0095] In addition, this display system can ensure that important information in an image is easy to see when it performs a sixth brightness control that determines the portion in which important information is to be displayed as a mask area and maintains the brightness level of the mask area regardless of the frequency of gaze thereon. (Other embodiments) Although the present disclosure has been described based on the embodiment, it is understood that the present disclosure is not limited to the embodiment or structure. The present disclosure also includes various modifications and modifications within the equivalent range. In addition, various combinations and forms, and other combinations and forms including only one element, more than one, or less than one, are also within the scope and concept of the present disclosure.

[0096] The control unit 12 and the method described in the present disclosure may be implemented by a special-purpose computer provided by configuring a processor and a memory programmed to perform one or more functions embodied in a computer program. Alternatively, the control unit 12 and the method described in the present disclosure may be implemented by a special-purpose computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit 12 and the method described in the present disclosure may be implemented by one or more special-purpose computers configured by a combination of a processor and a memory programmed to perform one or more functions and a processor configured with one or more hardware logic circuits. Also, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions executed by the computer.

[0097] In each of the above embodiments, it goes without saying that the elements constituting the embodiment are not necessarily essential, except when it is specifically stated that they are essential or when it is clearly considered essential in principle. In addition, in each of the above embodiments, when the numbers, values, amounts, ranges, etc. of the components of the embodiment are mentioned, they are not limited to the specific numbers, except when it is specifically stated that they are essential or when it is clearly limited to a specific number in principle. In addition, in each of the above embodiments, when the shapes, positional relationships, etc. of the components are mentioned, they are not limited to the shapes, positional relationships, etc., except when it is specifically stated that they are essential or when it is clearly limited to a specific shape, positional relationship, etc. in principle. [Explanation of symbols]

[0098] 12c Saliency map generator 12d Brightness control section 20 Display 30 In-vehicle equipment

Claims

1. A display control device that performs brightness control of a display (20), a saliency map generating unit (12c) for generating a saliency map based on image data to be displayed on the display; A display control device comprising: a brightness control unit (12d) that executes brightness control of the display based on a gaze frequency in the saliency map.

2. 2. The display control device according to claim 1, wherein the brightness control unit maintains the brightness of pixel groups corresponding to areas of the display area where the gaze frequency is higher than a predetermined threshold, and performs control to reduce or set to zero the brightness of pixel groups corresponding to areas where the gaze frequency is equal to or lower than the threshold.

3. The display control device according to claim 2 , wherein the luminance gradient between pixel groups corresponding to a plurality of areas of the display area with different gaze frequencies is set as a luminance gradient, and the luminance control unit performs luminance control that does not limit a lower limit value of the luminance gradient.

4. 3. The display control device according to claim 2, wherein the luminance gradient between pixel groups corresponding to a plurality of areas of the display area with different gaze frequencies is set as a luminance gradient, and the luminance control unit determines a lower limit value of the luminance gradient and performs luminance control such that the luminance gradient is equal to or greater than the lower limit value.

5. The display control device according to claim 2 , wherein the luminance control unit determines a lower limit value of luminance, and executes luminance control to set the luminance of an area of ​​the display area where luminance is to be controlled to be reduced to equal to or higher than the lower limit value.

6. 5. The display control device according to claim 4, wherein the luminance control unit calculates an average value of gradation values ​​of a plurality of sub-pixels with different luminescent colors constituting each pixel for a group of pixels constituting the display, and executes control to change the lower limit value of the luminance gradient in accordance with the average value.

7. 7. The display control device according to claim 6, wherein the luminance control unit sets a predetermined threshold for the average value as an average threshold, and when the average value is higher than the average threshold, executes control to make the luminance gradient smaller than when the average value is equal to or lower than the average threshold.

8. 6. The display control device according to claim 5, wherein the luminance control unit calculates an average value of gradation values ​​of a plurality of sub-pixels with different luminescent colors constituting each pixel of the display, and performs control to change the lower limit value of the luminance of an area where control to lower the luminance is performed in accordance with the average value.

9. 9. The display control device according to claim 8, wherein the luminance control unit sets a predetermined threshold for the average value as an average threshold, and when the average value is higher than the average threshold, performs control to make the lower limit value of luminance larger than when the average value is equal to or lower than the average threshold.

10. The display control device according to claim 1 , wherein the luminance control unit executes control to maintain luminance in an area of ​​the display area where a predetermined content is displayed, regardless of the gaze frequency.

11. A display (20) mounted on a vehicle; a saliency map generating unit (12c) for generating a saliency map based on image data to be displayed on the display; and a brightness control unit (12d) that controls the brightness of the display in accordance with a gaze frequency in the saliency map.

12. The in-vehicle display system of claim 11 , wherein the display is wider than a driver's seat of the vehicle.

13. 12. The automotive display system according to claim 11, wherein the luminance control unit determines a lower limit value of the luminance gradient, which is a gradient of luminance between pixel groups corresponding to a plurality of regions of the display area of ​​the display that have different gaze frequencies, and executes luminance control such that the luminance gradient is equal to or greater than the lower limit value, and also obtains information on a driver's state or a vehicle speed of the vehicle from an in-vehicle device (30) of the vehicle, and executes control to change the lower limit value of the luminance gradient based on the information.

14. The automotive display system according to claim 13, wherein the luminance control unit executes control to increase the luminance gradient when the vehicle is being manually driven and the vehicle speed exceeds a speed threshold, compared to when the vehicle is being automatically driven or the vehicle speed is equal to or less than the speed threshold.

15. The luminance control unit determines a lower limit value of luminance, and executes luminance control to set the luminance of an area of ​​the display area where luminance is controlled to be reduced to equal to or higher than the lower limit value, and also acquires information on the state of the driver of the vehicle or the vehicle speed of the vehicle from an in-vehicle device (30) of the vehicle, and executes control to change the lower limit value of luminance based on the information.

16. 16. The automotive display system according to claim 15, wherein the luminance control unit executes control to reduce the lower limit value of luminance when the vehicle is being manually driven and the vehicle speed is equal to or greater than a speed threshold value, compared to when the vehicle is being automatically driven or the vehicle speed is less than the speed threshold value.

Images