Display control method, display method, display control device, and display control system

Abstract

To provide a display control method capable of displaying objects properly, a display method, a display control device, and a display control system.SOLUTION: A display control method concerning the present disclosure includes the following steps: acquiring a first image corresponding to the field of view of a driver in the vehicle interior including the display area; generating a second image by adding a display object in the first display format on the display on the display area in the first image; estimating the attention state of the driver by applying one or more attention estimation models to the second image; and controlling the display format of the display object to be displayed in the predetermined display area of the display inside the vehicle depending on the estimation result of the attention state.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a display control method, a display control device, and a display control system.

Background Art

[0002] There is a case where a display such as a head-up display is controlled to display a display object in a predetermined display area in the vehicle interior. Thereby, information can be transmitted to the driver and driving support for the driver can be performed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

[0004] In order to appropriately transmit information to the driver, it is desirable to control the display so as to appropriately display the display object.

[0005] The present disclosure provides a display control method, a display method, a display control device, and a display control system capable of appropriately displaying a display object.

[0006] The display control method according to the present disclosure includes acquiring a first image corresponding to the driver's field of view in the vehicle interior including a display area, adding a display object by a display in a first display form to the display area in the first image to generate a second image, applying one or more attention estimation models to the second image to estimate the driver's attention state, and controlling the display form of the display object to be displayed by the display in the predetermined display area in the vehicle interior according to the estimation result of the attention state.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Embodiments for Carrying Out the Invention

[0008] Hereinafter, embodiments of a display control device according to the present disclosure will be described with reference to the drawings.

[0009] (First Embodiment) The display control method according to the first embodiment controls a display such as a head-up display to display a display object in a display area in the vehicle interior, and a device for appropriately displaying the display object is provided.

[0010] The display control method is executed in a vehicle 100 as shown in FIG. 1. FIG. 1 is a diagram showing an outline of a vehicle in which the display control method is executed.

[0011] The vehicle 100 is any moving body capable of traveling on a road surface 600. The vehicle 100 may be a two-wheeled vehicle, a three-wheeled vehicle, a four-wheeled vehicle, a moving body having five or more wheels, or a moving body having no wheels. Hereinafter, mainly, the case where the vehicle 10 is a four-wheeled vehicle will be mainly described. Also, the traveling direction of the vehicle 100 is defined as the X direction, the direction perpendicular to the road surface 600 is defined as the Z direction, and the direction perpendicular to the X direction and the Z direction is defined as the Y direction.

[0012] In the vehicle 100, a vehicle interior 110 is formed, surrounded by a substantially box-shaped vehicle body 100a. The driver 200 can board a driver's seat 103 in the vehicle interior 110. In a normal posture, the driver 200 can visually recognize a dotted-line visual field VF1. As shown in FIG. 2, the dotted-line visual field VF1 includes the windshield 101 on the +X side and members (for example, the steering wheel 106) around the windshield 101 in the XY direction. The steering wheel 106 can be operated by the driver 200.

[0013] Inside the passenger compartment 110 shown in FIG. 1, an imaging sensor 11 is mounted. The imaging sensor 11 has an imaging range VF2. The imaging range VF2 corresponds to the visual field VF1. Inside the passenger compartment 110, the imaging sensor 11 is arranged at an arbitrary position where an imaging range VF2 corresponding to the visual field VF1 can be secured. In FIG. 1, a headrest 104 is arranged on the +Z side of the driver's seat 103, and the imaging sensor 11 is connected via a frame 105 on the +Z side of the headrest 104. As shown in FIG. 2, the imaging range VF2 may substantially coincide with the visual field VF1. Thereby, the imaging sensor 11 can image the imaging range VF2 and acquire an image IM1 of the imaging range VF2. The image IM1 is an image corresponding to the visual field VF1 of the driver 200.

[0014] The vehicle 100 is further equipped with a display 18 and a display control device 1 for controlling the same. The display 18 can display a display object in a display area DA1 inside the passenger compartment 110. The display area DA1 may be located near the +Z side end of the steering wheel 106 on the windshield 101.

[0015] The configuration including the imaging sensor 11, the display control device 1, and the display 18 functions as a display control system 2 that executes a display control method. The display control system 2 can be configured hardware-wise as shown in FIG. 3. FIG. 3 shows the hardware configuration of the display control system 2.

[0016] In the display control system 2, the display control device 1 is connected between the imaging sensor 11 and the display 18.

[0017] The display control device 1 has an imaging interface (I / F) 12, a CPU 13, a volatile memory unit 14, a display interface (I / F) 15, a non-volatile memory unit 16, and a bus 18. The imaging interface 12, the CPU 13, the volatile memory unit 14, the display interface 15, and the non-volatile memory unit 16 are communicably connected to each other via the bus 18.

[0018] The CPU 13 comprehensively controls each part of the display control device 1.

[0019] The imaging interface 12 is communicably connected to the imaging sensor 11 via a communication medium such as a communication line. Under the control of the CPU 13, the imaging interface 12 performs interface operations for the imaging sensor 11.

[0020] The volatile memory unit 14 temporarily stores information. The volatile memory unit 14 can also be used as a working area for the CPU 13.

[0021] The non-volatile memory unit 16 stores information non-volatily. The non-volatile memory unit 16 may store a program 17 for executing the display control method.

[0022] The display interface 15 is communicably connected to the display 18 via a communication medium such as a communication line. Under the control of the CPU 13, the display interface 15 performs interface operations for the display 18.

[0023] The display 18 shown in FIG. 1 may be a head-up display. The display 18 is disposed in a space 102 provided on the -Z side of the windshield 101. The space 102 may be a dashboard. The space 102 has an opening on the +Z side. The display 18 can irradiate light from the -Z side of the windshield 101 through the opening to a display area DA1 on the windshield 101.

[0024] Thereby, when the display 18 projects the display object 300 onto the display area DA1, the display object 300 reflected by the windshield 101 as a display medium can be visually recognized by the driver 200. The display 18 projects a display object 400 as a virtual image onto a virtual screen 500 disposed in front of the vehicle 100. As shown in FIG. 2, the driver 200 will visually recognize the display object 400 through the windshield 101.

[0025] The display 18 can allow the driver 200 to visually recognize a display object 400 that indicates driving assistance information. The driving assistance information includes, for example, vehicle speed information, navigation information, pedestrian information, information on a vehicle ahead, lane departure information, and vehicle condition. The navigation information includes a right turn guidance, a left turn guidance, a straight-ahead guidance, a stop guidance, a parking guidance, a parking guidance, a right lane change guidance, a left lane change guidance, and the like. In FIG. 2, a case where the display 18 displays an arrow display object 400 indicating a right turn guidance is illustrated.

[0026] Thereby, the display 18 can transmit driving assistance information to the driver 200 and perform driving assistance for the driver 200 by displaying a display object in the display area DA1.

[0027] Next, the functional configuration of the display control device 1 will be described with reference to FIG. 4. FIG. 4 is a diagram showing the functional configuration of the display control device 1.

[0028] The display control device 1 includes an acquisition unit 4, a simulation unit 5, a caution estimation unit 6, a display determination unit 8, and a display control unit 9. In the display control device 1, each unit shown in FIG. 4 may be realized hardware-wise (for example, as a circuit), may be realized software-wise, or part may be realized hardware-wise and the remainder may be realized software-wise. When each unit shown in FIG. 4 is realized software-wise, the CPU 13 (see FIG. 3) may execute the program 17 to functionally construct each unit shown in FIG. 4 on the volatile storage unit 14 all at once at compile time or sequentially as the processing progresses.

[0029] The imaging sensor 11 images an imaging range VF2 (see FIG. 2) in the passenger compartment 110 and acquires an image IM1. When the imaging range VF2 corresponds to the visual field VF1, the image IM1 corresponds to the visual field VF1 of the driver 200 and can be regarded as a visual field image. The image IM1 includes the display area DA1. The imaging sensor 11 supplies the image IM1 to the display control device 1.

[0030] The acquisition unit 4 receives the image IM1 from the imaging sensor 11. The acquisition unit 4 can acquire a field-of-view image using the image IM1. The acquisition unit 4 may use the image IM1 as the field-of-view image as it is. The acquisition unit 4 supplies the image IM1 to the simulation unit 5.

[0031] The simulation unit 5 receives the image IM1 from the acquisition unit 4. The simulation unit 5 generates the image IM2 using the image IM1. The simulation unit 5 may generate the image IM2 by adding the display object 400 in the display form DF1 to the display area DA1 in the image IM1. The display form DF1 includes the brightness BR1. The simulation unit 5 supplies the image IM2 to the attention estimation unit 6.

[0032] The attention estimation unit 6 receives the image IM2 from the simulation unit 5. The attention estimation unit 6 includes one or more attention estimation models 7. The attention estimation unit 6 applies one or more attention estimation models 7 to the image IM2 to estimate the attention state of the driver 200. The attention estimation model 7 is a learned model that has been trained to estimate the attention state based on the mechanism of human cognition. In FIG. 4, the case where the attention estimation unit 6 includes the novelty estimation model 7_1 is illustrated. The novelty estimation model 7_1 is a learned model that has been trained to estimate novelty based on the mechanism of human cognition.

[0033] Novelty is a characteristic related to human attention and is based on the predictive coding theory and the free energy principle. A person's brain is always predicting the outside world. When the brain's prediction is off, in order to minimize the error between the brain's prediction of the outside world and the external stimulus perceived, attention is attracted because the person tries to actively incorporate the information at the location where the prediction is off. The greater the prediction error, or the change in the prediction error, the higher the novelty tends to be.

[0034] The novelty estimation model 7_1 can generate map information MP that shows a two-dimensional distribution of prediction errors corresponding to the current image by generating a current predicted image from visual field images at a plurality of past time points and comparing the current image with the predicted image. Since the higher the prediction error, the higher the novelty tends to be, the map information MP also shows a two-dimensional distribution of novelty.

[0035] The attention estimation unit 6 inputs visual field images at a plurality of past time points and the image IM2 to the attention estimation model 7. The attention estimation model 7 generates a current predicted image from visual field images at a plurality of past time points, and two-dimensionally obtains a feature amount related to the attention state of a person based on the image IM2 and the predicted image. The attention estimation model 7 estimates the attention state according to the distribution of the two-dimensional feature amount, and outputs the estimation result of the attention state to the attention estimation unit 6. The attention estimation unit 6 supplies the estimation result of the attention state to the display determination unit 8.

[0036] When the attention estimation model 7 is the novelty estimation model 7_1, the attention estimation unit 6 inputs visual field images at a plurality of past time points and the image IM2 to the novelty estimation model 7_1. The novelty estimation model 7_1 generates a current predicted image from visual field images at a plurality of past time points, two-dimensionally obtains a prediction error related to human cognition based on the image IM2 and the predicted image, and generates map information MP that shows a distribution of the two-dimensional prediction error. The novelty estimation model 7_1 outputs the map information MP to the attention estimation unit 6. The attention estimation unit 6 supplies an estimation result of the attention state including the image IM2 and the map information MP to the display determination unit 8.

[0037] The display determination unit 8 receives the estimation result of the attention state from the attention estimation unit 6. The display determination unit 8 determines the display form of the display object 400 to be displayed by the display 18 in the display area DA1 in the passenger compartment 110 according to the estimation result of the attention state.

[0038] When the estimation result of the attention state is the first estimation result, the display determination unit 8 changes the display form of the display object 400 from the display form DF1 to the display form DF2. The first estimation result indicates that excessive attention is concentrated on the display object 400 in the visual field VF1. The display form DF2 is a display form in which attention is more suppressed than the display form DF1.

[0039] When the estimation result of the attention state is the second estimation result, the display determination unit 8 maintains the display form of the display object 400 as the display form DF1. The second estimation result indicates that the degree of attention concentration on the display object 400 in the visual field VF1 is within an acceptable range.

[0040] For example, the display form DF1 may include displaying the display object 400 with the luminance BR1, and the display form DF2 may include displaying the display object 400 with the luminance BR2. The luminance BR2 is lower than the luminance BR1.

[0041] The display form DF1 may include displaying the display object 400 with the luminance BR1, and the display form DF2 may include displaying the display object while gradually increasing the luminance from the luminance BR2.

[0042] The display form DF1 may include displaying the display object with the color CL1, and the display form DF2 may include displaying the display object with the color CL2. The color CL1 may include the chroma and hue of the display object 400, or may include any combination of luminance, chroma, and hue. The color CL2 may be a color in which at least one of the chroma and hue is different from the color CL1 and attention is more suppressed than the color CL1. The color CL2 may be a color in which one or more of the luminance, chroma, and hue are different from the color CL1 and attention is more suppressed than the color CL1.

[0043] The display form DF1 may include immediately displaying the display object 400, and the display form DF2 may include displaying the display object 400 after the time Δt has elapsed. The time Δt can be determined experimentally in advance as a time sufficient to suppress attention.

[0044] When the estimation result of the attention state includes the image IM2 and the map information MP, the display determination unit 8 may specify the pattern of the attention state according to the image IM2 and the map information MP2. The display determination unit 8 may have a registered pattern 81 and display form information 82. The registered pattern 81 includes one or more attention state patterns experimentally determined in advance as patterns of attention states in which attention is excessively concentrated on the display object 400 in the visual field VF1. The display form information 82 may have the pattern of the attention state and the display form to be changed associated with one or more attention state patterns. Alternatively, the display form information 82 may have the pattern of the attention state and the amount of change in the display form associated with one or more attention state patterns.

[0045] When the pattern of the attention state matches the registered pattern 81, the display determination unit 8 refers to the display form information 82 and changes the display form of the display object 400 from the display form DF1 to the display form DF2. The display form DF2 is a display form in which attention is suppressed more than the display form DF1. In the display form information 82, a display form in which attention is suppressed for each of one or more attention state patterns may be experimentally obtained in advance, and the obtained display form may be included in a form associated with the attention state pattern as the display form to be changed. Alternatively, in the display form information 82, a display form in which attention is suppressed for each of one or more attention state patterns may be experimentally obtained in advance, and the difference between the obtained display form and the standard display form may be included in a form associated with the attention state pattern as the amount of change in the display form.

[0046] When the pattern of the attention state does not match the registered pattern 81, the display determination unit 8 maintains the display form of the display object 400 as the display form DF1.

[0047] The display determination unit 8 supplies the determined display form to the display control unit 9 as the display form of the display object 400.

[0048] The display control unit 9 receives the display form of the display object 400 from the display determination unit 8. The display control unit 9 generates a control signal according to the display form of the display object 400 and supplies it to the display unit 10.

[0049] The display unit 10 receives the control signal from the display control unit 9 and displays the display object 400 in the display area DA1 in the passenger compartment 110 in a display form according to the control signal. Thereby, when excessive attention is concentrated on the display object 400 in the visual field VF1 when the display object 400 is displayed in the display form DF1, the display object 400 can be displayed in the display area DA1 in a display form DF2 in which attention is suppressed more than the display form DF1.

[0050] Next, the display control method executed by the display control system 2 will be described with reference to FIG. 5. FIG. 5 is a flowchart showing the display control method.

[0051] In the display control system 2, the display control device 1 acquires, as a visual field image, an image IM1 obtained by imaging the imaging range VF2 with the imaging sensor 11 (S1). The display control device 1 adds the display object 400 in the display form DF1 to the display area DA1 in the image IM1 and generates an image IM2 as a simulation image (S2).

[0052] The display control device 1 applies the attention estimation model 7 to the image IM2 to estimate the attention state of the driver 200 (S3) and generates an estimation result of the attention state. The display control device 1 may apply the novelty estimation model 7_1 to the image IM2 to generate map information MP indicating a two-dimensional distribution of the prediction error corresponding to the image IM2, and generate an estimation result of the attention state including the image IM2 and the map information MP.

[0053] The display control device 1 performs display control processing according to the estimation result of the caution state (S4). In the display control processing (S4), according to the estimation result of the caution state, the display form of the display object 400 to be displayed on the display 18 in the display area DA1 in the passenger compartment 110 is determined, and the display 18 is controlled to display the display object 400 in the determined display form.

[0054] When the estimation result of the caution state includes the image IM2 and the map information MP, the display control device 1 may specify the pattern of the caution state. The display control device 1 determines whether the pattern of the caution state matches the registered pattern 81 (S5).

[0055] When the pattern of the caution state does not match the registered pattern 81 (No in S5), the display control device 1 determines the display form of the display object 400 as the display form DF1 according to the simulation image in S2. The display control device 1 controls the display 18 to display the display object 400 in the display form DF1 in the display area DA1 in the passenger compartment 110 (S6).

[0056] When the pattern of the caution state matches the registered pattern 81 (Yes in S5), the display control device 1 refers to the display form information 82 and determines the display form of the display object 400 as the display form DF2. The display form DF2 is a display form in which caution is suppressed more than the display form DF1 of the simulation image in S2. The display control device 1 controls the display 18 to display the display object 400 in the display form DF2 in the display area DA1 in the passenger compartment 110 (S7).

[0057] As described above, in the first embodiment, the display control method acquires a field-of-view image, generates a simulation image from the field-of-view image, applies an attention estimation model to the simulation image to estimate the attention state of the driver 200, and controls the display form of the display object 400 by the display 18 according to the estimation result. For example, when the pattern of the attention state matches the registered pattern 81, the display form of the display object 400 is changed from the display form DF1 of the simulation image to the display form DF2 in which attention is more suppressed. Thereby, when excessive attention is concentrated on the display object 400 in the visual field VF1 when the display object 400 is displayed in the display form DF1, the display object 400 can be displayed in the display area DA1 in the display form DF2 in which attention is more suppressed than the display form DF1. As a result, the degree of attention concentration on the display object 400 in the visual field VF1 can be kept within an allowable range.

[0058] Note that the concept of this embodiment is not limited to the display 18 (for example, a head-up display), and can be applied to any display within the field of view of the driver 200. Instead of the display 18, the display control system 2 may have displays 21 to 25 indicated by dotted lines in FIGS. 3 and 4, or may have displays 21 to 25 in addition to the display 18. The supply destination of the control signal from the display control device 1 may be any one of the displays 21 to 25 instead of the display 18. The displays 21 and 22 are respectively provided on the pillars 108 and 109 that support the roof portion 107 of the vehicle body 100a on both sides in the Y direction of the windshield 101, and are also called pillar displays. The display 23 is arranged near the front center in the passenger compartment 100 and is also called a center display. The display 24 displays a speedometer or the like and is also called a meter display. The display 25 functions as a substitute for the rearview mirror by displaying a rear image acquired by the imaging sensor, and is also called an electronic mirror.

[0059] Further, the novelty estimation model 7_1 shown in FIG. 4 may generate map information MP indicating the distribution of the variance of two-dimensional prediction errors instead of map information MP indicating the distribution of the values of two-dimensional prediction errors. In this case, the attention estimation unit 6 may receive the map information MP from the novelty estimation model 7_1 and estimate the attention state according to the distribution of the variance of two-dimensional prediction errors indicated by the map information MP.

[0060] Alternatively, the novelty estimation model 7_1 may generate map information MP indicating the distribution of the combination of the values and variances of two-dimensional prediction errors. In this case, the attention estimation unit 6 may receive the map information MP from the novelty estimation model 7_1 and estimate the attention state according to the distribution of the combination of the values and variances of two-dimensional prediction errors indicated by the map information MP.

[0061] Further, instead of being acquired by one imaging sensor, the field of view image may be acquired by synthesizing images of a plurality of imaging sensors. In this case, the display control system 2 may have a plurality of imaging sensors 19 and 20 shown by dotted lines in FIGS. 3 and 4 instead of the imaging sensor 11. Inside the passenger compartment 110 shown in FIG. 1, the imaging sensor 19 is arranged on the +Y side of the driver 200 and captures an image IM01 by imaging a region corresponding to the +Y side of the field of view VF1, and the imaging sensor 20 is arranged on the -Y side of the driver 200 and captures an image IM02 by imaging a region corresponding to the -Y side of the field of view VF1. The acquisition unit 4 shown in FIG. 4 may receive the image IM01 from the imaging sensor 19, receive the image IM02 from the imaging sensor 20, and acquire the image IM1 by synthesizing the image IM01 and the image IM02. The image IM1 can be regarded as a field of view image corresponding to the field of view VF1.

[0062] Even with such a plurality of imaging sensors 19 and 20, the display control system 2 can acquire a field of view image corresponding to the field of view VF1 of the driver 200.

[0063] Further, the attention estimation unit 6 shown in FIG. 4 may have a plurality of attention estimation models 7. The plurality of attention estimation models 7 may include, in addition to the novelty estimation model 7_1, a saliency estimation model 7_2, ···, and other attention estimation models 7_n. n is an arbitrary integer of 3 or more.

[0064] The saliency estimation model 7_2 is an algorithm that extracts prominent image features based on the mechanism of human perception. Saliency is based on the feature integration theory and is an attention index obtained as a visual feature different from the surroundings by classifying an image into each feature (brightness, color, orientation). There is also a definition that saliency is "the property that a sensory stimulus attracts bottom-up attention", but in this specification, saliency is the property that attention is attracted by the spatial property of a visual stimulus.

[0065] The attention estimation unit 6 may apply a plurality of attention estimation models 7 to the image IM2, obtain a plurality of estimation results of the attention state, synthesize them by weighted addition of the plurality of estimation results, and supply the synthesized estimation result to the display determination unit 8.

[0066] Such an attention estimation unit 6 can also estimate the attention state of the driver 200.

[0067] Further, as a first modification of the first embodiment, display control as shown in FIG. 6 may be performed. FIG. 6 is a diagram showing a display control method according to the first modification of the first embodiment.

[0068] For example, assume that a pattern of the attention state as shown by the image IM2 in FIG. 6(b) and the map information MP in FIG. 6(c) is included in advance in the registered pattern 81.

[0069] In S1 of FIG. 5, the imaging range VF2 is imaged, and an image IM1 as shown in FIG. 6(a) is acquired. The image IM1 shows a situation where there is no object on the road surface 600 for a certain period of time.

[0070] In S2 of FIG. 5, a display object 401 is added to the display area DA1 in the image IM1 with the luminance BR1, and an image IM2 as shown in FIG. 6(b) is generated.

[0071] In S3 of FIG. 5, the attention estimation model 7 is applied to the image IM2, the attention state of the driver 200 is estimated, and map information MP as shown in FIG. 6(c) is generated. The map information MP corresponds to the image IM2. In the map information MP, by referring to the image IM2, it is shown that a pattern PT1 with a novelty level equal to or higher than the threshold level Lth1 is included at a position corresponding to the display area DA1. The threshold level Lth1 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is overly concentrated.

[0072] In S5 of FIG. 5, it is determined that the pattern of the attention state specified by the image IM2 in FIG. 6(b) and the map information MP in FIG. 6(c) matches the registered pattern 81.

[0073] In S7 of FIG. 5, the luminance of the display object 400 is changed to a luminance BR2 lower than the luminance BR1. The display 18 is controlled with the luminance BR2, and by the display 18, as shown in FIG. 6(d), a display object 402 is displayed in the display area DA1 in the visual field VF1 with the luminance BR2.

[0074] Even with such display control, when the display object 400 is displayed with the luminance BR1 and attention is overly concentrated on the display object 400 in the visual field VF1, the display object 400 can be displayed in the display area DA1 with a luminance BR2 where attention is suppressed compared to the luminance BR1. As a result, the degree of attention concentration on the display object 400 in the visual field VF1 can be kept within an allowable range.

[0075] Alternatively, in S1 of FIG. 5, the imaging range VF2 is imaged, and an image IM1 as shown in FIG. 6(a) is acquired. The image IM1 shows a situation where there are no objects on the road surface 600 for a certain period of time.

[0076] In S3 of FIG. 5, the attention estimation model 7 is applied to the image IM2, the attention state of the driver 200 is estimated, and map information MP as shown in FIG. 6(c) is generated. The map information MP corresponds to the image IM2. In the map information MP, by referring to the image IM2, it is shown that a pattern PT1 with a novelty level equal to or higher than the threshold level Lth1 is included at a position corresponding to the display area DA1. The threshold level Lth1 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is overly concentrated.

[0077] In S5 of FIG. 5, it is determined that the pattern of the attention state specified by the image IM2 in FIG. 6(b) and the map information MP in FIG. 6(c) matches the registered pattern 81.

[0078] Also, as a second modification of the first embodiment, display control as shown in FIG. 7 may be performed. FIG. 7 is a diagram showing a display control method according to the second modification of the first embodiment.

[0079] For example, assume that a pattern of the attention state as shown by the image IM2 in FIG. 6(b) and the map information MP in FIG. 7(a) is included in the registered pattern 81 in advance.

[0080] In S1 of FIG. 5, the imaging range VF2 is imaged, and an image IM1 as shown in FIG. 6(a) is acquired. The image IM1 shows a situation where there are no objects on the road surface 600 for a certain period of time.

[0081] In S2 of FIG. 5, a display object 401 is added to the display area DA1 in the image IM1 with a luminance BR1, and an image IM2 as shown in FIG. 6(b) is generated.

[0082] In S3 of FIG. 5, the attention estimation model 7 is applied to the image IM2, the attention state of the driver 200 is estimated, and map information MP as shown in FIG. 7(a) is generated. The map information MP is the same as the map information MP in FIG. 6(c).

[0083] In S5 of FIG. 5, it is determined that the pattern of the attention state specified by the image IM2 in FIG. 6(b) and the map information MP in FIG. 7(a) matches the registered pattern 81.

[0084] In S7 of FIG. 5, it is determined that the luminance of the display object 400 gradually increases from the luminance BR2 lower than the luminance BR1. The display 18 is controlled to gradually increase from the luminance BR2. By the display 18, the display object 403 with the luminance BR2 shown in FIG. 7(b) → the display object 404 with the luminance BR3 (>BR2, <BR1) shown in FIG. 7(c) → the display object 405 with the luminance BR4 (>BR3, <BR1) shown in FIG. 7(d) are sequentially displayed in the display area DA1.

[0085] Even with such display control, when excessive attention is concentrated on the display object 400 in the visual field VF1 when the display object 400 is displayed at the luminance BR1, the display object 400 can be displayed in the display area DA1 so as to gradually increase from the luminance BR2 where the attention is suppressed compared to the luminance BR1. As a result, the degree of concentration of attention on the display object 400 in the visual field VF1 can be kept within an allowable range.

[0086] Also, as a third modification of the first embodiment, display control as shown in FIG. 8 may be performed. FIG. 8 is a diagram showing a display control method according to the third modification of the first embodiment.

[0087] For example, assume that the pattern of the attention state as shown by the image IM2 in FIG. 6(b) and the map information MP in FIG. 8(a) is included in the registered pattern 81 in advance.

[0088] In S1 of FIG. 5, the imaging range VF2 is imaged, and an image IM1 as shown in FIG. 6(a) is acquired. The image IM1 shows a situation where there are no objects on the road surface 600 for a certain period of time.

[0089] In S2 of FIG. 5, the display object 401 is added to the display area DA1 in the image IM1, and an image IM2 as shown in FIG. 6(b) is generated.

[0090] In S3 of FIG. 5, the attention estimation model 7 is applied to the image IM2, the attention state of the driver 200 is estimated, and map information MP as shown in FIG. 8(a) is generated. The map information MP is the same as the map information MP in FIG. 6(c).

[0091] In S5 of FIG. 5, it is determined that the pattern of the attention state specified by the image IM2 in FIG. 6(b) and the map information MP in FIG. 8(a) matches the registered pattern 81.

[0092] In S7 of FIG. 5, the timing at which the display object 400 starts to be displayed is changed to a timing delayed by a time Δt from the immediately subsequent timing. The display 18 is controlled to be delayed by the time Δt. By the display 18, as shown in FIG. 8(b), the display object is not displayed in the display area DA1 for the time Δt, and at the timing delayed by the time Δt, as shown in FIG. 8(c), the display object 405 is displayed in the display area DA1.

[0093] Even with such display control, when the display object 400 is immediately displayed and the attention is overly concentrated on the display object 400 in the visual field VF1, the display object 400 can be displayed in the display area DA1 at the timing delayed by the time Δt. As a result, the degree of attention concentration on the display object 400 in the visual field VF1 can be kept within an allowable range.

[0094] As a third modification of the first embodiment, display control as shown in FIG. 9 may be performed. FIG. 9 is a diagram showing a display control method according to the third modification of the first embodiment.

[0095] For example, assume that a pattern of an attention state as shown by the image IM2 in FIG. 9(b) and the map information MP in FIG. 9(c) is included in the registered pattern 81 in advance.

[0096] In S1 of FIG. 5, the imaging range VF2 is imaged, and an image IM1 as shown in FIG. 9(a) is acquired. The image IM1 shows a situation where the objects OB1 and OB2 to be noted are present at positions away from the display area DA1 on the road surface 600.

[0097] In S2 of FIG. 5, a display object 407 is added to the display area DA1 in the image IM1 with a luminance BR1, and an image IM2 as shown in FIG. 9(b) is generated.

[0098] In S3 of FIG. 5, the attention estimation model 7 is applied to the image IM2, the attention state of the driver 200 is estimated, and map information MP as shown in FIG. 9(c) is generated. The map information MP corresponds to the image IM2. In the map information MP, by referring to the image IM2, it is shown that a pattern PT2 with a novelty level equal to or higher than the threshold level Lth1 is included at a position corresponding to the display area DA1. The threshold level Lth1 can be determined experimentally in advance as the novelty level corresponding to the situation where attention is overly concentrated. Also, in the map information MP, by referring to the image IM2, it is shown that a pattern PT3 with a novelty level equal to or higher than the threshold level Lth2 is included at a position away from the display area DA1. The threshold level Lth2 can be determined experimentally in advance as the novelty level corresponding to the situation where attention is required.

[0099] In S5 of FIG. 5, it is determined that the pattern of the attention state specified by the image IM2 in FIG. 6(b) and the map information MP in FIG. 6(c) matches the registered pattern 81.

[0100] In S7 of FIG. 5, the luminance of the display object 400 is changed to a luminance BR2 lower than the luminance BR1. The display 18 is controlled with the luminance BR2, and by the display 18, as shown in FIG. 9(d), a display object 408 is displayed with the luminance BR2 in the display area DA1 in the visual field VF1.

[0101] Even with such display control, when the display object 400 is displayed at the luminance BR1 and it seems that excessive attention is concentrated on the display object 400 in the visual field VF1, the display object 400 can be displayed in the display area DA1 at a luminance BR2 where attention is suppressed compared to the luminance BR1. As a result, the degree of attention concentration on the display object 400 in the visual field VF1 can be kept within an allowable range, and attention can be appropriately directed to the objects OB1 and OB2 that should be noted.

[0102] As a fourth modification of the first embodiment, display control as shown in FIG. 10 may be performed. FIG. 10 is a diagram showing a display control method according to the fourth modification of the first embodiment.

[0103] For example, assume that a pattern of the attention state as shown by the image IM2 in FIG. 10(b) and the map information MP in FIG. 10(c) is included in advance in the registered pattern 81.

[0104] In S1 of FIG. 5, the imaging range VF2 is imaged, and an image IM1 as shown in FIG. 10(a) is acquired. The image IM1 shows a situation where the objects OB3 and OB4 to be noted exist at positions far from and close to the display area DA on the road surface 600.

[0105] In S2 of FIG. 5, the display object 409 is added to the display area DA1 in the image IM1 at the luminance BR1, and an image IM2 as shown in FIG. 10(b) is generated.

[0106] In S3 of FIG. 5, the attention estimation model 7 is applied to the image IM2, the attention state of the driver 200 is estimated, and map information MP as shown in FIG. 10(c) is generated. The map information MP corresponds to the image IM2. In the map information MP, by referring to the image IM2, it is shown that the pattern PT2 with a novelty level equal to or higher than the threshold level Lth1 is included at the position corresponding to the display area DA1. The threshold level Lth1 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is overly concentrated. Also, in the map information MP, by referring to the image IM2, it is shown that the patterns PT5 and PT6 with a novelty level equal to or higher than the threshold level Lth2 are included at positions close to and away from the display area DA1. The threshold level Lth2 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is required.

[0107] In S5 of FIG. 5, it is determined that the pattern of the attention state specified by the image IM2 in FIG. 10(b) and the map information MP in FIG. 10(c) matches the registered pattern 81.

[0108] In S7 of FIG. 5, the luminance of the display object 400 is changed to a luminance BR2 lower than the luminance BR1. The display 18 is controlled with the luminance BR2, and by the display 18, as shown in FIG. 10(d), the display object 410 is displayed in the display area DA1 in the visual field VF1 with the luminance BR2.

[0109] Even with such display control, when the display object 400 is displayed with the luminance BR1 and excessive attention is concentrated on the display object 400 in the visual field VF1, the display object 400 can be displayed in the display area DA1 with the luminance BR2 where attention is suppressed compared to the luminance BR1. As a result, the degree of attention concentration on the display object 400 in the visual field VF1 can be kept within the allowable range, and attention can be appropriately exerted on the objects OB3 and OB4 that should be noted.

[0110] As a fifth modification of the first embodiment, display control as shown in FIG. 11 may be performed. FIG. 11 is a diagram showing a display control method according to the fifth modification of the first embodiment.

[0111] For example, assume that a pattern of an attention state as shown by the image IM2 in FIG. 11(b) and the map information MP in FIG. 11(c) is included in the registration pattern 81 in advance.

[0112] In S1 of FIG. 5, the imaging range VF2 is imaged, and an image IM1 as shown in FIG. 11(a) is acquired. The image IM1 shows a situation where objects OB5 to OB9 to be noted exist over the entire road surface 600.

[0113] In S2 of FIG. 5, a display object 411 is added to the display area DA1 in the image IM1 with a luminance BR1, and an image IM2 as shown in FIG. 11(b) is generated.

[0114] In S3 of FIG. 5, the attention estimation model 7 is applied to the image IM2, the attention state of the driver 200 is estimated, and map information MP as shown in FIG. 11(c) is generated. The map information MP corresponds to the image IM2. In the map information MP, by referring to the image IM2, it is shown that a pattern PT2 with a novelty level equal to or higher than the threshold level Lth1 is included at a position corresponding to the display area DA1. The threshold level Lth1 can be determined experimentally in advance as a novelty level corresponding to the situation where attention is overly concentrated. Also, in the map information MP, by referring to the image IM2, it is shown that a pattern PT8 with a novelty level equal to or higher than the threshold level Lth2 is included in a wide range including the display area DA1. The threshold level Lth2 can be determined experimentally in advance as a novelty level corresponding to the situation where attention is required.

[0115] In S5 of FIG. 5, it is determined that the pattern of the attention state specified by the image IM2 in FIG. 11(b) and the map information MP in FIG. 11(c) matches the registration pattern 81.

[0116] In S7 of FIG. 5, the luminance of the display object 400 is changed to a luminance BR2 lower than the luminance BR1. The display 18 is controlled at the luminance BR2, and the display object 412 is displayed at the luminance BR2 in the display area DA1 in the visual field VF1 as shown in FIG. 11(d) by the display 18.

[0117] Even with such display control, when the display object 400 is displayed at the luminance BR1 and excessive attention is concentrated on the display object 400 in the visual field VF1, the display object 400 can be displayed in the display area DA1 at a luminance BR2 where attention is suppressed compared to the luminance BR1. As a result, the degree of concentration of attention on the display object 400 in the visual field VF1 can be kept within an allowable range, and attention can be appropriately directed to the objects OB5 to OB9 that need to be noticed.

[0118] As a sixth modification of the first embodiment, display control for a plurality of displays including the display 18 may be performed. For example, in addition to the display 18, the display control system 2 may have displays 21 to 23 shown by dotted lines in FIGS. 3 and 4. Display areas DA2 and DA3 of the displays 21 and 22 may be provided on the pillars 108 that support the roof portion 107 of the vehicle body 100a shown in FIG. 2 on both sides in the Y direction of the windshield 101. In addition, a display area DA4 of the display 23 may be provided on the -Z side of the windshield 101.

[0119] At this time, display control as shown in FIG. 12 may be performed. FIG. 12 is a diagram showing a display control method according to the sixth modification of the first embodiment.

[0120] For example, assume that a pattern of the attention state as shown by the image IM2 in FIG. 12(b) and the map information MP in FIG. 12(c) is included in advance in the registered pattern 81.

[0121] In S1 of FIG. 5, the imaging range VF2 is imaged, and an image IM1 as shown in FIG. 12(a) is acquired. The image IM1 shows a situation where there are no objects on the road surface 600 for a certain period of time.

[0122] In S2 of FIG. 5, a display object 401 is added to the display area DA1 in the image IM1 with the luminance BR1, a display object is added to the display area DA2 with the luminance BR11, and an image IM2 as shown in FIG. 12(b) is generated.

[0123] In S3 of FIG. 5, the attention estimation model 7 is applied to the image IM2, the attention state of the driver 200 is estimated, and map information MP as shown in FIG. 12(c) is generated. The map information MP corresponds to the image IM2. In the map information MP, by referring to the image IM2, it is shown that patterns PT9 and PT10 with a novelty level equal to or higher than the threshold level Lth1 are respectively included at positions corresponding to a plurality of display areas DA1 and DA2. The threshold level Lth1 can be determined experimentally in advance as the novelty level corresponding to the situation where attention is overly concentrated.

[0124] In S5 of FIG. 5, it is determined that the pattern of the attention state specified by the image IM2 in FIG. 12(b) and the map information MP in FIG. 12(c) matches the registered pattern 81.

[0125] In S7 of FIG. 5, the luminance of the display object 400 in the display area DA1 is changed to a luminance BR2 lower than the luminance BR1, and the luminance of the display object in the display area DA2 is changed to a luminance BR12 lower than the luminance BR11. The displays 18 and 21 are respectively controlled with the luminances BR2 and BR12. As shown in FIG. 6(d), the display object 402 is displayed with the luminance BR2 in the display area DA1 in the visual field VF1 by the display 18, and the display object is displayed with the luminance BR12 in the display area DA2 by the display 21.

[0126] Even with such display control, when the display objects in the plurality of display areas DA1 and DA2 are displayed with the brightnesses BR1 and BR11 and excessive attention is concentrated on the plurality of display objects in the visual field VF1, the display objects in the display areas DA1 and DA2 can be displayed in the display areas DA1 and DA12 with the brightnesses BR2 and BR12, where the attention is suppressed more than the brightnesses BR1 and BR11. As a result, the degree of attention concentration on the plurality of display objects in the visual field VF1 can be kept within the allowable range respectively.

[0127] As a seventh modification of the first embodiment, display control as shown in FIG. 13 may be performed. FIG. 13 is a diagram showing a display control method according to the seventh modification of the first embodiment.

[0128] For example, assume that a pattern of the attention state as shown by the image IM2 in FIG. 13(b) and the map information MP in FIG. 13(c) is included in advance in the registered pattern 81.

[0129] In S1 of FIG. 5, the imaging range VF2 is imaged, and an image IM1 as shown in FIG. 13(a) is acquired. The image IM1 shows a situation where the objects OB10 and OB11 to be noted exist between the display areas DA1 and DA2 on the road surface 600.

[0130] In S2 of FIG. 5, a display object 415 is added to the display area DA1 in the image IM1 with the brightness BR1, and a display object is added to the display area DA2 with the brightness BR11, and an image IM2 as shown in FIG. 13(b) is generated.

[0131] In S3 of FIG. 5, the attention estimation model 7 is applied to the image IM2, the attention state of the driver 200 is estimated, and map information MP as shown in FIG. 13(c) is generated. The map information MP corresponds to the image IM2. In the map information MP, by referring to the image IM2, it is shown that patterns PT11 and PT12 with a novelty level equal to or higher than the threshold level Lth1 are included at positions corresponding to the display areas DA1 and DA2. The threshold level Lth1 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is overly concentrated. Also, in the map information MP, by referring to the image IM2, it is shown that a pattern PT13 with a novelty level equal to or higher than the threshold level Lth2 is included at a position between the display areas DA1 and DA2. The threshold level Lth2 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is required.

[0132] In S5 of FIG. 5, it is determined that the pattern of the attention state specified by the image IM2 in FIG. 13(b) and the map information MP in FIG. 13(c) matches the registered pattern 81.

[0133] In S7 of FIG. 5, the luminance of the display object in the display area DA1 is changed to a luminance BR2 lower than the luminance BR1, and the luminance of the display object in the display area DA2 is changed to a luminance BR12 lower than the luminance BR11. The displays 18 and 21 are controlled by the luminances BR2 and BR12. As shown in FIG. 13(d), the display object 416 is displayed at the luminance BR2 in the display area DA1 in the visual field VF1 by the display 18, and the display object is displayed at the luminance BR12 in the display area DA2 by the display 21.

[0134] Even with such display control, when the display objects in the plurality of display areas DA1 and DA2 are displayed with the brightness BR1 and BR11 and excessive attention is concentrated on the plurality of display objects in the visual field VF1, the display objects in the display areas DA1 and DA2 can be displayed in the display areas DA1 and DA12 with the brightness BR2 and BR12, where the attention is suppressed more than the brightness BR1 and BR11. As a result, the degree of attention concentration on the plurality of display objects in the visual field VF1 can be kept within the allowable range respectively, and the attention can be appropriately paid to the objects OB10 and OB11 that should be noted.

[0135] (Second Embodiment) Next, a display control method according to the second embodiment will be described. Hereinafter, the description will focus on the parts different from the first embodiment.

[0136] In the first embodiment, the change in the display form of the display object according to the estimation result of the attention state is exemplified. In the second embodiment, the evaluation for determining the display form to be changed will be exemplified. For example, when attention should be paid to an object (e.g., a pedestrian) in the actual scene of the forward visual field that the driver should see, it is objectively evaluated whether the display object to be displayed on the display affects the driver's attention state.

[0137] As shown in FIG. 14, the display control device 101 of the display control system 102 further includes an evaluation unit 124 and a generation unit 125 with respect to the display control device 1 (see FIG. 4). FIG. 14 is a diagram showing the functional configuration of the display control device 101.

[0138] In the display control system 102, each part of the display control device 101 shown in FIG. 14 may be realized hardware-wise (e.g., as a circuit), software-wise, or some parts may be realized hardware-wise and the rest software-wise. When each part shown in FIG. 14 is realized software-wise, the CPU 13 (see FIG. 3) may execute the program 17 to functionally construct each part shown in FIG. 14 on the volatile memory unit 14 either collectively at compile time or sequentially as the processing progresses.

[0139] The display control device 101 operates in the same manner as in the first embodiment after shipment, but performs an evaluation for determining the display form before shipment.

[0140] For example, in the display control device 101, the simulation unit 5 supplies the images IM1 and IM2 to the attention estimation unit 6. The attention estimation unit 6 applies one or more attention estimation models 7 to each of the images IM1 and IM2 to estimate the attention state of the driver 200.

[0141] When the attention estimation model 7 is the novelty estimation model 7_1, the attention estimation unit 6 inputs the visual field images at a plurality of past time points and the image IM1 to the novelty estimation model 7_1. The novelty estimation model 7_1 generates a current predicted image from the visual field images at a plurality of past time points, two-dimensionally obtains a prediction error regarding human cognition based on the image IM1 and the predicted image, and generates map information MP1 indicating the distribution of the two-dimensional prediction error. The novelty estimation model 7_1 outputs the map information MP1 to the attention estimation unit 6. The attention estimation unit 6 supplies the estimation result of the attention state including the image IM1 and the map information MP1 to the evaluation unit 124 as the estimation result of the attention state with respect to the image IM1.

[0142] Similarly, the attention estimation unit 6 inputs the visual field images at a plurality of past time points and the image IM2 into the novelty estimation model 7_1. The novelty estimation model 7_1 generates a current predicted image from the visual field images at a plurality of past time points, two-dimensionally obtains a prediction error related to human perception based on the image IM2 and the predicted image, and generates map information MP2 indicating the distribution of the two-dimensional prediction error. The novelty estimation model 7_1 outputs the map information MP2 to the attention estimation unit 6. The attention estimation unit 6 supplies the estimation result of the attention state including the image IM2 and the map information MP2 to the display determination unit 8 as the estimation result of the attention state with respect to the image IM2.

[0143] The evaluation unit 124 evaluates the influence on the attention state of the image IM1 of the display object 400 according to the estimation result of the attention state with respect to the image IM1 and the estimation result of the attention state with respect to the image IM2. The evaluation unit 124 performs convolution integration on the values in the map information MP1 for the area near the object to be noted and the area near the display object 400. The evaluation unit 124 performs convolution integration on the values in the map information MP2 for the area near the object to be noted and the area near the display object 400. The evaluation unit 124 may calculate the influence degree of the display object 400 on the image IM1 by dividing the difference between the integrated value of the map information MP1 and the integrated value of the map information MP2 by the maximum integrated value in the map information MP1. The evaluation unit 124 supplies the evaluation result to the generation unit 125. The evaluation result may include the pattern of the attention state corresponding to the display object 400, the display form of the display object 400, and the influence degree thereby.

[0144] The generation unit 125 determines a display form according to the evaluation result and generates display form information 82. If the degree of influence included in the evaluation result is within the allowable range, the generation unit 125 may generate the display form information 82 so as to include the display form of the display object 400 included in the evaluation result in association with the pattern of the attention state included in the evaluation result. Alternatively, if the generation unit 125 receives an instruction to adopt the display object and its display form according to the evaluation result, the generation unit 125 may generate the display form information 82 so as to include the display form of the display object 400 included in the evaluation result in association with the pattern of the attention state included in the evaluation result. Thereby, the generation unit 125 can generate the display form information so as to include a display form DF2 in which attention is suppressed compared to the display form DF1 of the image IM1.

[0145] The generation unit 125 supplies the display form information to the display determination unit 8. In response thereto, the display determination unit 8 may generate the display form information 82. Alternatively, if the display determination unit 8 already holds the display form information 82, the display determination unit 8 updates the display form information 82 with the supplied display form information. For example, the display determination unit 8 may update the display form information 82 by adding the supplied display form information to the display form information 82.

[0146] Also, as shown in FIG. 15, the display control method executed in the display control system 102 is different from the first embodiment in the following points. FIG. 15 is a flowchart showing the display control method according to the second embodiment.

[0147] In the display control system 102, the display control device 101 captures the imaging range VF2 with the imaging sensor 11 and acquires the image IM1 as a field-of-view image from the imaging sensor 11 (S11).

[0148] For example, the imaging range VF2 is imaged, and an image IM1 as shown in FIG. 16(a) is acquired. FIG. 16 is a diagram showing a display control method according to the second embodiment. The image IM1 shows a situation where the object OB13 to be noted exists at a position away from the display area DA1 on the road surface 600. In FIG. 16(a), for simplicity, a portion near the display area DA1 in the image IM1 is illustrated.

[0149] Alternatively, the imaging range VF2 is imaged, and an image IM1 as shown in FIG. 17(a) is acquired. FIG. 17 is a diagram showing a display control method according to the second embodiment. The image IM1 shows a situation where the object OB13 to be noted exists at a position away from the display area DA1 on the road surface 600. In FIG. 17(a), for simplicity, a portion near the display area DA1 in the image IM1 is illustrated.

[0150] After S11, the processes of S12 to S13 and the process of S14 are performed in parallel.

[0151] In the processes of S12 to S13, the display control device 101 adds the display object 400 in the display form DF1 to the display area DA1 in the image IM1 and generates the image IM2 as a simulation image (S2).

[0152] For example, the display object 417 is added to the display area DA1 in the image IM1 with the luminance BR1, and an image IM2 as shown in FIG. 16(b) is generated. In FIG. 16(b), for simplicity, a portion near the display area DA1 in the image IM2 is illustrated.

[0153] Alternatively, the display object 418 is added to the display area DA1 in the image IM1 with the luminance BR2, and an image IM2 as shown in FIG. 17(b) is generated. In FIG. 17(b), for simplicity, a portion near the display area DA1 in the image IM2 is illustrated.

[0154] The display control device 101 applies the attention estimation model 7 to the image IM2 to estimate the attention state of the driver 200 (S3) and generates an estimation result of the attention state for the image IM2.

[0155] For example, the attention estimation model 7 is applied to the image IM2 in FIG. 16(b), the attention state of the driver 200 is estimated, and map information MP2 as shown in FIG. 16(c) is generated. The map information MP2 corresponds to the image IM2. In the map information MP2, by referring to the image IM2, it is shown that a pattern PT14 with a novelty level equal to or higher than the threshold level Lth1 is included at a position corresponding to the display area DA1. The threshold level Lth1 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is overly concentrated. Also, in the map information MP2, by referring to the image IM2, it is shown that a pattern PT15 with a novelty level equal to or higher than the threshold level Lth2 is included at a position away from the display area DA1. The threshold level Lth2 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is required.

[0156] Alternatively, the attention estimation model 7 is applied to the image IM2 in FIG. 17(b), the attention state of the driver 200 is estimated, and map information MP2 as shown in FIG. 17(c) is generated. The map information MP2 corresponds to the image IM2. In the map information MP2, by referring to the image IM2, it is shown that a pattern PT14b with a novelty level equal to or higher than the threshold level Lth2 is included at a position corresponding to the display area DA1. The threshold level Lth2 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is required. Also, in the map information MP2, by referring to the image IM2, it is shown that a pattern PT15b with a novelty level equal to or higher than the threshold level Lth1 is included at a position away from the display area DA1. The threshold level Lth1 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is overly concentrated.

[0157] On the other hand, in the process of S14, the display control device 101 applies the attention estimation model 7 to the image IM1 to estimate the attention state of the driver 200 and generates an estimation result of the attention state with respect to the image IM1.

[0158] For example, the attention estimation model 7 is applied to the image IM1 in FIG. 16(a), the attention state of the driver 200 is estimated, and map information MP1 as shown in FIG. 16(d) is generated. The map information MP1 corresponds to the image IM1. In the map information MP1, by referring to the image IM1, it is shown that a pattern PT14a with a novelty level equal to or higher than the threshold level Lth2 is included at a position corresponding to the display area DA1. The threshold level Lth1 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is overly concentrated. Also, in the map information MP1, by referring to the image IM1, it is shown that a pattern PT15a with a novelty level equal to or higher than the threshold level Lth2 is included at a position away from the display area DA1. The threshold level Lth2 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is required.

[0159] Alternatively, the attention estimation model 7 is applied to the image IM1 in FIG. 17(a), the attention state of the driver 200 is estimated, and map information MP1 as shown in FIG. 17(d) is generated. The map information MP1 corresponds to the image IM1. In the map information MP1, by referring to the image IM1, it is shown that a pattern PT14a with a novelty level equal to or higher than the threshold level Lth2 is included at a position corresponding to the display area DA1. The threshold level Lth1 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is overly concentrated. Also, in the map information MP1, by referring to the image IM1, it is shown that a pattern PT15a with a novelty level equal to or higher than the threshold level Lth2 is included at a position away from the display area DA1. The threshold level Lth2 can be experimentally determined in advance as the novelty level corresponding to the situation where attention is required.

[0160] When both the processes of S12 to S13 and the process of S14 are completed, the display control device 101 evaluates the influence on the attention state of the display object 400 with respect to the image IM1 according to the estimation result of the attention state with respect to the image IM1 and the estimation result of the attention state with respect to the image IM2.

[0161] That is, the display control device 101 compares the estimation result of the attention state obtained in S11 with the estimation result of the attention state obtained in S3 (S15), and performs an evaluation process according to the comparison result (S16).

[0162] In the evaluation process (S16), the display control device 101 evaluates the influence on the attention state of the display object 400 according to the comparison result. The display control device 101 performs convolution integration on the values in the map information MP1 for the area near the object to be noted and the area near the display object 400. The display control device 101 performs convolution integration on the values in the map information MP2 for the area near the object to be noted and the area near the display object 400. The display control device 101 may calculate the influence degree on the image IM1 of the display object 400 by dividing the difference between the integrated value of the map information MP1 and the integrated value of the map information MP2 by the maximum integrated value in the map information MP1.

[0163] For example, the values in the map information MP2 shown in FIG. 16(c) are convolution integrated for the area near the object OB12 and the area near the display object 417. The values in the map information MP2 are convolution integrated for the area near the object OB12 and the area near the display object 417. The difference between the integrated value of the map information MP1 and the integrated value of the map information MP2 is divided by the maximum integrated value in the map information MP1, and as shown in FIG. 16(e), the influence degree on the image IM1 of the display object 400 is calculated. In FIG. 16(e), an influence degree of 70% is exemplified as the evaluation result.

[0164] FIG. 16 shows an example in which, when there is no display object 417, pattern PT15a with a novelty level equal to or higher than the threshold level Lth2 corresponds to a pedestrian moving, and pattern PT14a with a novelty level equal to or higher than the threshold level Lth2 corresponds to a "stop" road sign. In FIG. 16, when the display object 417 is superimposed on the real scene in the forward field of view, the fact that pattern PT15 with a novelty level equal to or higher than the threshold level Lth2 is displayed thinner than pattern PT15a with a novelty level equal to or higher than the threshold level Lth2 indicates that the degree of influence attracting attention to the area of pattern PT15a with a novelty level equal to or higher than the threshold level Lth2 corresponding to a pedestrian moving is decreasing. Also, the fact that pattern PT14 with a novelty level equal to or higher than the threshold level Lth1 has a larger area and is displayed darker than pattern PT14a with a novelty level equal to or higher than the threshold level Lth2 indicates that the degree of influence attracting attention to the area of pattern PT14a with a novelty level equal to or higher than the threshold level Lth2 corresponding to a road sign is extremely increasing.

[0165] From this, in a situation where pedestrians need to be paid more attention than road signs, the attention to pedestrians becomes less likely to be attracted than when the display object 417 is not displayed, and the attention to road signs that do not need to attract more attention than pedestrians (although road signs can be actively viewed by drivers based on their knowledge of traffic rules, it is difficult to pay attention to all pedestrians moving around) becomes more likely to be attracted than when the display object 417 is not displayed. When the influence degree of the display object 417 is 45% from the statistical distribution of the influence degree on pedestrians and 25% from the statistical distribution of the influence degree on road signs, the total can be 70%. Alternatively, without separating objects such as pedestrians and road signs, the sum of the product of the area and density of the region of pattern PT14a where the novelty level when the display object 417 is not displayed is above the threshold level Lth2 and the region of pattern PT15a where the novelty level is above the threshold level Lth2 is compared with the sum of the product of the area and density of the region of pattern PT14 where the novelty level when the display object 417 is displayed is above the threshold level Lth1 and the region of pattern PT15 where the novelty level is above the threshold level Lth2. Here, as the technology for separating objects, a technology known as scene segmentation technology may be used.

[0166] Alternatively, the values in the map information MP2 shown in FIG. 17(c) are convolutionally integrated for the region near the object OB12 and the region near the display object 417. The values in the map information MP2 are convolutionally integrated for the region near the object OB12 and the region near the display object 417. The difference between the integrated value of the map information MP1 and the integrated value of the map information MP2 is divided by the maximum integrated value in the map information MP1, and as shown in FIG. 17(e), the influence degree on the image IM1 of the display object 400 is calculated. In FIG. 17(e), as an evaluation result, an influence degree of 10% is exemplified.

[0167] FIG. 17 shows an example where, when there is no display object 418, pattern PT15a with a novelty level equal to or higher than the threshold level Lth2 corresponds to a pedestrian moving, and pattern PT14a with a novelty level equal to or higher than the threshold level Lth2 corresponds to a "Stop" road sign. In FIG. 17, when the display object 418 is superimposed on the real scene in the forward field of view, the fact that pattern PT15b with a novelty level equal to or higher than the threshold level Lth1 is displayed with substantially the same area and thickness as pattern PT15a with a novelty level equal to or higher than the threshold level Lth2 indicates that there is no change in the degree of attracting attention to the area of pattern PT15a corresponding to the pedestrian moving, where the novelty level is equal to or higher than the threshold level Lth2. Also, the fact that pattern PT14b with a novelty level equal to or higher than the threshold level Lth2 is displayed with substantially the same area and the same density as pattern PT14a with a novelty level equal to or higher than the threshold level Lth2 indicates that the degree of attracting attention to the area of pattern PT14a corresponding to the road sign, where the novelty level is equal to or higher than the threshold level Lth2, is substantially the same.

[0168] From this, in a situation where attention must be paid to the pedestrian rather than the road sign, the attention to the pedestrian remains unchanged when the display object 417 is not displayed, and the attention to the road sign, which does not need to attract more attention than the pedestrian, also remains unchanged when the display object 418 is not displayed. The influence degree of the display object 418 can be set to a total of 10% when it is 5% from the statistical distribution of the influence degree on the pedestrian and 5% from the statistical distribution of the influence degree on the road sign. Alternatively, without separating objects such as pedestrians and road signs, the sum of the products of the area and density of the regions of pattern PT14a with a novelty level equal to or higher than the threshold level Lth2 and pattern PT15a with a novelty level equal to or higher than the threshold level Lth2 when the display object 418 is not displayed may be compared with the sum of the products of the area and density of the regions of pattern PT14b with a novelty level equal to or higher than the threshold level Lth2 and pattern PT15b with a novelty level equal to or higher than the threshold level Lth1 when the display object 418 is displayed. Here, a technique for separating objects may use a technique known as a scene segmentation technique.

[0169] As shown in the examples of FIGS. 16 and 17, by comparing the influence degrees of variations of display objects with respect to the video of the actual scene in the forward field of view, it can be quantitatively evaluated objectively based on objective criteria that the display object 418 is superior to the display object 417.

[0170] The display control device 101 determines a display form according to the evaluation result in S13 and generates or updates display form information 82 (S17).

[0171] When the display form information 82 has not been generated, if the influence degree included in the evaluation result is within the allowable range, the display control device 101 may generate the display form information 82. The display control device 101 may generate display form information 82 including the display form of the display object 400 included in the evaluation result in association with the pattern of the attention state included in the evaluation result.

[0172] For example, assume that the allowable range of the influence degree is 0% or more and less than 15%. When the evaluation result includes "influence degree 70%" shown in FIG. 16(e), assuming that the influence degree included in the evaluation result is outside the allowable range, the display form (for example, luminance BR1) of the display object 417 is not adopted as the information to be included in the display form information 82.

[0173] Alternatively, when the evaluation result includes "influence degree 10%" shown in FIG. 17(e), assuming that the influence degree included in the evaluation result is within the allowable range, the display form (for example, luminance BR2) of the display object 418 is adopted as the information to be included in the display form information 82. The patterns of the attention states shown in FIGS. 16(b) and 16(c) are added as the registered pattern 18. Along with that, display form information 82 including the display form (for example, luminance BR2) of the display object 418 shown in FIG. 17(b) can be generated in association with the patterns of the attention states shown in FIGS. 16(b) and 16(c).

[0174] When the display form information 82 has been generated, if the degree of influence included in the evaluation result is within the allowable range, the display control device 101 may update the display form information 82. The display control device 101 may update the display form information 82 by adding the display form of the display object 400 included in the evaluation result corresponding to the pattern of the caution state included in the evaluation result.

[0175] For example, assume that the allowable range of the degree of influence is 0% or more and less than 15%. When the evaluation result includes "degree of influence 70%" shown in FIG. 16(e), since the degree of influence included in the evaluation result is out of the allowable range, the display form (for example, brightness BR1) of the display object 417 is not adopted as the information to be added to the display form information 82.

[0176] Alternatively, when the evaluation result includes "degree of influence 10%" shown in FIG. 17(e), since the degree of influence included in the evaluation result is within the allowable range, the display form (for example, brightness BR2) of the display object 418 is adopted as the information to be added to the display form information 82. The patterns of the caution states shown in FIGS. 16(b) and 16(c) are added as the registered pattern 18. At the same time, the display form (for example, brightness BR2) of the display object 418 shown in FIG. 17(b) is added corresponding to the patterns of the caution states shown in FIGS. 16(b) and 16(c), whereby the display form information 82 can be updated.

[0177] After that, the display control device 101 performs the processes of S1 to S7 in FIG. 5 (S18).

[0178] For example, when the pattern of the caution state matches the registered pattern 81 (Yes in S5), the display control device 101 refers to the display form information 82 generated or updated in S17 and determines the display form of the display object 400 as the display form DF2. The display form DF2 is a display form in which caution is suppressed compared to the display form DF1 of the simulation image in S2. The display control device 101 controls the display 18 to display the display object 400 in the display area DA1 in the passenger compartment 110 in the display form DF2 (S7).

[0179] As described above, in the second embodiment, the display control method acquires a field-of-view image, generates a simulation image from the field-of-view image, applies an attention estimation model to each of the simulation image and the field-of-view image, and evaluates the influence on the attention state by adding a display object in the display form DF2. As a result, when excessive attention is concentrated on the display object 400 in the field of view VF1 when the display object 400 is displayed in the display form DF1, according to the evaluation result of the display form DF2, the display form of the display object 400 with suppressed attention can be determined to be the display form DF2 from the display form DF1. As a result, display form information 82 including the determined display form DF2 can be generated. For example, when the pattern of the attention state matches the registered pattern 81, referring to the display form information 82, the display form of the display object 400 can be changed from the display form DF1 of the simulation image to the display form DF2 with more suppressed attention.

[0180] Note that, as a first modification example of the second embodiment, an evaluation for determining the display form may be further performed in consideration of the correlation between the driver's line of sight and the attention state. In this case, as shown in FIG. 18, the display control system 202 has a display control device 201 instead of the display control device 101 (see FIG. 14) and further has an imaging sensor 226. The display control device 201 further has a line-of-sight detection unit 227. FIG. 18 is a diagram showing the functional configuration of the display control device 201 according to the first modification example of the second embodiment.

[0181] The imaging sensor 226 has an imaging range VF12. The imaging range VF12 includes the pupil of the driver 200's eyeball. The imaging sensor 226 acquires an image of the pupil. The image of the pupil includes information about the direction of the driver 200's line of sight. The imaging sensor 226 supplies an image signal indicating the image of the pupil to the line-of-sight detection unit 227.

[0182] The line-of-sight detection unit 227 acquires an image signal from the imaging sensor 226. The line-of-sight detection unit 227 extracts line-of-sight information about the direction of the driver 200's line of sight from the image signal. The line-of-sight detection unit 227 supplies the line-of-sight information to the evaluation unit 224.

[0183] The evaluation unit 224 receives the line-of-sight information from the line-of-sight detection unit 227 and the estimation result of the attention state with respect to the image IM2 from the attention estimation unit 6. The evaluation unit 224 may generate correlation information 2241 indicating the correlation between the driver's line of sight and the driver's attention state according to the information about the direction of the driver 200's line of sight and the estimation result of the attention state with respect to the image IM2.

[0184] For example, it is assumed that in the image IM1 where the object OB13 to be noted is located at a position away from the display area DA1 on the road surface 600, a display object 418 is added to the display area DA1 and the image IM2 shown in FIG. 19(a) is generated. The evaluation unit 224 specifies the position PS1 of the viewpoint corresponding to the line of sight EL of the driver 200 according to the line-of-sight information. The evaluation unit 224 can obtain the position PS1 of the driver 200's viewpoint as the position of the intersection of the driver 200's line of sight EL and the virtual screen 500 (see FIG. 1). In FIG. 19(a), the case where the position PS1 of the viewpoint is on the object OB13 is illustrated. The evaluation unit 224 specifies the position PS11 of the center of the pattern PT17 whose novelty level in the map information MP2 shown in FIG. 19(e) is equal to or higher than the threshold level Lth1. The evaluation unit 224 may obtain the distance D1 between the position PS1 and the position PS11.

[0185] Similarly, the image IM2 shown in FIG. 19(b) is generated, and the evaluation unit 224 specifies the position PS2 of the viewpoint corresponding to the line of sight ELa of the driver 200 according to the line-of-sight information. In FIG. 19(b), the case where the position PS1 of the viewpoint is on the display object 418 is illustrated. The evaluation unit 224 specifies the position PS12 of the center of the pattern PT16a whose novelty level in the map information MP2 shown in FIG. 19(f) is equal to or higher than the threshold level Lth1. The evaluation unit 224 may obtain the distance D2 between the position PS2 and the position PS12.

[0186] The image IM2 shown in FIG. 19(c) is generated, and the evaluation unit 224 specifies the position PS3 of the viewpoint corresponding to the line of sight ELb of the driver 200 according to the line-of-sight information. In FIG. 19(c), a case where the position PS3 of the viewpoint is away from both the object OB13 and the display object 418 is illustrated. The evaluation unit 224 specifies that there is no pattern in the map information MP2 shown in FIG. 19(g) whose novelty level is equal to or higher than the threshold level Lth1.

[0187] The image IM2 shown in FIG. 19(d) is generated, and the evaluation unit 224 specifies the position PS4 of the viewpoint corresponding to the line of sight ELc of the driver 200 according to the line-of-sight information. In FIG. 19(d), a case where the position PS4 of the viewpoint is on the object OB13 is illustrated. The evaluation unit 224 specifies the position PS14 of the pattern PT16c in the map information MP2 shown in FIG. 19(h) whose novelty level is equal to or higher than the threshold level Lth1. The evaluation unit 224 may obtain the distance D4 between the position PS4 and the position PS14.

[0188] The evaluation unit 224 obtains the strength of the correlation between the line of sight of the driver 200 and the attention state of the driver according to the position of the viewpoint in the image IM2 shown in FIGS. 19(a), 19(b), and 19(d) and the position of the pattern in the map information MP2 shown in FIGS. 19(e), 19(f), and 19(h) whose novelty level is equal to or higher than the threshold level Lth1. Since there is no pattern in the map information MP2 shown in FIG. 19(g) whose novelty level is equal to or higher than the threshold level Lth1, the image IM2 shown in FIG. 19(c) and the map information MP2 shown in FIG. 19(g) may be excluded from the evaluation target. The evaluation unit 224 generates and holds the correlation information 2241 including the strength of the correlation.

[0189] The evaluation unit 224 may obtain the average distance AD1 by averaging the distances D1, D2, and D4 as a value indicating the strength of the correlation. The closer the average distance AD1 is to 0, the stronger the correlation between the line of sight of the driver 200 and the attention state of the driver. The evaluation unit 224 may generate the correlation information 2241 including the average distance AD1 and store it in the non-volatile memory unit 16 (see FIG. 3).

[0190] Also, the display control method executed by the display control system 202 differs from the second embodiment in the following points as shown in FIG. 20. FIG. 20 is a flowchart showing the display control method according to the first modification of the second embodiment.

[0191] After S11, the display control device 201 acquires line-of-sight information indicating the line of sight of the driver 200 (S21). For example, the imaging range VF12 is imaged by the imaging sensor 226 and an image of the pupil of the driver 200's eyeball is acquired, and line-of-sight information about the direction of the driver 200's line of sight is extracted from the image signal indicating the image of the pupil.

[0192] The display control device 201 acquires correlation information 2241 indicating the correlation between the driver's line of sight and the driver's attention state (S22). The display control device 201 may acquire the correlation information 2241 by reading it from the non-volatile storage unit 16.

[0193] After S22, the processes of S12 to S24 and the processes of S14 to S25 are performed in parallel.

[0194] In the processes of S12 to S24, the display control device 201 adds the display object 400 in the display form DF1 to the display area DA1 in the image IM1 to generate the image IM2 as a simulation image (S12).

[0195] The display control device 201 applies the attention estimation model 7 to the image IM2 to estimate the attention state of the driver 200 (S13) and generates an estimation result of the attention state for the image IM2. For example, the display control device 201 may apply the novelty estimation model 7_1 to the image IM2 to generate map information MP2 indicating a two-dimensional distribution of the prediction error corresponding to the image IM2, and generate an estimation result of the attention state including the image IM2 and the map information MP2.

[0196] The display control device 201 corrects the estimation result of the attention state in S13 according to the line-of-sight information in S21 and the correlation information in S22 (S23). The display control device 201 may correct the estimation result of the attention state by using the average distance AD1 included in the correlation information, the distance D5 between the position of the viewpoint corresponding to the line-of-sight EL included in the line-of-sight information and the position of the pattern included in the estimation result in S13. For example, the display control device 201 may subtract the distance D5 from the average distance AD1 to obtain a difference DF2, and correct the position of the pattern in the map information MP2 so that the difference DF2 is canceled, thereby correcting the estimation result of the attention state in S13.

[0197] The display control device 201 updates the correlation information (S24). For example, the display control device 201 may calculate the average distance AD2 by averaging the distances D1, D2, D4, and D5. The display control device 201 may generate correlation information 2241 including the average distance AD2 and store it in the non-volatile memory unit 16 (see FIG. 3) by overwriting. Thereby, the correlation information 2241 can be updated by overwriting.

[0198] On the other hand, in the processes of S14 to S25, the display control device 201 applies the attention estimation model 7 to the image IM1 to estimate the attention state of the driver 200 (S14), and generates an estimation result of the attention state for the image IM1. For example, the display control device 201 may apply the novelty estimation model 7_1 to the image IM2 to generate map information MP2 indicating a two-dimensional distribution of the prediction error corresponding to the image IM2, and generate an estimation result of the attention state including the image IM2 and the map information MP2.

[0199] The display control device 201 corrects the estimation result of the attention state in S14 according to the line-of-sight information in S21 and the correlation information in S22 (S25). The display control device 201 may correct the estimation result of the attention state by using the average distance AD1 included in the correlation information, the distance D6 between the position of the viewpoint corresponding to the line of sight EL included in the line-of-sight information, and the position of the pattern included in the estimation result in S14. For example, the display control device 201 may subtract the distance D6 from the average distance AD1 to obtain a difference DF1, and correct the position of the pattern in the map information MP1 so that the difference DF1 is canceled, thereby correcting the estimation result of the attention state in S14.

[0200] When all of the processes of S12 to S24 and the processes of S14 to S25 are completed, the display control device 201 evaluates the influence on the attention state of the image IM1 of the display object 400 according to the estimation result of the attention state with respect to the image IM1 and the estimation result of the attention state with respect to the image IM2.

[0201] That is, the display control device 201 compares the estimation result of the attention state corrected in S23 and the estimation result of the attention state corrected in S25 (S27), and performs an evaluation process according to the comparison result (S28).

[0202] In the evaluation process (S28), the display control device 201 calculates the degree of influence on the image IM1 of the display object 400 according to the comparison result (S29). The display control device 201 performs convolution integration on the values in the map information MP1 for the area near the object to be noted and the area near the display object 400. The display control device 201 performs convolution integration on the values in the map information MP2 for the area near the object to be noted and the area near the display object 400. The display control device 201 may calculate the degree of influence on the image IM1 of the display object 400 by dividing the difference between the integrated value of the map information MP1 and the integrated value of the map information MP2 by the maximum integrated value in the map information MP1.

[0203] If the degree of influence is less than the threshold value (Yes in S30), the display control device 201 determines the display form according to the evaluation result in S28, and generates or updates the display form information 82 (S17).

[0204] If the degree of influence is greater than or equal to the threshold value (No in S30), the display control device 201 skips S17.

[0205] After that, the display control device 201 performs the processes of S1 to S7 in FIG. 5 (S18).

[0206] Even with such a display control method, when the display object 400 is displayed in the display form DF1 and attention is excessively concentrated on the display object 400 in the visual field VF1, the display form of the display object 400 in which attention is suppressed more than the display form DF1 can be determined as the display form DF2 according to the evaluation result.

[0207] Further, as a second modification of the second embodiment, an evaluation may be performed to determine the display form by further considering the correlation between the driver's subjective evaluation and the attention state. In this case, as shown in FIG. 21, the display control system 302 has a display control device 301 instead of the display control device 101 (see FIG. 14). The display control device 301 further has an input unit 328. FIG. 21 is a diagram showing the functional configuration of the display control device 301 according to the second modification of the second embodiment.

[0208] The input unit 328 receives subjective evaluation information regarding the subjective evaluation of the attention state in the visual field image from the driver 200. The input unit 328 supplies the subjective evaluation information to the evaluation unit 224.

[0209] The evaluation unit 224 receives the subjective evaluation information from the input unit 328 and receives the estimation result of the attention state for the image IM2 from the attention estimation unit 6. The evaluation unit 224 may generate correlation information 3241 indicating the correlation between the driver's subjective evaluation and the driver's attention state according to the subjective evaluation information and the estimation result of the attention state for the image IM2.

[0210] For example, in an image IM1 where an object OB13 to be noted exists at a position away from a display area DA1 on a road surface 600, it is assumed that a display object 418 is added to the display area DA1 in the image IM1, and an image IM2 shown in FIG. 19(a) is generated. It is assumed that the subjective evaluation information indicates that the driver 200's attention is concentrated on the object OB13. The evaluation unit 224 specifies that a pattern PT17 in which the novelty level in the map information MP2 shown in FIG. 19(e) is equal to or higher than a threshold level Lth1 corresponds to the object OB13. The evaluation unit 224 evaluates that the driver's subjective evaluation corresponds to the attention state estimated by the evaluation unit 224 based on the image IM2 shown in FIG. 19(a), the map information MP2 shown in FIG. 19(e), and the subjective evaluation information, and that the correlation between the two is strong.

[0211] Similarly, it is assumed that an image IM2 shown in FIG. 19(b) is generated. It is assumed that the subjective evaluation information indicates that the driver 200's attention is concentrated on the display object 418. The evaluation unit 224 specifies that a pattern PT16a in which the novelty level in the map information MP2 shown in FIG. 19(f) is equal to or higher than the threshold level Lth1 corresponds to the display object 418. The evaluation unit 224 evaluates that the driver's subjective evaluation corresponds to the attention state estimated by the evaluation unit 224 based on the image IM2 shown in FIG. 19(b), the map information MP2 shown in FIG. 19(f), and the subjective evaluation information, and that the correlation between the two is strong. The evaluation unit 224 may divide the number of times the two match by the number of evaluation times to obtain a coincidence probability as a value indicating the strength of the correlation between the two.

[0212] The evaluation unit 224 may generate correlation information 2241 indicating that the correlation between the driver's subjective evaluation and the attention state is strong and store it in the non-volatile memory unit 16 (see FIG. 3). The evaluation unit 224 may generate correlation information 2241 including the coincidence probability and store it in the non-volatile memory unit 16.

[0213] In addition, as shown in FIG. 22, the display control method executed by the display control system 202 is different from the second embodiment in the following points. FIG. 22 is a flowchart showing a display control method according to a second modification of the second embodiment.

[0214] After S11, the display control device 301 acquires subjective evaluation information indicating the subjective evaluation of the driver 200 (S31). For example, the subjective evaluation information is acquired by receiving from the driver 200 subjective evaluation information regarding the subjective evaluation of the attention state in the field of view image.

[0215] The display control device 301 acquires correlation information 3241 indicating the correlation between the subjective evaluation of the driver and the attention state of the driver (S32). The display control device 301 may acquire the correlation information 3241 by reading it from the non-volatile storage unit 16.

[0216] After S32, the processes of S12 to S34 and the processes of S14 to S35 are performed in parallel.

[0217] In the processes of S12 to S34, the display control device 201 adds the display object 400 in the display area DA1 in the image IM1 in the display form DF1 to generate the image IM2 as a simulation image (S12).

[0218] The display control device 301 applies the attention estimation model 7 to the image IM2 to estimate the attention state of the driver 200 (S13) and generates an estimation result of the attention state for the image IM2. For example, the display control device 301 may apply the novelty estimation model 7_1 to the image IM2 to generate map information MP2 indicating a two-dimensional distribution of the prediction error corresponding to the image IM2, and generate an estimation result of the attention state including the image IM2 and the map information MP2.

[0219] The display control device 301 corrects the estimation result of the attention state in S13 according to the subjective evaluation information in S31 and the correlation information in S32 (S33). When the correlation information indicates that the correlation between the driver's subjective evaluation and the estimation result of the attention state is strong (for example, when the coincidence probability is equal to or higher than a predetermined value), the display control device 301 may correct the estimation result of the attention state so as to correspond to the subjective evaluation information in S31. For example, if the object being attended to in the estimation result of the attention state matches the subjective evaluation information, the display control device 301 may leave the estimation result of the attention state in S13 as it is. If the object being attended to in the estimation result of the attention state does not match the subjective evaluation information, the display control device 301 may change the estimation result of the attention state in S13 so as to match the subjective evaluation information.

[0220] The display control device 301 updates the correlation information (S34). For example, the display control device 301 may increment the number of evaluations, divide the number of times the two match by the number of evaluations to obtain a coincidence probability, generate correlation information 3241 including the coincidence probability, and overwrite and store it in the non-volatile storage unit 16. Thereby, the correlation information 3241 can be updated by overwriting.

[0221] On the other hand, in the processes of S14 to S35, the display control device 301 applies the attention estimation model 7 to the image IM1 to estimate the attention state of the driver 200 (S14), and generates an estimation result of the attention state for the image IM1. For example, the display control device 301 may apply the novelty estimation model 7_1 to the image IM2 to generate map information MP2 indicating a two-dimensional distribution of the prediction error corresponding to the image IM2, and generate an estimation result of the attention state including the image IM2 and the map information MP2.

[0222] The display control device 301 corrects the estimation result of the attention state in S14 according to the subjective evaluation information in S31 and the correlation information in S32 (S35). When the correlation information indicates that the correlation between the driver's subjective evaluation and the estimation result of the attention state is strong (for example, when the coincidence probability is equal to or greater than a predetermined value), the display control device 301 may correct the estimation result of the attention state so as to correspond to the subjective evaluation information in S31. For example, if the object being attended to in the estimation result of the attention state matches the subjective evaluation information, the display control device 301 may leave the estimation result of the attention state in S14 as it is. If the object being attended to in the estimation result of the attention state does not match the subjective evaluation information, the display control device 301 may change the estimation result of the attention state in S14 so as to match the subjective evaluation information.

[0223] When all of the processes of S12 to S34 and the processes of S14 to S35 are completed, the display control device 301 evaluates the influence on the attention state of the image IM1 of the display object 400 according to the estimation result of the attention state for the image IM1 and the estimation result of the attention state for the image IM2.

[0224] That is, the display control device 301 compares the estimation result of the attention state corrected in S33 and the estimation result of the attention state corrected in S35 (S37), and performs an evaluation process according to the comparison result (S38).

[0225] In the evaluation process (S38), the display control device 301 calculates the degree of influence on the image IM1 of the display object 400 according to the comparison result (S39). The display control device 301 performs convolution integration on the areas near the object to be noted and the area near the display object 400 in the map information MP1. The display control device 301 performs convolution integration on the areas near the object to be noted and the area near the display object 400 in the map information MP2. The display control device 301 may calculate the degree of influence on the image IM1 of the display object 400 by dividing the difference between the integrated value of the map information MP1 and the integrated value of the map information MP2 by the maximum integrated value in the map information MP1.

[0226] If the degree of influence is less than the threshold value (Yes in S40), the display control device 301 determines the display form according to the evaluation result in S38, and generates or updates the display form information 82 (S17).

[0227] If the degree of influence is greater than or equal to the threshold value (No in S40), the display control device 301 skips S17.

[0228] After that, the display control device 301 performs the processes of S1 to S7 in FIG. 5 (S18).

[0229] Even with such a display control method, when the display object 400 is displayed in the display form DF1 and attention is excessively concentrated on the display object 400 in the visual field VF1, according to the evaluation result, the display form of the display object 400 in which attention is suppressed more than the display form DF1 can be determined as the display form DF2.

[0230] Further, as a third modification of the second embodiment, display control as shown in FIG. 23 may be performed. FIG. 23 is a flowchart showing a display control method according to the third modification of the second embodiment.

[0231] In the display control system 302, the display control device 301 selects an image to be evaluated among a plurality of images IM1 captured by the imaging sensor 11 as the visual field image (S41). For example, the display control device 301 acquires a plurality of images IM1 captured by the imaging sensor 11. The display control device 301 may receive a selection instruction to select one image IM1 among the plurality of images IM1, and select the image IM1 selected by the selection instruction as the visual field image.

[0232] For example, an image display screen 10a and an information display screen 10b may be displayed on the display unit 10 (see FIG. 3) shown in FIG. 24(a). FIG. 24 is a diagram showing a display control method according to the third modification of the second embodiment. The information display screen 10b includes a scene input field 10b1, a content input field 10b2, and a driving influence display field 10b3.

[0233] The scene input field 10b1 can receive a selection instruction for selecting one of the plurality of images IM1 as the scene to be evaluated. The scene input field 10b1 may display thumbnail images of the plurality of images IM1 in a drop-down manner according to the selection operation of the scene input field 10b1, and may receive a selection instruction for the image IM1 corresponding to the thumbnail image according to the selection operation of the thumbnail image. Identification information (for example, 〇〇 right fork) of the image IM1 selected by the selection instruction may be displayed in the scene input field 10b1.

[0234] The display control device 301 selects display content (S42). The display content includes a display object and its display form. For example, a plurality of display contents are pre-generated and stored in the non-volatile storage unit 16 (see FIG. 3). The display control device 301 may receive a selection instruction for selecting one of the plurality of display contents, and select the display content selected by the selection instruction as the display content to be evaluated.

[0235] For example, in the display unit 10 shown in FIG. 24(a), the content input field 10b2 of the information display screen 10b can receive a selection instruction for selecting one of the plurality of display contents as the pattern to be evaluated. The content input field 10b2 may display thumbnail images of the plurality of display contents in a drop-down manner according to the selection operation of the content input field 10b2, and may receive a selection instruction for the display content corresponding to the thumbnail image according to the selection operation of the thumbnail image. Identification information (for example, A pattern) of the display content selected by the selection instruction may be displayed in the content input field 10b2.

[0236] After S42, the processes of S43 to S44 and the process of S45 are performed in parallel.

[0237] In the processes of S43 to S44, the display control device 301 adds the display content of S42 to the display area DA1 in the image IM1 to generate an image IM2 as a simulation image (S43). The display content includes displaying the display object 418 in the display form DF2.

[0238] For example, in the display unit 10 shown in FIG. 24(b), the image IM2 is displayed on the image display screen 10a. In the display area DA1 of the image IM2, the display object 418 is displayed in the display form DF2.

[0239] The display control device 301 applies the attention estimation model 7 to the image IM2 to estimate the attention state of the driver 200 (S44), and generates an estimation result of the attention state for the image IM2. For example, the display control device 301 may apply the novelty estimation model 7_1 to the image IM2 to generate map information MP2 indicating a two-dimensional distribution of prediction errors corresponding to the image IM2, and generate an estimation result of the attention state including the image IM2 and the map information MP2.

[0240] For example, the attention estimation model 7 is applied to the image IM2 in FIG. 24(b), the attention state of the driver 200 is estimated, and map information MP2 as shown in FIG. 24(c) is generated. In the map information MP2, by referring to the image IM2, it is shown that the pattern PT16b with a novelty level equal to or higher than the threshold level Lth2 is included at a position away from the display area DA1. The threshold level Lth2 can be determined experimentally in advance as the novelty level corresponding to the need for attention.

[0241] On the other hand, in the process of S45, the display control device 301 applies the attention estimation model 7 to the image IM1 to estimate the attention state of the driver 200, and generates an estimation result of the attention state for the image IM1.

[0242] For example, the attention estimation model 7 is applied to the image IM1 in FIG. 24(a), the attention state of the driver 200 is estimated, and map information MP1 as shown in FIG. 24(d) is generated. In the map information MP1, by referring to the image IM1, it is shown that a pattern PT16b with a novelty level equal to or higher than the threshold level Lth2 is included at a position away from the display area DA1. The threshold level Lth2 can be experimentally determined in advance as the novelty level corresponding to the need for attention.

[0243] When the processes of S43 to S44 and the process of S45 are both completed, the display control device 301 evaluates the influence on the attention state of the image IM1 of the display object 400 according to the estimation result of the attention state for the image IM1 and the estimation result of the attention state for the image IM2.

[0244] That is, the display control device 101 compares the estimation result of the attention state obtained in S11 with the estimation result of the attention state obtained in S3 (S46), and performs an evaluation process according to the comparison result (S47).

[0245] In the evaluation process (S47), the display control device 301 calculates the degree of influence on the image IM1 of the display content according to the comparison result (S48), and displays the calculated degree of influence (S49).

[0246] For example, in the display unit 10 shown in FIG. 24(e), the degree of influence is displayed in the driving influence display column 10b3. In FIG. 24(e), the case where the degree of influence "10%" is displayed in the driving influence display column 10b3 is illustrated.

[0247] The display control device 301 receives an instruction regarding the adoption of the display content, and determines whether to adopt the display content according to the instruction (S50). When the display control device 301 receives an instruction indicating the adoption of the display content, assuming that it adopts the display content (Yes in S50), according to the evaluation result of S47, it determines the display form and generates or updates the display form information 82 (S17).

[0248] When the display control device 301 receives an instruction indicating non-adoption of the display content, it skips S17 on the assumption that it will not adopt the display content (No in S50).

[0249] After that, the display control device 301 performs the processes of S1 to S7 in FIG. 5 (S18).

[0250] Even with such a display control method, when the display object 400 is displayed in the display form DF1 and excessive attention is concentrated on the display object 400 in the visual field VF1, according to the evaluation result, the display form of the display object 400 in which attention is suppressed more than the display form DF1 can be determined as the display form DF2.

[0251] Also, as a fourth modification of the second embodiment, display control as shown in FIG. 25 may be performed. FIG. 25 is a flowchart showing a display control method according to the fourth modification of the second embodiment.

[0252] In the display control system 302, the display control device 301 sets an initial value of 0 for a parameter N for counting the number of loop iterations, and after performing S41 and S42 (see FIG. 23), selects a display timing (S51). For example, a plurality of display contents are pre-generated and stored in the non-volatile storage unit 16 (see FIG. 3). Each display content includes a display object and its display form, and the display form may include a display timing. The display control device 301 may receive an instruction for selecting one display timing from among a plurality of display timings, and select the display timing selected by the instruction as the display timing to be evaluated.

[0253] For example, in the display unit 10 shown in Fig. 26(a), the information display screen 10c includes a content input field 10b1, a timing input field 10b2, and a driving influence display field 10c3. In the content input field 10b2, the identification information of the display content selected according to the content selection instruction in S42 (for example, A pattern) can be displayed. The display content may include a display form DF2. The timing input field 10b2 can accept a bar movement operation and a bar thickness selection operation. By the bar movement operation, the timing when the display object 400 starts to be displayed can be changed from the immediately following timing to a timing delayed by a time Δt. By the bar thickness selection operation, the display time ΔT for displaying the display object 400 can be changed. In Fig. 26(a), a case is illustrated where the bar thickness selection operation is performed so that the display time ΔT becomes relatively short. The display form DF2 includes a relatively short display time ΔT.

[0254] Alternatively, in the display unit 10 shown in Fig. 27(a), the information display screen 10c includes a content input field 10b1, a timing input field 10b2, and a driving influence display field 10c3. In the content input field 10b2, the identification information of the display content selected according to the content selection instruction in S42 (for example, B pattern) can be displayed. The display content may include a display form DF1. In Fig. 27(a), a case is illustrated where the bar thickness selection operation is performed so that the display time ΔT becomes relatively long. The display form DF1 includes a relatively long display time ΔT.

[0255] After S51, the processes of S43 to S44 and the process of S45 are performed in parallel.

[0256] In the processes of S43 to S44, the display control device 301 adds the display content of S42 to the display area DA1 in the image IM1 to generate an image IM2 as a simulation image (S43).

[0257] For example, in the display unit 10 shown in FIG. 26(b), an image IM2 is displayed on the image display screen 10a. In the display area DA1 of the image IM2, a display object 418 is displayed in a display form DF2. The display form DF2 includes a relatively short display time ΔT.

[0258] Alternatively, in the display unit 10 shown in FIG. 27(b), an image IM2 is displayed on the image display screen 10a. In the display area DA1 of the image IM2, a display object 418 is displayed in a display form DF1. The display form DF1 includes a relatively long display time ΔT.

[0259] The display control device 301 applies the attention estimation model 7 to the image IM2 to estimate the attention state of the driver 200 (S44), and generates an estimation result of the attention state for the image IM2. For example, the display control device 301 may apply the novelty estimation model 7_1 to the image IM2 to generate map information MP2 indicating a two-dimensional distribution of prediction errors corresponding to the image IM2, and generate an estimation result of the attention state including the image IM2 and the map information MP2.

[0260] For example, the attention estimation model 7 is applied to the image IM2 in FIG. 26(b), the attention state of the driver 200 is estimated, and map information MP2 as shown in FIG. 26(c) is generated. In the map information MP2, by referring to the image IM2, it is shown that a pattern PT16b with a novelty level equal to or higher than the threshold level Lth2 is included at a position away from the display area DA1. The threshold level Lth2 can be determined experimentally in advance as a novelty level corresponding to the need for attention.

[0261] Alternatively, the attention estimation model 7 is applied to the image IM2 in FIG. 27(b), the attention state of the driver 200 is estimated, and map information MP2 as shown in FIG. 27(c) is generated. In the map information MP2, by referring to the image IM2, it is shown that a pattern PT16b with a novelty level equal to or higher than the threshold level Lth2 is included at a position away from the display area DA1. The threshold level Lth2 can be determined experimentally in advance as a novelty level corresponding to the need for attention.

[0262] On the other hand, in the process of S45, the display control device 301 applies the attention estimation model 7 to the image IM1, estimates the attention state of the driver 200, and generates an estimation result of the attention state for the image IM1.

[0263] For example, the attention estimation model 7 is applied to the image IM1 in FIG. 26(a), the attention state of the driver 200 is estimated, and map information MP1 as shown in FIG. 26(d) is generated. In the map information MP1, by referring to the image IM1, it is shown that the pattern PT16c with a novelty level equal to or higher than the threshold level Lth2 is included at a position away from the display area DA1. The threshold level Lth2 can be experimentally determined in advance as the novelty level corresponding to the need for attention.

[0264] Alternatively, the attention estimation model 7 is applied to the image IM1 in FIG. 27(a), the attention state of the driver 200 is estimated, and map information MP1 as shown in FIG. 27(d) is generated. In the map information MP1, by referring to the image IM1, it is shown that the pattern PT16c with a novelty level equal to or higher than the threshold level Lth2 is included at a position away from the display area DA1. The threshold level Lth2 can be experimentally determined in advance as the novelty level corresponding to the need for attention.

[0265] When the processes of S43 to S44 and the process of S45 are both completed, the display control device 301 compares the estimation result of the attention state obtained in S44 with the estimation result of the attention state obtained in S45 (S46), and performs an evaluation process according to the comparison result (S52).

[0266] In the evaluation process (S52), the display control device 301 calculates the influence degree on the image IM1 of the display content according to the comparison result (S53), and displays the calculated influence degree (S54).

[0267] For example, in the display unit 10 shown in FIG. 26(e), the degree of influence is displayed in the driving influence display column 10c3. In FIG. 26(e), a case where the degree of influence "10%" is displayed in the driving influence display column 10b3 is illustrated.

[0268] Alternatively, in the display unit 10 shown in FIG. 27(e), the degree of influence is displayed in the driving influence display column 10c3. In FIG. 27(e), a case where the degree of influence "40%" is displayed in the driving influence display column 10b3 is illustrated.

[0269] The display control device 301 receives an instruction regarding the adoption of display content and determines whether to adopt the display content according to the instruction (S55). When the display control device 301 receives an instruction indicating the adoption of display content, assuming that it adopts the display content (Yes in S55), according to the evaluation result of S52, it determines the display form and generates or updates the display form information 82 (S17).

[0270] When the display control device 301 receives an instruction indicating non - adoption of the display content, assuming that it does not adopt the display content (No in S55), it skips S17.

[0271] Then, the display control device 301 increments a parameter N for counting the number of loops (S56) and determines whether the parameter N exceeds the number threshold (S57). The number threshold can be determined experimentally in advance.

[0272] If the parameter N is less than or equal to the number threshold (No in S57), the display control device 301 returns the process to S41.

[0273] If the parameter N exceeds the number threshold (Yes in S57), the display control device 301 performs the processes of S1 - S7 in FIG. 5 (S18).

[0274] Even with such a display control method, when the display object 400 is displayed in the display form DF1 and excessive attention is concentrated on the display object 400 in the visual field VF1, according to the evaluation result, the display form of the display object 400 in which attention is suppressed more than the display form DF1 can be determined as the display form DF2.

[0275] Also, as a fifth modification of the second embodiment, display control as shown in FIG. 28 may be performed. FIG. 28 is a flowchart showing a display control method according to the fifth modification of the second embodiment.

[0276] In the display control system 302, the display control device 301 sets an initial value of 0 for the parameter N for counting the number of loops, and after performing S41 to S46 (see FIG. 25), performs a part of the evaluation process (S61) according to the comparison result. That is, the display control device 301 calculates the degree of influence on the image IM1 of the display content according to the comparison result in S46 (S53).

[0277] The display control device 301 increments the parameter N for counting the number of loops (S63), and determines whether the parameter N exceeds the number threshold (S64).

[0278] If the parameter N is less than or equal to the number threshold (No in S64), the display control device 301 returns the process to S41.

[0279] If the parameter N exceeds the number threshold (Yes in S64), the display control device 301 performs the remaining part of the evaluation process (S61). That is, the display control device 301 calculates a display timing range in which the degree of influence falls within the allowable range using the N degrees of influence calculated in S62 (S65).

[0280] For example, an image display screen 10d and an information display screen 10e may be displayed on the display unit 10 shown in FIG. 29. A plurality of images MI2 whose influence degrees fall within an allowable range may be displayed on the image display screen 10d. In FIG. 29, a case where five images IM2_1 to IM2_5 whose influence degrees fall within the allowable range are displayed is illustrated. The information display screen 10e includes a content input field 10e2, a plurality of timing input fields 10e2_1 to 10e2_5, and a driving influence input field 10e3.

[0281] The content input field 10e1 of the information display screen 10e can receive a selection instruction for selecting one of the plurality of display contents as an evaluation target pattern. The content input field 10e1 may display thumbnail images of the plurality of display contents in a drop-down manner according to a selection operation of the content input field 10e1, and may receive a selection instruction for the display content corresponding to the thumbnail image according to a selection operation of the thumbnail image. Identification information (for example, pattern A) of the display content selected by the selection instruction may be displayed in the content input field 10e1.

[0282] The driving influence input field 10e3 can receive an allowable range of the influence degree. The driving influence input field 10e3 may display a plurality of allowable ranges in a drop-down manner according to a selection operation of the driving influence input field 10e3, and may receive an allowable range according to a selection operation of the allowable range. An allowable range (<15% for example) selected by the selection instruction may be displayed in the content input field 10e1.

[0283] The plurality of timing input fields 10e2_1 to 10e2_5 correspond to the plurality of images IM2_1 to IM2_5. Each timing input field 10e2 can receive a bar movement operation regarding the display timing of the corresponding image IM2. Further, in each timing input field 10e2, a range of the display timing whose influence degree falls within the allowable range is indicated by a frame 10e4 within the range where the bar movement operation is possible.

[0284] The display control device 301 determines the timing using the display timing range calculated in S65, calculates the degree of influence (S66), and displays the timing and the degree of influence (S67).

[0285] For example, in the display unit 10 shown in FIG. 26(e), the display timing is displayed in the timing input field 10b2, and the degree of influence is displayed in the driving influence display field 10c3. In FIG. 26(e), a case where the degree of influence "10%" is displayed in the driving influence display field 10b3 is illustrated.

[0286] Even with such a display control method, when the display object 400 is displayed in the display form DF1 and excessive attention is concentrated on the display object 400 in the visual field VF1, according to the evaluation result, the display form of the display object 400 in which attention is suppressed more than the display form DF1 can be determined as the display form DF2.

[0287] (Third Embodiment) Next, a display method according to the third embodiment will be described. Hereinafter, the description will focus on the parts different from the first and second embodiments.

[0288] In the second embodiment, an example of performing an evaluation for determining the display form to be changed is given. In the third embodiment, an example of performing the evaluation and displaying the evaluation result is given.

[0289] For example, as shown in FIG. 30, the display control method executed by the display control system 102 is different from the second embodiment in the following points. FIG. 30 is a flowchart showing the display control method according to the third embodiment. In FIG. 30, S17 and S18 in FIG. 15 are replaced by S19 and S20 (see FIG. 30).

[0290] After S11 to S16 are performed in the same manner as in the second embodiment, the display control device 101 displays the evaluation result of S16 on the display unit 10 (see FIG. 3) (S19). The evaluation result of S16 may be the calculation result of the degree of influence as shown in FIG. 16(e) or FIG. 17(e), or may be the calculation result of the degree of influence as shown in FIG. 24(e) and the superimposed display of the image IM2 and the map information MP2, or may be the calculation result of the degree of influence as shown in FIG. 26(e), FIG. 27(e), FIG. 29, the display of timing, and the superimposed display of the image IM2 and the map information MP2.

[0291] When the display control device 101 receives an instruction to include the display form of S19 in the registration pattern in response to the display of S19, it records the display form of S19 as the registration pattern 81 used when performing display control in FIG. 5 (S18).

[0292] As described above, in the third embodiment, the display method acquires a field of view image, generates a simulation image from the field of view image, applies an attention estimation model to each of the simulation image and the field of view image, evaluates the influence on the attention state by adding a display object in the display form DF2, and displays the evaluation result. Also by this, when the display object 400 is displayed in the display form DF1 and excessive attention is concentrated on the display object 400 in the field of view VF1, the display form of the display object 400 in which attention is suppressed more than the display form DF1 can be determined as the display form DF2 according to the evaluation result of the display form DF2.

[0293] Note that, as a first modification example of the third embodiment, in the display control method executed by the display control system 102, the processes in which S17 and S18 in FIG. 20 are replaced with S19 and S20 (see FIG. 30) may be performed.

[0294] Alternatively, as a second modification example of the third embodiment, in the display control method executed by the display control system 102, the processes in which S17 and S18 in FIG. 22 are replaced with S19 and S20 (see FIG. 30) may be performed.

[0295] Alternatively, as a third modification of the third embodiment, in the display control method executed by the display control system 102, the processes in which S17 and S18 in FIG. 23 are replaced with S19 and S20 (see FIG. 30) may be performed.

[0296] Alternatively, as a fourth modification of the third embodiment, in the display control method executed by the display control system 102, the processes in which S17 and S18 in FIG. 25 are replaced with S19 and S20 (see FIG. 30) may be performed.

[0297] Alternatively, as a fifth modification of the third embodiment, in the display control method executed by the display control system 102, the processes in which S17 and S18 in FIG. 28 are replaced with S19 and S20 (see FIG. 30) may be performed.

[0298] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and its equivalent scope.

Explanation of Reference Numerals

[0299] 1 Display control device 2 Display control system 4 Acquisition unit 5 Simulation unit 6 Attention estimation unit 8 Display determination unit 9 Display control unit 11, 19, 20 Imaging sensor 18, 21, 22, 23 Display

Claims

1. Obtaining a first image corresponding to the driver's field of view in the vehicle interior including a display area; Adding a display object by a display in a first display form to the display area in the first image to generate a second image; Applying one or more attention estimation models to the second image to estimate the driver's attention state; Controlling the display form of the display object to be displayed by the display in the display area in the vehicle interior according to the estimation result of the attention state; A display control method including the above.

2. Controlling the display form includes: When the estimation result of the attention state is a first estimation result, changing the display form of the display object from the first display form to a second display form in which attention is suppressed more than the first display form. The display control method according to Claim 1.

3. Controlling the display form further includes: When the estimation result of the attention state is a second estimation result, maintaining the display form of the display object in the first display form. The display control method according to Claim 2.

4. The estimation includes generating map information indicating a two-dimensional distribution of prediction errors corresponding to the second image. Controlling the display form includes: When a pattern of the attention state according to the second image and the map information matches a pre-registered pattern, controlling the display form of the display object from the first display form to a second display form in which attention is suppressed. The display control method according to Claim 1.

5. Controlling the display form further includes: When a pattern of the attention state according to the second image and the map information does not match a pre-registered pattern, controlling the display form of the display object in the first display form. The display control method according to Claim 4.

6. The first display form includes displaying the display object at a first luminance. The second display form includes displaying the display object at a second luminance lower than the first luminance. The display control method according to Claim 2.

7. The first display form includes displaying the display object at a first luminance. The second display form includes displaying the display object while gradually increasing the luminance from a second luminance lower than the first luminance. The display control method according to Claim 2.

8. The first display form includes displaying the display object in a first color, and the second display form includes displaying the display object in a second color that is different from the first color in terms of one or more of chroma, luminance, and hue. The display control method according to claim 2.

9. The first display form includes displaying the display object after a first period of time has elapsed, and the second display form includes displaying the display object after a second period of time longer than the first period of time has elapsed. The display control method according to claim 2.

10. Obtaining a first image corresponding to the driver's field of view in the vehicle interior including a display area, adding a display object by a display in the display area of the first image in a first display form to generate a second image, applying one or more attention estimation models to the first image to estimate the driver's attention state with respect to the first image, applying the one or more attention estimation models to the second image to estimate the driver's attention state with respect to the second image, evaluating the influence of the display object on the attention state of the first image according to the estimation result of the attention state with respect to the first image and the estimation result of the attention state with respect to the second image, controlling the display form of the display object to be displayed by the display in the display area of the vehicle interior according to the evaluation result. A display control method including the above.

11. The evaluation is evaluating the influence of the display object on the attention state of the first image according to the estimation result of the attention state with respect to the first image, the estimation result of the attention state with respect to the second image, and the correlation between the driver's line of sight and the driver's attention state. The display control method according to claim 10.

12. The evaluation is evaluating the influence of the display object on the attention state of the first image according to the estimation result of the attention state with respect to the first image, the estimation result of the attention state with respect to the second image, and the correlation between the driver's subjective evaluation and the driver's attention state. The display control method according to claim 10.

13. The estimation with respect to the first image is generating first map information indicating a two-dimensional distribution of a value of a prediction error corresponding to the first image, a variance of the prediction error, or a combination of the value and the variance of the prediction error; the estimation for the second image includes generating second map information indicating a two-dimensional distribution of a value of a prediction error corresponding to the second image, a variance of the prediction error, or a combination of the value and the variance of the prediction error The display control method according to claim 10.

14. acquiring line-of-sight information regarding a driver's line of sight; acquiring correlation information indicating a correlation between the driver's line of sight and the driver's attention state; further comprising the estimation for the first image further includes correcting the first map information according to the line-of-sight information and the correlation information; the estimation for the second image further includes correcting the second map information according to the line-of-sight information and the correlation information The display control method according to claim 13.

15. acquiring evaluation information regarding a subjective evaluation of annoyance by the driver; acquiring correlation information indicating a correlation between the subjective evaluation of annoyance and the driver's attention state; further comprising the estimation for the first image further includes correcting the first map information according to the evaluation information and the correlation information; the estimation for the second image further includes correcting the second map information according to the evaluation information and the correlation information The display control method according to claim 13.

16. an acquisition unit that acquires a first image corresponding to a driver's field of view in a vehicle interior including a display area; a generation unit that adds a display object by a display in a first display form to the display area in the first image to generate a second image; an estimation unit that applies one or more attention estimation models to the second image to estimate a driver's attention state; a display determination unit that determines a display form of the display object to be displayed by the display in the display area in the vehicle interior according to an estimation result of the attention state; a display control unit that controls the display to display the display object in the determined display form; A display control device comprising the above.

17. an imaging sensor; a display control device that receives an image acquired by the imaging sensor; a display controlled by the display control device; comprising the display control device is An acquisition unit that acquires a first image corresponding to the driver's field of view in the vehicle interior including the display area; A generation unit that adds a display object by a display in a first display form to the display area in the first image to generate a second image; An estimation unit that applies one or more attention estimation models to the second image to estimate the driver's attention state; A display determination unit that determines the display form of the display object to be displayed by the display in the display area in the vehicle interior according to the estimation result of the attention state; A display control unit that controls the display to display the display object in the determined display form; Having A display control system.

18. Acquiring a first image corresponding to the driver's field of view in the vehicle interior including the display area; Adding a display object by a display in a first display form to the display area in the first image to generate a second image; Applying one or more attention estimation models to the first image to estimate the driver's attention state with respect to the first image; Applying the one or more attention estimation models to the second image to estimate the driver's attention state with respect to the second image; Evaluating the influence of the display object on the attention state of the first image according to the estimation result of the attention state with respect to the first image and the estimation result of the attention state with respect to the second image; Displaying the evaluation result on the display; A display method including.

Images