Information display method and information display device

Abstract

To display information that allows a crew member to quickly recognize the target to be focused among displayed images.SOLUTION: A processor (10) is configured to: acquire a captured image using an on-vehicle camera (2); use a crew member monitoring device (5) to determine a visual line direction of a crew member; display a first display image on a display unit (20), which includes the captured image of a traveling direction of a vehicle and is not based on the visual line direction of the crew member; generate, when a difference between the crew member's visual line direction and the vehicle's traveling direction is greater than or equal to a predetermined value, second display information (G2) in which mark images (M1X, M1Y, M2X, M2Y, M3, M4) indicating a gaze area according to the visual line direction of the crew member are superimposed onto a second display image (C2) containing the captured image of the visual line direction of the crew member; and switch, when the second display information (G2) is generated, a display content from the first display image and displays the second display information (G2) on the display unit (20).SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information display method and an information display device. [Background technology]

[0002] A device is known that changes the priority of images to be displayed based on the environmental conditions around the vehicle and the driving conditions, and when a signal is received from a changeover switch, switches the images to be displayed in order of priority (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-15436 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above technique has a problem in that the passenger cannot quickly check the image of the area he or she wants to focus on.

[0005] The problem to be solved by the present invention is to display information that enables a passenger to quickly check an image of an area that the passenger wishes to focus on. [Means for solving the problem]

[0006] The present invention solves the above problem by displaying a first display image which includes an image captured in the direction of the vehicle's travel and is independent of the occupant's line of sight, and when the difference between the occupant's line of sight and the vehicle's direction of travel is a predetermined value or greater, generating second display information in which a mark image indicating the gaze area according to the occupant's line of sight is superimposed on a second display image which includes an image captured in the direction of the occupant's line of sight, and switching the display content to display the second display information. Effect of the Invention

[0007] According to the present invention, it is possible to display information that enables an occupant to quickly check an image of an area that the occupant wishes to focus on. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a block diagram showing an example of a hardware configuration of the information display device. [Diagram 2] 13 is a flowchart illustrating an example of an information display process procedure. [Diagram 3] FIG. 11 is a first diagram showing an example of display information. [Figure 4] FIG. 2 is a second diagram showing an example of display information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] As shown in FIG. 1, the information display device 1 of this embodiment includes a processor 10 that executes display control, a display 20, and a communication device 30. The information display device 1 cooperates with a camera 2, a plurality of sensors (group) 3, a vehicle information acquisition device 4, an occupant monitoring device 5, and a navigation device 6. These devices may be mounted on the vehicle or may be brought into the vehicle. The processor 10 and these devices are connected via a communication device 30 such as a controller area network (CAN) or other in-vehicle LAN, and exchange information with each other. The information display device 1 also cooperates with external devices such as a server outside the vehicle and a roadside device via the communication device 30 such as a wireless communication device, and acquires captured images captured by the roadside device and the positions and distances of objects measured by the roadside device.

[0010] A single or multiple cameras 2 capture images of the surroundings of the vehicle in all directions. The cameras include an image sensor with an image sensor such as a CCD, an ultrasonic camera, and an infrared camera. The cameras 2 include at least a front camera 2a that captures images of the front of the vehicle, a rear camera 2b that captures images of the rear or rear side of the vehicle, a right camera 2c that captures images of the right side, the front right side, and the rear right side of the vehicle, and a left camera 2d that captures images of the left side, the front left side, and the rear left side of the vehicle. The type of the camera 2 is not limited as long as it can capture images of the vehicle in all directions. A single camera 2 mounted on a base having a rotation mechanism may be used. The camera 2 captures images of other vehicles, motorcycles, pedestrians, animals, signs, objects such as signs, billboards, lanes, signs on the road surface, and signs on the roadside in all directions (front, rear, left and right sides) of the vehicle. The captured images are provided to the processor 10. The processor 10 extracts information about objects included in the captured image using image processing technology known at the time of filing, and obtains information (display content) indicated by the object, attributes of the object (stationary object, moving object, vehicle, motorcycle, pedestrian, adult, child, animal, etc.) and its behavior (movement direction, movement speed, movement acceleration). The processor 10 also obtains road signs (stop, pedestrian crossing), traffic light display content and traffic rules extracted from the captured image using image processing technology known at the time of filing. The processor 10 judges information indicated by traffic lights in the direction of travel, road signs that require the driver or other passengers' attention, or objects (objects) approaching within a specified distance as objects that the passenger should gaze at. Of the detected objects (objects), the one at which the passenger directs his or her gaze is judged to be the gaze target described below.

[0011] The sensors 3 acquire information from in-vehicle devices and external devices according to their respective functions, and transmit the acquired detection information to the processor 10. The sensors 3 include distance measurement sensors that can detect the presence of objects around the vehicle, the positions and position changes of the objects, the sizes (width and height) of the objects, and the occupied areas of the objects. The distance measurement sensors include laser radar, millimeter wave radar, LiDAR (light detection and ranging) units, and ultrasonic radar. The sensors 3 also include a GPS (Global Positioning System) unit, a gyro sensor, a vehicle speed sensor, and the like, which are arranged in the vehicle, acquire information on the vehicle state over time, and transmit the acquired detection information to the processor 10. The vehicle state includes the current position, attitude, orientation, traveling direction, speed, and acceleration of the vehicle.

[0012] The vehicle information acquisition device 4 calculates the current position, attitude, speed, acceleration, behavior, and traveling direction of the vehicle based on the information acquired from the sensor 3, and provides the information to the processor 10. The processor 10 calculates the relative distance and relative speed between the vehicle and objects (objects) around the vehicle, taking into account the speed of the vehicle and the like.

[0013] The occupant monitoring device 5 includes an in-vehicle camera 51 that captures an image of an occupant in the vehicle cabin. The occupant monitoring device 5 uses the in-vehicle camera 51 to analyze the eyes and eyeball movements of the occupants, including the driver, and observes (monitors) the gaze direction of the occupants over time. The in-vehicle camera 51 may be an eye camera that projects light onto the cornea, detects the orientation of the occupant's left and right eyeballs and the degree of pupil opening by the reflected light, and measures the movement of the eyeball. The occupant monitoring device 5 may determine the gaze direction by taking into account the movement of the occupant's head and face. The occupant monitoring device 5 provides the occupant's gaze to the processor 10 in association with time. The "occupant" in this embodiment is the driver. The "occupant" in this embodiment includes the driver, the person responsible for driving the autonomous vehicle, the driving manager of the autonomous vehicle, or other vehicle passengers.

[0014] The navigation device 6 is equipped with a GPS (Global Positioning System) unit, and refers to map information 61 to calculate a route to a set destination. The processor 10 refers to the route of the navigation device 6 to determine the traveling direction of the vehicle. The navigation device 6 provides the determined traveling direction to the processor 10. The processor 10 may use the traveling direction obtained from the navigation device 6 to determine the traveling direction of the vehicle.

[0015] A processor 10 of the information display device 1 displays information to a vehicle occupant using a display 20. The processor 10 includes a ROM (Read Only Memory) 12 that stores a program for controlling information processing and information display, a CPU (Central Processing Unit) 11 that executes the program stored in the ROM 12, and a RAM (Random Access Memory) 13 that functions as an accessible storage device. The processor 10 executes each function by working in cooperation with software for realizing at least the display control function of the first display image, the display control function of the second display information, and the display switching control function of the second display information, and with each hardware shown in FIG. 1. The display 20 displays information generated by the processor 10 on its display surface under the control of the processor 10. The display 20 may include a touch panel type input device 21. The input device 21 may be configured as a switch, a button, or the like. The display 20 may also include a built-in speaker that functions as the output device 22. The speaker as the output device 22 may be provided independently of the display. The output device 22 may be configured as a lamp that lights up to notify information.

[0016] Each function will be explained according to the flowchart in Figure 2. The display control function of the first display image generates a first display image related to the traveling direction of the vehicle and displays it on the display surface of the display 20. The display control function of the first display image may generate first display information using the first display image and display this on the display 20. First, the processor 10 acquires an image from the camera 2 (S1). The images acquired from the multiple cameras 2a to 2d are synthesized into a single continuous image based on a common coordinate axis. The images captured by the cameras 2a to 2d have their overlapping parts removed, image distortions are corrected according to the imaging characteristics of the cameras 2a to 2d, and the positions of the captured images are adjusted so that they are continuous, and a series of images showing an omnidirectional image of the vehicle are synthesized. The unit images (each pixel) of the synthesized series of images can be identified by common coordinates. For this reason, the unit images (each pixel) of the synthesized image are associated with a direction based on the position of the vehicle and the vehicle occupant. In other words, a specific direction or a direction of a specific width based on the position of the vehicle or the occupant is associated with a specific unit image (each pixel) or a specific image area. In other words, if a specific direction or a range of a specific direction along the occupant's line of sight is identified, an image of the area visible in front of the occupant's line of sight is identified from the image of the synthesized omnidirectional image (all around the vehicle). The position of the image of the object (gaze target) included in the synthesized image can also be specified by the common coordinate value. The position of the occupant may be the position of the occupant determined by the occupant monitoring device 5, or may be defined as the position of the driver's seat in the vehicle or the seating position of each occupant in the vehicle.

[0017] The processor 10 acquires from the vehicle information acquisition device 4 vehicle information indicating the traveling state of the vehicle based on the detection information of the sensor 3 (S2), and determines the traveling direction of the vehicle from the vehicle information (S3). The processor 10 skips S4 until second display information described later is generated, acquires captured images in the traveling direction of the vehicle (S5), and generates a first display image (S6). The processor 10 displays the first display image on the display 20 as default display information (S10). FIG. 3(a) shows an example of a first display image C1 displayed on the display surface 20a of the display 20. As shown in FIG. The first display image is an image that does not depend on the line of sight of the occupant monitored by the occupant monitoring device 5. In other words, the first display image is a display image whose display content does not change according to the change in the line of sight of the occupant. The first display image is an image showing an image in a direction according to the traveling direction of the vehicle, and is not an image showing an image in the line of sight of the occupant. As shown in the figure, the display surface 20a displays a first display image C1 including a captured image in the traveling direction of the vehicle. As shown in the figure, guidance information and driving support information provided by the navigation device 6 may be added to the first display image C1. First display information (not shown) in which a first mark image described later is superimposed on the first display image C1 may be displayed.

[0018] The display control function of the second display information generates the second display information and displays it on the display surface 20a of the display 20. The second display information is generated using a second display image including a captured image in the line of sight of the occupant. The processor 10 acquires line of sight information of the occupant detected using the in-vehicle camera 51 from the occupant monitoring device 5 (S11). The line of sight information is information indicating the direction of the occupant's line of sight. The processor 10 skips the processing of S12, which is additionally executed, and determines the line of sight direction of the occupant based on the information acquired using the occupant monitoring device 5 (S13).

[0019] The process of S12 is a process for confirming the reliability of the gaze information detected by the occupant monitoring device 5. As an example of the process, in the process for determining the gaze direction of the occupant, the processor 10 causes the occupant monitoring device 5 to measure the time during which the gaze of the occupant was directed in a predetermined direction, and determines whether or not the predetermined direction is detected for a time period equal to or longer than the predetermined time (S12). The predetermined direction may be defined as one angle, or as an angular range of a predetermined width. This angular range of a predetermined width is defined as a range in which it is possible to determine that a person is looking in the predetermined direction. If the gaze of the occupant has been directed in the predetermined direction for a predetermined time or longer (YES in S12), the predetermined direction is determined to be the gaze direction of the occupant (S13). Even if the gaze of the occupant is directed in a direction other than the traveling direction of the vehicle, it may simply be a case of the occupant simply looking at the direction, and may be a behavior that is not based on the occupant's intention to gaze in that direction. The occupant feels annoyed when the display information is switched in a complicated manner due to the occupant looking at the direction for just a moment. In this example, the direction in which the occupant has been looking for a predetermined period of time or more is determined to be the direction in which the occupant wants to gaze, and the gaze direction of the occupant based on the occupant's intention is detected. By taking into account the occupant's gaze time in this way, the gaze direction of the occupant based on the occupant's intention can be determined with high accuracy.

[0020] Vehicle occupants, particularly drivers and driving managers, always pay attention to the entire surroundings of the vehicle according to the ever-changing situation. For this reason, it is not easy to determine the gaze direction based on the occupant's true intention based on the detected gaze direction of the occupant. For this reason, in this processing example, at the timing when an input specifying the gaze start timing is received from the occupant via the input device 21 (S14), the gaze direction of the occupant is determined using the occupant monitoring device 5 (S13). When the occupant has the intention to check the visually recognized object, the occupant inputs specifying the gaze start timing via the input device 21. The input device 21 may be one using a touch sensor of the display 20 or a switch provided on the steering wheel. Basically, the occupant directs their gaze in the direction of travel, so that even if they notice an object they want to gaze at, they cannot stare at it. For this reason, by receiving an input from the occupant spontaneously specifying the gaze start timing, the gaze direction based on the occupant's true intention is determined, and an image in that gaze direction is displayed on the display 20, so that the occupant can check the content.

[0021] The processor 10 compares the traveling direction of the vehicle obtained in S3 with the line of sight of the occupant obtained in S13 (S4), and if the difference between the line of sight of the occupant and the traveling direction of the vehicle is equal to or greater than a predetermined value (Yes in S4), acquires an image corresponding to the line of sight of the occupant determined in S13 (S15). Then, the processor 10 generates a second display image including an image captured in the line of sight of the occupant (S16). The second display image includes a still image or a moving image. The processor 10 generates the second display image by identifying an image of a region of a predetermined angle of view based on the line of sight of the occupant from a series of captured images around the vehicle and acquiring an image within this range. The second display image includes coordinate information for identifying the position of a unit image (pixel). The second display image including an image captured in the line of sight of the occupant is highly likely to include a gaze target of interest to the occupant. By displaying the second display information using this second display image, it is expected that the occupant who sees it will confirm the gaze target.

[0022] When the second display image includes an object image predicted to be a gaze target, it is preferable to extract the image and display it to the occupant. The processor 10 extracts the gaze target from the generated second display image. The gaze target to be extracted is an object that may be of interest to the occupant, and includes other vehicles, motorcycles, pedestrians, animals, signs, billboards, and other objects (objects), lanes, signs on the road surface, and signs on the roadside. Methods known at the time of filing, such as pattern matching, object shape recognition, and text recognition, can be used to extract an image of the gaze target from the second display image. Examples of gaze targets include signs informing of detours due to road construction plans, signs informing of road closures due to events such as marathons, guidance for newly constructed facilities, and other vehicles approaching from the side or rear. The processor 10 generates, as the second display image, a still image of an image including the extracted gaze target, or a tracking image captured over time by tracking the gaze target. If the occupant wants to confirm information indicated by the gaze target, it is preferable to confirm it with a still image. Also, if the occupant wants to confirm the movement of the gaze target, it is preferable to confirm it with a tracking image. The tracking image is a video. The video includes a frame-by-frame mode in which still images captured over time are sequentially switched and displayed. From the viewpoint of the occupant, there are cases where the occupant wants to confirm the gaze target of interest with a still image, and cases where the occupant wants to confirm the movement of the moving gaze target with a video. By making it possible to select either a video or a still image as the display mode of the image, it is possible to display information generated using an image corresponding to the gaze target.

[0023] In particular, when an occupant notices a gaze target approaching the vehicle from the rear side, the occupant wants to monitor the movement of the gaze target. In this situation, the occupant wants to pay attention to the movement of the gaze target approaching the vehicle from the rear side while checking the forward direction, which is the traveling direction. In this embodiment, when an input requesting tracking of the gaze target is received from the occupant (S21), a second display image including tracking images captured over time by tracking the gaze target approaching the vehicle is generated (S16). The occupant can check the tracking image that tracks the movement of the gaze target approaching the vehicle. For example, in a situation where another vehicle approaches the vehicle near a merging point to merge, the occupant wants to look in the traveling direction to monitor the distance between the vehicle and the preceding vehicle, but also wants to pay attention to the movement of the other vehicle approaching from the side. In such a situation, the occupant can check the movement of the gaze target by the second display image including the tracking image.

[0024] 3(b) shows an example of the second display information G2 using the second display image C2. This second display image C2 is generated when the occupant's line of sight is directed toward a road extending to the right of the traveling direction. The second display information G2 is an image in which mark images M1X and M1Y, which will be described later, are superimposed on the second display image C2, which includes a captured image in the line of sight direction. The processor 10 also displays the second display information G2 generated using the second display image C2 including the captured image in the line of sight of the occupant in a display section set on the side of the line of sight of the occupant on the display surface of the display 20. For example, when the line of sight of the occupant is directed to the right of the traveling direction, as shown in FIG. 3(c), the second display information G2 is displayed in the section 20aR on the right side of the display surface 20a of the display 20. The first display image C1 related to the traveling direction may be displayed in the section 20aL on the left side, or route guidance information of the navigation device 6 may be displayed. In this way, by displaying the image seen by the occupant to the right in the section 20aR on the right side of the display surface 20a, the occupant can check the image in the direction in which he or she is directing his or her gaze, without feeling uncomfortable, by the display of the display 20. Although not shown, when the line of sight of the occupant is directed to the left of the traveling direction, the second display information G2 may be displayed in the section (not shown) on the left side of the display surface 20a of the display 20.

[0025] Also, as shown in FIG. 3(d), the processor 10 may reduce the second display information G2 and display it in a window area 20aU set in a part of the display surface 20a. The window area 20aU is a part of the display surface 20a and is a display area having a relatively small area with respect to the display surface 20a. The main display surface 20a may display a first display image C1 in the traveling direction of the vehicle, or may display route guidance information of the navigation device 6. The processor 10 generates the second display image according to an input from the occupant requesting an enlarged or reduced display of the second display image. The input command is input via the input device 21. When the processor 10 receives an input from the occupant requesting an enlarged or reduced display of the second display image (S22), the processor 10 generates an enlarged or reduced second display image (S16). In the display process (S10) on the display 20, the processor 10 alternates between the display surface 20a and the window area 20aU in response to the input command from the occupant input via the input device 21. For example, when an enlargement / reduction instruction from the occupant is input, the display contents are switched from the state of FIG. 3(b) to reduce the second display information G2 in the occupant's line of sight direction and display it in the window area 20aU, and the first display image C1 in the traveling direction of the vehicle is enlarged and displayed on the display surface 20a, as shown in FIG. 3(d). When an enlargement / reduction instruction from the occupant is input again, the first display image C1 in the traveling direction of the vehicle may be reduced and displayed in the window area 20aU, and the second display information G2 in the line of sight direction of the occupant may be enlarged and displayed on the display surface 20a (not shown). This allows the image that the occupant wants to see to be enlarged and displayed. Also, when there are multiple directions that the occupant wants to see, the occupant can enlarge or reduce the image according to his / her instruction and display multiple images together on one display surface 20a. Images in two different directions that the occupant cannot actually see can be displayed on one display surface 20a.

[0026] Next, the processor 10 generates a mark image (S17). The mark image is an image that indicates the position of the gaze area in the second display image according to the line of sight of the occupant. The second mark image is displayed in the area of ​​the second display image where the occupant is directing his / her gaze. The "gaze area" indicated by the second mark image is the area in the direction in which the occupant is directing his / her gaze. The processor 10 generates second display information by superimposing the second mark image on the second display image (S18). For ease of identification, in this specification, the mark image generated and superimposed on the second display image is referred to as the second mark image, and the mark image generated and superimposed on the first display image is referred to as the first mark image.

[0027] Passengers who prioritize visibility in the direction of travel may be unable to stare at an object of interest even if they see it, and may have to pass it by without checking the information it presents. It is difficult to identify the object of interest and obtain the information it presents after passing it. Also, even if passengers want to pay attention to an object approaching from a direction other than the direction of travel, they may have to prioritize checking the distance between their vehicle and the vehicle ahead. Human vision is limited, making it difficult to check different directions at the same time. In the present invention, it is considered that a situation in which the difference between the occupant's line of sight and the vehicle's direction of travel becomes equal to or greater than a predetermined value is a situation in which the occupant has visually recognized an object that he or she wishes to check, and that there is a high probability that the object that the occupant wishes to check is present in the line of sight at this time. However, simply displaying the second display image including the captured image according to the line of sight makes it difficult for the occupant to accurately and quickly identify the object that the occupant wants to see from the second display image. The processor 10 defines the gaze direction of the occupant at the time when the difference between the gaze direction of the occupant and the traveling direction of the vehicle becomes equal to or greater than a predetermined value as a gaze area, and displays second display information on the display 20 in which a mark image indicating the gaze area is superimposed on a position corresponding to the gaze area of ​​the second display image (S10). That is, by observing the change in the gaze direction of the occupant, the processor 10 identifies the time when the occupant viewed an object of concern, identifies the direction in which the object exists as the gaze area, and displays second display information on the display 20 indicating, by the mark image, the area (gaze area) in which the object of concern to the occupant exists at the identified time. Using the mark image as a guide, the occupant can quickly identify and confirm the position of the image of the gaze area to which the occupant is directing his or her gaze from the second display image including the captured image in the gaze direction. Incidentally, since the occupant understands the contents of the object in relation to the overall image, it is preferable that the second display image showing the overall image is an image with a predetermined range (predetermined angle of view) in terms of aiding the occupant's recognition. The horizontal angle of view of the second display image can be set with reference to the human field of view (160 degrees to 200 degrees). In this embodiment, by displaying a mark image in a superimposed manner, it is possible to provide second display information that allows the occupant to quickly grasp the gaze area to which the occupant is directing his or her gaze while using image information of a wide range without narrowing the range of the second display image. In this embodiment, the range of the gaze area is narrower than the range of the second display image. The gaze area is a range of angle A with the gaze direction as the central axis, and the area shown by the second display image is a range of angle B (> angle A) with the gaze direction as the central axis. Angle A and angle B are the viewing angles with the gaze direction as the axis. The relationship between angle A and angle B can be determined according to the size of display surface 20a of display 20 and the detection accuracy of the gaze direction of occupant monitoring device 5.

[0028] It is predicted that an object (gaze target) that the occupant wants to see is present in the direction of the occupant's line of sight. Here, an example of the process of generating a mark image (S17) that takes into account the presence of the gaze target will be described. The processor 10 judges whether or not the second display image includes a gaze target (S19). The gaze target is an object included in the second display image to which the occupant directs his / her gaze. The object includes a pedestrian, another vehicle, a traffic sign, a signboard, a guide, and a facility. The second display image is an area in the direction in which the occupant directs his / her gaze, and the captured image in this direction is highly likely to include an object that the occupant wants to check. The object included in the second display image is judged to be a gaze target in which the occupant is interested. The processor 10 judges whether or not the second display image includes a pedestrian, another vehicle, a traffic sign, a signboard, a guide, or a facility that is evaluated as an object. Then, the processor 10 extracts the gaze target from the second display image and specifies its position on the image. The processor 10 judges the attribute of the gaze target (pedestrian, automobile, motorcycle) by a pattern matching method using the characteristics of the object (human limbs, vehicle shape), and specifies the position of the gaze target on the image. The processor 10 may extract an object that occupies a large proportion of the image as the gaze target, or may recognize text or figures and extract an object that displays them as the gaze target.

[0029] When the processor 10 can extract the gaze target from the second display image (YES in S19), it determines the size of the gaze target (S20). The size of the gaze target includes the width and height of the gaze target. The processor 10 generates a second mark image indicating a gaze area corresponding to the position of the gaze target in the second display image or the area occupied by the gaze target (S17). The area occupied by the gaze target is defined by the width and height of the gaze target. The second mark image is a graphic image indicating a gaze area corresponding to the position of the gaze target on the image in the second display image. The "gaze area" indicated by the second mark image is the area in which the gaze target exists in the direction in which the occupant is looking. The processor 10 generates second display information by superimposing the second mark image on the second display image (S18) and displays this on the display 20 (S10). It is not easy for the occupant to immediately identify the gaze target that interests him from the second display image in the line of sight. In other words, the occupant cannot quickly catch the gaze target by simply displaying the second display image including the captured image in the line of sight of the occupant. For this reason, the processor 10 displays a mark image indicating a gaze area according to the position of the gaze target to which the occupant directs his gaze. Using this mark image as a guide, the occupant can quickly confirm the gaze target. In this manner, in this process, the second display image including the captured image in the line of sight is generated, and the position of the gaze target included in this second display image is indicated by a superimposed mark image. This makes it possible to display second display information that allows the occupant to quickly confirm the object to be gazed at without narrowing the range of the image displayed by the second display information. When the gaze target cannot be extracted from the second display image (NO in S19), the processor 10 displays the second display image on the display 20 without generating a second mark image (S10).

[0030] 4(a) to 4(d) show examples of the second display information G2 in which the second mark images M1X, M1Y, M2X, M2Y, M3, and M4 are superimposed on the second display image C2. As shown in FIG. 4(a), the processor 10 generates a second mark image (M1X, M1Y) including a vertical axis M1Y extending in the vertical direction of the display surface 20a of the display 20 and a horizontal axis M1X perpendicular to the vertical axis M1Y. The intersection of the vertical axis M1Y and the horizontal axis M1X corresponds to the position of the occupant's line of sight on the display surface 20a of the display 20. The vertical axis M1Y extends from the top end to the bottom end in the vertical direction of the second display image C2, and the horizontal axis M1X extends from the right end to the left end of the second display image C2. The vertical axis M1Y and the horizontal axis M1X that cross the second display image C2 vertically and horizontally can identify the position of the gaze area in the two-dimensional second display image C2. The occupant can quickly check the gaze area in the direction in which the occupant is looking in the second display information G2 by referring to the second mark image (M1X, M1Y). The second display information G2 ensures a wide range of the image displayed, while the second mark image (M1X, M1Y) allows the user to confirm the position or direction of the gaze area. The processor 10 can also generate a second mark image (M1X, M1Y) indicating a gaze area according to the position of the gaze target in the second display image C2. In the example shown in FIG. 4(a), a pedestrian waiting to cross the intersection is present near the intersection of the vertical axis M1Y and the horizontal axis M1X. The horizontal position of the gaze target is specified by the vertical axis M1Y, and the vertical position of the gaze target is specified by the horizontal axis M1X, and the location of the gaze target can be specified by the intersection. The location of the gaze target in the two-dimensional second display image C2 can be specified by the vertical axis M1Y and the horizontal axis M1X that run vertically and across the second display image C2. In this way, the occupant can quickly confirm the gaze target from the second display information G2 in the area where the line of sight is directed by referring to the mark image (M1X, M1Y). The second display information G2 ensures a wide range of the image to be displayed, while indicating the vertical and horizontal positions of the gaze target within the range by the mark image (M1X, M1Y), so that the occupant can quickly confirm the gaze target. The second mark image may be superimposed on the second display image C2 so that the intersection of the vertical axis M1Y and the horizontal axis M1X of the second mark image corresponds to the position of the road surface (legs) where the gaze target starts moving earliest if the gaze target is a moving object, or corresponds to the center or center of gravity of the image area occupied by the gaze target.

[0031] As shown in FIG. 4(b), the processor 10 generates a mark image (M2X, M2Y) including a vertical frame M2Y extending in the vertical direction Y of the display surface 20a of the display 20 and a horizontal frame M2X extending in the horizontal direction X, and the intersection area of ​​the vertical frame M2Y and the horizontal frame M2X is displayed at a position corresponding to the line of sight of the occupant on the display surface 20a of the display 20. The vertical frame M2Y is a strip-shaped mark image that extends in the vertical direction Y and has a width in the horizontal direction X perpendicular to the vertical direction Y. The horizontal frame M2X is a strip-shaped mark image that extends in the horizontal direction X and has a width in the vertical direction Y. The intersection area of ​​the vertical frame M2Y and the horizontal frame M2X corresponds to the range of the line of sight of the occupant on the display surface 20a of the display 20. The vertical frame M2Y extends from the upper end to the lower end of the second display image C2 in the vertical direction, and the horizontal frame M2X extends from the right end to the left end of the second display image C2. The vertical frame M2Y and horizontal frame M2X that run vertically and horizontally across the second display image C2 allow the location of the gaze area in the two-dimensional second display image C2 to be identified. By referring to the second mark image (M2X, M2Y), the occupant can quickly confirm the gaze area in the second display information G2 to which the occupant is directing his or her gaze. While a wide range of the second display information G2 is displayed, the gaze area can be confirmed by the second mark image (M2X, M2Y). The processor 10 can also generate a second mark image (M2X, M2Y) indicating a gaze area corresponding to the position of the gaze target in the second display image C2. In the example shown in FIG. 4(b), a pedestrian is present near the intersection area of ​​the vertical frame M2Y and the horizontal frame M2X, waiting to cross the intersection. The vertical frame M2Y and the horizontal frame M2X that run vertically and horizontally across the second display image C2 allow the position and area of ​​the gaze target in the two-dimensional second display image C2 to be identified. As shown in the figure, the width of the vertical frame M2Y corresponds to the area (width) of the gaze target in the horizontal direction X. The horizontal frame M2X is a strip-shaped mark image that extends in the horizontal direction X and has a width in the vertical direction Y. The width of the horizontal frame M2X corresponds to the area (height) of the gaze target in the vertical direction Y. FIG. 4(b) shows the second display information G2 in which frame-shaped or strip-shaped mark images (M2X, M2Y) arranged according to the area where the gaze target exists are superimposed on the second display image C2. It can be seen that a pedestrian waiting to cross the intersection exists in the intersection area of ​​the vertical frame M2Y and the horizontal frame M2X. The vertical frame M2Y has a width according to the area where the pedestrian exists, and the horizontal frame M2X has a width (height) according to the area where the pedestrian exists, i.e., the height of the pedestrian. Humans cannot immediately recognize an object when only the attention point on an image is shown, but can quickly recognize the object if they can know the size and height of the object. Although the second display information G2 displays a wide range of image, the vertical and horizontal widths of the gaze target existing therein are shown by the mark images (M2X, M2Y), so that the occupant can quickly confirm the gaze target. In particular, in this example, the horizontal position and width of the gaze target in the second display information G2 can be specified by the vertical frame M2Y, and the vertical position and height of the gaze target can be specified by the horizontal frame M2X. By displaying the intersection area of ​​the vertical frame M2Y and the horizontal frame M2X, the occupant can know the location, size, height, and width of the gaze target in the second display information G2. In this way, the occupant can obtain information on the position, size, height, and width of the gaze target by referring to the mark image (M2X, M2Y), so that the occupant can quickly confirm the gaze target to which the gaze is directed from within the second display information G2.

[0032] As shown in FIG. 4(c), an elliptical mark image M3 may be superimposed on the display image. The processor 10 generates a mark image M3 in which the inner extent of an ellipse is displayed at a position on the display surface 20a of the display 20 corresponding to the line of sight of the occupant. If there is a width in the detected line of sight of the occupant, the circumference of the ellipse can indicate the line of sight area of ​​the occupant. By referring to the second mark image M3, the occupant can quickly confirm the gaze area in the second display information G2 to which the occupant is directing his / her gaze. While a wide range of image is shown by the second display information G2, the location of the gaze area can be clearly indicated by the second mark image M3. The processor 10 can also generate a second mark image M3 indicating a gaze area corresponding to the position of the gaze target in the second display image C2. As shown in Fig. 4(c), the horizontal position and width of the gaze target can be specified based on the positions of the circumference of the ellipse at both ends of the short axis of the ellipse, and the vertical position and height of the gaze target can be specified based on the positions of the circumference of the ellipse at both ends of the long axis of the ellipse. The occupant can recognize the position, width, height, and size of the gaze target based on the position of the circumference of the ellipse intersecting the long and short axes of the ellipse, or the area enclosed by the ellipse. Since the gaze target is assumed to exist or be installed on the ground parallel to the road surface, when generating the second mark image M3 indicating the presence area (width and height) of the gaze target, it is preferable to arrange one of the long and short axes of the ellipse perpendicular to the road surface. The height of the gaze target can be indicated by the length of the long or short axis of the ellipse, and the width of the gaze target can be indicated by the length of the short or long axis of the ellipse. From the viewpoint of clearly recognizing the width and height of the gaze target, it is preferable that the second mark image M3 is an ellipse whose long and short axes have different lengths. However, when the width and height of the gaze target are the same, the second mark image M3 of this example includes a circular form in which the short and long axes are equal.

[0033] As shown in FIG. 4(d), a rectangular mark image M4 may be superimposed on the display image. The processor 10 generates a mark image M4 whose interior extent is displayed at a position on the display surface 20a of the display 20 corresponding to the line of sight of the occupant. The mark image M4 is a rectangle with two sets of opposing sides parallel to each other. If there is width in the detected line of sight of the occupant, the line of sight area of ​​the occupant can be indicated by the four sides of the quadrangle. By referring to the second mark image M4, the occupant can quickly confirm the gaze area in the second display information G2 to which the occupant is directing his / her gaze. While a wide range of image is displayed in the second display information G2, the gaze area can be quickly confirmed by the second mark image M4. The processor 10 can also generate a second mark image M4 indicating a gaze area according to the position of the gaze target in the second display image C2. As shown in FIG. 4(d), the horizontal position and width of the gaze target can be specified by the positions of a pair of short sides of the rectangle, and the vertical position and height of the gaze target can be specified by the positions of a pair of long sides of the rectangle. The occupant can recognize the position, width, height, and size of the gaze target by the vertex positions of the rectangle or the positions of the four sides of the rectangle. Since the gaze target is assumed to exist or be installed on the ground parallel to the traveling road surface, it is preferable to arrange a pair of sides of the rectangle perpendicular to the traveling road surface and make the other pair of sides parallel to the traveling road surface when generating the second mark image M4 indicating the existence area (width and height) of the gaze target. The height of the gaze target can be indicated by one pair of sides of the rectangle, and the width of the gaze target can be indicated by the length of the other pair of sides. The existence position, height, width, and size of the gaze target can be recognized by the positions of the short and long sides of the rectangle, or the area surrounded by the rectangle. In this way, the occupant can quickly confirm the gaze target included in the second display information G2 in the area where the gaze is directed, by using information on the position, size, height, and width of the gaze target.

[0034] In the above, a process has been described in which the object of gaze is extracted from the second display image C2, the second mark images M2Y, M2X are generated, and the second mark images M2Y, M2X are superimposed on the second display image C2 to generate the second display information G2, but a similar process can also be performed on the first display image. The processor 10 can generate a first mark image showing the detected object and generate the first display information by superimposing the first mark image on the first display image. The process of superimposing the first mark image on the first display image can be omitted, and is indicated by a dashed arrow in the flowchart of FIG. 2. After generating the first display image (S6), the processor 10 judges whether or not the first display image includes an object to be watched (S7), and if the object is not included (NO in S7), displays the first display image on the display 20 as it is (S10). If the object is included in the first display image (YES in S7), the processor 10 generates the first mark image (S8) and generates the first display information by superimposing the first mark image on the first display image (S9). The processor 10 displays the first display image or the first display information on the display 20 as a default display image (S10). The form of the first mark image is common to the form of the second mark image described above, and the description thereof is incorporated herein by reference. The object indicated by the first mark image is an obstacle that exists in the traveling direction of the vehicle and is information (signboards) that is essential for route guidance to the destination. On the other hand, the gaze target indicated by the second mark image is an object (gaze target) that exists in a line of sight that differs from the traveling direction of the vehicle by a predetermined angle or more and that the occupant wants to gaze at (directs his / her gaze at). In this way, the gaze target indicated by the first mark image is different from the object indicated by the second mark image.

[0035] The display switching control function switches the display between the first display image (or the first display information) and the second display information. As a basic process, as shown in Fig. 2, when the difference between the traveling direction of the vehicle and the line of sight of the occupant is less than a predetermined value (NO in S4), the processor 10 acquires a captured image in the traveling direction of the vehicle (S5), generates a first display image including the captured image (S6), and displays the first display image on the display 20 (S10). The first display information may be the first display image to which navigation guidance or the like has been added. The first display information may be information in which a first mark image indicating an object is superimposed on the first display image. On the other hand, if the difference between the vehicle's direction of travel and the occupant's line of sight is greater than or equal to a predetermined value (YES in S4), an image captured in the direction of the occupant's line of sight is obtained (S15), a second display image C2 including this is obtained (S16), a second mark image is generated (S17), and second display information G2 is generated by superimposing the second mark image indicating the gaze area on the second display image C2 (S18), and this is displayed on the display 20 (S10).

[0036] In S4, a "predetermined value" that is a threshold value of the difference between the traveling direction of the vehicle and the line of sight of the occupant can be set to a value that is experimentally determined to be an intentional change in line of sight with respect to the traveling direction. The "predetermined value" may be experimentally set according to the vehicle model. The "predetermined value" may be experimentally set according to the road type (highway, vehicle-only road, urban road, near an intersection) of the road on which the vehicle is traveling and the traffic volume. The "predetermined value" may be experimentally set according to the traveling speed of the vehicle. When evaluating the driver's line of sight direction, it may be set according to the driver's driving experience and skills.

[0037] When the second display information is generated, the display switching control function switches the current display content, for example, the first display information, and displays the second display information on the display 20. The display content is switched by at least temporarily stopping or reducing the display of the current display content, and displaying the second display information C2 on the entire display surface 20a of the display 20 or enlarging it. The current display content is switched by changing the display mode so that the second display information is more easily recognized than the current display content. After switching, the second display information is shown to the occupant in a display area larger than the display content displayed before switching. When the occupant directs his / her gaze in a direction other than the traveling direction, it is determined that a gaze target that interests the occupant is present in that gaze direction, and the second display information G2 in which a second mark image indicating the gaze area is superimposed on the second display image C2 is displayed preferentially. By displaying the second display information preferentially, the gaze target on which the occupant is directing his / her gaze can be quickly confirmed. The current display content may be the first display image C1 or the first display information, or may be a route guidance image of the navigation device 6.

[0038] Even if the occupant looks in a direction different from the traveling direction, there are cases where the occupant wants to continue displaying the first display image C1 (see FIG. 3(a)) including the captured image in the traveling direction of the vehicle. When the processor 10 receives an input from the occupant requesting display of the first display image C1 (S23), it switches the currently displayed content, generates the first display image C1 using the captured image in the traveling direction of the host vehicle (S6), and displays the first display information or the first display information on the display 20 (S10).

[0039] Furthermore, in this embodiment, a notification regarding the gaze target is performed. The processor 10 calculates the amount of change in the relative position of the extracted gaze target with respect to the vehicle. If the gaze target is included in the second display image G2 (YES in S19) and the amount of change in the relative position of the gaze target with respect to the vehicle is equal to or greater than a predetermined value (YES in S24), the processor 10 further checks whether the gaze target approaches the vehicle (S25). If the amount of change in the relative position of the gaze target with respect to the vehicle is equal to or greater than a predetermined value (YES in S24) and the gaze target approaches the vehicle (YES in S25), the processor 10 notifies the occupant of the approach of the gaze target via the display 20 or the output device 22 (S26). The processor 10 may display a tracking image of the gaze target on the display 20 to notify the occupant of the approach of the gaze target. The processor 10 may notify the approach of the gazed-at target by voice through a speaker serving as the output device 22, may notify the approach of the gazed-at target by displaying text on the display 20, or may notify the approach of the gazed-at target by lighting a lamp serving as the output device 22. In this way, when the relative position changes and the gazed-at target approaches the vehicle, the occupants can be notified of this. On the other hand, when the amount of change in the relative position is less than the predetermined value (NO in S24), normal processing is performed and the process proceeds to S17. If the amount of change in the relative position of the gaze target with respect to the vehicle is equal to or greater than a predetermined value (YES in S24), even if the gaze target does not approach the vehicle (NO in S25), the behavior of the gaze target is notified to the occupant via the display 20 or the output device 22 (S27). The behavior is approach or separation from the vehicle. The processor 10 may display a tracking image of the gaze target on the display 20 to notify the occupant of the movement of the gaze target. The processor 10 may notify the behavior (approach / separation) of the gaze target by voice via a speaker as the output device 22, may display text on the display 20 to notify the behavior (approach / separation) of the gaze target, or may notify the behavior (approach / separation) of the gaze target by a change in the lighting state of a lamp as the output device 22. When there is a change in the positional relationship between the gaze target and the vehicle, the occupant can know the change. Whether it is approach or separation, the occupant wants to know the movement of the gaze target of interest. According to this processing example, it is possible to notify the occupants of the presence of an object to be watched that may affect the movement of the vehicle. [Explanation of symbols]

[0040] 1...information display device, 10...processor, 11...CPU, 12...ROM, 13...RAM, 20...display, 21...input device, 22...output device, 30...communication device, 2...camera, 2a...front camera, 2b...rear camera, 2c...right camera, 2d...left camera, 3...sensor(s), 4...vehicle information acquisition device, 5...occupant monitoring device, 6...navigation device, 61...map information

Claims

1. 1. An information display method for displaying information to an occupant of a vehicle using an information display device having a processor and a display, comprising: The processor, Obtaining an image of the surroundings of the vehicle using a camera provided in the vehicle; determining a gaze direction of the occupant using an occupant monitoring device; displaying, on the display, a first display image including the captured image in the traveling direction of the vehicle and not depending on the line of sight of the occupant; when a difference between the line of sight of the occupant and a traveling direction of the vehicle is equal to or greater than a predetermined value, second display information is generated in which a mark image indicating a gaze area according to the line of sight of the occupant is superimposed on a second display image including the captured image in the line of sight of the occupant, When the second display information is generated, the display content is switched from the first display image to the second display information on the display.

2. The processor, The information display method according to claim 1, further comprising generating the mark image including a vertical axis extending in the vertical direction of the display surface of the display and a horizontal axis perpendicular to the vertical axis, the intersection of the vertical axis and the horizontal axis being displayed at a position corresponding to the line of sight of the occupant on the display surface of the display.

3. The processor, 2. The information display method according to claim 1, further comprising: a strip-shaped vertical frame extending in a vertical direction of a display surface of the display and having a width in a horizontal direction perpendicular to the vertical direction; and a strip-shaped horizontal frame extending in the horizontal direction and having a width in the vertical direction, the mark image being displayed at a position corresponding to the line of sight of the occupant on the display surface of the display, the intersection area of ​​the vertical frame and the horizontal frame.

4. The processor, Extracting a gaze target from the generated second display image; The information display method according to claim 1 , further comprising generating the mark image indicating a gaze area corresponding to a position in the second display image where the gaze target exists or an area occupied by the gaze target.

5. The processor, Extracting a gaze target from the generated second display image; The information display method according to claim 1 , wherein the second display image is generated to include a still image of an image including the extracted gaze target or a tracking image captured over time by tracking the gaze target.

6. The processor, The information display method according to claim 1, wherein in the process of determining the occupant's gaze direction, the occupant monitoring device measures the time that the occupant's gaze is directed in a predetermined direction, and the predetermined direction for which the measured time is equal to or longer than a predetermined time is determined to be the occupant's gaze direction.

7. The processor, The information display method according to claim 1 , further comprising the step of: determining a line of sight direction of the occupant using the occupant monitoring device at a timing when an input designating a gaze start timing is received from the occupant.

8. The processor, Extracting a gaze target from the generated second display image; The information display method according to claim 1, wherein when an input requesting tracking of the gaze target is received from the occupant, the second display image is generated including a tracking image captured over time by tracking the gaze target.

9. The processor, The information display method according to claim 1 , wherein, when an input is received from the occupant requesting an enlarged or reduced display of the second display image, the second display image is enlarged or reduced and displayed.

10. The processor, The information display method according to claim 1 , wherein when an input requesting display of the first display image is received from the occupant, the current display content is switched and the first display image is displayed on the display.

11. The processor, Extracting a gaze target from the generated second display image; The information display method according to claim 1 , further comprising: calculating an amount of change in a relative position of the extracted object of interest with respect to the vehicle; and, if the amount of change in the relative position is equal to or greater than a predetermined value, notifying the occupant of the behavior of the object of interest.

12. The processor, The information display method according to claim 11, further comprising the step of notifying the occupant of the approach of the gazed upon target when the gazed upon target approaches the vehicle.

13. The processor, The information display method according to claim 1, wherein the second display information generated using the second display image including the captured image in the line of sight of the occupant is displayed in a display section set on the side of the display surface of the display in the line of sight of the occupant.

14. An information display device that includes a processor and a display and displays information to an occupant of a vehicle, The processor, Obtaining an image of the surroundings of the vehicle using a camera provided in the vehicle; determining a gaze direction of the occupant using an occupant monitoring device; displaying, on the display, a first display image including the captured image in the traveling direction of the vehicle and not depending on the line of sight of the occupant; when a difference between the line of sight of the occupant and a traveling direction of the vehicle is equal to or greater than a predetermined value, second display information is generated in which a mark image indicating a gaze area according to the line of sight of the occupant is superimposed on a second display image including the captured image in the line of sight of the occupant, When the second display information is generated, the information display device switches display content from the first display image and displays the second display information on the display.

Images