Display controller, head-up display device, program, and display system for vehicle

Abstract

To minimize annoyance arising because identical information is repeatedly provided, and uneasiness arising because important alerting information may be missed or identical information is not provided.SOLUTION: A display controller 30 controls a display device 50 that displays an image, which shows a specific object recognized from a foreground image of a vehicle, by superposing the image on the foreground image. The display controller includes an image acquisition unit 301 that acquires a foreground image of a vehicle to be imaged, an image recognition unit 302 that recognizes a specific object, a point-of-view position detection unit 303 that detects a point-of-view position of an occupant who has boarded a vehicle, and a control unit 300 that determines, based on the position of the recognized specific object and the detected occupant's point-of-view position, whether the occupant is viewing the specific object, identifies the position of an image of the specific object in an image formation area according to a result of the determination, and implements control so that the image of the specific object is formed at the identified position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a display control device that controls a display device that recognizes a specific object such as a road sign from an image of the foreground of a vehicle and displays an image showing the recognized specific object superimposed on the foreground image. [Background technology]

[0002] For example, Patent Document 1 describes a vehicle display device that includes a gaze detection means for detecting the gaze direction of a user of the vehicle (e.g., a passenger who is in the vehicle and driving the vehicle) and an object identification means for identifying an object that the user is looking at based on the gaze direction detected by the gaze detection means and a color image captured by an imaging means, and an object recognition means for recognizing objects other than the object identified by the object identification means.

[0003] According to the vehicle display device described in Patent Document 1, if the user of the vehicle is provided with the same information again even though he or she has already viewed it once, the user will find the provided information annoying. Therefore, by providing information on objects other than those that the user of the vehicle has already viewed, the annoyance can be eliminated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-182306 A (paragraph

[0005] ,

[0006] reference) Summary of the Invention [Problem to be solved by the invention]

[0005] However, the vehicle display device described in Patent Document 1 provides information about objects other than those that have been viewed once, and since information about objects that have been viewed once is no longer provided, there is a concern that important information that warns drivers of their safety, such as road signs, may be overlooked. Furthermore, this may cause the user to wonder why information that has been provided to the user many times in the past is no longer being provided, which may cause the user to feel uneasy.

[0006] The object of the present invention is to provide a display control device, etc., which, when a user views a specific object that calls attention to itself, such as a road sign, displays an image indicating the presence of the object in an inconspicuous position in the user's field of vision, thereby reducing the annoyance of being provided with the same information multiple times and the anxiety of overlooking important information that calls attention to itself or not being provided with the same information.

[0007] Other objects of the present invention will become apparent to those skilled in the art by referring to the following exemplary aspects and best modes, as well as the accompanying drawings. [Means for solving the problem]

[0008] In order to facilitate an understanding of the outline of the present invention, embodiments according to the present invention will be exemplified below.

[0009] A first aspect is a display control device that recognizes a specific object from an image of the front view of a vehicle and controls a display device to superimpose an image showing the recognized specific object on the foreground image, and includes an image acquisition unit that acquires an image of the front view of the vehicle to be captured, an image recognition unit that recognizes the specific object, a viewpoint position detection unit that detects the viewpoint position of an occupant riding in the vehicle, and a control unit that determines whether the occupant is viewing the specific object based on the position of the specific object recognized by the image recognition unit and the viewpoint position of the occupant detected by the viewpoint position detection unit, identifies an imaging position in an imaging area of ​​the image of the specific object in accordance with the determination result, and controls the imaging of the specific object at the identified imaging position.

[0010] In the first aspect, the control unit calculates the imaging position of the image of the generated specific object in the imaging area according to whether or not the occupant is viewing a specific object such as a road sign, which is determined from the position of the specific object recognized by the image recognition unit and the viewpoint position of the occupant detected by the viewpoint position detection unit, and performs control to image the image of the specific object at the imaging position. In this way, by performing control to image the image of the specific object in the imaging area by making the imaging position different between when the occupant is viewing the specific object and when he or she is not viewing the specific object, it is possible to appropriately alert the occupant. For example, when the occupant is viewing a specific object such as a road sign, the image of the specific object is formed at an imaging position that is not conspicuous in the occupant's field of view, thereby eliminating the annoyance caused by the image that calls attention being formed repeatedly at a conspicuous position in the occupant's field of view. In addition, the chance of overlooking a specific object such as a road sign, which is important for driving a vehicle, is reduced, and the occupant's anxiety about not displaying the same image that calls attention repeatedly can be eliminated.

[0011] The "specific object" here refers to, for example, road signs such as stop signs, speed signs, and stop entry signs, or structures that the occupant needs to check when driving the vehicle, such as a vehicle ahead, a traffic control device, etc. Also, "determining whether the occupant is viewing a specific object" can be realized, for example, by calculating the line of sight from the occupant's viewpoint position detected by the viewpoint position detection unit, and calculating the occupant's gaze point in an imaging area that is virtually set in front of the vehicle from the obtained line of sight of the occupant.

[0012] Furthermore, "specifying the imaging position in the imaging area" means, for example, correcting the captured image based on the mounting position of the camera and the viewpoint position of the occupant to an image seen from the viewpoint position of the occupant, allocating an imaging area to the corrected image, and specifying the imaging position of an image showing a specific object for the allocated imaging area. The correction here means compensating for the deviation between the viewpoint position of the occupant seated in the driver's seat and the mounting position of the camera, and means calculating the imaging position when an image showing a specific object is formed in the imaging area, that is, the imaging position based on the viewpoint position of the occupant, based on a coordinate system based on the mounting position of the camera and a coordinate system based on the viewpoint of the occupant.

[0013] In a second aspect dependent on the first aspect, when it is determined that the occupant is viewing the specific object, the control unit may perform control to form an image of the specific object at a position indicated by a visual image distance (the distance indicated by the length of a line segment from the occupant's viewpoint position to the center of the image of the specific object formed in the imaging area) that is smaller than a default value.

[0014] Here, the "default value" is, for example, 8 [m], assuming a case where a head-up display device with a display distance of 4-8 [m] (for example, as shown in Figures 2(a) and 2(b), an oblique imaging plane in which the distance from the occupant's viewpoint EP to the upper end PU of the image plane HD is 8 [m], and the distance from the occupant's viewpoint EP to the lower end PD of the image plane HD is 4 [m]) is used as the display device.

[0015] In the second aspect, when it is determined that the occupant is viewing a specific object, the control unit sets a default value (VID) defined by, for example, 8 [m]. TH The system controls the imaging of a specific object, such as a road sign, at a position indicated by a visual image distance (VID: Visual Image Distance) smaller than the visual image distance (VID is the distance indicated by the length of the line segment from the occupant's viewpoint (eye point EP) to the center of the image of the specific object formed on the imaging plane (imaging area) that is virtually set in front of the vehicle). In other words, when the occupant is viewing a specific object, the VID THBy projecting an image (an image showing a specific object) that calls attention to a location indicated by a VDA that is smaller than the actual VDA (in other words, a location that is difficult to see within the occupant's field of vision), the occupant is freed from the annoyance of being provided with multiple warnings.

[0016] For example, the contents of road signs are important in driving situations, and if the occupants overlook them, it could lead to an accident. Therefore, the VID system is designed to display such information in a position that is easily visible to the occupants. TH On the other hand, for example, in the driving environment of the vehicle shown in FIG. 4(a), the display image is displayed at a prominent position within the field of vision of the occupant at each successive intersection, which may be annoying when the occupant is viewing a specific object such as a road sign. TH By forming an image of a specific object at a position that satisfies the condition that the VID is smaller than the target image, the image becomes less visible within the field of vision of the occupant, but it is possible to eliminate the annoyance when the occupant is viewing a specific object such as a road sign.

[0017] In addition, in a third aspect that is dependent on the first aspect, when it is determined that the occupant is viewing the specific object, the control unit may perform control to form an image of the specific object at a vertical angle of view indicated by a look-down angle (a first line of sight angle formed by a first reference line extending perpendicular to an up-down, left-right plane that is virtually set in front of the vehicle from the occupant's viewpoint and a straight line from the occupant's viewpoint toward the center of an image of the specific object when the vehicle is viewed from above) that is greater than a first default value.

[0018] Here, the "first preset value" is, for example, 4.5 degrees, which assumes the case where a head-up display device using an oblique imaging surface with a VID of 4-8 [m] and an LDA of 4.5 degrees is used as the display device. In the third aspect, when it is determined that the occupant is viewing a specific object such as a road sign, the control unit sets the first preset value (LDA TH) at a vertical angle of view VA indicated by a look-down angle (for example, as shown in FIG. 2(a), a first line of sight angle (LDA: Look Down Angle) is an angle between a first reference line (LDR) extending perpendicularly to a vertical / horizontal plane (VDP) virtually set in front of the vehicle from the occupant's viewpoint EP, and a straight line (HDC) from the occupant's viewpoint EP toward the center of an image of a specific object formed on an image plane (HD) when the vehicle is viewed from above). In other words, for example, by forming an image showing a specific object, such as a road sign, at a position away from the front, which is the same as the position of the angle of view where an image of a speedometer or the like is normally displayed, the occupant is freed from the annoyance of being given warnings repeatedly.

[0019] For example, the contents of road signs are important in driving situations, and if the occupants overlook them, it could lead to an accident. Therefore, the LDA is designed to display the relevant information in a position that is easily visible to the occupants. TH On the other hand, in the vehicle driving environment shown in FIG. 4(a), the image is displayed at a prominent position in the driver's field of vision at each successive intersection, which may be annoying when the driver is viewing a road sign. Therefore, the LDA TH By forming an image of a specific object, such as a road sign, at a position with a vertical angle of view indicated by an LDA larger than (the first default value), the angle from the front is increased (moving it away from the front, in other words, reducing the display distance), making it less visible within the occupant's field of vision, but it is possible to eliminate the annoyance when the occupant is viewing the road sign.

[0020] In addition, in a fourth aspect dependent on the first aspect, when it is determined that the occupant is viewing the specific object, the control unit may perform control to form an image of the specific object at a horizontal angle of view indicated by a lookover angle (a second line of sight angle formed by a second reference line extending perpendicularly to an up-down, left-right plane that is virtually set in front of the vehicle from the viewpoint position of the occupant and a straight line toward the center of the image of the specific object) whose absolute value is greater than a second default value.

[0021] Here, the "second preset value" is defined as, for example, 0 degrees, assuming that a head-up display device with an oblique imaging surface having a VID of 4-8 [m] and an LDA (look-down angle) of 4.5 degrees is used as the display device. In the fourth aspect, when it is determined that the occupant is viewing a specific object, the control unit sets the second preset value (LOA TH 2B, the second line of sight angle (LOA; Look Over Angle) is the angle between a second reference line (LOR) that extends perpendicularly to a vertical / horizontal / lateral plane VDP that is virtually set in front of the vehicle from the occupant's viewpoint EP and a straight line VL2 that points from the occupant's viewpoint EP toward the center of the image of the specific object formed on the image plane when the vehicle is viewed from the side. That is, the second default value LOA is used to control the position of the horizontal angle of view HA indicated by the look-over angle (LOA; Look Over Angle) that is the angle between a second reference line (LOR) that extends perpendicularly to a vertical / lateral plane VDP that is virtually set in front of the vehicle from the occupant's viewpoint EP and a straight line VL2 that points toward the center of the image of the specific object formed on the image plane when the vehicle is viewed from the side. TH By forming an image showing a specific object, such as a road sign, at a position where the absolute value of the LOA is greater than the absolute value of the LOA (in other words, at a position that is difficult to see within the occupant's field of vision), the occupant can be relieved from the annoyance of being provided with multiple warnings.

[0022] For example, the contents of road signs are important in driving situations, and if the occupants overlook them, it could lead to an accident. Therefore, the LOA is set so that the information displayed on the road signs is easily visible to the occupants. THOn the other hand, in the driving environment shown in FIG. 4(a), the display image is displayed at a prominent position within the field of vision of the occupant at each successive intersection, which may be annoying when the occupant is viewing a road sign. Therefore, the LOA TH By forming an image of a specific object, such as a road sign, at a position that satisfies the condition that the absolute value of the LOA is greater than the actual LOA, the image becomes less visible within the field of vision of the occupant, but it is possible to eliminate the annoyance when the occupant is viewing the specific object, such as a road sign.

[0023] In addition, LOA is LOA TH The image formation position on the image plane is determined based on the result of comparing the absolute values ​​of the two. For example, as shown in Fig. 4(b), when a passenger notices a specific object, the image showing the specific object is moved away from the position when the passenger looks straight ahead (LOA is 0 degrees), and the image can be moved to the left (+ direction) or to the right (- direction), in the sense that it is an absolute value comparison.

[0024] In a fifth aspect that is dependent on the first aspect, when it is determined that the occupant is viewing the specific object, the control unit may perform control to form an image of the specific object at an imaging position of the imaging area that shows the same value as either the visual image distance of the image that is normally displayed or the look-down angle (a first line of sight angle formed by a first reference line extending perpendicular to an up-down, left-right plane that is virtually set from the occupant's viewpoint in front of the vehicle and a straight line from the occupant's viewpoint toward the center of the image of the specific object when the vehicle is viewed from above).

[0025] In the fifth aspect, when it is determined that the occupant is viewing a specific object, the control unit performs control to form an image for calling attention at an imaging position showing the same value as either the VID or LDA of an image (normal display image) of, for example, a speedometer, etc., which is normally displayed. In this way, when the occupant is viewing a specific object, by forming an image for calling attention at a position that is easily visible within the occupant's field of vision, in other words, at a prominent position, it is possible to reduce overlooking a specific object such as an important road sign in a driving scene, and to reduce the annoyance of the same information being displayed multiple times and the anxiety of not displaying the same information multiple times.

[0026] A sixth aspect is a head-up display device that forms a virtual image on an imaging surface that is virtually set in front of a vehicle and allows occupants in the vehicle to view the virtual image superimposed on a foreground of the vehicle, and includes an image acquisition unit that acquires an image of the field of view in front of the vehicle that is captured, an image recognition unit that recognizes a specific object from the acquired captured image, a viewpoint position detection unit that detects the viewpoint position of the occupant, and a control unit that determines whether the occupant is viewing the specific object based on the position of the specific object recognized by the image recognition unit and the viewpoint position of the occupant detected by the viewpoint position detection unit, calculates the imaging position of the virtual image on the imaging surface in accordance with the determination result, and controls the imaging of the specific object at the calculated imaging position.

[0027] In a sixth aspect, the control unit calculates an imaging position on the image plane of the image of the specific object, which is generated according to whether or not the occupant is viewing a specific object such as a road sign, which is determined from the position of the specific object recognized by the image recognition unit and the viewpoint position of the occupant detected by the viewpoint position detection unit, and performs control to image the image of the specific object at the calculated imaging position. In this way, by performing control to image and display the image of the specific object at a different imaging position on the image plane depending on whether the occupant is viewing the specific object or not, it is possible to provide a head-up display device that can appropriately call attention. For example, when an occupant is viewing a specific object such as a road sign, an image of the specific object can be focused at an inconspicuous imaging position in the occupant's field of vision, thereby eliminating the annoyance of repeatedly displaying images that call attention to the object. On the other hand, when an occupant is viewing a specific object, an image of the specific object can be displayed at the same imaging position as the image that is normally displayed, thereby reducing the chance that the occupant will overlook a specific object such as a road sign that is important for driving, and also eliminating the occupant's anxiety about not being shown the same attention-grabbing image multiple times.

[0028] A seventh aspect is a program for a display control device that controls a display device to recognize a specific object from an image of the front view of the vehicle and superimpose the recognized specific object on the foreground image, and causes a processor of the display control device to execute the following processes: acquiring an image of the front view of the vehicle to be captured, recognizing the specific object, detecting the viewpoint position of an occupant riding in the vehicle, determining whether the occupant is viewing the specific object from the recognized position of the specific object and the detected viewpoint position of the occupant, and specifying the position of the image of the specific object in an imaging area in accordance with the determination result, and forming an image of the specific object at the specified imaging position.

[0029] In the seventh aspect, the processor sequentially reads and executes the program from the memory, thereby making it possible to control the imaging position of the image of the specific object in the imaging area to be different depending on whether the occupant is viewing the specific object or not, thereby making it possible to appropriately alert the occupant. For example, when the occupant is viewing a specific object such as a road sign, an image showing the specific object is formed at an inconspicuous imaging position in the occupant's field of view, thereby eliminating the annoyance of the image repeatedly being imaged at a prominent position in the occupant's field of view. At this time, by forming the image showing the specific object at the same imaging position as the image normally displayed, for example, the chance of overlooking a specific object such as a road sign that is important for driving the vehicle is reduced, and the occupant's anxiety about not being shown the same image repeatedly to alert the occupant can be eliminated.

[0030] An eighth aspect is a display system for a vehicle including a head-up display device and a display control device that controls the head-up display device, wherein the display control device includes an image acquisition unit that acquires an image of the foreground of the vehicle to be captured, an image recognition unit that recognizes the specific object, a viewpoint position detection unit that detects the viewpoint position of an occupant aboard the vehicle, and a control unit that determines whether the occupant is viewing the specific object based on the recognized position of the specific object and the detected viewpoint position of the occupant, identifies an imaging position of the image of the specific object based on the determination result, and controls the imaging of the image at the identified imaging position, and the head-up display device, under the control of the control unit, superimposes the image of the specific object formed on an imaging surface that is virtually set in front of the vehicle onto the foreground of the vehicle, allowing the occupant to view it.

[0031] In an eighth aspect, the display control device specifies an imaging position on an imaging surface (imaging area) of an image of a specific object generated according to whether or not an occupant is viewing a specific object, which is determined from the position of the specific object recognized by the image recognition unit and the viewpoint position of the occupant detected by the viewpoint position detection unit, and performs control to image the image of the specific object at the specified imaging position. In this way, by performing control to display the image of the specific object at a different imaging position on the imaging surface depending on whether or not the occupant is viewing the specific object, it is possible to provide a vehicular display system that can appropriately call attention to the driver's attention. For example, if the occupant does not see a road sign, an image of the road sign can be imaged at an inconspicuous imaging position in the occupant's field of vision, eliminating the annoyance of repeatedly being imaged as a warning to the driver. On the other hand, if the occupant sees a road sign, the road sign can be imaged at the same imaging position as the image that is normally displayed, reducing the chance of overlooking a road sign that is important for driving and eliminating the occupant's anxiety about not being shown the same warning image multiple times.

[0032] Those skilled in the art will easily understand that the exemplified embodiments according to the present invention can be further modified without departing from the spirit of the present invention. [Brief description of the drawings]

[0033] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a vehicle display system to which a display control device according to an embodiment of the present invention is applied. [Diagram 2] FIG. 2(a) is a side view used to explain the look-down angle (LDA) when the vehicle is viewed from the side, and FIG. 2(b) is a top view used to explain the look-over angle (LOA) when the vehicle is viewed from above. [Diagram 3] 4 is a flowchart showing an operation of a basic processing procedure of the display control device according to the embodiment of the present invention. [Figure 4]FIG. 4(a) is a diagram cited to explain the vehicle driving environment, and FIG. 4(b) is a diagram showing an example of an image layout displayed on a display device by a display control device that is an embodiment of the present invention. [Diagram 5] 1 is a block diagram showing a configuration of a head-up display device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] The best mode described below is used to easily understand the present invention. Therefore, those skilled in the art should be aware that the present invention is not unduly limited by the mode described below (hereinafter, referred to as the present mode).

[0035] (Configuration of the embodiment) FIG. 1 is a block diagram showing an example of the configuration of a vehicle display system 100 to which a display control device 30 of this embodiment is applied.

[0036] As shown in FIG. 1, a vehicle display system 100 includes an in-vehicle monitoring device 10, a navigation device 20, and a display control device 30 of this embodiment, all of which are capable of bidirectional information transmission via an I / O interface 40.

[0037] The vehicle-mounted monitoring device 10 is composed of sensors that monitor the state of the vehicle including the driving environment, and includes, for example, a camera 11, a GPS (Global Positioning System) 12, a LiDAR (Light Detection And Ranging) 13, and a behavior sensor 14 such as an IMU (Inertial Measurement Unit). Information detected by the vehicle-mounted monitoring device 10 is output to a navigation device 20 and to a display control device 30 of this embodiment via an I / O interface 40.

[0038] The camera 11 photographs at least the vehicle's forward field of view (actual view) and the face (eyes) of the driver who is the viewer, and the GPS 12 locates the current position of the vehicle. The LiDAR 13 uses near-infrared light, visible light, and ultraviolet light to, for example, shine light on an obstacle or the like present in front of the vehicle photographed by the camera 11, captures the reflected light with an optical sensor, and determines the distance to the obstacle based on the time difference. The IMU, which is one of the behavior sensors 14, measures the driving situation and attitude of the vehicle using a three-axis acceleration sensor and a three-axis angular velocity sensor (gyro sensor) (acceleration [m / s 2 The behavior sensor 14 can detect translational motion in three axial directions from the angular velocity [deg / s] and rotational motion from the angular velocity [deg / s]. The behavior sensor 14 also includes a vehicle speed sensor that detects the speed of the vehicle, a steering angle sensor that detects the steering angle of the steering wheel, etc.

[0039] The navigation device 20 uses a Global Navigation Satellite System (GNSS) or a gyro sensor to obtain map information from a map information DB 200 (described later) or by wireless communication with the outside of the vehicle, and performs route guidance for the vehicle, etc. The navigation device 20 transfers a signal to the display device 50 via the I / O interface 40 at an appropriate timing to prompt display output based on the route guidance.

[0040] The map information (map information DB 200) can acquire and store the latest map information by, for example, communication with an external center (not shown) via a V2X (Vehicle to X) type communication system. Here, the map information is mapping data that has been digitized to represent the driving environment of the vehicle 1. The mapping data is preferably digital data of a particularly high-precision dynamic map. Note that a dynamic map is a digital map that combines a huge amount of dynamic information that changes from moment to moment, such as traffic regulations, construction information, accidents, congestion, pedestrians, and traffic lights information, with static information, such as high-precision three-dimensional position information (road surface information, lane information, three-dimensional structures such as road signs, etc.).

[0041] The display control device 30 of this embodiment recognizes a specific object from a foreground image of the vehicle, and controls the display device 50 to superimpose an image showing the recognized specific object on the foreground image. Here, the specific object is, for example, a preceding vehicle, a road sign, a traffic controller, etc. For this reason, the display control device 30 of this embodiment includes a control unit 300, an image acquisition unit 301, an image recognition unit 302, a viewpoint position detection unit 303, and a storage unit 304.

[0042] The image acquisition unit 301 acquires an image of the front view of the vehicle captured by the camera 11 of the in-vehicle monitoring device 10 and outputs it to the control unit 300. The image recognition unit 302 can recognize a specific object such as a road sign, and can recognize the specific object from the foreground image (the forward field of view of the vehicle) captured by the camera 11 of the in-vehicle monitoring device 10 by pattern matching based on an image of a road sign shown in FIG. 4(b) that is stored in advance in the storage unit 304. The image showing the specific object recognized by the image recognition unit 302 is output to the control unit 300 together with its position information.

[0043] The viewpoint position detection unit 303 detects the viewpoint position of an occupant in the vehicle and outputs it to the control unit 300. The viewpoint position detection unit 303 can detect the viewpoint position of the occupant with high accuracy, for example, by performing image recognition of the eye (pupil image) of the occupant sitting in the driver's seat of the vehicle captured by the camera 11 of the in-vehicle monitoring device 10. The viewpoint position information of the occupant detected by the viewpoint position detection unit 303 is output to the control unit 300.

[0044] The control unit 300 determines whether or not the occupant is viewing a specific object based on the specific object recognized by the image recognition unit 302 and the occupant's viewpoint position detected by the viewpoint position detection unit 303, identifies the imaging position of the image of the specific object based on the determination result, and performs control to display the image of the specific object at the identified imaging position.

[0045] For this reason, the control unit 300 generates display information including the recognized specific object, and performs control to write the display information in synchronization with the update timing to a VRAM (Video RAM) area (a copy of the imaging area) allocated to a partial area of ​​the storage unit 304 described later. The control unit 300 also controls a display distance (hereinafter, simply referred to as VID) and a predetermined reference display distance (hereinafter, simply referred to as VID TH ), a look-down angle (hereinafter, simply referred to as LDA) which is a first line-of-sight angle, and a predetermined first line-of-sight angle reference value (hereinafter, simply referred to as LDA TH A look-over angle (hereinafter, simply referred to as LOA) which is a second line of sight angle and a predetermined second line of sight angle reference value (hereinafter, simply referred to as LOA TH VID TH >VID,LDA TH <LDA,LOA TH When any of the conditions <|LOA| is satisfied, control is performed to write the generated image showing a specific object in the imaging area at the imaging position that satisfies the condition.

[0046] Fig. 2(a) shows a side view of a vehicle, which is used to explain LDA, and Fig. 2(b) shows a top view of a vehicle, which is used to explain LOA. Both Fig. 2(a) and Fig. 2(b) are explanatory diagrams for a case where a head-up display device (HUD device) having a normal image surface is used as the display device 50. In Fig. 2(a), RD indicates the road surface, EP indicates the eye point of the occupant, and VA indicates the vertical angle of view, and in Fig. 2(b), HA indicates the horizontal angle of view.

[0047] Here, the VID is represented by the length of a line segment from the viewpoint position EP of the occupant to the center of an image of a specific object formed in an imaging area that is virtually set in front of the vehicle.

[0048] 2(a), the LDA is the line of sight angle between a first reference line (dashed line LDR) that extends perpendicularly to a vertical, horizontal, and lateral plane (VDP) that is virtually set in front of the vehicle from the occupant's viewpoint EP and a straight line (wavy line HDC) that extends from the occupant's viewpoint EP toward the center of an image of a specific object formed on the image plane HD when the vehicle is viewed from above. For example, when the occupant is not gazing at a specific object, the LDA is 4.5 degrees, and when the occupant is gazing at a specific object, the LDA is 5.5 degrees.

[0049] Also, as shown in FIG. 2(b), for example, the LOA is the line of sight angle between a second reference line (dashed line LOR) extending perpendicularly to a vertical, horizontal, and lateral plane (VDP) that is virtually set in front of the vehicle from the occupant's viewpoint EP and a straight line (dashed line HDC) that points from the occupant's viewpoint EP toward the center of an image of a specific object formed on an image plane HD when the vehicle is viewed from the side. For example, when the occupant is not gazing at a specific object, the LOA is +3 degrees (3 degrees to the left), and when the occupant is gazing at a specific object, the LDA is 3+5 degrees (8 degrees to the left). In another example, when the occupant does not notice a specific object, the LOA is -3 degrees (3 degrees to the right), and when the occupant notices a specific object, the LDA is -3-5 degrees (8 degrees to the right).

[0050] In addition, VID TH For example, 8[m], and LDA TH For example, if the LOA is 4.5 degrees, TH is, for example, 0 degrees. These values ​​are based on the assumption that a HUD device with VID of 4-8 [m] (an oblique image plane in which the distance from the occupant's viewpoint EP to the upper end PU of the image plane HD is 8 [m] and the distance from the occupant's viewpoint EP to the lower end PD of the image plane HD is 4 [m]) and LDA of 4.5 degrees is used as the display device.

[0051] The explanation will be returned to Fig. 1. For example, the control unit 300 calculates the line of sight direction from the viewpoint position EP of the occupant detected by the viewpoint position detection unit 303, and calculates the gaze point in an image formation area that is virtually set in front of the vehicle from the obtained line of sight direction of the occupant, thereby making it possible to determine whether the occupant is viewing a specific object (for example, see WO2012 / 077713).

[0052] In addition, the control unit 300 can calculate the imaging position of the specific object based on, for example, the position of the specific object recognized by the image recognition unit 302 and a preset correction value in order to specify the imaging position in the imaging area of ​​the image showing the specific object depending on whether the occupant is viewing the specific object. Here, the correction value is a value set based on the amount of deviation between the position of the eye box (occupant's viewpoint position EP) where the viewpoint of the occupant seated in the driver's seat is assumed to be located and the mounting position of the camera 11 of the in-vehicle monitoring device 10. That is, the control unit 300 specifies the imaging position when an image showing the specific object is formed in the imaging area, that is, the imaging position based on the occupant's viewpoint position EP, by calculation, based on a coordinate system based on the mounting position of the camera 11 of the in-vehicle monitoring device 10 on the vehicle and a coordinate system based on the occupant's viewpoint position EP. In this way, a technology for determining the imaging position when an image is formed in an imaging area based on the mounting position of the camera 11 and the occupant's viewpoint position EP, that is, the imaging position when an image is formed based on the occupant's viewpoint position EP, is described in detail, for example, in Patent Publication No. 2021-142770.

[0053] The control unit 300 also controls the reading out of display information including an image showing a specific object written in a VRAM area allocated to a partial area of ​​the memory unit 304 described later in synchronization with the display timing, and the display on the display device 50.

[0054] When it is determined that the occupant is viewing a specific object, the control unit 300 sets a default value (VID THIt is possible to control the focusing of an image of a specific object, such as a road sign, at an imaging position indicated by a visual image distance (VID: the distance indicated by the length of a line segment from the occupant's viewpoint EP to the center of the image of a specific object focused on an imaging plane (imaging area) that is virtually set in front of the vehicle) that is smaller than VID.

[0055] In addition, when it is determined that the occupant is viewing a specific object, the control unit 300 sets the first preset value (LDA TH It is also possible to control the focusing of an image of a specific object, such as a road sign, at a position indicated by a vertical angle of view VA indicated by a look-down angle (LDA, which is the angle between a first reference line (see LDR in FIG. 2(a)) extending perpendicular to an up-down, left-right plane (see VDP in FIG. 2(a)) that is virtually set in front of the vehicle from the occupant's viewpoint EP, and a straight line (see HDC in FIG. 2(a)) toward the center of an image of a specific object focused on the image plane HD (imaging area) from the occupant's viewpoint EP when the vehicle is viewed from above) that is greater than 100°.

[0056] Furthermore, when it is determined that the occupant is viewing a specific object, the control unit 300 sets a second preset value (LOA TH It is also possible to control the focusing of an image of a specific object, such as a road sign, at the horizontal angle of view HA indicated by the look-over angle (LOA, which is the angle between a second reference line (see LOR in Figure 2(b)) extending perpendicular to the up-down, left-right plane (see VDP in Figure 2(b)) that is virtually set in front of the vehicle from the occupant's viewpoint and a straight line (see HDC in Figure 2(b)) from the occupant's viewpoint EP when looking at the vehicle from the side) toward the center of the image of the specific object focused on the image plane) whose absolute value is greater than the look-over angle (LOA).

[0057] In addition, when the control unit 300 determines that the occupant is viewing a specific object, it can also control the imaging of the specific object, such as a road sign, at a corresponding position (imaging position) in an imaging area that shows the same value as either the VID or LDA of an image that is normally displayed, such as a speedometer.

[0058] The storage unit 304 is a memory in which, for example, a static RAM, a dynamic RAM, or a flash memory is implemented, to which a program area and a work area are assigned. Here, the program area stores a program for the display control device 30 to acquire an image of the front view of the vehicle to be captured, detect the viewpoint position of the occupant, recognize a specific object, generate an image of the recognized specific object, determine whether the occupant is viewing the specific object from the position of the recognized specific object and the detected viewpoint position of the occupant, specify the imaging position of the image of the specific object according to the determination result, and display the image of the specific object at the specified position. In addition, the work area also stores display information including an image showing a specific object generated during the execution of the above-mentioned program, and this display information is expanded in a VRAM area assigned to a part of the work area, and the display content is updated sequentially in synchronization with the update timing.

[0059] The I / O interface 40 communicates with the navigation device 20 in addition to the display control device 30 in accordance with, for example, a Controller Area Network (CAN) standard (also referred to as CAN communication). Note that the communication standard adopted by the I / O interface 40 is not limited to CAN, and includes, for example, wired communication interfaces such as CANFD (CAN with Flexible Data Rate), LIN (Local Interconnect Network), Ethernet (registered trademark), MOST (Media Oriented Systems Transport: MOST is a registered trademark), UART (Universal Asynchronous Receiver Transmitter), or USB (Universal Serial Bus), or an in-vehicle communication (internal communication) interface that is a short-range wireless communication interface within several tens of meters, such as a personal area network (PAN) such as a Bluetooth (registered trademark) network, or a local area network (LAN) such as an 802.11x Wi-Fi (registered trademark) network.

[0060] In addition, the I / O interface 40 may include an external communication (external communication) interface such as a wide area communication network (e.g., an Internet communication network) based on cellular communication standards such as a wireless wide area network (WWAN0, IEEE802.16-2004 (WiMAX: Worldwide Interoperability for Microwave Access)), IEEE802.16e-based (Mobile WiMAX), 4G, 4G-LTE, LTE Advanced, and 5G.

[0061] The display device 50 is, for example, a head-up display (HUD device) housed in the dashboard of the vehicle. The HUD device projects an image onto the windshield as a projection target member of the vehicle, and can display a virtual image at a position a predetermined distance away from the occupant. The display device 50 is not limited to the HUD device, and may be a center information display (CID) provided in the center of the vehicle, or a head-mounted display device (HMD device) mounted on the head of the occupant.

[0062] (Operation of the embodiment) Fig. 3 is a flowchart showing the operation of the basic processing procedure of the display control device 30 of this embodiment. Fig. 4(a) is a diagram cited for explaining the driving environment of the vehicle, and Fig. 4(b) is a diagram showing an example of the layout of an image displayed on the display device 50 by the display control device 30. Hereinafter, the operation of the display control device 30 of this embodiment shown in Fig. 1 will be described in detail with reference to Fig. 3 and Figs. 4(a) and (b).

[0063] In the following description, it is assumed that the vehicle CR is traveling on a road RD as shown in Fig. 4(a), and the occupant sitting in the driver's seat sees road signs RS1 and RS2 (stop signs) in succession, and the display is controlled depending on whether the occupant recognizes the road sign (specific object). In other words, in successive intersections, a road sign is displayed in a prominent position within the occupant's field of vision every time, which may be annoying if the occupant recognizes the road sign.

[0064] In the flowchart of FIG. 3, in the display control device 30 of this embodiment, first, the image acquisition unit 301 acquires a captured image of the foreground of the vehicle CR captured by the camera 11 of the in-vehicle monitoring device 10 (step ST101). Here, the camera 11 is mainly attached to the upper part of the windshield or the front grill of the vehicle CR, and is used to acquire information on the forward visibility of the vehicle CR. Next, the image recognition unit 302 detects a specific object from the captured image acquired by the image acquisition unit 301, for example, by a method such as pattern matching based on an image showing the specific object stored in advance in the storage unit 304 (step ST102). The image recognition unit 302 performs detection by updating the image captured by the camera 11 at a constant period.

[0065] Next, the image recognition unit 302 recognizes a road sign from the detected specific object (step ST103). As an example of a road sign, in addition to a stop structure installed on the roadside (see RS1 and RS2 in FIG. 4(a)), letters and pictures drawn on the road encouraging a vehicle to stop are also considered to be road signs. In addition to stop signs, the types of road signs include various road signs such as maximum speed signs, no crossing signs, and no vehicle entry signs. The image showing the road sign recognized by the image recognition unit 302 is output to the control unit 300 together with its position information.

[0066] Next, the viewpoint position detection unit 303 can detect the viewpoint position of the occupant with high accuracy by, for example, performing image recognition of the eye (pupil image) of the occupant sitting in the driver's seat of the vehicle captured by the camera 11 of the in-vehicle monitoring device 10 (step ST104). The viewpoint position information of the occupant detected by the viewpoint position detection unit 303 is output to the control unit 300.

[0067] Subsequently, the control unit 300 specifies the position within the imaging region of the road sign from the road sign recognized by the image recognition unit 302 and the viewpoint position information of the occupant detected by the viewpoint position detection unit 303 (step ST105). When specifying this position, the control unit 300 calculates, for example, the imaging position of a specific object based on the position of the recognized specific object and a preset correction value. Here, the correction value is a value set based on the deviation amount between the viewpoint position of the occupant seated in the driver's seat and the mounting position of the camera 11 in the in-vehicle monitoring device 10. That is, the control unit 300 is based on the coordinate system with the mounting position of the camera 11 of the in-vehicle monitoring device 10 as a reference and the coordinate system with the viewpoint position of the occupant as a reference, and calculates the imaging position when displaying an image showing the road sign in the imaging region, that is, the imaging position based on the viewpoint position of the occupant.

[0068] Next, the control unit 300 determines the imaging position of the road sign recognized by the image recognition unit 302 (step ST106). When determining the imaging position, if it is determined that the occupant recognizes a specific object, the control unit 300, when at least one of the conditions of VID TH >VID,LDA TH <LDA,LOA TH <│LOA│ is satisfied, performs control to image the image of a specific object such as the road sign recognized by the image recognition unit 302 at the corresponding position of the imaging region that satisfies the condition (when VID TH >VID is satisfied, the imaging position indicated by VID, LDA TH <When LDA is satisfied, the imaging position of the vertical viewing angle VA indicated by LDA, LOA TH <When │LOA│ is satisfied, the imaging position of the horizontal viewing angle HA indicated by the absolute value of LOA) (step ST106). Here, the reason for comparing only the absolute value of LOA is, for example, as shown in FIG. 4(b), when the occupant notices a specific object, from the viewpoint of moving away from the position (when LOA is 0 degrees) when looking straight at the image showing the specific object, it is based on the fact that it is okay to move away in the left direction (+ direction) or the right direction (- direction).

[0069] Here, the control unit 300 calculates the gaze direction from the occupant's gaze position detected by the gaze position detection unit 303 (if the display device 50 is a HUD device, the gaze position within the imaging area of ​​the HUD device on the windshield is regarded as the occupant's gaze), and calculates the gaze point in an imaging area that is virtually set in front of the vehicle from the obtained gaze direction of the occupant, thereby making it possible to determine whether or not the occupant is viewing a specific object.

[0070] Checking road signs is important when driving, and if the occupants overlook it, it could lead to an accident. Therefore, the VID system is designed to display relevant information in a position that is easily visible to the occupants. TH , LDA TH , L.O.A. TH On the other hand, in the driving environment of the vehicle shown in FIG. 4(a), for example, the display image is displayed at a prominent position within the field of vision of the occupant at each successive intersection, which may be annoying when the occupant is viewing a road sign. TH The position where VID is smaller than LDA TH The position that satisfies the condition of larger LDA, LOA TH By forming and displaying an image of the recognized road sign at a position that satisfies at least one of the conditions that the absolute value of the LOA is larger than the actual value, the image becomes less visible within the occupant's field of vision, but it is possible to eliminate annoyance when the occupant is viewing a specific object such as a road sign.

[0071] It is possible to eliminate the annoyance by not displaying images showing specific objects such as road signs, but in this case there is a possibility that the occupants will feel a new sense of anxiety due to the image that has always been displayed no longer being displayed. Therefore, by changing the imaging position of the image as described above, it is possible to realize an appropriate display format when the occupant is viewing a road sign.

[0072] Next, the control unit 300 generates display information including an image showing a road sign to be displayed on the display device 50, and images the image at the imaging position determined in step ST106 (step ST107). An example of the display information generated at this time being displayed on the screen is shown in FIG. 4(b). In FIG. 4(b), the imaging area is IA, and the image showing the road sign is SI. Here, as described above, VID TH VID is smaller than LDA TH LDA is greater than or LOA is greater than TH As an example of an imaging position that satisfies at least one of the above conditions, the absolute value of LOA is larger than that of the image of the speedometer (SP) as shown in Figure 4(b), where the image is formed at the position indicated by the VID and LDA, which are the same as the position normally displayed, such as the image of the speedometer (SP).

[0073] In addition, in Figure 4(b), RS1' is an image corresponding to the road sign RS1 (stop) in Figure 4(a), RS2' is an image corresponding to the road sign RS2 (stop) in Figure 4(a), and RD' is an image corresponding to the road RD in Figure 4(a).

[0074] Finally, the control unit 300 writes the generated display information to a VRAM area allocated to a partial area of ​​the storage unit 304, and the display device 50 reads the display information written to the VRAM area in synchronization with the display timing and displays the display on the screen according to the contents shown in Fig. 4(b) (step ST108). In this way, in a scene where the occupant continuously sees a specific object such as a road sign (see Fig. 4(a)), by performing the display shown in Fig. 4(b), it is possible to simultaneously eliminate the annoyance of being provided with the same warning many times and the anxiety that occurs when no warning is issued.

[0075] In order to perform the above-mentioned control, the display control device 30 of the present embodiment is implemented with, for example, a built-in memory (ROM / RAM) or an external processor, and the processor reads out and sequentially executes a program recorded in the memory, thereby executing the functions of the control unit 300, the image acquisition unit 301, the image recognition unit 302, and the viewpoint position determination unit 303. That is, the display control device 30 acquires an image of the front view of the vehicle to be captured, detects the viewpoint position of the occupant, recognizes a specific object, and determines whether the occupant is viewing the specific object based on the position of the recognized specific object and the detected viewpoint position of the occupant. The graphic controller specifies the position of the image of the specific object in the imaging area according to the determination result, and forms and displays the image of the specific object at the specified position. Also, at least a part of the above-mentioned functions can be realized by hardware such as an FPGA (Field Programmable Gate Array) or a logic circuit, without relying on a processor.

[0076] (Modification) According to the display control device 30 of the present embodiment described above, the display control device 30 acquires an image of the front view of the vehicle to be captured, detects the viewpoint position of the occupant, recognizes a specific object, determines whether the occupant is viewing the specific object based on the position of the recognized specific object and the detected viewpoint position of the occupant, specifies the position of the image of the specific object in the imaging area according to the determination result, and forms and displays the image of the specific object at the specified position, but by incorporating these functions into the HUD device, the functions of the display control device 30 can be realized by the HUD device alone. In this case, the processing load of the display control device 30 can be reduced.

[0077] In this case, the HUD device is configured with a HUD control unit 51 and an image display unit 52, as shown in an example configuration in FIG.

[0078] The HUD control unit 51 includes an image acquisition unit 511 for acquiring an image of the front view of the vehicle to be captured, a viewpoint position detection unit 513 for detecting the viewpoint position of the occupant, an image recognition unit 512 for recognizing a specific object, and a control unit 510 for determining whether the occupant is viewing the specific object from the recognized position of the specific object and the detected viewpoint position of the occupant, specifying the position of the image of the specific object in the imaging area according to the determination result, and forming and displaying the image of the specific object at the specified position. It also includes a storage unit 514 to which an image showing the specific object and a VRAM area are assigned for image recognition and display.

[0079] The image display unit 52 is mainly composed of a light source consisting of a light-emitting diode mounted on a wiring board, a projection unit 521 including a relay optical system, and a TFT (Thin Film Transistor Liquid Crystal) type liquid crystal display element 522 located on the emission side (directly above) of the light source so as to transmit illumination light from the light source to form display light, and can output display light by transmitting light emitted from the light source through the liquid crystal display element 522. The liquid crystal display element 522 is a display device that forms a desired image based on display information (drive signal) generated under the control of the HUD control unit 51, and displays the image formed on an imaging surface (imaging area) virtually set in front of the vehicle in a superimposed manner on the forward field of vision, allowing the viewer to view it.

[0080] According to the HUD device 50A of the present embodiment, the HUD control unit 51 (control unit 510) calculates the imaging position on the image plane of the image of the specific object generated according to whether or not the occupant is viewing the specific object, which is determined from the position of the specific object recognized by the image recognition unit 512 and the viewpoint position of the occupant detected by the viewpoint position detection unit 513, and performs control to drive the image display unit 52 (projection unit 521) by imaging the image of the specific object at the calculated imaging position. In this way, by performing control to image and display the image of the specific object at a different imaging position on the image plane depending on whether the occupant is viewing the specific object or not, it is possible to provide a HUD device 50A that can appropriately call attention and reduce the processing load of the display control device 30.

[0081] (Effects of the embodiment) As described above, the display control device 30 of this embodiment is a display control device 30 that recognizes a specific object from a front view image of the vehicle and controls the display device 50 to display an image showing the recognized specific object superimposed on the front view image. The display control device 30 includes, for example, an image acquisition unit 301 that acquires an image of the front view of the vehicle to be captured, an image recognition unit 302 that recognizes the specific object, a viewpoint position detection unit 303 that detects the viewpoint position of an occupant who boards the vehicle, and a control unit 300 that determines whether or not the occupant is viewing the specific object from the position of the specific object recognized by the image recognition unit 302 and the viewpoint position of the occupant detected by the viewpoint position detection unit 303, specifies the imaging position in the imaging area of ​​the image of the specific object according to the determination result, and performs control to image and display the image of the specific object at the specified imaging position, as shown in FIG.

[0082] According to the display control device 30 of this embodiment, the control unit 300 calculates the imaging position of the image of the generated specific object in the imaging area according to, for example, whether or not the occupant is viewing the specific object, which is determined from the position of the specific object recognized by the image recognition unit 302 and the viewpoint position of the occupant detected by the viewpoint position detection unit 303, and performs control to image the image of the specific object at the imaging position. In this way, by performing control to display the image of the specific object in the imaging area at a different imaging position depending on whether the occupant is viewing the specific object or not, it is possible to appropriately alert the occupant. For example, when the occupant is viewing a specific object such as a road sign, the image of the specific object is imaged at an inconspicuous imaging position in the occupant's field of view, thereby eliminating the annoyance of the image that calls attention to the occupant being displayed repeatedly at a prominent position in the occupant's field of view. At this time, by imaging the image of the specific object in the same position as the image that is normally displayed, the chance of overlooking a specific object, such as a road sign, which is important when driving the vehicle is reduced, and the occupants' anxiety about not being shown the same attention-grabbing image multiple times is alleviated.

[0083] Furthermore, in the display control device 30 of this embodiment, when the control unit 300 determines that the occupant is viewing a specific object, it performs control to form an image of the specific object, such as a road sign, at an imaging position indicated by a visual image distance (VID indicated by the length of the line segment from the occupant's viewpoint to the center of the image of the specific object formed in an imaging area virtually set in front of the vehicle) that is smaller than a default value.

[0084] According to the display control device 30 of this embodiment, when it is determined that the occupant is viewing a specific object, the control unit 300 sets the distance to, for example, 8 [m] (default value VID TH The system controls the imaging of a specific object, such as a road sign, at a position indicated by a VID (VID indicated by the length of the line segment from the occupant's viewpoint (eye point EP) to the center of the image of the specific object formed on the imaging plane (imaging area) that is virtually set in front of the vehicle) that is smaller than the VID. In other words, when the occupant is viewing a specific object, the VID TH By projecting an image (an image showing a specific object) that calls attention to the location indicated by a VID smaller than the actual VID (in other words, a location that is difficult to see within the occupant's field of vision), the occupant is freed from the annoyance of being provided with multiple warnings.

[0085] For example, the contents of road signs are important in driving situations, and if the occupants overlook them, it could lead to an accident. Therefore, the VID system is designed to display such information in a position that is easily visible to the occupants. TH On the other hand, for example, in the driving environment of the vehicle shown in FIG. 4(a), the display image is displayed at a prominent position within the field of vision of the occupant at each successive intersection, which may be annoying when the occupant is viewing a road sign. Therefore, TH By forming and displaying an image showing a specific object at a position that satisfies the condition that the VID is smaller than the target, the specific object becomes less visible within the field of vision of the occupant, but it is possible to eliminate the annoyance when the occupant is viewing a specific object such as a road sign.

[0086] In addition, in the display control device 30 of this embodiment, when it is determined that the occupant is viewing a specific object, the control unit 300 sets the first default value (LDA) defined as, for example, 4.5 degrees. TH By forming an image of a specific object, such as a road sign, at a position of the vertical angle of view VA indicated by the look-down angle (LDA, which is the first line of sight angle that is the angle between a first reference line (LDR) extending perpendicularly to the vertical, horizontal, and lateral plane (VDP) that is virtually set in front of the vehicle from the occupant's viewpoint EP and a straight line (HDC) directed from the occupant's viewpoint EP toward the center of the image of the specific object formed on the image plane (HD) when the vehicle is viewed from above) that is greater than 100°, the occupant is freed from the annoyance of being provided with multiple warnings.

[0087] For example, the contents of road signs are important in driving situations, and if the occupants overlook them, it could lead to an accident. Therefore, the LDA is designed to display the relevant information in a position that is easily visible to the occupants. TH On the other hand, in the driving environment of the vehicle shown in FIG. 4(a), the display image is displayed at a prominent position within the field of vision of the occupant at each successive intersection, which may be annoying when the occupant is viewing a road sign. Therefore, the LDA TH By forming and displaying an image of a road sign at a position that satisfies the condition that the LDA is greater than the road sign, the road sign becomes less visible within the field of vision of the occupant, but at the same time, it is possible to eliminate the annoyance when the occupant is viewing the road sign.

[0088] In addition, in the display control device 30 of this embodiment, when it is determined that the occupant is viewing a specific object, the control unit 300 sets the second default value (LOA) defined as 0 degrees, for example. THBy forming an image of a specific object, such as a road sign, at a position of the horizontal angle of view HA (in other words, a position that is difficult to see within the occupant's field of view) indicated by the absolute value of the look-over angle (LOA, which is a second line of sight angle, which is the angle between a second reference line (LOR) extending perpendicularly to a vertical / lateral plane VDP that is virtually set in front of the vehicle from the occupant's viewpoint EP and a straight line VL2 directed from the occupant's viewpoint EP toward the center of an image of a specific object formed on the image plane when the vehicle is viewed from the side), which has an absolute value greater than the horizontal angle of view HA (in other words, a position that is difficult to see within the occupant's field of view)., the occupant is freed from the annoyance of being provided with multiple warnings.

[0089] For example, the contents of road signs are important in driving situations, and if the occupants overlook them, it could lead to an accident. Therefore, the LOA is set so that the information displayed on the road signs is easily visible to the occupants. TH On the other hand, in the driving environment shown in FIG. 4(a), the display image is displayed at a prominent position within the field of vision of the occupant at each successive intersection, which may be annoying when the occupant is viewing a road sign. Therefore, the LOA TH By forming an image of a road sign or a specific object at a position that satisfies the condition that the absolute value of the LOA is greater than the LOA, it becomes difficult to see within the field of vision of the occupant, but it is possible to eliminate the annoyance when the occupant is viewing the road sign or the specific object. TH The image position on the imaging surface is determined based on the result of comparing the absolute values ​​of the two. For example, as shown in Fig. 4(b), when a passenger notices a specific object, the image showing the specific object is moved away from the position when the passenger looks straight ahead (LOA is 0 degrees), and the image can be moved to the left (+ direction) or to the right (- direction), which is an absolute value comparison.

[0090] According to the display control device 30 of the present embodiment, when it is determined that the occupant is viewing a specific object, the control unit 300 performs control to form an image for calling attention at an imaging position showing the same value as either the VID or LDA of an image (normal display image) of a speedometer (SP) or the like, as shown in Fig. 4(b), for example. In this way, when the occupant is viewing a specific object, by displaying an image for calling attention at a position that is easily visible within the occupant's field of vision, in other words, at a prominent position, it is possible to reduce overlooking a specific object such as an important road sign in a driving scene, and to reduce the annoyance of the same information being displayed multiple times and the anxiety of not displaying the same information multiple times.

[0091] 5, the head-up display device 50A of the present embodiment is a HUD device 50A that forms a virtual image on an image plane virtually set in front of the vehicle and allows a passenger in the vehicle to view the virtual image superimposed on the foreground of the vehicle. The HUD device 50A (HUD control unit 51) includes an image acquisition unit 511 that acquires an image of the forward field of view of the vehicle to be captured, an image recognition unit 512 that recognizes a specific object from the acquired captured image, a viewpoint position detection unit 513 that detects the viewpoint position of the passenger, and a control unit 510 that determines whether the passenger is viewing the specific object based on the position of the specific object recognized by the image recognition unit 512 and the viewpoint position of the passenger detected by the viewpoint position detection unit 513, calculates the imaging position of the virtual image on the image plane according to the determination result, and performs control to image the image of the specific object at the calculated imaging position.

[0092] According to the HUD device 50A of this embodiment, the control unit 510 calculates the imaging position on the image plane of the image of the specific object generated according to whether or not the occupant is viewing the specific object, which is determined from the position of the specific object recognized by the image recognition unit 512 and the viewpoint position of the occupant detected by the viewpoint position detection unit 513, and performs control to image the image of the specific object at the calculated imaging position. In this way, by performing control to image and display the image of the specific object at a different imaging position on the image plane depending on whether the occupant is viewing the specific object or not, it is possible to provide the HUD device 50A that can appropriately call attention. For example, when the occupant does not see a specific object such as a road sign, an image of the specific object can be focused at an inconspicuous imaging position in the occupant's field of vision, eliminating the annoyance of repeatedly displaying images that call attention to the object. On the other hand, when the occupant does see a specific object, an image of the specific object can be displayed at the same imaging position as the image that is normally displayed, reducing the chance that a specific object such as a road sign that is important for driving will be overlooked, and also eliminating the occupant's anxiety about not seeing the same attention-grabbing image repeatedly.

[0093] The program of the present embodiment is, for example, a program of a display control device that controls a display device 50 that recognizes a specific object from a foreground image of a vehicle and displays the recognized specific object superimposed on the foreground image, as shown in Fig. 1. The program causes a processor of the display control device 30 to execute, for example, as shown in Fig. 3, a process of acquiring a captured foreground image of the vehicle (see step ST101), a process of recognizing a specific object (see steps ST102 and ST103), a process of detecting a viewpoint position of an occupant who boards the vehicle (step ST104), a process of determining whether or not the occupant is viewing the specific object from the position of the recognized specific object and the detected viewpoint position of the occupant, and a process of specifying a position in the imaging area of ​​the image of the specific object according to the determination result (steps ST105 and ST106), and a process of forming and displaying an image of the specific object at the specified position (steps ST107 and ST108).

[0094] According to the program of the present embodiment, the processor sequentially reads out and executes the program from the memory, thereby making it possible to control the imaging of the specific object in the imaging area to be differently imaged and displayed depending on whether the occupant is viewing the specific object or not, thereby making it possible to appropriately alert the occupant. For example, when the occupant is viewing a specific object, an image showing the specific object is displayed at an inconspicuous imaging position in the occupant's field of view, thereby eliminating the annoyance of the image repeatedly displaying the image that calls attention at a prominent position in the occupant's field of view. At this time, the image showing the specific object is displayed at, for example, the same position as the imaging position of the image normally displayed, thereby reducing the chance of overlooking a specific object such as a road sign that is important for driving the vehicle, and also dispelling the occupant's anxiety about not displaying the same image repeatedly to call attention.

[0095] 1, the vehicular display system 100 includes a head-up display device (display device 50) and a display control device 30 that controls the head-up display device. The vehicular display system 100 includes an image acquisition unit 301 that acquires an image of the front view of the vehicle to be captured, an image recognition unit 302 that recognizes a specific object, a viewpoint position detection unit 303 that detects the viewpoint position of a passenger who boards the vehicle, and a control unit 300 that determines whether the passenger is viewing the specific object from the recognized position of the specific object and the detected viewpoint position of the passenger, specifies the imaging position of the virtual image according to the determination result, and controls the imaging of the virtual image at the specified imaging position. The head-up display device superimposes the virtual image formed on the imaging surface virtually set in front of the vehicle on the front view of the vehicle under the control of the control unit 300, allowing the passenger to view it.

[0096] According to the vehicular display system 100 of this embodiment, the display control device 30 specifies the imaging position on the imaging plane (imaging area) of the image of the specific object generated according to whether or not the occupant is viewing a specific object such as a road sign, which is determined from the position of the specific object recognized by the image recognition unit 302 and the viewpoint position of the occupant detected by the viewpoint position detection unit 303, and performs control to image the image of the specific object at the specified imaging position. In this way, by performing control to display the image of the specific object at a different imaging position on the imaging plane depending on whether the occupant is viewing the specific object or not, it is possible to provide a vehicular display system 100 that can appropriately call attention to the driver's attention. For example, when an occupant does not see a particular object, an image showing that particular object can be imaged at an inconspicuous position in the occupant's field of vision, eliminating the annoyance of having images that call attention to the object displayed repeatedly. On the other hand, when an occupant sees a particular object, an image of that particular object can be imaged at the same position as the image that is normally displayed, reducing the chance that the occupant will overlook a road sign that is important for driving and also eliminating the primary concern of not having the same attention-grabbing image displayed repeatedly.

[0097] The present invention is not limited to the above-described exemplary embodiments, and those skilled in the art could easily modify the above-described exemplary embodiments to the extent that they fall within the scope of the claims. [Explanation of symbols]

[0098] 10···In-vehicle monitoring device, 11···Camera, 12···GPS, 13···LiADR, 14···Behavior sensor, 20···Navigation device, 30···Display control device, 40···I / O interface, 50···Display device, 50A···Head-up display device (HUD device), 51···HUD control unit, 52···Image display unit, 100···Vehicle display system, 200···Map information DB, 300···Control unit, 301···Image acquisition unit, 302···Image recognition unit, 303···Viewpoint position detection unit, 304···Memory unit, 510···Control unit, 511···Image acquisition unit, 512···Image recognition unit, 513···Viewpoint position determination unit, 514···Memory unit, 521···Projection unit, 522···Liquid crystal display element

Claims

1. A display control device that controls a display device to recognize a specific object from a front view image of a vehicle and superimpose an image showing the recognized specific object on the front view image, an image acquisition unit that acquires an image of a front view of the vehicle; an image recognition unit that recognizes the specific object; A viewpoint position detection unit that detects a viewpoint position of a passenger in the vehicle; a control unit that determines whether or not the occupant is viewing the specific object based on the position of the specific object recognized by the image recognition unit and the viewpoint position of the occupant detected by the viewpoint position detection unit, specifies an imaging position in an imaging area of ​​the image of the specific object according to a result of the determination, and performs control to form an image of the specific object at the specified imaging position; A display control device comprising:

2. The control unit is A display control device as described in claim 1, which, when it is determined that the occupant is viewing the specific object, controls the imaging of the specific object at an imaging position indicated by a visual image distance (the distance indicated by the length of a line segment from the occupant's viewpoint position to the center of the image of the specific object formed in an imaging area) that is smaller than a default value.

3. The control unit is 2. The display control device according to claim 1, wherein when it is determined that the occupant is viewing the specific object, control is performed to form an image of the specific object at a vertical angle of view indicated by a look-down angle (a first line of sight angle formed by a first reference line extending perpendicular to an up-down, left-right plane that is virtually set from the occupant's viewpoint in front of the vehicle and a straight line from the occupant's viewpoint toward the center of the image of the specific object when the vehicle is viewed from above) that is greater than a first default value.

4. The control unit is 2. The display control device according to claim 1, wherein when it is determined that the occupant is viewing the specific object, control is performed to form an image of the specific object at a horizontal angle of view indicated by a lookover angle (a second line of sight angle formed by a second reference line extending perpendicular to an up-down, left-right plane that is virtually set in front of the vehicle from the occupant's viewpoint position and a straight line toward the center of the image of the specific object) whose absolute value is greater than a second default value.

5. The control unit is 2. A display control device as described in claim 1, which, when it is determined that the occupant is viewing the specific object, controls the image of the specific object to be formed at a position indicating the same value as either the image visual distance of the image normally displayed, or the look-down angle (a first line of sight angle formed by a first reference line extending perpendicular to an up-down, left-right plane that is virtually set from the occupant's viewpoint in front of the vehicle, and a straight line from the occupant's viewpoint toward the center of the image of the specific object when the vehicle is viewed from above).

6. A head-up display device that forms an image on an image plane that is virtually set in front of a vehicle and allows a passenger in the vehicle to visually recognize the image by superimposing it on a foreground of the vehicle, An image acquisition unit that acquires an image of a forward field of view of the vehicle; an image recognition unit that recognizes a specific object from the acquired captured image; A viewpoint position detection unit that detects a viewpoint position of the occupant; a control unit that determines whether or not the occupant is viewing the specific object based on the position of the specific object recognized by the image recognition unit and the viewpoint position of the occupant detected by the viewpoint position detection unit, calculates an imaging position of the image showing the specific object on the image plane in accordance with the determination result, and controls the imaging of the specific object at the calculated imaging position.

7. A program for a display control device that controls a display device to recognize a specific object from a front view image of a vehicle and superimpose an image of the specific object recognized on the front view image, A processor included in the display control device, A process of acquiring an image of a front view of the vehicle to be captured; A process of recognizing the specific object; A process of detecting a viewpoint position of an occupant in the vehicle; a process of determining whether or not the occupant is viewing the specific object based on the recognized position of the specific object and the detected viewpoint position of the occupant, and specifying a position of the specific object in an imaging area of ​​an image based on the determination result; and a program for executing a process of forming and displaying an image of the specific object at the specified position.

8. A display system for a vehicle including a head-up display device and a display control device that controls the head-up display device, The display control device includes: an image acquisition unit for acquiring an image of a front view of a vehicle to be captured; an image recognition unit that recognizes the specific object; A viewpoint position detection unit that detects a viewpoint position of a passenger in the vehicle; a control unit that determines whether or not the occupant is viewing the specific object based on the recognized position of the specific object and the detected viewpoint position of the occupant, specifies an imaging position of the image of the specific object in accordance with a result of the determination, and performs control to form the image at the specified imaging position, The head-up display device includes: A display system for a vehicle, in which an image of the specific object formed on an imaging surface virtually set in front of the vehicle is superimposed on the foreground of the vehicle under the control of the control unit, allowing the occupants to view it.

Images