Display device, mobile object, display method, and program
The display device projects virtual images in the direction of traffic signs and routes, addressing the need for wide angles and high costs in existing systems by using narrow angles effectively.
Patent Information
- Application Number
- JP2020203002
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-12-26
- Filing Date
- 2020-12-07
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2040-12-07
AI Technical Summary
Existing display devices require a wide projection angle of view to superimpose virtual images on traffic signs, leading to increased size and cost.
A display device that generates a projected image including a virtual sign image based on the position of traffic signs and routes, determining the direction of existence and projecting the image in that direction, allowing narrow projection angles to indicate the target route or sign.
Enables the indication of traffic signs and routes with a narrow projection angle of view, reducing device size and cost while maintaining visibility.
Smart Images

Figure 0007577988000001 
Figure 0007577988000002 
Figure 0007577988000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a display device, a mobile object, a display method, and a program. [Background technology]
[0002] 2. Description of the Related Art Display devices such as head-up displays are known that display a virtual image of a projected image including various information on the front windshield or the like of a vehicle in a superimposed manner on the view in front of the vehicle, allowing the driver of the vehicle to visually recognize the virtual image.
[0003] In addition, a technology has been disclosed that recognizes traffic signs such as road signs and road markings in front of the vehicle based on a forward image of the vehicle acquired by a camera mounted on the vehicle, and indicates to the driver of the vehicle the road to which the traffic sign applies by superimposing a virtual image of a virtual sign image on the position of the road visible in the foreground (see, for example, Patent Document 1). Summary of the Invention [Problem to be solved by the invention]
[0004] However, in the technology of Patent Document 1, when a route such as a road to which a traffic sign is applied or a traffic sign is viewed near the edge of the foreground, a wide projection angle is required for the display device in order to superimpose a virtual image of a predetermined image such as a virtual sign image at the position of the traffic sign in the foreground, which may increase the size and cost of the display device.
[0005] An object of the present invention is to display a route to which a traffic sign is applied or a traffic sign even with a narrow projection angle. [Means for solving the problem]
[0006] A display device according to one aspect of the present invention is a display device that displays a virtual image of a projected image, and includes an image generating unit that generates the projected image including a predetermined image acquired based on a traffic sign, and a direction indicating unit that outputs an existence direction in which either a target route to which the traffic sign is applied or the traffic sign exists based on a position of the traffic sign and a position of a route ahead of a moving body, and causes the image generating unit to generate the projected image including the existence direction; a direction target determination unit that determines, based on the traffic sign, either the target route or the traffic sign as a target of the direction of existence; Equipped with. Effect of the Invention
[0007] Even with a narrow projection angle, the route to which the traffic sign applies or the traffic sign can be shown. [Brief description of the drawings]
[0008] [Figure 1] 1 is a diagram showing a configuration example in which a display device according to an embodiment is mounted on a vehicle; [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a projection unit. [Diagram 3] 1 is a block diagram showing an example of the configuration of an in-vehicle system in which a display device is mounted in a vehicle. [Figure 4] FIG. 2 is a block diagram showing a configuration example of a detection device. [Diagram 5] 1 is a block diagram showing an example of a functional configuration of an image processing device; [Figure 6] FIG. 2 is a block diagram of an example of a hardware configuration of a control unit according to the embodiment. [Figure 7] 4 is a block diagram of an example of a functional configuration of a control unit according to the first embodiment. FIG. [Figure 8] 5 is a flowchart showing an example of processing by a control unit according to the first embodiment. [Figure 9] FIG. 2 is a diagram showing an example of a forward image. [Figure 10] FIG. 4 is a diagram showing an example of a projected image. [Figure 11] FIG. 2 is a diagram showing an example of a front view of a vehicle according to the first embodiment. [Figure 12]12 is a diagram showing an example of a foreground scene changed from the state shown in FIG. 11 due to a movement of the vehicle. [Figure 13] FIG. 4 is a diagram illustrating a wide-angle projection unit. [Figure 14] 13 is a diagram illustrating an example of the functional configuration of a control unit according to a second embodiment. FIG. [Figure 15] 10 is a flowchart showing an example of processing by a control unit according to the second embodiment. [Figure 16] FIG. 11 is a diagram showing an example of a front view of a vehicle according to a second embodiment. [Figure 17] FIG. 17 is a diagram showing an example of a foreground scene changed from the state shown in FIG. 16 due to a movement of the vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, an embodiment of the invention will be described with reference to the drawings. In the drawings, the same components are designated by the same reference numerals, and duplicated descriptions may be omitted.
[0010] The display device according to the embodiment displays a virtual image of a projected image including various information superimposed on a front windshield (windshield) or the like, and allows a driver of a moving body such as a vehicle to visually confirm the virtual image.
[0011] In addition, a projected image is generated that includes a predetermined image, such as a virtual sign image obtained based on a traffic sign, and based on the position of the traffic sign and the position of the route ahead of the moving body, the direction in which either the target route to which the traffic sign applies or the traffic sign exists is shown to the driver.
[0012] By showing the direction, it is not necessary to widen the projection angle to superimpose a virtual image of a predetermined image such as a virtual sign image at the position of the target route visible in the foreground. This makes it possible to show either the target route or a traffic sign even if the projection angle of the display device is narrow.
[0013] <Terminology in the embodiment> (image) This includes both still images and moving images, and also includes video.
[0014] (foreground) The term "foreground" refers to the scenery ahead of a moving body as viewed by the driver of the moving body. For example, the scenery ahead as viewed by the driver through the front windshield of the vehicle corresponds to the foreground.
[0015] (Driver) This refers to a person who drives or operates a moving object. For example, a vehicle driver.
[0016] (traffic sign) A traffic sign refers to an object that provides necessary information to those who use traffic. For example, a traffic sign includes a display member that is a member that displays information, an image that displays information, or paint that displays information. More specifically, a traffic sign includes road signs or road markings. A road sign is a display board that is installed near or above a road and provides necessary information to those who use the road, such as vehicle drivers. A road marking is paint that is installed on the surface of a road to provide information for the purpose of arranging, guiding, or regulating traffic. A traffic sign displays predetermined figures, symbols, letters, etc. according to the information to be provided.
[0017] (Mobile) This refers to a movable object. For example, a mobile body includes a land mobile body that can move on land, such as an automobile, a vehicle, a train, a steam train, or a forklift, an aerial mobile body that can move in the air, such as an airplane, a balloon, or a drone, and a water mobile body that can move on water, such as a ship, a vessel, a steamship, or a boat.
[0018] (route) This refers to the road, route, way, or direction along which a moving body travels. For example, a route includes a traffic route, a direction of travel, or a running direction, etc., a road, passage, way, running path, or railroad track along which a land moving body travels, an air route or airway along which an air moving body travels, and a waterway, sea route, or seaway along which a water moving body travels.
[0019] (Virtual sign image) It refers to an image of a virtual traffic sign, and refers to an image that displays at least one of a figure corresponding to the figure displayed by the traffic sign or a character corresponding to the character displayed by the traffic sign. For example, if the traffic sign is a "No Entry" road sign, an image with the same pattern as the pattern displayed by the "No Entry" road sign corresponds to the virtual sign image.
[0020] (Projected image) This refers to an image that is projected for the driver to see. The projected image includes information provided to the driver, such as the vehicle's traveling speed information and virtual sign images.
[0021] (Prescribed image) The predetermined image refers to an image included in the projected image. For example, the virtual sign image and a text image showing text obtained by replacing a traffic sign with words correspond to the predetermined image.
[0022] (Predetermined position) This refers to the position at which the above-mentioned predetermined image is placed within the projected image when the traffic sign moves out of the foreground as a result of the movement of the mobile object and information about the traffic sign can no longer be acquired.
[0023] (Target route) This refers to the route to which the information indicated by the traffic sign applies. For example, if a traffic sign indicates that a speed limit of 60km / h applies to a road on which a vehicle is traveling, the road to which this 60km / h speed limit applies corresponds to the target route. The road in this case is called the target road. The target road is an example of a target route.
[0024] (direction of existence) This refers to the direction in which either the target route or the traffic sign is located, as seen from the driver's perspective.
[0025] Hereinafter, an embodiment will be described using as an example a head-up display (HUD) mounted on a vehicle capable of traveling on a road. Here, a road is an example of a route, a vehicle is an example of a moving body, and a HUD is an example of a display device.
[0026] <Configuration example> 1 is a diagram showing an example of a configuration in which a display device 1 according to an embodiment is mounted on a vehicle 8. As shown in FIG.
[0027] The display device 1 is embedded inside a dashboard provided in a vehicle 8, and projects an image toward a front windshield 71 from an exit window 3 provided on the upper surface of the display device 1. The projected image is displayed as a virtual image I in the positive Y direction from the front windshield 71, and is viewed by a driver V of the vehicle 8. By viewing the virtual image I, the driver V can obtain information useful for driving with minimal eye movement while keeping his or her gaze on other vehicles or the road surface in the positive Y direction of the vehicle 8.
[0028] The display device 1 may be installed on the dashboard, the ceiling, the sun visor, or the like, as long as it can project an image onto the front windshield 71.
[0029] As shown in FIG. 1, the display device 1 includes a projection unit 10 and a control unit 20 as main components. Known projection methods of the projection unit 10 include a laser scanning method and a panel method. The laser scanning method is a method in which a laser beam emitted from a laser light source is scanned by a two-dimensional scanning device to form an intermediate image (a real image projected on a screen, which will be described later). The panel method is a method in which an intermediate image is formed by an imaging device such as a liquid crystal panel, a DMD (Digital Micro-mirror Device) panel, or a vacuum fluorescent display (VFD). Projected images include both those projected by the laser scanning method and those projected by the panel method.
[0030] Unlike the panel type, which forms an image by illuminating the entire screen and then partially blocking it, the laser scanning type can assign light emission / non-emission to each pixel, so it is advantageous in that it can generally form a high-contrast image. Since high contrast improves visibility, it has been revealed that vehicle occupants can view information with fewer attention resources than a panel type HUD.
[0031] In particular, with the panel method, light that cannot be completely blocked is projected even in areas where there is no information, and a display frame is projected in the area where the HUD can display an image (this phenomenon is called a postcard). With the laser scanning method, this phenomenon does not occur, and only the content can be projected. In particular, in AR (Augmented Reality), the reality is improved when the generated image is superimposed on the real landscape.
[0032] AR is a term that translates to "augmented reality," and refers to a technology that virtually expands the world in front of you by overlaying images of non-existent objects onto real scenery. However, even if it is a panel-type HUD, it is acceptable as long as it can display information in a way that requires less attention resources (less fatigue).
[0033] Fig. 2 is a diagram showing an example of the configuration of the projection unit 10. The projection unit 10 mainly includes a light source unit 101, an optical deflector 102, a mirror 103, a screen 104, and a concave mirror 105. Note that Fig. 2 shows only the main components, and the projection unit 10 may include components other than those shown in the figure, or may not include some of the components shown in the figure.
[0034] The light source unit 101 includes three laser light sources (hereinafter referred to as LDs: laser diodes) corresponding to Red (R), Green (G), and Blue (B), a coupling lens, an aperture, a combining element, a lens, etc., and combines the laser beams emitted from the three LDs and guides them toward the reflecting surface of the optical deflector 102. The laser beams guided to the reflecting surface of the optical deflector 102 are two-dimensionally deflected by the optical deflector 102.
[0035] The optical deflector 102 may be a single minute mirror that oscillates about two orthogonal axes, or two minute mirrors that oscillate or rotate about one axis. The optical deflector 102 may be a MEMS (Micro Electro Mechanical Systems) mirror manufactured by a semiconductor process or the like. The optical deflector 102 may be driven by an actuator using the deformation force of a piezoelectric element as a driving force. However, the optical deflector 102 may also be a galvanometer mirror, a polygon mirror, or the like.
[0036] The laser beam two-dimensionally deflected by the optical deflector 102 is incident on the mirror 103 and is reflected by the mirror 103 to draw a two-dimensional image (intermediate image) on the surface (surface to be scanned) of the screen 104. A concave mirror or the like can be used as the mirror 103, but a convex mirror or a plane mirror may also be used. Deflecting the direction of the laser beam with the optical deflector 102 and the mirror 103 allows the projection unit 10 to be made smaller and the arrangement of the components to be flexibly changed.
[0037] As the screen 104, it is preferable to use a microlens array or a micromirror array that has a function of diverging a laser beam at a desired divergence angle, but a diffusion plate that diffuses the laser beam, a transmission plate or a reflection plate with a smooth surface, etc. may also be used. In general, the components from the light source unit 101 to the screen 104 are called a HUD device. However, other components may also be included.
[0038] The laser beam emitted from the screen 104 is reflected by the concave mirror 105 and projected onto the windshield 71. The concave mirror 105 has a function similar to that of a lens, and has a function of forming an image at a predetermined focal length. For this reason, if the concave mirror 105 is a lens, the image on the screen 104 corresponding to the object is formed at a position of distance R2 determined by the focal length of the concave mirror 105. Therefore, when viewed from a passenger such as the driver V of the vehicle 8, the virtual image I is displayed at a position of distance R1+R2 from the windshield 71. If the distance from the driver V of the vehicle 8 to the windshield 71 is R3, in FIG. 2, the virtual image I is displayed (formed) at a position of distance R (=R1+R2+R3) from the viewpoint E of the driver V of the vehicle 8.
[0039] At least a part of the light beam toward the front windshield 71 is reflected toward the viewpoint E of the driver V of the vehicle 8. As a result, the driver V of the vehicle 8 can view the virtual image I, which is an enlarged version of the intermediate image on the screen 104, through the front windshield 71. In other words, as viewed from the driver V of the vehicle 8, the virtual image I, which is an enlarged version of the intermediate image, is displayed through the front windshield 71.
[0040] Generally, the front windshield 71 is not flat but slightly curved. Therefore, the imaging position of the virtual image I is determined not only by the focal length of the concave mirror 105 but also by the curved surface of the front windshield 71, but the distance R is determined substantially by the distance R1+R2 as described above. To form the virtual image I at a distance so as to reduce the line of sight movement, the distance R1 or R2 is increased. One method for increasing the distance R1 is to fold the optical path with a mirror, and one method for increasing the distance R2 is to adjust the focal length of the concave mirror 105.
[0041] In addition, because the influence of the front windshield 71 causes optical distortion in which the horizontal line of the intermediate image has a convex shape upward or downward, it is preferable that at least one of the mirror 103 and the concave mirror 105 is designed and positioned so as to correct the distortion. Alternatively, it is preferable that the projected image is corrected taking the distortion into consideration.
[0042] Also, a combiner may be disposed as a transmissive reflecting member closer to viewpoint E than front windshield 71. Even if light from concave mirror 105 is irradiated onto the combiner, information can be presented as virtual image I in the same way as when light from concave mirror 105 is irradiated onto front windshield 71.
[0043] <Configuration Example of In-Vehicle System 2 in Which Display Device 1 is Mounted> Next, an in-vehicle system 2 in which the display device 1 is mounted on a vehicle 8 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the configuration of the in-vehicle system 2 in which the display device 1 is mounted on a vehicle 8.
[0044] The in-vehicle system 2 includes a car navigation system 11 that communicates via an in-vehicle network NW such as a CAN (Controller Area Network) bus, an engine ECU (Electronic Control Unit) 12, a display device 1, a brake ECU 13, a steering ECU 14, an image processing device 15, and a detection device 16.
[0045] The car navigation system 11 includes a Global Navigation Satellite System (GNSS) such as a Global Positioning System (GPS), detects the current location of the vehicle 8, and displays the position of the vehicle 8 on an electronic map.
[0046] The system also accepts input of a departure point and a destination, searches for a route from the departure point to the destination, and displays the route on an electronic map, and provides guidance to the occupants of the vehicle 8 on the direction of travel before changing course by voice, text (displayed on a display), animation, or the like. The car navigation system 11 may communicate with a server via a mobile phone network, etc. In this case, the server can transmit the electronic map to the vehicle 8 and perform route searches.
[0047] The engine ECU 12 determines the ideal fuel injection amount, advances / retards the ignition timing, controls the valve mechanism, etc., based on information from each sensor and the situation of the vehicle 8. It also determines the need for gear changes by referring to a map in which gear change lines are determined for the relationship between the current vehicle speed and accelerator opening. The engine ECU 12 combines these to control acceleration and deceleration when following a preceding vehicle. Note that an electric motor may be used as the power source together with or without the engine.
[0048] The brake ECU 13 controls the braking force for each wheel of the vehicle 8 even without the driver V of the vehicle 8 operating the brake pedal, including ABS (Antilock Braking System) control, braking control when following a preceding vehicle, automatic braking based on the TTC (Time To Collision) with an obstacle, and control to maintain a stopped state when starting on a slope.
[0049] The steering ECU 14 detects the direction and amount of steering of the steering wheel by the driver V of the vehicle 8, and performs power steering control to add steering torque in the steering direction. Also, the steering ECU 14 steers the vehicle 8 in a direction to avoid deviation from the driving lane, in a direction to keep the vehicle in the center of the driving lane, or in a direction to avoid approaching an obstacle, even without steering wheel operation by the driver V of the vehicle 8.
[0050] The detection device 16 is equipped with various sensors such as a forward camera 164, and can perform various detections such as capturing an image of the front of the vehicle 8. Here, the forward camera 164 is attached to the vehicle 8 and its position and inclination are adjusted so that the forward image corresponds to the foreground viewed by the driver V through the front windshield 71.
[0051] The image processing device 15 performs image processing on the forward image of the vehicle 8 captured by the forward camera 164, obtains information used to generate the projected image, and outputs it to the display device 1.
[0052] <Configuration Example of Detection Device 16> Next, the configuration of the detection device 16 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the configuration of the detection device 16. The detection device 16 includes a vehicle speed sensor 161 for detecting the vehicle speed displayed by the display device 1, a vehicle information sensor 162 for acquiring vehicle information displayed by the display device 1, a radar sensor 163 and a front camera 164 for detecting obstacles, an occupant status information sensor 165 for acquiring occupant status information that is information about occupants, a VICS receiving device 166 (registered trademark: VICS: Vehicle Information and Communication System Center) for receiving traffic congestion information and the like from the outside, and an external communication device 167 for connecting to the Internet and the like.
[0053] It should be noted that the sensors provided in the detection device 16 do not need to be present all together in the detection device 16, but may be mounted on the vehicle.
[0054] The vehicle speed sensor 161 detects a magnet that rotates with the shaft of the drive train using a sensor unit fixed to the vehicle body, and generates a pulse wave proportional to the rotation speed. The vehicle speed can be detected based on the number of pulses per unit time.
[0055] The vehicle information sensor 162 includes one or more sensors that detect vehicle information other than the vehicle speed sensor 161. For example, these include a fuel gauge sensor, a shift lever position sensor, an odometer, a trip meter, a blinker sensor, and a water temperature sensor. These may have a general configuration and may be capable of acquiring each vehicle information. The fuel gauge sensor detects the current remaining fuel.
[0056] The shift lever position sensor detects the position of the shift lever operated by the driver V of the vehicle 8. The odometer accumulates the distance traveled by the vehicle 8 to provide the total distance traveled. The trip meter provides the distance traveled in a section from the time the driver V of the vehicle 8 initialized it to the present time.
[0057] The turn signal sensor detects the direction of the turn signal operated by the driver V of the vehicle 8. The water temperature sensor detects the temperature of the engine coolant. These are merely examples of information that can be obtained from the vehicle, and other information that can be obtained from the vehicle 8 can also be vehicle information. For example, in an electric vehicle or hybrid vehicle, the remaining battery charge, the amount of regenerated power, the amount of power consumed, etc. can also be obtained.
[0058] The front camera 164 is an imaging device that captures an image in front of the vehicle 8 (in a direction corresponding to the positive Y direction in FIG. 1). The front camera 164 is attached, for example, to the rear side of the rearview mirror or in its vicinity.
[0059] The front camera 164 may be a monocular camera or a stereo camera. In the case of a monocular camera or a stereo camera capable of acquiring distance information, the radar sensor 163 may not be required. However, by providing the radar sensor 163 in addition to the front camera 164 capable of acquiring distance information, the distance information of the front camera 164 and the distance information of the radar sensor 163 are fused (integrated), and the disadvantages of each are complemented to acquire highly accurate distance information. In addition to the radar sensor 163 and the front camera 164, a sonic sensor (ultrasonic sensor) or the like may be provided.
[0060] The radar sensor 163 transmits radar to the surroundings of the vehicle 8, such as in front, to the sides, and behind the vehicle 8, and receives the radar reflected back from objects. It may be placed anywhere that can detect obstacles around the vehicle 8. The radar sensor 163 detects the distance to an object based on the time from transmission to reception, and also uses a Time Of Flight (TOF) method that detects the direction of an object based on the direction of radar irradiation.
[0061] LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) is known as a TOF radar sensor. There is also the FMCW (Frequency Modulation Continuous Wave) method, which generates a mixed wave of the received wave and the transmitted wave while continuously increasing the frequency of the transmitted wave, and converts the beat frequency of the mixed wave generated by the slight difference in frequency into distance. In the FMCW method, the direction of an object is estimated by detecting the phase shift of the received wave with multiple receiving antennas.
[0062] The occupant status information sensor 165 is a sensor that detects occupant status information directly or indirectly detected from an occupant of the vehicle 8. A representative example is a face camera. The face camera captures an image of the face of the occupant of the vehicle 8 and performs face authentication to specify or identify the occupant of the vehicle 8. It also becomes possible to detect the face direction and line of sight direction from the face image.
[0063] In addition, the occupant status information sensor 165 may be, for example, an electrocardiogram sensor, a heart rate sensor, a blood pressure sensor, a body temperature sensor, a pulse sensor, a respiration sensor, a sweat sensor, a blink sensor, a pupil sensor, an electroencephalogram sensor, or a myoelectric potential sensor. As the occupant status information sensor 165, for example, a wristwatch-type wearable terminal (smart watch) worn by the occupant of the vehicle may be used.
[0064] The VICS receiver 166 receives radio waves distributed by the VICS. The VICS is a system that transmits road traffic information such as congestion and traffic regulations to an in-vehicle device in real time using FM multiplex broadcasting and beacons. The external communication device 167 connects to the Internet via networks such as 3G, 4G, 5G, LTE, and wireless LAN to receive various types of information. For example, it can receive weather information such as rain, snow, and fog.
[0065] It may also receive news, music, videos, etc. The external communication device 167 may also obtain, for example, traffic light status information and the time until the traffic light changes. In this way, the VICS receiving device 166 and the external communication device 167 may perform road-to-vehicle communication. In addition, the external communication device 167 may obtain information detected by another vehicle 6 through vehicle-to-vehicle communication.
[0066] In addition, the advanced driving system (ADAS) may not only display information and warn, but also control the vehicle 8. In this case, the ECU cooperates with the engine ECU 12, the brake ECU 13, and the steering ECU 14 to perform various driving assistance based on distance information of an obstacle detected by at least one of the radar sensor 163 and the front camera 164. For example, acceleration / deceleration control when following a preceding vehicle, automatic braking, avoidance of deviation from a driving lane, lane keeping driving, and steering to avoid obstacles are performed. For such control, the ECU recognizes road paint such as white lines from the image captured by the front camera 164.
[0067] In acceleration / deceleration control during following, the ECU controls power and braking to maintain a target distance according to the vehicle speed. In automatic braking, depending on the TTC, it will give a warning, display a message encouraging the driver to press the brake pedal, and retract the seat belt and apply collision avoidance braking if there is a high possibility of a collision. In avoiding deviation from the driving lane, the driving assistance ECU 36 recognizes white lines (traffic lane markings) from the captured image and applies steering torque in the opposite direction to the direction of deviation from the driving lane.
[0068] In lane keeping, the center of the driving lane is set as the target driving line, and a steering torque proportional to the deviation from the target driving line is applied in the opposite direction to the deviation. In obstacle avoidance steering, if it is determined that a collision cannot be avoided by braking, a driving line to avoid the obstacle is determined, and a steering torque to travel along this driving line is applied.
[0069] In addition, for example, when changing lanes, if the radar sensor 163 or the front camera 164 detects a vehicle traveling alongside in an area (blind spot area) that is not reflected in the door mirror in the adjacent lane, a warning is given to the occupants. This type of support is called a blind spot monitor.
[0070] <Example of functional configuration of image processing device 15> Next, a description will be given of the functional configuration of the image processing device 15. The image processing device 15 executes a process of calculating and acquiring position information and the like of roads and traffic signs ahead of the vehicle 8, and outputting the information to the display device 1.
[0071] Fig. 5 is a block diagram showing an example of a functional configuration of the image processing device 15. As shown in Fig. 5, the image processing device 15 includes a sign area extraction unit 151, a road area extraction unit 152, a sign position acquisition unit 153, and a road position acquisition unit 154.
[0072] The sign area extraction unit 151 extracts an image area including a traffic sign in the forward image of the vehicle 8 as a sign image, and outputs the sign image to the sign position acquisition unit 153. The road area extraction unit 152 extracts an image area including a road in the forward image of the vehicle 8 as a road image, and outputs the road image to the road position acquisition unit 154. Image processing such as pattern matching and edge detection can be applied to extract the sign image and the road image. When multiple roads and multiple signs are included in the forward image, a sign image and a road image are extracted for each of the multiple signs and the multiple roads. The sign image includes coordinate information indicating the position of the sign image in the forward image, and the road image includes coordinate information indicating the position of the road image in the forward image.
[0073] Furthermore, an image recognition method using machine learning can also be applied as one of the methods by which the sign area extraction unit 151 and the road area extraction unit 152 extract image areas. Machine learning is a technology for making a computer acquire human-like learning capabilities, and refers to a technology in which a computer autonomously generates algorithms required for judgments such as data identification from learning data that is previously loaded, and applies these to new data to make predictions.
[0074] The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods. Machine learning techniques include perceptron, deep learning, support vector machine, logistic regression, naive Bayes, decision tree, random forest, etc.
[0075] The sign position acquisition unit 153 acquires, based on the sign image, position information of a traffic sign in the image ahead of the vehicle 8 as coordinate information. The road position acquisition unit 154 acquires, based on the road image, position information of a road in the image ahead of the vehicle 8 as coordinate information. When a plurality of roads and a plurality of signs are included in the image ahead of the vehicle 8, position information is acquired for each of the plurality of roads and each of the plurality of signs.
[0076] The image processing device 15 outputs to the display device 1 the sign image extracted by the sign area extraction unit 151, the position information of the traffic sign acquired by the sign position acquisition unit 153, and the position information of the road acquired by the road position acquisition unit 154.
[0077] When no traffic sign is included in the forward image, the image processing device 15 outputs no-sign information indicating this, such as coordinates (0, 0) as information on the sign position. Also, when a road is no longer included in the forward image, the image processing device 15 may output no-road information.
[0078] <Hardware configuration example of control unit 20> Next, the hardware configuration of the control unit 20 will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the hardware configuration of the control unit 20. As shown in Fig. 6, the control unit 20 includes a field-programmable gate array (FPGA) 201, a central processing unit (CPU) 202, a read only memory (ROM) 203, a random access memory (RAM) 204, an interface (I / F) 205, a bus line 206, an LD driver 207, a MEMS controller 208, and a hard disk drive (HDD) 210. The FPGA 201, the CPU 202, the ROM 203, the RAM 204, the I / F 205, and the HDD 210 are electrically connected to each other via the bus line 206.
[0079] The CPU 202 controls each function of the control unit 20. The ROM 203 stores a program 203p that the CPU 202 executes to control each function of the control unit 20. The program 203p is loaded into the RAM 204, which is used as a work area for the CPU 202 to execute the program 203p. The RAM 204 also has an image memory 209.
[0080] The image memory 209 is used to generate an image to be projected as the virtual image I. The I / F 205 is an interface for communicating with the image processing device 15 and the detection device 16, and is connected to, for example, a CAN (Controller Area Network) bus or Ethernet (registered trademark) of the vehicle 8.
[0081] The HDD 210 is a non-volatile memory that stores various data such as programs for executing processes by the control unit 20 and image data of traffic signs, and is a large-capacity storage device. The HDD may be an SSD (Solid State Drive) or the like.
[0082] The FPGA 201 controls the LD driver 207 based on the image generated by the CPU 202. The LD driver 207 drives the LD of the light source unit 101 of the projection unit 10 to control the light emission of the LD according to the image. The FPGA 201 operates the optical deflector 102 of the projection unit 10 via the MEMS controller 208 so that the laser beam is deflected in a direction according to the pixel position of the image.
[0083] [First embodiment] <Example of functional configuration of control unit 20> Next, the functional configuration of the control unit 20 according to the first embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of the functional configuration of the control unit 20. As shown in Fig. 7, the control unit 20 includes an image storage unit 21, a recognition unit 22, a direction object determination unit 23, a direction table storage unit 24, a direction indication unit 25, an image generation unit 26, and an output unit 27.
[0084] Among these, the functions of the recognition unit 22, the direction object determination unit 23, the direction indication unit 25, and the image generation unit 26 are realized by the CPU 202 executing a predetermined program, etc. Also, the functions of the image storage unit 21 and the direction table storage unit 24 are realized by the HDD 210, etc., and the function of the output unit 27 is realized by the I / F 205, etc.
[0085] The image storage unit 21 is a database that stores virtual sign images corresponding to traffic signs. Each of the stored traffic sign images is provided with identification information for identifying the traffic sign. The virtual sign image may be a small image in a range that can be viewed by the driver V when the driver V is caused to view a virtual image of the projected image by being placed in the projected image.
[0086] The image storage unit 21 preferably stores images of all traffic signs currently used in road traffic, but is not limited to this, and may store only a portion of those that are frequently used. Also, it is preferable that the stored images are updated periodically to accommodate changes in traffic signs and the addition of new traffic signs.
[0087] This update work is performed by downloading the latest data from the Internet or the like via the external communication device 167. The update may be performed automatically at regular intervals, or the driver V of the vehicle 8 may operate the in-vehicle system 2 to perform the update work manually.
[0088] The identification unit 22 refers to the image storage unit 21 based on the sign image input from the image processing device 15, and acquires a virtual sign image and identification information corresponding to the traffic sign indicated by the sign image. As an example, the identification unit 22 performs a pattern matching process between the input sign image and the images of each traffic sign pre-stored in the image storage unit 21, and acquires an image having a similarity equal to or greater than a predetermined threshold as a virtual sign image. It can also acquire the identification information assigned to the acquired virtual sign image.
[0089] The identification unit 22 outputs the acquired virtual sign image to the image generation unit 26, and also outputs the identification information to the direction object determination unit 23.
[0090] The direction target determination unit 23 determines, based on the traffic sign identification information input from the identification unit 22, either the direction in which the target road exists or the direction in which the traffic sign exists as the direction of existence to be shown to the driver V. In other words, it determines either the target road or the traffic sign as the target of the direction of existence. Then, it outputs information indicating the target of the direction of existence determined to the direction indication unit 25.
[0091] Specifically, the direction target determination unit 23 makes the above determination based on the identification information of the traffic sign input from the identification unit 22, by referring to the direction table storage unit 24 which stores a correspondence table showing the correspondence between predetermined identification information and the direction of existence.
[0092] For example, in the case of traffic signs applied to roads such as no entry or stop signs, the target road is the target of the direction of existence. On the other hand, in the case of traffic signs that call attention to the surrounding environment of the road, such as the presence of a school nearby or a pedestrian crossing nearby, and assist with traffic rules, the traffic sign is the target of the direction of existence. Note that for speed limit traffic signs, either the target road or the traffic sign may be the target of the direction of existence, and it is preferable that which one is to be selected be determined in advance by settings of the manufacturer of the display device 1, the manufacturer of the vehicle 8, the user of the display device 1, or the like.
[0093] The direction indication unit 25 has a function of indicating to the driver V the direction of presence, which is the direction in which either the target road to which the traffic sign applies or the traffic sign itself is located, based on the position of the traffic sign and the position of the road ahead of the vehicle 8.
[0094] Specifically, the direction indication unit 25 receives the position information of the traffic signs and roads from the image processing device 15, and also receives the target information of the existing direction from the direction target determination unit .
[0095] When the target of the existence direction is a target road, the direction indication unit 25 determines the road to which the traffic sign is applied as the target road based on the position of the traffic sign and the position of the road ahead of the vehicle 8. Then, using a predetermined reference position and coordinate information of the target road, it calculates and acquires the direction in which the target road exists relative to the reference position as the existence direction θ.
[0096] On the other hand, when the target of the presence direction is a traffic sign, the direction indication unit 25 uses the reference position and the coordinate information of the traffic sign to calculate and obtain the direction of the traffic sign relative to the reference position as the presence direction θ.
[0097] Here, the reference position may be the viewpoint (eye) of the driver V or a position corresponding to the viewpoint. In this case, the direction in which the target road or traffic sign exists as seen from the driver V corresponds to the presence direction.
[0098] The direction indication unit 25 outputs the existence direction θ to the image generation unit 26, and indicates the existence direction θ to the driver V by causing the image generation unit 26 to generate a projected image including the existence direction θ.
[0099] The image generating unit 26 has a function of generating a projected image in which a virtual sign image is arranged within the image. Specifically, the image generating unit 26 generates a projected image of a predetermined image size in which a reference position is aligned with respect to a forward image, and in which a virtual sign image is arranged in a direction of the existence direction θ from the reference position. This predetermined image size is preferably an image size in which the size of the virtual image of the projected image is smaller than that of the foreground in order to narrow the projection angle of the projection unit 10 onto the front windshield 71.
[0100] The virtual sign image is placed in the projected image by replacing the image area in the direction of the existence direction θ from the reference position with the virtual sign image. The image generating unit 26 can also place other information such as a vehicle speed display in the projected image. Furthermore, when no traffic sign is included in the forward image, the image generating unit 26 generates a projected image that does not include the virtual sign image.
[0101] The output unit 27 outputs image data of the projected image generated by the image generation unit 26 to the projection unit 10. That is, the output unit 27 controls the LD driver 207 and the MEMS controller 208 to cause the projection unit 10 to project the projected image generated by the image generation unit 26 and display it as a virtual image.
[0102] <Example of processing by the control unit 20> Next, the flow of processing by the control unit 20 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of processing by the control unit 20.
[0103] First, in step S81, the recognition unit 22 inputs a sign image from the image processing device 15.
[0104] Next, in step S82, the identification unit 22 acquires a virtual sign image and identification information corresponding to the traffic sign indicated by the sign image based on the sign image, by referring to the image storage unit 21. Then, the identification unit 22 outputs the acquired virtual sign image to the image generation unit 26, and outputs the identification information to the direction object determination unit 23.
[0105] Next, in step S83, the direction target determination unit 23 determines either the target road or the traffic sign as the target of the existing direction to be indicated to the driver V, based on the identification information of the traffic sign input from the identification unit 22. Then, information indicating the target of the existing direction thus determined is output to the direction indication unit 25.
[0106] Next, in step S84, the direction indication unit 25 inputs the position information of the traffic signs and the roads from the image processing device 15. Note that this step S84 may be performed at any timing between the above-mentioned steps S81 to S83.
[0107] Next, in step S85, an existence direction in which either a target road to which the traffic sign is applied or the traffic sign itself exists is acquired. If the target of the existence direction is the target road, the direction indication unit 25 determines the road to which the traffic sign is applied as the target road based on the position of the traffic sign and the position of the road ahead of the vehicle 8. Then, using a predetermined reference position and coordinate information of the target road, the direction in which the target road exists relative to the reference position is calculated and acquired as an existence direction θ.
[0108] On the other hand, when the target of the presence direction is a traffic sign, the direction indication unit 25 uses the reference position and the coordinate information of the traffic sign to calculate and obtain the direction of the traffic sign relative to the reference position as the presence direction θ.
[0109] Thereafter, the direction indicating unit 25 outputs the existence direction θ to the image generating unit 26.
[0110] Next, in step S86, the image generator 26 generates a projected image in which the virtual marker image is arranged in the direction of the existence direction θ from the reference position.
[0111] Next, in step S87, the output unit 27 outputs the image data of the projected image generated by the image generation unit 26 to the projection unit 10.
[0112] In this way, the display device 1 can generate a projected image for indicating to the driver the direction in which either the target road to which the traffic sign is applied or the traffic sign itself exists, and can display a virtual image of the projected image by the projection unit 10.
[0113] The projected image is updated as needed in response to the forward image that changes with the movement of the vehicle 8, and a virtual image of the updated projected image is superimposed on the foreground. When a traffic sign is no longer included in the forward image, a virtual image of the projected image that does not include the virtual sign image is superimposed on the foreground.
[0114] <Application example of the first embodiment> Next, application examples of the display device 1 will be described using specific examples of a forward image, a projected image, and a foreground in which the projected image is displayed in a superimposed manner.
[0115] 9 is a diagram showing an example of a forward image 80 captured by forward camera 164. The forward image 80 includes a leading vehicle 81, traffic signs 82 and 83, and roads 84 and 85. The traffic sign 82 is a road sign that applies to road 84, and the traffic sign 83 is a road sign that applies to road 85.
[0116] The sign area extraction unit 151 in the image processing device 15 extracts, from the forward image 80, a sign image 82a including a traffic sign 82 and a sign image 83a including a traffic sign 83, and outputs each of them to the sign position acquisition unit 153. In addition, the road area extraction unit 152 extracts, from the forward image 80, a road image 84a including a road 84 and a road image 85a including a road 85, and outputs each of them to the road position acquisition unit 154.
[0117] The sign position acquisition unit 153 acquires coordinate information of a sign position 82b, which is the position of the traffic sign 82 in the sign image 82a. It also acquires coordinate information of a sign position 83b, which is the position of the traffic sign 83 in the sign image 83a. One example of such a sign position is the position of a pixel that is on the most positive X-direction side and the most negative Y-direction side among the pixels that constitute the traffic sign in the sign image. Which pixel position is to be the sign position is determined in advance.
[0118] Similarly, the road position acquisition unit 154 acquires coordinate information of a road position 84b, which is the position of the road 84 in the road image 84a. It also acquires coordinate information of a road position 85b, which is the position of the road 85 in the road image 85a. One example of such a road position is a position corresponding to the center of gravity of the road image. It is predetermined which pixel position is to be the road position.
[0119] The identification unit 22 appropriately corrects the sign images 82a and 83a by pre-processing such as image rotation so as to facilitate pattern matching processing. Then, based on the sign images 82a and 83a, the image storage unit 21 is referenced to acquire a virtual sign image and identification information.
[0120] 9, traffic signs 82 and 83 are traffic signs indicating one-way streets, and so the recognition unit 22 acquires one-way virtual sign images as virtual sign images corresponding to the traffic signs 82 and 83, respectively. Alternatively, a no-entry virtual sign image indicating that entry onto the road is prohibited because it is a one-way street may be acquired as a virtual sign image corresponding to a one-way street. Such rules for acquiring virtual sign images are determined in advance.
[0121] Since a one-way traffic sign is a traffic rule that applies to a road, the direction target determination unit 23 determines the target road as the target in the direction of the existence of each of the traffic signs 82 and 83 based on the identification information of each of the traffic signs 82 and 83 input from the identification unit 22.
[0122] The direction indication unit 25 determines the road 84 closest to the sign position 82b in the forward image 80 as the target road of the traffic sign 82, and determines the road 84 closest to the sign position 83b as the target road of the traffic sign 83. In the example of Fig. 9, the roads 84 and 85 are at approximately the same distance from the sign position 82b, but the overlap with the target road of the traffic sign 83 is taken into consideration and the road 84 is determined as the target road.
[0123] Reference position V in Fig. 9 C is a position corresponding to the viewpoint (eye) of the driver V. The coordinates of the road position 84b are (X1, Y1), the coordinates of the road position 85b are (X2, Y2), and the reference position V C Coordinates (X C , Y C ), the direction indication unit 25 calculates the direction θ1 of the road 84 and the direction θ2 of the road 85 using the following equations (1) and (2), respectively. θ1=arctan{(Y1-Y C ) / (X1-X C )} ···(1) θ2=arctan{(Y2-Y C ) / (X2-X C )} ···(2) 10 is a diagram showing an example of a projected image 90 generated by the image generating unit 26. The projected image 90 includes within the image a virtual sign image 91 arranged in a direction θ1 in which a road 84 to which a traffic sign 82 is applied exists, a graphic image 92, and a vehicle speed image 93 arranged at a predetermined position. Here, the virtual sign image 91 is an example of a predetermined image.
[0124] The virtual sign image 91 virtually displays a no entry road sign indicating that entry is prohibited because it is a one-way street. The graphic image 92 is a graphic image indicating the direction of presence θ1. The vehicle speed image 93 is a text image indicating the vehicle speed of the vehicle 8 detected by the vehicle speed sensor 161. The vehicle speed image 93 is fixed at a predetermined position, and its position does not change even if the vehicle 8 moves and the forward image changes. Such a vehicle speed image 93 is an example of a fixed image.
[0125] In this embodiment, the graphic image 92 shows an example of a method of indicating a direction by displaying an arrow and a three-dimensional cone-shaped figure, but any other method may be used as long as the direction can be indicated. For example, a text image showing a direction can be displayed by displaying words such as "Right" or "Left front". In this case, for example, the display device 1 can display a text image which is a text or language such as "Right front", "Right front", or "Direction at an angle θ1 degrees to the right of the horizontal" in place of the arrow and the three-dimensional cone-shaped figure, or together with the arrow and the three-dimensional cone-shaped figure. The text image 94 showing the text "Right front" in FIG. 10 is an example of such a text image.
[0126] 11 is a diagram showing an example of a foreground 100 of a vehicle 8 in this embodiment. The foreground 100 is an image visually recognized by a driver V, which is generated when the display device 1 projects a projected image 90 onto the front windshield 71 and a virtual image 90' of the projected image 90 is superimposed on the projected image 90. As shown in FIG. 11, the size of the virtual image 90' is smaller than that of the foreground 100.
[0127] A virtual sign virtual image 91', which is a virtual image of the virtual sign image 91, and a graphic virtual image 92', which is a virtual image of the graphic image 92, are displayed in a direction along the existence direction θ1 of a road 84, which is a target road to which the traffic sign 82 is applied. In addition, a vehicle speed virtual image 93', which is a virtual image of a vehicle speed image 93, is displayed at a predetermined position.
[0128] By displaying the virtual sign virtual image 91' corresponding to the traffic sign 82 in a direction along the existence direction θ1, the driver V can recognize the traffic sign 82 and the road 84 to which the traffic sign 82 applies without moving his or her line of sight.
[0129] However, the virtual sign virtual image 91' does not have to be displayed strictly in the direction along the existence direction θ1, and may be displayed in a direction slightly shifted from the existence direction θ1 as long as the driver V can recognize the direction in which the target road or traffic sign exists.
[0130] Here, for example, when attempting to display a virtual sign virtual image 91' or a graphic virtual image 92' by superimposing it on a road 84 in the foreground 100, a virtual image larger than the virtual image 90' is required, and a wider projection angle is required for the projection unit 10.
[0131] In contrast, in the embodiment, by indicating the direction θ1 in which the road 84 exists, the driver V is able to recognize the road 84 to which the traffic sign 82 applies without superimposing a virtual sign virtual image 91' or a graphic virtual image 92' at the position where the road 84 is visible in the foreground 100.
[0132] 12 is a diagram showing an example of the foreground 100 after the vehicle 8 has moved from the state shown in FIG. 11. The foreground 100 changes in accordance with the movement of the vehicle 8, and the direction in which the road 84 exists as seen by the driver V also changes from θ1 to θ 11 The position where the virtual sign virtual image 91' is displayed changes in accordance with the change in the direction of the presence.
[0133] The position of the virtual graphic image 92' changes according to the change in the direction of the vehicle 8, and the shape of the virtual graphic image 92' also changes. The three-dimensional cone shape of the virtual graphic image 92' changes, so that the direction can be indicated with more reality than when the direction is indicated by a simple arrow or the like. The position of the virtual vehicle speed image 93' is fixed regardless of the movement of the vehicle 8, and only the numerical value indicating the vehicle speed changes.
[0134] The virtual sign virtual image 91' and the graphic virtual image 92' are arranged around the vehicle speed virtual image 93' on the line connecting the reference position Vc and the road position 84b, and their positions change on the circumference around the vehicle speed virtual image 93' depending on the direction of existence.
[0135] Although an example is shown in Figs. 9 and 10 showing only the direction θ1 of the road 84 to which the traffic sign 82 is applied, the direction θ2 of the road 85 to which the traffic sign 83 is applied can also be shown in the same way.
[0136] <Functions and Effects of Display Device 1> Next, the effects of the display device 1 will be described.
[0137] Some display devices, such as head-up displays, recognize traffic signs, such as road signs and road markings, ahead of the vehicle based on an image of the road ahead captured by a camera mounted on the vehicle, and superimpose a virtual sign image on the position of the road visible in the foreground.
[0138] However, in such a device, when a road to which a traffic sign is applied or a traffic sign is visually recognized near an edge of the foreground, etc., it is required to superimpose and display a virtual image of a virtual sign image at the position of the traffic sign in the foreground. For example, it is required to superimpose and display a virtual image larger in size than the virtual image 90' of the projected image in FIG.
[0139] Here, Fig. 13 is a diagram for explaining the widening of the angle of the projection unit 10. As shown in Fig. 13, in order to allow the driver V to view a virtual image I' that is larger in size than the virtual image I, a projection angle of view φ' wider than the projection angle of view φ for the virtual image I is required. When the projection angle of view is widened, the display device may become larger, or a complex and expensive projection lens may be required to ensure the quality of the virtual image.
[0140] In contrast, in this embodiment, the direction in which either the target road or the traffic sign exists is shown to the driver V. For example, a virtual sign image is arranged in the image along the direction of existence, and the image generating unit 26 generates a projected image in which a figure indicating the direction of existence is included in the image, and a virtual image of this projected image is displayed to show the direction of existence.
[0141] By indicating the direction, it is not necessary to superimpose a virtual image of a predetermined image such as a virtual sign image at the position of the target road visually recognized in the foreground, and it is not necessary to widen the projection angle of view. As a result, even with a narrow projection angle of view such as the projection angle of view φ in FIG. 13, it is possible to indicate either the target road or the traffic sign.
[0142] Meanwhile, there are traffic signs that are applied to roads, such as no entry or stop signs, and auxiliary signs that call attention to the surrounding environment of the road, such as the presence of a school nearby or a pedestrian crossing nearby. In the case of traffic signs that are applied to roads, it is better to be able to recognize the target road, but in the case of auxiliary traffic signs, it is better to be able to recognize the traffic sign itself rather than the target road.
[0143] In contrast, the present embodiment includes a direction target determination unit that determines either a target road or a traffic sign as a target of the existing direction based on the traffic sign. This makes it possible to indicate the existing direction of the target road when applied to a road, and to indicate the existing direction of the traffic sign when applied to an auxiliary traffic sign, and to provide the driver V with appropriate information according to the traffic sign.
[0144] In this embodiment, the direction of existence is indicated, which changes in accordance with the movement of the vehicle 8. This makes it possible to provide the driver V with appropriate information in accordance with the change in the situation due to the movement of the vehicle 8.
[0145] In this embodiment, the virtual sign image 91 is arranged around a fixed image such as a vehicle speed image 93 fixedly arranged in the projected image along the direction of presence to indicate the direction of presence. Fixed images are often visually recognized by the driver V, and are often fixed in positions that are easy to see. By arranging the virtual sign image 91 around such a fixed image, the direction of presence can be indicated so that the driver V can easily see it.
[0146] In the present embodiment, an example has been shown in which the virtual sign image 91 and the graphic image 92 are displayed together as the predetermined image, but either one of them may be displayed. The graphic image 92 may be a simple arrow graphic image or a character image indicating an existence direction such as an angle. In addition, an example has been shown in which the position of the virtual sign image 91 is changed along the existence direction to indicate the existence direction, but the existence direction may be indicated by only rotating the virtual sign image 91 without changing the position.
[0147] In the present embodiment, the image processing device 15 acquires position information of traffic signs and roads based on the image captured by the front camera 164, but the present invention is not limited to this. Position information of roads and traffic signs ahead of the vehicle 8 may be acquired using information input from the car navigation system 11. Furthermore, the vehicle 8 may be provided with a gyro sensor that detects the attitude of the vehicle, and the detection value of this gyro sensor may be used.
[0148] Although the image storage unit 21 stores virtual sign images, the present invention is not limited to this. It may store text images that express the information indicated by the traffic sign in text, or it may store graphic images that display a simplified version of the traffic sign.
[0149] Furthermore, an example has been shown in which the identification unit 22 acquires the virtual sign image and identification information by referring to the image storage unit 21, and the direction target determination unit 23 determines the target of the direction of existence by referring to the direction table storage unit 24, but the present invention is not limited to this. The identification unit 22 and the direction target determination unit 23 may realize their functions by the method using machine learning described above.
[0150] In the present embodiment, an example is shown in which the direction of existence θ is obtained by two-dimensional image processing using an image captured by the monocular front camera 164, but the present invention is not limited to this. The direction of existence θ may be obtained by three-dimensional image processing based on a projection relationship using distance information to at least one of a target road or a traffic sign detected by a stereo camera or the like as the front camera 164 and a known positional relationship between the vehicle 8 and the driver V. In this way, the driver V can be made to visually recognize a virtual image of a predetermined image with higher reality, and the direction of existence can be indicated.
[0151] [Second embodiment] Next, a display device 1a according to a second embodiment will be described.
[0152] When the vehicle 8 moves, the traffic sign may no longer be included in the foreground of the vehicle 8, and the traffic sign information may no longer be acquired by the forward camera 164. In this case, the virtual sign image corresponding to the traffic sign for which the information is no longer acquired may be hidden, but in the case of a traffic sign applied to a certain section of the road such as a speed limit, it may be preferable to make the virtual image of the virtual sign image visible to the driver V even after the information is no longer acquired.
[0153] In this embodiment, when traffic sign information is no longer acquired by the front camera 164 due to the movement of the vehicle 8, a projected image is generated in which the virtual sign image is fixed at a predetermined position within the image. This allows the driver V to visually recognize the virtual image of the virtual sign image even when the information is no longer acquired.
[0154] In addition, depending on the traffic sign, a projected image in which the virtual sign image is fixed at a predetermined position within the image is generated, or a projected image that does not include the virtual sign image is generated. This makes it possible to select, depending on the traffic sign, whether to display a virtual image of the virtual sign image or not to display the virtual sign image when information is no longer acquired.
[0155] <Example of functional configuration of control unit 20> Fig. 14 is a block diagram showing an example of a functional configuration of a control unit 20a included in the display device 1a. As shown in Fig. 14, the control unit 20a includes a continuation table storage unit 28, an information holding unit 29, a direction indication unit 25a, and an image generation unit 26a. The direction indication unit 25a also includes a determination unit 241 and a selection unit 242.
[0156] Of these, the function of the continuation table storage unit 28 is realized by the HDD 210 and the like, and the function of the information holding unit 29 is realized by the RAM 204 and the like.
[0157] The continuation table storage unit 28 stores a correspondence table showing the correspondence between predetermined identification information and display continuation information. Here, the display continuation information refers to information indicating either fixing the virtual sign image at a predetermined position or hiding the virtual sign image. For example, when fixing the virtual sign image, the display continuation information is set to 1, and when hiding the virtual sign image, the display continuation information is set to 0.
[0158] The information storage unit 29 temporarily stores the identification information of the traffic sign before the movement of the vehicle 8. Since the vehicle 8 is constantly moving while traveling, it is preferable to set the identification information of the traffic sign before the movement of the vehicle 8 to the identification information one cycle before a predetermined point in the imaging cycle of the front camera 164, since this makes it possible to obtain the identification information immediately before the movement.
[0159] The direction indication unit 25a inputs the identification information from the direction object determination unit 23. Then, when traffic sign information is acquired, the identification information is output to the information storage unit 29 to be stored. On the other hand, when traffic sign information is no longer acquired, the display continuation information is output to the image generation unit 26a. The direction indication unit 25a has other functions similar to those of the direction indication unit 25.
[0160] The determination unit 241 in the direction indication unit 25a determines whether or not traffic sign information has been acquired based on the sign position information input from the image processing device 15, and outputs the determination result to the selection unit 242.
[0161] Specifically, when a traffic sign is not included in the foreground of the vehicle 8 and traffic sign information is not acquired by the front camera 164, the image processing device 15 outputs no sign information as described in Fig. 5. When the determination unit 241 receives no sign information, it determines that traffic sign information has not been acquired.
[0162] When traffic sign information is not acquired, the selection unit 242 selects, based on the traffic sign, either to fix the virtual sign image at a predetermined position or to hide the virtual sign image, and outputs the selection result to the image generation unit 26.
[0163] Specifically, when it is determined that the traffic sign information has not been acquired, the selection unit 242 accesses the information storage unit 29 to acquire the identification information of the traffic sign temporarily stored in the information storage unit 29. Then, based on the identification information, the selection unit 242 refers to the continuation table storage unit 28 to acquire display continuation information. This display continuation information corresponds to the above selection result.
[0164] When display continuation information (e.g., 1) indicating that the virtual sign image is to be fixed at a predetermined position is input, the image generating unit 26a generates a projected image in which the virtual sign image is fixed at a predetermined position within the image. When display continuation information (e.g., 0) indicating that the virtual sign image is not to be displayed is input, the image generating unit 26a generates a projected image that does not include a virtual sign image. On the other hand, when traffic sign information is acquired (when the sign position information is not no-sign information), the image generating unit 26a generates a projected image in which the virtual sign image is positioned in the presence direction θ from the reference position.
[0165] <Example of processing by control unit 20a> Next, the flow of processing by the control unit 20a will be described with reference to Fig. 15. Fig. 15 is a flow chart showing an example of processing by the control unit 20a. The processing from steps S151 to S154 in Fig. 15 is similar to the processing from steps S81 to S84 in Fig. 8, so duplicated explanations will be omitted here. In addition to the above steps, explanations of parts that overlap with the explanation in Fig. 8 will be omitted as appropriate.
[0166] In step S155, the determination unit 241 determines, based on the sign position information input from the image processing device 15, whether or not traffic sign information has been acquired.
[0167] If it is determined in step S155 that the identification information of the traffic sign has been acquired (Yes in step S155), then in step S156, the direction indication unit 25a outputs the identification information of the traffic sign to the information storage unit 29 to be stored therein.
[0168] Next, in step S157, the direction indication unit 25a obtains the direction θ in which either the target road or the traffic sign is present, and outputs this to the image generation unit 26a.
[0169] Next, in step S158, the image generator 26a generates a projected image in which the virtual marker image is arranged in the direction of the existence direction θ from the reference position. After that, the process proceeds to step S163.
[0170] On the other hand, if it is determined in step S155 that traffic sign information has not been acquired (step S155, No), in step S159, the selection unit 242 accesses the information storage unit 29 and acquires the identification information of the traffic sign temporarily stored in the information storage unit 29.
[0171] Next, in step S160, the selection unit 242 refers to the continuation table storage unit 28 based on the identification information to obtain display continuation information.
[0172] Next, in step S161, the image generating unit 26a determines whether or not to continue displaying the traffic sign based on the display continuation information.
[0173] If it is determined in step S161 that the virtual sign image should be continuously displayed (Yes in step S161), the image generating unit 26a generates a projected image in which the virtual sign image is fixed at a predetermined position in the image in step S162. On the other hand, if it is determined that the virtual sign image should not be continuously displayed (No in step S161), the image generating unit 26a generates a projected image that does not include the virtual sign image in step S163.
[0174] Next, in step S164, the output unit 27 outputs to the projection unit 10 the projected image generated by the image generation unit 26a.
[0175] In this way, when traffic sign information is no longer acquired by the front camera 164 due to the movement of the vehicle 8, the display device 1a can generate either a projected image in which the virtual sign image is fixed at a predetermined position within the image, or a projected image that does not include a virtual sign image, depending on the traffic sign. Then, a virtual image of the projected image by the projection unit 10 can be displayed.
[0176] The virtual sign image fixed at a predetermined position can be made invisible by the image generating unit 26a when, for example, new traffic sign information is acquired in the forward image due to the movement of the vehicle 8.
[0177] <Application example of the second embodiment> Next, application examples of the display device 1a will be described using specific examples of a forward image, a projected image, and a foreground image in which the projected image is superimposed and displayed.
[0178] 16 is a diagram showing an example of a foreground 100 of a vehicle 8 in this embodiment. The foreground 160 is an image visually recognized by the driver V, in which a virtual image 171 of a projected image is superimposed and displayed by the display device 1a projecting an image onto the front windshield 71. In addition to the virtual image 171, the foreground 160 includes a traffic sign 172, a virtual sign virtual image 173, a graphic virtual image 174, and a vehicle speed virtual image 175.
[0179] In the example of FIG. 16, a traffic sign is defined as an object of the direction of presence, and the direction of presence θ 12 A virtual sign virtual image 173 and a graphic virtual image 174 are displayed along the line. A vehicle speed virtual image 175 is displayed fixedly at a predetermined position.
[0180] Next, Fig. 17 is a diagram showing an example of the foreground 160 after the vehicle 8 has moved from the state of Fig. 16. The foreground 160 changes in response to the movement of the vehicle 8, and no longer includes the traffic sign 172 in Fig. 16. In other words, the state is one in which traffic sign information is not acquired.
[0181] In the example shown in Fig. 17, the traffic sign is a road sign of a speed limit, which is applied to a certain section of the road, so it is preferable to display it continuously even after the information of the traffic sign is no longer acquired. For this reason, the correspondence table of the continuation table storage unit 28 is determined so that the road sign of the speed limit is continuously displayed. Based on this correspondence table, the display device 1a fixes the virtual sign virtual image 183 next to the vehicle speed virtual image 182 and continuously displays it. Displaying it next to the vehicle speed virtual image 182 is preferable because it makes it easier for the driver V to compare the vehicle speed of the vehicle 8 with the speed limit.
[0182] 17 shows an example in which the virtual sign virtual image 183 is fixedly displayed next to the vehicle speed virtual image 182, but the display position of the virtual sign virtual image 183 is not limited to this and may be any position. A position where the driver V can easily see is preferable.
[0183] <Functions and Effects of the Display Device 1a> As described above, in this embodiment, when traffic sign information cannot be acquired due to the movement of the vehicle 8, a projected image is generated in which the virtual sign image is fixed at a predetermined position within the image. This allows the driver V to continue to view the virtual image of the virtual sign image even when traffic sign information cannot be acquired.
[0184] In the present embodiment, either the virtual sign image is fixed at a predetermined position or the virtual sign image is hidden depending on the traffic sign. As a result, when traffic sign information is no longer acquired, either the virtual image of the virtual sign image fixed at a predetermined position is displayed to the driver V or the virtual sign image is hidden depending on the traffic sign. Then, appropriate information can be provided to the driver V depending on the traffic sign.
[0185] <Other suitable examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.
[0186] In the above-described embodiment, the control unit 20 may be configured to have some or all of the functions of the image processing device 15. Also, some of the functions of the control unit 20 may be configured to be provided by other components such as the image processing device 15, the engine ECU 12, the detection device 16, etc. Also, it is possible to configure the information stored in the image storage unit 21, the direction table storage unit 24, etc. to be stored in an external server, so that the stored information can be referenced by accessing the external server via a network.
[0187] In addition, in the embodiment, a road is an example of a route, a vehicle is an example of a moving body, and a HUD is an example of a display device, but the present invention is not limited to these. The embodiment can be applied to routes other than roads, moving bodies other than vehicles, and display devices other than HUDs.
[0188] The embodiment also includes a display method. For example, the display method is a display method by a display device that allows a driver of a moving body to visually recognize a virtual image of a projected image, and includes a step of generating the projected image including a predetermined image acquired based on a traffic sign, which is at least one of a road sign and a road marking, by an image generating unit, and a step of outputting an existence direction of either a target road to which the traffic sign is applied or the traffic sign based on a position of the traffic sign and a position of a road ahead of the moving body, and causing the image generating unit to generate the projected image including the existence direction. By such a display method, it is possible to obtain the same effect as the above-mentioned display device.
[0189] The embodiment also includes a program. For example, the program causes a computer to execute a process in which an image generating unit generates a projected image including a predetermined image acquired based on a traffic sign, which is at least one of a road sign and a road marking, and outputs an existence direction of either a target road to which the traffic sign is applied or the traffic sign based on a position of the traffic sign and a position of a road ahead of a moving object, and causes the image generating unit to generate the projected image including the existence direction. Such a program can provide the same effect as the display device described above.
[0190] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above. [Explanation of symbols]
[0191] 1 Display device 2. In-vehicle systems 8 Vehicles (an example of a moving object) 10 Projection part 15 Image processing device 151 Sign area extraction part 152 Road area extraction part 153 Sign position acquisition unit 154 Road position acquisition unit 16 Detection device 164 Front Camera 20 Control section 21 Image storage section 22 Identification section 23 Directional Object Determination Unit 24 Direction table storage section 25 Direction indicator 26 Image Generation Unit 27 Output section 28 Continuation table storage section 29 Information Holding Department 71 Front Windshield 80 Front Image 82 Traffic signs 84 Road (example of route) 90 Projected image 90' Virtual Image 91 Virtual sign image (example of a specified image) 91' Virtual Sign 92 Graphic Images 92' Graphical Virtual Image 93 Vehicle speed image 93' Vehicle speed image 100 foreground 210 HDD Vc reference position θ, θ1, θ2, θ 11 , θ 12 direction of existence φ Projection angle of view [Prior art documents] [Patent documents]
[0192] [Patent Document 1] JP 2018-200626 A
Claims
1. A display device that displays a virtual image of a projected image, an image generating unit that generates the projected image including a predetermined image acquired based on a traffic sign; a direction indicating unit that outputs a direction in which either a target route to which the traffic sign is applied or the traffic sign exists based on a position of the traffic sign and a position of a route ahead of a moving object, and causes the image generating unit to generate the projected image including the direction in which the traffic sign exists; A direction target determination unit that determines either the target route or the traffic sign as a target of the direction of existence based on the traffic sign. Display device.
2. A display device for displaying a virtual image of a projected image, comprising: an image generating unit that generates the projected image including a predetermined image acquired based on a traffic sign; a direction indicating unit that outputs a direction in which either a target route to which the traffic sign is applied or the traffic sign exists based on a position of the traffic sign and a position of a route ahead of the moving body, and causes the image generating unit to generate the projected image including the direction in which the traffic sign exists; The direction indicator indicates the direction of existence that changes in accordance with the movement of the moving object. Display device.
3. A display device for displaying a virtual image of a projected image, comprising: an image generating unit that generates the projected image including a predetermined image acquired based on a traffic sign; a direction indicating unit that outputs a direction in which either a target route to which the traffic sign is applied or the traffic sign exists based on a position of the traffic sign and a position of a route ahead of the moving body, and causes the image generating unit to generate the projected image including the direction in which the traffic sign exists; The direction indication unit indicates the direction of existence by causing the image generation unit to generate the projected image including a graphic image or a text image indicating the direction of existence. Display device.
4. A display device for displaying a virtual image of a projected image, comprising: an image generating unit that generates the projected image including a predetermined image acquired based on a traffic sign; a direction indicating unit that outputs a direction in which either a target route to which the traffic sign is applied or the traffic sign exists based on a position of the traffic sign and a position of a route ahead of the moving body, and causes the image generating unit to generate the projected image including the direction in which the traffic sign exists; The direction indication unit indicates the presence direction by causing the image generation unit to generate the projected image in which the predetermined image is arranged along the presence direction. Display device.
5. A display device for displaying a virtual image of a projected image, comprising: an image generating unit that generates the projected image including a predetermined image acquired based on a traffic sign; a direction indicating unit that outputs a direction in which either a target route to which the traffic sign is applied or the traffic sign exists based on a position of the traffic sign and a position of a route ahead of the moving body, and causes the image generating unit to generate the projected image including the direction in which the traffic sign exists; The direction indication unit indicates the presence direction by causing the image generation unit to generate the projected image in which the predetermined image is arranged along the presence direction around a fixed image fixedly arranged in the projected image. Display device.
6. The direction of presence is a direction in which either the target route or the traffic sign is present as seen from the driver of the moving object. The display device according to claim 1 .
7. The predetermined image is a virtual sign image showing at least one of a figure and a character corresponding to the traffic sign. The display device according to claim 1 .
8. An identification unit for identifying the traffic sign is provided. The display device according to claim 1 .
9. The image generating unit includes: When the information on the traffic sign is no longer acquired due to the movement of the moving object, the projected image is generated in which the predetermined image is fixed at a predetermined position within the image. The display device according to claim 4 or 5.
10. The image generating unit includes: When the information on the traffic sign is no longer acquired due to the movement of the moving body, the projected image is generated based on the traffic sign, either in which the predetermined image is fixed at a predetermined position within the image, or in which the predetermined image is not included. The display device according to claim 4 or 5.
11. A mobile object comprising the display device according to any one of claims 1 to 10.
12. A display method for displaying a virtual image of a projected image by a display device, comprising: generating the projected image including a predetermined image acquired based on a traffic sign by an image generating unit; a step of outputting a direction in which either a target route to which the traffic sign is applied or the traffic sign exists based on a position of the traffic sign and a position of a route ahead of the moving body, and causing the image generating unit to generate the projected image including the direction of existence; determining, based on the traffic sign, either the target route or the traffic sign as a target of the direction of existence; Display method.
13. an image generating unit generates a projected image including a predetermined image acquired based on a traffic sign; outputting a direction in which either a target route to which the traffic sign is applied or the traffic sign exists based on a position of the traffic sign and a position of a route ahead of the moving body, and generating the projected image including the direction of existence by the image generating unit; Based on the traffic sign, either the target route or the traffic sign is determined as a target in the direction of existence. A program that causes a computer to carry out processing.
Citation Information
Patent Citations
Vehicle display control device and vehicle display unit
JP2018156574A
Vehicle display control device and display control program
JP2018200626A
Head-up display device, navigation device, and display method
JP2019032243A
Head-up display device
JP2019059248A
Virtual image display unit, virtual image display method, and virtual image display program
JP2019061088A
