Information processing apparatus and information processing method

Abstract

To present the content of driving assistance in an understandable manner.SOLUTION: An information processing apparatus includes a display control section that controls display of a driving assistance image obtained by adding, to an image indicating a state of a periphery of a vehicle, assistance information to be added to an object within the image to assist driving, and that controls a color and a shape of the assistance information based on a state of the object with respect to the vehicle. The present technology can be applied to, for example, a vehicle.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present technology relates to an information processing apparatus and an information processing method, and more particularly, to an information processing apparatus and an information processing method capable of presenting the content of driving support in an easy-to-understand manner.

Background Art

[0002] Conventionally, a technique has been proposed in which an object in an image around a host vehicle is surrounded by a frame having a size or thickness corresponding to a collision risk and displayed (see, for example, Patent Document 1).

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When driving is supported using an image as in the invention described in Patent Document 1, it is desirable to present the content of driving support in an easy-to-understand manner.

[0005] The present technology has been made in view of such a situation, and enables the content of driving support to be presented in an easy-to-understand manner.

Means for Solving the Problems

[0006] An information processing apparatus according to a first aspect of the present technology controls the display of a driving support image in which support information for supporting driving is added to an image showing the state around a vehicle, and is added to an object in the image, and includes a display control unit that controls the color and shape of the support information based on the state of the object with respect to the vehicle.

[0007] In the information processing method according to the second aspect of the present technology, an information processing apparatus controls the display of a driving support image in which support information for an object around the vehicle is added to an image showing the state around the vehicle, and controls the color and shape of the support information based on the state of the object with respect to the vehicle.

[0008] The information processing apparatus according to the second aspect of the present technology includes a display control unit that controls the display of a driving support image in which support information for supporting driving, which is grounded or displayed on the ground, is added to an image showing the state around the vehicle.

[0009] In the information processing method according to the second aspect of the present technology, an information processing apparatus controls the display of a driving support image in which support information for supporting driving, which is grounded or displayed on the ground in the image, is added to an image showing the state around the vehicle.

[0010] In the first aspect of the present technology, the display of a driving support image in which support information for supporting driving, which is added to an object in the image, is added to an image showing the state around the vehicle is controlled, and the color and shape of the support information are controlled based on the state of the object with respect to the vehicle.

[0011] In the second aspect of the present technology, the display of a driving support image in which support information for supporting driving, which is grounded or displayed on the ground in the image, is added to an image showing the state around the vehicle is controlled.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments for carrying out the present technology will be described. The description will be made in the following order. 1. Embodiment 2. Modification Example 3. Others

[0014] <<1. Embodiment>> With reference to FIGS. 1 to 17, embodiments of the present technology will be described.

[0015] <Configuration Example of Vehicle 1> FIG. 1 shows a configuration example of the functions of vehicle 1 to which the present technology is applied, mainly the functions related to the present technology.

[0016] Vehicle 1 is equipped with a vehicle control system 11. The vehicle control system 11 includes a control unit 21, an external sensor 22, a GNSS (Global Navigation Satellite System) receiver 23, an in-vehicle sensor 24, a vehicle sensor 25, an input unit 26, a communication unit 27, and a display unit 28.

[0017] The control unit 21 includes one or more ECUs (Electronic Control Units). The control unit 21 executes various processes of the vehicle 1 and controls each part.

[0018] The external sensor 22 includes various sensors used for detecting various information in the external world (the outside world) of the vehicle 1, and supplies sensor data from each sensor to the control unit 21. The type and number of sensors included in the external sensor 22 are not particularly limited.

[0019] For example, the external sensor 22 includes a camera 41, a radar 42, a LiDAR 43, and a sonar 44. The number and type of the camera 41, the radar 42, the LiDAR 43, and the sonar 44 are not particularly limited.

[0020] For example, the imaging method of the camera 41 is not particularly limited. For example, cameras of various imaging methods such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, and an infrared camera, which are imaging methods capable of distance measurement, are applied to the camera 41 as needed. Not limited to this, the camera 41 may be simply for acquiring a captured image regardless of distance measurement.

[0021] The GNSS receiver 23 receives GNSS signals from GNSS satellites and supplies them to the control unit 21.

[0022] The in-vehicle sensor 24 includes various sensors used for detecting various information inside the vehicle, and supplies sensor data from each sensor to the control unit 21. The type and number of sensors included in the in-vehicle sensor 24 are not particularly limited.

[0023] For example, the in-vehicle sensor 24 includes a camera 51 and a microphone 52. The number and type of the camera 51 and the microphone 52 are not particularly limited. Also, the imaging method of the camera 51 is not particularly limited, similar to the camera 51 of the external sensor 22.

[0024] The vehicle sensor 25 includes various sensors used for detecting the state of the vehicle 1, and supplies sensor data from each sensor to the control unit 21. The type and number of sensors included in the vehicle sensor 25 are not particularly limited.

[0025] For example, the vehicle sensor 25 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU (Inertial Measurement Unit)) that integrates them. For example, the vehicle sensor 25 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the operation amount of the accelerator pedal, and a brake sensor that detects the operation amount of the brake pedal. For example, the vehicle sensor 25 includes a rotation sensor that detects the rotation speed of the engine or the motor, a tire pressure sensor that detects the air pressure of the tire, a slip rate sensor that detects the slip rate of the tire, and a wheel speed sensor that detects the rotation speed of the wheel. For example, the vehicle sensor 25 includes a battery sensor that detects the remaining amount and temperature of the battery, and a shock sensor that detects an external shock.

[0026] The input unit 26 includes an input device for a passenger to input data, instructions, etc., generates an input signal based on the data, instructions, etc. input by the input device, and supplies it to the control unit 21. The input unit 26 may include, for example, not only an input device that a passenger directly operates to input data, instructions, etc., but also an input device that inputs instructions, etc. by voice or gesture.

[0027] The communication unit 27 is equipped with various communication devices, communicates with various in-vehicle and out-of-vehicle devices, other vehicles, servers, base stations, etc., and transmits and receives various types of data. The communication unit 27 can communicate using a plurality of communication methods as necessary.

[0028] The display unit 28 is equipped with various display devices and displays visual information. The number and type of display devices included in the display unit 28 are not particularly limited. As the display devices included in the display unit 28, for example, a display device that presents visual information by displaying an image itself or a projector device that presents visual information by projecting an image can be applied. Note that the display device may be a device that displays visual information within the user's field of view, such as a head-up display, a transmissive display, or a wearable device having an AR (Augmented Reality) function, in addition to a display device having a normal display.

[0029] FIG. 2 schematically shows an arrangement example of a part of the functions of the vehicle 1 in FIG. 1.

[0030] The vehicle 1 includes an ECU 71, a front sensing unit 72, a sonar 73, a ToF camera 74, and a display unit 75.

[0031] The ECU 71 constitutes, for example, a part of the control unit 21 in FIG. 1 and is provided in front of the vehicle 1.

[0032] The front sensing unit 72 constitutes, for example, a part of the external sensor 22 in FIG. 1 and is provided in front of the vehicle 1. The front sensing unit 72 includes, for example, a part of the camera 41, the radar 42, and the LiDAR 43 of the external sensor 22 and performs sensing in front of the vehicle 1.

[0033] The sonar 73 constitutes, for example, a part of the sonar 44 of the external sensor 22 in FIG. 1 and is provided in front of and below the vehicle 1. The sonar 73 performs sensing near the road surface in front of the vehicle 1.

[0034] The ToF camera 74 forms part of, for example, the camera 51 of the in-vehicle sensor 24 in FIG. 1 and is provided on the dashboard of the vehicle 1. The ToF camera 74 photographs the passengers (e.g., the driver) inside the vehicle.

[0035] The display unit 75 forms part of, for example, the display unit 28 in FIG. 1 and is provided on the front surface of the dashboard of the vehicle 1.

[0036] FIG. 3 schematically shows the vicinity of the dashboard 101 in the front of the interior of the vehicle 1.

[0037] Near the dashboard 101, a ToF camera 74, a display unit 75, and a head-up display (only the display 113 is shown in the figure) are provided.

[0038] The ToF camera 74 is provided on the dashboard 101 slightly closer to the driver's seat than the center in the left-right direction. The ToF camera 74 photographs, for example, a range including at least the head of the driver sitting in the driver's seat.

[0039] The display unit 75 is provided on the front surface of the dashboard 101 so as to extend horizontally in front of the driver's seat and the passenger seat and below the windshield 102. The display unit 75 has a configuration in which the displays 111L, 111R, 112L, and 112R are continuous in the left-right direction and integrated. The displays 111L, 111R, 112L, and 112R can each perform individual displays independently or perform integrated displays.

[0040] The displays 111L and 111R extend horizontally from near the left end of the driver's seat to near the right end of the passenger seat in front of the driver's seat and the passenger seat and below the windshield 102, facing rearward (rearward of the vehicle 1) as viewed from the driver's seat or the passenger seat. The display 111L is disposed in front of the driver's seat. The display 111R is disposed between the driver's seat and the passenger seat and in front of the passenger seat.

[0041] The displays 112L and 112R are provided substantially symmetrically on the left and right ends of the display unit 75. The display 112L is angled inward toward the vehicle interior at the left end of the display unit 75, facing diagonally rearward to the right (diagonally rearward to the right of the vehicle 1) as viewed from the driver's seat or the passenger seat. The display 112R is angled inward toward the vehicle interior at the right end of the display unit 75, facing diagonally rearward to the left (diagonally rearward to the left of the vehicle 1) as viewed from the driver's seat or the passenger seat.

[0042] The displays 111L and 111R display, for example, information for assisting driving, an image of the surroundings of the vehicle 1, infotainment-related information, etc. For example, mainly information for the driver is displayed on the display 111L. For example, infotainment-related information such as audio, video, websites, maps, etc. is displayed on the display 111R.

[0043] Also, as will be described later, a driving support image is displayed on the display 111L. The driving support image is, for example, an image in which support information for assisting driving is added to an image showing the state of the surroundings (e.g., the front) of the vehicle 1.

[0044] Displays 112L and 112R are mainly used as digital outer mirrors (electronic side mirrors) to replace conventional side mirrors. That is, Displays 112L and 112R are used in a CMS (Camera Monitoring System). For example, Display 112L displays an image of the rear left diagonal of Vehicle 1 captured by Camera 41. Display 112R displays an image of the rear right diagonal of Vehicle 1 captured by Camera 41.

[0045] The head-up display includes a Display 113 (hereinafter referred to as the HUD Display 113) provided in front of the driver's seat. For example, the HUD Display 113 may be constituted by a part of the windshield 102, or may be provided separately from the windshield 102. In the latter case, for example, the HUD Display 113 is attached to the windshield 102. Then, by using AR technology, visual information is projected onto the HUD Display 113, so that the visual information is superimposed within the driver's field of view.

[0046] The HUD Display 113 displays information for assisting driving, for example.

[0047] FIG. 4 shows a configuration example of functions realized by the control unit 21. The control unit 21 includes an information acquisition unit 151, a vehicle control unit 152, and a display control unit 153. The information acquisition unit 151 includes a detection unit 161, a self-position estimation unit 162, and an external information collection unit 163. The detection unit 161 includes an external state detection unit 171, a vehicle state detection unit 172, and a passenger state detection unit 173.

[0048] The external state detection unit 171 executes detection processing of the external state based on the sensor data from the external sensor 22. For example, the external state detection unit 171 detects or estimates the presence, contour, size, shape, position, movement, attributes (type, etc.) of objects around the vehicle 1. Objects around the vehicle 1 include, for example, other vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. The external state detection unit 171 detects or estimates, for example, the environment around the vehicle 1. The environment around the vehicle 1 includes, for example, weather, temperature, humidity, brightness, road surface conditions, etc. The external state detection unit 171 supplies information indicating the result of the detection processing to the vehicle control unit 152 and the display control unit 153.

[0049] The vehicle state detection unit 172 executes detection processing of the state of the vehicle 1 based on the sensor data from the vehicle sensor 25 and the information from the vehicle control unit 152. For example, the vehicle state detection unit 172 detects or estimates the driving state of the vehicle 1, the state of each part of the vehicle 1, the operation state of the vehicle 1's automated driving, etc. The vehicle state detection unit 172 supplies information indicating the result of the detection processing to the vehicle control unit 152 and the display control unit 153.

[0050] The passenger state detection unit 173 executes detection processing of the passenger's state based on the sensor data from the in-vehicle sensor 24. For example, the passenger state detection unit 173 detects or estimates the passenger's posture and line-of-sight direction. The passenger's posture includes, for example, the direction of the passenger's face, the position of the face, and the position of the eyes. For example, the passenger state detection unit 173 detects or estimates the passenger's physical condition, arousal level, concentration level, fatigue level, drunkenness level, driving operation, etc. For example, the passenger state detection unit 173 recognizes the passenger's gestures and speech content. The passenger state detection unit 173 supplies information indicating the result of the detection processing to the vehicle control unit 152 and the display control unit 153.

[0051] The self-position estimation unit 162 estimates the self-position of the vehicle 1 based on the GNSS signal from the GNSS receiver 23. The self-position estimation unit 162 supplies information indicating the estimation result to the vehicle control unit 152 and the display control unit 153.

[0052] Note that the self-position estimation unit 162 may estimate the self-position of the vehicle 1 using other technologies such as SLAM (Simultaneous Localization and Mapping).

[0053] The external information collection unit 163 communicates with various in-vehicle and out-of-vehicle devices, other vehicles, servers, base stations, etc. via the communication unit 27 to collect external information regarding the outside of the vehicle 1. The external information includes, for example, map information, traffic information, weather information, and information around the vehicle 1. The information around the vehicle 1 includes, for example, sightseeing information, information on various facilities such as commercial facilities, etc. The external information collection unit 163 supplies the collected external information to the vehicle control unit 152 and the display control unit 153.

[0054] The vehicle control unit 152 executes control of each part of the vehicle 1 based on various information supplied from the information acquisition unit 151 and input signals from the input unit 26, etc. For example, the vehicle control unit 152 executes control of the steering system, braking system, drive system, body system, various lights, car horn, etc. of the vehicle 1. For example, the vehicle control unit 152 executes control of level 1 to level 5 driving automation (driving support) of the vehicle 1. The vehicle control unit 152 supplies information indicating the state of the vehicle 1 to the information acquisition unit 151 based on the control state of the vehicle 1.

[0055] The display control unit 153 controls the display of various display images by the display unit 28. The display control unit 153 includes an image generation unit 181 and an output control unit 182.

[0056] The image generation unit 181 generates a display image to be displayed on the display unit 28 based on various information supplied from the information acquisition unit 151 and input signals from the input unit 26, etc. The display image includes the above-described driving support image. The image generation unit 181 supplies the display image to the output control unit 182.

[0057] The output control unit 182 controls the display of the display image by the display device by controlling the output of the display image to the display device provided in the display unit 28.

[0058] <Driving support image display process> Next, with reference to the flowchart of FIG. 5, the driving support image display process executed by the vehicle 1 will be described.

[0059] This process starts, for example, when a passenger instructs the display of a driving support image via the input unit 26, and ends when the passenger instructs the stop of the display of the driving support image.

[0060] In step S1, the vehicle 1 performs sensing of the external environment. Specifically, each sensor of the external sensor 22 performs sensing of the external environment and supplies the sensor data obtained by the sensing to the control unit 21. The external state detection unit 171 of the control unit 21 executes a detection process of the external state based on the sensor data from the external sensor 22. The external state detection unit 171 supplies information indicating the result of the detection process to the vehicle control unit 152 and the display control unit 153.

[0061] In step S2, the vehicle 1 executes a detection process of the state of the vehicle 1. Specifically, each sensor of the vehicle sensor 25 detects the state of the vehicle 1 and supplies the sensor data indicating the detection result to the control unit 21. The vehicle control unit 152 supplies information indicating the state of each part of the vehicle 1 to the control unit 21. The vehicle state detection unit 172 of the control unit 21 executes a detection process of the state of the vehicle 1 based on the sensor data from the vehicle sensor 25 and the information from the vehicle control unit 152. The vehicle state detection unit 172 supplies information indicating the result of the detection process to the vehicle control unit 152 and the display control unit 153.

[0062] In step S3, the vehicle 1 executes a detection process of the state of the passenger. Specifically, the in-vehicle sensor 24 performs sensing inside the vehicle and supplies the sensor data obtained by the sensing to the control unit 21. The passenger state detection unit 173 of the control unit 21 executes a detection process of the state of the passenger based on the sensor data from the in-vehicle sensor 24. The passenger state detection unit 173 supplies information indicating the result of the detection process to the vehicle control unit 152 and the display control unit 153.

[0063] In step S4, the vehicle 1 executes self-position estimation processing. Specifically, the GNSS receiver 23 receives GNSS signals from GNSS satellites and supplies them to the control unit 21. The self-position estimation unit 162 estimates the self-position of the vehicle 1 based on the GNSS signals. The self-position estimation unit 162 supplies information indicating the estimation result to the vehicle control unit 152 and the display control unit 153.

[0064] In step S5, the vehicle 1 collects external information. Specifically, the external information collection unit 163 communicates with an external server or the like via the communication unit 27 and collects external information regarding the outside of the vehicle 1. The external information collection unit 163 supplies the collected external information to the vehicle control unit 152 and the display control unit 153.

[0065] Note that the processes of steps S1 to S5 do not necessarily have to be executed in this order. For example, the order of the processes may be changed, or a plurality of processes may be executed in parallel. Also, in each loop of the driving support image display process, the processes of steps S1 to S5 do not necessarily have to be executed in their entirety every time, and some processes may be omitted.

[0066] In step S6, the image generation unit 181 generates a driving support image. For example, the image generation unit 181 generates an image (hereinafter referred to as a front image) showing the state in front of the vehicle 1 by CG (computer graphics) based on at least one or more of the result of the detection process of the state of the outside, the result of the detection process of the state of the vehicle 1, the result of the detection process of the state of the occupant, the estimation result of the self-position of the vehicle 1, and the external information.

[0067] Note that the front image may be an image that reproduces the state around the vehicle 1 in detail, or an image that reproduces the state around the vehicle 1 in a simplified manner. Also, the front image includes, for example, an image indicating the vehicle 1 which is the own vehicle.

[0068] Further, for example, the image generation unit 181 generates a driving support image by adding, by means of CG, support information for assisting driving to the front image based on at least one or more of the result of the detection process of the state of the external world, the result of the detection process of the state of the vehicle 1, the result of the detection process of the state of the passengers, the estimated result of the self-position of the vehicle 1, and the external information.

[0069] Furthermore, for example, the image generation unit 181 controls the display mode of the support information based on at least one or more of the result of the detection process of the state of the external world, the result of the detection process of the state of the vehicle 1, the result of the detection process of the state of the passengers, the estimated result of the self-position of the vehicle 1, and the external information. For example, the image generation unit 181 controls the display mode of the support information for the object in the driving support image based on the state of the object with respect to the vehicle 1. More specifically, for example, the image generation unit 181 controls at least one of the color and shape of the support information for the object in the driving support image based on at least one or a combination of a plurality of the distance, relative speed, moving direction, other behaviors, and size of the object with respect to the vehicle 1.

[0070] The image generation unit 181 supplies the driving support image to the output control unit 182.

[0071] In step S7, the vehicle 1 displays the driving support image. Specifically, the output control unit 182 supplies the driving support image to the display 111L. The display 111L displays the driving support image.

[0072] Note that specific examples of the driving support image will be described later with reference to FIGS. 6 to 17.

[0073] Thereafter, the process returns to step S1, and the processes after step S1 are executed.

[0074] <Specific Example of Driving Support Image> Next, with reference to FIGS. 6 to 17, specific examples of the driving support image and the support information will be described.

[0075] <Display Example of Support Information for Other Vehicles> First, with reference to FIGS. 6 to 8, an example of displaying assistance information for other vehicles will be described. FIGS. 6 to 8 schematically show examples of driving assistance images.

[0076] In the driving assistance image of FIG. 6, vehicles 201 to 205 are displayed. Vehicle 201 is an image showing the host vehicle (vehicle 1), and vehicles 202 to 205 are images showing other vehicles around the host vehicle.

[0077] Vehicles 201 to 205 are traveling in the same direction on a road with three lanes on one side. Vehicles 201 and 204 are traveling in the center lane. Vehicles 202 and 205 are traveling in the left lane. Vehicle 203 is traveling in the right lane. The inter-vehicle distances from vehicle 201 are in the order of vehicle 202, vehicle 203, vehicle 204, and vehicle 205 from the shortest.

[0078] On the road surface around vehicle 201, an outline 201A, which is assistance information indicating the position of vehicle 201 as the host vehicle, is displayed. The outline 201A is substantially elliptical and surrounds the periphery of vehicle 201. The color of the outline 201A is set to, for example, blue.

[0079] On the road surface around vehicle 202, an outline 202A, which is assistance information for calling attention to vehicle 202, is displayed. The outline 202A L-shapedly surrounds the right rear of vehicle 202, which is a part of the periphery of vehicle 202 close to the front (traveling direction) of vehicle 201, that is, a part where vehicle 201 has a high risk of collision or contact. The color of the outline 202A is set to, for example, orange.

[0080] On the road surface around vehicle 203, an outline 203A, which is assistance information for calling attention to vehicle 203, is displayed. The outline 203A L-shapedly surrounds the left rear of vehicle 203, which is a part of the periphery of vehicle 203 close to the front (traveling direction) of vehicle 201, that is, a part where vehicle 201 has a high risk of collision or contact. The color of the outline 202A is set to, for example, yellow.

[0081] On the road surface around the vehicle 204, an outline 204A, which is assistance information for attracting attention to the vehicle 204, is displayed. The outline 204A U-shapedly surrounds the rear of the vehicle 204, which is a part of the area around the vehicle 204 that is close to the front (travel direction) of the vehicle 201, that is, the part where there is a high risk of the vehicle 201 colliding or coming into contact. The color of the outline 202A is set to green, for example.

[0082] Since the vehicle 205 is separated from the vehicle 201 by a predetermined distance or more, no outline is particularly displayed around the vehicle 205.

[0083] Here, the display mode (for example, color and shape) of the outline for other vehicles (for example, vehicles 202 to 205) changes according to at least one of the relative position and movement with respect to the host vehicle (for example, vehicle 201).

[0084] For example, the display mode of the outline for other vehicles changes based on the moving direction of the other vehicle with respect to the host vehicle. More specifically, for example, when the movement of the other vehicle in the direction of cutting in (hereinafter referred to as the cutting-in direction) into the traveling direction (for example, forward) of the host vehicle is detected or estimated (predicted), the display mode of the outline for the other vehicle changes.

[0085] For example, FIGS. 6 to 8 show changes in the display mode of the outline 202A when the vehicle 202 is about to cut in from the left lane to the center lane in front of the vehicle 201.

[0086] Specifically, the vehicle 202 is about to change lanes from the left lane to the center lane in which the vehicle 201 is traveling in front of the vehicle 201. Therefore, the vehicle 202 approaches the vehicle 201, and the risk of collision or contact increases.

[0087] On the contrary, the shape of the outline 202A for the vehicle 202 changes so as to spread in the cutting-in direction (the direction in which the vehicle 202 changes lanes).

[0088] For example, in the example of FIG. 7, the right end of the vehicle 202 is approaching the demarcation line between the left lane and the center lane.

[0089] In contrast, compared with the example of FIG. 6, the range of the outline 202A has expanded. Specifically, the range that surrounds the right rear of the vehicle 202 in an L shape has expanded. That is, the range in which the outline 202A surrounds the vehicle 202 has expanded to the front and left of the vehicle 202. Also, the outline 202A has expanded in the cut-in direction, that is, the direction in which the vehicle 202 changes lanes.

[0090] For example, in the example of FIG. 8, a part of the vehicle 202 has entered the center lane.

[0091] In contrast, compared with the example of FIG. 7, the range of the outline 202A has further expanded. Specifically, the range that surrounds the right rear of the vehicle 202 in an L shape has further expanded. That is, the range in which the outline 202A surrounds the vehicle 202 has further expanded to the front and left of the vehicle 202. Also, the outline 202A has further expanded in the cut-in direction, that is, the direction in which the vehicle 202 changes lanes.

[0092] Also, as the risk of the vehicle 202 approaching the vehicle 201 and colliding with or contacting the vehicle 201 increases, for example, the color of the outline 202A changes to a color that reminds of more danger (for example, red).

[0093] Here, with reference to FIG. 9, an example of the relationship between the inter-vehicle distance between the host vehicle (for example, the vehicle 201) and other vehicles (for example, the vehicles 202 to 205) and the color of the outline will be described. Note that in FIG. 9, the difference in the color of the outline is represented by patterns.

[0094] For example, when the inter-vehicle distance ≥ L3, since the risk of the host vehicle colliding with or contacting another vehicle is extremely low, the color of the outline is set to green. Note that the shade of the color of the outline (green) changes according to the inter-vehicle distance. For example, the color of the outline becomes lighter as the inter-vehicle distance increases, and the outline disappears when the distance reaches a certain level or more.

[0095] For example, when L2 ≤ inter-vehicle distance < L3, since the risk of the host vehicle colliding with or contacting another vehicle is low, the color of the outline is set to yellow.

[0096] For example, when L1 ≤ inter-vehicle distance < L2, since the risk of the host vehicle colliding with or contacting another vehicle is moderate, the color of the outline is set to orange.

[0097] For example, when the inter-vehicle distance < L1, since the risk of the host vehicle colliding with or contacting another vehicle is high, the color of the outline is set to red.

[0098] In this way, the color of the outline for another vehicle changes based on the degree of risk corresponding to the inter-vehicle distance from the host vehicle.

[0099] Note that, for example, when the function of displaying the outline for another vehicle is available but turned off, the color of the outline is set to white regardless of the inter-vehicle distance.

[0100] For example, when the function of displaying the outline for another vehicle is unavailable, the outline is not displayed regardless of the inter-vehicle distance.

[0101] As described above, based on the degree of risk of the host vehicle colliding with or contacting another vehicle, at least one of the shape and color of the outline for another vehicle changes, and the degree of risk for another vehicle is presented clearly.

[0102] For example, other vehicles become visible as if they are floating up from the surroundings due to the outline, and the driver can surely recognize the presence of other vehicles. Further, based on the change in the shape of the outline, the driver can predict the movement of other vehicles and avoid collision or contact with other vehicles. Furthermore, by changing the color of the outline based on the degree of danger, the driver is appropriately alerted to other vehicles, and collision or contact with other vehicles is avoided.

[0103] <Example of Display of Support Information for Pedestrians> Next, with reference to FIG. 10, an example of the display of support information for pedestrians will be described. FIG. 10 schematically shows an example of a driving support image.

[0104] In the driving support image of FIG. 10, for example, support information is displayed for pedestrians existing within a range of a predetermined distance in the traveling direction (for example, the front) of the vehicle 201.

[0105] Specifically, for example, outlines 221A and 221B are displayed for the pedestrian 221 who is crossing the crosswalk in front of the vehicle 201 from right to left.

[0106] The outline 221A constitutes the outline of the upper body of the pedestrian 221.

[0107] The outline 221B is displayed on the road surface around the pedestrian 221. The outline 221B surrounds the front and left sides of the pedestrian 221, which are the parts close to the front surface (traveling direction) of the vehicle 201 among the surroundings of the pedestrian 221, that is, the parts where the risk of the vehicle 201 colliding or coming into contact is high. Also, since the pedestrian 221 is walking in the direction of cutting into the front (traveling direction) of the vehicle 201, the outline 221B spreads in the cutting-in direction (front of the pedestrian 221).

[0108] Also, outlines 222A and 222B are displayed for the pedestrian 222 who is walking on the sidewalk adjacent to the driving lane of the vehicle 201.

[0109] Outline 222A constitutes the contour of the upper body of the pedestrian 222.

[0110] Outline 222B is displayed on the road surface around the pedestrian 222. Outline 222B arc-surrounds the left side of the pedestrian 222, which is the part close to the front (traveling direction) of the vehicle 201 among the surroundings of the pedestrian 222, that is, the part where there is a high risk of the vehicle 201 colliding or coming into contact.

[0111] The colors of Outline 221A to Outline 222B change based on the degree of risk of the vehicle 201 colliding or coming into contact with the pedestrian.

[0112] For example, in this example, the degree of risk of the vehicle 201 colliding or coming into contact with the pedestrian 221 is higher than the degree of risk of colliding or coming into contact with the pedestrian 222.

[0113] In contrast, for example, the colors of Outline 221A and Outline 221B for the pedestrian 221 are set to orange. For example, the colors of Outline 222A and Outline 222B for the pedestrian 222 are set to yellow.

[0114] In this way, based on the degree of risk of the host vehicle colliding or coming into contact with the pedestrian, one or both of the shape and color of the outline for the pedestrian change, and the degree of risk for the pedestrian is presented clearly. Thereby, the driver is appropriately alerted to the pedestrian, and a collision or contact with the pedestrian is avoided.

[0115] <Display Example of Support Information for Traffic Lights> Next, with reference to FIGS. 11 to 13, a display example of support information for traffic lights will be described. FIGS. 11 to 13 schematically show examples of driving support images.

[0116] FIG. 11 shows a display example of support information when the traffic light 241 installed at the intersection in front of the vehicle 201 is a green light.

[0117] In this case, an outline 241A surrounding the traffic signal 241 is displayed. For example, the color of the outline 241A is set to the same blue as the color of the traffic signal 241.

[0118] Thereby, the driver can surely recognize the presence of the traffic signal 241 and that the traffic signal 241 is a green signal.

[0119] FIG. 12 shows an example of the display of assistance information when the traffic signal 241 is a yellow signal.

[0120] In this case, similar to the example of FIG. 11, an outline 241A surrounding the traffic signal 241 is displayed. The color of the outline 241A is set to, for example, the same yellow as the color of the traffic signal 241.

[0121] Also, a deceleration zone 242, which is a virtual object for suppressing the speed of the vehicle 201, is displayed. The deceleration zone 242 is displayed on the road surface so as to longitudinally cross the intersection in front of the vehicle 201 within the travel lane of the vehicle 201. The deceleration zone 242 includes a plurality of lines extending in the left-right direction, and each line is arranged at a predetermined interval in the front-rear direction. The color of each line is set to, for example, the same yellow as the color of the traffic signal 241.

[0122] Thereby, the driver can surely recognize the presence of the traffic signal 241 and that the traffic signal 241 is a yellow signal. Also, the deceleration zone 242 prompts the driver to suppress (decelerate) the approaching speed to the intersection.

[0123] FIG. 13 shows an example of the display of assistance information when the traffic signal 241 is a red signal.

[0124] In this case, similar to the examples of FIGS. 11 and 12, an outline 241A surrounding the traffic signal 241 is displayed. The color of the outline 241A is set to, for example, the same red as the color of the traffic signal 241.

[0125] In addition, a wall 243, which is a virtual object for suppressing the progress of the vehicle 201, i.e., the entry of the vehicle 201 into the intersection, is displayed. The wall 243 is grounded on the road surface and is displayed so as to extend vertically from the position of the line at the very front of the speed bump 242 in FIG. 12 (the side closest to the vehicle 201). Further, the wall 243 displays the waiting time of the red signal of the traffic signal 241. The color of the wall 243 is set to the same red as the color of the traffic signal 241, for example.

[0126] As a result, the driver can surely recognize the presence of the traffic signal 241 and that the traffic signal 241 is a red signal. Also, the wall 243 prompts the driver to stop in front of the intersection and not enter the intersection. Furthermore, the driver can know the waiting time of the red signal.

[0127] <Display Example of Support Information Regarding Close Call Points> Next, with reference to FIG. 14, a display example of support information regarding close call points such as intersections without traffic signals will be described. A close call point is, for example, a point where a dangerous event such as an accident has occurred in the past or a point where an accident is likely to occur, and is a point where the driver needs to pay special attention. FIG. 14 schematically shows an example of a driving support image.

[0128] For example, the image generation unit 181 detects a close call point based on map information and the like, and generates a driving support image including support information for the detected close call point.

[0129] FIG. 14 shows a display example of support information when the vehicle 201 reaches in front of an intersection without a traffic signal, which is a close call point.

[0130] For example, a speed bump 261 similar to the speed bump 242 in FIG. 12 is displayed on the road surface so as to longitudinally cross the intersection in front of the vehicle 201 in the traveling lane of the vehicle 201 in the front-rear direction.

[0131] Also, an icon 262, which is a virtual object representing a person riding a bicycle, is displayed so as to be in contact with the road surface of the road extending rightward from the intersection. For example, the icon 262 is displayed to alert the driver to the risk of jaywalking regardless of the presence or absence of a bicycle.

[0132] This enables the driver to surely recognize the near-miss points and pay attention to jaywalking and the like.

[0133] <Example of Display of Support Information for Vehicle Travel Route> Next, with reference to FIGS. 15 to 17, an example of the display of support information for the travel route of the vehicle 201 will be described. FIGS. 15 to 17 schematically show examples of driving support images.

[0134] Note that the travel route of the vehicle 201 is set based on, for example, map information included in external information.

[0135] FIGS. 15 to 17 show an example in which the travel route of the vehicle 201 turns right at the intersection ahead.

[0136] In the driving support image of FIG. 15, support information 281 indicating the travel route of the vehicle 201 is displayed at the intersection ahead of the vehicle 201.

[0137] The support information 281 includes a plurality of virtual objects representing arrows indicating the travel route of the vehicle 201. Each virtual object is grounded so as to stand on the road surface within the intersection ahead of the vehicle 201 in the driving lane of the vehicle 201 and is arranged in the right-turn direction. A shadow is displayed at the lower end of each virtual object, emphasizing that each virtual object is grounded.

[0138] In this way, by grounding each virtual object of the support information 281, the positional relationship between the support information 281 and the intersection becomes clear, and the position where the vehicle 201 turns right becomes easy to understand. Thereby, for example, the driver can surely turn right without making a mistake in the travel route at the intersection ahead.

[0139] In the driving support image of FIG. 16, at the intersection in front of the vehicle 201, support information 301 indicating the driving route of the vehicle 201 is displayed.

[0140] The support information 301 includes a plurality of virtual objects modeled after a plurality of arrow plates indicating the direction of travel installed at a construction site or the like. Each virtual object is grounded so as to stand on the road surface within the intersection in front of the vehicle 201 in the driving lane of the vehicle 201, and is arranged so as to overlap along the right turn direction.

[0141] In this way, by grounding each virtual object of the support information 301, the positional relationship between the support information 301 and the intersection becomes clear, and the position where the vehicle 201 makes a right turn becomes easy to understand. Thereby, for example, the driver can surely make a right turn without making a mistake in the driving route at the intersection ahead.

[0142] In the driving support image of FIG. 17, at the intersection in front of the vehicle 201, a guide 321 which is a virtual object indicating the driving route of the vehicle 201 is displayed.

[0143] The guide 321 is grounded so as to stand on the road surface within the intersection in front of the vehicle 201 in the driving lane of the vehicle 201. Further, the guide 321 indicates the driving route of the vehicle 201 by means of a gesture.

[0144] In this way, by grounding the guide 321, the positional relationship between the guide 321 and the intersection becomes clear, and the position where the vehicle 201 makes a right turn becomes easy to understand. Thereby, for example, the driver can surely make a right turn without making a mistake in the driving route at the intersection ahead.

[0145] As described above, the content of the driving support is presented clearly by the support information in the driving support image. Thereby, the driver can accurately recognize the content of the driving support and act appropriately according to the driving support.

[0146] <<2. Modification Example>> Next, a modification example of the above-described embodiment of the present technology will be described.

[0147] <Modification Example Regarding Driving Support Image>

[0148] For example, the driving support image may include an image showing the surrounding state other than in front of the vehicle 1. For example, when the vehicle 1 is moving backward, the driving support image may include an image showing the state behind the vehicle 1.

[0149] For example, the object to be outlined is not limited to the other vehicles and pedestrians described above. For example, an outline may be displayed for other moving objects such as bicycles, or an outline may be displayed for stationary objects such as obstacles.

[0150] For example, in addition to the relative distance from the host vehicle, or instead of the relative distance, the risk of the host vehicle colliding with or contacting an object is calculated based on other elements, and the display mode (e.g., color and shape) of the support information may be changed based on the risk. Such elements include, for example, relative speed and the like. Further, for example, the risk may be calculated based on the content and operation status of the function of the vehicle 1 (e.g., ADAS (Advanced Driver Assistance System)).

[0151] For example, when the vehicle 1 is traveling in a place other than a road, the support information may be grounded or displayed on the ground other than the road surface.

[0152] In the above description, an example in which the driving support image is generated by CG has been shown. However, for example, the driving support image may be generated by superimposing support information on an image taken of the surroundings of the vehicle 1.

[0153] <Other Modification Examples> The position of the display for displaying the driving support image is not necessarily limited to the above-described example. For example, the driving support image may be displayed on display 111R, display 112L, display 112R, or display 113. For example, the driving support image may be displayed so as to be continuous between display 111L and display 112R.

[0154] The configuration example of the display of the display unit 75 can be changed as appropriate. For example, display 111L and display 111R may be connected to form one display. For example, display 111R may be divided into two at the front between the driver's seat and the passenger seat and in front of the passenger seat.

[0155] This technology can be applied not only to vehicles but also to moving bodies traveling on roads.

[0156] <<3. Others>> <Configuration Example of Computer> The above-described series of processes can be executed by hardware or by software. When the series of processes are executed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer incorporated in dedicated hardware and, for example, a general-purpose personal computer that can execute various functions by installing various programs.

[0157] FIG. 18 is a block diagram showing a configuration example of the hardware of a computer that executes the above-described series of processes by a program.

[0158] In computer 1000, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0159] The bus 1004 is further connected to an input / output interface 1005. The input / output interface 1005 is connected to an input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010.

[0160] The input unit 1006 includes an input switch, buttons, a microphone, an imaging device, etc. The output unit 1007 includes a display, a speaker, etc. The storage unit 1008 includes a hard disk, a non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0161] In the computer 1000 configured as described above, the CPU 1001 loads and executes, for example, a program recorded in the storage unit 1008 via the input / output interface 1005 and the bus 1004 into the RAM 1003, whereby the above-described series of processes are performed.

[0162] The program executed by the computer 1000 (CPU 1001) can be recorded and provided, for example, on a removable medium 1011 as a package medium or the like. Also, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0163] In the computer 1000, the program can be installed in the storage unit 1008 via the input / output interface 1005 by mounting the removable medium 1011 on the drive 1010. Also, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Additionally, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0164] Note that the program executed by the computer may be a program that is processed in time series according to the order described in this specification, or a program that is processed in parallel or at a necessary timing such as when a call is made.

[0165] In addition, in this specification, a system means a collection of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and one device in which a plurality of modules are housed in one housing are both systems.

[0166] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various changes can be made without departing from the gist of the present technology.

[0167] For example, the present technology can take a configuration of cloud computing in which one function is shared and jointly processed by a plurality of devices via a network.

[0168] In addition, each step described in the above flowchart can be executed by one device or shared and executed by a plurality of devices.

[0169] Furthermore, when a plurality of processes are included in one step, the plurality of processes included in that one step can be executed by one device or shared and executed by a plurality of devices.

[0170] <Example of configuration combination> The present technology can also take the following configuration.

[0171] (1) A display control unit that controls the display of a driving support image in which support information for assisting driving is added to an image showing the state around the vehicle, and controls the color and shape of the support information based on the state of the object with respect to the vehicle An information processing apparatus to be provided. (2) The support information includes an outline surrounding at least a part of the periphery of the object. The information processing apparatus according to (1) above. (3) The outline surrounds at least a part of the periphery of the object on the ground around the object. The information processing apparatus according to (2) above. (4) The display control unit controls the shape of the outline based on the moving direction of the object with respect to the vehicle. The information processing apparatus according to (3) above. (5) When the object moves in the intrusion direction, which is the direction in which the object intrudes into the traveling direction of the vehicle, the display control unit expands the outline in the intrusion direction. The information processing apparatus according to (4) above. (6) The outline surrounds the part of the object close to the vehicle. The information processing apparatus according to any one of (3) to (5) above. (7) The display control unit controls at least one of the color and shape of the support information based on the risk of the object colliding with or contacting the vehicle. The information processing apparatus according to any one of (1) to (6) above. (8) The risk is based on the distance between the vehicle and the object. The information processing apparatus according to (7) above. (9) The display control unit generates the driving support image by computer graphics, The driving support image includes an image of the vehicle. The information processing apparatus according to any one of (1) to (8) above. (10) The information processing apparatus controls the display of a driving support image in which support information for an object around the vehicle is added to an image showing the state around the vehicle. Controlling the color and shape of the support information based on the state of the object with respect to the vehicle Information processing method. (11) A display control unit that controls the display of a driving support image in which support information for assisting driving is added to an image showing the state around the vehicle and is grounded or displayed on the ground An information processing device comprising the same. (12) The support information includes a virtual object that is grounded or displayed on the ground and suppresses the speed or progress of the vehicle. The information processing device according to (11) above. (13) The virtual object is used to suppress entry into or the entry speed at an intersection. The information processing device according to (12) above. (14) The display control unit matches the color of the virtual object with the color of the traffic signal installed at the intersection. The information processing device according to (13) above. (15) The support information includes an outline that surrounds at least a part of the periphery of the traffic signal, and the display control unit matches the color of the outline with the color of the traffic signal. The information processing device according to (14) above. (16) The support information includes a virtual object that is grounded on the ground in the traveling direction of the vehicle and indicates the route of the vehicle. The information processing device according to any one of (11) to (15) above. (17) The support information includes a shadow of the virtual object displayed on the ground. The information processing device according to (16) above. (18) The virtual object is grounded on the ground within the running lane of the vehicle within the intersection in the traveling direction. The information processing device according to (16) or (17) above. (19) The display control unit generates the driving support image by computer graphics, The driving support image includes an image of the vehicle The information processing apparatus according to any one of (11) to (18). (20) The information processing apparatus controls the display of a driving support image in which support information for assisting driving is added to an image showing the state around the vehicle and is grounded or displayed on the ground in the image. Information processing method.

[0172] Note that the effects described in this specification are merely examples and are not limiting, and there may be other effects.

Description of Reference Numerals

[0173] 1 Vehicle, 11 Vehicle control system, 21 Control unit, 22 External sensor, 23 GNSS receiver, 24 In-vehicle sensor, 25 Vehicle sensor, 28 Display unit, 41 Camera, 75 Display unit, 111L to 113 Displays, 151 Information acquisition unit, 153 Display control unit, 161 Detection unit, 162 Self-position estimation unit, 163 External information collection unit, 171 External state detection unit, 172 Vehicle state detection unit, 173 Occupant state detection unit, 181 Image generation unit, 182 Output control unit

Claims

1. A display control unit that controls the display of a driving support image in which support information for assisting driving is added to an object in an image showing the state around a vehicle, and controls the color and shape of the support information based on the state of the object with respect to the vehicle. An information processing apparatus comprising the same.

2. The support information includes an outline surrounding at least a part of the periphery of the object. The information processing apparatus according to claim 1.

3. The outline surrounds at least a part of the periphery of the object on the ground around the object. The information processing apparatus according to claim 2.

4. The display control unit controls the shape of the outline based on the moving direction of the object with respect to the vehicle. The information processing apparatus according to claim 3.

5. When the object moves in the intrusion direction, which is the direction in which the object intrudes into the traveling direction of the vehicle, the display control unit expands the outline in the intrusion direction. The information processing apparatus according to claim 4.

6. The outline surrounds the part of the object close to the vehicle. The information processing apparatus according to claim 3.

7. The display control unit controls at least one of the color and shape of the support information based on the degree of risk that the object collides with or contacts the vehicle. The information processing apparatus according to claim 1.

8. The degree of risk is based on the distance between the vehicle and the object. The information processing apparatus according to claim 7.

9. The display control unit generates the driving support image by computer graphics, The driving support image includes an image of the vehicle. The information processing apparatus according to claim 1.

10. An information processing apparatus, controls the display of a driving support image in which support information for an object around the vehicle is added to an image showing the state around the vehicle, controls the color and shape of the support information based on the state of the object with respect to the vehicle. An information processing method.

11. A display control unit that controls the display of a driving support image in which support information for assisting driving is added to an image showing the state around a vehicle and is grounded or displayed on the ground. An information processing apparatus comprising the same.

12. The support information includes a virtual object that is grounded or displayed on the ground and is used to suppress the speed or progress of the vehicle. The information processing apparatus according to claim 11.

13. The virtual object is used to suppress entry into an intersection or the entry speed. The information processing apparatus according to claim 12.

14. The display control unit matches the color of the virtual object to the color of the traffic signal installed at the intersection. The information processing apparatus according to claim 13.

15. The support information includes an outline surrounding at least a part of the periphery of the traffic signal. The display control unit matches the color of the outline to the color of the traffic signal. The information processing apparatus according to claim 14.

16. The support information includes a virtual object that is grounded on the ground in the traveling direction of the vehicle and indicates the route of the vehicle. The information processing apparatus according to claim 11.

17. The support information includes the shadow of the virtual object displayed on the ground. The information processing apparatus according to claim 16.

18. The virtual object is grounded on the ground within the travel lane of the vehicle within the intersection in the traveling direction. The information processing apparatus according to claim 16.

19. The display control unit generates the driving support image by computer graphics, The driving support image includes an image of the vehicle. The information processing apparatus according to claim 11.

20. An information processing apparatus, controls the display of a driving support image in which support information for assisting driving is added to an image showing the state around the vehicle and is grounded or displayed on the ground in the image. An information processing method.

Images