Information processing device, information processing method, and program
The information processing device predicts obstacle movements and calculates danger levels to effectively present risk information to vehicle users, addressing the shortcomings of conventional systems by providing intuitive and enhanced safety through AR-enhanced danger prediction.
Patent Information
- Application Number
- PCT/JP2024/015585
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-10
- Filing Date
- 2024-04-19
- Publication Date
- 2024-11-14
AI Technical Summary
Conventional vehicle safety systems fail to adequately predict the movement of obstacles and calculate the degree of danger based on predicted movements, leading to insufficient information for users to avoid potential hazards.
An information processing device with a prediction unit that forecasts the future movement of objects and accident information, a calculation unit that determines the danger level, and a presentation control unit that displays danger prediction information to users, enhancing the presentation of risk information using AR technology.
Provides users with intuitive and effective danger avoidance information by visually presenting the predicted movement and danger level of objects, thereby improving vehicle safety through enhanced risk awareness.
Smart Images

Figure JP2024015585_14112024_PF_FP_ABST
Abstract
Description
Information processing device, information processing method, and program
[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that can more appropriately present information for avoiding danger to users of moving objects such as vehicles.
[0002] In recent years, the introduction of Augmented Reality (AR) technology into vehicles has been progressing (see, for example, Patent Document 1). For example, there is a technology in which, when an obstacle is detected by a sensor, a template representing the object is superimposed on an image of the surroundings of the vehicle.
[0003] International Publication No. 2021 / 079975
[0004] In conventional technology, the degree of danger based on the predicted results of vehicle movement and the position of an object is expressed using the size of a template, etc. However, since the movement of obstacles is not predicted, the degree of danger is not calculated based on the predicted results of object movement, and the predicted results of object movement are not presented to the vehicle user, it is possible that sufficient information to avoid danger is not displayed.
[0005] The present technology has been made in consideration of such circumstances, and makes it possible to more appropriately present information for avoiding danger to users of moving bodies such as vehicles.
[0006] An information processing device according to one aspect of the present technology includes a prediction unit that predicts movement information indicating future movements of objects around a moving body and accident information indicating the movement of the object in the event of an accident, a calculation unit that calculates the degree of danger of the object based on the movement information and the accident information, and a presentation control unit that presents danger prediction information indicating the degree of danger of the object to a user of the moving body.
[0007] In an information processing method according to one aspect of the present technology, an information processing device predicts movement information indicating the future movement of objects around a moving body and accident information indicating the movement of the object if an accident occurs, calculates the degree of danger of the object based on the movement information and the accident information, and presents danger prediction information indicating the degree of danger of the object to a user of the moving body.
[0008] A program according to one aspect of the present technology causes a computer to execute a process of predicting movement information indicating the future movement of objects around a moving body and accident information indicating the movement of the objects in the event of an accident, calculating the degree of danger of the objects based on the movement information and the accident information, and presenting danger prediction information indicating the degree of danger of the objects to a user of the moving body.
[0009] In one aspect of the present technology, movement information indicating the future movement of objects around a moving body and accident information indicating the movement of the object in the event of an accident are predicted, the degree of danger of the object is calculated based on the movement information and the accident information, and danger prediction information indicating the degree of danger of the object is presented to the user of the moving body.
[0010] 1 is a block diagram showing an example of the configuration of a vehicle control system. FIG. 2 is a diagram showing an example of a sensing area. FIG. 3 is a block diagram showing an example of the configuration of a vehicle control system to which the present technology is applied. FIG. 4 is a flowchart illustrating processing performed by the vehicle control system. FIG. 5 is a diagram showing an example of risk prediction information. FIG. 6 is a diagram showing an example of risk prediction information indicating the risk level of a moving vehicle. FIG. 7 is a diagram showing an example of risk prediction information indicating the risk level of a pedestrian crossing a crosswalk. FIG. 8 is a diagram showing an example of risk prediction information indicating the risk level of a bicycle. FIG. 9 is a diagram showing an example of risk prediction information indicating the risk level of a stopped vehicle. FIG. 10 is a diagram showing a display example of risk prediction information. FIG. 11 is a diagram showing a first modified example of risk prediction information. FIG. 12 is a diagram showing a second modified example of risk prediction information. FIG. 13 is a diagram showing a display example of risk prediction information for ensuring the visibility of risk prediction information. FIG. 14 is a diagram showing a display example of risk prediction information for ensuring the visibility of an object. FIG. 15 is a diagram showing an example of hazardous environment information. FIG. 16 is a diagram showing another example of hazardous environment information. FIG. 17 is a block diagram showing an example of the configuration of computer hardware.
[0011] Hereinafter, embodiments of the present technology will be described in the following order: 1. Configuration example of a vehicle control system 2. Embodiment 3. Modification
[0012] 1. Configuration Example of Vehicle Control System FIG. 1 is a block diagram showing a configuration example of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.
[0013] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to automated driving of the vehicle 1. This automated driving includes automated driving of levels 1 to 5, and remote driving and remote assistance of the vehicle 1 by a remote driver.
[0014] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, a driving automation control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.
[0015] The vehicle control ECU 21, communication unit 22, map information storage unit 23, position information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, memory unit 28, driving automation control unit 29, DMS 30, HMI 31, and vehicle control unit 32 are connected to each other so as to be able to communicate with each other via a communication network 41. The communication network 41 is configured, for example, by an in-vehicle communication network or bus conforming to a digital two-way communication standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). Different communication networks 41 may be used depending on the type of data being transmitted. For example, CAN may be used for data related to vehicle control, and Ethernet may be used for large-volume data. In addition, each part of the vehicle control system 11 may be directly connected without going through the communication network 41, using wireless communication intended for communication over relatively short distances, such as near field communication (NFC) or Bluetooth (registered trademark).
[0016] In the following description, when each unit of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described that the vehicle control ECU 21 and the communication unit 22 communicate with each other.
[0017] The vehicle control ECU 21 is configured with various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The vehicle control ECU 21 controls the entire or part of the functions of the vehicle control system 11.
[0018] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 22 can perform communication using a plurality of communication methods.
[0019] The following provides an overview of communication with the outside of the vehicle that can be performed by the communication unit 22. The communication unit 22 communicates with a server (hereinafter referred to as an external server) or the like on an external network via a base station or an access point using a wireless communication method such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the Internet, a cloud network, or a network specific to a carrier. The communication method used by the communication unit 22 with the external network is not particularly limited as long as it is a wireless communication method that enables digital two-way communication at a communication speed equal to or higher than a predetermined distance.
[0020] Furthermore, for example, the communication unit 22 can communicate with a terminal located near the vehicle using P2P (Peer to Peer) technology. The terminal located near the vehicle can be, for example, a terminal worn by a mobile object moving at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed location in a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 22 can also perform V2X communication. V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian.
[0021] The communication unit 22 can receive, for example, a program for updating software that controls the operation of the vehicle control system 11 from the outside (over the air). The communication unit 22 can also receive map information, traffic information, information about the surroundings of the vehicle 1, and the like from the outside. For example, the communication unit 22 can also transmit information about the vehicle 1 and information about the surroundings of the vehicle 1 to the outside. Information about the vehicle 1 that the communication unit 22 transmits to the outside includes, for example, data indicating the state of the vehicle 1 and the recognition result by the recognition unit 73. Furthermore, for example, the communication unit 22 performs communication corresponding to a vehicle emergency notification system such as e-call.
[0022] For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System) (registered trademark)) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.
[0023] The following provides an overview of communication with the vehicle interior that can be performed by the communication unit 22. The communication unit 22 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 22 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wireless communication, such as wireless LAN, Bluetooth, NFC, or Wireless USB (WUSB). The communication unit 22 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 22 can communicate with each device in the vehicle using wired communication via a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wired communication, such as Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI) (registered trademark), or Mobile High-Definition Link (MHL).
[0024] Here, the in-vehicle device that communicates with the communication unit 22 refers to, for example, a device in the vehicle that is not connected to the communication network 41. Possible in-vehicle devices include, for example, a mobile device or wearable device carried by a user in the vehicle, such as a driver, and an information device that is brought into the vehicle and temporarily installed.
[0025] The map information storage unit 23 stores one or both of a map acquired from an external source and a map created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, and the like.
[0026] Examples of high-precision maps include dynamic maps, point cloud maps, and vector maps. A dynamic map is a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and is provided to the vehicle 1 from an external server or the like. A point cloud map is a map made up of a point cloud (point cloud data). A vector map is a map adapted to automated driving by associating traffic information such as the positions of lanes and traffic lights with the point cloud map.
[0027] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created in the vehicle 1 based on sensing results from the camera 51, radar 52, LiDAR 53, etc. as a map for matching with a local map described later, and stored in the map information storage unit 23. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of several hundred square meters relating to the planned route along which the vehicle 1 will travel is acquired from the external server or the like in order to reduce communication capacity.
[0028] The location information acquisition unit 24 receives GNSS (Global Navigation Satellite System) signals from GNSS satellites and acquires location information of the vehicle 1. The acquired location information is supplied to the driving automation control unit 29. Note that the method used by the location information acquisition unit 24 is not limited to using GNSS signals, and it may also acquire location information using a beacon, for example.
[0029] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.
[0030] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. Without being limited to this, the external recognition sensor 25 may be configured to include one or more types of sensors selected from the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 is not particularly limited as long as the number is a number that can be realistically installed on the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of sensing areas of each sensor included in the external recognition sensor 25 will be described later.
[0031] The imaging method of the camera 51 is not particularly limited. For example, cameras of various imaging methods capable of distance measurement, such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera, can be applied to the camera 51 as needed. However, the camera 51 may simply acquire an image without distance measurement.
[0032] Furthermore, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment for the vehicle 1. The environmental sensor is a sensor for detecting the environment such as weather, climate, brightness, etc., and may include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.
[0033] Furthermore, for example, the external recognition sensor 25 includes a microphone used to detect sounds around the vehicle 1 and the location of sound sources.
[0034] The interior sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the interior sensor 26 are not particularly limited as long as they are of types and numbers that can be realistically installed in the vehicle 1.
[0035] For example, the interior sensor 26 may include one or more types of sensors selected from the group consisting of a camera, radar, a seating sensor, a steering wheel sensor, a microphone, and a biometric sensor. The camera included in the interior sensor 26 may be a camera using any of various imaging methods capable of measuring distances, such as a Time of Flight (ToF) camera, a stereo camera, a monocular camera, or an infrared camera. The camera included in the interior sensor 26 may also be a camera simply for acquiring captured images, regardless of distance measurement. The biometric sensor included in the interior sensor 26 may be provided, for example, on a seat, a steering wheel, or the like, and detect various types of biometric information of the user.
[0036] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the vehicle sensor 27 are not particularly limited as long as they are of types and numbers that can be realistically installed on the vehicle 1.
[0037] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these sensors. For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining battery charge and temperature, and an impact sensor that detects external impacts.
[0038] The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The storage unit 28 is used, for example, as an electrically erasable programmable read-only memory (EEPROM) and a random access memory (RAM). Examples of storage media that can be used include a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, and a magneto-optical storage device. The storage unit 28 stores various programs and data used by each component of the vehicle control system 11. For example, the storage unit 28 includes an event data recorder (EDR) and a data storage system for automated driving (DSSAD), and stores information about the vehicle 1 before and after an event such as an accident, and information acquired by the in-vehicle sensors 26.
[0039] The driving automation control unit 29 controls the driving automation function of the vehicle 1. For example, the driving automation control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.
[0040] The analysis unit 61 performs an analysis process of the vehicle 1 and the surrounding situation. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.
[0041] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25 and matches the local map with the high-precision map to estimate the self-position of the vehicle 1. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.
[0042] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technique such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size and the occupancy status of objects is indicated on a grid-by-grid basis. The occupancy status of objects is indicated, for example, by the presence or absence of an object and its probability of existence. The local map is also used, for example, in the detection process and recognition process of the situation outside the vehicle 1 by the recognition unit 73.
[0043] The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and the sensor data from the vehicle sensor 27 .
[0044] The sensor fusion unit 72 performs sensor fusion processing to obtain information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include compounding, integration, fusion, and association.
[0045] The recognition unit 73 executes a detection process for detecting the situation outside the vehicle 1 and a recognition process for recognizing the situation outside the vehicle 1 .
[0046] For example, the recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.
[0047] Specifically, for example, the recognition unit 73 performs detection processing and recognition processing of objects around the vehicle 1. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, and movement (for example, the type of action, direction of movement, and speed of movement) of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object and to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated and may overlap.
[0048] For example, the recognition unit 73 detects objects around the vehicle 1 by performing clustering to classify a point cloud based on sensor data from the radar 52, the LiDAR 53, or the like into clusters of points. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.
[0049] For example, the recognition unit 73 performs tracking to follow the movement of clusters of point clouds classified by clustering, thereby detecting the movement of objects around the vehicle 1. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.
[0050] For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. based on image data supplied from the camera 51. The recognition unit 73 may also recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.
[0051] For example, the recognition unit 73 can perform a recognition process of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the result of estimation of the self-position by the self-position estimation unit 71, and the result of recognition of objects around the vehicle 1 by the recognition unit 73. Through this process, the recognition unit 73 can recognize the positions and states of traffic lights, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes that can be traveled, etc.
[0052] For example, the recognition unit 73 can perform a recognition process of the environment around the vehicle 1. The surrounding environment to be recognized by the recognition unit 73 may include weather, temperature, humidity, brightness, and road surface conditions.
[0053] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route tracking processing.
[0054] The route planning includes global path planning and local path planning. Global path planning includes a process of planning a rough route from a start to a goal. Local path planning, also called trajectory planning, includes a process of generating a trajectory that allows the vehicle 1 to proceed safely and smoothly in the vicinity of the vehicle 1 on the planned route, taking into account the motion characteristics of the vehicle 1.
[0055] Path following is a process of planning an operation for safely and accurately traveling along a route planned by a route plan within a planned time. The behavior planning unit 62 can, for example, calculate a target speed and a target angular velocity of the vehicle 1 based on the results of this path following process.
[0056] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62 .
[0057] For example, the operation control unit 63 controls a steering control unit 81, a brake control unit 82, and a drive control unit 83 included in a vehicle control unit 32 described later, to perform lateral vehicle motion control and longitudinal vehicle motion control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 performs control aimed at driver assistance functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, host vehicle collision warning, host vehicle lane departure warning, and driving automation such as driving without operation by the driver or a remote driver.
[0058] The DMS 30 performs processes such as authenticating the driver and recognizing the driver's state based on sensor data from the in-vehicle sensors 26 and input data input to the HMI 31 (described later). Examples of the driver's state to be recognized include physical condition, level of alertness, level of concentration, level of fatigue, line of sight, level of intoxication, driving operation, and posture.
[0059] The DMS 30 may be configured to perform authentication processing for users other than the driver and recognition processing for the status of the users. Furthermore, for example, the DMS 30 may be configured to perform recognition processing for the status inside the vehicle based on sensor data from the in-vehicle sensor 26. Examples of the status inside the vehicle that may be recognized include temperature, humidity, brightness, and odor.
[0060] The HMI 31 inputs various data and instructions and presents various data to the user.
[0061] The following provides an overview of data input via the HMI 31. The HMI 31 includes input devices for a person to input data. The HMI 31 generates input signals based on data, instructions, and the like input via the input devices and supplies the signals to each component of the vehicle control system 11. The HMI 31 includes, as input devices, controls such as a touch panel, buttons, switches, and levers. The HMI 31 may also include input devices that allow information to be input by voice, gestures, or other means other than manual operation. Furthermore, the HMI 31 may use, as input devices, externally connected devices such as a remote control device using infrared or radio waves, or a mobile or wearable device compatible with the operation of the vehicle control system 11.
[0062] The presentation of data by the HMI 31 will be briefly described. The HMI 31 generates visual information, auditory information, and tactile information for the user or the outside of the vehicle. The HMI 31 also performs output control, controlling the output, output content, output timing, output method, etc. of each piece of generated information. The HMI 31 generates and outputs, as visual information, information indicated by images or lights, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image showing the situation around the vehicle 1. The HMI 31 also generates and outputs, as auditory information, information indicated by sounds, such as voice guidance, warning sounds, and warning messages. The HMI 31 also generates and outputs, as tactile information, information imparted to the user's sense of touch by, for example, force, vibration, movement, etc.
[0063] Examples of output devices that the HMI 31 uses to output visual information include a display device that displays an image on its own to present visual information and a projector device that projects an image to present visual information. The display device may be a head-up display, a see-through display, or a display with an AR (Augmented Reality) function, in addition to a display device with a normal display. The display device may also be a device that displays visual information within the user's field of view, such as a wearable device with an AR function. Examples of wearable devices include eyeglass displays, smartphones, and smartwatches. The HMI 31 can also use display devices included in a navigation system, an instrument panel, a camera monitoring system (CMS), an electronic mirror, lamps, and the like installed in the vehicle 1 as output devices that output visual information. Furthermore, the HMI 131 can also use retrofit display devices, such as an on-dash monitor, a tablet device, a drive recorder, or a display for the drive recorder, as output devices that output visual information.
[0064] Output devices that output visual information are provided on the front of the dashboard in front of the driver's seat and passenger seat, on the console between the driver's seat and passenger seat, on the windshield, on the steering wheel, on the back of the driver's seat and passenger seat, etc.
[0065] As an output device for the HMI 31 to output auditory information, for example, an audio speaker, a headphone, or an earphone can be applied.
[0066] The HMI 31 may output tactile information using a haptic element that uses haptic technology. The haptic element may be provided on a part that the user touches, such as a steering wheel or a seat. The tactile information may also be output by a smartphone, smartwatch, or other device carried by the user.
[0067] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.
[0068] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.
[0069] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an antilock brake system (ABS), a regenerative brake mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.
[0070] The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 83 includes, for example, a drive ECU for controlling the drive system, and an actuator for driving the drive system.
[0071] The body system control unit 84 detects and controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a body system ECU that controls the body system systems, an actuator that drives the body system systems, etc.
[0072] The light control unit 85 detects and controls the states of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 85 includes a light ECU that controls the lights, an actuator that drives the lights, and the like.
[0073] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, and the like.
[0074] Fig. 2 is a diagram showing an example of a sensing area by the camera 51, radar 52, LiDAR 53, ultrasonic sensor 54, etc. of the external recognition sensor 25 in Fig. 1. Note that Fig. 2 shows a schematic view of the vehicle 1 as seen from above, with the left end side being the front end (front) side of the vehicle 1 and the right end side being the rear end (rear) side of the vehicle 1.
[0075] Sensing area 101F and sensing area 101B show examples of sensing areas of the ultrasonic sensors 54. Sensing area 101F covers the periphery of the front end of the vehicle 1 with multiple ultrasonic sensors 54. Sensing area 101B covers the periphery of the rear end of the vehicle 1 with multiple ultrasonic sensors 54.
[0076] The sensing results in the sensing area 101F and the sensing area 101B are used, for example, for parking assistance for the vehicle 1.
[0077] Sensing area 102F to sensing area 102B show examples of sensing areas of a short-range or medium-range radar 52. Sensing area 102F covers a position farther in front of the vehicle 1 than sensing area 101F. Sensing area 102B covers a position farther behind the vehicle 1 than sensing area 101B. Sensing area 102L covers the periphery behind the left side of the vehicle 1. Sensing area 102R covers the periphery behind the right side of the vehicle 1.
[0078] The sensing results in sensing area 102F are used, for example, to detect vehicles, pedestrians, and the like that are present in front of the vehicle 1. The sensing results in sensing area 102B are used, for example, for a collision prevention function behind the vehicle 1. The sensing results in sensing area 102L and sensing area 102R are used, for example, to detect objects in blind spots on the sides of the vehicle 1.
[0079] Sensing areas 103F to 103B show examples of sensing areas sensed by camera 51. Sensing area 103F covers a position farther in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position farther in the rear of vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery of the left side of vehicle 1. Sensing area 103R covers the periphery of the right side of vehicle 1.
[0080] The sensing results in the sensing area 103F can be used for, for example, recognition of traffic lights and traffic signs, lane departure prevention assistance systems, and automatic headlight control systems. The sensing results in the sensing area 103B can be used for, for example, parking assistance and surround view systems. The sensing results in the sensing areas 103L and 103R can be used for, for example, surround view systems.
[0081] Sensing area 104 shows an example of the sensing area of LiDAR 53. Sensing area 104 covers a position farther ahead of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.
[0082] The sensing results in the sensing area 104 are used to detect objects such as surrounding vehicles, for example.
[0083] A sensing area 105 shows an example of the sensing area of the long-range radar 52. The sensing area 105 covers a position further ahead of the vehicle 1 than the sensing area 104. On the other hand, the sensing area 105 has a narrower range in the left-right direction than the sensing area 104.
[0084] The sensing results in the sensing area 105 are used for, for example, adaptive cruise control (ACC), emergency braking, collision avoidance, and the like.
[0085] The sensing areas of the cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 included in the external recognition sensor 25 may have various configurations other than those shown in FIG. 2 . Specifically, the ultrasonic sensors 54 may also sense the sides of the vehicle 1, and the LiDAR 53 may sense the rear of the vehicle 1. The installation positions of the sensors are not limited to the examples described above. The number of each sensor may be one or more.
[0086] 2. Embodiment Next, an embodiment of the present technology will be described with reference to FIGS. 3 to 10. FIG.
[0087] FIG. 3 is a block diagram showing an example of the configuration of a vehicle control system 11 to which the present technology is applied.
[0088] The vehicle control system 11 in Fig. 3 includes the above-described configuration (external recognition sensor 25 and recognition unit 73), an information processing unit 201 that calculates the degree of danger of objects around the vehicle 1 and generates danger prediction information indicating the degree of danger of the objects, and a display unit 202 that displays the danger prediction information. Fig. 3 shows the configuration of the portion of the vehicle control system 11 that is involved in displaying the danger prediction information.
[0089] As described above, the recognition unit 73 acquires sensor data from the external recognition sensor 25 and performs recognition processing of objects around the vehicle 1 based on the sensor data using AI (Artificial Intelligence) trained using deep learning or other machine learning, a neural network, etc. Through the recognition processing, the recognition unit 73 recognizes, for example, the type, attribute, state, position, orientation, movement speed, and size of objects around the vehicle 1.
[0090] The type of object indicates whether the object is a pedestrian, a vehicle, a ball, a guardrail, a road sign, an obstacle, etc. The recognition unit 73 recognizes not only moving objects such as pedestrians, vehicles, and balls, but also stationary objects such as guardrails, signs, and obstacles.
[0091] When the type of object is a pedestrian, the attribute of the object indicates whether the pedestrian is a child, an elderly person, etc. When the type of object is a vehicle, the attribute of the object indicates the model and color of the vehicle, etc. The state of the object indicates that the pedestrian as an object is walking while looking at a smartphone, that the brake lights of the vehicle as an object are on, etc.
[0092] The recognition unit 73 supplies the recognition results of the objects around the vehicle 1 to the information processing unit 201. Note that the recognition unit 73 may obtain information about the objects around the vehicle 1 from equipment other than the vehicle 1 via the communication unit 22 and perform recognition processing based on the information.
[0093] The information processing unit 201 is a part of the function of the HMI 31. The information processing unit 201 is made up of a prediction unit 211, a risk calculation unit 212, and an image generation unit 213.
[0094] The prediction unit 211 predicts the future movement of objects around the vehicle 1 and the movement of the objects in the event of an accident, based on the object recognition results by the recognition unit 73 .
[0095] Specifically, the prediction unit 211 predicts the range that the object will reach in a few seconds as the future movement of the object. The range that the object will reach in a few seconds is basically predicted based on the position, orientation, and movement speed of the object.
[0096] The range that an object will reach in a few seconds may be predicted based on a time series of the object's movement. For example, the range that an object will reach in a few seconds may be predicted based on the amount of movement of the object over a recent predetermined period. The range that an object will reach in a few seconds may also be predicted taking into account the state of the object, such as a pedestrian walking while looking at their smartphone or a vehicle's brake lights being on.
[0097] The prediction unit 211 predicts the possibility of an accident occurring and the movement of an object if an accident occurs (the range that the object will reach in a few seconds) based on the attributes of the object, such as, for example, "children are likely to suddenly run out into the road" or "elderly people are likely to fall while crossing the road." The prediction unit 211 can also predict the possibility of an accident occurring and the movement of an object if an accident occurs, taking into account the position of a stationary object, such as, for example, "children are unlikely to suddenly run out into the road because there is a guardrail."
[0098] The prediction unit 211 supplies movement information indicating the future movement of the object and accident information indicating the movement of the object if an accident occurs to the risk calculation unit 212 and the image generation unit 213. The accident information includes information indicating the possibility of an accident occurring. The accident information may also include prediction difficulty, which is the degree of difficulty in predicting the future movement of the object.
[0099] The risk calculation unit 212 calculates the risk of the object based on the object recognition result by the recognition unit 73 and the movement information and accident information supplied from the prediction unit 211. For example, the possibility of a collision between the object and the vehicle 1 is estimated as the risk based on the range where the object is predicted to arrive in a few seconds and the moving direction and moving speed of the vehicle 1.
[0100] The risk calculation unit 212 supplies the risk of the object to the image generation unit 213 .
[0101] The image generation unit 213 generates risk prediction information based on the movement information and accident information supplied from the prediction unit 211 and the risk level of the object supplied from the risk level calculation unit 212. Specifically, the image generation unit 213 controls the display mode of the risk prediction information based on at least one of the movement information, the accident information, and the risk level. The display mode of the risk prediction information includes color, shape (width, side inclination, direction), transparency, etc.
[0102] The image generating unit 213 generates an image including the risk prediction information. For example, the image generating unit 213 superimposes the risk prediction information on a captured image showing the surroundings of the vehicle 1. The image generating unit 213 functions as a presentation control unit that controls the display unit 202 to present an image to the user of the vehicle 1.
[0103] The display unit 202 is configured by, for example, a display installed inside the vehicle. The display unit 202 displays an image under the control of the image generation unit 213, and presents the danger prediction information to the user of the vehicle 1.
[0104] All or part of the functions of the recognition unit 73 and the information processing unit 201 may be provided in a device other than the vehicle 1, such as a cloud.
[0105] Next, the processing performed by the vehicle control system 11 having the above configuration will be described with reference to the flowchart in Figure 4. The processing in Figure 4 is started, for example, when an operation to start the vehicle 1 and begin driving is performed, for example, when an ignition switch, power switch, start switch, or the like of the vehicle 1 is turned on. The processing in Figure 4 is ended, for example, when an operation to end driving the vehicle 1 is performed, for example, when an ignition switch, power switch, start switch, or the like of the vehicle 1 is turned off.
[0106] In step S1 , the external recognition sensor 25 senses the situation around the vehicle 1 .
[0107] In step S2, the recognition unit 73 recognizes objects around the vehicle 1 based on the sensor data of the external recognition sensor.
[0108] In step S3, the information processing unit 201 determines whether the recognition unit 73 has recognized an object.
[0109] If it is determined in step S3 that an object has been recognized, the process proceeds to step S4. In step S4, the prediction unit 211 predicts the future movement of the object recognized by the recognition unit 73 and the possibility of an accident occurring.
[0110] In step S5 , the risk calculation unit 212 calculates the risk of the object recognized by the recognition unit 73 based on the recognition result by the recognition unit 73 and the prediction result by the prediction unit 211 .
[0111] After the degree of danger of the object is calculated in step S5, the process proceeds to step S6. If it is determined in step S3 that the object has not been recognized, steps S4 and S5 are skipped and the process proceeds to step S6.
[0112] In step S6, the image generation unit 213 generates an image to be presented to the user of the vehicle 1 by the display unit 202. If the recognition unit 73 recognizes an object, the image generation unit 213 generates an image to be presented by the display unit 202, for example, by superimposing danger prediction information on a captured image showing the periphery of the vehicle 1. If the recognition unit 73 does not recognize an object, the image generation unit 213 uses, for example, the captured image showing the periphery of the vehicle 1 as the image to be presented by the display unit 202.
[0113] In step S7, the display unit 202 presents an image to the user of the vehicle 1 in accordance with the control of the image generation unit 213.
[0114] FIG. 5 is a diagram showing an example of risk prediction information.
[0115] As shown in Fig. 5, when a pedestrian as an object is standing still or is predicted to stand still, the danger prediction information A1 has, for example, a rectangular shape with long sides connecting the pedestrian's head to feet. Also, as shown in Fig. 5, when a pedestrian is moving or is predicted to move, the danger prediction information A1 has, for example, a trapezoid shape with rounded corners and sides connecting the pedestrian's head to feet.
[0116] The danger prediction information A1 has a bulging shape at the top of the edge on the side (left side) in the direction of movement in which the pedestrian is predicted to move, and the edge on the side in the direction of movement in the danger prediction information A1 becomes bulging the wider the range in which the pedestrian is predicted to reach in a few seconds.
[0117] In other words, the width (length of the upper side) of the danger prediction information A1 indicates the magnitude of the pedestrian's future movement. The direction of the bulge of the danger prediction information A1 indicates the predicted direction of the pedestrian's movement. Furthermore, the outer angle (slope of the side facing the predicted direction of the pedestrian's movement) of the vertex below (lower left) the predicted direction of the pedestrian's movement in the danger prediction information A1 also indicates the magnitude of the pedestrian's future movement.
[0118] The color of the danger prediction information A1 indicates the level of danger to the pedestrian. If the level of danger is low, the danger prediction information A1 is displayed in, for example, cyan, and if the level of danger is medium, the danger prediction information A1 is displayed in, for example, yellow. If the level of danger is high, the danger prediction information A1 is displayed in, for example, orange. For example, the danger prediction information A1 is given a gradation such that the transparency increases as it gets closer to the pedestrian.
[0119] The width of the danger prediction information A1 or the slope of the edge in the danger prediction information A1 that faces the pedestrian's predicted movement direction may indicate information other than the magnitude of the pedestrian's future movement. The width of the danger prediction information A1 may indicate, for example, the amplitude of fluctuation in the pedestrian's future movement. For example, the position of the edge in the danger prediction information A1 that faces the pedestrian's predicted movement direction indicates the upper limit of the range that the pedestrian may reach in a few seconds, and the position of the edge opposite to the pedestrian's predicted movement direction indicates the lower limit of that range.
[0120] Furthermore, the width of the danger prediction information A1 may indicate, for example, the degree of danger to the pedestrian, and the slope of the edge of the danger prediction information A1 on the side of the predicted direction of movement of the pedestrian may indicate the predicted movement speed of the pedestrian.
[0121] FIG. 6 is a diagram showing an example of risk prediction information indicating the risk level of a traveling vehicle.
[0122] If vehicle C1 is predicted to travel leftward at high speed and block the progress of vehicle 1, the vehicle control system 11, for example, calculates a high risk level for vehicle C1 and displays risk prediction information A11 indicating the risk level for vehicle C1 in orange, superimposed on a portion of vehicle C1, as shown in Fig. 6. Because vehicle C1 is predicted to travel at high speed, the vehicle control system 11, for example, increases the width of the risk prediction information A11 and decreases the slope of the left side of the risk prediction information A11.
[0123] FIG. 7 is a diagram showing an example of risk prediction information indicating the risk level of a pedestrian crossing a crosswalk.
[0124] If pedestrian P1 is predicted to cross the crosswalk and move leftward, the vehicle control system 11 may, for example, calculate a medium level of risk for pedestrian P1 and display risk prediction information A12 indicating the risk level for pedestrian P1 in yellow, superimposing it on a portion of pedestrian P1, as shown in Fig. 7. Because pedestrian P1 is predicted to move at a low speed, the vehicle control system 11 may, for example, reduce the width of the risk prediction information A12 and increase the slope of the left side of the risk prediction information A12.
[0125] 7, a point Po1 may be displayed superimposed on the head of the pedestrian P1. Displaying the point Po1 makes it easier for users of the vehicle 1 to know that a pedestrian is present. If the presence of the pedestrian P1 around the vehicle 1 is recognized, but the direction of the pedestrian P1 cannot be recognized or the future movement of the pedestrian P1 cannot be predicted, the danger prediction information A12 may not be displayed and only the point Po1 may be displayed.
[0126] FIG. 8 is a diagram showing an example of risk prediction information indicating the degree of risk for bicycles.
[0127] If bicycle B1 is predicted to jump out to the right and cross in front of vehicle 1, vehicle control system 11, for example, calculates a high level of danger for bicycle B1 and displays danger prediction information A13 indicating the level of danger for the bicycle in orange, superimposed on a portion of the bicycle, as shown in Figure 8.
[0128] Because bicycle B1 is predicted to move at high speed relative to vehicle C1, vehicle control system 11 increases the width of risk prediction information A13 and decreases the slope of the right side of risk prediction information A13, for example. Because bicycle B1 is predicted to move toward the rear right (to the front right of vehicle 1), risk prediction information A13 is drawn according to perspective so that the user of vehicle 1 can easily see that bicycle B1 is moving toward the rear right.
[0129] As shown in FIG. 8, a point Po2 may be displayed superimposed on the head of the person riding the bicycle B1.
[0130] FIG. 9 is a diagram showing an example of risk prediction information indicating the risk level of a stopped vehicle.
[0131] If vehicle C2 is predicted to be stopped in a location that will not hinder the progress of vehicle 1, the vehicle control system 11, for example, calculates a low level of danger for vehicle C2 and displays danger prediction information A14 indicating the level of danger for vehicle C2 in cyan, as shown in Figure 9.
[0132] The risk of a stopped vehicle or a stationary obstacle is indicated, for example, by columnar risk prediction information arranged at the corners of a bounding box that surrounds the bottom surface of the vehicle or obstacle.
[0133] The above-mentioned danger prediction information may be displayed on a display installed inside vehicle 1, superimposed on a captured image showing the surroundings of vehicle 1, or may be displayed on a head-up display so as to be superimposed on a real object.
[0134] The display mode of the risk prediction information is not limited to the display modes described with reference to FIGS.
[0135] FIG. 10 is a diagram showing an example of display of danger prediction information.
[0136] 10, danger prediction information A21 to A24 are displayed superimposed on a captured image showing the surroundings of vehicle 1. Danger prediction information A21 indicates the danger level of a pedestrian moving in the direction behind as seen by the user of vehicle 1, danger prediction information A22 indicates the danger level of a pedestrian crossing a crosswalk, danger prediction information A23 indicates the danger level of a pedestrian about to cross a crosswalk, and danger prediction information A24 indicates the danger level of a vehicle stopped on the road.
[0137] In addition, if an excessive amount of danger prediction information is presented to the user of vehicle 1, it may cause confusion for the user, so if there are a large number of objects around vehicle 1, only danger prediction information indicating a risk level higher than a predetermined threshold may be presented.
[0138] As described above, in the vehicle control system 11 of the present technology, movement information indicating the future movement of objects around the vehicle 1 and accident information indicating the movement of the objects in the event of an accident are predicted, the degree of danger of the objects is calculated based on the movement information and the accident information, and danger prediction information indicating the degree of danger of the objects is presented to the user of the vehicle 1.
[0139] The vehicle control system 11 does not simply issue a warning that a dangerous object has been detected, but visually indicates which object is moving in which direction and how dangerous each object is. Therefore, the vehicle control system 11 can suitably present information for avoiding danger to the user of the vehicle 1. This allows the user of the vehicle 1 to intuitively know of potential danger, and the vehicle control system 11 can achieve safe driving of the vehicle 1.
[0140] 3. Modifications Variations of hazard prediction information In the above, an example was described in which the range where an object is predicted to reach in a few seconds is indicated by two-dimensional hazard prediction information. In cases where there is a possibility of an accident, such as an elderly person suddenly falling or a child suddenly changing direction of movement, the two-dimensional hazard prediction information may not be able to simultaneously express the range where the object is predicted to move under normal circumstances and the range where the object is predicted to move in the event of an accident. Therefore, the range where the object is predicted to reach in a few seconds may be indicated by three-dimensional hazard prediction information.
[0141] FIG. 11 is a diagram showing a first modified example of the danger prediction information.
[0142] As shown in A of Figure 11, the shape of the danger prediction information A51 is approximately spherical, enveloping the entire pedestrian as an object, and the danger prediction information A51 is given a gradation in which the transparency gradually increases from the front to the back of the pedestrian.
[0143] If there is a high possibility that the pedestrian will suddenly move to the left as seen from the pedestrian's perspective, rather than directly ahead, the possibility that the pedestrian will suddenly move to the left can be expressed by, for example, expanding the part of the spherical surface of the danger prediction information A51 that is to the left as seen from the pedestrian's perspective, as shown in B of Figure 11.
[0144] FIG. 12 is a diagram showing a second modified example of the danger prediction information.
[0145] It is also possible to display multiple planes of danger prediction information corresponding to multiple movement directions in which a single object is predicted to move. In the example of Fig. 12, four pieces of danger prediction information A61-1 to A61-4 are superimposed on one pedestrian. The danger prediction information A61-1 indicates the range in the front direction where the pedestrian is predicted to reach in a few seconds, the danger prediction information A61-2 indicates the range in the left direction where the pedestrian is predicted to reach in a few seconds, the danger prediction information A61-3 indicates the range in the right direction where the pedestrian is predicted to reach in a few seconds, and the danger prediction information A61-4 indicates the range in the rear direction where the pedestrian is predicted to reach in a few seconds.
[0146] The danger prediction information A61-1 to A61-4 may be displayed in one color according to the danger level of the pedestrian.Alternatively, each of the danger prediction information A61-1 to A61-4 may be displayed in a color according to the danger level when the pedestrian moves in each of the forward, backward, leftward, and rightward directions.
[0147] Example of Ensuring Visibility FIG. 13 is a diagram showing an example of displaying danger prediction information to ensure the visibility of the danger prediction information.
[0148] As shown by hatching in Figure 13, image processing may be performed to reduce the brightness or saturation of the entire captured image showing the surroundings of vehicle 1, and danger prediction information A21 to A24 may be superimposed on this image.
[0149] The danger prediction information may be difficult to see if it is superimposed on a captured image showing the surroundings of the vehicle 1. By lowering the brightness or saturation of the captured image, it is possible to ensure the visibility of the danger prediction information.
[0150] Note that image processing to reduce brightness or saturation may be applied to the entire captured image, or may be applied only to the periphery of the portion of the captured image where the danger prediction information is superimposed.
[0151] FIG. 14 is a diagram showing an example of displaying danger prediction information to ensure the visibility of objects.
[0152] When the danger prediction information A71 is displayed so as to envelop the entire object, the transparency of the area along the outline of the object in the danger prediction information A71 may be increased, as shown in Fig. 14. When the danger prediction information is superimposed on an object, the object itself may become difficult to see. By increasing the transparency of the area around the object in the danger prediction information, it is possible to ensure the visibility of the object on which the danger prediction information is superimposed.
[0153] When two-dimensional danger prediction information is displayed, in order to ensure the visibility of the object, it is preferable to superimpose the danger prediction information not directly on the object, but at a position slightly shifted from the object, such as around the object.
[0154] Example of Considering Driving Environment The future movement of an object and the possibility of an accident occurring may be predicted based on not only the type, attribute, state, position, orientation, moving speed, and size of the object, but also the driving environment of the vehicle 1. Furthermore, the degree of danger of the object may be calculated based on the driving environment of the vehicle 1.
[0155] The driving environment of the vehicle 1 includes the location, time of day, weather, and the like. For example, in a school zone, there is a higher possibility that children will run out into the road. In rainy weather or snowfall, the braking distance of the vehicle 1 increases when the brakes are applied. In addition, in rainy weather or snowfall, there is a higher possibility that pedestrians will fall. In this way, the driving environment of the vehicle 1 is taken into consideration when predicting the future movement of an object, the possibility of an accident occurring, and calculating the degree of danger of the object.
[0156] When the vehicle 1 is traveling in a traveling environment where the danger level of objects is higher than normal, the danger prediction information may be displayed with more emphasis than when the vehicle 1 is traveling in a normal traveling environment. Also, dangerous environment information indicating the danger level of the traveling environment may be displayed.
[0157] FIG. 15 is a diagram illustrating an example of hazardous environment information.
[0158] For example, when the road surface is wet due to rain, dangerous environment information A81, which indicates the high level of danger of the road surface with a color, is displayed superimposed on a portion of the road surface as shown in Fig. 15. When the level of danger is high, the dangerous environment information A81 is displayed in orange, for example.
[0159] Along with the dangerous environment information A81, an icon and text indicating that it is raining are displayed, for example, in the lower right corner of the screen of the display unit 202, thereby alerting the user of the vehicle 1 that the road surface is wet and there is a high possibility of an accident occurring.
[0160] FIG. 16 is a diagram showing another example of the hazardous environment information.
[0161] If there is a blind spot around vehicle 1, such as the back of a building, as seen from the user of vehicle 1, dangerous environment information A91, A92 is displayed superimposed on the blind spot, as shown in Fig. 16. In the dangerous environment information A91, A92, for example, if a pedestrian suddenly appears from the blind spot, the width of the range that the pedestrian is predicted to reach in a few seconds is displayed, and the degree of danger of the pedestrian is displayed in color.
[0162] In this way, in places with poor visibility such as intersections, it is possible to display dangerous environment information to alert the user of vehicle 1 even if actual pedestrians, vehicles, etc. are not recognized.
[0163] The present technology can be applied to a variety of products. For example, the present technology may be realized as a device mounted on any type of moving object, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, or a robot.
[0164] Regarding the computer, the above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.
[0165] 17 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program. The information processing unit 201 is, for example, a PC having a configuration similar to that shown in FIG.
[0166] The CPU 501 , ROM (Read Only Memory) 502 , and RAM 503 are interconnected by a bus 504 .
[0167] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc., and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Also connected to the input / output interface 505 are a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives removable media 511.
[0168] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.
[0169] The program executed by the CPU 501 is installed in the storage unit 508 by being recorded on, for example, a removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.
[0170] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.
[0171] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.
[0172] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.
[0173] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.
[0174] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.
[0175] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.
[0176] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.
[0177] Example of configuration combinations The present technology can also be configured as follows.
[0178] (1) An information processing device comprising: a prediction unit that predicts movement information indicating future movement of an object around a moving body and accident information indicating the movement of the object when an accident occurs; a calculation unit that calculates a degree of danger of the object based on the movement information and the accident information; and a presentation control unit that presents danger prediction information indicating the degree of danger of the object to a user of the moving body. (2) The information processing device described in (1), wherein the prediction unit predicts the movement information based on a position, orientation, and movement speed of the object. (3) The information processing device described in (1) or (2), wherein the prediction unit predicts the movement information based on a time series of the movement of the object. (4) The information processing device described in any of (1) to (3), wherein the prediction unit predicts the accident information based on attributes of the object. (5) The information processing device described in any of (1) to (4), wherein the prediction unit predicts the accident information based on an environment around the moving body. (6) The information processing device according to (5), wherein the environment includes a location where the moving object moves, a time period during which the moving object moves, and weather. (7) The information processing device according to any of (1) to (6), wherein the presentation control unit displays the danger prediction information by superimposing it on the object or a periphery of the object. (8) The information processing device according to (7), wherein the presentation control unit controls a display manner of the danger prediction information based on at least one of the danger level of the object, the movement information, and the accident information. (9) The information processing device according to (8), wherein the presentation control unit controls a color of the danger prediction information based on the danger level of the object. (10) The information processing device according to (9), wherein the presentation control unit controls a shape of the danger prediction information based on a predicted moving direction of the object. (11) The information processing device according to (10), wherein the presentation control unit displays a plurality of pieces of danger prediction information corresponding to a plurality of moving directions in which one object is predicted to move by superimposing them on one object or a periphery of one object.(12) The information processing device according to (11), wherein the calculation unit calculates a degree of danger of the object for each of the movement directions of the object, and the presentation control unit controls the color of each of the plurality of pieces of danger prediction information based on the degree of danger of the object calculated for each of the movement directions. (13) The information processing device according to any of (8) to (12), wherein the presentation control unit controls a shape of the danger prediction information based on a range in which the object is predicted to move. (14) The information processing device according to any of (8) to (13), wherein the presentation control unit controls an angle of a side included in the danger prediction information based on a predicted movement speed of the object. (15) The information processing device according to any of (8) to (12) and (14), wherein the presentation control unit controls the shape of the danger prediction information based on the degree of danger of the object. (16) The information processing device according to any of (1) to (15), wherein the presentation control unit presents to the user hazardous environment information indicating a degree of danger in an environment surrounding the moving body. (17) The information processing device according to any of (1) to (16), wherein the prediction unit is provided in the mobile body and predicts the movement information and the accident information based on sensor data from a sensor that senses the surroundings of the mobile body. (18) The information processing device according to any of (1) to (17), wherein the prediction unit predicts the movement information and the accident information based on information about the object acquired from equipment other than the mobile body. (19) An information processing method, wherein an information processing device predicts movement information indicating future movements of objects around the mobile body and accident information indicating movements of the objects in the event of an accident, calculates a degree of danger of the objects based on the movement information and the accident information, and presents danger prediction information indicating the degree of danger of the objects to a user of the mobile body. (20) A program for causing a computer to execute the following process: predicting movement information indicating the future movement of objects around a moving body and accident information indicating the movement of the object in the event of an accident; calculating the degree of danger of the object based on the movement information and the accident information; and presenting danger prediction information indicating the degree of danger of the object to a user of the moving body.
[0179] 1 Vehicle, 11 Vehicle control system, 25 External recognition sensor, 73 Recognition unit, 201 Information processing unit, 202 Display unit, 211 Prediction unit, 212 Risk calculation unit, 213 Image generation unit
Claims
1. An information processing device comprising: a prediction unit that predicts movement information indicating future movements of objects around a moving body and accident information indicating the movement of the object if an accident occurs; a calculation unit that calculates a degree of danger of the object based on the movement information and the accident information; and a presentation control unit that presents danger prediction information indicating the degree of danger of the object to a user of the moving body.
2. The information processing device according to claim 1, wherein the prediction unit predicts the motion information based on the position, orientation, and moving speed of the object.
3. The information processing device according to claim 1, wherein the prediction unit predicts the motion information based on a time series of the motion of the object.
4. The information processing device according to claim 1, wherein the prediction unit predicts the accident information based on attributes of the object.
5. The information processing device according to claim 1, wherein the prediction unit predicts the accident information based on the surrounding environment of the moving object.
6. The information processing device according to claim 5, wherein the environment includes a location where the mobile object moves, a time period during which the mobile object moves, and weather.
7. The information processing device according to claim 1, wherein the presentation control unit displays the danger prediction information by superimposing it on the object or on the periphery of the object.
8. The information processing device according to claim 7, wherein the presentation control unit controls a display mode of the danger prediction information based on at least one of the danger level of the object, the movement information, and the accident information.
9. The information processing device according to claim 8, wherein the presentation control unit controls a color of the danger prediction information based on the degree of danger of the object.
10. The information processing device according to claim 9, wherein the presentation control unit controls a shape of the danger prediction information based on a direction in which the object is predicted to move.
11. The information processing device described in claim 10, wherein the presentation control unit displays a plurality of pieces of danger prediction information corresponding to a plurality of moving directions in which a single object is predicted to move, superimposed on the single object or around the single object.
12. The information processing device described in claim 11, wherein the calculation unit calculates the degree of danger of the object for each of the movement directions of the object, and the presentation control unit controls the color of each of the multiple pieces of danger prediction information based on the degree of danger of the object calculated for each of the movement directions.
13. The information processing device according to claim 8, wherein the presentation control unit controls a shape of the danger prediction information based on a range in which the object is predicted to move.
14. The information processing device according to claim 8, wherein the presentation control unit controls an angle of a side included in the danger prediction information based on a predicted moving speed of the object.
15. The information processing device according to claim 8, wherein the presentation control unit controls a shape of the danger prediction information based on the degree of danger of the object.
16. The information processing device according to claim 1, wherein the presentation control unit presents to the user dangerous environment information indicating a degree of danger in the environment surrounding the moving object.
17. The information processing device according to claim 1, wherein the prediction unit is provided in the moving body and predicts the movement information and the accident information based on sensor data from a sensor that senses the surroundings of the moving body.
18. The information processing device according to claim 1, wherein the prediction unit predicts the movement information and the accident information based on information about the object acquired from equipment other than the moving body.
19. An information processing method in which an information processing device predicts movement information indicating future movements of objects around a moving body and accident information indicating the movement of the object if an accident occurs, calculates the degree of danger of the object based on the movement information and the accident information, and presents danger prediction information indicating the degree of danger of the object to a user of the moving body.
20. A program for causing a computer to execute a process of predicting movement information indicating the future movement of objects around a moving body and accident information indicating the movement of said objects if an accident occurs, calculating the degree of danger of said objects based on said movement information and said accident information, and presenting danger prediction information indicating the degree of danger of said objects to a user of said moving body.
Citation Information
Patent Citations
Danger place information display unit for
JP2007051973A
Obstacle course predicting method, device and program
JP2008123217A
Moving object prediction device and program
JP2011248445A
Image displaying system, and method, for on-vehicle use
JP2017021546A
Mobile body behavior prediction device and mobile body behavior prediction method
JP2019109691A
