INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD AND PROGRAM

Abstract

The present technology relates to an information processing device, an information processing method and a program that enable information to be presented more advantageously to a user of a moving body, such as a vehicle, in order to avoid a hazard. The information processing device, according to one aspect, comprises: a prediction unit that forecasts motion information indicating the future movement of an object near the moving body, and accident information indicating the object's movement upon the occurrence of an accident; a calculation unit that calculates the object's hazard level based on the motion and accident information; and a presentation control unit that presents hazard prediction information indicating the object's hazard level to a user of the moving body. The present technology can be applied, for example, to moving bodies such as vehicles.

Description

Technical field

[0001] The present technology relates to an information processing device, an information processing method and a program, and in particular an information processing device, an information processing method and a program that enable information to be presented more appropriately to a user of a moving body, such as a vehicle, in order to avoid a hazard. State of the art

[0002] In recent years, there has been an increasing adoption of augmented reality (AR) technology in vehicles (see, for example, patent literature 1). For instance, one technology involves overlaying a template identifying the object onto an image of the vehicle's surroundings when a sensor detects an obstacle. List of cited documents Patent literature

[0003] Patent literature 1: WO 2021 / 079975 Disclosure of the invention Technical problem

[0004] Conventional technology expresses a hazard level, based on a prediction of a vehicle's movement and an object's position, through a template size and / or similar metrics. However, since the movement of the obstacle itself is not predicted, the hazard level is not calculated based on a prediction of the object's movement, or the prediction of the object's movement is not presented to the vehicle user. Consequently, in some cases, insufficient information for hazard avoidance is displayed.

[0005] The present technology was developed with regard to the circumstances described above and aims to enable information to be appropriately presented to a user of a moving body, such as a vehicle, to avoid a hazard. Solution to the problem

[0006] An information processing device according to one aspect of the present technology includes a prediction unit that predicts motion information indicating a future movement of an object in the vicinity of a moving body, and accident information indicating a movement of the object in a case where an accident occurs; a calculation unit that calculates a hazard level of the object based on the motion information and the accident information; and a presentation control unit that presents hazard prediction information indicating the hazard level of the object to a user of the moving body.

[0007] An information processing procedure according to one aspect of the present technology is carried out by an information processing device, wherein the information processing procedure includes: predicting motion information indicating a future movement of an object in the vicinity of a moving body, and accident information indicating a movement of the object in a case where an accident occurs; calculating a hazard level of the object based on the motion information and the accident information; and presenting, to a user of the moving body, hazard prediction information indicating the hazard level of the object.

[0008] A program according to one aspect of the present technology causes a computer to perform processing that includes: predicting motion information indicating a future movement of an object in the vicinity of a moving body, and accident information indicating a movement of the object in a case where an accident occurs; calculating a hazard level of the object based on the motion information and the accident information; and presenting, to a user of the moving body, hazard prediction information indicating the hazard level of the object.

[0009] According to the aspect of the present technology, the motion information, which indicates the future movement of the object in the vicinity of the moving body, and the accident information, which indicates the movement of the object in a case where an accident occurs, are calculated; the degree of danger of the object is predicted based on the motion information and the accident information; and the hazard prediction information, which indicates the degree of danger of the object, is presented to the user of the moving body. Brief description of the drawings [ Fig. 1] Fig. Figure 1 is a block diagram showing a configuration example for a vehicle control system. [ Fig. 2] Fig. Figure 2 is a diagram showing an example of detection areas. [ Fig. 3] Fig. Figure 3 is a block diagram showing a configuration example for a vehicle control system to which the present technology is applied. [ Fig. 4] Fig. Figure 4 is a flowchart illustrating a processing procedure performed by the vehicle control system. [ Fig. 5] Fig. Figure 5 is a diagram showing an example of hazard prediction information. [ Fig. 6] Fig. Figure 6 is a diagram showing an example of the hazard prediction information that indicates the hazard level of a moving vehicle. [ Fig. 7] Fig. Figure 7 is a diagram showing an example of the hazard prediction information that indicates the level of danger to a pedestrian crossing an intersection. [ Fig. 8] Fig. Figure 8 is a diagram showing an example of the hazard prediction information that indicates the hazard level of a bicycle. [ Fig. 9] Fig. Figure 9 is a diagram showing an example of the hazard prediction information that indicates the level of danger to a vehicle at a stopping point. [ Fig. 10] Fig. Figure 10 is a diagram showing an example display of hazard prediction information. [ Fig. 11] Fig. Figure 11 is a diagram showing a first modified example of the hazard prediction information. [ Fig. 12] Fig. Figure 12 is a diagram showing a second modified example of the hazard prediction information. [ Fig. 13] Fig. Figure 13 is a diagram showing a display example of hazard prediction information to ensure the visibility of the hazard prediction information. [ Fig. 14] Fig. Figure 14 is a diagram showing an example of how to display hazard prediction information to ensure the visibility of an object. [ Fig. 15] Fig. Figure 15 is a diagram showing an example of hazardous environment information. [ Fig. 16] Fig. Figure 16 is a diagram showing another example of the hazard environment information. [ Fig. 17] Fig. Figure 17 is a block diagram showing a configuration example for computer hardware. Implementation methods of the invention

[0010] The following is a description of ways in which the present revelation can be carried out. Descriptions are given in the following order. 1. Configuration example for a vehicle control system 2. Design 3. Modified Examples <1. Configuration example for a vehicle control system>

[0011] Fig. Figure 1 is a block diagram showing a configuration example for a vehicle control system 11, which is an example of a control system for a mobile body to which the present technology is applied.

[0012] The vehicle control system 11 is provided in a vehicle 1 and performs processing related to the automated driving of the vehicle 1. The automated driving can include levels 1 to 5 and can include remote assistance and / or remote driving of the vehicle 1 by a remote driver.

[0013] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information accumulation unit 23, a position information acquisition unit 24, an external detection sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a storage unit 28, a driving automation control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31 and a vehicle control unit 32.

[0014] The vehicle control ECU 21, the communication unit 22, the map information accumulation unit 23, the position information acquisition unit 24, the external detection sensor 25, the vehicle-internal sensor 26, the vehicle sensor 27, the storage unit 28, the driving automation control unit 29, the DMS 30, the HMI 31, and the vehicle control unit 32 are communicatively interconnected via a communication network 41. The communication network 41 includes, for example, a vehicle-internal communication network, a bus, and / or the like, which conform to a digital bidirectional communication standard, such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), or FlexRay. ® or Ethernet ®The communication network 41 can be used selectively depending on the type of data to be transmitted. For example, CAN can be used for vehicle control data, and Ethernet can be used for large volumes of data. It should be noted that the individual units of the vehicle control system 11 may be connected not via the communication network 41, but directly via wireless communication, which assumes communication over a relatively short distance, such as near-field communication (NFC) and Bluetooth. ® .

[0015] It should be noted that in the following, when the respective units of the vehicle control system 11 communicate with each other via the communication network 41, the communication network 41 will not be described. For example, if the vehicle control ECU 21 and the communication unit 22 communicate with each other via the communication network 41, it will simply be described that the vehicle control ECU 21 and the communication unit 22 are communicating with each other.

[0016] The vehicle control ECU 21 is implemented, for example, by various processors, such as a CPU (Central Processing Unit) and / or an MPU (Micro Processing Unit). The vehicle control ECU 21 controls all or some of the functions of the vehicle control system 11.

[0017] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, and / or the like, to transmit and receive various types of data. At this time, the communication unit 22 can communicate using a variety of communication technologies.

[0018] A communication process that the communication unit 22 can perform with devices outside the vehicle is described schematically. The communication unit 22 communicates with a server or similar device located on an external network (hereinafter referred to as an external server) via a base station or access point using a wireless communication technology, examples of which include 5G (5th Generation Mobile Communication System), LTE (Long Term Evolution), DSRC (Dedicated Short Range Communications), and the like. The external network with which the communication unit 22 communicates is, for example, the internet, a cloud network, a company-specific network, or the like.The communication technology used by the communication unit 22 to communicate with the external network is not particularly restricted, as long as it is a wireless communication technology that enables digital bidirectional communication at a communication speed equal to or greater than a predetermined speed and at a distance equal to or greater than a predetermined distance.

[0019] Furthermore, the communication unit 22 can, for example, communicate with terminal devices located near the vehicle using peer-to-peer (P2P) technology. These terminal devices include, for example, a device carried by a body moving at a relatively low speed, such as a pedestrian or bicycle, a device installed in a fixed position in a shop or similar location, and / or an MTC (machine-type communication) terminal. In addition, the communication unit 22 can also perform V2X communication.V2X communication refers to communication between a vehicle and another entity, examples of which include vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside device or the like, vehicle-to-home communication with a house, vehicle-to-pedestrian communication with an end device or the like carried or transported by a pedestrian, and the like.

[0020] For example, the communication unit 22 can receive a program for updating software to control the operation of the vehicle control system 11 from an external source (over the air). Furthermore, the communication unit 22 can receive map information, traffic information, information about the vehicle 1's surroundings, and / or the like from an external source. Additionally, the communication unit 22 can transmit information relating to the vehicle 1, information about the vehicle 1's surroundings, and / or the like to the outside. Examples of information relating to the vehicle 1 transmitted externally by the communication unit 22 include data indicating the vehicle 1's status, a detection result from a detection unit 73, and the like. The communication unit 22 also performs communication that supports a vehicle emergency call system, such as eCall.

[0021] For example, the communication unit receives 22 electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System)). ® ) transmitted using a radio wave beacon, an optical beacon, an FM multiplex broadcast and / or the like.

[0022] The communication that the communication unit 22 can perform with the vehicle's internal devices is described schematically. The communication unit 22 can communicate with the vehicle's internal devices, for example, using wireless communication. Specifically, the communication unit 22 can communicate wirelessly with a vehicle's internal device using any wireless communication technology that enables digital bidirectional communication at a communication speed equal to or greater than a predetermined speed. Examples of wireless communication technologies include wireless LAN, Bluetooth, NFC, and WUSB (Wireless USB). The communication unit 22 is not limited to these technologies and can also communicate with the vehicle's internal devices using wired communication.For example, the communication unit 22 can communicate with the vehicle's internal devices via wired communication over a cable connected to a connection port (not shown). The communication unit 22 can communicate with the vehicle's internal devices using any wired communication technology that enables digital bidirectional communication at a communication speed equal to or higher than a predetermined speed, with examples of wired communication technologies being USB (Universal Serial Bus) and HDMI (High-Definition Multimedia Interface). ® and MHL (Mobile High-definition Link) are included.

[0023] In this context, in-vehicle devices with which the communication unit 22 communicates refer, for example, to devices that are not connected to the communication network 41 in the vehicle. In-vehicle devices include, for example, a mobile device or a wearable device worn by a user in the vehicle, such as a driver, an information device that is brought into the vehicle and temporarily installed there, and the like.

[0024] The map information accumulation unit 23 accumulates a map acquired from an external source and a map created by the vehicle 1. For example, the map information accumulation unit 23 accumulates a three-dimensional high-precision map, a global map that has lower precision than the high-precision map and covers a wide area, and / or the like.

[0025] The high-precision map is, for example, a dynamic map, a point cloud map, a vector map, or similar. A dynamic map, for instance, is a map with four layers of dynamic, semi-dynamic, semi-static, and static information, and is provided to vehicle 1 from an external server or similar source. A point cloud map is a map that incorporates a point cloud (point group data). A vector map, for example, is a map in which traffic information, such as the positions of lanes and / or traffic lights, is associated with a point cloud map for adaptation to automated driving.

[0026] The point cloud map and the vector map can, for example, be provided by an external server or the like, or they can be created by the vehicle 1 as a map that is to be aligned with a local map described later, based on acquisition results from a camera 51, a radar 52, a LiDAR 53, and the like, and can be accumulated in the map information accumulation unit 23. Furthermore, in a case where a high-precision map is provided by an external server or the like, map data of several hundred square meters with regard to a planned path on which the vehicle 1 is to travel can, for example, be acquired from the external server or the like in order to reduce the communication overhead.

[0027] The position information acquisition unit 24 receives signals from a GNSS (Global Navigation Satellite System) satellite to acquire position information for the vehicle 1. The acquired position information is then transmitted to the driving automation control unit 29. It should be noted that the position information acquisition unit 24 is not limited to the use of GNSS signals, and the position information can also be acquired, for example, using a beacon.

[0028] The external detection sensor 25 includes various sensors used to detect a situation outside the vehicle 1 and provides sensor data from the sensors to the respective units of the vehicle control system 11. The external detection sensor 25 can include any type and any number of sensors.

[0029] For example, the external detection sensor 25 can include the camera 51, the radar 52, the LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. The external detection sensor 25 is not limited to this and can include one or more types of sensors from the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of cameras 51, radars 52, LiDAR 53, and ultrasonic sensors 54 is not particularly limited, as long as they can be installed in the vehicle 1 in practice. Furthermore, the type of sensor included in the external detection sensor 25 is not limited to this example, and the external detection sensor 25 can include a different type of sensor.An example of the detection range of each sensor included in the external detection sensor 25 will be described later.

[0030] It should be noted that the imaging method of Camera 51 is not particularly limited. For example, cameras that use various imaging methods capable of distance measurement, such as a Time-of-Flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera, can be applied to Camera 51 as needed. Camera 51 is not limited to this and can function simply as a camera for capturing an image independently of distance measurement.

[0031] Furthermore, the external detection sensor 25 can, for example, include an environmental sensor for detecting an environment in relation to the vehicle 1. The environmental sensor is a sensor for detecting the environment, such as weather, a meteorological phenomenon, and brightness, and can, for example, include various sensors, such as a raindrop sensor, a fog sensor, a sunshine sensor, a snow sensor, and an illuminance sensor.

[0032] Additionally, the external detection sensor 25 includes, for example, a microphone that is used to detect a sound in the vicinity of the vehicle 1, a position of a sound source and / or the like.

[0033] The vehicle-internal sensor 26 includes various sensors for detecting information within the vehicle and delivers sensor data from the various sensors to the respective units of the vehicle control system 11. The type and number of different sensors included in the vehicle-internal sensor 26 are not particularly limited, as long as the types and numbers allow for the practical installation of the sensors in the vehicle 1.

[0034] For example, the vehicle's internal sensor 26 can include one or more types of sensors, such as a camera, radar, seat sensor, steering wheel sensor, microphone, and biological sensor. The camera included in the vehicle's internal sensor 26 could, for example, be a camera using various imaging techniques capable of distance measurement, such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, or an infrared camera. The camera included in the vehicle's internal sensor 26 is not limited to this; it could simply be a camera for capturing an image independently of distance measurement. The biological sensor included in the vehicle's internal sensor 26 could, for example, be located in a seat, steering wheel, or similar component and could detect various types of biological information from a user.

[0035] The vehicle sensor 27 includes various sensors for detecting the condition of the vehicle 1 and delivers sensor data from the various sensors to the respective units of the vehicle control system 11. The type and number of different sensors included in the vehicle sensor 27 are not particularly limited, as long as the types and numbers allow for the practical installation of the sensors in the vehicle 1.

[0036] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyroscope), and an inertial measurement unit (IMU) that integrates these sensors. For example, the vehicle sensor 27 includes a steering wheel sensor that detects the steering angle of a steering wheel, a yaw rate sensor, an accelerator pedal sensor that detects the degree of actuation of an accelerator pedal, and a brake sensor that detects the degree of actuation of a brake pedal. For example, the vehicle sensor 27 includes a rotation sensor that detects the rotational speed of a power machine or motor, an air pressure sensor that detects the air pressure of a tire, a slip rate sensor that detects the slip rate of a tire, and a wheel speed sensor that detects the rotational speed of a wheel.For example, the vehicle sensor 27 includes a battery sensor that detects the remaining charge and temperature of a battery, and an impact sensor that detects an external impact.

[0037] The memory unit 28 includes a non-volatile storage medium and / or a volatile storage medium and stores data and a program. The memory unit 28 is used, for example, as an EEPROM (Electrically Erasable Programmable Read-Only Memory) and a RAM (Random Access Memory), and a magnetic storage device, such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, and a magneto-optical storage device, can be used as the storage medium. The memory unit 28 stores various programs and data that are used by the respective units of the vehicle control system 11.For example, the storage unit 28 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving) and stores information from the vehicle 1 before and after an event, such as an accident, and / or information captured by the vehicle's internal sensor 26.

[0038] The driving automation control unit 29 controls a driving automation function of the vehicle 1. For example, the driving automation control unit 29 includes an analysis unit 61, an action planning unit 62 and an operating control unit 63.

[0039] The analysis unit 61 performs an analysis of the situation of vehicle 1 and / or a situation in the environment of vehicle 1. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and the detection unit 73.

[0040] The self-position estimation unit 71 estimates the self-position of vehicle 1 based on sensor data from the external detection sensor 25 and the high-precision map accumulated in the map information accumulation unit 23. For example, the self-position estimation unit 71 estimates the self-position of vehicle 1 by generating a local map based on sensor data from the external detection sensor 25 and comparing the local map with the high-precision map. The center of the axle of the rear wheel pair is used as a reference for the position of vehicle 1.

[0041] For example, the local map is a three-dimensional, high-precision map created using a technology such as SLAM (Simultaneous Localization and Mapping), an occupancy grid map, and / or the like. The three-dimensional, high-precision map is, for example, the point cloud map described above, or the like. The occupancy grid map is a map in which a three-dimensional or two-dimensional space around Vehicle 1 is divided into grids of a predetermined size, and an occupancy state of an object is specified for each grid. The occupancy state of an object is specified, for example, by its presence or absence, or by a probability of its presence. The local map is also used, for example, by the detection and recognition processing of a situation outside of Vehicle 1 by the detection unit 73.

[0042] It should be noted that the self-position estimation unit 71 can estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and the sensor data from the vehicle sensor 27.

[0043] The sensor fusion unit 72 performs sensor fusion processing to combine a variety of different types of sensor data (for example, image data supplied by the camera 51 and sensor data supplied by the radar 52) to obtain information. Methods for combining different types of sensor data include composition, integration, fusion, association, and the like.

[0044] The detection unit 73 performs detection processing to detect a situation outside of vehicle 1 and recognition processing to recognize a situation outside of vehicle 1.

[0045] For example, the detection unit 73 performs the detection processing and the recognition processing of a situation outside the vehicle 1 based on information from the external detection sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72 and / or the like.

[0046] In particular, the recognition unit 73 performs, for example, detection processing, recognition processing, and / or the like on an object in the vicinity of vehicle 1. Detection processing of an object is, for example, processing to detect the presence or absence of an object, its size, shape, position, movement (for example, the content of the movement, its direction, and its speed), and / or the like. Recognition processing of an object is, for example, processing to recognize an attribute, such as the type of object, and / or to identify a specific object. Detection processing and recognition processing are not necessarily clearly separated and can be duplicated.

[0047] For example, the detection unit 73 detects an object in the vicinity of vehicle 1 by performing clustering to classify a point cloud, based on sensor data from the radar 52, the LiDAR 53, or the like, into clusters of point groups. Accordingly, the presence or absence, size, shape, and / or position of an object in the vicinity of vehicle 1 are detected.

[0048] For example, the detection unit 73 detects the movement of an object in the vicinity of vehicle 1 by tracking the movement of clusters of point groups classified by clustering. Accordingly, the speed and direction of travel (the motion vector) of the object in the vicinity of vehicle 1 are detected.

[0049] For example, the recognition unit 73 detects or recognizes a vehicle, a person, a bicycle, an obstacle, a structure, a road, a traffic light, a traffic sign, a road traffic sign and / or the like based on the image data supplied by the camera 51. Furthermore, the recognition unit 73 recognizes the type of object in the vicinity of the vehicle 1 by performing recognition processing, such as semantic segmentation.

[0050] For example, based on a map accumulated in the map information accumulation unit 23, an estimated self-position by the self-position estimation unit 71, and an object detection result in the vicinity of the vehicle 1 by the recognition unit 73, the recognition unit 73 can perform traffic rule processing in the vicinity of the vehicle 1. Through this processing, the recognition unit 73 can recognize the position and state of a traffic light, the content of a traffic sign and road sign, the content of a traffic regulation, a drivable lane, and / or the like.

[0051] For example, the detection unit 73 can perform a detection processing of the environmental conditions in the vicinity of vehicle 1. Weather, temperature, humidity, brightness, road surface condition, and the like are assumed to be the environmental conditions in the vicinity to be detected by the detection unit 73.

[0052] Action planning unit 62 creates an action plan for vehicle 1. For example, action planning unit 62 creates an action plan by performing path planning and path tracking processing.

[0053] It should be noted that path planning includes both global and local path planning. Global path planning involves processing to plan a rough path from the start to the destination. Local path planning, also known as trajectory planning, involves processing to generate a trajectory near vehicle 1, along which vehicle 1 can travel safely and without disruption, taking into account the movement characteristics of vehicle 1.

[0054] Path tracking is a processing process used to plan operations for the safe and accurate navigation of a path planned by path planning within a planned time. For example, the action planning unit 62 can calculate a target speed and a target angular velocity for vehicle 1 based on the result of this path tracking processing.

[0055] The operating control unit 63 controls the operation of vehicle 1 in order to achieve the action plan created by the action planning unit 62.

[0056] For example, the operating control unit 63 controls a steering control unit 81, a brake control unit 82, and a drive control unit 83, which are contained in the vehicle control unit 32 described later, to perform vehicle lateral motion control and vehicle longitudinal motion control so that the vehicle 1 travels on the trajectory calculated by the trajectory planning. For example, the operating control unit 63 performs control for the following purposes: driver assistance functions, such as collision avoidance or impact mitigation, control of a distance between vehicles, vehicle speed control, vehicle collision warning, and lane departure warning; and driving automation for driving without the intervention of a driver or a remote driver.

[0057] The DMS 30 performs driver authentication processing, driver condition detection processing, and / or similar processes based on sensor data from the vehicle's internal sensor 26, input data fed into the HMI 31 described later, and / or similar data. Examples of driver condition to be detected include physical condition, alertness level, concentration level, fatigue level, line of sight direction, intoxication level, driving behavior, posture, and similar factors.

[0058] It should be noted that the DMS 30 can perform authentication processing for a user other than the driver and user state detection processing. Furthermore, the DMS 30 can perform situation detection processing within the vehicle based on sensor data from the vehicle's internal sensor 26. Examples of situations to be detected within the vehicle include temperature, humidity, brightness, odor, and the like.

[0059] The HMI 31 receives various types of data, instructions, and the like as input and presents different types of data to a user.

[0060] Data input into the HMI 31 is described schematically. The HMI 31 includes an input device used by a person to enter data. The HMI 31 generates an input signal based on the data, instructions, and / or the like entered through the input device and delivers the input signal to the respective units of the vehicle control system 11. The HMI 31 includes, for example, a control element such as a touch panel, a button, a switch, and / or a lever as the input device. The HMI 31 is not limited to this and may also include an input device that allows information to be entered by a method other than manual operation, such as speech, gestures, and / or the like.Furthermore, the HMI 31 can, for example, use a remote control device that uses infrared beams or radio waves, and / or an external connection device, such as a mobile device and a wearable device that is adaptable to the operation of the vehicle control system 11, as an input device.

[0061] The presentation of data by the HMI 31 is described schematically. The HMI 31 generates visual, acoustic, and tactile information for the user or for the exterior of the vehicle. Furthermore, the HMI 31 performs output control to manage the output, output content, output timing, output method, and / or the like for each element of generated information. For example, the HMI 31 generates and outputs visual information such as an operating screen, a status indicator for vehicle 1, a warning indicator, an image (e.g., a monitoring image) indicating a situation in the vicinity of vehicle 1, and information conveyed by light. Similarly, the HMI 31 generates and outputs acoustic information such as sound, including voice guidance, a warning tone, and a warning message.Furthermore, the HMI 31 generates and outputs tactile information, for example, information that is transmitted to a user's sense of touch through force, vibration, movement and / or the like.

[0062] The output device to which the HMI 31 sends the visual information can be, for example, a display device that presents the visual information by showing an image alone, and / or a projector device that presents the visual information by projecting an image. It should be noted that, in addition to the display device being a standard display, it could also be, for example, a head-up display, a transmissive display, or a display with an augmented reality (AR) function. Alternatively, the display device could be a device that displays visual information within a user's field of vision, such as a wearable device with AR functionality. This wearable device could include eyeglasses, a smartphone, a smartwatch, and the like.Furthermore, a display device included in a navigation device, instrument panel, camera monitoring system (CMS), electronic mirror, light, and / or the like provided in the vehicle 1 can also be used as the output device to which the HMI 31 outputs visual information. Additionally, a retrofitted display device, such as a dashboard-mounted monitor, a tablet device, a trip recording device, and a trip recording device display, can also be used as the output device to which the HMI 131 outputs visual information.

[0063] The output device to which the acoustic information is output is provided on a front surface of a dashboard in front of the driver's seat or a passenger seat, in a console provided between the driver's seat and the passenger seat, on a windscreen, a steering wheel, a rear surface of the driver's seat or the passenger seat and / or the like.

[0064] The output device to which the HMI 31 outputs the acoustic information can be, for example, an audio loudspeaker, headphones and / or earphones.

[0065] The output device to which the HMI 31 sends the tactile information can be, for example, a haptic element using haptic technology. This haptic element is located, for instance, on a part that the user touches, such as a steering wheel or a seat. It should be noted that the tactile information can also be output via a smartphone, smartwatch, and / or similar device carried by the user.

[0066] The vehicle control unit 32 controls the respective units of the vehicle 1. The vehicle control unit 32 includes the steering control unit 81, the brake control unit 82, the drive control unit 83, a body system control unit 84, a lighting control unit 85 and a horn control unit 86.

[0067] The steering control unit 81 performs detection, control, and / or similar functions on the state of the steering system of vehicle 1. The steering system includes, for example, a steering mechanism comprising a steering wheel and the like, an electric power steering system, and / or the like. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, and / or the like.

[0068] The brake control section 82 performs detection, control, and / or similar functions on the condition of the vehicle's braking system 1. The braking system includes, for example, a braking mechanism comprising a brake pedal and the like, an ABS (Anti-lock Braking System), a regenerative braking mechanism, and / or the like. The brake control unit 82 includes, for example, a brake ECU that controls the braking system, an actuator that drives the braking system, and / or the like.

[0069] The vehicle control unit 83 performs detection, control, and / or similar functions on the state of the vehicle 1's drive system. The drive system includes, for example, an accelerator pedal, a power generation device for generating power for an internal combustion engine, a drive motor, or the like, a power transmission mechanism for transferring the power to the wheels, and / or the like. The drive control unit 83 includes, for example, a drive ECU that controls the drive system, an actuator that drives the drive system, and / or the like.

[0070] The body system control unit 84 performs detection, control, and / or similar functions on the state of the vehicle's body system 1. The body system includes, for example, a keyless entry system, a smart key system, power windows, a power seat, air conditioning, an airbag, a seat belt, a gearshift lever, and / or similar components. The body system control unit 84 includes, for example, a body system ECU that controls the body system, an actuator that drives the body system, and / or similar components.

[0071] The lighting control unit 85 detects, controls, and / or otherwise manipulates the states of various lights on the vehicle 1. Examples of lights to be controlled by the lighting control unit 85 include a headlight, a taillight, a fog light, a turn signal indicator, a brake light, a projector light, a bumper indicator, and the like. The lighting control unit 85 includes a lighting ECU that controls a light, an actuator that drives the light, and / or the like.

[0072] The horn control unit 86 performs detection, control, and / or similar functions regarding the state of the vehicle horn of vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, and / or similar components.

[0073] Fig. Figure 2 is a diagram showing an example of the detection ranges of the camera 51, the radar 52, the LiDAR 53, the ultrasonic sensor 54 and the like of the external detection sensor 25. Fig. 1 shows. It should be noted that Fig. 2 shows the vehicle 1 schematically viewed from above, with the left end being the front end (front) of the vehicle 1 and the right end being the rear end (rear) of the vehicle 1.

[0074] A detection range 101F and a detection range 101B are examples of the detection ranges of the ultrasonic sensor 54. Detection range 101F covers the periphery of the front end of the vehicle 1 by means of the plurality of ultrasonic sensors 54. Detection range 101B covers the periphery of the rear end of the vehicle 1 by means of the plurality of ultrasonic sensors 54.

[0075] The recording results in the recording area 101F and the recording area 101B are used, for example, in a parking assistance system for vehicle 1 and the like.

[0076] A detection range 102F, a detection range 102B, a detection range 102L, and a detection range 102R provide examples of detection ranges for radar 52 at short or medium range. Detection range 102F covers a position further in front of vehicle 1 than detection range 101F. Detection range 102B covers a position further behind vehicle 1 than detection range 101B. Detection range 102L covers the rear periphery of the left side of vehicle 1. Detection range 102R covers the rear periphery of the right side of vehicle 1.

[0077] The detection result in detection area 102F is used, for example, for the detection of a vehicle, pedestrian, or similar object located in front of vehicle 1. The detection result in detection area 102B is used, for example, for a collision avoidance function or similar function located behind vehicle 1. The detection results in detection areas 102L and 102R are used, for example, for the detection of an object in the blind spots to the side of vehicle 1.

[0078] A detection area 103F, a detection area 103B, a detection area 103L, and a detection area 103R provide examples of detection areas for camera 51. Detection area 103F covers a position further in front of vehicle 1 than detection area 102F. Detection area 103B covers a position further behind vehicle 1 than detection area 102B. Detection area 103L covers the periphery of the left side of vehicle 1. Detection area 103R covers the periphery of the right side of vehicle 1.

[0079] The detection result in detection area 103F can be used, for example, for the detection of a traffic light and / or a traffic sign, in a lane keeping assist system, and / or an automatic headlight control system. The detection result in detection area 103B can be used, for example, in a parking assist system and / or in a surround view system. The detection results in detection areas 103L and 103R can be used, for example, in a surround view system.

[0080] A detection area 104 provides an example of a detection area of ​​the LiDAR 53. Detection area 104 covers a position further in front of vehicle 1 than detection area 103F. On the other hand, detection area 104 has a narrower area in the left / right direction than detection area 103F.

[0081] The detection result in detection area 104 is used, for example, when detecting an object such as a peripheral vehicle.

[0082] A detection range of 105 provides an example of the detection range of radar 52 for a long distance. Detection range 105 covers a position further in front of vehicle 1 than detection range 104. On the other hand, detection range 105 has a narrower area in the left / right direction than detection range 104.

[0083] The detection result in detection area 105 is used, for example, in ACC (Adaptive Cruise Control), emergency braking, collision avoidance and / or the like.

[0084] It should be noted that the detection ranges of the sensors, which include the cameras 51, the radar 52, the LiDAR 53 and the ultrasonic sensor 54 contained in the external detection sensor 25, have different configurations than those shown in Fig. The installation positions of the sensors are not limited to the examples described above. Furthermore, the number of sensors can be one or more. 2. Design

[0085] Next, an embodiment of the present technology will be described with reference to Fig. 3 to 10 described.

[0086] Fig. Figure 3 is a block diagram showing a configuration example of the vehicle control system 11 to which the present technology is applied.

[0087] The in Fig. The vehicle control system 11 shown in Figure 3 includes, in addition to the configuration described above (the external detection sensor 25 and the detection unit 73), an information processing unit 201, which calculates the hazard level of an object in the vicinity of the vehicle 1 and generates hazard prediction information indicating the hazard level of the object, and a display unit 202, which displays the hazard prediction information. It should be noted that Fig. Figure 3 shows a configuration of part of the vehicle control system 11, which relates to the display of hazard prediction information.

[0088] As described above, the detection unit 73 acquires sensor data from the external detection sensor 25 and performs object recognition processing of an object in the vicinity of the vehicle 1 based on the sensor data using AI (Artificial Intelligence), which is trained using deep learning or other machine learning, a neural network, and / or the like. For example, the detection unit 73 recognizes the type, attribute, state, position, orientation, speed of movement, and size of an object in the vicinity of the vehicle 1 through its recognition processing.

[0089] The object type indicates whether the object is a pedestrian, a vehicle, a ball, a guardrail, a traffic sign, an obstacle, or the like. The recognition unit 73 recognizes not only moving objects, such as a pedestrian, a vehicle, and a ball, but also stationary objects, such as a guardrail, a traffic sign, and an obstacle.

[0090] If the object type is a pedestrian, the object's attribute specifies whether the pedestrian is a child, an elderly person, and / or similar. If the object type is a vehicle, the object's attribute specifies the vehicle's type, color, and / or similar characteristics. The object's state specifies a condition in which a pedestrian corresponding to an object is walking while looking at a smartphone, a condition in which the vehicle's brake light is illuminated, and / or similar.

[0091] The detection unit 73 delivers the detection result of the object in the vicinity of vehicle 1 to the information processing unit 201. It should be noted that the detection unit 73 can acquire information relating to the object in the vicinity of vehicle 1 from devices other than vehicle 1 via the communication unit 22 and perform the detection processing based on this information.

[0092] The information processing unit 201 is part of the function of the HMI 31. The information processing unit 201 includes a prediction unit 211, a hazard degree calculation unit 212 and a video generation unit 213.

[0093] Based on the object detection result obtained by the detection unit 73, the prediction unit 211 predicts a future movement of an object in the vicinity of vehicle 1 and a movement of the object in a case where an accident occurs.

[0094] Specifically, the prediction unit 211 forecasts the area the object will reach in a few seconds, indicating its future movement. This area is predicted based on the object's position, orientation, and speed of movement.

[0095] It should be noted that the area the object will reach in a few seconds can alternatively be predicted based on the object's movement over time. For example, the area the object will reach in a few seconds can be predicted based on the amount of movement the object made during a recent predetermined period. Alternatively, the area the object will reach in a few seconds can be predicted taking into account the object's state, such as a pedestrian walking while looking at a smartphone or a vehicle's brake light being illuminated.

[0096] For example, the Predictive Unit 211 predicts the likelihood of an accident occurring and the movement of an object in the event of an accident (the area the object will reach in a few seconds) based on the object's attribute, such as "it is very likely that children will suddenly run into the road" or "it is very likely that elderly people will fall while crossing the road." The Predictive Unit 211 can also predict the likelihood of an accident occurring and the movement of an object in the event of an accident, taking into account the position of a stationary object, such as "it is unlikely that children will suddenly run into the road because of the guardrail."

[0097] The prediction unit 211 provides motion information, indicating the future movement of the object, and accident information, indicating the object's movement in the event of an accident, to the hazard calculation unit 212 and the video generation unit 213. The accident information includes details indicating the likelihood of an accident occurring. Alternatively, the accident information can include a prediction difficulty rating, which represents the degree of difficulty in predicting the object's future movement.

[0098] The hazard level calculation unit 212 calculates the hazard level of the object based on the object detection result obtained by the detection unit 73 and the movement and accident information supplied by the prediction unit 211. For example, the possibility of a collision between the object and vehicle 1 is estimated as the hazard level based on the area that the object is predicted to reach in a few seconds and the direction and speed of movement of vehicle 1.

[0099] The hazard level calculation unit 212 provides the hazard level of the object to the video generation unit 213.

[0100] The video generation unit 213 generates hazard prediction information based on the motion and accident information supplied by the prediction unit 211 and the hazard level of the object supplied by the hazard level calculation unit 212. Specifically, the video generation unit 213 controls a display mode for the hazard prediction information based on the motion information and / or the accident information and / or the hazard level. The display mode for the hazard prediction information includes color, shape (width, slant, direction), transparency, and / or the like.

[0101] The video generation unit 213 generates a video that includes the hazard prediction information. For example, the video generation unit 213 overlays the hazard prediction information onto a recorded video showing the surroundings of vehicle 1. The video generation unit 213 acts as a presentation control unit that controls the display unit 202 to present a video to the user of vehicle 1.

[0102] The display unit 202 includes, for example, a display installed inside the vehicle. The display unit 202 shows a video, controlled by the video generation unit 213, and presents hazard prediction information to the vehicle's user.

[0103] All or some of the functions of the Recognition Unit 73 and the Information Processing Unit 201 can be provided in a device other than the Vehicle 1, such as a cloud.

[0104] Next, processing is carried out by the vehicle control system 11, which has the configuration described above, with reference to the in Fig. The flowchart shown in section 4 describes the process. For example, the process shown in Fig. The processing shown in section 4 is started when an operation is performed to activate vehicle 1 for starting driving, that is, when, for example, an ignition switch, an on / off switch, a start switch, or the like of vehicle 1 is turned on. Furthermore, the processing shown in Fig. The processing shown in section 4 is terminated when an operation is performed to stop the driving of vehicle 1, that is, when, for example, the ignition switch, the on / off switch, the start switch or the like of vehicle 1 is switched off.

[0105] In step S1, the external detection sensor 25 detects a situation in the vicinity of the vehicle 1.

[0106] In step S2, the detection unit 73 detects an object in the vicinity of the vehicle 1 based on the sensor data from the external detection sensor.

[0107] In step S3, the information processing unit 201 determines whether the recognition unit 73 has recognized an object.

[0108] If step S3 determines that an object has been detected, processing proceeds to step S4. In step S4, the prediction unit 211 forecasts the future movement of the object detected by the detection unit 73 and the possibility of an accident occurring.

[0109] In step S5, the hazard level calculation unit 212 calculates the hazard level of the object detected by the detection unit 73 based on the detection result obtained by the detection unit 73 and the prediction result obtained by the prediction unit 211.

[0110] After the hazard level of the object has been calculated in step S5, processing proceeds to step S6. However, if it is determined in step S3 that no object was detected, processing in steps S4 and S5 is skipped, and processing proceeds to step S6.

[0111] In step S6, the video generation unit 213 creates a video to be presented to the user of vehicle 1 by the display unit 202. If an object is detected by the detection unit 73, the video generation unit 213 creates a video to be presented by the display unit 202, for example, by overlaying hazard prediction information onto a recorded video showing the surroundings of vehicle 1. If no object is detected by the detection unit 73, the video generation unit 213 uses, for example, the captured video showing the surroundings of vehicle 1 unchanged as the video to be presented by the display unit 202.

[0112] In step S7, the display unit 202 presents the video to the user of vehicle 1 under control of the video generation unit 213.

[0113] Fig. Figure 5 is a diagram showing an example of hazard prediction information.

[0114] As in Fig. As shown in Figure 5, when a pedestrian corresponding to an object is stationary or predicted to stop, the hazard prediction information A1, for example, has a square shape with long sides connecting the pedestrian's head and feet. Furthermore, as shown in Figure 5, when a pedestrian corresponding to an object is stationary or predicted to stop, the hazard prediction information A1 has a square shape with long sides connecting the pedestrian's head and feet. Fig. 5 shows that if the pedestrian is moving or is predicted to move, the hazard prediction information A1, for example, shows a trapezoidal shape with rounded corners and sides connecting the pedestrian's head and feet.

[0115] The hazard prediction information A1 has a bulging shape on an upper section of its side on the direction-of-movement side (left side) where the pedestrian is predicted to be moving, and the side of the hazard prediction information A1 on the direction-of-movement side bulges more as the area that the pedestrian is predicted to reach in a few seconds becomes larger.

[0116] In other words, the width (the length of the upper side) of the hazard prediction information A1 indicates the magnitude of the pedestrian's future movement. Furthermore, the direction of the bulge in the hazard prediction information A1 indicates the predicted direction of the pedestrian's movement. Additionally, the exterior angle of the vertex (the slope of the side on the side corresponding to the predicted direction of the pedestrian's movement) on the lower side of the hazard prediction information A1 in the predicted direction of the pedestrian's movement (lower left side) also indicates the magnitude of the pedestrian's future movement.

[0117] The color of the hazard prediction information A1 indicates the level of danger posed by the pedestrian. For example, at a low level of danger, the hazard prediction information A1 is displayed in cyan, at a medium level of danger, it is displayed in yellow, and at a high level of danger, it is displayed in orange. The hazard prediction information A1 is graduated, so that, for example, its transparency increases as one approaches the pedestrian.

[0118] It should be noted that the width of the hazard prediction information A1, or the slope of the side of the hazard prediction information A1 on the side of the predicted direction of pedestrian movement, can indicate information other than the magnitude of the pedestrian's future movement. For example, the width of the hazard prediction information A1 can indicate the amplitude of the pedestrian's future movement. For example, the position of the side of the hazard prediction information A1 on the side of the predicted direction of pedestrian movement indicates an upper limit to the area the pedestrian can reach in a few seconds, and the position of the side opposite the predicted direction of pedestrian movement indicates a lower limit to the area.

[0119] Furthermore, the width of the hazard prediction information A1 can, for example, indicate the degree of danger posed by the pedestrian. The slope of the side of the hazard prediction information A1 on the side corresponding to the predicted direction of movement of the pedestrian can indicate the predicted speed of the pedestrian.

[0120] Fig. Figure 6 is a diagram showing an example of the hazard prediction information that indicates the hazard level of a moving vehicle.

[0121] If it is predicted that vehicle C1 will veer left at high speed and block the path of vehicle 1, the vehicle control system 11, for example, calculates a high hazard level for vehicle C1 and overlays hazard prediction information A11, indicating the hazard level of vehicle C1, onto a part of vehicle C1 and displays it in orange, as shown in Fig. Figure 6 shows that, since it is predicted that the vehicle C1 will travel at a high speed, the vehicle control system 11, for example, increases the width of the hazard prediction information A11 and reduces the slope of the side of the hazard prediction information A11 on the left side.

[0122] Fig. Figure 7 is a diagram showing an example of the hazard prediction information that indicates the level of danger to a pedestrian crossing an intersection.

[0123] If it is predicted that pedestrian P1 will cross the intersection and move in the left direction, the vehicle control system 11, for example, calculates a medium hazard level for pedestrian P1 and overlays hazard prediction information A12, which indicates the hazard level of pedestrian P1, onto a part of pedestrian P1 and displays it in yellow, as in Fig. Figure 7 shows that, since it is predicted that pedestrian P1 is moving at a low speed, the vehicle control system 11, for example, reduces the width of the hazard prediction information A12 and increases the slope of the side of the hazard prediction information A12 on the left side.

[0124] It should be noted that, as in Fig. Figure 7 shows that a point Po1 can be displayed in a state where it is superimposed on the head of pedestrian P1. Displaying point Po1 likely indicates that the presence of the pedestrian is detected by the user of vehicle 1. If the presence of pedestrian P1 around vehicle 1 is detected, but the orientation of pedestrian P1 is not detected or the future movement of pedestrian P1 is unpredictable, only point Po1 can be displayed without the hazard prediction information A12.

[0125] Fig. Figure 8 is a diagram showing an example of the hazard prediction information that indicates the hazard level of a bicycle.

[0126] If it is predicted that bicycle B1 will suddenly move to the right and in front of vehicle 1, the vehicle control system 11, for example, calculates a high hazard level for bicycle B1 and overlays hazard prediction information A13, which indicates the hazard level of the bicycle, onto a part of the bicycle and displays it in orange, as in Fig. 8 shown.

[0127] Since it is predicted that bicycle B1 will move at high speed relative to vehicle C1, the vehicle control system 11, for example, increases the width of the hazard prediction information A13 and reduces the slant of the side of the hazard prediction information A13 on the right side. Since it is predicted that bicycle B1 will move in the rear right direction (the right front of vehicle 1), the hazard prediction information A13 is drawn in perspective so that the user of vehicle 1 can easily see that bicycle B1 is moving in the rear right direction.

[0128] It should be noted that, as in Fig. Figure 8 shows that a point Po2 can be displayed in a state where it is superimposed on the head of a person riding the bicycle B1.

[0129] Fig. Figure 9 is a diagram showing an example of the hazard prediction information that indicates the level of danger to a vehicle at a stopping point.

[0130] If it is predicted that vehicle C2 will stop at a location that does not impede the movement of vehicle 1, the vehicle control system 11, for example, calculates a low hazard level for vehicle C2 and displays hazard prediction information A14, which indicates the hazard level of vehicle C2 in cyan, as shown in Fig. 9 shown.

[0131] The degree of danger posed by a vehicle at a stopping point or a stationary obstacle is indicated, for example, by column-shaped hazard prediction information arranged at the corners of a boundary box surrounding a lower surface of the vehicle or obstacle.

[0132] The hazard prediction information, as described above, can be superimposed on a recorded video showing the surroundings of vehicle 1, which is to be displayed on a display installed in vehicle 1, or can be displayed on a head-up display so that it is superimposed on a real object.

[0133] It should be noted that the display mode of the hazard prediction information does not refer to the information referred to in Fig. The display modes described in 5 to 9 are limited.

[0134] Fig. Figure 10 is a diagram showing an example display of hazard prediction information.

[0135] In the Fig. In the example shown, hazard prediction information A21 to A24 is displayed in a state where it is superimposed on a recorded video showing the surroundings of vehicle 1. Hazard prediction information A21 indicates the hazard level of a pedestrian moving in the direction behind the vehicle 1's occupant, and hazard prediction information A22 indicates the hazard level of a pedestrian crossing an intersection. Hazard prediction information A23 indicates the hazard level of a pedestrian about to cross the intersection, and hazard prediction information A24 indicates the hazard level of a vehicle stopped on the road.

[0136] It should be noted that, since presenting an excessive amount of hazard prediction information to the user of vehicle 1 may irritate the user when there are a large number of objects in the vicinity of vehicle 1, only the hazard prediction information indicating the level of hazard that is higher than a predetermined threshold may be presented.

[0137] As described above, in the vehicle control system 11, according to the present technology, the motion information, which indicates a future movement of an object in the vicinity of the vehicle 1, and the accident information, which indicates a movement of the object in a case where an accident occurs, are calculated, the degree of danger of the object is predicted based on the motion information and the accident information, and the hazard prediction information, which indicates the degree of danger of the object, is presented to the user of the vehicle 1.

[0138] The vehicle control system 11 not only warns that a dangerous object has been detected, but also visually displays which object is moving in which direction and how dangerous each object is. Therefore, the vehicle control system 11 can provide the vehicle 1 user with beneficial information for avoiding hazards. This allows the vehicle 1 user to intuitively recognize potential dangers, and the vehicle control system 11 can ensure safe driving of the vehicle 1. <3. Modified Examples> • Variations in Hazard Forecast Information

[0139] The above descriptions were based on the example where the area an object is predicted to reach within a few seconds is specified using planar hazard prediction information. In cases where there is a possibility of an accident occurring, such as an elderly person suddenly falling or a child suddenly changing direction, it can be difficult for planar hazard prediction information to simultaneously express the area an object is predicted to move within under normal circumstances and the area the object is predicted to move within in the event of an accident. In this respect, the area an object is predicted to reach within a few seconds can be specified using three-dimensional hazard prediction information.

[0140] Fig. Figure 11 is a diagram showing a first modified example of the hazard prediction information.

[0141] As in A of Fig. As shown in Figure 11, the hazard prediction information A51 is essentially spherical to represent the entire pedestrian corresponding to an object, and a gradation is applied to the hazard prediction information A51 such that the transparency gradually increases from the front direction to the rear direction of the pedestrian.

[0142] If the pedestrian is highly likely to suddenly move to the left from the pedestrian's perspective instead of forward, the possibility of the pedestrian suddenly moving to the left is expressed, for example, by causing part of the spherical surface of the hazard prediction information A51 to bulge in the left direction from the pedestrian's perspective, as shown in B of Fig. 11 shown.

[0143] Fig. Figure 12 is a diagram showing a second modified example of the hazard prediction information.

[0144] Multiple elements of planar hazard prediction information, each corresponding to directions of movement of several directions in which an object is predicted to move, can be displayed. In the example of Fig. In Figure 12, four elements of hazard prediction information, A61-1 to A61-4, are superimposed on a pedestrian. Hazard prediction information A61-1 indicates an area in the forward direction that the pedestrian is predicted to reach in a few seconds, hazard prediction information A61-2 indicates an area in the left direction that the pedestrian is predicted to reach in a few seconds, hazard prediction information A61-3 indicates an area in the right direction that the pedestrian is predicted to reach in a few seconds, and hazard prediction information A61-4 indicates an area in the rear direction that the pedestrian is predicted to reach in a few seconds.

[0145] Hazard prediction information A61-1 to A61-4 can be displayed in a color according to the pedestrian's hazard level. Alternatively, each hazard prediction information A61-1 to A61-4 can be displayed in a color corresponding to the hazard level if the pedestrian is moving in the corresponding direction: front, back, left, or right.

[0146] • Example of ensuring visibility Fig. Figure 13 is a diagram showing a display example of hazard prediction information to ensure the visibility of the hazard prediction information.

[0147] As in Fig. As shown in Figure 13, image processing can be performed to reduce the brightness or chrominance of the entire recorded video showing the environment of vehicle 1, and hazard prediction information A21 to A24 can be superimposed on this video.

[0148] The visibility of hazard prediction information can be impaired if it is overlaid on an unprocessed recorded video showing the surroundings of vehicle 1. Reducing the brightness or chrominance of the recorded video can ensure the visibility of the hazard prediction information.

[0149] It should be noted that image processing to reduce brightness or chrominance can be applied to the entire captured video or only to the periphery of the part of the captured video where the hazard prediction information is superimposed.

[0150] Fig. Figure 14 is a diagram showing an example of how to display hazard prediction information to ensure the visibility of an object.

[0151] If the hazard prediction information A71 is displayed in such a way that it shows the entire object, the transparency of the area along the outline of the object in the hazard prediction information A71 can be increased, as shown in Fig. Figure 14 shows that when hazard prediction information is overlaid on an object, the object's visibility can be impaired. Increasing the transparency of a portion of the object's periphery within the hazard prediction information can ensure the visibility of the object overlaid with the hazard prediction information.

[0152] When displaying planar hazard prediction information, to ensure the visibility of the object, it is preferable to overlay the hazard prediction information not directly onto the object, but onto a position slightly different from the object, such as around the object. • Example that takes a driving environment into account

[0153] The future movement of an object and the possibility of an accident can be predicted based not only on the object's type, attributes, condition, position, orientation, speed, and size, but also on the driving environment of vehicle 1. Furthermore, the object's hazard level can also be calculated based on vehicle 1's driving environment.

[0154] The driving environment of vehicle 1 includes the location, time of day, weather, and similar factors. For example, in a school zone, there is a good chance that children will run into the street. In rainy or snowy weather, vehicle 1's braking distance will be longer. Furthermore, in rainy or snowy weather, there is a risk of pedestrians falling. Thus, the prediction of an object's future movement, the possibility of an accident, and the calculation of the object's hazard level are performed, taking vehicle 1's driving environment into account.

[0155] It should be noted that if vehicle 1 is moving in a driving environment where the hazard level of an object is higher than under normal circumstances, the hazard prediction information may be displayed more prominently compared to a case where vehicle 1 is moving in a normal driving environment. Alternatively, hazard environment information may be displayed, indicating the hazard level of the driving environment.

[0156] Fig. Figure 15 is a diagram showing an example of the hazard environment information.

[0157] For example, when the road surface is wet from rain, hazard environment information A81, which uses color to indicate that the hazard level of the road surface is high, is displayed in a state where it is overlaid on a portion of the road surface, as in Fig. 15 is shown. For example, in cases of high hazard level, the hazard environment information A81 is displayed in orange.

[0158] For example, by displaying a symbol and text indicating that it is raining in a lower right corner of the screen of display unit 202, together with the hazard environment information A81, the user of vehicle 1 is reminded that the possibility of an accident occurring is high because the road surface is wet.

[0159] Fig. Figure 16 is a diagram showing another example of the hazard environment information.

[0160] If there are blind spots in the vicinity of vehicle 1 from the perspective of the vehicle's user, such as the back of a building, elements of hazard environment information A91 and A92 are displayed in a state where they are superimposed on the blind spots, as shown in Fig. 16 shown. In the elements of hazard environment information A91 and A92, for example, if a pedestrian walks out from the place that is a blind spot, the area that is predicted to reach the pedestrian in a few seconds is indicated by the width, and the degree of danger to the pedestrian is indicated by the color.

[0161] In this way, in places with poor visibility, such as intersections, even if an actual pedestrian, vehicle or the like is not detected, it is possible to display the hazard environment information to warn the user of vehicle 1. • Miscellaneous

[0162] The present technology can be applied to various products. For example, the present technology can be implemented as a device mounted on any type of mobile body, such as a motor vehicle, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an aircraft, a drone, a ship, or a robot. • With regard to a computer

[0163] The processing sequence described above can be executed by hardware or by software. If the processing sequence is executed by software, a program configuring the software is installed on a computer integrated into dedicated hardware, a general-purpose personal computer, and / or the like, from a medium on which the program is recorded.

[0164] Fig. Figure 17 is a block diagram showing a hardware configuration example of a computer that executes the processing sequence described above through a program. For example, the Information Processing Unit 201 includes a PC with a configuration similar to that shown in Figure 17. Fig. 17 configurations shown.

[0165] A CPU 501, a ROM (Read Only Memory) 502 and a RAM 503 are connected to each other via a bus 504.

[0166] An input / output interface 505 is also connected to bus 504. Connected to the input / output interface 505 are an input unit 506, which includes a keyboard, mouse, and / or the like, and an output unit 507, which includes a display, speaker, and / or the like. Additionally, connected to the input / output interface 505 are a storage unit 508, which includes a hard disk, non-volatile memory, and / or the like, a communication unit 509, which includes a network interface and / or the like, and a drive 510, which drives a removable medium 511.

[0167] In the computer configured as described above, the CPU 501 loads a program stored in the memory unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executes it, for example to carry out the processing sequence described above.

[0168] The program to be executed by the CPU 501 is, for example, recorded on the removable medium 511 or provided via a wired or wireless transmission medium, such as a local network, the Internet and a digital broadcast, to be installed in the storage unit 508.

[0169] It should be noted that the program to be executed by the computer may be a program in which the processing is carried out in time sequences in the order described in this document, or a program in which the processing is carried out in parallel or at required times, such as during a call.

[0170] In this document, the system refers to an assemblage of several components (devices, modules (components), and the like), and it is irrelevant whether all components are located in the same housing. Accordingly, several devices, each housed in separate housings and interconnected via a network, and a single device containing several modules in a single housing, each constitute a system.

[0171] The effects described in this document are merely examples and are not limited; other effects may also be achieved.

[0172] Furthermore, the embodiment of the present technology is not limited to the embodiment described above and can be modified in various ways without deviating from the basic idea of ​​the present technology.

[0173] For example, the present technology can adopt a cloud computing configuration in which a single function is shared to be processed cooperatively by multiple facilities over a network.

[0174] Furthermore, the respective steps described above in the flowchart can be carried out by a single facility or used jointly by several facilities for execution.

[0175] Furthermore, if multiple processing operations are included in a single step, the multiple processing operations included in that single step can be performed by a single facility or divided among multiple facilities for execution. • Combination example of configurations

[0176] The present technology can also assume the following configurations. (1) An information processing facility comprising: a prediction unit that predicts motion information indicating a future movement of an object in the vicinity of a moving body, and accident information indicating a movement of the object in a case where an accident occurs; a calculation unit that calculates a hazard level of the object based on movement information and accident information; and A presentation control unit that presents hazard prediction information to a user of the moving body, indicating the level of danger posed by the object. (2) The information processing facility according to (1), wherein The prediction unit forecasts the movement information based on the object's position, orientation, and speed of movement. (3) The information processing facility according to (1) or (2), wherein The prediction unit forecasts motion information based on the object's movement in time sequences. (4) The information processing facility according to one of (1) to (3), wherein The prediction unit predicts accident information based on an attribute of the object. (5) The information processing facility according to one of (1) to (4), wherein The prediction unit forecasts accident information based on the environment of the moving body. (6) The information processing facility according to (5), wherein The environment includes a place where the moving body is located, a time of day during which the moving body is moving, and weather conditions. (7) The information processing facility according to one of (1) to (6), wherein The presentation control unit displays the hazard prediction information in a state where it is superimposed on the object or the object's environment. (8) The information processing facility according to (7), wherein The presentation control unit controls a display mode of hazard prediction information based on the hazard level of the object and / or the movement information and / or the accident information. (9) The information processing facility according to (8), wherein The presentation control unit controls the color of the hazard prediction information based on the hazard level of the object. (10) The information processing facility according to (9), wherein The presentation control unit controls a form of hazard prediction information based on a direction of movement in which the object is predicted to move. (11) The information processing facility according to (10), wherein The presentation control unit displays several of the elements of hazard prediction information in a state where they are superimposed on the object or the object's environment, wherein the several elements of hazard prediction information are several of the elements of hazard prediction information, each corresponding to several of the directions of movement in which the object is predicted to move. (12) The information processing facility according to (11), wherein The calculation unit calculates the degree of danger of the object for each of the object's directions of movement and The presentation control unit controls the color of each of the multiple elements of hazard prediction information based on the hazard level of the object, which is calculated for each of the directions of movement. (13) The information processing facility according to one of (8) to (12), wherein The presentation control unit controls a form of hazard prediction information based on an area in which the object is predicted to move. (14) The information processing facility according to one of (8) to (13), wherein The presentation control unit controls an angle formed by one side of the hazard prediction information, based on a predicted movement speed of the object. (15) The information processing facility according to one of (8) to (12) and (14), wherein The presentation control unit controls a form of hazard prediction information based on the hazard level of the object. (16) The information processing facility according to one of (1) to (15), wherein The presentation control unit provides the user with hazard environment information, indicating the level of danger of an environment surrounding the moving body. (17) The information processing facility according to one of (1) to (16), wherein The prediction unit predicts the motion information and the accident information based on sensor data from a sensor that is provided on the moving body and that detects the environment of the moving body. (18) The information processing facility according to one of (1) to (17), wherein The prediction unit predicts movement information and accident information based on information relating to the object that is captured by devices other than the moving body. (19) Information processing operation carried out by an information processing establishment, wherein the information processing operation includes: Predictions of motion information, indicating a future movement of an object in the vicinity of a moving body, and accident information, indicating a movement of the object in a case where an accident occurs; Calculating the hazard level of the object based on movement information and accident information; and Presenting, for a user of the moving body, hazard prediction information that indicates the level of danger of the object. (20) Program for causing a computer to perform processing which includes: Predictions of motion information, indicating a future movement of an object in the vicinity of a moving body, and accident information, indicating a movement of the object in a case where an accident occurs; Calculating the hazard level of the object based on movement information and accident information; and Presenting, for a user of the moving body, hazard prediction information that indicates the level of danger of the object. Reference symbol list 1 vehicle 11 Vehicle control system 25 external detection sensor 73 Recognition unit 201 Information Processing Unit 202 Display unit 211 Forecast Unit 212 Hazard Degree Calculation Unit 213 Video generation unit QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] WO 2021 / 079975

[0003]

Claims

[1] Information processing equipment comprising the following: a prediction unit that predicts motion information indicating a future movement of an object in the vicinity of a moving body, and accident information indicating a movement of the object in a case where an accident occurs; a calculation unit that calculates a hazard level of the object based on movement information and accident information; and A presentation control unit that presents hazard prediction information to a user of the moving body, indicating the level of danger posed by the object. [2] Information processing device according to claim 1, wherein the prediction unit predicts the motion information based on a position, an orientation and a speed of movement of the object. [3] Information processing device according to claim 1, wherein the prediction unit predicts the motion information based on the movement of the object in time sequences. [4] Information processing device according to claim 1, wherein the prediction unit predicts the accident information based on an attribute of the object. [5] Information processing device according to claim 1, wherein the prediction unit predicts the accident information based on an environment of the moving body. [6] Information processing device according to claim 5, wherein the environment includes a location where the moving body is located, a time of day during which the moving body is located, and weather conditions. [7] Information processing device according to claim 1, wherein the presentation control unit displays the hazard prediction information in a state in which it is superimposed on the object or the object's environment. [8] Information processing device according to claim 7, wherein the presentation control unit controls a display mode of the hazard prediction information based on the hazard level of the object and / or the movement information and / or the accident information. [9] Information processing device according to claim 8, wherein the presentation control unit controls a color of the hazard prediction information based on the hazard level of the object. [10] Information processing device according to claim 9, wherein the presentation control unit controls a form of hazard prediction information based on a direction of movement in which the object is predicted to move. [11] Information processing device according to claim 10, wherein the presentation control unit displays several of the elements of hazard prediction information in a state in which they are superimposed on the object or the object's environment, wherein the several elements of the hazard prediction information are several of the elements of the hazard prediction information, each corresponding to several of the directions of movement in which the object is predicted to move. [12] Information processing device according to claim 11, wherein The calculation unit calculates the degree of danger of the object for each of the object's directions of movement and The presentation control unit controls the color of each of the multiple elements of hazard prediction information based on the hazard level of the object, which is calculated for each of the directions of movement. [13] Information processing device according to claim 8, wherein the presentation control unit controls a form of hazard prediction information based on an area in which the object is predicted to move. [14] Information processing device according to claim 8, wherein the presentation control unit controls an angle formed by one side of the hazard prediction information based on a predicted movement speed of the object. [15] Information processing device according to claim 8, wherein the presentation control unit controls a form of hazard prediction information based on the hazard level of the object. [16] Information processing device according to claim 1, wherein the presentation control unit provides the user with hazard environment information indicating a hazard level of an environment of the moving body. [17] Information processing device according to claim 1, wherein the prediction unit predicts the motion information and the accident information based on sensor data from a sensor provided on the moving body and which detects the environment of the moving body. [18] Information processing device according to claim 1, wherein the prediction unit predicts the motion information and the accident information based on information relating to the object that is acquired by devices other than the moving body. [19] Information processing procedure carried out by an information processing facility, wherein the information processing procedure comprises: Predictions of motion information, indicating a future movement of an object in the vicinity of a moving body, and accident information, indicating a movement of the object in a case where an accident occurs; Calculating the hazard level of the object based on movement information and accident information; and Presenting, for a user of the moving body, hazard prediction information that indicates the level of danger of the object. [20] Program to cause a computer to perform processing which includes: Predictions of motion information, indicating a future movement of an object in the vicinity of a moving body, and accident information, indicating a movement of the object in a case where an accident occurs; Calculating the hazard level of the object based on movement information and accident information; and Presenting, for a user of the moving body, hazard prediction information that indicates the level of danger of the object.

Images