Information processing device, information processing method, and program

Abstract

The present technology pertains to an information processing device, an information processing method, and a program that make it possible to more appropriately present information for avoiding danger to a user of a moving body such as a vehicle. An information processing device according to the present technology comprises: a prediction unit that predicts movement information indicating future movement of an object around a moving body; a calculation unit that calculates a degree of risk of the object based on the movement information; and a presentation control unit that controls a notification sound that is presented to a user of the moving body and corresponds to the type of the object, based on the degree of risk. The present technology can be applied to, for example, a moving body such as a vehicle.

Description

TECHNICAL FIELD

[0001] The present technology relates to an information processing apparatus, an information processing method, and a program, and more particularly, to an information processing apparatus, an information processing method, and a program that enable more favorable presentation of information for avoiding danger to a user of a mobile body such as a vehicle. BACKGROUND

[0002] In the past, a technology has been proposed that uses a sound image positioning technology to position a sound image of a warning sound such as a buzzer sound in a direction of an object existing around a vehicle, to cause a user of the vehicle to recognize the direction of the object (for example, see Patent Literature 1).

[0003] LIST OF CITATIONS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: WO2021 / 079975 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] In the conventional technology, even when a warning sound is heard, the user cannot know the type of the object existing around the vehicle. In addition, in the conventional technology, there is a possibility that the user will be distracted by the continuous warning sound.

[0008] The present technology was made in view of the above circumstances, and aims to enable more favorable presentation of information for avoiding danger to a user of a mobile body such as a vehicle.

[0009] SOLUTION TO PROBLEM

[0010] An information processing apparatus according to one aspect of the present technology includes a prediction unit that predicts movement information indicating future movement of an object around a mobile body, a calculation unit that calculates a degree of danger of the object based on the movement information, and a presentation control unit that controls presentation to a user of the mobile body of a notification sound corresponding to a type of the object based on the degree of danger.

[0011] An information processing method according to one aspect of the present technology is executed by an information processing apparatus, and includes predicting movement information indicating future movement of an object around a mobile body, calculating a degree of danger of the object based on the movement information, and controlling presentation to a user of the mobile body of a notification sound corresponding to a type of the object based on the degree of danger.

[0012] According to one aspect of the present technology, a computer performs a process comprising: predicting movement information indicating future movement of objects around a moving body; calculating the danger level of the objects based on the movement information; and controlling a notification sound presented to a user of the moving body and corresponding to the type of the object based on the danger level.

[0013] According to this aspect of the technology, motion information is predicted to indicate the future movement of objects around a moving body, the danger level of the objects is calculated based on the motion information, and a notification sound corresponding to the type of object is presented to the user of the moving body based on the danger level. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating an example configuration of a vehicle control system.

[0015] Figure 2 This is a diagram showing an example of the sensing area.

[0016] Figure 3 This is a block diagram illustrating a configuration example of a vehicle control system to which this technology is applied.

[0017] Figure 4 This is a flowchart illustrating the processes performed by the vehicle control system.

[0018] Figure 5 This is a diagram showing an example of speaker arrangement.

[0019] Figure 6 This is a diagram showing another example of speaker arrangement.

[0020] Figure 7 This is a diagram showing an example of an output notification sound.

[0021] Figure 8 This is a diagram illustrating an example of visual information.

[0022] Figure 9 This is another example of visual information.

[0023] Figure 10 This is a block diagram illustrating an example of a computer's hardware configuration. Detailed Implementation

[0024] The following sections will describe the modes used to implement this technology. The descriptions will be presented in the following order.

[0025] 1. Configuration example of vehicle control system

[0026] 2. Example

[0027] 3. Modify the example

[0028] <1. Example of Vehicle Control System Configuration>

[0029] Figure 1 This is a block diagram illustrating a configuration example of a vehicle control system 11, which is an example of a mobile body control system to which this technology is applied.

[0030] A vehicle control system 11 is installed in the vehicle 1 and performs processing related to the driving automation of the vehicle 1. The driving automation includes driving automation at levels 1 to 5, as well as remote assistance and / or remote driving of the vehicle 1 by a remote driver.

[0031] The vehicle control system 11 includes a vehicle control ECU (electronic control unit) 21, a communication unit 22, a map information accumulation unit 23, a location information acquisition unit 24, an external identification sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a storage unit 28, a driving automation control unit 29, a DMS (driver monitoring system) 30, an HMI (human-machine interface) 31, and a vehicle control unit 32.

[0032] The vehicle control ECU 21, communication unit 22, map information accumulation unit 23, location information acquisition unit 24, external identification sensor 25, in-vehicle sensor 26, vehicle sensor 27, storage unit 28, driving automation control unit 29, DMS 30, HMI 31, and vehicle control unit 32 are communicatively connected to each other via communication network 41. Communication network 41 includes, for example, compliant with CAN (Controller Area Network), LIN (Local Internet Connection Network), LAN (Local Area Network), and FlexRay. ® or Ethernet ® Vehicle communication networks, buses, etc., that are digital two-way communication standards. The communication network 41 can be selectively used depending on the type of data to be transmitted. For example, CAN can be used for data related to vehicle control, and Ethernet can be used for large-capacity data. Note that the individual units of the vehicle control system 11 can communicate directly via wireless communication (such as Near Field Communication (NFC) and Bluetooth) assuming relatively short distance communication, instead of via the communication network 41. ® (Use this to connect)

[0033] Note that in the following text, when the various units of the vehicle control system 11 communicate with each other via the communication network 41, the communication network 41 will not be described. For example, when the vehicle control ECU 21 and the communication unit 22 communicate with each other via the communication network 41, the communication between the vehicle control ECU 21 and the communication unit 22 will be simply described.

[0034] The vehicle control ECU 21 is implemented by various processors, such as a CPU (central processing unit) and / or an MPU (microprocessor unit). The vehicle control ECU 21 controls all or part of the functions of the vehicle control system 11.

[0035] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., to send and receive various types of data. At this time, the communication unit 22 can use a variety of communication technologies for communication.

[0036] The communication unit 22's ability to communicate with devices outside the vehicle will be described schematically. The communication unit 22 communicates with servers (hereinafter referred to as external servers) existing on external networks via a base station or access point using wireless communication technology. Examples of such wireless communication technologies include 5G (5th Generation Mobile Communication System), LTE (Long Term Evolution), and DSRC (Dedicated Short Range Communication). The external network with which the communication unit 22 communicates is, for example, the Internet, cloud networks, or corporate private networks. There are no particular limitations on the communication technology used by the communication unit 22 to communicate with external networks, as long as it is a wireless communication technology capable of two-way digital communication at a speed equal to or greater than a predetermined speed and over a distance equal to or greater than a predetermined distance.

[0037] Furthermore, for example, communication unit 22 can use P2P (peer-to-peer) technology to communicate with terminals located near the vehicle. Terminals located near the vehicle include, for example, terminals worn by mobile entities moving at relatively low speeds (such as pedestrians or bicycles), terminals installed in fixed locations in shops, etc., and / or MTC (machine-type communication) terminals. Additionally, communication unit 22 can also perform V2X communication. V2X communication refers to communication between a vehicle and another entity, examples of which include vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with roadside equipment, vehicle-to-home communication with a household, and vehicle-to-pedestrian communication with terminals carried or worn by pedestrians.

[0038] For example, communication unit 22 can receive programs from the outside (via air) for updating the software used to control the operation of vehicle control system 11. Additionally, communication unit 22 can receive map information, traffic information, and information about the surrounding environment of vehicle 1 from the outside. Furthermore, for example, communication unit 22 can transmit information related to vehicle 1 and information about the surrounding environment of vehicle 1 to the outside. Examples of vehicle-related information transmitted to the outside via communication unit 22 include data indicating the status of vehicle 1 and identification results obtained by identification unit 73. Furthermore, communication unit 22 performs communications supporting a vehicle emergency call system, an example of which is eCall.

[0039] For example, communication unit 22 receives information from a road traffic information communication system (VICS, Vehicle Information and Communication System) that uses radio wave beacons, optical beacons, FM multiplexed broadcasts, etc. ® Electromagnetic waves sent.

[0040] The communication unit 22 will be schematically described in relation to in-vehicle devices. The communication unit 22 can communicate with in-vehicle devices, for example, using wireless communication. For instance, the communication unit 22 can wirelessly communicate with in-vehicle devices using any wireless communication technology that enables bidirectional digital communication at a speed equal to or greater than a predetermined speed. Examples of wireless communication technologies include wireless LAN, Bluetooth, NFC, and WUSB (Wireless USB). The communication unit 22 is not limited to this, and it can also communicate with in-vehicle devices using wired communication. For example, the communication unit 22 can communicate with in-vehicle devices via a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with in-vehicle devices using any wired communication technology that enables bidirectional digital communication at a speed equal to or greater than a predetermined speed. Examples of wired communication technologies include USB (Universal Serial Bus) and HDMI (High-Definition Multimedia Interface). ® And MHL (Mobile High Definition Link).

[0041] Here, the in-vehicle device that communicates with the communication unit 22 refers to, for example, a device in the vehicle that is not connected to the communication network 41. Examples of in-vehicle devices include, for instance, mobile or wearable devices carried by users in the vehicle (such as the driver), information devices brought into the vehicle and temporarily installed there.

[0042] The map information accumulation unit 23 accumulates maps acquired from external sources and / or maps created by the vehicle 1. For example, the map information accumulation unit 23 accumulates three-dimensional high-precision maps, global maps with lower precision than high-precision maps but covering a wide area, etc.

[0043] High-precision maps include, for example, dynamic maps, point cloud maps, and vector maps. Dynamic maps, for example, include four layers: dynamic information, semi-dynamic information, semi-static information, and static information, and are provided to vehicles from external servers. Point cloud maps are maps that include point clouds (point cluster data). Vector maps, for example, are maps that associate traffic information such as the location of lanes and / or traffic lights with point cloud maps to adapt to driving automation.

[0044] Point cloud maps and vector maps can be provided, for example, from external servers, or can be created by vehicle 1 based on sensing results from cameras 51, radar 52, LiDAR 53, etc., to match the local map described later, and can be accumulated in map information accumulation unit 23. Furthermore, in the case of providing high-precision maps from external servers, for example, acquiring map data of several hundred square meters regarding the planned route to be traveled by vehicle 1 from an external server, in order to reduce communication overhead.

[0045] The location information acquisition unit 24 receives GNSS signals from GNSS (Global Navigation Satellite System) satellites to obtain the location information of vehicle 1. The acquired location information is provided to the driving automation control unit 29. Note that the location information acquisition unit 24 is not limited to using GNSS signals, and may also use beacons, for example, to obtain location information.

[0046] The external identification sensor 25 includes various sensors for identifying conditions outside the vehicle 1 and provides sensor data from the sensors to various units of the vehicle control system 11. The external identification sensor 25 may include any type and any number of sensors.

[0047] For example, the external identification sensor 25 may include a camera 51, a radar 52, a LiDAR (light detection and ranging / laser imaging detection and ranging) 53, and an ultrasonic sensor 54. The external identification sensor 25 is not limited to this, and may include one or more types of sensors selected from the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54. There is no particular limitation on the number of cameras 51, radar 52, LiDAR 53, and ultrasonic sensors 54, as long as they can be practically installed in the vehicle 1. Furthermore, the types of sensors included in the external identification sensor 25 are not limited to this example, and the external identification sensor 25 may include other types of sensors. An example of the sensing area of ​​each sensor included in the external identification sensor 25 will be described later.

[0048] Note that there are no particular limitations on the imaging method of camera 51. For example, cameras using various imaging methods capable of performing distance measurements (such as ToF (Time-of-Flight) cameras, stereo cameras, monocular cameras, and infrared cameras) can be applied to camera 51 as needed. Camera 51 is not limited to this and can be a camera used simply to acquire captured images regardless of distance measurement.

[0049] Additionally, for example, the external identification sensor 25 may include an environmental sensor for detecting the environment surrounding the vehicle 1. The environmental sensor is a sensor for detecting environmental conditions such as weather, meteorological phenomena, and brightness, and may include, for example, various sensors such as rain sensors, fog sensors, sunlight sensors, snow sensors, and illuminance sensors.

[0050] Additionally, for example, the external identification sensor 25 includes a microphone for detecting sounds around the vehicle 1, the location of sound sources, etc.

[0051] The in-vehicle sensors 26 include various sensors for detecting information inside the vehicle and provide sensor data from these sensors to various units of the vehicle control system 11. There are no particular limitations on the type and number of sensors included in the in-vehicle sensors 26, as long as the type and number allow for actual installation of the sensors in the vehicle 1.

[0052] For example, the in-vehicle sensor 26 may include one or more types of sensors selected from cameras, radar, seating sensors, steering wheel sensors, microphones, and biosensors. As a camera included in the in-vehicle sensor 26, for example, a camera using various imaging methods capable of performing distance measurements, such as a ToF camera, a stereo camera, a monocular camera, and an infrared camera, may be used. The cameras included in the in-vehicle sensor 26 are not limited to these, and the camera may be a camera used simply to acquire captured images regardless of distance measurement. The biosensors included in the in-vehicle sensor 26 are disposed in, for example, seats, steering wheels, etc., and detect various types of biometric information of the user.

[0053] Vehicle sensor 27 includes various sensors for detecting the state of vehicle 1, and provides sensor data from these sensors to various units of vehicle control system 11. There are no particular limitations on the type and number of sensors included in vehicle sensor 27, as long as the type and number allow for actual installation of the sensors in vehicle 1.

[0054] For example, vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyroscope sensor), and an inertial measurement unit (IMU) integrating these sensors. For example, vehicle sensor 27 includes a steering angle sensor for detecting the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor for detecting the amount of operation of the accelerator pedal, and a brake sensor for detecting the amount of operation of the brake pedal. For example, vehicle sensor 27 includes a rotation sensor for detecting the rotational speed of the engine or electric motor, a tire pressure sensor for detecting tire pressure, a slip rate sensor for detecting tire slip ratio, and a wheel speed sensor for detecting the rotational speed of the wheels. For example, vehicle sensor 27 includes a battery sensor for detecting the remaining battery level and temperature, and an impact sensor for detecting external impacts.

[0055] Storage unit 28 includes at least one of non-volatile or volatile storage media and stores data and programs. Storage unit 28 may function as, for example, EEPROM (Electrically Erasable Programmable Read-Only Memory) and RAM (Random Access Memory), and magnetic storage devices such as HDD (Hard Disk Drive), semiconductor storage devices, optical storage devices, and magneto-optical storage devices may be used as storage media. Storage unit 28 stores various programs and data used by the various units of the vehicle control system 11. For example, storage unit 28 includes EDR (Event Data Recorder) and / or DSSAD (Data Storage System for Automated Driving), and stores information about vehicle 1 before and after events such as accidents and / or information acquired by in-vehicle sensors 26.

[0056] The driving automation control unit 29 controls the driving automation functions of the vehicle 1. For example, the driving automation control unit 29 includes an analysis unit 61, a motion planning unit 62, and an operation control unit 63.

[0057] The analysis unit 61 performs analysis and processing of the condition of vehicle 1 and / or the condition of the area surrounding vehicle 1. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and an identification unit 73.

[0058] The self-position estimation unit 71 estimates the vehicle 1's own position based on sensor data from the external identification sensor 25 and the high-precision map accumulated in the map information accumulation unit 23. For example, the self-position estimation unit 71 estimates the vehicle 1's own position by generating a local map based on sensor data from the external identification sensor 25 and matching the local map with the high-precision map. For example, the center of the rear wheel axle is used as a reference for the vehicle 1's position.

[0059] For example, a local map is a high-precision 3D map created using techniques such as SLAM (Simultaneous Localization and Mapping) and occupancy grid mapping. A high-precision 3D map is, for example, the point cloud map described above. An occupancy grid map is a map that divides the 3D or 2D space around vehicle 1 into grids of a predetermined size and indicates the occupancy status of an object on a grid-by-grid basis. For example, the occupancy status of an object can be indicated by its presence, absence, or probability of presence. The local map is also used, for example, for detection and recognition processing of the external conditions of vehicle 1 by the recognition unit 73.

[0060] Note that the self-position estimation unit 71 can estimate the self-position of vehicle 1 based on the position information obtained by the position information acquisition unit 24 and the sensor data from the vehicle sensor 27.

[0061] The sensor fusion unit 72 performs sensor fusion processing, which combines multiple types of sensor data (e.g., image data from camera 51 and sensor data from radar 52) to obtain information. Methods for combining different types of sensor data include compositing, integration, fusion, correlation, etc.

[0062] The identification unit 73 performs detection processing of the external condition of the vehicle 1 and identification processing of the external condition of the vehicle 1.

[0063] For example, the identification unit 73 performs detection and identification processing of the external conditions of the vehicle 1 based on information from the external identification sensor 25, information from its own position estimation unit 71, and information from the sensor fusion unit 72.

[0064] Specifically, for example, the recognition unit 73 performs object detection processing, recognition processing, etc., around the vehicle 1. Object detection processing includes, for example, detecting the presence or absence of objects, their size, shape, position, movement (e.g., operation content, direction of movement, and speed of movement). Object recognition processing includes, for example, recognizing attributes such as the type of object and / or recognizing a specific object. Detection processing and recognition processing are not necessarily clearly separated and may be overlapping.

[0065] For example, the identification unit 73 detects objects around the vehicle 1 by performing clustering to classify point clouds based on sensor data from radar 52, LiDAR 53, etc., into clusters of point groups. Thus, the presence or absence, size, shape, and / or location of objects around the vehicle 1 are detected.

[0066] For example, the recognition unit 73 detects the movement of objects around vehicle 1 by tracking the movement of clusters of points classified by clustering. Therefore, it detects the speed and direction of travel (movement vector) of objects around vehicle 1.

[0067] For example, the recognition unit 73 detects or identifies vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road signs, etc., based on image data provided from the camera 51. Additionally, the recognition unit 73 identifies the types of objects surrounding the vehicle 1 by performing recognition processing such as semantic segmentation.

[0068] For example, the recognition unit 73 can perform traffic rule recognition processing around vehicle 1 based on the map accumulated in the map information accumulation unit 23, the estimation result of its own position by the self-position estimation unit 71, and the recognition result of objects around vehicle 1 by the recognition unit 73. Through this processing, the recognition unit 73 can recognize the position and status of traffic lights, the content of traffic signs and road signs, the content of traffic rules, drivable lanes, etc.

[0069] For example, the recognition unit 73 can perform recognition processing of the environment surrounding the vehicle 1. Assume that weather, temperature, humidity, brightness, road surface condition, etc. are considered as the surrounding environment to be recognized by the recognition unit 73.

[0070] Motion planning unit 62 creates a motion plan for vehicle 1. For example, motion planning unit 62 creates a motion plan by performing path planning and path following processes.

[0071] Note that path planning includes global path planning and local path planning. Global path planning involves planning a coarse path from the starting point to the destination. Local path planning, also known as trajectory planning, involves generating a trajectory near vehicle 1, which allows vehicle 1 to travel safely and smoothly along the planned path given its motion characteristics.

[0072] Path following is a process of planning operations to ensure safe and accurate travel on a path planned through path planning within a planning time. For example, the motion planning unit 62 can calculate the target speed and target angular velocity of vehicle 1 based on the results of this path following process.

[0073] In order to implement the motion plan created by the motion planning unit 62, the operation control unit 63 controls the operation of the vehicle 1.

[0074] For example, the operation control unit 63 controls the steering control unit 81, braking control unit 82, and drive control unit 83 included in the vehicle control unit 32, described later, to perform lateral vehicle motion control and longitudinal vehicle motion control, so that the vehicle 1 travels on a trajectory calculated through trajectory planning. For example, the operation control unit 63 performs controls for purposes such as driver assistance functions, including collision avoidance or impact mitigation, inter-vehicle distance control, vehicle speed control, vehicle collision warning, and lane departure warning; and driving automation for driving without a driver or with a remote driver.

[0075] Based on sensor data from in-vehicle sensor 26 and input data input to HMI 31 (described later), DMS 30 performs driver authentication processing and driver state recognition processing. Examples of driver states to be recognized include physical condition, alertness, concentration, fatigue, gaze direction, intoxication level, driving actions, posture, etc.

[0076] Note that DMS 30 can perform authentication processing for users other than the driver and user status recognition processing. Furthermore, for example, DMS 30 can perform vehicle interior condition recognition processing based on sensor data from in-vehicle sensor 26. Examples of vehicle interior conditions to be recognized include temperature, humidity, brightness, odor, etc.

[0077] HMI 31 receives various types of data, instructions, etc. as input and presents various types of data to the user.

[0078] The data input to HMI 31 will be described schematically. HMI 31 includes input devices used by humans to input data. HMI 31 generates input signals based on data, instructions, etc., input through the input devices and provides the input signals to the various units of vehicle control system 11. HMI 31 includes, for example, operating elements (such as touch panels, buttons, switches, and / or joysticks) as input devices. HMI 31 is not limited to this and may also include input devices that enable information to be input via methods other than manual operation (such as voice, gestures, etc.). In addition, HMI 31 may use, for example, remote control devices using infrared or radio waves and / or external connection devices suitable for the operation of vehicle control system 11 (such as mobile devices and wearable devices) as input devices.

[0079] The data presentation performed by HMI 31 will be illustrated schematically. HMI 31 generates visual, auditory, and tactile information for the user or for the vehicle's exterior. Furthermore, HMI 31 performs output control, controlling the output, content, timing, and method of each generated information. As visual information, HMI 31 generates and outputs, for example, operation screens, vehicle 1 status displays, warning displays, images such as surveillance images indicating the surroundings of vehicle 1, and information indicated by light. As auditory information, HMI 31 generates and outputs, for example, information indicated by sound, such as voice guidance, warning sounds, and warning messages. Furthermore, as tactile information, HMI 31 generates and outputs information that provides tactile sensations to the user, such as force, vibration, and movement.

[0080] As an output device for HMI 31 to output visual information, for example, a display device that presents visual information by displaying images on its own and / or a projector device that presents visual information by projecting images can be used. Note that in addition to display devices that include conventional displays, display devices can also be, for example, head-up displays, transmissive displays, or displays with AR (augmented reality) functionality. Alternatively, the display device can be a device that displays visual information in the user's field of vision, such as a wearable device with AR functionality. Wearable devices include glasses-type displays, smartphones, smartwatches, etc. In addition, display devices included in navigation devices, dashboards, CMS (camera monitoring system), electronic mirrors, lights, etc., installed in vehicle 1 can also be used as output devices for HMI 31 to output visual information. Furthermore, modified display devices such as dashboard monitors, tablet terminals, driving recorders, and displays for driving recorders can also be used as output devices for HMI 131 to output visual information.

[0081] The output device that outputs auditory information is located on the front surface of the dashboard in front of the driver's seat or passenger seat, in the console between the driver's seat and passenger seat, on the windshield, on the steering wheel, on the back of the driver's seat or passenger seat, etc.

[0082] As an output device to which the HMI 31 outputs auditory information, an audio speaker, headphones, and / or earphones can be used, for example.

[0083] As an output device for the HMI 31 to output tactile information, a tactile element using tactile technology can be applied, for example. The tactile element is located on, for example, a part that the user touches, such as a steering wheel or seat. Note that tactile information can also be output via a smartphone, smartwatch, or similar device carried by the user.

[0084] The vehicle control unit 32 controls various units of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0085] The steering control unit 81 performs functions such as detecting and controlling the state of the steering system of vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, electric power steering, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, and an actuator that drives the steering system, etc.

[0086] The brake control unit 82 performs functions such as detecting and controlling the status of the braking system of vehicle 1. The braking system includes, for example, a braking mechanism including a brake pedal, ABS (anti-lock braking system), regenerative braking mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the braking system, and actuators that drive the braking system, etc.

[0087] The drive control unit 83 performs functions such as detecting and controlling the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force for an internal combustion engine, a drive electric motor, etc., and a drive force transmission mechanism for transmitting drive force to the wheels. The drive control unit 83 includes, for example, a drive ECU for controlling the drive system and an actuator for driving the drive system.

[0088] The body system control unit 84 performs functions such as detecting and controlling the status of the vehicle 1's body systems. These body systems include, for example, keyless entry systems, smart key systems, power windows, power seats, air conditioning, airbags, seat belts, and gear shift levers. The body system control unit 84 includes, for example, a body system ECU that controls the body systems and actuators that drive the body systems.

[0089] The lighting control unit 85 performs functions such as detecting and controlling the status of various lights in vehicle 1. Examples of lights controlled by the lighting control unit 85 include headlights, taillights, fog lights, turn signals, brake lights, projector lights, bumper displays, etc. The lighting control unit 85 includes a lighting ECU for controlling the lights and actuators for driving the lights.

[0090] The horn control unit 86 performs functions such as detecting and controlling the status of the vehicle horn of vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the vehicle horn and an actuator that drives the vehicle horn.

[0091] Figure 2 It is shown Figure 1 A diagram illustrating an example of the sensing area of ​​the external identification sensor 25, including camera 51, radar 52, LiDAR 53, ultrasonic sensor 54, etc. Note that... Figure 2 The vehicle 1 is schematically shown as seen from above, with the left side being the front (front) side of the vehicle 1 and the right side being the rear (rear) side of the vehicle 1.

[0092] Sensing areas 101F and 101B are examples of the sensing areas of the ultrasonic sensor 54. Sensing area 101F covers the periphery of the front end of the vehicle 1 via multiple ultrasonic sensors 54. Sensing area 101B covers the periphery of the rear end of the vehicle 1 via multiple ultrasonic sensors 54.

[0093] The sensing results in sensing areas 101F and 101B are used for, for example, parking assistance for vehicle 1.

[0094] Sensing areas 102F, 102B, 102L, and 102R indicate examples of the sensing areas of radar 52 for short or medium range. Sensing area 102F covers a greater distance in front of vehicle 1 than sensing area 101F. Sensing area 102B covers a greater distance behind vehicle 1 than sensing area 101B. Sensing area 102L covers the rear perimeter of the left side of vehicle 1. Sensing area 102R covers the rear perimeter of the right side of vehicle 1.

[0095] The sensing results in sensing area 102F are used for, for example, detecting vehicles, pedestrians, etc., present in front of vehicle 1. The sensing results in sensing area 102B are used for, for example, collision prevention functions at the rear of vehicle 1. The sensing results in sensing areas 102L and 102R are used for, for example, detecting objects in the blind spot on the side of vehicle 1.

[0096] Sensing areas 103F, 103B, 103L, and 103R indicate examples of the sensing areas of camera 51. Sensing area 103F covers a greater distance in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a greater distance behind vehicle 1 than sensing area 102B. Sensing area 103L covers the left perimeter of vehicle 1. Sensing area 103R covers the right perimeter of vehicle 1.

[0097] The sensing results in sensing area 103F can be used, for example, for traffic light and / or traffic sign recognition, lane departure prevention assist systems, and / or automatic headlight control systems. The sensing results in sensing area 103B can be used, for example, for parking assist and / or surround view systems. The sensing results in sensing areas 103L and 103R can be used, for example, for surround view systems.

[0098] Sensing area 104 indicates an example of the sensing area of ​​LiDAR 53. Sensing area 104 covers a greater distance in front of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.

[0099] The sensing results in sensing area 104 are used, for example, to detect objects such as surrounding vehicles.

[0100] Sensing area 105 indicates an example of the sensing area of ​​radar 52 used for long-range applications. Sensing area 105 covers a greater distance in front of vehicle 1 than sensing area 104. On the other hand, sensing area 105 has a narrower range in the left-right direction than sensing area 104.

[0101] The sensing results in sensing area 105 are used for applications such as ACC (Adaptive Cruise Control), emergency braking, and collision avoidance.

[0102] Note that the sensing area of ​​the sensors, including the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54 included in the external identification sensor 25, can be taken in addition to... Figure 2 Various configurations beyond those shown are also possible. Specifically, ultrasonic sensor 54 can also sense the sides of vehicle 1, and LiDAR 53 can sense the rear of vehicle 1. The sensor mounting locations are not limited to the examples described above. Furthermore, the number of sensors can be one or more.

[0103] 2. Example

[0104] Next, we will refer to Figures 3 to 7 Description of embodiments of this technology.

[0105] Figure 3 This is a block diagram illustrating a configuration example of a vehicle control system 11 to which this technology is applied.

[0106] Figure 3 The vehicle control system 11 shown includes the above-described configuration (external identification sensor 25 and identification unit 73), an information processing unit 201 that calculates the hazard level of objects around the vehicle 1 and generates a notification sound indicating the hazard level of the objects, a sound material database 202, and a sound presentation unit 203 that presents the notification sound. Note that... Figure 3 The configuration of the portion of the vehicle control system 11 related to the presentation of notification sounds is shown.

[0107] The external identification sensor 25 includes multiple microphones arranged to surround the vehicle 1 at its outer edge, such as a camera 51, radar 52, LiDAR 53, ultrasonic sensor 54, etc. Note that the external identification sensor 25 may include a microphone array comprising multiple microphones. The external identification sensor 25 can use the multiple microphones to collect sound while separating sound according to the direction of arrival of sound around the vehicle 1.

[0108] As described above, the recognition unit 73 acquires sensor data from the external recognition sensor 25 and performs object recognition processing based on the sensor data using AI (artificial intelligence) learned through deep learning or other machine learning, neural networks, etc. The recognition unit 73 identifies, for example, the type, attributes, state, position, orientation, speed, and size of objects around the vehicle 1 through the recognition processing.

[0109] The object type indicates whether the object is a pedestrian, vehicle, ball, guardrail, road sign, obstacle, etc. The recognition unit 73 not only recognizes moving objects such as pedestrians, vehicles, and balls, but also recognizes stationary objects such as guardrails, signs, and obstacles.

[0110] When the object type is a pedestrian, the object's attributes indicate whether the pedestrian is a child, elderly person, etc. When the object type is a vehicle, the object's attributes indicate the vehicle's type, color, etc. The object's state indicates the state of the pedestrian walking while looking at a smartphone, the state of the vehicle's brake lights being illuminated, etc.

[0111] The identification unit 73 provides the identification results of objects around the vehicle 1 to the information processing unit 201. Note that the identification unit 73 can obtain information about objects around the vehicle 1 from devices other than the vehicle 1 via the communication unit 22, and perform identification processing based on that information.

[0112] The information processing unit 201 is part of the functionality of the HMI 31. The information processing unit 201 includes a prediction unit 211, a hazard calculation unit 212, and a sound generation unit 213.

[0113] Based on the object recognition results obtained by the recognition unit 73, the prediction unit 211 predicts the future movement of objects around the vehicle 1 and the movement of objects in the event of an accident.

[0114] Specifically, prediction unit 211 predicts the range an object will reach within a few seconds in normal time as the object's future movement. This prediction is essentially based on the object's position, orientation, and velocity.

[0115] Note that alternatively, the range an object will reach in a few seconds within a normal timeframe can be predicted based on its movement over a time series. For example, the range an object will reach in a few seconds within a normal timeframe can be predicted based on the amount of movement it has made during the most recent predetermined time period. Alternatively, the state of the object (such as a pedestrian walking while looking at a smartphone, or a vehicle with its brake lights on) can be considered to predict the range an object will reach in a few seconds within a normal timeframe.

[0116] For example, prediction unit 211 predicts the probability of an accident and the movement of the object in the event of an accident (the range the object will reach in a few seconds) based on the object's properties (such as "there is a high probability that a child will suddenly run into the road" or "there is a high probability that an elderly person will fall while crossing the road"). Prediction unit 211 may also consider the position of stationary objects (such as "it is unlikely that a child will suddenly run into the road because of the guardrail") to predict the probability of an accident and the movement of the object in the event of an accident.

[0117] The prediction unit 211 provides movement information indicating the future movement of an object, and accident information indicating the movement of the object in the event of an accident, to the hazard calculation unit 212 and the sound generation unit 213. The accident information includes information indicating the likelihood of an accident occurring. Alternatively, the accident information may include prediction difficulty, which is the degree of difficulty in predicting the future movement of an object.

[0118] The hazard calculation unit 212 calculates the hazard level of an object based on the object identification results obtained by the identification unit 73 and the movement and accident information provided by the prediction unit 211. For example, based on the range that the object is predicted to reach within a few seconds and the direction and speed of the vehicle 1, the probability of a collision between the object and the vehicle 1 is estimated as the hazard level.

[0119] The danger calculation unit 212 provides the danger level of the object to the sound generation unit 213.

[0120] The sound generation unit 213 generates audio data for a notification sound based on the movement and accident information provided by the prediction unit 211 and the danger level of the object provided by the danger level calculation unit 212.

[0121] Since the position of the object is identified by the recognition unit 73, the sound generation unit 213 essentially acquires the sound from the position of the object (the direction in which the object is seen from the vehicle 1) from the external recognition sensor 25. When the object is a car, for example, the engine sound is acquired; when the object is a pedestrian, for example, the footstep sound is acquired; and when the object is a bicycle, for example, the chain sound is acquired.

[0122] The sound generation unit 213 causes the sound presentation unit 203 to output the sound coming from the location of the object (actually the sound emitted by the object) as a notification sound.

[0123] When the object is an electric vehicle or a pedestrian, these sounds are not captured in some cases because the volume of the engine sound or footsteps emitted by the electric vehicle or pedestrian is low. Furthermore, when the object is a ball, almost no sound is emitted from the ball. Therefore, for example, in some cases, the sound of the ball rolling is not captured. When no sounds are captured from objects around vehicle 1, the sound generation unit 213 generates audio data of the type of sound that evokes the object based on pre-provided audio data of sounds obtained from the sound material database 202.

[0124] The generated audio data includes sounds such as "honking" (cause of a car horn), "roaring" (cause of a motorcycle engine), "jingling" (cause of a bicycle bell), "walking" (cause of a pedestrian), and "rolling" (cause of a ball). For example, the sound of a car could be obtained by pre-recording sounds emitted by a car, pre-recording sounds emitted by other objects, or it could be an electronic sound.

[0125] The sound generation unit 213 causes the sound presentation unit 203 to output a sound of the type that evokes an object as a notification sound. In this way, the sound generation unit 213 functions as a presentation control unit that controls the sound presentation unit 203 to present the notification sound to the user of the vehicle 1.

[0126] Based on the degree of danger of an object, the sound generation unit 213 controls the volume of the notification sound used to notify the object. For example, the sound generation unit 213 sets the volume of the notification sound used to notify an object with a danger level greater than a first threshold to high, and sets the volume of the notification sound used to notify an object with a danger level less than the first threshold to low.

[0127] Note that when the danger level is greater than the second threshold (second threshold > first threshold), warning sounds such as beeping sounds can be combined with sounds that are actually emitted from the object or sounds that evoke the object, and the combined sound can be output as a notification sound.

[0128] Furthermore, as will be described later, the sound generation unit 213 controls the position of the sound image (virtual sound source) of the notification sound based on the position of the object.

[0129] Audio data of various types of sounds that evoke objects are pre-recorded in the sound material database 202.

[0130] The sound presentation unit 203 includes, for example, multiple speakers installed in a vehicle. Under the control of the sound generation unit 213, the sound presentation unit 203 outputs a notification sound to present the notification sound to the user of the vehicle 1.

[0131] All or some of the functions of the identification unit 73 and the information processing unit 201 can be set in a device other than the vehicle 1, such as the cloud.

[0132] Next, refer to Figure 4 The flowchart shown illustrates the processes performed by the vehicle control system 11 with the above configuration. For example, when an operation to start the vehicle 1 to begin driving is performed, that is, when the ignition switch, power switch, starter switch, etc. of the vehicle 1 are turned on, the process begins... Figure 4The process is shown in the diagram. Furthermore, for example, when an operation to end driving of vehicle 1 is performed, i.e., when the ignition switch, power switch, starter switch, etc., of vehicle 1 are turned off, the process ends. Figure 4 The processing shown.

[0133] In step S1, the external identification sensor 25 senses the conditions around the vehicle 1.

[0134] In step S2, the identification unit 73 identifies objects around the vehicle 1 based on sensor data from the external identification sensor.

[0135] In step S3, the information processing unit 201 determines whether the recognition unit 73 has recognized the object.

[0136] If no object is identified in step S3, the process returns to step S1 and the conditions around vehicle 1 are sensed until an object is identified.

[0137] On the other hand, when it is determined in step S3 that an object has been identified, the process proceeds to step S4. In step S4, the prediction unit 211 predicts the future movement of the object identified by the recognition unit 73.

[0138] In step S5, the hazard calculation unit 212 calculates the hazard of the object identified by the identification unit 73 based on the identification result obtained by the identification unit 73 and the prediction result obtained by the prediction unit 211.

[0139] In step S6, the sound generation unit 213 generates audio data for a notification sound based on the degree of danger.

[0140] In step S7, the sound presentation unit 203, under the control of the sound generation unit 213, presents a notification sound to the user of vehicle 1. After presenting the notification sound in step S7, the process returns to step S1 and performs subsequent processing.

[0141] Figure 5 This is a diagram showing an example of speaker arrangement.

[0142] For example, the sound presentation unit 203 includes eight speakers 203F to 203B. Note that in Figure 5 In the diagram, the positions of speakers 203F to 203B are schematically indicated by dashed circles.

[0143] Speakers 203F to 203B are arranged inside vehicle 1 to surround the driver D1, who is sitting in the driver's seat.

[0144] Specifically, speaker 203FL is positioned to the left front of driver D1, near the front of the passenger seat door of vehicle 1, and speaker 203FR is positioned to the right front of driver D1, near the front of the driver's seat door of vehicle 1. Speaker 203F is positioned near the midpoint between speaker 203FL and speaker 203FR, and also near the center of the dashboard of vehicle 1.

[0145] Speaker 203BL is located to the left rear of driver D1, slightly forward of the center of the left rear door of vehicle 1, and speaker 203BR is located to the right rear of driver D1, slightly forward of the center of the right rear door of vehicle 1.

[0146] Speaker 203L is located near the midpoint between speaker 203FL and speaker 203BL, and also near the rear end of the passenger seat door of vehicle 1. Speaker 203R is located near the midpoint between speaker 203FR and speaker 203BR, and also near the rear end of the driver's seat door of vehicle 1.

[0147] Speaker 203B is positioned near the midpoint between speakers 203BL and 203BR, and also near the center of the rear seat. Speakers 203F and 203B, as well as speakers 203L and 203R, are arranged opposite each other. Speakers 203F to 203B are all mounted facing inward toward the interior of vehicle 1.

[0148] Speakers 203F to 203B are used, for example, to output sound that is completely directed into the interior of the vehicle (to all passengers in the vehicle). Speakers 203F to 203B achieve 360° true audio. ® (Hereinafter referred to as 360RA) and stereo sound.

[0149] Here, 360RA is a technology in which sound images (virtual sound sources) are individually arranged at arbitrary locations in a spherical space, and the sound output is controlled so that the listener perceives the sound as emanating from various directions of the sound images. For example, by using 360RA to present notification sounds, passengers can identify the location of objects around vehicle 1 (the direction of the objects as seen from vehicle 1).

[0150] Note that the number and location of the speakers arranged in vehicle 1 are not limited to those referenced. Figure 5 The quantity and location are described. For example, stereo sound can be achieved using stereo speakers or nine or fewer surround speakers. This technology can also be applied to systems capable of positioning the sound image at any location within a predetermined range, rather than in all directions. Stereo sound can be achieved with a single speaker, or this technology can be applied to vehicles equipped with a single speaker that does not support stereo sound.

[0151] Figure 6 This is a diagram showing another example of speaker arrangement.

[0152] like Figure 6 As shown, multiple speakers can be arranged at different heights. Note that... Figure 6 This is a schematic perspective view of the interior of vehicle 1 as seen from the left. Similarly, in the figure, with... Figure 5 The corresponding parts are represented by the same reference numerals.

[0153] exist Figure 6 In the example, with Figure 5 Compared to the example, the following speakers have been added: speaker 203FLU (not shown), speaker 203FRU, speaker 203FLD (not shown), speaker 203FRD, speaker 203BLD (not shown), and speaker 203BRD.

[0154] Speaker 203FRU is positioned near the upper right end of the windshield of vehicle 1, at a position higher than that of speaker 203FR, and speaker 203FRD is positioned below speaker 203FR, near the lower end of the driver's side door. Speaker 203BRD is positioned below speaker 203BR, near the lower end of the right rear door.

[0155] Note that although not shown in the figure, loudspeakers 203FLU, 203FLD, and 203BLD are arranged in positions substantially opposite to loudspeakers 203FRU, 203FRD, and 203BRD on the inner left side.

[0156] Speakers 203FLU, 203FRU, 203FLD, 203FRD, 203BLD, and 203BRD are all mounted facing the interior of vehicle 1.

[0157] For example, by outputting sound from speakers 203FRU and 203FR, the sound image can be positioned higher than vehicle 1 (or driver D1). Alternatively, by outputting sound from speakers 203FRU and 203FR, the sound image can be positioned lower than vehicle 1 (or driver D1).

[0158] Figure 7 This is a diagram showing an example of an output notification sound.

[0159] like Figure 7 As shown in A, when vehicle C1 is located to the right front of vehicle 1, a notification sound is output so that, for example, the sound emitted from vehicle 1 can be heard from the direction of vehicle 1 (to the right front).

[0160] In addition, such as Figure 7 As shown in B, when pedestrian P1 is in front of the left side of vehicle 1, a notification sound is output so that, for example, the sound emitted by pedestrian P1 can be heard from the direction of pedestrian P1 (front left direction).

[0161] In this manner, the sound generation unit 213 generates audio data to be provided to each speaker of the sound presentation unit 203, such that the acoustic image of the notification sound used to notify objects present around the vehicle 1 is positioned at a location corresponding to the direction of the corresponding object using the vehicle 1 as a reference.

[0162] Since the sound emitted by the object or the sound that evokes the type of object is output so that it can be heard from the direction of the object, the user of vehicle 1 can intuitively grasp what type of object is approaching vehicle 1 from which direction.

[0163] Note that the characteristics of the notification sound can be controlled based on the distance between vehicle 1 and the object. For example, when the object is located far from vehicle 1, the output notification sound may have its high-frequency components removed, or the pitch of the notification sound may be increased.

[0164] The position of the notification sound image in the vertical direction can be controlled based on the distance between vehicle 1 and the object. For example, when the object is located far from vehicle 1, the notification sound image is positioned higher than a reference position, and when the distance between vehicle 1 and the object is moderate, the notification sound image is positioned at the reference position. When the object is located close to vehicle 1, the notification sound image is positioned lower than the reference position.

[0165] The position of the notification sound image in the height direction can be controlled based on the degree of danger of the object.

[0166] The position of the notification sound image can be controlled based on the prediction of the object's future movement (movement information). For example, outputting a "whoosh" sound as the notification sound causes the position of the sound image to move along the predicted trajectory of the object's movement.

[0167] The characteristic quantities of the notification sound can be controlled based on the fluctuation of the danger level. For example, when the fluctuation of the danger level is large (when the accuracy of the prediction of the object's movement is low), reverberation processing or echo processing, which is used to reproduce the reverberation of sounds in caves, can be applied to the notification sound.

[0168] As described above, in this technology, motion information indicating the future movement of objects around vehicle 1 is predicted, the hazard level of the objects is calculated based on the motion information, and the characteristic quantities of a notification sound presented to the user of vehicle 1 and corresponding to the type of object are controlled based on the hazard level. The characteristic quantities of the notification sound include volume, frequency components, pitch, effects (e.g., reverberation or echo), etc.

[0169] The vehicle control system 11 does not merely warn of the presence of dangerous objects around the vehicle 1, but rather presents an audible notification, allowing the user of the vehicle 1 to understand the degree of danger, type, and direction of the object. Therefore, the vehicle control system 11 can advantageously present information to the user of the vehicle 1 to help avoid danger. This allows the user of the vehicle 1 to intuitively grasp potential hazards, and thus the vehicle control system 11 can ensure the safe driving of the vehicle 1.

[0170] <3. Modification Example>

[0171] • Example of presenting notification sounds only to the driver

[0172] The sound generation unit 213 can also use audio signal processing technology to generate audio data, so that the notification sound can only be heard by the driver of the vehicle 1. In this case, users who are not driving and are sitting in the passenger seat or rear seat cannot hear the notification sound, thus achieving a comfortable vehicle interior space for users who are not driving.

[0173] • Example of presenting a notification sound for notifying objects not visible from vehicle 1

[0174] When information about objects around vehicle 1 can be obtained from a device other than vehicle 1, a notification sound can be presented to notify objects that exist in locations not actually visible from vehicle 1. For example, at an intersection with poor visibility, when it is predicted that another vehicle will suddenly appear in front of vehicle 1, a sound emanating from that vehicle or a sound that evokes that vehicle can be output so that the sound can be heard from the direction of that vehicle.

[0175] Examples of noise cancellation

[0176] Depending on the driving environment of vehicle 1, users inside the vehicle may be able to hear external sounds. For example, when vehicle 1 is driving near a construction site, users inside the vehicle may be able to hear construction noise. In this case, the notification sound presented by the vehicle control system 11 will mix with the external sounds, making it difficult for users of vehicle 1 to hear the notification sound.

[0177] In this regard, a notification sound can be presented after noise cancellation is performed on the external sound to notify objects of a high degree of danger. Note that when noise cancellation is difficult to perform, the volume of the notification sound can be increased so that the notification sound has a higher volume than the external sound, and a higher volume notification sound can be presented.

[0178] • Examples of linking notification sound and visual information

[0179] The vehicle control system 11 can also present a notification sound along with visual information indicating the degree of danger of objects around the vehicle 1.

[0180] Figure 8 This is a diagram illustrating an example of visual information.

[0181] exist Figure 8 In the example shown, hazard prediction information A21 to A24, each indicating the degree of danger of objects around vehicle 1, is displayed as a status overlaid on a captured video showing the environment around vehicle 1.

[0182] For example, hazard prediction information A21 uses color to indicate the hazard level of a pedestrian P11 moving in the rear direction as seen from the user of vehicle 1, and uses shape to indicate the predicted range of movement of pedestrian P11. Hazard prediction information A22 uses color to indicate the hazard level of a pedestrian P12 crossing the crosswalk, and uses shape to indicate the predicted range of movement of pedestrian P12. Hazard prediction information A23 uses color to indicate the hazard level of a pedestrian P13 about to cross the crosswalk, and uses shape to indicate the predicted range of movement of pedestrian P13. Hazard prediction information A24 uses color to indicate the hazard level of a vehicle C11 stopped on the road.

[0183] Points Po1 to Po3 are displayed as superimposed on the heads of pedestrians P11 to P13, respectively.

[0184] For example, a display installed inside vehicle 1 shows captured video overlaid with hazard prediction information A21 to A24 and points Po to Po3. By presenting a notification sound along with the captured video and hazard prediction information displayed on the display, the user of vehicle 1 can easily determine the source of the notification sound.

[0185] Figure 9 This is another example of visual information.

[0186] exist Figure 9In the example shown, a CG (computer graphics) video is displayed, reproducing vehicle 1 itself and the objects and environment surrounding vehicle 1. The CG video includes a waveform W1, which indicates the direction in which the notification sound is heard and the volume (danger level) of the notification sound. Waveform W1 is a circular line around vehicle 1, and the portion of this line corresponding to the direction in which the notification sound is heard is displayed as a wavy line. The louder the notification sound, the larger the wave in the portion of waveform W1 corresponding to the direction in which the notification sound is heard.

[0187] For example, such as Figure 9 As shown, when the cyclist P21 is located to the left rear of the vehicle 1, a notification sound for notifying the cyclist P21 (bicycle) is presented at a high volume, so the wave in the portion of waveform W1 corresponding to the direction of the cyclist P21 using the vehicle 1 as a reference becomes larger.

[0188] In addition, such as Figure 9 As shown, when pedestrian P22 is in front of the right of vehicle 1, a notification sound for notifying pedestrian P22 is presented at a low volume, so the wave in waveform W1 corresponding to the direction of pedestrian P22 using vehicle 1 as a reference becomes smaller.

[0189] Note that in waveform W1, for example, orange is used to display the portion corresponding to the direction of each object using vehicle 1 as a reference, and other portions are displayed, for example, in cyan. The color of the portion in waveform W1 corresponding to the direction of each object using vehicle 1 as a reference can be determined based on the hazard level of each object.

[0190] In this way, by presenting the notification sound together with a CG video of waveform W1 that includes the direction in which the notification sound is heard and the volume (danger level) of the notification sound, the user of vehicle 1 can more easily grasp the source of the notification sound.

[0191] ·other

[0192] This technology can be applied to a variety of products. For example, it can be implemented as a device mounted on any type of mobile body, such as cars, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobile devices, airplanes, drones, boats, or robots.

[0193] About computers

[0194] The above series of processes can be performed by hardware or by software. When the series of processes are performed by software, the program configuring the software is installed on a medium containing the program in a computer, general-purpose personal computer, or similar hardware.

[0195] Figure 10This is a block diagram illustrating an example of a computer hardware configuration that performs the above-described series of processes via a program. The information processing unit 201 includes, for example, components having... Figure 10 The configuration shown is similar to that of a PC.

[0196] CPU 501, ROM (Read-Only Memory) 502 and RAM 503 are interconnected via bus 504.

[0197] Input / output interface 505 is also connected to bus 504. Input unit 506, including keyboard, mouse, etc., and output unit 507, including display, speaker, etc., are connected to input / output interface 505. Storage unit 508, including hard disk, non-volatile memory, etc., communication unit 509, including network interface, etc., and driver 510 for driving removable medium 511 are also connected to input / output interface 505.

[0198] In a computer configured as described above, for example, CPU 501 loads a program stored in storage unit 508 into RAM 503 via input / output interface 505 and bus 504 and executes the program to perform the series of processes described above.

[0199] For example, the program executed by CPU 501 may be recorded on removable medium 511 or provided via wired or wireless transmission media such as a local area network, the Internet, and digital broadcasting, so as to install the program in storage unit 508.

[0200] Note that a program to be executed by a computer can be a program in which processing is performed sequentially in the order described in this specification, or a program in which processing is performed in parallel or at timed intervals as required, such as when invoked.

[0201] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), and it is irrelevant whether all components are housed in the same housing. Therefore, multiple devices housed in separate housings and interconnected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0202] The effects described in this manual are merely examples and not limitations, and other effects may also be achieved.

[0203] Furthermore, embodiments of this technology are not limited to the above embodiments, and various modifications can be made without departing from the spirit of this technology.

[0204] For example, this technology can be configured using cloud computing, where a single function is shared so that multiple devices can work collaboratively over a network.

[0205] Furthermore, the steps described in the flowchart above can be performed by a single device or can be performed by multiple devices shared by the device.

[0206] In addition, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device or can be shared by multiple devices.

[0207] • Configuration combination examples

[0208] This technology can also be configured as follows.

[0209] (1) An information processing device, comprising:

[0210] A prediction unit that predicts motion information indicating the future movement of objects around the moving body;

[0211] A calculation unit that calculates the danger level of the object based on the movement information; and

[0212] A presentation control unit controls the presentation of a notification sound to the user of the moving object based on the level of danger, and the notification sound corresponds to the type of the object.

[0213] (2) The information processing apparatus according to (1), wherein,

[0214] The notification sound includes a sound emitted from the object.

[0215] (3) The information processing apparatus according to (1) or (2), wherein,

[0216] The notification sound includes a sound that evokes the type of the object.

[0217] (4) The information processing apparatus according to (3), wherein,

[0218] The presentation control unit generates a sound that evokes the type of the object based on pre-provided sounds.

[0219] (5) The information processing apparatus according to any one of (1) to (4), wherein,

[0220] The presentation control unit controls the volume of the notification sound based on the level of danger.

[0221] (6) The information processing apparatus according to (5), wherein,

[0222] The presentation control unit:

[0223] A notification sound corresponding to the type of object with a hazard level below a first threshold is presented at a volume lower than a reference volume.

[0224] A notification sound corresponding to the type of object with a danger level greater than a first threshold is presented at a volume higher than the reference volume.

[0225] (7) The information processing apparatus according to any one of (1) to (6), wherein,

[0226] When the danger level is greater than the second threshold, the presentation control unit presents the notification sound, including a warning sound.

[0227] (8) The information processing apparatus according to any one of (1) to (7), wherein,

[0228] The presentation control unit controls the position of the audio-visual image of the notification sound based on the position of the object.

[0229] (9) The information processing apparatus according to (8), wherein,

[0230] The presentation control unit positions the sound image at a location corresponding to the orientation of the object determined using the moving body as a reference.

[0231] (10) The information processing apparatus according to (9), wherein,

[0232] The presentation control unit also presents visual information indicating the degree of danger.

[0233] (11) The information processing apparatus according to (10), wherein,

[0234] The presentation control unit presents a CG video as the visual information, the CG video including a waveform indicating the position of the sound image of the notification sound and the volume of the notification sound, and showing the objects around the moving body.

[0235] (12) The information processing apparatus according to any one of (8) to (11), wherein,

[0236] The presentation control unit moves the position of the sound image along the trajectory predicted to be followed by the object's movement.

[0237] (13) The information processing apparatus according to any one of (8) to (12), wherein,

[0238] The presentation control unit controls the position of the sound image in the height direction based on the distance between the moving body and the object.

[0239] (14) The information processing apparatus according to any one of (1) to (13), wherein,

[0240] The presentation control unit only presents the notification sound to the driver of the moving vehicle.

[0241] (15) The information processing apparatus according to any one of (1) to (14), wherein,

[0242] The presentation control unit performs noise cancellation for external sounds.

[0243] (16) The information processing apparatus according to any one of (1) to (15), wherein,

[0244] The prediction unit also predicts accident information indicating the movement of the object in the event of an accident, and

[0245] The calculation unit calculates the degree of danger based on the movement information and the accident information.

[0246] (17) The information processing apparatus according to (16), wherein,

[0247] The presentation control unit controls the effect of the notification sound based on the fluctuation of the danger level.

[0248] (18) An information processing method executed by an information processing device, the information processing method comprising:

[0249] Predicting movement information that indicates the future movement of objects around a moving body;

[0250] The danger level of the object is calculated based on the movement information; and

[0251] The notification sound presented to the user of the moving object is controlled based on the level of danger and corresponds to the type of the object.

[0252] (19) A program for causing a computer to perform a process, said process comprising:

[0253] Predicting movement information that indicates the future movement of objects around a moving body;

[0254] The danger level of the object is calculated based on the movement information; and

[0255] The notification sound presented to the user of the moving object is controlled based on the level of danger and corresponds to the type of the object.

[0256] Reference tag list

[0257] 1 vehicle

[0258] 11 Vehicle Control System

[0259] 25 External identification sensors

[0260] 73 Identification Units

[0261] 201 Information Processing Unit

[0262] 202 Sound Material Database

[0263] 203 sound presentation units

[0264] 211 Prediction Unit

[0265] 212 Hazard Calculation Unit

[0266] 213 Sound Generation Unit

Claims

1. An information processing apparatus, comprising: A prediction unit that predicts motion information indicating the future movement of objects around the moving body; A calculation unit that calculates the danger level of the object based on the movement information; as well as A presentation control unit controls the presentation of a notification sound to the user of the moving object based on the level of danger, and the notification sound corresponds to the type of the object.

2. The information processing apparatus according to claim 1, wherein, The notification sound includes a sound emitted from the object.

3. The information processing apparatus according to claim 1, wherein, The notification sound includes a sound that evokes the type of the object.

4. The information processing apparatus according to claim 3, wherein, The presentation control unit generates a sound that evokes the type of the object based on pre-provided sounds.

5. The information processing apparatus according to claim 1, wherein, The presentation control unit controls the volume of the notification sound based on the level of danger.

6. The information processing apparatus according to claim 5, wherein, The presentation control unit: A notification sound corresponding to the type of object with a hazard level below a first threshold is presented at a volume lower than a reference volume. A notification sound corresponding to the type of object with a danger level greater than a first threshold is presented at a volume higher than the reference volume.

7. The information processing apparatus according to claim 1, wherein, When the danger level is greater than the second threshold, the presentation control unit presents the notification sound, including a warning sound.

8. The information processing apparatus according to claim 1, wherein, The presentation control unit controls the position of the audio-visual image of the notification sound based on the position of the object.

9. The information processing apparatus according to claim 8, wherein, The presentation control unit positions the sound image at a location corresponding to the orientation of the object determined using the moving body as a reference.

10. The information processing apparatus according to claim 9, wherein, The presentation control unit also presents visual information indicating the degree of danger.

11. The information processing apparatus according to claim 10, wherein, The presentation control unit presents a CG video as the visual information, the CG video including a waveform indicating the position of the sound image of the notification sound and the volume of the notification sound, and showing the objects around the moving body.

12. The information processing apparatus according to claim 8, wherein, The presentation control unit moves the position of the sound image along the trajectory predicted to be followed by the object's movement.

13. The information processing apparatus according to claim 8, wherein, The presentation control unit controls the position of the sound image in the height direction based on the distance between the moving body and the object.

14. The information processing apparatus according to claim 1, wherein, The presentation control unit only presents the notification sound to the driver of the moving vehicle.

15. The information processing apparatus according to claim 1, wherein, The presentation control unit performs noise cancellation for external sounds.

16. The information processing apparatus according to claim 1, wherein, The prediction unit also predicts accident information indicating the movement of the object in the event of an accident, and The calculation unit calculates the degree of danger based on the movement information and the accident information.

17. The information processing apparatus according to claim 16, wherein, The presentation control unit controls the effect of the notification sound based on the fluctuation of the danger level.

18. An information processing method executed by an information processing device, the information processing method comprising: Predicting movement information that indicates the future movement of objects around a moving body; The danger level of the object is calculated based on the movement information; as well as The notification sound presented to the user of the moving object is controlled based on the level of danger and corresponds to the type of the object.

19. A program for causing a computer to perform a process, said process comprising: Predicting movement information that indicates the future movement of objects around a moving body; The danger level of the object is calculated based on the movement information; as well as The notification sound presented to the user of the moving object is controlled based on the level of danger and corresponds to the type of the object.

Images