Information processing device, information processing method, program, mobile device, and information processing system

Abstract

This technology pertains to an information processing device, an information processing method, a program, a moving device, and an information processing system that enable improvement in the accuracy in recognizing the direction in which an object is present in the vicinity of the moving device, such as a vehicle. The information processing device is provided with: a recognition unit that performs a detection process for an object in the vicinity of the moving device; and a warning sound control unit that, on the basis of the position of the detected object, controls the position of a sound image and the feature quantity of a first warning sound indicating the presence of the object. This technology can be applied to a moving device, such as a vehicle, for example.

Description

[Technical Field]

[0001] The present technology relates to an information processing device, an information processing method, a program, a mobile device, and an information processing system, and in particular to an information processing device, an information processing method, a program, a mobile device, and an information processing system that are suitable for use when outputting a warning sound for an object around a mobile device. [Background technology]

[0002] Conventionally, a technology has been proposed that uses sound image localization control to localize the sound image of a warning sound in the direction of an object around the vehicle, thereby allowing the driver to recognize the direction in which the object is located (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-253721 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Patent Document 1 that uses acoustic localization control to localize the sound image of a warning sound in the direction of an object present around the vehicle, it is desirable to improve the accuracy of recognizing the direction in which the object is present.

[0005] The present technology has been made in consideration of such circumstances, and aims to improve the accuracy of recognizing the direction in which an object exists around a mobile device such as a vehicle. [Means for solving the problem]

[0006] An information processing device according to a first aspect of the present technology includes a recognition unit that performs a process of detecting an object around a mobile device, and a warning sound control unit that controls, based on the position of the detected object, a feature value and a position of a sound image of a first warning sound that indicates the presence of the object.

[0007] An information processing method according to a first aspect of the present technology includes a step of detecting an object around a mobile device, and controlling the feature value and the position of a sound image of a warning sound indicating the presence of the object based on the position of the detected object.

[0008] A program according to a first aspect of the present technology causes a computer to execute a process including a step of detecting an object around a mobile device and controlling, based on the position of the detected object, the feature of a warning sound indicating the presence of the object and the position of a sound image of the warning sound.

[0009] In a first aspect of the present technology, a detection process for an object around a mobile device is performed, and based on the position of the detected object, the feature amount and the position of a sound image of a warning sound indicating the presence of the object are controlled.

[0010] A mobile device according to a second aspect of the present technology includes a recognition unit that performs a process of detecting surrounding objects, and a warning sound control unit that controls, based on the position of the detected object, the feature amount and the position of a sound image of a warning sound that indicates the presence of the object.

[0011] In a second aspect of the present technology, a process of detecting a surrounding object is performed, and based on the position of the detected object, a feature amount and a position of a sound image of a warning sound indicating the presence of the object are controlled.

[0012] An information processing system according to a third aspect of the present technology includes a sensor unit that senses the surroundings of a mobile device, a recognition unit that performs a process of detecting objects around the mobile device based on the sensing results of the sensor unit, a warning sound control unit that controls the feature values ​​and the position of a sound image of a warning sound indicating the presence of the object based on the position of the detected object, and a speaker that outputs the warning sound under the control of the warning sound control unit.

[0013] In a third aspect of the present technology, sensing of the surroundings of a mobile device is performed, and based on the sensing results, a detection process for objects around the mobile device is performed, and based on the position of the detected object, the features and position of the sound image of a warning sound indicating the presence of the object are controlled, and the warning sound is output. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a vehicle control system. [Figure 2] FIG. 1 is a diagram illustrating an example of installation positions of a camera, a LiDAR, a radar, and an ultrasonic sensor. [Figure 3] FIG. 2 is a schematic diagram of the interior of the vehicle viewed from the right. [Figure 4] FIG. 2 is a schematic view of the interior of the vehicle as seen from the front. [Figure 5] FIG. 2 is a block diagram showing a configuration related to a warning sound control unit. [Figure 6] 10 is a flowchart illustrating a warning sound control process. [Figure 7] FIG. 10 is a diagram illustrating an example of setting zones in a planar direction. [Figure 8] FIG. 10 is a diagram showing the directions from which warning sounds for objects in each zone can be heard. [Figure 9] 10A and 10B are diagrams for explaining a warning sound for a vehicle approaching from the rear left. [Figure 10] 10A and 10B are diagrams for explaining a warning sound for a vehicle approaching from the rear left. [Figure 11] 10A and 10B are diagrams for explaining a warning sound for a vehicle approaching from the rear left. [Figure 12] FIG. 10 is a diagram illustrating an example of setting zones in the height direction. [Figure 13] FIG. 10 is a diagram illustrating an example of setting zones in a planar direction. [Figure 14] FIG. 10 is a diagram illustrating an example of setting zones in a planar direction. [Figure 15] FIG. 10 is a diagram illustrating an example of setting zones in a planar direction. [Figure 16]10A and 10B are diagrams illustrating an example of controlling the feature amount of the warning sound based on the zone in which an object exists. [Figure 17] 10A and 10B are diagrams illustrating an example of controlling the feature amount of the warning sound based on the zone in which an object exists. [Figure 18] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present technology will be described in the following order. 1. Example of vehicle control system configuration 2. Embodiment 3. Variations 4.Other

[0016] <<1. Example of vehicle control system configuration>> FIG. 1 is a block diagram showing an example of the configuration of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.

[0017] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to driving assistance and automatic driving of the vehicle 1.

[0018] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, a driving assistance / autonomous driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0019] The vehicle control ECU 21, communication unit 22, map information storage unit 23, position information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, memory unit 28, cruise assist / autonomous driving control unit 29, driver monitoring system (DMS) 30, human-machine interface (HMI) 31, and vehicle control unit 32 are connected to each other so as to be able to communicate with each other via a communication network 41. The communication network 41 is configured, for example, by an in-vehicle communication network or bus conforming to a digital two-way communication standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). Different communication networks 41 may be used depending on the type of data being transmitted. For example, CAN may be applied to data related to vehicle control, and Ethernet may be applied to large-volume data. In addition, each part of the vehicle control system 11 may be directly connected without going through the communication network 41, using wireless communication intended for communication over relatively short distances, such as near field communication (NFC) or Bluetooth (registered trademark).

[0020] In the following description, when each unit of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described that the vehicle control ECU 21 and the communication unit 22 communicate with each other.

[0021] The vehicle control ECU 21 is configured with various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The vehicle control ECU 21 controls the entire or part of the functions of the vehicle control system 11.

[0022] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 22 can perform communication using a plurality of communication methods.

[0023] The following provides an overview of communication with the outside of the vehicle that can be performed by the communication unit 22. The communication unit 22 communicates with a server (hereinafter referred to as an external server) or the like present on an external network via a base station or an access point using a wireless communication method such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the Internet, a cloud network, or a network specific to a carrier. The communication method used by the communication unit 22 with the external network is not particularly limited as long as it is a wireless communication method that enables digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0024] Furthermore, for example, the communication unit 22 can communicate with a terminal located near the vehicle using P2P (Peer To Peer) technology. The terminal located near the vehicle can be, for example, a terminal worn by a mobile object moving at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed position in a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 22 can also perform V2X communication. V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit, or the like, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian, or the like.

[0025] The communication unit 22 can receive, for example, a program for updating software that controls the operation of the vehicle control system 11 from the outside (over the air). The communication unit 22 can also receive map information, traffic information, information about the surroundings of the vehicle 1, and the like from the outside. For example, the communication unit 22 can also transmit information about the vehicle 1 and information about the surroundings of the vehicle 1 to the outside. Information about the vehicle 1 that the communication unit 22 transmits to the outside includes, for example, data indicating the state of the vehicle 1, the recognition result by the recognition unit 73, and the like. Furthermore, for example, the communication unit 22 performs communication corresponding to a vehicle emergency notification system such as e-call.

[0026] For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System) (registered trademark)) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.

[0027] The following provides an overview of communication with the vehicle interior that can be performed by the communication unit 22. The communication unit 22 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 22 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wireless communication, such as wireless LAN, Bluetooth, NFC, or WUSB (Wireless USB). The communication unit 22 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 22 can communicate with each device in the vehicle using wired communication via a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wired communication, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), or MHL (Mobile High-Definition Link).

[0028] Here, the in-vehicle device refers to, for example, a device that is in the vehicle and is not connected to the communication network 41. Possible in-vehicle devices include, for example, mobile devices and wearable devices carried by passengers such as the driver, and information devices that are brought into the vehicle and temporarily installed.

[0029] The map information storage unit 23 stores one or both of a map acquired from an external source and a map created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, and the like.

[0030] Examples of high-precision maps include dynamic maps, point cloud maps, and vector maps. A dynamic map is a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and is provided to the vehicle 1 from an external server or the like. A point cloud map is a map made up of a point cloud (point group data). A vector map is a map that associates traffic information such as the positions of lanes and traffic lights with a point cloud map, and is adapted to ADAS (Advanced Driver Assistance System) and AD (Autonomous Driving).

[0031] The point cloud map and vector map may be provided, for example, from an external server or the like, or may be created by the vehicle 1 based on sensing results from the camera 51, radar 52, LiDAR 53, etc. as a map for matching with a local map (described later) and stored in the map information storage unit 23. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred square meters relating to the planned route along which the vehicle 1 will travel is acquired from the external server or the like in order to reduce communication capacity.

[0032] The location information acquisition unit 24 receives GNSS signals from GNSS (Global Navigation Satellite System) satellites and acquires location information of the vehicle 1. The acquired location information is supplied to the driving assistance / autonomous driving control unit 29. Note that the location information acquisition unit 24 is not limited to a method using GNSS signals, and may acquire location information using a beacon, for example.

[0033] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0034] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. Without being limited to this, the external recognition sensor 25 may be configured to include one or more types of sensors from the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of cameras 51, radar 52, LiDAR 53, and ultrasonic sensors 54 is not particularly limited as long as it is a number that can be realistically installed on the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0035] The imaging method of camera 51 is not particularly limited. For example, cameras of various imaging methods, such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, and an infrared camera, which are imaging methods capable of distance measurement, can be applied to camera 51 as needed. However, the invention is not limited to this, and camera 51 may simply be used to obtain a photographed image, regardless of distance measurement.

[0036] Furthermore, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment for the vehicle 1. The environmental sensor is a sensor for detecting the environment such as weather, climate, brightness, etc., and may include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0037] Furthermore, for example, the external recognition sensor 25 includes a microphone used to detect sounds around the vehicle 1 and the positions of sound sources.

[0038] The in-vehicle sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the in-vehicle sensor 26 are not particularly limited as long as they are of types and numbers that can be realistically installed in the vehicle 1.

[0039] For example, the interior sensor 26 may include one or more types of sensors selected from the group consisting of a camera, radar, a seating sensor, a steering wheel sensor, a microphone, and a biometric sensor. The camera included in the interior sensor 26 may be a camera using any of various imaging methods capable of measuring distances, such as a ToF camera, a stereo camera, a monocular camera, or an infrared camera. However, the camera included in the interior sensor 26 may simply be one that acquires captured images without measuring distances. The biometric sensor included in the interior sensor 26 may be provided, for example, on a seat, steering wheel, or the like, and detects various types of biometric information of a passenger such as a driver.

[0040] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the vehicle sensor 27 are not particularly limited as long as they are of types and numbers that can be realistically installed on the vehicle 1.

[0041] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these. For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining battery level and temperature, and an impact sensor that detects external impacts.

[0042] The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The storage unit 28 is used, for example, as an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and the storage medium may be a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage unit 28 stores various programs and data used by each component of the vehicle control system 11. For example, the storage unit 28 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and stores information about the vehicle 1 before and after an event such as an accident, and information acquired by the in-vehicle sensors 26.

[0043] The driving assistance / automatic driving control unit 29 controls driving assistance and automatic driving of the vehicle 1. For example, the driving assistance / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0044] The analysis unit 61 performs an analysis process of the vehicle 1 and the surrounding situation. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0045] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25 and estimates the self-position of the vehicle 1 by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.

[0046] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technology such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size and the occupancy state of an object is indicated on a grid-by-grid basis. The occupancy state of an object is indicated, for example, by the presence or absence of an object and its probability of existence. The local map is also used, for example, in the detection process and recognition process of the situation outside the vehicle 1 by the recognition unit 73.

[0047] The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and the sensor data from the vehicle sensor 27.

[0048] The sensor fusion unit 72 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and association.

[0049] The recognition unit 73 executes a detection process for detecting the situation outside the vehicle 1 and a recognition process for recognizing the situation outside the vehicle 1.

[0050] For example, the recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0051] Specifically, for example, the recognition unit 73 performs detection processing and recognition processing of objects around the vehicle 1. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0052] For example, the recognition unit 73 performs clustering to classify a point cloud based on sensor data from the radar 52, the LiDAR 53, or the like into clusters of points, thereby detecting objects around the vehicle 1. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.

[0053] For example, the recognition unit 73 performs tracking to follow the movement of clusters of point clouds classified by clustering, thereby detecting the movement of objects around the vehicle 1. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.

[0054] For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. based on image data supplied from the camera 51. The recognition unit 73 may also recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0055] For example, the recognition unit 73 can perform recognition processing of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the result of estimation of the self-position by the self-position estimation unit 71, and the result of recognition of objects around the vehicle 1 by the recognition unit 73. Through this processing, the recognition unit 73 can recognize the position and state of traffic signals, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes that can be traveled, etc.

[0056] For example, the recognition unit 73 can perform a recognition process of the environment around the vehicle 1. The surrounding environment to be recognized by the recognition unit 73 may include weather, temperature, humidity, brightness, and road surface conditions.

[0057] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route following processing.

[0058] Global path planning is a process for planning a rough route from the start to the goal. This route planning also includes a process for generating a trajectory (local path planning) that allows the vehicle 1 to proceed safely and smoothly in the vicinity of the vehicle 1, taking into account the motion characteristics of the vehicle 1 on the planned route.

[0059] Path following is a process of planning an operation for safely and accurately traveling along a route planned by a route plan within a planned time. The behavior planning unit 62 can, for example, calculate a target speed and a target angular velocity of the vehicle 1 based on the results of this path following process.

[0060] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0061] For example, the operation control unit 63 controls a steering control unit 81, a brake control unit 82, and a drive control unit 83 included in a vehicle control unit 32 described later, to perform acceleration / deceleration control and direction control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 performs cooperative control aimed at realizing ADAS functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, collision warning for the host vehicle, and lane departure warning for the host vehicle. For example, the operation control unit 63 performs cooperative control aimed at automatic driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0062] The DMS 30 performs processes such as authenticating the driver and recognizing the driver's state based on sensor data from the in-vehicle sensors 26 and input data input to the HMI 31 (described later). Possible driver states to be recognized include, for example, physical condition, level of alertness, level of concentration, level of fatigue, line of sight, level of intoxication, driving operation, and posture.

[0063] The DMS 30 may be configured to perform authentication processing for passengers other than the driver and recognition processing for the conditions of the passengers. Furthermore, for example, the DMS 30 may be configured to perform recognition processing for the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Possible conditions inside the vehicle to be recognized include, for example, temperature, humidity, brightness, and odor.

[0064] The HMI 31 inputs various data and instructions, and presents various data to the driver, etc.

[0065] The following provides an overview of data input via the HMI 31. The HMI 31 includes an input device for a person to input data. The HMI 31 generates input signals based on data, instructions, and the like input via the input device, and supplies the signals to each component of the vehicle control system 11. The HMI 31 includes, as input devices, controls such as a touch panel, buttons, switches, and levers. However, the HMI 31 may also include input devices that allow information to be input by voice, gestures, or other means other than manual operation. Furthermore, the HMI 31 may use, as input devices, externally connected devices such as a remote control device that uses infrared or radio waves, or a mobile or wearable device that is compatible with the operation of the vehicle control system 11.

[0066] The presentation of data by the HMI 31 will be briefly described. The HMI 31 generates visual information, auditory information, and tactile information for the occupant or the outside of the vehicle. The HMI 31 also performs output control, controlling the output, output content, output timing, output method, etc. of each piece of generated information. The HMI 31 generates and outputs, as visual information, information indicated by images or lights, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image showing the situation around the vehicle 1. The HMI 31 also generates and outputs, as auditory information, information indicated by sounds, such as voice guidance, warning sounds, and warning messages. The HMI 31 also generates and outputs, as tactile information, information imparted to the occupant's sense of touch by, for example, force, vibration, movement, etc.

[0067] Examples of output devices that the HMI 31 uses to output visual information include a display device that displays an image on its own to present visual information and a projector device that projects an image to present visual information. The display device may be a device that displays visual information within the passenger's field of vision, such as a head-up display, a see-through display, or a wearable device with an AR (Augmented Reality) function, in addition to a display device with a normal display. The HMI 31 may also use display devices provided in a navigation system, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, a lamp, or the like provided in the vehicle 1 as output devices that output visual information.

[0068] As an output device for the HMI 31 to output auditory information, for example, an audio speaker, headphones, or earphones can be applied.

[0069] For example, a haptic element using haptic technology can be applied as an output device for the HMI 31 to output tactile information. The haptic element is provided on a part of the vehicle 1 that an occupant touches, such as a steering wheel or a seat.

[0070] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0071] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0072] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative brake mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0073] The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 83 includes, for example, a drive ECU for controlling the drive system, and an actuator for driving the drive system.

[0074] The body system control unit 84 detects and controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a body system ECU that controls the body system systems, an actuator that drives the body system systems, etc.

[0075] The light control unit 85 detects and controls the states of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 85 includes a light ECU that controls the lights, an actuator that drives the lights, and the like.

[0076] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, and the like.

[0077] <<2. Embodiments>> Next, an embodiment of the present technology will be described with reference to FIGS.

[0078] <Configuration example of the external recognition sensor 25 and the in-vehicle sensor 26> 2 shows an example of the installation positions of the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54 provided in the external recognition sensor 25 of the vehicle 1, and the cameras provided in the interior sensor 26. In this example, the external recognition sensor 25 includes cameras 51FC1 to 51BC3, radars 52FC to 52BR, LiDARs 53F to 53B, and ultrasonic sensors 54FL1 to 54BR. The interior sensor 26 includes cameras 101L to 102.

[0079] The camera 51FC1 is provided near the center of the front end of the vehicle 1. The camera 51FC1 captures images ahead of the vehicle 1. For example, a fisheye camera is used as the camera 51FC1. The camera 51FC1 is used for, for example, ADAS and surround view. The surround view is a function that displays, for example, an image of the surroundings of the vehicle 1 or an overhead image of the surroundings of the vehicle 1 seen from above.

[0080] The camera 51FC2 is provided near the center of the front of the interior of the vehicle 1. The camera 51FC2 captures an image of the area in front of the vehicle 1 through the windshield. The camera 51FC2 is used in, for example, an ADAS. Images from the camera 51FC2 are recorded in, for example, a drive recorder.

[0081] The cameras 51FL and 51FR are arranged on the left and right at a predetermined distance apart in the front of the interior of the vehicle 1, forming a stereo camera. The cameras 51FL and 51FR capture images of the area in front of the vehicle 1 through the windshield. The cameras 51FL and 51FR are used, for example, in ADAS. The cameras 51FL and 51FR have a higher resolution than the camera 51FC2, for example.

[0082] The cameras 51SL1 to 51SL4 are provided on the left side of the vehicle 1 near the front end of the driver's seat door.

[0083] The camera 51SL1 captures an image diagonally forward to the left of the vehicle 1. The camera 51SL1 is used in, for example, an ADAS.

[0084] The camera 51SL2 captures images to the left of the vehicle 1. The camera 51SL2 is, for example, a fisheye camera with a wider angle than the cameras 51SL1, 51SL3, and 51SL4. The camera 51SL2 is used for, for example, ADAS and surround view.

[0085] The camera 51SL3 captures an image diagonally rearward to the left of the vehicle 1. For example, a camera with a higher resolution than the camera 51SL1 is used as the camera 51SL3. The camera 51SL3 is used, for example, in an ADAS.

[0086] Camera 51SL4 photographs the area diagonally rear to the left of vehicle 1. The optical axis of camera 51SL4 faces in a direction closer to the rear of vehicle 1 than the optical axis of camera 51SL3. In other words, the optical axis of camera 51SL3 faces in a direction closer to the left (lateral direction) of vehicle 1 than the optical axis of camera 51SL4. Camera 51SL4 is used, for example, in a CMS.

[0087] The cameras 51SR1 to 51SR4 are provided on the right side of the vehicle 1 near the front end of the passenger door.

[0088] The camera 51SR1 captures an image diagonally forward to the right of the vehicle 1. The camera 51SR1 is used in, for example, an ADAS.

[0089] The camera 51SR2 captures images to the right of the vehicle 1. For example, a fisheye camera with a wider angle than the cameras 51SR1, 51SR3, and 51SR4 is used as the camera 51SR2. The camera 51SR2 is used for, for example, ADAS and surround view.

[0090] The camera 51SR3 captures an image diagonally rearward to the right of the vehicle 1. For example, a camera with a higher resolution than the camera 51SR1 is used as the camera 51SR3. The camera 51SR3 is used, for example, in an ADAS.

[0091] Camera 51SR4 photographs the area diagonally rear to the right of vehicle 1. The optical axis of camera 51SR4 faces in a direction closer to the rear of vehicle 1 than the optical axis of camera 51SR3. In other words, the optical axis of camera 51SR3 faces in a direction closer to the right (lateral direction) of vehicle 1 than the optical axis of camera 51SR4. Camera 51SR4 is used, for example, in a CMS.

[0092] The camera 51BC1 is provided near the center of the rear end of the vehicle 1. The camera 51BC1 captures an image behind the vehicle 1. For example, a fisheye camera is used as the camera 51BC1. The camera 51BC1 is used for, for example, ADAS and surround view.

[0093] The camera 51BC2 and the camera 51BC3 are provided near the rear center of the interior of the vehicle 1. The camera 51BC2 and the camera 51BC3 capture images of the rear of the vehicle 1 through the rear window. The camera 51BC2 is used, for example, in a CMS. The camera 51BC3 is used, for example, in an ADAS. Images from the camera 51BC3 are recorded, for example, in a drive recorder.

[0094] The radar 52FC is provided near the center of the front end of the vehicle 1. The radar 52FC senses the area ahead of the vehicle 1.

[0095] The radar 52FL is provided near the left end of the front of the vehicle 1. The radar 52FL senses the area diagonally forward to the left of the vehicle 1.

[0096] The radar 52FR is provided near the right end of the front of the vehicle 1. The radar 52FR performs sensing in the diagonally forward right direction of the vehicle 1.

[0097] The radar 52BC is provided near the center of the rear end of the vehicle 1. The radar 52BC senses the area behind the vehicle 1.

[0098] The radar 52BL is provided near the left end of the rear end of the vehicle 1. The radar 52BL senses the area diagonally rearward and to the left of the vehicle 1.

[0099] The radar 52BR is provided near the right end of the rear end of the vehicle 1. The radar 52BR performs sensing of the area diagonally rear right of the vehicle 1.

[0100] The LiDAR 53F is provided near the center of the front end of the vehicle 1. The LiDAR 53F senses the area ahead of the vehicle 1.

[0101] The LiDAR 53L is provided in front of the left side of the vehicle 1. The LiDAR 53L performs sensing of the left direction of the vehicle 1.

[0102] The LiDAR 53R is provided in front of the right side of the vehicle 1. The LiDAR 53R performs sensing in the right direction of the vehicle 1.

[0103] The LiDAR 53B is provided near the center of the rear end of the vehicle 1. The LiDAR 53B performs sensing behind the vehicle 1.

[0104] The ultrasonic sensor 54FL1 is provided slightly to the left of the center of the front end of the vehicle 1. The ultrasonic sensor 54FL1 performs sensing in the front of the vehicle 1 slightly to the left.

[0105] The ultrasonic sensor 54FL2 is provided near the left end of the front of the vehicle 1. The ultrasonic sensor 54FL1 senses the area diagonally forward and to the left of the vehicle 1.

[0106] The ultrasonic sensor 54FR1 is provided slightly to the right of the center of the front end of the vehicle 1. The ultrasonic sensor 54FR1 performs sensing in the front of the vehicle 1 slightly to the right.

[0107] The ultrasonic sensor 54FR2 is provided near the right end of the front of the vehicle 1. The ultrasonic sensor 54FR1 senses the vehicle 1 in a diagonal direction to the right in front of the vehicle 1.

[0108] The ultrasonic sensor 54SL1 is provided on the front left side of the body of the vehicle 1. The ultrasonic sensor 54SL1 senses the area in front of the vehicle 1 in the left direction.

[0109] The ultrasonic sensor 54SL2 is provided at the rear of the left side of the body of the vehicle 1. The ultrasonic sensor 54SL2 is located at the rear of the vehicle 1 and performs sensing in the left direction.

[0110] The ultrasonic sensor 54SR1 is provided in front of the right side of the body of the vehicle 1. The ultrasonic sensor 54SR1 senses the area in front of the vehicle 1 in the right direction.

[0111] The ultrasonic sensor 54SR2 is provided at the rear of the right side of the body of the vehicle 1. The ultrasonic sensor 54SR2 is located at the rear of the vehicle 1 and performs sensing in the right direction.

[0112] The ultrasonic sensor 54BL is provided near the left end of the rear end of the vehicle 1. The ultrasonic sensor 54BL senses the area diagonally rearward and leftward of the vehicle 1.

[0113] The ultrasonic sensor 54BR is provided near the right end of the rear end of the vehicle 1. The ultrasonic sensor 54BR senses the vehicle 1 in a diagonal direction obliquely rearward to the right.

[0114] The camera 101L is installed in the vehicle diagonally above the right of the driver's seat. The camera 101L takes images of the area around the driver's seat. For example, a ToF camera is used as the camera 101L.

[0115] The camera 101R is installed in the vehicle diagonally above and to the left of the passenger seat. The camera 101R takes images of the area around the passenger seat. For example, a ToF camera is used as the camera 101R.

[0116] The camera 102 is installed on the dashboard inside the vehicle, slightly to the left of the center, and captures images of the area around the driver's seat.

[0117] <Speaker installation position> Next, examples of the installation positions of speakers inside the vehicle 1 will be described.

[0118] As shown in Fig. 3, a speaker 203FL is embedded in the door 202FL on the driver's seat 201FL side, slightly forward of the backrest of the driver's seat 201FL. Although not shown, a speaker 203FR is embedded in the door 202FR on the passenger's seat 201FR side, slightly forward of the backrest of the passenger's seat 201FR. A speaker 203BL is embedded in the door 202BL on the left side of the rear seat 201BL, slightly forward of the backrest of the rear seat 201BL. Although not shown, a speaker 203BR is embedded in the door 202BR on the right side of the rear seat 201BR, slightly forward of the backrest of the rear seat 201BR.

[0119] As shown in Fig. 4, a seat speaker 204FL is embedded below the headrest of the driver's seat 201FL. A seat speaker 204FR is embedded below the headrest of the passenger seat 201FR. A seat speaker 204BL is embedded in the left rear seat 201BL. A seat speaker 204BR is embedded in the right rear seat 201BR.

[0120] Hereinafter, when there is no need to distinguish between the driver's seat 201FL and the rear seat 201BR, they will simply be referred to as seats 201. Hereinafter, when there is no need to distinguish between the doors 202FL and 202BR, they will simply be referred to as doors 202. Hereinafter, when there is no need to distinguish between the speakers 203FL and 203BR, they will simply be referred to as speakers 203. Hereinafter, when there is no need to distinguish between the seat speakers 204FL and 204BR, they will simply be referred to as seat speakers 204.

[0121] The speakers 203 embedded in each door 202 are used, for example, to output sound to the entire vehicle interior (all passengers in the vehicle). Furthermore, each speaker 203 realizes 360 Reality Audio (registered trademark, hereinafter referred to as 360RA).

[0122] Here, 360RA is a technology that individually places sound images (virtual sound sources) of each sound at any position within a spherical space and controls the sound output so that the listener hears each sound as if it is coming from the direction of each sound image. By using 360RA, for example, passengers can enjoy videos, music, etc. with realistic sound in the car. It can also allow passengers to recognize the positions of objects around the vehicle 1.

[0123] On the other hand, the seat speaker 204 of each seat 201 is used, for example, to output private sounds mainly for the individual passenger sitting in each seat 201. In other words, the sounds output from each seat speaker 204 are individually controlled.

[0124] The shape of each seat 201 and the position of each seat speaker 204 are adjusted so that people of various heights (sitting heights) can clearly hear the sound from the seat speaker 204 of each seat 201.

[0125] Furthermore, this speaker placement is an example and can be changed. For example, the speaker may be placed on the dashboard at the front of the vehicle 1.

[0126] <Warning sound control section> FIG. 5 shows an example of a configuration related to a warning sound control unit 251, which is part of the functions of the HMI 31.

[0127] The warning sound control unit 251 controls the speaker 203 and the seat speaker 204 to control the output of the warning sound based on the recognition results by the recognition unit 73, the action plan created by the action planning unit 62, and the control status of each part of the vehicle 1 by the vehicle control unit 32.

[0128] Here, the warning sound includes, for example, a warning sound for notifying passengers of the vehicle 1 of a dangerous situation, and a warning sound for notifying objects (e.g., bicycles, pedestrians, etc.) around the vehicle 1 of a dangerous situation. The former warning sound is, for example, a warning sound that indicates the presence of an object or a dangerous location around the vehicle 1 and alerts passengers inside the vehicle. The latter warning sound is, for example, a warning sound that indicates the presence of the vehicle 1 to objects around the vehicle 1 and alerts the objects around the vehicle 1 to be careful.

[0129] As will be described later, the warning sound control unit 251 controls the position of the sound image (virtual sound source) of the warning sound and the feature quantities of the warning sound. Here, the feature quantities of the warning sound include, for example, volume (amplitude), pitch (frequency), tone (waveform, frequency components), rhythm, melody, and harmony.

[0130] <Warning sound control processing> Next, the warning sound control process executed by the vehicle 1 will be described with reference to the flowchart of FIG.

[0131] This process starts, for example, when an operation is performed to start the vehicle 1 and start driving, such as when an ignition switch, a power switch, or a start switch of the vehicle 1 is turned on. This process ends, for example, when an operation is performed to end driving of the vehicle 1, such as when an ignition switch, a power switch, or a start switch of the vehicle 1 is turned off.

[0132] In step S1, the recognition unit 73 performs a process of detecting objects around the vehicle 1. For example, as described above, the recognition unit 73 performs a process of detecting objects around the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc. In this way, the recognition unit 73 detects the positions of objects around the vehicle 1. Note that this object detection process may include the object recognition process described above.

[0133] Here, the recognition unit 73 may use curvature information and gradient information of the road on which the vehicle 1 is traveling to correct the detection result of the object position, thereby improving the detection accuracy.

[0134] For example, when the vehicle 1 is traveling on a curved road, the recognition unit 73 estimates the curvature of the road based on an input image from the camera 51. Alternatively, when the vehicle 1 is traveling on a curved road, the recognition unit 73 detects the actual curvature of the road based on a high-precision map such as a dynamic map. Then, the recognition unit 73 corrects the detection results of the positions of other vehicles traveling on the same curved road based on the curvature of the road.

[0135] For example, when the vehicle 1 is traveling on a slope, the recognition unit 73 estimates the gradient of the road based on sensor data from an acceleration sensor included in the vehicle sensor 27. Alternatively, when the vehicle 1 is traveling on a slope, the recognition unit 73 detects the actual gradient of the road based on a high-precision map such as a dynamic map. Then, the recognition unit 73 corrects the detection result of the pitch direction position of other vehicles traveling on the same slope based on the gradient of the road.

[0136] Furthermore, the recognition unit 73 detects the movement and speed of the object (for example, the relative speed with respect to the vehicle 1) as needed. Furthermore, the recognition unit 73 recognizes attributes of the object around the vehicle 1, such as the type, size, and shape, as needed.

[0137] In step S2, the recognition unit 73 determines whether or not an object has been detected based on the result of the processing in step S1. If it is determined that an object has not been detected, the processing returns to step S1.

[0138] Thereafter, the processes of steps S1 and S2 are repeatedly executed until it is determined in step S2 that an object has been detected.

[0139] On the other hand, if it is determined in step S2 that an object has been detected, the process proceeds to step S3.

[0140] In step S3, the recognition unit 73 determines whether or not an object exists in the zone around the vehicle 1.

[0141] Specifically, the recognition unit 73 sets a zone around the vehicle 1 to define the range in which the warning sound is to be output and the direction in which the warning sound is to be output.

[0142] An example of a method for setting zones is shown in Figure 7. In this example, a vehicle 1 is traveling on a road with at least three lanes on each side, and zones ZL1 to ZR4 are set around the vehicle 1.

[0143] In the following description, the axis pointing in the traveling direction of the vehicle 1 is referred to as the X axis, and the axis perpendicular to the X axis and pointing to the left of the vehicle 1 is referred to as the Y axis.

[0144] Zones ZL1 to ZR4 are rectangular areas of approximately the same size. Zones ZC1, ZC2, and ZC4 are located in the driving lane of vehicle 1. Zone ZC2 is located immediately behind vehicle 1. Zone ZC1 is set behind zone ZC2. Zone ZC4 is located immediately before vehicle 1.

[0145] Zones ZL1 to ZL4 are located in the lane to the left of the lane in which vehicle 1 is traveling. Zones ZR1 to ZR4 are located in the lane to the right of the lane in which vehicle 1 is traveling. Zones ZL1 and ZR1 are located in the same position as zone ZC1 in the X-axis direction. Zones ZL2 and ZR2 are located in the same position as zone ZC2 in the X-axis direction. Zones ZL3 and ZR3 are located in the same position as vehicle 1 in the X-axis direction. Zones ZL4 and ZR4 are located in the same position as zone ZC4 in the X-axis direction.

[0146] For example, if the detected object is located outside the range of zones ZL1 to ZR4, the recognition unit 73 determines that the object is not present in the zone, and the process returns to step S1.

[0147] Thereafter, the processes of steps S1 to S3 are repeatedly executed until it is determined in step S3 that an object is present in the zone around the vehicle 1.

[0148] On the other hand, in step S3, if the detected object is located within the range of any one of zones ZL1 to ZR4, the recognition unit 73 determines that the object is present in the zone around the vehicle 1, and the process proceeds to step S4.

[0149] In step S4, the vehicle 1 outputs a warning sound from the direction of the zone where the object is present. Specifically, the recognition unit 73 supplies the warning sound control unit 251 with information indicating the zone where the object is present, the type and size of the object, etc.

[0150] The warning sound control unit 251 generates warning sound data for outputting a warning sound for a detected object. The warning sound control unit 251 supplies the generated warning sound data to the speakers 203FL to 203BR, thereby causing the speakers 203FL to 203BR to output the warning sound.

[0151] At this time, the warning sound control unit 251 sets the position of the sound image (virtual sound source) of the warning sound so that the warning sound is heard from the direction of the zone where the object is detected at a reference position (for example, the driver's head) in the vehicle 1, as shown by the arrow in each zone in Fig. 8. This allows the driver to intuitively recognize the direction of the detected object.

[0152] 9 to 11 show an example of a method for controlling a warning sound for a vehicle 301 approaching from the rear left.

[0153] Fig. 9 shows the sound field A1 of the warning sound when the vehicle 301 is in zone ZL1, Fig. 10 shows the sound field A1 of the warning sound when the vehicle 301 is in zone ZL2, and Fig. 11 shows the sound field A1 of the warning sound when the vehicle 301 is in zone ZL3.

[0154] The apex of the conical sound field A1 indicates the position of the sound image (virtual sound source). The direction in which the cone spreads from the apex of the sound field A1 indicates the direction in which the sound field spreads.

[0155] In this way, the direction of sound field A1 is set to be approximately the same as the direction in which the head of the driver of vehicle 1 is viewed from vehicle 301. Furthermore, the sound image of the warning sound is set at a position a predetermined distance away from the driver's head in the direction of vehicle 301. Therefore, sound field A1 moves so that it faces in a direction approximately the same as the direction in which the head of the driver of vehicle 1 is viewed from vehicle 301. Furthermore, sound field A1 moves so that the distance between the sound image of the warning sound and the driver's head is approximately proportional to the distance between vehicle 301 and vehicle 1.

[0156] Therefore, the warning sound allows the driver to recognize the approach of the vehicle 301. Furthermore, the movement of the warning sound allows the driver to intuitively recognize the speed, distance, and direction of the vehicle 301.

[0157] Note that the zones may be divided not only in the horizontal direction but also in the height direction (vertical direction) as shown in Fig. 12. For example, Fig. 12 shows an example in which zone ZL1 is divided into zones ZL1L to ZL1H in the height direction. Then, the warning sound control unit 251 sets the height position of the sound image of the warning sound for the detected object based on the size, which is one of the attributes of the object detected in zone ZL1.

[0158] Here, the size of an object is represented by, for example, at least one of the width, depth, and height of the object.

[0159] For example, if the object is small, such as a motorbike, bicycle, or pedestrian, the position of the sound image of the warning sound is set so that the warning sound will be heard from the direction of zone ZL1L at the reference position. For example, if the object is medium-sized, such as a passenger car, the position of the sound image of the warning sound is set so that the warning sound will be heard from the direction of zone ZL1M at the reference position. For example, if the object is large, such as a large vehicle such as a truck or bus, the position of the sound image of the warning sound is set so that the warning sound will be heard from the direction of zone ZL1H at the reference position.

[0160] In this way, even if an object is detected within the same horizontal zone, the height of the warning sound image changes depending on the size of the object, allowing the driver to intuitively recognize not only the position but also the size of the detected object.

[0161] Furthermore, by recognizing the size of the object, the driver can predict the degree of danger of the object. Here, the degree of danger of the object refers to the degree of danger that the vehicle 1 will be harmed by the object, or the degree of danger that the vehicle 1 will harm the object. For example, the driver can predict that the higher the position of the sound image of the warning sound, the higher the danger that the vehicle 1 will be harmed by the object due to a collision or the like, and the lower the danger that the vehicle 1 will harm the object. Conversely, the driver can predict that the lower the position of the sound image of the warning sound, the lower the danger that the vehicle 1 will be harmed by the object due to a collision or the like, and the higher the danger that the vehicle 1 will harm the object.

[0162] Thereafter, the process returns to step S1, and the processes from step S1 onwards are executed.

[0163] In this way, the driver can hear the warning sound from the direction in which the object is detected, improving the accuracy of recognizing the direction in which the object is located. Furthermore, by moving the sound image of the warning sound not only horizontally but also vertically, the driver can recognize the size of the object and its degree of danger.

[0164] <<3. Modifications>> Hereinafter, modifications of the above-described embodiment of the present technology will be described.

[0165] <Zone Variations> The area around the vehicle 1 that the driver should pay attention to changes depending on the situation in which the vehicle 1 is placed. Therefore, the recognition unit 73 may change the zone settings based on the situation in which the vehicle 1 is placed. In other words, the recognition unit 73 may change the zone settings based on the area in which the driver should pay attention.

[0166] Here, changing the zone settings means changing the location, number, size, shape, etc. of the zones. Furthermore, the situation in which the vehicle 1 is placed includes, for example, the location where the vehicle 1 is traveling, the environment around the vehicle 1, the driving operation of the vehicle 1, the traveling state of the vehicle 1, etc.

[0167] For example, when vehicle 1 is about to change lanes, there is less need for the driver to pay attention to vehicles in the lane in which vehicle 1 is traveling. Therefore, when vehicle 1 is about to change lanes, recognition unit 73 may, for example, not set zones ZC1, ZC2, and ZC4 in the lane in which vehicle 1 is traveling, among the zones in FIG.

[0168] For example, when the vehicle 1 is traveling on a road with two lanes in each direction and the lane in which zones ZR1 to ZR4 in Fig. 7 are located is an oncoming lane, the recognition unit 73 may, for example, not set zones ZR1 to ZR4. Alternatively, when the driver needs to pay close attention to vehicles in the oncoming lane due to, for example, a narrow road or poor forward visibility, the recognition unit 73 may, for example, set zones ZR1 to ZR4.

[0169] For example, when the vehicle 1 is traveling in an urban area, the speed of the vehicle 1 and surrounding vehicles is relatively slow, and there are many pedestrians, so the driver needs to be more careful of people jumping out in front of the vehicle 1 than of people jumping out behind the vehicle 1. In response to this, the recognition unit 73 may set zones ZL11 to ZR14 around the vehicle 1, for example, as shown in FIG.

[0170] Zones ZL11 to ZR14 are rectangular areas of approximately the same size. Zones ZC11, ZC13, and ZC14 ​​are located in the driving lane of vehicle 1. Zone ZC11 is located immediately behind vehicle 1. Zone ZC13 ​​is located immediately before vehicle 1. Zone ZC14 ​​is located in front of zone ZC13.

[0171] Zones ZL11 to ZL14 are located in the lane to the left of the lane in which vehicle 1 is traveling. Zones ZR11 to ZR14 are located in the lane to the right of the lane in which vehicle 1 is traveling. Zones ZL11 and ZR11 are located in the same position as zone ZC11 in the X-axis direction. Zones ZL12 and ZR12 are located in the same position as vehicle 1 in the X-axis direction. Zones ZL13 and ZR13 are located in the same position as zone ZC13 ​​in the X-axis direction. Zones ZL14 and ZR14 are located in the same position as zone ZC14 ​​in the X-axis direction.

[0172] 13, the zone behind the vehicle 1 is narrower and the zone in front of the vehicle 1 is wider than the example in FIG.

[0173] For example, when the vehicle 1 is backing up, it is necessary to pay more attention to the area behind the vehicle 1. In response to this, the recognition unit 73 may set zones ZL21 to ZR25 around the vehicle 1, for example, as shown in FIG.

[0174] Zones ZL21 to ZR25 are rectangular areas of approximately the same size. Zones ZC21 to ZC23 and zone ZC25 are located in the driving lane of vehicle 1. Zone ZC23 is located immediately behind vehicle 1. Zone ZC22 is located behind zone ZC23. Zone ZC21 is located behind zone ZC22. Zone ZC25 is located immediately before vehicle 1.

[0175] Zones ZL21 to ZL25 are located in the lane to the left of the lane in which vehicle 1 is traveling. Zones ZR21 to ZR25 are located in the lane to the right of the lane in which vehicle 1 is traveling. Zones ZL21 and ZR21 are located in the same position as zone ZC21 in the X-axis direction. Zones ZL22 and ZR22 are located in the same position as zone ZC22 in the X-axis direction. Zones ZL23 and ZR23 are located in the same position as zone ZC23 in the X-axis direction. Zones ZL24 and ZR24 are located in the same position as vehicle 1 in the X-axis direction. Zones ZL25 and ZR25 are located in the same position as zone ZC25 in the X-axis direction.

[0176] 14, the zone behind the vehicle 1 is wider than in the example of FIG.

[0177] For example, when the vehicle 1 is parked in a parking lot, there is a high possibility that other vehicles will approach from the side of the vehicle 1 rather than from the front or rear of the vehicle, and it is therefore necessary to pay more attention to the sides of the vehicle 1. In response to this, the recognition unit 73 may set zones ZL31A to ZR32C around the vehicle 1, for example, as shown in FIG.

[0178] Zones ZL31A to ZR32C are rectangular areas of approximately the same size. Zone ZL31A is located to the left of vehicle 1. Zone ZL31B is located to the left of zone ZL31A. Zone ZL31C is located to the left of zone ZL31B. Zone ZR31A is located to the right of vehicle 1. Zone ZR31B is located to the right of zone ZR31A. Zone ZR31C is located to the left of zone ZR31B. Zone ZC32 is located immediately before vehicle 1. Zone ZL32A is located to the left of zone ZC32. Zone ZL32B is located to the left of zone ZL32A. Zone ZL32C is located to the left of zone ZL32B. Zone ZR32A is located to the right of zone ZC32. Zone ZR32B is located to the right of zone ZR32A, and zone ZR32C is located to the left of zone ZR32B.

[0179] Thus, in the example of FIG. 15, the zones are wider on the sides of the vehicle 1 than in the other examples.

[0180] As described above, the zone settings may be changed depending on the situation of the vehicle 1. This makes it possible to output warning sounds for objects around the vehicle 1 more appropriately depending on the situation.

[0181] In the above description, an example has been given in which the zones have the same size and shape, but the zones may have different sizes and shapes.

[0182] Furthermore, for example, the warning sound control unit 251 may set the position of the sound image of the warning sound simply based on the position of the detected object, without setting a zone.

[0183] <Modifications regarding warning sounds> For example, the recognition unit 73 may estimate the degree of danger of the detected object, and the warning sound control unit 251 may control the feature amount of the warning sound based on the estimated degree of danger of the object.

[0184] For example, the warning sound control unit 251 changes the feature amount of the warning sound so as to more strongly attract the driver's attention as the zone in which the object is detected approaches, i.e., as the detected object approaches the vehicle 1. As a result, the driver can more reliably recognize the presence and direction of the object and direct his or her attention as the danger level of the detected object increases.

[0185] For example, comparing zone ZL41 on the left rear and zone ZR41 on the right rear of vehicle 1 in Fig. 16 with zone ZL51 on the left rear and zone ZR51 on the right rear of vehicle 1 in Fig. 17, zones ZL41 and ZR41 are closer to vehicle 1 than zones ZL51 and ZR51. Therefore, for example, when an object is present in zone ZL41 or zone ZR41, the warning sound control unit 251 makes the warning sound higher (higher frequency) than when the object is present in zone ZL51 or zone ZR51.

[0186] For example, the faster the relative speed of the object in the direction approaching the vehicle 1 relative to the vehicle 1, the higher the risk of the vehicle 1 colliding with the object. Therefore, even if an object is detected in the same zone, for example, the warning sound control unit 251 increases the volume of the warning sound as the relative speed of the detected object in the direction approaching the vehicle 1 increases.

[0187] Furthermore, for example, the warning sound control unit 251 may change the feature amount of the warning sound depending on whether the vehicle 1 is more likely to be harmed by the detected object or more likely to harm the detected object. For example, if the detected object is a truck, bus, passenger car, etc., it is determined that the vehicle 1 is more likely to be harmed. For example, if the detected object is a motorbike, bicycle, pedestrian, etc., it is determined that the vehicle 1 is more likely to harm the detected object.

[0188] This allows the driver to intuitively recognize whether their own vehicle (vehicle 1) is the one receiving or causing harm, and the driver can take a more appropriate response to the detected object depending on whether their own vehicle is the one receiving or causing harm.

[0189] Furthermore, for example, if the vehicle 1 is equipped with a speaker or the like that outputs sound to the surrounding area, the warning sound control unit 251 may control the output of a warning sound that indicates the presence of the vehicle 1 to a detected object based on the degree of risk that the vehicle 1 may cause harm to the detected object. This makes it possible to alert motorbikes, bicycles, pedestrians, and the like around the vehicle 1, for example, and to prevent accidents from occurring.

[0190] Furthermore, for example, the warning sound control unit 251 may change the feature amount of the warning sound based on the attributes of the detected object. For example, the warning sound control unit 251 may change the feature amount of the warning sound based on the type or size of the detected object. This allows the driver to intuitively recognize the attributes of the detected object.

[0191] Furthermore, when multiple objects are detected simultaneously, the warning sound control unit 251 may be configured to simultaneously output multiple warning sounds for each object. In this case, the warning sound control unit 251 may change the feature amount of each warning sound to make it easier to distinguish between the warning sounds for each object. For example, as described above, the warning sound control unit 251 may change the feature amount of the warning sound for each object based on the degree of danger of each object.

[0192] For example, when multiple detected objects are traveling in a convoy, the warning sound control unit 251 may treat the multiple objects traveling in the convoy as a single object and output a warning sound for the object that has been treated as a single object.

[0193] Furthermore, for example, when the vehicle 1 is performing a predetermined operation, the warning sound control unit 251 may output a warning sound when an object is detected within the zone.

[0194] For example, the warning sound control unit 251 may output a warning sound when an object is detected within the zone when the vehicle 1 attempts to change lanes or turn (e.g., when the turn signal of the vehicle 1 is turned on). In this case, the warning sound is used as one function of Lane Change Decision Aid Systems (LCDAS).

[0195] For example, the warning sound control unit 251 may be configured to output a warning sound when an object is detected within the zone when the vehicle 1 is about to back up (for example, when the gear of the vehicle 1 is set to reverse).

[0196] Furthermore, for example, when the recognition unit 73 detects a dangerous location, the warning sound control unit 251 may output a warning sound for the dangerous location in the same manner as when the above-mentioned object is detected.

[0197] Here, dangerous places are assumed to be, for example, places where accidents frequently occur, places where there is a high possibility of an accident occurring, etc. Places where there is a high possibility of an accident occurring are assumed to be, for example, near the entrance to a park or a school, or near a construction site.

[0198] Also, for example, when vehicle 1 is operating autonomously, vehicle 1 will basically automatically avoid danger and drive safely, so it is not necessarily necessary to output a warning sound for detected objects as described above.

[0199] On the other hand, for example, when the vehicle 1 is performing an operation such as changing lanes, turning, parking, overtaking, or starting during autonomous driving, it is assumed that the vehicle 1 detects an obstacle and cancels the operation. In this case, for example, the warning sound control unit 251 may be configured to output a warning sound for the detected object. For example, if the vehicle 1 turns on a turn signal to turn right but the vehicle 1 does not turn right, the warning sound control unit 251 may be configured to output a warning sound for the object that caused the vehicle 1 to cancel the right turn. This allows the occupants to recognize the reason why the vehicle 1 canceled the operation, providing a sense of security.

[0200] Furthermore, for example, when the vehicle 1 detects a highly dangerous object and automatically performs a risk avoidance operation such as pre-crash safety, the warning sound control unit 251 may output a warning sound for the object that caused the risk avoidance operation. This allows the passengers to recognize the cause of the risk avoidance operation performed by the vehicle 1, and provides a sense of security.

[0201] Furthermore, for example, the recognition unit 73 can detect an object at an intersection or the like through road-to-vehicle communication even in a blind spot where detection is difficult with the external recognition sensor 25. In this case, too, the warning sound control unit 251 can output a warning sound for the detected object in the same manner as described above.

[0202] Furthermore, for example, if a warning sound is continuously output to vehicles around the vehicle 1 during a traffic jam, the passengers may find it annoying. In response to this, for example, the recognition unit 73 may recognize vehicles in a traffic jam, and the warning sound control unit 251 may exclude vehicles in a traffic jam from the targets of the warning sound. Furthermore, when an object other than a vehicle in a traffic jam is detected in the zone, the warning sound control unit 251 may output a warning sound for that object. This makes it possible to output a warning sound to, for example, a motorbike or the like that is passing by a vehicle in a traffic jam.

[0203] Furthermore, for example, each seat speaker 204 may be used to output a warning sound only to a specific passenger (for example, the driver).

[0204] <Other variations> This technology can be used with sound image localization technologies other than 360RA.

[0205] The type of vehicle to which the present technology can be applied is not particularly limited. In addition, the present technology can also be applied to mobile devices other than vehicles, such as personal mobility devices, transport robots, airplanes, ships, construction machinery, and agricultural machinery.

[0206] <<4. Other>> <Example of computer configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0207] FIG. 18 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0208] In the computer 1000, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0209] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a recording unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0210] The input unit 1006 includes an input switch, a button, a microphone, an image sensor, etc. The output unit 1007 includes a display, a speaker, etc. The recording unit 1008 includes a hard disk, a non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0211] In the computer 1000 configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program recorded in the recording unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0212] The program executed by the computer 1000 (CPU 1001) can be provided by being recorded on a removable medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0213] In the computer 1000, the program can be installed in the recording unit 1008 via the input / output interface 1005 by inserting the removable medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the recording unit 1008. Alternatively, the program can be installed in the ROM 1002 or the recording unit 1008 in advance.

[0214] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0215] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0216] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0217] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0218] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0219] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0220] <Configuration combination example> The present technology can also be configured as follows.

[0221] (1) a recognition unit that performs a process of detecting objects around the mobile device; a warning sound control unit that controls a feature amount and a position of a sound image of a first warning sound that indicates the presence of the object based on the detected position of the object; An information processing device comprising: (2) the recognition unit detects a relative velocity of the object with respect to the mobile device; The warning sound control unit further controls a feature amount of the first warning sound based on the relative speed of the object. The information processing device according to (1) above. (3) the recognition unit recognizes an attribute of the object; The warning sound control unit further controls at least one of a feature amount and a position of a sound image of the first warning sound based on an attribute of the object. The information processing device according to (1) or (2). (4) the attributes of the object include a size of the object; The warning sound control unit controls the height position of the sound image of the first warning sound based on the size of the object. The information processing device according to (3) above. (5) The recognition unit estimates a degree of danger of the object, The warning sound control unit further controls a feature amount of the first warning sound based on the degree of danger of the object. The information processing device according to any one of (1) to (4). (6) the recognition unit estimates whether there is a higher possibility that the mobile device will harm the object or that the mobile device will be harmed by the object; The warning sound control unit changes a feature amount of the first warning sound depending on whether the mobile device is more likely to harm the object or more likely to be harmed by the object. The information processing device according to (5) above. (7) The warning sound control unit controls output of a second warning sound that indicates the presence of the mobile device to the object based on the degree of risk that the mobile device will harm the object. The information processing device according to (5) or (6). (8) When a plurality of objects are detected, the warning sound control unit controls the vehicle to simultaneously output a plurality of first warning sounds, each indicating the presence of each of the objects. The information processing device according to any one of (1) to (7). (9) The recognition unit estimates a degree of danger of each of the objects, The warning sound control unit controls the feature amount of each of the first warning sounds based on the degree of danger of each of the objects. The information processing device according to (8). (10) When the plurality of objects are traveling in a convoy, the warning sound control unit controls the plurality of objects to be regarded as one object and to output the first warning sound. The information processing device according to (8) or (9). (11) the recognition unit sets a plurality of zones around the mobile device and detects the zone in which the object exists; The warning sound control unit controls a feature amount and a position of a sound image of the first warning sound based on the zone where the object exists. The information processing device according to any one of (1) to (10). (12) The recognition unit changes the setting of the zone based on the situation in which the mobile device is placed. The information processing device according to (11) above. (13) The warning sound control unit controls the mobile device to output the first warning sound when the object is detected while the mobile device is performing a predetermined operation. The information processing device according to any one of (1) to (12). (14) The warning sound control unit controls the mobile device to output the first warning sound when the mobile device stops a predetermined operation due to the detected object while the mobile device is operating autonomously. The information processing device according to any one of (1) to (13). (15) the recognition unit recognizes objects around the mobile device through road-to-vehicle communication; The warning sound control unit controls to output the first warning sound when the object is recognized in a blind spot of the mobile device through the road-to-vehicle communication. The information processing device according to any one of (1) to (14). (16) The recognition unit acquires information about a road on which the mobile device is traveling based on at least one of a sensor provided in the mobile device and a map of the surroundings of the mobile device, and corrects the position of the object based on the information about the road. The information processing device according to any one of (1) to (15). (17) Performing a detection process of objects around the mobile device; Based on the detected position of the object, the feature amount of the warning sound indicating the presence of the object and the position of the sound image are controlled. An information processing method comprising the steps. (18) Performing a detection process of objects around the mobile device; Based on the detected position of the object, the feature amount of the warning sound indicating the presence of the object and the position of the sound image are controlled. A program that causes a computer to execute a process that includes steps. (19) a recognition unit that performs a process of detecting surrounding objects; a warning sound control unit that controls a feature amount and a position of a sound image of a warning sound indicating the presence of the object based on the detected position of the object; A mobile device comprising: (20) a sensor unit that senses the surroundings of the mobile device; a recognition unit that performs a process of detecting objects around the mobile device based on the sensing result of the sensor unit; a warning sound control unit that controls a feature amount and a position of a sound image of a warning sound indicating the presence of the object based on the detected position of the object; a speaker that outputs the warning sound under the control of the warning sound control unit; An information processing system comprising:

[0222] The effects described in this specification are merely examples and are not limiting, and other effects may also be present. [Explanation of symbols]

[0223] 1 vehicle, 11 vehicle control system, 25 external recognition sensor, 31 HMI, 51 camera, 52 radar, 53 LiDAR, 54 ultrasonic sensor, 73 recognition unit, 62 action planning unit, 63 operation control unit, 202FL to 202BR speaker, 251 warning sound control unit

Claims

1. a recognition unit that performs a process of detecting an object around the mobile device and estimates whether the mobile device is more likely to harm the object or to be harmed by the object; a warning sound control unit that controls a feature value and a position of a sound image of a first warning sound indicating the presence of the object based on the position of the detected object, and changes the feature value of the first warning sound depending on whether the mobile device is more likely to harm the object or more likely to be harmed by the object; An information processing device comprising:

2. the recognition unit detects a relative velocity of the object with respect to the mobile device; The warning sound control unit further controls a feature amount of the first warning sound based on the relative speed of the object. The information processing device according to claim 1 .

3. the recognition unit recognizes an attribute of the object; The warning sound control unit further controls at least one of a feature amount and a position of a sound image of the first warning sound based on an attribute of the object. The information processing device according to claim 1 .

4. the attributes of the object include a size of the object; The warning sound control unit controls the height position of the sound image of the first warning sound based on the size of the object. The information processing device according to claim 3 .

5. The recognition unit estimates a degree of danger of the object, The warning sound control unit further controls a feature amount of the first warning sound based on the degree of danger of the object. The information processing device according to claim 1 .

6. The warning sound control unit controls output of a second warning sound that indicates the presence of the mobile device to the object based on the degree of risk that the mobile device will harm the object. The information processing device according to claim 1 .

7. When a plurality of objects are detected, the warning sound control unit controls the vehicle to simultaneously output a plurality of first warning sounds, each indicating the presence of one of the objects. The information processing device according to claim 1 .

8. The recognition unit estimates a degree of danger of each of the objects, The warning sound control unit controls a feature amount of each of the first warning sounds based on a degree of danger of each of the objects. The information processing device according to claim 7 .

9. When the plurality of objects are traveling in a convoy, the warning sound control unit controls the plurality of objects to be regarded as one object and to output the first warning sound. The information processing device according to claim 7 .

10. the recognition unit sets a plurality of zones around the mobile device and detects the zone in which the object exists; The warning sound control unit controls a feature amount and a position of a sound image of the first warning sound based on the zone where the object exists. The information processing device according to claim 1 .

11. The recognition unit changes the setting of the zone based on the situation in which the mobile device is placed. The information processing device according to claim 10.

12. The warning sound control unit controls the mobile device to output the first warning sound when the object is detected while the mobile device is performing a predetermined operation. The information processing device according to claim 1 .

13. The warning sound control unit controls the mobile device to output the first warning sound when the mobile device stops a predetermined operation due to the detected object while the mobile device is operating automatically. The information processing device according to claim 1 .

14. the recognition unit recognizes objects around the mobile device through road-to-vehicle communication; The warning sound control unit controls to output the first warning sound when the object is recognized in a blind spot of the mobile device through the road-to-vehicle communication. The information processing device according to claim 1 .

15. The recognition unit acquires information about a road on which the mobile device is traveling based on at least one of a sensor provided in the mobile device and a map of the surroundings of the mobile device, and corrects the position of the object based on the information about the road. The information processing device according to claim 1 .

16. An information processing device comprising: performing a process of detecting objects around the mobile device; Estimating whether the mobile device is more likely to harm the object or to be harmed by the object; controlling a feature value and a position of a sound image of a warning sound indicating the presence of the object based on the detected position of the object; changing a feature value of the warning sound depending on whether the mobile device is more likely to harm the object or whether the mobile device is more likely to be harmed by the object; An information processing method including:

17. performing a process of detecting objects around the mobile device; Estimating whether the mobile device is more likely to harm the object or to be harmed by the object; controlling a feature value and a position of a sound image of a warning sound indicating the presence of the object based on the detected position of the object; changing a feature value of the warning sound depending on whether the mobile device is more likely to harm the object or whether the mobile device is more likely to be harmed by the object; A program for causing a computer to execute a process including the above.

18. a recognition unit that performs a process of detecting surrounding objects and estimates whether there is a higher possibility that the user will cause harm to the object or that the user will be harmed by the object; a warning sound control unit that controls a feature value and a position of a sound image of a warning sound indicating the presence of the object based on the position of the detected object, and changes the feature value of the warning sound depending on whether there is a high possibility that the object will cause harm or whether there is a high possibility that the object will cause harm; and A mobile device comprising:

19. a sensor unit that senses the surroundings of the mobile device; a recognition unit that performs a detection process of an object around the mobile device based on the sensing result of the sensor unit, and estimates whether the mobile device is more likely to harm the object or to be harmed by the object; a warning sound control unit that controls a feature value and a position of a sound image of a warning sound indicating the presence of the object based on the position of the detected object, and changes the feature value of the warning sound depending on whether the mobile device is more likely to harm the object or more likely to be harmed by the object; a speaker that outputs the warning sound under the control of the warning sound control unit; An information processing system comprising:

Images