Information processing device, information processing system, and program

Abstract

To make it possible to achieve quick and smooth parking support in a manner close to a human sense regardless of an environment of a parking space.SOLUTION: An information processing device generates a three-dimensional semantic segmentation image having depth data and class information classified with a three-dimensional segmentation process to the image data by a pixel unit of the image data, retrieves a parking space where parking is allowed on the basis of the three-dimensional segmentation image, and controls vehicles by planning a route to the retrieved parking space. The present disclosure can be applied to a parking support system.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that are capable of realizing quick and smooth parking assistance that is close to human senses, regardless of the environment of the parking space. [Background technology]

[0002] In recent years, interest in parking assistance systems has been growing. Parking assistance systems are required in various daily driving situations, and there is a demand for safer, more comfortable, and more convenient parking assistance systems.

[0003] For example, a technology has been proposed that detects a parking space based on a camera image and provides parking assistance for parking in the detected parking space (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-111803 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology described in Patent Document 1, the parking space is not detected while taking depth information into consideration.

[0006] Therefore, in an environment where it is difficult to detect a parking space without being able to recognize depth, it may not be possible to distinguish the parking space with a sense similar to that used by humans to detect a parking space, and there is a risk that smooth parking assistance may not be achieved.

[0007] The present disclosure has been made in consideration of such circumstances, and in particular, is directed to realizing quick and smooth parking assistance that is closer to human senses, regardless of the environment of the parking space. [Means for solving the problem]

[0008] An information processing device and program according to one aspect of the present disclosure includes an image acquisition unit that acquires image data captured by a camera within a predetermined distance from a vehicle, a depth data acquisition unit that acquires depth data of a range that at least partially overlaps with the imaging range of the image data, a 3D semantic segmentation processing unit that generates a 3D semantic segmentation image in which class information and depth data are set on a pixel-by-pixel basis of the image data, and a parking space detection unit that detects a parking space based on the 3D semantic segmentation image.

[0009] An information processing method of one aspect of the present disclosure is an information processing method that includes an image acquisition process that acquires image data captured by a camera within a predetermined distance from a vehicle, a depth data acquisition process that acquires depth data of a range that at least partially overlaps with the imaging range of the image data, a 3D semantic segmentation process that generates a 3D semantic segmentation image in which class information and depth data are set on a pixel-by-pixel basis of the image data, and a parking space detection process that detects a parking space based on the 3D semantic segmentation image.

[0010] In one aspect of the present disclosure, image data captured by a camera within a predetermined distance from a vehicle is acquired, depth data is acquired from a range that at least partially overlaps with the imaging range of the image data, a 3D semantic segmentation image in which class information and depth data are set for each pixel of the image data is generated, and a parking space is detected based on the 3D semantic segmentation image. [Brief description of the drawings]

[0011] [Figure 1] FIG. 13 is a diagram illustrating limitations of a parking assistance function. [Diagram 2] FIG. 1 is a diagram illustrating an example of driving assistance using a parking assistance function. [Diagram 3] FIG. 1 is a diagram illustrating an overview of a parking assistance function to which the technology of the present disclosure is applied. [Figure 4] FIG. 1 is a block diagram showing an example of the configuration of a vehicle control system. [Diagram 5] FIG. 2 is a diagram illustrating an example of a sensing region. [Figure 6] FIG. 2 is a diagram illustrating an example of the configuration of a parking assistance control unit according to the present disclosure. [Figure 7] 7 is a diagram illustrating an example of the configuration of a 3D semantic segmentation processing unit in FIG. 6. [Figure 8] FIG. 1 is a diagram illustrating 3D semantic segmentation information. [Figure 9] FIG. 11 is a diagram illustrating an example of detecting a parking space. [Figure 10] FIG. 11 is a diagram illustrating an example of detecting a parking space. [Figure 11] 13 is a flowchart illustrating a 3D semantic segmentation information generation process. [Figure 12] 4 is a flowchart illustrating a parking assistance process. [Figure 13] 13 is a flowchart illustrating a parking space search mode process. [Figure 14] 11 is a flowchart illustrating a parking mode process. [Figure 15] FIG. 1 is a diagram illustrating an example of the configuration of a general-purpose personal computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0013] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Overview of this Disclosure 2. Example of vehicle control system configuration 3. Example configuration of a parking assistance control unit that realizes the parking assistance function disclosed herein 4. Software implementation example

[0014] <<1. Overview of this Disclosure>> An overview of a technology that applies the technology of the present disclosure to achieve quick and smooth parking assistance that is close to human senses, regardless of the environment of the parking space, will be described.

[0015] Parking assistance functions, such as recognizing parking spaces and automatically parking, or guiding the vehicle to the optimal parking path, have already been commercialized in various forms, but each has various restrictions.

[0016] For example, with the first parking assistance function, as shown by the dotted line in Figure 1, it is possible to park in a parking space regardless of whether the vehicle has been there before, that is, even in a parking space that has not been registered in advance in the vehicle, but it is not possible to park in a parking space that does not have a white line or has a faded white line.

[0017] Also, for example, the second parking assistance function, as shown by the dashed line in FIG. 1, allows parking in parking spaces with and without white lines, regardless of whether there are white lines or not, but it only functions for parking spaces that the vehicle has been to before, i.e., parking spaces that have been registered in advance in the vehicle.

[0018] That is, to realize the parking assistance function described above, conditions are set according to whether or not the target parking space has a white line and whether or not the parking space has been registered in advance. In Fig. 1, the horizontal axis indicates the degree of whether or not there is a white line (whether or not there is a white line), and the vertical axis indicates the degree of whether or not the function has been performed (whether or not it has been registered in advance).

[0019] Therefore, in the parking assistance function of the present disclosure, as shown by the solid lines in FIG. 1, by appropriately recognizing the surrounding conditions such as the parking space and the direction of the parked vehicles (parallel, vertical), regardless of whether the target parking space has been registered in advance in the vehicle (whether the vehicle has been there before) and regardless of whether there is a white line, quick and smooth parking assistance is realized, just like parking operations performed by a human being.

[0020] For example, as shown in the left part of FIG. 2, parking assistance will be considered when a vehicle C1 having sensors Sc1-1 and Sc1-2 such as cameras provided on the left and right sides of the body is parked in a parking space SP1.

[0021] In FIG. 2, the direction of a convex portion indicated by the isosceles of an isosceles triangle mark in the figure is the front of the vehicle C1.

[0022] As shown in the left part of Figure 2, vehicle C1 needs to pass in front of parking space SP1 at least once in order to detect the location of parking space SP1 as an available space in which to park using sensor Sc1-1 mounted on the left side.

[0023] Also, for example, as shown in the right part of FIG. 2, parking assistance will be considered when a vehicle C2 having sensors Sc11-1 and Sc11-2 such as ultrasonic sensors provided on the left and right sides of the body is parked in a parking space SP2.

[0024] In the case of the right side of Figure 2, too, the vehicle C2 needs to pass near the parking space SP2 at least once in order to detect the parking space SP1 that is an available space for parking using the sensor Sc11-1 attached to the left side of the vehicle C2.

[0025] That is, as explained with reference to FIG. 2, when considering parking assistance by providing cameras or ultrasonic sensors on the left and right sides of the main body, it is necessary for the vehicle to pass in front of or near the target parking space in advance.

[0026] For this reason, in the example of parking assistance described with reference to Figure 2, it is not possible to visually search for a target vacant space, determine the searched vacant space as the target parking space, and start the parking operation as a human would.

[0027] For this reason, when parking using the above-mentioned parking assistance, the vehicle will continue to circle the parking lot until it passes in front of or beside an available parking space.

[0028] In this case, in some cases, the person in the vehicle may be able to visually confirm an available parking space, but may still drive around searching for a parking space even in areas where they know there are no available spaces at all.

[0029] As a result, it may take unnecessary time to complete parking, which may cause the person in the vehicle to feel that the parking operation is not quick and smooth, but rather feels unnatural.

[0030] Therefore, in a parking assistance function to which the technology disclosed herein is applied, the situation around the vehicle is grasped through object recognition processing using 3D (three-dimensional) semantic segmentation, and the parking space is identified before the parking operation is performed.

[0031] For example, as shown in FIG. 3, a sensor Sc31 that detects images and depth data is provided in front of a vehicle C11.

[0032] First, in the vehicle C11, the surrounding situation is recognized by performing object recognition processing using 3D semantic segmentation based on the depth data of the front detected by the sensor Sc31 and the image. Next, in the vehicle C11, based on the object recognition result, a vacant space as a parking target is searched for within a range from the vehicle C11 to a position a predetermined distance ahead (for example, a position 15 m to 30 m ahead).

[0033] Then, when an empty space is found from the search results within a range a predetermined distance ahead of the vehicle C11, the searched empty space is recognized as the parking space SP11 that is the parking target, and a parking path for parking, for example as shown by the thick solid line in the figure, is calculated, and the vehicle C11 is controlled to operate along the calculated parking path.

[0034] By implementing such parking operations, parking assistance is provided in a manner similar to that of a human parking operation, in that an empty space is visually searched for, and the parking operation is performed when the searched empty space is recognized as a parking space.

[0035] As a result, it is possible to realize quick and smooth parking assistance that does not seem strange to the human occupants.

[0036] In addition, by using object recognition processing based on 3D semantic segmentation to recognize the surrounding situation and identify the parking space, it is possible to provide comfortable parking assistance in parking spaces located in various locations, regardless of the parking space's environment.

[0037] <<2. Example of vehicle control system configuration>> FIG. 4 is a block diagram showing an example of the configuration of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.

[0038] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to driving assistance and automatic driving of the vehicle 1.

[0039] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a GNSS (Global Navigation Satellite System) receiving unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a recording unit 28, a driving assistance / automatic driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0040] The processor 21, the communication unit 22, the map information storage unit 23, the GNSS receiving unit 24, the external recognition sensor 25, the in-vehicle sensor 26, the vehicle sensor 27, the recording unit 28, the driving support / automatic driving control unit 29, the driver monitoring system (DMS) 30, the human machine interface (HMI) 31, and the vehicle control unit 32 are connected to each other so as to be able to communicate with each other via a communication network 41. The communication network 41 is configured by an in-vehicle communication network or a bus conforming to a digital two-way communication standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). The communication network 41 may be used depending on the type of data to be communicated. For example, CAN is applied to data related to vehicle control, and Ethernet is applied to large-capacity data. In addition, each part of the vehicle control system 11 may be directly connected to each other without going through the communication network 41, using wireless communication intended for communication over relatively short distances, such as near field communication (NFC) or Bluetooth (registered trademark).

[0041] In the following description, when each unit of the vehicle control system 11 communicates with each other via the communication network 41, the description of the communication network 41 will be omitted. For example, when the processor 21 and the communication unit 22 communicate with each other via the communication network 41, it will be simply described that the processor 21 and the communication unit 22 communicate with each other.

[0042] The processor 21 is configured with various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The processor 21 controls the vehicle control system 11 as a whole.

[0043] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 22 can communicate using a plurality of communication methods.

[0044] The following provides an overview of communication with the outside of the vehicle that can be performed by the communication unit 22. The communication unit 22 communicates with a server (hereinafter referred to as an external server) on an external network via a base station or an access point using a wireless communication method such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the Internet, a cloud network, or a network specific to a carrier. The communication method by which the communication unit 22 communicates with the external network is not particularly limited as long as it is a wireless communication method that allows digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0045] Also, for example, the communication unit 22 can communicate with a terminal existing near the vehicle by using P2P (Peer To Peer) technology. The terminal existing near the vehicle can be, for example, a terminal attached to a moving body moving at a relatively low speed such as a pedestrian or a bicycle, a terminal installed at a fixed position in a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 22 can also perform V2X communication. The V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside device, vehicle-to-home communication with a home, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian.

[0046] The communication unit 22 can receive, for example, a program for updating software that controls the operation of the vehicle control system 11 from the outside. The communication unit 22 can further receive map information, traffic information, information about the surroundings of the vehicle 1, and the like from the outside. Also, for example, the communication unit 22 can transmit information about the vehicle 1, information about the surroundings of the vehicle 1, and the like to the outside. Information about the vehicle 1 that the communication unit 22 transmits to the outside includes, for example, data indicating the state of the vehicle 1, a recognition result by the recognition unit 73, and the like. Furthermore, for example, the communication unit 22 performs communication corresponding to a vehicle emergency notification system such as e-Call.

[0047] The communication with the inside of the vehicle that can be performed by the communication unit 22 will be briefly described. The communication unit 22 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 22 can wirelessly communicate with each device in the vehicle using a communication method that allows digital bidirectional communication at a communication speed equal to or higher than a predetermined speed by wireless communication, such as wireless LAN, Bluetooth (registered trademark), NFC, or WUSB (Wireless USB). Not limited to this, the communication unit 22 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 22 can communicate with each device in the vehicle using wired communication via a cable connected to a connection terminal not shown. The communication unit 22 can communicate with each device in the vehicle using a communication method that allows digital bidirectional communication at a communication speed equal to or higher than a predetermined speed by wired communication, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), or MHL (Mobile High-definition Link).

[0048] Here, the in-vehicle device refers to, for example, a device in the vehicle that is not connected to the communication network 41. Assumed in-vehicle devices include, for example, a mobile device or a wearable device carried by a passenger such as a driver, and an information device brought into the vehicle and temporarily installed therein.

[0049] For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System) (registered trademark)) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.

[0050] The map information storage unit 23 stores one or both of a map acquired from an external source and a map created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that has lower precision than a high-precision map and covers a wide area, and the like.

[0051] The high-precision map is, for example, a dynamic map, a point cloud map, a vector map, etc. The dynamic map is, for example, a map consisting of four layers of dynamic information, semi-dynamic information, semi-static information, and static information, and is provided to the vehicle 1 from an external server or the like. The point cloud map is a map composed of a point cloud (point cloud data). Here, the vector map refers to a map adapted to an ADAS (Advanced Driver Assistance System) in which traffic information such as the positions of lanes and traffic lights is associated with the point cloud map.

[0052] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created by the vehicle 1 as a map for matching with a local map described later based on sensing results from the radar 52, the LiDAR 53, or the like, and stored in the map information storage unit 23. In addition, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred meters square regarding the planned route along which the vehicle 1 will travel is acquired from the external server or the like in order to reduce communication capacity.

[0053] The GNSS receiver 24 receives GNSS signals from GNSS satellites and acquires position information of the vehicle 1. The received GNSS signals are supplied to the driving assistance / automated driving control unit 29. Note that the GNSS receiver 24 is not limited to a method using GNSS signals, and may acquire position information using a beacon, for example.

[0054] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0055] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. Not limited to this, the external recognition sensor 25 may include one or more types of sensors among the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of the cameras 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 is not particularly limited as long as it is a number that can be practically installed in the vehicle 1. In addition, the type of the sensor included in the external recognition sensor 25 is not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0056] The imaging method of the camera 51 is not particularly limited as long as it is an imaging method that allows distance measurement. For example, the camera 51 can be a camera of various imaging methods such as a ToF (Time of Flight) camera, a stereo camera, a monocular camera, an infrared camera, etc., as necessary. However, the camera 51 may be a camera simply for acquiring a photographed image (captured image) without regard to distance measurement.

[0057] Furthermore, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment for the vehicle 1. The environmental sensor is a sensor for detecting the environment such as the weather, climate, and brightness, and may include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0058] Furthermore, for example, the external recognition sensor 25 includes a microphone used to detect sounds around the vehicle 1 and the positions of sound sources.

[0059] The in-vehicle sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the in-vehicle sensor 26 are not particularly limited as long as they are the number that can be realistically installed in the vehicle 1.

[0060] For example, the in-vehicle sensor 26 may include one or more types of sensors selected from a camera, a radar, a seating sensor, a steering wheel sensor, a microphone, and a biosensor. The camera included in the in-vehicle sensor 26 may be a camera using various imaging methods capable of measuring distances, such as a ToF camera, a stereo camera, a monocular camera, and an infrared camera. The camera included in the in-vehicle sensor 26 may be a camera simply for acquiring captured images, regardless of distance measurement. The biosensor included in the in-vehicle sensor 26 is provided, for example, on a seat, a steering wheel, or the like, and detects various types of bioinformation of a passenger such as a driver.

[0061] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the vehicle sensor 27 are not particularly limited as long as they are the number that can be practically installed in the vehicle 1.

[0062] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these. For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of operation of an accelerator pedal, and a brake sensor that detects the amount of operation of a brake pedal. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or the motor, an air pressure sensor that detects the air pressure of the tires, a slip ratio sensor that detects the slip ratio of the tires, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining charge and temperature of the battery, and an impact sensor that detects an impact from the outside.

[0063] The recording unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The recording unit 28 is used, for example, as an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and the storage medium may be a magnetic storage device such as a hard disc drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The recording unit 28 records various programs and data used by each part of the vehicle control system 11. For example, the recording unit 28 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and records information about the vehicle 1 before and after an event such as an accident.

[0064] The driving support / automatic driving control unit 29 controls driving support and automatic driving of the vehicle 1. For example, the driving support / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63. In addition, the driving support / automatic driving control unit 29 realizes the function of a parking support control unit 201 ( FIG. 6 ) that realizes a parking support function of the present disclosure, which will be described later.

[0065] The analysis unit 61 performs an analysis process of the vehicle 1 and the surrounding situation. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0066] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25, and estimates the self-position of the vehicle 1 by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.

[0067] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map (OGM) created using a technique such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the above-mentioned point cloud map. The occupancy grid map is a map in which a three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size, and the occupancy state of an object is indicated on a grid-by-grid basis. The occupancy state of an object is indicated, for example, by the presence or absence of an object and the probability of its existence. The local map is also used, for example, in detection processing and recognition processing of the external situation of the vehicle 1 by the recognition unit 73.

[0068] The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the GNSS signal and the sensor data from the vehicle sensor 27.

[0069] The sensor fusion unit 72 performs a sensor fusion process to obtain new information by combining a plurality of different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and association.

[0070] The recognition unit 73 executes a detection process for detecting the situation outside the vehicle 1 and a recognition process for recognizing the situation outside the vehicle 1.

[0071] For example, the recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, and the like.

[0072] Specifically, for example, the recognition unit 73 performs detection processing and recognition processing of objects around the vehicle 1. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0073] For example, the recognition unit 73 detects objects around the vehicle 1 by performing clustering to classify a point cloud based on sensor data from the LiDAR 53, the radar 52, or the like into clusters of points. This allows the presence or absence, size, shape, and position of objects around the vehicle 1 to be detected.

[0074] For example, the recognition unit 73 performs tracking to follow the movement of the clusters of the point cloud classified by clustering, thereby detecting the movement of the objects around the vehicle 1. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.

[0075] For example, the recognition unit 73 performs object recognition processing such as semantic segmentation on the image data supplied from the camera 51 to recognize the type of object around the vehicle 1.

[0076] Note that possible objects to be detected or recognized by the recognition unit 73 include, for example, vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, and road markings.

[0077] For example, the recognition unit 73 can perform recognition processing of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the estimation result of the self-position by the self-position estimation unit 71, and the recognition result of the objects around the vehicle 1 by the recognition unit 73. Through this processing, the recognition unit 73 can recognize the positions and states of traffic signals, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes on which travel is possible.

[0078] For example, the recognition unit 73 can perform a recognition process of the environment around the vehicle 1. The surrounding environment to be recognized by the recognition unit 73 may include weather, temperature, humidity, brightness, and road surface conditions.

[0079] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route following processing.

[0080] Global path planning is a process for planning a rough route from the start to the goal. This route planning also includes a process for local path planning, which is called trajectory planning and allows the vehicle 1 to proceed safely and smoothly in the vicinity of the vehicle 1, taking into account the motion characteristics of the vehicle 1 on the route planned by the route planning.

[0081] Path following is a process of planning an operation for safely and accurately traveling along a route planned by a route plan within a planned time. The action planning unit 62 can calculate a target speed and a target angular velocity of the vehicle 1, for example, based on the result of the path following process.

[0082] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0083] For example, the operation control unit 63 controls a steering control unit 81, a brake control unit 82, and a drive control unit 83 included in a vehicle control unit 32 described later, and performs acceleration / deceleration control and direction control so that the vehicle 1 proceeds along a trajectory calculated by a trajectory plan. For example, the operation control unit 63 performs cooperative control for the purpose of realizing ADAS functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, collision warning for the vehicle itself, and lane departure warning for the vehicle itself. For example, the operation control unit 63 performs cooperative control for the purpose of automatic driving that autonomously drives without the driver's operation.

[0084] The DMS 30 performs authentication processing of the driver and recognition processing of the driver's state based on the sensor data from the in-vehicle sensor 26 and input data input to the HMI 31 (described later), etc. In this case, the driver's state to be recognized by the DMS 30 may be, for example, physical condition, alertness level, concentration level, fatigue level, line of sight direction, level of intoxication, driving operation, posture, etc.

[0085] The DMS 30 may perform authentication processing of passengers other than the driver and recognition processing of the conditions of the passengers. For example, the DMS 30 may perform recognition processing of the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Conditions inside the vehicle to be recognized include, for example, temperature, humidity, brightness, odor, and the like.

[0086] The HMI 31 inputs various data and instructions, and displays various data to the driver, etc.

[0087] The input of data by the HMI 31 will be briefly described. The HMI 31 includes an input device for a person to input data. The HMI 31 generates an input signal based on data, instructions, and the like input by the input device, and supplies the input signal to each part of the vehicle control system 11. The HMI 31 includes, as input devices, operators such as a touch panel, a button, a switch, and a lever. Without being limited to this, the HMI 31 may further include an input device that allows information to be input by a method other than manual operation, such as voice or gesture. Furthermore, the HMI 31 may use, as an input device, an externally connected device such as a remote control device using infrared rays or radio waves, or a mobile device or wearable device compatible with the operation of the vehicle control system 11.

[0088] The presentation of data by the HMI 31 will be briefly described. The HMI 31 generates visual information, auditory information, and tactile information for the passenger or the outside of the vehicle. The HMI 31 also performs output control to control the output, output contents, output timing, output method, etc. of each of the generated information. The HMI 31 generates and outputs, as visual information, information indicated by images or lights, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image indicating the situation around the vehicle 1. The HMI 31 also generates and outputs, as auditory information, information indicated by sounds, such as voice guidance, warning sounds, and warning messages. The HMI 31 also generates and outputs, as tactile information, information given to the passenger's sense of touch by, for example, force, vibration, movement, etc.

[0089] Examples of the output device to which the HMI 31 outputs visual information include a display device that displays an image to present visual information, and a projector device that projects an image to present visual information. The display device may be a device that displays visual information within the field of vision of the passenger, such as a head-up display, a transmissive display, or a wearable device with an AR (Augmented Reality) function, in addition to a display device having a normal display. The HMI 31 may also use display devices such as a navigation device, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, and lamps provided in the vehicle 1 as output devices that output visual information.

[0090] As an output device to which the HMI 31 outputs auditory information, for example, an audio speaker, a headphone, or an earphone can be applied.

[0091] For example, a haptic element using haptic technology can be applied as an output device to which the HMI 31 outputs haptic information. The haptic element is provided on a part of the vehicle 1 that an occupant touches, such as a steering wheel or a seat.

[0092] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0093] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a control unit such as an ECU that controls the steering system, and an actuator that drives the steering system.

[0094] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative brake mechanism, etc. The brake control unit 82 includes, for example, a control unit such as an ECU that controls the brake system.

[0095] The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, etc. The drive control unit 83 includes, for example, a control unit such as an ECU that controls the drive system.

[0096] The body system control unit 84 detects and controls the states of the body system of the vehicle 1. The body system includes, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a control unit such as an ECU that controls the body system.

[0097] The light control unit 85 detects and controls the states of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 85 includes a control unit such as an ECU that controls the lights.

[0098] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a control unit such as an ECU that controls the car horn.

[0099] Fig. 5 is a diagram showing an example of a sensing area by the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 of the external recognition sensor 25 in Fig. 4. In Fig. 5, the vehicle 1 is shown as viewed from above, with the left end side being the front end of the vehicle 1 and the right end side being the rear end of the vehicle 1.

[0100] Sensing area 101F and sensing area 101B show examples of sensing areas of the ultrasonic sensors 54. The sensing area 101F covers the periphery of the front end of the vehicle 1 by the multiple ultrasonic sensors 54. The sensing area 101B covers the periphery of the rear end of the vehicle 1 by the multiple ultrasonic sensors 54.

[0101] The sensing results in the sensing area 101F and the sensing area 101B are used for, for example, parking assistance for the vehicle 1.

[0102] Sensing area 102F to sensing area 102B show examples of sensing areas of a short-range or medium-range radar 52. Sensing area 102F covers a position farther in front of the vehicle 1 than sensing area 101F. Sensing area 102B covers a position farther in the rear of the vehicle 1 than sensing area 101B. Sensing area 102L covers the rear periphery of the left side of the vehicle 1. Sensing area 102R covers the rear periphery of the right side of the vehicle 1.

[0103] The sensing results in the sensing area 102F are used, for example, to detect vehicles, pedestrians, and the like that exist in front of the vehicle 1. The sensing results in the sensing area 102B are used, for example, for a collision prevention function behind the vehicle 1. The sensing results in the sensing area 102L and the sensing area 102R are used, for example, to detect objects in blind spots on the sides of the vehicle 1.

[0104] Sensing area 103F to sensing area 103B show examples of sensing areas by camera 51. Sensing area 103F covers a position farther in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position farther in the rear of vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery of the left side of vehicle 1. Sensing area 103R covers the periphery of the right side of vehicle 1.

[0105] The sensing results in the sensing area 103F can be used for, for example, recognition of traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. The sensing results in the sensing area 103B can be used for, for example, parking support and surround view systems. The sensing results in the sensing areas 103L and 103R can be used for, for example, surround view systems.

[0106] Sensing area 104 illustrates an example of the sensing area of ​​the LiDAR 53. Sensing area 104 covers a position farther forward than sensing area 103F in front of the vehicle 1. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.

[0107] The sensing results in the sensing area 104 are used to detect objects such as surrounding vehicles, for example.

[0108] Sensing area 105 shows an example of the sensing area of ​​the long-distance radar 52. Sensing area 105 covers a position farther ahead of the vehicle 1 than sensing area 104. On the other hand, sensing area 105 has a narrower range in the left-right direction than sensing area 104.

[0109] The sensing results in the sensing area 105 are used for, for example, ACC (Adaptive Cruise Control), emergency braking, collision avoidance, and the like.

[0110] The sensing areas of the cameras 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 included in the external recognition sensor 25 may have various configurations other than those shown in FIG. 5. Specifically, the ultrasonic sensor 54 may sense the sides of the vehicle 1, and the LiDAR 53 may sense the rear of the vehicle 1. The installation positions of the sensors are not limited to the above-mentioned examples. The number of sensors may be one or more.

[0111] <<3. Configuration example of a parking assistance control unit that realizes the parking assistance function of the present disclosure>> Next, a configuration example of the parking assist control unit 201 that realizes the parking assist function of the present disclosure will be described with reference to FIG.

[0112] The parking assistance control unit 201 is realized by the driving assistance / automatic driving control unit 29 in the vehicle control system 11 described above.

[0113] The parking assistance control unit 201 realizes a parking assistance function based on image data, depth data (ranging results), and radar detection results supplied from the cameras 202-1 to 202-q, the ToF cameras 203-1 to 203-r, and the radars 204-1 to 204-s.

[0114] In addition, when there is no particular need to distinguish between the cameras 202-1 to 202-q, the ToF cameras 203-1 to 203-r, and the radars 204-1 to 204-s, they will simply be referred to as the camera 202, the ToF camera 203, and the radar 204, and the other components will be referred to in the same manner.

[0115] The camera 202 and the ToF camera 203 correspond to the camera 51 in FIG. 4, and the radar 204 corresponds to the radar 52 in FIG.

[0116] In addition, the parking assistance control unit 201 may realize the parking assistance function using detection results of various configurations of the external recognition sensor 25 and the vehicle sensor 27 in FIG. 4 as well as the cameras 202-1 to 202-q, the ToF cameras 203-1 to 203-r, and the radars 204-1 to 204-s in FIG. 6.

[0117] Based on the image data and depth data supplied from the camera 202, the ToF camera 203, and the radar 204, the parking assistance control unit 201 associates the image data with the depth data on a pixel-by-pixel basis.

[0118] The parking assistance control unit 201 performs 3D semantic segmentation processing using information including image data, depth data, and radar detection results, in addition to the depth data associated on a pixel-by-pixel basis, and generates a 3D semantic segmentation image in which the depth data and object recognition results are associated on a pixel-by-pixel basis.

[0119] The parking assistance control unit 201 searches for a parking space based on the 3D semantic segmentation image, and assists the vehicle 1 in parking the vehicle 1 in the searched parking space.

[0120] At this time, the parking assistance control unit 201 operates in two operation modes, a parking space search mode and a parking mode, to realize the parking assistance function.

[0121] That is, first, the parking assistance control unit 201 operates in a parking space search mode, searches for a parking space based on the 3D semantic segmentation image, and stores the search result.

[0122] Then, when a parking space is searched for, the parking assistance control unit 201 switches the operation mode to a parking mode, plans a route to the searched parking space, and controls the operation of the vehicle 1 so that parking is completed along the planned route.

[0123] By implementing the parking assistance function in this manner, the parking space is searched for and then the parking operation is performed, just like when a human parks, making it possible to realize quick and smooth parking assistance.

[0124] More specifically, the parking assistance control unit 201 is made up of an analysis unit 261, an action planning unit 262, and an operation control unit 263.

[0125] The analysis unit 261 is a component corresponding to the analysis unit 61 in FIG. 4, the behavior planning unit 262 is a component corresponding to the behavior planning unit 62 in FIG. 4, and the operation control unit 263 is a component corresponding to the operation control unit 63 in FIG.

[0126] The analysis unit 261 includes a self-position estimation unit 271 , a sensor fusion unit 272 , and a recognition unit 273 .

[0127] The self-location estimation unit 271, the sensor fusion unit 272, and the recognition unit 273 are configured to correspond to the self-location estimation unit 71, the sensor fusion unit 72, and the recognition unit 73 in FIG. 4, respectively.

[0128] The self-position estimation unit 271 includes a SLAM processing unit 301 and an OMG storage unit 302 as functions for implementing the automatic parking assistance process.

[0129] The SLAM (Simultaneous Localization and Mapping) processing unit 301 simultaneously performs self-location estimation and creates a surrounding map to realize an automatic parking assistance function. Specifically, the SLAM processing unit 301 creates a self-location estimation and a three-dimensional map of the surroundings as, for example, an occupancy grid map (OGM) based on information (hereinafter simply referred to as integrated information) obtained by integrating (fusing or combining) information from multiple sensors supplied by the sensor fusion unit 272, and stores the map in the OMG storage unit 302.

[0130] The OMG storage unit 302 stores the OMG created by the SLAM processing unit 301, and supplies it to the action planning unit 262 as necessary.

[0131] The recognition unit 273 includes an object detection unit 321, an object tracking unit 322, a 3D semantic segmentation processing unit 323, and a context awareness unit 324.

[0132] The object detection unit 321 detects an object by detecting, for example, the presence or absence, size, shape, position, movement, etc. of the object based on the integrated information supplied from the sensor fusion unit 272. The object tracking unit 322 tracks the object detected by the object detection unit 321.

[0133] The 3D semantic segmentation processing unit 323 executes three-dimensional semantic segmentation (3D semantic segmentation) based on the image data captured by the camera 202, the depth data (ranging result) detected by the ToF camera 203, and the detection result of the radar 204, and generates a 3D semantic segmentation result. Note that a detailed configuration of the 3D semantic segmentation processing unit 323 will be described later with reference to FIG. 7.

[0134] The context awareness unit 324 is composed of a recognizer that has undergone machine learning using, for example, a deep neural network (DNN), and recognizes a situation (e.g., a parking space) from the relationships between objects based on the results of 3D semantic segmentation.

[0135] More specifically, the context awareness unit 324 includes a parking space detection unit 324a, which detects parking spaces from the relationships between objects based on the 3D semantic segmentation results.

[0136] The parking space detection unit 324a recognizes and detects parking spaces from the mutual relationships between multiple object recognition results, based on the 3D semantic segmentation results, for example, a space surrounded by a frame such as a white line large enough to park a vehicle, a space large enough to park a vehicle even without white lines and with wheel stops, and a space large enough to park a vehicle between a vehicle and a pillar.

[0137] The behavior planning unit 262 includes a route planning unit 351, and when a parking space is detected by the recognition unit 273, plans a route from the current position of the vehicle to parking in the detected parking space. The operation control unit 263 controls the operation of the vehicle 1 to realize the action plan created by the behavior planning unit 262, for parking in the parking space recognized by the recognition unit 273.

[0138] <Example of the configuration of the 3D semantic segmentation processing unit> Next, an example configuration of the 3D semantic segmentation processing unit 323 will be described with reference to FIG.

[0139] The 3D semantic segmentation processing unit 323 includes a pre-processing unit 371, an image feature extraction unit 372, a monocular depth estimation unit 373, a 3D anchor grid generation unit 374, a dense fusion processing unit 375, a pre-processing unit 376, a point cloud feature extraction unit 377, a radar detection result feature extraction unit 378, and a type determination unit 379.

[0140] The pre-processing unit 371 performs predetermined pre-processing (such as contrast correction and edge emphasis) on the image data supplied in time series from the camera 202 , and outputs the result to an image feature extraction unit 372 and a type determination unit 379 .

[0141] The image feature extraction unit 372 extracts image features from the pre-processed image data as image features and outputs them to the monocular depth estimation unit 373, the point cloud feature extraction unit 377, and the type determination unit 379.

[0142] The monocular depth estimation unit 373 estimates monocular depth (distance measurement image) based on the image feature amount, and outputs it as dense depth data (depth data based on an image) to the dense fusion processing unit 375. The monocular depth estimation unit 373 estimates monocular depth (distance measurement image) using, for example, information on the distance from a vanishing point in one piece of two-dimensional image data to a feature point from which an image feature amount is extracted, and outputs it to the dense fusion processing unit 375 as dense depth data.

[0143] The 3D anchor grid generation unit 374 generates a 3D anchor grid in which three-dimensional anchor positions are arranged in a grid pattern based on the distance measurement results detected by the ToF camera 203, and outputs the 3D anchor grid to the dense fusion processing unit 375.

[0144] The dense fusion processing unit 375 generates dense fusion by fusing the 3D anchor grid supplied from the 3D anchor grid generation unit 374 and the dense depth supplied from the monocular depth estimation unit 373, and outputs it to the point cloud feature extraction unit 377.

[0145] The pre-processing unit 376 performs pre-processing such as noise removal on the point cloud data consisting of the distance measurement results supplied from the ToF camera 203 , and outputs the result to a point cloud feature extraction unit 377 and a type determination unit 379 .

[0146] The point cloud feature extraction unit 377 extracts point cloud features from the preprocessed point cloud data supplied from the preprocessing unit 376 based on the image features supplied from the image feature extraction unit 372 and the dense fusion supplied from the dense fusion processing unit 375, and outputs the point cloud features to the type determination unit 379.

[0147] The radar detection result feature quantity extraction unit 378 extracts radar detection result feature quantities from the detection result of the radar 204 supplied from the radar 204 , and outputs the same to a type determination unit 379 .

[0148] Based on the point cloud data (depth data) supplied from the pre-processing unit 376, the type determination unit 379 associates the depth data in each pixel of the image data supplied from the pre-processing unit 371, thereby associating the depth data in pixel units in the image data. At this time, the type determination unit 379 may generate depth data by combining the point cloud data and the radar detection result of the radar 204 (depth data (point cloud) based on the radar detection result) as necessary, and associate the depth data in pixel units in the image data. This makes it possible to complement the ToF camera 203. That is, even in a scene where the reliability of the ToF camera 203 falls below a predetermined threshold due to fog or the like, it is possible to measure the distance without decreasing the reliability below a predetermined threshold by using the radar detection result of the radar 204 using radio waves. In this case, it is desirable that the radar 204 is a so-called imaging radar having a high resolution comparable to that of the camera 202. In addition, by using the radar detection result by the radar 204, the relative speed with respect to the object can be calculated, so it becomes possible to determine whether the target object is moving or not, and by adding speed information on a pixel-by-pixel basis, it becomes possible to improve the object recognition performance. More specifically, by implementing object recognition processing using six parameters (x, y, z, vx, vy, vz) for each pixel, it becomes possible to improve the object recognition accuracy. Note that vx, vy, vz are speed information in the x, y, and z directions, respectively.

[0149] In addition, the type determination unit 379 is composed of a recognizer that performs 3D semantic segmentation processing using machine learning such as DNN (deep neural network), and performs object recognition processing on a pixel-by-pixel basis on the image data based on the image features supplied from the image feature extraction unit 372, the point cloud features supplied from the point cloud feature extraction unit 377, and the radar detection result features supplied from the radar detection result feature extraction unit 378, to identify the type (class).

[0150] That is, the type determination unit 379 associates depth data (x, y, z) in pixel units shown in the grid of image P11, for example, as shown in Fig. 8. Furthermore, the type determination unit 379 performs 3D semantic segmentation processing in pixel units based on the depth data, image feature amount, point cloud feature amount, and radar detection result feature amount to determine the type (class) (seg). Then, the type determination unit 379 sets 3D semantic segmentation information (x, y, z, seg) by associating the type determination result with the depth data in pixel units.

[0151] The type determination unit 379 generates an image consisting of 3D semantic segmentation information (x, y, z, seg) set in pixel units as a 3D semantic segmentation image. As a result, the 3D semantic segmentation image is an image in which an area for each type (class) is formed within the image.

[0152] Classes recognized as objects by object recognition processing include, for example, roadways, sidewalks, pedestrians, bicycle and motorbike riders, vehicles, trucks, buses, motorbikes, bicycles, buildings, walls, guardrails, bridges, tunnels, poles, traffic signs, traffic signals, white lines, etc. For example, a pixel classified as a vehicle as a class has pixels similarly classified as a vehicle class around it, and the area formed by these pixels forms an image that is visible as a vehicle as a whole. Therefore, areas for each class classified on a pixel-by-pixel basis are formed in the image.

[0153] Note that an example has been described in which the type determination unit 379 determines the type by performing 3D semantic segmentation processing on a pixel-by-pixel basis based on the depth data, image features, point cloud features, and radar detection result features.

[0154] However, instead of the depth data, 2D (two-dimensional) image data and image feature values ​​alone may be used to perform 2D semantic segmentation processing to determine the type on a pixel-by-pixel basis, and then the type may be associated with the depth data on a pixel-by-pixel basis to realize 3D semantic segmentation processing. Also, 3D semantic segmentation processing may be performed without using radar detection result feature values.

[0155] <Parking space detection by context awareness unit (part 1)> Next, detection of a parking space by the context awareness unit 324 will be described.

[0156] As described above, the 3D semantic segmentation information is composed of depth data and type for each pixel in the captured image data.

[0157] Therefore, the context awareness unit 324 identifies relationships between objects using the results of type determination on a pixel-by-pixel basis in an image of the surroundings based on 3D semantic segmentation information through a process called context awareness processing, and recognizes the surrounding situation from the relationships between the identified objects.

[0158] In this example, the context awareness unit 324 includes a parking space detection unit 324a, and controls the parking space detection unit 324a to perform context awareness processing based on 3D semantic segmentation information, thereby detecting parking spaces in the image based on the relationships between objects.

[0159] For example, in the case of image P31 in Figure 9, the parking space detection unit 324a recognizes the area indicated by the solid line as a parking space based on the pixel-by-pixel depth data and type, the positional relationship between the pillar Ch11 and the parked vehicle Ch12 in image P31, the size and shape of the space, etc.

[0160] Also, for example, in the case of image P32 in Figure 9, the parking space detection unit 324a recognizes the area indicated by solid lines as a parking space based on the pixel-by-pixel depth data and type, the spacing between the multiple wheel stoppers Ch21 in image P32, and the size and shape of the space.

[0161] Furthermore, for example, in the case of image P33 in Figure 9, the parking space detection unit 324a recognizes the area indicated by the solid line as a parking space based on the pixel-by-pixel depth data and type, the spacing between the foldable wheel chocks Ch31 and the white lines Ch32 in the coin parking lot in image P33, and the size of the space.

[0162] Also, for example, in the case of image P34 in Figure 9, the parking space detection unit 324a recognizes the area indicated by the solid line as a parking space based on the pixel-by-pixel depth data and type, the positional relationship between the vehicle Ch41 and the parking pillar Ch42 in image P34, and the size and shape of the space.

[0163] In this way, the context awareness unit 324 (the parking space detection unit 324a) functions as a recognizer that has undergone machine learning such as DNN (deep neural network), and detects parking spaces based on the relationship between multiple recognition results based on the depth data and type (class) information contained in the 3D semantic segmentation information.

[0164] In addition, the recognizer that has undergone machine learning using a DNN to realize the context awareness unit 324 (the parking space detection unit 324a) and the recognizer that has undergone machine learning using a DNN to realize the type determination unit 379 may be different or the same.

[0165] Then, the parking space detection unit 324a of the context awareness unit 324 detects a parking space that is an empty space in which no vehicle is parked, among the areas detected as parking spaces described with reference to Figure 9, as a target parking space that serves as a parking target for the parking assistance function.

[0166] <Parking space detection by context awareness unit (part 2)> Although a method for detecting a parking space in a parking lot has been described above, an example of detecting a parking space on a road rather than in a parking lot will be described.

[0167] For example, consider the case where an image P31 as shown in FIG.

[0168] Image P31 is an image captured from vehicle C51 in the overhead view shown in the right part of Fig. 10, with the upper part of the figure facing forward. In image P51, vehicles C61 to C64 are parallel parked on the left side of the road.

[0169] 3D semantic segmentation information consisting of depth data and class is set for each pixel in image P31. Therefore, as shown in the right part of the figure, the parking space detection unit 324a of the context awareness unit 324 can recognize the distance from the vehicle C51 to the right rear of each of the vehicles C61 and C62 in image P31, and can recognize the presence of the parking space SP51 from the difference in distance.

[0170] More specifically, for example, the parking space detection unit 324a of the context awareness unit 324 can specify the size of the rectangular vacant space indicated by the dotted line in the figure based on the distance DL between the right front end of the vehicle C62 and the white line, the distance to the right front end of the vehicle C62, the width DS of the visible right rear end of the vehicle C61, and the distance to the right rear end of the vehicle C61. Specifically, the size of the vacant space is specified using both a parking space recognition process using a feature extractor learned offline based on 3D semantic segmentation information and a parking space recognition process using distance information such as the distance DL and the width DS.

[0171] Therefore, when the parking space detection unit 324a of the context awareness unit 324 can recognize from the size of the identified vacant space that the vehicle C51 can be parked, it recognizes the rectangular vacant space indicated by the dotted line in the figure as the parking space SP51.

[0172] In this way, based on an image such as that shown in image P31 of FIG. 10 and the corresponding 3D semantic segmentation information, the parking space detection unit 324a of the context awareness unit 324 is able to recognize the empty space between vehicles C61, C62 as a parking space.

[0173] As a result, using only the two-dimensional image P31, it is difficult to recognize an empty space such as parking space SP51 approximately 15 to 30 m ahead; however, by setting 3D semantic segmentation information, it becomes possible to properly detect parking space SP51 even in situations where only the edge portion of the right rear of the vehicle body is visible.

[0174] <3D Semantic Segmentation Image Generation Processing> Next, a 3D semantic segmentation image generation process for generating a 3D semantic segmentation image by the 3D semantic segmentation processing unit 323 will be described with reference to the flowchart in FIG.

[0175] In step S 11 , the preprocessing unit 371 acquires captured image data supplied from the camera 202 .

[0176] In step S12, the 3D anchor grid generation unit 374 and the pre-processing unit 376 acquire depth data (ranging results) sensed by the ToF camera 203.

[0177] In step S13, the radar detection result feature amount extraction unit 378 acquires the detection result of the radar 204 as the radar detection result.

[0178] In step S14, the pre-processing unit 371 performs predetermined pre-processing (such as contrast correction and edge emphasis) on the image data supplied from the camera 202, and outputs the result to the image feature amount extraction unit 372 and the type determination unit 379.

[0179] In step S15 , the image feature extraction unit 372 extracts image features from the preprocessed image data as image features and outputs them to the monocular depth estimation unit 373 , the point cloud feature extraction unit 377 , and the type determination unit 379 .

[0180] In step S16, the monocular depth estimation unit 373 estimates monocular depth (ranging image) based on the image feature amount, and outputs it to the dense fusion processing unit 375 as dense depth (depth data based on the image).

[0181] In step S17, the 3D anchor grid generation unit 374 generates a 3D anchor grid by gridding three-dimensional anchor positions based on the depth data (ranging results) detected by the ToF camera 203, and outputs the 3D anchor grid to the dense fusion processing unit 375.

[0182] In step S18, the dense fusion processing unit 375 fuses the 3D anchor grid supplied from the 3D anchor grid generation unit 374 and the dense depth supplied from the monocular depth estimation unit 373 to generate dense fusion data and outputs it to the point cloud feature extraction unit 377.

[0183] In step S19, the preprocessing unit 376 performs preprocessing such as noise removal on point cloud data consisting of depth data (ranging results) supplied from the ToF camera 203, and outputs the result to the point cloud feature extraction unit 377 and the type determination unit 379.

[0184] In step S20, the point cloud feature extraction unit 377 extracts point cloud features from the preprocessed point cloud data supplied from the preprocessing unit 376 based on the image features supplied from the image feature extraction unit 372 and the dense fusion data supplied from the dense fusion processing unit 375, and outputs the point cloud features to the type determination unit 379.

[0185] In step S 21 , the radar detection result feature quantity extraction unit 378 extracts radar detection result feature quantities from the detection result of the radar 204 supplied from the radar 204 , and outputs the radar detection result feature quantities to the type determination unit 379 .

[0186] In step S22, the type determination unit 379 matches the depth data for each pixel of the image data supplied from the pre-processing unit 371 based on the point cloud data supplied from the pre-processing unit 376, thereby matching the depth data on a pixel-by-pixel basis in the image data.

[0187] In step S23, the type determination unit 379 performs object recognition processing using 3D semantic segmentation processing based on the pixel-by-pixel depth data, the image features supplied from the image feature extraction unit 372, the point cloud features supplied from the point cloud feature extraction unit 377, and the radar detection result features supplied from the radar detection result feature extraction unit 378, determines the type (class) of the image data on a pixel-by-pixel basis, and generates 3D semantic segmentation information (x, y, z, seg).

[0188] In step S24, the type determination unit 379 generates an image in which 3D semantic segmentation information (x, y, z, seg) is associated with each pixel of the captured image data as a 3D semantic segmentation image, and stores it in chronological order.

[0189] In step S25, it is determined whether or not a stopping operation has been performed, and if a stopping operation has not been performed, the process returns to step S11. That is, the processes of steps S11 to S25 are repeated until a stopping operation is performed.

[0190] Then, in step S25, if a stop operation is instructed, the process ends.

[0191] Through the above processing, 3D semantic segmentation information is set on a pixel-by-pixel basis for each piece of image data captured in time series, and the process of generating an image consisting of the 3D semantic segmentation information as a 3D semantic segmentation image is repeated and stored sequentially.

[0192] The 3D semantic segmentation images generated sequentially in time series are then used to realize the parking assistance processing described below.

[0193] In addition, the above processing allows the 3D semantic segmentation images to be stored sequentially in chronological order, independent of other processing, making it possible to use the 3D semantic segmentation images generated in chronological order in other processing.

[0194] For example, the object detection unit 321 may detect an object based on a 3D semantic segmentation image. The object tracking unit 322 may track an object using the 3D semantic segmentation image for the object detection result of the object detection unit 321. Furthermore, in the parking mode process described later, it is also possible to use the 3D semantic segmentation image around the parking space for processing to check whether or not an obstacle has been found in the parking space and the parking state has changed to one in which parking is not possible, until parking of the vehicle in the parking space is completed.

[0195] <Parking assistance processing> Next, the parking assistance process will be described with reference to the flowchart of FIG.

[0196] In step S41, the parking assist control unit 201 determines whether or not to start the parking assist process. For example, the parking assist control unit 201 may determine whether or not to start the parking assist process based on whether or not an input device such as the HMI 31 that instructs the start of the parking assist process has been operated.

[0197] Furthermore, the parking assistance control unit 201 may determine whether or not to start the parking assistance process based on whether or not information indicating the entrance to the parking lot is detected from the 3D semantic segmentation image.

[0198] If it is determined in step S41 that the start of the parking assistance process has been instructed, the process proceeds to step S42.

[0199] In step S42, the parking assistance control unit 201 controls the HMI 31 to present information indicating that the parking assistance process has started.

[0200] In step S43, the parking assistance control unit 201 sets the operation mode to the parking space search mode, and controls the HMI 31 to present that the current operation mode is the parking space search mode.

[0201] In step S44, the parking assistance control unit 201 controls the context awareness unit 324 in the recognition unit 273 of the analysis unit 261 to execute a parking space search mode process, search for a parking space, and register the parking space as a search result. The parking space search mode process will be described in detail later with reference to the flowchart of FIG.

[0202] In step S45, the parking assistance control unit 201 determines whether or not a parking space where parking is possible is registered in the context awareness unit 324. Note that the meaning of "registered" does not mean that a parking space at home or the like is registered in advance, but means that it is temporarily registered in the context awareness unit 324.

[0203] If it is determined in step S45 that an available parking space has been registered, the process proceeds to step S46.

[0204] In step S46, the parking assistance control unit 201 switches and sets the operation mode to the parking mode, and controls the HMI 31 to present that the operation mode is the parking mode.

[0205] In step S47, the parking assistance control unit 201 controls the action planning unit 262 to execute the parking mode process and park the vehicle. The parking mode process will be described in detail later with reference to the flowchart of FIG.

[0206] In step S48, the parking assist control unit 201 determines whether the parking assist process is completed. More specifically, the parking assist control unit 201 determines whether the parking assist process is completed by the parking mode process or by a notification that parking is not possible due to the absence of a parking space.

[0207] If it is determined in step S48 that the parking assistance process has ended, the process proceeds to step S49.

[0208] In step S49, the parking assist control unit 201 controls the HMI 31 to present that the parking assist process has ended.

[0209] In step S50, the parking assistance control unit 201 determines whether or not a stopping operation has been performed to stop the operation of the vehicle 1, and if it is determined that a stopping operation of the vehicle 1 has not been performed, the processing returns to step S41.

[0210] On the other hand, if it is determined in step S45 that no available parking space is registered, the process proceeds to step S51.

[0211] In step S51, the parking assistance control unit 201 controls the HMI 31 to present that there is no parking space, and the process proceeds to step S48.

[0212] Also, in step S41, if the start of the parking assistance process has not been instructed, the process proceeds to step S50.

[0213] That is, if the start of the parking assistance process is not instructed and the vehicle 1 is not stopped, the processes of steps S41 and S50 are repeated.

[0214] Then, in step S50, if the vehicle 1 has been stopped, the process ends.

[0215] Through the above processing, the parking space search mode searches for a parking space based on 3D semantic segmentation information, and if a parking space is searched for and registered, the vehicle 1 is parked in the parking space registered as the search result.

[0216] Also, if a parking space is not searched for and no available parking space is registered, a message is displayed indicating that there is no parking space and parking is not possible.

[0217] With the above-mentioned processing, the parking space search mode processing is performed first, and then the parking mode processing is performed. As a result, the vehicle first finds a parking space in the vicinity and then performs parking operations, just like when a human parks the vehicle, making it possible to realize quick and smooth parking assistance.

[0218] <Parking space search mode processing> Next, the parking space search mode process will be described with reference to the flowchart of FIG.

[0219] In step S71, the context awareness unit 324 acquires a 3D semantic segmentation image registered in the 3D semantic segmentation processing unit 323.

[0220] In step S72, the context awareness unit 324 controls the parking space detection unit 324a to search for a parking space based on the 3D semantic segmentation image. Here, the parking space search process by the parking space detection unit 324a of the context awareness unit 324 is, for example, the context awareness process described with reference to Fig. 9 and Fig. 10.

[0221] In step S73, the context awareness unit 324 registers (updates) the searched parking space and its position information as a result of the parking space search by the parking space detection unit 324a based on the 3D semantic segmentation image.

[0222] Through the above process, parking spaces are searched for by context awareness processing based on the 3D semantic segmentation image, and are sequentially registered in the parking space detection unit 324a.

[0223] Furthermore, since the same process will be repeated as long as the parking space search mode continues, if a parking space that was once searched for as available for parking becomes no longer available for parking due, for example, to the appearance of an obstacle or the parking of another vehicle, the registered information for the parking space will be deleted and updated.

[0224] Similarly, even if a space has not been searched for as a possible parking space, if a vehicle drives away from the parking space during the parking space search mode and the space is searched for as a new parking space, the space will be newly registered.

[0225] Furthermore, in the case where multiple parking spaces are searched for, the position information of the multiple parking spaces is registered.

[0226] <Parking mode processing> Next, the parking mode process will be described with reference to the flowchart of FIG.

[0227] In step S91, the behavior planning unit 262 reads out the position information of the parking space registered in the parking space detection unit 324a. When the position information of multiple parking spaces is registered, the position information of the multiple parking spaces is read out.

[0228] In step S92, the action planning unit 262 sets the parking space that is the shortest distance from the vehicle's position as the destination parking space from among the parking space position information that has been read out. At this time, there may be a step of asking the user to confirm whether it is OK to set the parking space that is the shortest distance as the destination parking space. Alternatively, multiple parking spaces may be displayed on a display disposed in the vehicle cabin, and the user may select the destination parking space from the multiple parking spaces.

[0229] In step S93, the behavior planning unit 262 controls the route planning unit 351 to plan a route to the destination parking space as a parking route.

[0230] In step S94, the behavior planning unit 262 controls the HMI 31 to present the parking route to the destination parking space, which has been planned by the route planning unit 351.

[0231] In step S95, the behavior planning unit 262 controls the operation control unit 263 to operate the vehicle 1 along the parking path.

[0232] In step S96, the action planning unit 262 determines whether or not parking is complete. If it is determined in step S96 that parking is not complete, the process proceeds to step S97.

[0233] In step S97, the behavior planning unit 262 determines whether the destination parking space is in a state where parking is not possible, for example, based on the latest 3D semantic segmentation image. That is, it is determined whether parking is not possible due to an obstacle being found in the destination parking space while moving along the parking path, or another vehicle entering the destination parking space while moving along the parking path.

[0234] In step S96, if the destination parking space is not unparkable, the process returns to step S94. That is, the process of steps S94 to S97 is repeated until parking is completed, unless the destination parking space is unparkable, by performing an operation along the parking path.

[0235] If it is determined in step S97 that parking is no longer possible in the destination parking space, the process proceeds to step S98.

[0236] In step S98, the behavior planning unit 262 determines whether or not the read parking space position information includes position information of other parking spaces.

[0237] If it is determined in step S98 that the read parking space position information includes position information of other parking spaces, the process returns to step S92.

[0238] That is, the new parking space, the destination parking space, and the parking route are reset, and the processes of steps S94 to S97 are repeated.

[0239] Then, if it is determined in step S96 that parking is completed, the process proceeds to step S100.

[0240] In step S100, the action planning unit 262 controls the HMI 31 to display an image notifying the completion of parking.

[0241] Also, in step S98, if there are no other parking spaces, the process proceeds to step S99.

[0242] In step S99, the behavior planning unit 262 controls the HMI 31 to present an image notifying that a parking space cannot be found and parking is not possible.

[0243] Through the above processing, the parking space with the closest location information to the vehicle among the location information registered as parking spaces is set as the destination parking space, a parking path is planned, and operations are performed along the parking path, thereby realizing automatic parking.

[0244] In this case, if an obstacle is detected in the destination parking space while parking the vehicle toward the destination parking space, or if parking becomes impossible due to another vehicle parking there, and if the position information of other parking spaces is registered, the parking space closest to the vehicle among the other parking spaces will be reset as the destination parking space and parking will be performed there.

[0245] Through the above series of processes, depth data (three-dimensional position) and object type (class) are set for each pixel in the image based on the image captured by the camera 202, the depth data (ranging results) detected by the ToF camera 203, and the detection results of the radar 204, and a 3D semantic segmentation image is generated.

[0246] In addition, parking spaces are detected and their location information is registered using context awareness processing based on 3D semantic segmentation images.

[0247] Then, based on the position information of the registered parking space, a parking route is set and the operation for parking is controlled.

[0248] This allows the type (class) to be determined on a pixel-by-pixel basis by combining image features with point cloud features and radar detection result features, making it possible to achieve more accurate type determination.

[0249] In addition, because the parking space can be specified by the context awareness processing using the 3D semantic segmentation image with high accuracy type determination, it becomes possible to appropriately search for various parking spaces regardless of the environment of the parking space. In addition, since the parking space can be set using the image captured by the vehicle camera, the parking path can be set after the parking space is set, so it becomes possible to set the parking path without passing near the parking space.

[0250] As a result, it is possible to identify a parking space based on information in the image captured by the camera, then set a parking path and park the vehicle, making it possible to achieve quick and smooth parking assistance similar to that achieved when a human parks the vehicle.

[0251] <<4. Software execution example>> The above-mentioned series of processes can be executed by hardware, but can also be executed by software. When the series of processes are executed by software, the programs constituting the software are installed from a recording medium into a computer built into dedicated hardware, or into, for example, a general-purpose computer capable of executing various functions by installing various programs.

[0252] 15 shows an example of the configuration of a general-purpose computer. This personal computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0253] An input unit 1006 consisting of input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 for outputting a processing operation screen and an image of the processing result to a display device, a storage unit 1008 consisting of a hard disk drive for storing programs and various data, and a communication unit 1009 consisting of a LAN (Local Area Network) adapter and the like for executing communication processing via a network such as the Internet are connected to the input / output interface 1005. Also connected is a drive 1010 for reading and writing data from and to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0254] The CPU 1001 executes various processes according to a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory and installed in a storage unit 1008, and loaded from the storage unit 1008 to a RAM 1003. The RAM 1003 also stores data and the like necessary for the CPU 1001 to execute various processes, as appropriate.

[0255] In a computer configured as described above, the CPU 1001 loads a program stored in the memory unit 1008, for example, into the RAM 1003 via the input / output interface 1005 and the bus 1004, and executes the program, thereby performing the series of processes described above.

[0256] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0257] In the computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by mounting the removable storage medium 1011 in the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be pre-installed in the ROM 1002 or the storage unit 1008.

[0258] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0259] 15. The CPU 1001 in FIG. 15 realizes the function of the parking assistance control unit 201 in FIG.

[0260] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0261] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present disclosure.

[0262] For example, the present disclosure can take the form of cloud computing in which a single function is shared and processed collaboratively by multiple devices via a network.

[0263] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0264] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0265] The present disclosure can also be configured as follows.

[0266] <1> an image acquisition unit that acquires image data of an area within a predetermined distance from the vehicle, the image data being captured by a camera; a depth data acquisition unit that acquires depth data of a range that at least partially overlaps with an imaging range of the image data; A 3D semantic segmentation processing unit that generates a 3D semantic segmentation image in which class information and depth data are set for each pixel of the image data; a parking space detection unit that detects a parking space based on the 3D semantic segmentation image; An information processing device comprising: <2> The class information is information classified by a semantic segmentation process based on the image data. <1> The information processing device described in <3> The 3D semantic segmentation processing unit generates the pixel-by-pixel depth data based on the image data and the depth data, associates the depth data with each pixel of a 2D semantic segmentation image generated based on the image data, and generates a 3D semantic segmentation image. <1> The information processing device described in <4> The parking space detection unit further determines whether the detected parking space is available for parking. <1> The information processing device described in <5> The parking space detection unit further determines whether the detected parking space is suitable for parking the vehicle based on the size of the vehicle. <4> The information processing device described in <6> The parking space detection unit further determines whether the detected parking space is suitable for parking based on a distance between a plurality of objects present within an imaging range of the image data. <4> The information processing device described in <7> The 3D semantic segmentation processing unit performs 3D semantic segmentation processing based on time-series image data within a range of a predetermined distance from the vehicle and time-series depth data within a range that at least partially overlaps with an imaging range of the image data; The parking space detection unit estimates a movement of an object present in an imaging range of the image data based on the generated time-series 3D semantic segmentation images, and updates information regarding available parking spaces where the vehicle can be parked based on the estimated movement of the object present in the imaging range of the image data. <1> The information processing device described in <8> The 3D semantic segmentation processing unit generates the depth data in pixel units based on the image data and the depth data, associates the depth data in pixel units with the image data, executes the 3D semantic segmentation processing on the image data, and sets class information in pixel units to generate the 3D semantic segmentation image consisting of a plurality of regions. <7> The information processing device described in <9> The depth data is generated using an ultrasonic sensor, a LiDAR, an optical ranging sensor, a stereo camera, a monocular camera, or an infrared camera. <1> The information processing device described in <10> The vehicle further includes a radar for acquiring speed information of an object moving within a predetermined distance range from the vehicle. <1> The information processing device described in <11> The parking space search unit further includes: <4> The information processing device described in <12> When the number of available parking spaces registered in the parking space search unit is multiple, the parking control unit sets a nearest available parking space among the multiple available parking spaces as a target parking space, plans a route to the target parking space, and controls an operation of the vehicle related to parking along the planned route. <11> The information processing device described in <13> The parking control unit further includes: setting a parking space available for parking selected by a user from among the one or more parking spaces available for parking registered in the parking space search unit as a target parking space; planning a route to the target parking space; and controlling an operation of the vehicle related to parking along the planned route. <11> The information processing device described in <14> performing an image acquisition process for acquiring image data captured by a camera within a predetermined distance from the vehicle; performing a depth data acquisition process for acquiring depth data of a range that at least partially overlaps with an imaging range of the image data; A 3D semantic segmentation process is performed to generate a 3D semantic segmentation image in which class information and depth data are set for each pixel of the image data; A parking space detection process is performed to detect a parking space based on the 3D semantic segmentation image. An information processing method comprising: <15> an image acquisition unit that acquires image data of an area within a predetermined distance from the vehicle, the image data being captured by a camera; a depth data acquisition unit that acquires depth data of a range that at least partially overlaps with an imaging range of the image data; A 3D semantic segmentation processing unit that generates a 3D semantic segmentation image in which class information and depth data are set for each pixel of the image data; a parking space detection unit that detects a parking space based on the 3D semantic segmentation image; programs that make a computer function. [Explanation of symbols]

[0267] 201 Parking assistance control unit, 202, 202-1 to 202-q, 203, 203-1, 203-r ToF camera, 204, 204-1 to 204-s, 261 Analysis unit, 262 Action planning unit, 271 Self-position estimation unit, 272 Sensor fusion unit, 273 Recognition unit, 301 SLAM processing unit, 302 OGM memory unit, 321 Object detection unit, 322 Object tracking unit, 323 3D semantic segmentation processing unit, 324 Context awareness unit, 324a Parking space detection unit, 351 Path planning unit, 371 Preprocessing unit, 372 Image feature extraction unit, 373 Monocular depth estimation unit, 374 3D anchor grid generation unit, 375 Dense fusion processing unit, 376 Preprocessing section, 377 Point cloud feature extraction section, 378 Radar detection result feature extraction section, 379 Type determination section

Claims

1. an image acquisition unit that acquires image data of an area within a predetermined distance from the vehicle, the image data being captured by a camera; a depth data acquisition unit that acquires depth data of a range that at least partially overlaps with an imaging range of the image data; A 3D semantic segmentation processing unit that generates a 3D semantic segmentation image in which class information and depth data are set for each pixel of the image data; a parking space detection unit that detects a parking space based on the 3D semantic segmentation image; An information processing device comprising:

2. The class information is information classified by a semantic segmentation process based on the image data. The information processing device according to claim 1 .

3. The 3D semantic segmentation processing unit generates the pixel-by-pixel depth data based on the image data and the depth data, associates the depth data with each pixel of a 2D semantic segmentation image generated based on the image data, and generates a 3D semantic segmentation image. The information processing device according to claim 1 .

4. The parking space detection unit further determines whether the detected parking space is available for parking. The information processing device according to claim 1 .

5. The parking space detection unit further determines whether the detected parking space is suitable for parking the vehicle based on the size of the vehicle. The information processing device according to claim 4.

6. The parking space detection unit further determines whether the detected parking space is suitable for parking based on a distance between a plurality of objects present within an imaging range of the image data. The information processing device according to claim 4.

7. The 3D semantic segmentation processing unit performs 3D semantic segmentation processing based on time-series image data within a range of a predetermined distance from the vehicle and time-series depth data within a range that at least partially overlaps with an imaging range of the image data, The parking space detection unit estimates a movement of an object present in an imaging range of the image data based on the generated time-series 3D semantic segmentation image, and updates information regarding available parking spaces where the vehicle can be parked based on the estimated movement of the object present in the imaging range of the image data. The information processing device according to claim 1 .

8. The 3D semantic segmentation processing unit generates the depth data in pixel units based on the image data and the depth data, associates the depth data in pixel units with the image data, executes the 3D semantic segmentation process on the image data, and sets class information in pixel units to generate the 3D semantic segmentation image consisting of a plurality of regions. The information processing device according to claim 7.

9. The depth data is generated using an ultrasonic sensor, a LiDAR, an optical ranging sensor, a stereo camera, a monocular camera, or an infrared camera. The information processing device according to claim 1 .

10. The vehicle further includes a radar for acquiring speed information of an object moving within a predetermined distance range from the vehicle. The information processing device according to claim 1 .

11. The parking space search unit further includes: The information processing device according to claim 4.

12. When the number of available parking spaces registered in the parking space search unit is multiple, the parking control unit sets a nearest available parking space among the multiple available parking spaces as a target parking space, plans a route to the target parking space, and controls an operation of the vehicle related to parking along the planned route. The information processing device according to claim 11.

13. The vehicle further includes a parking control unit that sets a parking space selected by a user from among one or more available parking spaces registered in the parking space search unit as a target parking space, plans a route to the target parking space, and controls an operation of the vehicle related to parking along the planned route. The information processing device according to claim 11.

14. performing an image acquisition process for acquiring image data captured by a camera within a predetermined distance from the vehicle; performing a depth data acquisition process for acquiring depth data of a range that at least partially overlaps with an imaging range of the image data; A 3D semantic segmentation process is performed to generate a 3D semantic segmentation image in which class information and depth data are set for each pixel of the image data; A parking space detection process for detecting a parking space based on the 3D semantic segmentation image; An information processing method comprising:

15. an image acquisition unit that acquires image data of an area within a predetermined distance from the vehicle, the image data being captured by a camera; a depth data acquisition unit that acquires depth data of a range that at least partially overlaps with an imaging range of the image data; A 3D semantic segmentation processing unit that generates a 3D semantic segmentation image in which class information and depth data are set for each pixel of the image data; a parking space detection unit that detects a parking space based on the 3D semantic segmentation image; programs that make a computer function.

Images